Trailing 2-1 from the first leg, Slaven Bilic's side won 4-2 on aggregate at London Stadium courtesy of Senegalese midfielder Kouyate's first-half double and Algerian winger Sofiane Feghouli's late strike.
Elsewhere, Portugal's Euro 2016 final hero Eder was brought down to earth with a bump as his side Lille were eliminated by Qabala of Azerbaijan.
Formerly known as the Olympic Stadium, West Ham's new 60 000-seater ground hosted the athletics events at the 2012 Olympic Games in London.
It has been converted to a football stadium at a cost of 272 million pounds ($362.7 million, 325.8 million euros), partly funded by the British taxpayer.
Thursday's game was West Ham's first as hosts since leaving Upton Park – their home for 112 years – at the end of last season.
They are due to officially open the stadium in a glamour friendly against Italian champions Juventus on Sunday.
With the stadium due to continue being used as an athletics venue, a retractable bottom tier of seating will be moved into place for West Ham's matches and green fabric placed over the running track.
West Ham, missing the rested Dimitri Payet, took an eighth-minute lead when Kouyate flicked home left-back Sam Byram's low cross from close range.
It made the Senegal international the first scorer at the ground and he added another in the 25th minute, toeing the ball in from Andy Carroll's knock-down to register his first career brace in his 321st match.
This is dummy text. It is not meant to be read. Accordingly, it is difficult to figure out when to end it. But then, this is dummy text. It is not meant to be read. Period.
ConversionConversion EmoticonEmoticon