The north African side took a shock lead when Sofiane Feghouli scored from the spot after being fouled by Tottenham defender Jan Vertonghen.
Manchester United's Fellaini, a second-half substitute, equalised with a looping header shortly after coming on.
And Mertens scored the winner when he finished off a swift counter-attack.
The 80th-minute strike prompted an eruption of relief from the Belgium bench, not least from coach Marc Wilmots, who had spent most of the match looking understandably pensive.
His side had 65% possession but for most of the game found it difficult to penetrate an Algeria defence marshalled by former Rangers centre-half Madjid Bougherra.
The European side, the seeded team in the group and widely tipped to do well in Brazil after an impressive qualifying campaign, fielded an imposing starting XI, with the Chelsea pair of Eden Hazard and Romelu Lukaku in attack, while Spurs' Mousa Dembele and former Chelsea man Kevin de Bruyne sat in midfield.
Pre-tournament fifth favourites with some bookmakers, they looked anything but for 69 minutes.
They were rocked in the 24th minute when forward Feghouli was hauled down by Vertonghen in the area as he aimed to convert Faouzi Ghoulam's cross.
The Valencia player recovered to sidefoot a tame penalty past another Blues player Thibaut Courtois, who dived the wrong way. That was Algeria's first World Cup goal since 1986 - 506 minutes of football.
Belgium's best early chances were from long distance and fell to Zenit St Petersburg's Axel Witsel, who had two efforts saved before failing to convert a header.
The pattern changed with the Belgium substitutions. First to test the goal was 19-year-old Lille striker Divock Origi, who forced a brilliant low save from Rais Mbolhi.
Gaps began to grow in a tiring Algeria defence, and a fresh Fellaini took advantage of that when he outmuscled marker Carl Medjani and flicked his effort beyond the reach of Mbolhi.
The winner came from a move that showed off Belgium's attack at its best.
De Bruyne tackled Feghouli in his own half and when the ball found its way to Hazard on the left, the PFA young player of the year sped down the wing and fed Mertens, who slammed his shot high into the net.
Belgium almost grabbed a third through Fellaini, but Mbolhi produced another good save from point-blank range.
Belgium coach Marc Wilmots:
"It was a tough match but we were patient throughout and at the end we were rewarded.
"We stayed calm after doing a small mistake in the first half. We got a new momentum. We saw that Algeria were getting more and more tired. At half-time people were a bit down, but I said 'don't worry, we will score and we will change totally the match'."
Algeria coach Vahid Halilhodzic:
"It's a major disappointment. I would have liked to have a couple of tougher and more resilient players to resist their attacks but I don't have much of a choice.
"We missed a great opportunity here. But in the second half it was a bit tough. We left too much space to the Belgian team and the two goals scored against us were to be expected."
