Dexter Lembikisa opened the scoring with a thunderous strike before Jordan Hugill haunted his old club by adding a second.
Norwich pulled one back through Christian Fassnacht but the Millers survived waves of second-half attacks to clinch a 2-1 victory.
The Canaries had previously picked up three wins and one draw in the current Championship campaign.
Rotherham showed early on they had no desire to sit back, with a first league win evidently the target.
They gave the visitors a very early scare when Ollie Rathbone released Andre Green, but Angus Gunn was quick to race out of his box and challenge.
Norwich were not dominant but did advance quickly when in possession. Dimitris Giannoulis surged down the left flank at speed before crossing for Gabriel Sara, who fired off target.
The best opportunities of the opening exchanges fell to Rotherham and Fred Onyedinma was unlucky to see Gunn tip his well-struck effort on to the post.
It was the Millers who eventually broke the deadlock, a just reward for their bold approach.
The ball fell kindly to Lembikisa on the edge of the box, who unleashed a powerful drive that rendered Gunn helpless.
Norwich attempted to rally as the first-half progressed but lacked cutting edge, as Ashley Barnes and Adam Idah shots were kept out relatively comfortably.
Rotherham, on the other hand, showed a clinical edge in the final third.
Hugill, a former Norwich forward, latched on to a Cafu cross before steering a header home to double the advantage.
Norwich flew out of the traps in the second half, reducing the deficit just six minutes after the restart.
Jonathan Rowe danced through defenders before finding Fassnacht, who stabbed home.
Momentum was then firmly behind the Canaries, although their desire to attack in numbers left them looking vulnerable on the counter-attack.
Adam Idah twisted and turned his way into space inside the box, but his drilled effort was held by Johansson.
A wicked deflection on a Stacey effort came close to wrong-footing Rotherham’s goalkeeper, but he displayed impressive reflexes to react.
Rathbone was a driving force for Rotherham throughout and the midfielder’s attacking intent opened up an excellent opportunity for Green.
He surged into the box with some deft footwork before his blocked cross fell into Green’s path. The winger’s shot was well-struck but whistled agonisingly wide of the post.
Rotherham found themselves pinned back in the latter stages but survived the Norwich rally, sending their fans home having seen a league win for the first time this term.