After trailing 1-0 to a 17th-minute goal from Kasey McAteer, which arrived just a minute or so after the hosts saw a 'goal' from Jordan Hugill controversially ruled out, the Millers produced an excellent response and did not feel sorry for themselves.
They deservedly levelled when Fred Onyedinma glanced home a header from Andre Green's cross and threatened to turn over their high-flying visitors and end their 100 per cent record.
But City regrouped and netted a late winner when a mistake from Cohen Bramall was seized upon by McAteer, who sealed a day to remember when he struck for a second time - his first two senior goals for the Foxes.
Taylor, whose side have picked up just point from the club's opening four Championship games, said: "We are in a little bit of pain right now because apart from the first 45 minutes v Stoke, we have competed in every game and showed signs of being a really team, but not picked up the points we feel we have deserved.
"Moments have gone against us. We have played a big part in that and even today, I think that was on show. We got a little bit weaker as the game went on and we needed to see out a couple of defensive moments better than we did.
"But we more than matched one of the best teams in the league.
"We just needed a little bit more off the pitch in terms of the bench and on the pitch as the game went on."
"Football is a funny game. All of a sudden if you (keep) playing well and losing, the confidence factor is one you cannot ignore.
"Especially as that group are quite young and naive at times. People talk about the levels of opposition, that's irrelevant. We have to be where we need to be, which is a front foot team."
Taylor revealed that he spoke to referee Simon Hooper to clear up another moment of controversy for the Millers when Hugill's goal was ruled out.
A vicious inswinging corner from Cafu grazed the head of Cameron Humphreys and the ball flew towards the far corner of the net with Hugill lurking at the far post.
The Millers striker wheeled away to celebrate, after getting a faint flick, before Hooper, in consultation with an assistant referee, spoiled the hosts' fun and chalked off the goal.
The initial thought was that it was down to a perceived handball or offside. But replays did seem to indicate that Dexter Lembikisa had fouled Mads Hermansen in the build-up.
Taylor continued: "It was disallowed for two reasons, offside and a foul on the goalkeeper.
"Both were marginal. If that's a foul, then a lot is going to get given from set-piece moments.
"The irony is that (if) we've affected the goalkeeper, we've had a free header from four yards out and it's gone in off the post.
"Players have jobs on set-pieces and in fairness to our player, he has probably done his job.
"You've seen if there was a foul and an offside, the pain for us is we were celebrating for a long period of time and their communications weren't working.
"Hence why it took a little bit longer to make that decision. All of a sudden, the feel of the whole place and on the pitch is happening again."