The presenter, 65, left the castle camp to go to the hospital on Wednesday night after becoming ill.
And he's now had to quit the ITV programme because leaving Gwrych Castle meant he had "broken the COVID bubble".
The star said he was "gutted" to be out of the competition so soon.
Read more: All the I'm A Celebrity winners so far
He said in a message to viewers: "Richard here – firstly just to say that I’m absolutely FINE.
"I started to feel briefly unwell in the small hours of the morning and was taken to hospital as a precaution.
"By leaving the camp, I had consequently broken the COVID ‘bubble’ and as such I’ve had to leave the castle and all the wonderful celebs that remain in the camp.
"Obviously I’m gutted to be leaving so soon but the safety of all the campmates is the number one priority."
Madeley said he was "incredibly excited" to watch his fellow campers continue their journey, joking that he was even more excited "at the prospect of being somewhere a little warmer than Gwrych Castle".
"Thank you to everyone who supported me on my brief but completely unforgettable adventure – I’ve made some great friends and honestly had the time of my life," he said.
Read more: Meet the I'm A Celebrity 2021 campmates
A spokesperson for the ITV show said: "Our strict COVID protocols means Richard has sadly had to leave the show as he is no longer in the bubble the celebrities form when entering the castle.
"He has been a truly brilliant camp mate and we thank him for being part of this series."
During Wednesday's episode, viewers had watched Madeley take on the Kitchen Nightmares trial, in which he had to crawl through rotten fruit and critters as he searched for stars to exchange for meals for his co-stars.
He was disappointed to only win four of the 10 stars available, although his campmates assured him they were pleased with how he'd done.
Watch: Who is I'm A Celeb's Richard Madeley?