It looked like the top seed would make quick work of Rogers in the rain-delayed encounter after romping through the first set in 26 minutes.
But world number 116 Rogers kept her composure and got the better of Williams as the first meeting between the two big hitting Americans turned into a slugfest.
Rogers sealed the win when Williams sent a backhand long on match point.
Next up for Rogers will be Switzerland’s Jil Teichman.
Williams was hoping the tournament would help her to get into top form for the U.S. Open, which starts on Aug. 31.
But having been stretched to three sets in all three of her matches this week, the 38-year-old American knows she has plenty of work to do if she is to fulfil her dream of winning that elusive 24th Grand Slam title at Flushing Meadows.
“I could play a lot better, I think that’s the main thing I can take from this match,” Williams said.
In the other quarter-final of the day, 16-year-old Coco Gauff battled back to defeat Tunisian Ons Jabeur 4-6 6-4 6-1 and will next face fellow American Jennifer Brady.