QPR won 4-1 in the clubs’ last meeting, but it was the Robins who took the initiative in this entertaining encounter.
Grant McCann gave Cheltenham the lead after just two minutes with a fantastic long range effort that gave Chris Day no chance at all.
Graham Fyfe was handed his debut at left wing-back and the Scotsman enjoyed a productive start to his Cheltenham career after making the move down south.
Ian Holloway’s side equalised on 15 minutes after Steve Book parried a Paul Furlong free-kick and Gareth Ainsworth, signed from Cardiff in the summer, was on hand to stab the ball past Book, who will be disappointed that none of his defenders reacted more quickly.
Furlong was then sent off after receiving two yellow cards, the first for delaying the taking of the free-kick and the second for a late lunge on Michael Duff.
Gould introduced Bertrand Cozic at the start of the second half in place of Dave Bird and the Frenchman was soon making his mark in midfield as he attempts to earn an extended contract. Cozic looked a cut above at times and will have given his manager plenty of food for thought.
Damian Spencer was denied by Day after outpacing the Rangers defence and Day also blocked a shot from the impressive Cozic as Cheltenham pressed for a second.
But it was 10-man QPR however that stole a late winner with another stunning goal at the same end as McCann’s spectacular opener.
Richard Langley had looked dangerous throughout and he fired home a 25-yard rocket that consigned Cheltenham to their fourth first round exit in their five years in the League Cup.