It's hard to make a general rule on this, it really depends on the position.
You speak of fear of them having a Queen on the board when you do not, but you have two Rooks, and they have none.
It's usually a pretty even game and hard to win, I would say that a very open game favours the Queen because she has more meneuverability but if the game is very restrictive to where anyone can move and your Rooks are in a battery up the only active file then the Queen is dead meat.
As soon as you let your Rooks stray from the protection of each other you are weakening your position, and potentially leaving yourself open. You must also keep your King very safe from the Queen otherwise the chances of a draw by perpetual/repeated position are very high.
I know without a specific game to analyze, it's often hard to state hard-and-fast whether a specific exchange is beneficial or not. Still, I've often come up against the situation where my opponent's in-line rooks are attacked by my queen and another piece (say, a bishop on the diagonal behind my queen). Thus, I can take the first rook only by saccing my queen first and then recapturing with the trailing bishop. Is this a good exchange?
My logic dictates that being ahead material is a good thing and that giving up 9 points to gain (10-12, depending on what scoring system you believe in) is worth it. On the other hand, there's a little voice inside me that is horrified by the thought of my opponent still having a queen on the board when I don't. Some of this might stem from confidence in playing my own rooks, but I'd like to hear from the experience of the masses.
In the same situation (an open middlegame that's fairly even and a few minor pieces are gone), would you give up your queen to gain two rooks?