I don't believe you can generalize. However it's less important to castle when the queens are off the board.
When Queens are traded really early in the game, is it better to not Castle?

I've been cut to pieces when I didn't castle, even with the queens off the board early on, even as White. Typically the two bishops come in from different angles while my king is stuck in the middle, maybe hiding behind a single center pawn at d3 or e3, while I try desperately to bring my rooks into play. It's no fun. Therefore I often still find it necessary to castle, though not always. Circumstances dictate, mostly depending on how many pieces are left on the board: in general, the fewer pieces, the safer it is to ignore castling.
Having said that, if I know the opening well and/or if I'm in a position similar to the openings I know well, especially where I can hide behind a bishop pawn, like stepping behind a pawn at f6 as Black via ...Kf7, or where I will have to worry about being checked by only one rook at a time, whereupon I can trade rooks even though temporarily blocking one rook in, then I know I'm safe and I don't worry about castling. In fact, when playing against a computer in those situations, the computer still works to castle even when it's not necessary, which puts the program at a disadvantage since it puts its king near the corner instead of centralizing it for the upcoming endgame. In such cases it's actually better not to castle!
P.S.--Here's an example of what I mean...

Yeah, I'm with Fiveofwords. I almost always still try to castle after a Queen trade. I just wondered if anyone else has ever had success hurrying into an endgame without Castling.

As as ways it depends. If only the queens and maybe a few pawns have been exchanged it is generally still a good idea to castle.
It is less important if the queens are gone, but it isn't really an "end game" where you want to "activate the king" if most or all of the minor pieces and rooks are still on the board. You still need "king safety" and castling is a good way to get that.
Now if you king is relatively safe behind a closed center and you have a way to quickly trade off a bunch of other pieces, skipping castling might be a useful way to save a tempo. But otherwise, no get the king safe.

It is tricky to generalize this question, but I'd say if you're playing a blitz game, and you are unfamiliar with the opening, castle to be on the safe side. On the other hand, I think we learn more by taking risks in chess (and in life!)

In other words, is it best to just start moving your King to the center of the board and let your Rooks connect that way instead?
On the rare occasion when a game of mine has had an early queen trade, I havent castled. It has turned out well, but you cant just make a random generalization. A lot will depend on the position.
In other words, is it best to just start moving your King to the center of the board and let your Rooks connect that way instead?