The only way to permanently lose the right to castle would be having to capture with the king. If you are recapturing something that took a pawn the rook can now get past the king anyway and so you easily castle by hand.
If it was a trade of queens (so your king had to capture the enemy queen) then you generally aren't concerned since the queens are gone.
It all really depends on the situation on the board though. There are some times where a sacrificed knight to ruin the opponents castling leads to a big attack on the king.
Say you're in a situation where you can either lose a pawn or lose your right to castle. It's obviously very important to generalise here but let's say the hypothetical situation is maybe 10 moves in with 6 points worth of material exchanged. What would you choose? Is it worth less than a pawn? Half a pawn, quarter of a pawn? How can you judge when it's important?