This is what I think as well. Taking too long to castle can be a bad thing, obviously, but there are times where taking a long time to castle (or even not castling at all) is perfectly fine.
The way I see it, castling should generally only be done if one of the following things is true:
1. Your opponent is attacking one side of the board heavily
2. The center is open(ing)
3. Your opponent is threatening to take away your castling rights and launch a big attack.
4. You just don't see a better move on the board.
Even if you lose the right to castle, it's not necessarily the end of the world. Take this position from one of my games, for example, where the loss of my castling rights didn't mean much:
I am finally learning chess. I do not understand why castling early is so important. example. I develop early and castle king side. Lets say after 5 moves. Does that just let your opponent just set up first and attack that side? Is not better to wait as long as possible to see where their attack is heading?
600 ELO player at best