castling on the queen side

Castling on the queen side seems fraught with danger to me.  If white for example, castles on queen side, and black's rook or queen gets through to a1, that would be the end of the game.  I think it is far easier to get through the "castle" if it's done on the queen's side.