Open file confusion

Meant to bring black bishop to g6 so if white takes there is an open file for my rook on h8, but if I castle kingside, the rook is not  in front of the open file, the king is closer to the open file, does this mean I should castle Queenside, is their something I am missing? Or do I not castle so the rook has that open file?

@1

"Meant to bring black bishop to g6" ++ Better trade it: Bxd3. Bf5 is your bad bishop: on the same color as your pawns d5 and f7. Bd3 is his good bishop, not on the color of his pawns e3 and c5. So Bxd3 is a favourable trade for you.

Another option is to play ...e6, intending to recapture Bxf5 exf5 with firm control over central square e4. ...e6 also opens the diagonal for Bf8 and threatens ...Bxc5 and prepares ...O-O.

"if white takes there is an open file for my rook on h8" ++ What would you do with that file?

"but if I castle kingside, the rook is not in front of the open file, the king is closer to the open file" ++ Bg6 Bxg6 hxg6 and ...O-O is a bad idea indeed, put your king in danger.

"I should castle Queenside"
++ No, O-O-O is not safe. Your pawn c7 is missing and his Bf4 is dangerous.

"Or do I not castle so the rook has that open file?"
++ Leaving the king in the center is dangerous.