"Problem mode" and "game mode" are things you made up, so I have no way of responding to that.
Engines take positions as input, and assign a value to each legal move as output.
Interfaces can implement a "game mode" on top of this functionality by ending a game when time runs out and by not allowing the user to take back moves (etc), but this doesn't change the operation of the engine which is simply assigning values to legal moves.
That's not to say there aren't more versatile (or specialized) engines and interfaces, but if you had such an uncommon and specialized piece of software (one that rigorously tests whether castling is legal), you'd have mentioned it.
Ok, I tested a few interfaces I have. Some assume castling is illegal and some don't. Mostly it seems to depend on whether you're starting with a full board or empty board. Castling is legal when starting with full board, and illegal when starting with empty.
But none allow it to be ambiguous and try to figure it out afterwards. After giving an engine a position, the FEN notation will tell you how the engine sees it. This is regardless of how you set up the position (manually, with FEN, or otherwise).