It's because the best laid schemes o' Mice an' Men gang aft agley.
I haven't really paid attention, but is it because you know where you're from, so the other's flag only is up so you can see where he's from?
Maybe your flag is on your opponent's screen and his isn't.
when we play your flag is next to your name
How do I change the union jack with the Scotland flag?