Absolutely agree. It's just that we're always going to meet someone who doesn't know the rule and then accuses you of making it up. The way en passant works, it's just ripe for a certain type to say just that because when he tries it, *three moves* later, and you say, "You can't do that now," he'll say you're changing the rules as you go along, blah, blah, blah.
I asked a friend once if he knows en passant before we played for the first time and he just looked at me blankly with a sarcastic look on his face. He said, "Are you kidding me with a stupid question like that? I've been playing since I was a kid."
Damned if you do, damned if you don't!
Another thing is what we're dealing with here is casual chess players. They don't really take the game seriously and might just play a game when family is around for Christmas. And these are the ones who insist that they can put their king next to yours, that you have to say "check," that a pawn can move two spaces only on the first move of the game, etc. En passant is just special because it doesn't come up as often as the other rules of the game.
I've never had a situation where en passant capture was more satisfying. In the blitz game linked, I went from a loss to a win because of an opponent's mistake. He or she had a sense of humor about it but I can't imagine playing the same game in a coffee shop against a stranger and being told I'm making that rule up! Whenever I play someone new, I can't help this intense compulsion to just ask, "You know what en passant is, right?" to avoid any situations.
The move in question is 49 . . . a5.
https://www.chess.com/live/game/1823665092