Feels like I always lose a rematch
Probably psychological because your opponent just lost and now they are super focused to try and win that second game. This determination might make them more likely to win.
This can backfire for them too though. If they let emotions get to them like this, then perhaps they are also more likely to tilt in that second game if you give them a lot of resistance and this might make them more likely to blunder as well.
Psychology aside, rematches or not, you are no more (or less) likely to lose against them than you were in the first game. It is the same player and their ability didn't significantly change.