The same has happened to me and my friend a few days ago.
Unexpected behavior when playing Balanced/Odd games

A chess friend challenged me today for a handicap game where I am the weaker player. However, I was the one with white and the handicap. It's obviously a bug. Sent bug report. i think for handicap rematching, when it's the stronger players turn to play black, he should also still have the handicap with black.

I didn't realize until now that this topic was started in October last year. It's a long lasting problem, and it's obviously still not fixed. Will it?

Got a new challenge today from a much stronger player. Also in this challenge I'm supposed to play white and with me as the weaker player having the handicap for a more UNBALANCED/odds game. Hope it will be fixed. This is getting tiring.

https://support.chess.com/article/295-how-do-i-report-a-bug-or-make-a-suggestion
Hi,
Maybe worth reporting as a bug and hopefully it will be looked into.
Thanks
Still a bug.