The easiest thing to do is just click on the follow game icon after the new game has started, but maybe chess.com should make this easier, you can suggest this with Help->Make a suggestion.
How to keep observing one player's game

@aanval22 thanks your reply, but the problems is I cant find where's the follow game icon lol.
https://imgur.com/0ijrBAI
When I observing one player's game and the game ended, if he didn't rematch but searching for new opponent, the observing mode didn't switch automatically.
Any way to auto switch?