Is it legit to let time pass on purpose ?

I had a game where my opponent was behind. I had not much time, like 20 sec left, but he lost his queen.

He gave a good fight, but despite I was short in time I got his bishop as well and we traded one rook.

Then he offered a draw and I refused.

Then he stooped playing.

When he had 20 seconds left he finally did a move and game continued. I obviously won.

At first I was a bit upset because it felt like he wished I would drop the game.

But on the second thought , it is his clock, and it might be just a way of him to add  confusion and phsycological elements to the game.

What is the correct point of view on this ?

This is bad sportsmanship by your opponent, known as stalling (letting your clock run out when you know you're lost). If your opponent does this, at the end of the game, click the thumbs down button in the game chat and report the player