It could be for one of two reasons.
One: You're playing games with an increment, on this site they are represented by the number after the "|". E.g. 5|5 means five minutes plus five seconds for every move made. You have been playing such games.
Two: Lag. This has happened to me a lot when I'm lagging. The server may not register your move until a long time after your screen shows that you've made it. When you re-sync your time will appear to jump down and your opponents will jump up. This may be the first indication that you're lagging.
Why do my opponents "win" time on their clock? I was winning on time, and then all of a sudden the clock of my opponent went up from 12 secs to 19 to 35...?