I believe it's a network issue...same thing happens to me...there's a red/yellow/green bar that's usually next to your name and if it's red it means that you have a laggy connection...so when your opponent moves there can be a significant delay until your computer receives that message..causing you to lose some time.
If you don't want to risk losing on time standard chess (30 minute) or 15/10 minute games are better! (You'll lost 25% of your time in bullet lol)
I've started to see a strange phenomenon where big chunks of my time disappear when it's not my turn in live chess -- not ticking down second by second, but large chunks (like 20-30 seconds worth) just disappear and it isn't even my turn. A few times I've even "run out of time" although it wasn't my turn and I had plenty of time left when I completed my last move.
Has anyone else seen this? Is it possibly related to a network issue (I'm on a new network since I've seen this start happening).