Cheating on Live Chess?

This has happened before to me and I'm wondering what is going on; how this is possible.  Both of our connections were green but my opponent's time would go down to 3.1 seconds or lower and then back up to 20 seconds.  wtf?!  He ate through two minutes of my time and ending up winning because of it.  How is this possible?  Before I chalked it up to a bad connection but now I don't think that is what it is.  Anyone had this happen before?