How to Catch a Chess Cheater

As I see it, the problem with this method to detect cheating is that it assumes a player of a given rating always plays at that rating.  It doesn't account for the possibility that a player can have a Trefler-like result and play well above his rating.  The moves, after all, are right there in front of you.  If you just happen to find the right ones, you too can play at a GM level.

In other words, performance is a bell-curve, not an absolute.  If an infinite number of monkeys play Magnus Carlsen, a few of them will beat him.  

Apparently some people do not take well to being corrected when they are wrong. That was just odd. Why are you so angry, son?

<#16> 2075 isn't THAT far away from master... as he was amazingly impressive otherwise, I think it's ok to round that corner. It would have looked funny if he was 1600...

<#17> Fluctuations between your rating and your true strength, and between diverse chess programs, lose their importance when you consider the HUGE difference between observed play (IPR) and actual elo rating that the tests are looking for - which are many hundreds of points. 

It's like comparing my chess rating to that of Carlsen and then saying - but wait, we didn't take into account a couple of draws that Carlsen played which will only be rated in July. 

<#19> you are right :-) "Smart" cheating, such as tweaking your level consciously up by just a hundred, a hundred and fifty points, is tougher and slower to detect, if possible at all. 

<Obit> - from the article it would seem that results like are being caught, with the cheater playing at five standard deviations above his actual rating, are one in five million... so you'll have one Trefler here and there, but you will mostly catch the cheaters. 

The system gives a clear indication of strong candidate for cheating. It doesn't replace human judgment on behavior after they are caught. 

Were Trefler accused of cheating, he would probably acquit himself in a way very different from the behavior of people such as Ivanov, "Von Hermann" and the others. 

I looked up von hermann & didn't find anything cheater related. Was the name spelled right?

He registered with the name John Von Neumann

Yeah, what I meant.. :-)

Hermann was Colonel Landa's trusty partner. I guess I keep mixing the two. Both came to a rather grim end...

Oh yeah! John Von Nuemann. I was actually there at that tournament (World Open) & saw him playing & around in the halls & so forth. I was just visiting the tournament but not playing. I lived in Philadelphia at the time. He wore a rediculous looking headset & never was seen without it on his head. & he had dreadlocks so if you can visualize the scene he stood out like a neon sign, just sitting & bobbing his head like he was jammin or something. He was unrated & it was supposed to be his 1st tournament (!!) & he was playing at weak GM level. I think that was 1993. If he was cheating & whoever was helping him had been smarter they would've at least gotten a decent chessplayer to try the scam because I heard Goichberg got him to do a simple test position or two that a class D or C player should be able to solve & he couldn't do it, had no idea. He would also spend an incredible amount of time on simple recaptures occasionally, stuff that seemed very odd. 

At that same tournament (I'm pretty sure it was the same one) I actually got to sit & talk with Rustam Kamsky himself for about half an hour. I used to smoke & was having a cigarette outside one of the small entrances & he came out to smoke. We ended up chatting about family & similar things (he brought it up) & how important family relationships were, etc. Just the 2 of us talking & smokin. He thought I was making a mistake living in Philadephia while my family was home in NC. Haha. It was kind of cool really. He was quite nice & friendly, unlike all the stuff everybody reads about. Yes, I was actually getting family relationship advice from Rustam Kamsky himself!! We smoked a couple, wished each other luck & said the usual "take cares"  & then he went back in to check & see how Gata was doing. 

I had not heard this anecdote specifically before (so thanks for the story!), but this was exactly what I was thinking when I read the article. A 'lesser' player can have a great tournament....and many very strong players aren't GMs because they don't participate in enough FIDE events to get the norms etc., but that doesn't mean master level sleepers don't exist. Also someone rated 2100 could vanish from the chess scene for a few years, study a ton, suddenly scream back into the competitive world, and start winning everything. 

......I'm not sure that I can suggest a better way to cut down on engines in earpieces, but it feels pretty icky to be evaluated and expected to perform according to the evaluation on pain of disqualification for uncharacteristic awesomeness. 

I think that a 2100 rated player playing at 2500 strength isn't even five standard deviations... 

so he would have to otherwise show suspicious behavior, on top of performing at 2500. 

........I'm a culture chick, not a math chick :)

Thanks for the feedback.

Let me correct a misconception.IPR is, in fact, unreliable. IPR is like a "grade" on an anti-cheating exam, and if one were to compute it using only a few moves, then this would be like grading an exam with only a few questions: not enough data for reliablility. In fact, IPR and Z-score are two totally different metrics, and Regan uses the Z-score to detect cheaters; he does not use the IPR to detect cheaters.

The "typical performance" for each Elo is found by Regan by "training" his algorithms on hundreds of thousands of moves played by thousands of players. I refer you to Regan's papers if you want a more technical definition.

To simply compare an accused cheater's IPR to his Elo would be unreliable to say the least. IPR is more of a fun metric than a valid statistical tool. The Z-score is what uses all this trained data. Perhaps I didn't make this clear enough in the article.

The IPR is based ONLY on finding the parameters s & c. The Z-score is more akin to a standard deviation. To quote the article: "If Regan knows a player’s Elo before subjecting the player’s moves to an anti-cheating exam, he can compare how well each moves’ partial credit matches the typical partial credit earned by a player with that Elo. Regan represents this difference as a z-score, which is a fancy name for the ratio of how many standard deviations a player’s test performance is from that player’s typical Elo performance. The greater the z-score, the more likely a person has cheated." Each move's partial credit gets compared to the engine's best move, and the average error of all moves is used to create the z-score.

The difficulty with writing an article like this is that some people want just the conclusions and would rather not read the other 5000 words of profile and discussion about artificial intelligence and the differences about how humans and computers "think" about chess. So I apologize to people like r_k_ting who found the article meandering. But other, more literary types, might appreciate the "meanderings" and profile. Combining the technical with the human interest was not an easy task. I can't please everybody.

Unless the z-score is way off the charts like Ivanov's, nowhere does Regan say that his statistics should be the sole determination of cheating. Physics experiments typically accept z-scores of 5 or greater as "proof" that a law of nature exists. That's pretty reliable. But Regan himself says that physical or psychological evidence, in addition to a large z-score, is necessary to convict a cheater. Or multiple "high" z-scores in multiple tournaments in a row would also be "proof." I haven't studied the latest FIDE-ACP anti-cheating proposal for Tromso, but it can be found here: http://www.chessprofessionals.org/content/draft-fideacp-anti-cheating-proposal.

As I wrote in the article, Regan is working on technique to catch "selective-move" cheaters, guys who use their engines on only a single move per game. To catch these guys would certainly require some non-statistical evidence. But once this evidence is established, then the TD could ask Regan, without telling him what exact move he had in mind, "Ken, I think this guy cheated on one move in this game." Then, like a police lineup, Regan would report back to the TD which move he thought was the culprit. If the two moves match, then the physical/psychological evidence plus the police linup evidence could be damning.

Thank you all for reading the article. I hope you enjoyed it (or at least most of it). I know it's long. But I'm not out to please everyone.

Hg

Excellent article Howard!

I fear I am going to remain sceptical about this sort of thing for a long time.

However the two or three well publicised cases of cheating by obtaining access to computer analysis while playing make it clear that efforts have to be made to guarantee the honesty of results. If the sort of paranoia seen on this site ever spread to otb tournaments it would be a great pity and might well put people off participating. If (widely shared) suspicion started to reach the upper levels of the game sponsors would certainly be even thinner on the ground than they already are.

So I guess I have to be glad off Dr Regan's work. I hope, however, that this sort of thing does not come to be seen as an alternative to robust rules keeping electronic devices of all kinds as far away from the playing areas as possible. In that sort of precaution I have much more faith.

From what I understood, 'z-score' means how many standard deviations a player's IPR went away from his player's rating...

put otherwise, the "z-score" is based on the following:

1) his IPR; 

2) his elo. 

Did I miss anything?!

Solskytz, The IPR is based on the parameters 's' and 'c'. (Go back to the article for a recap of how this is done.) To find the z-score, Regan first finds a very accurate IPR, using hundreds of thousands of moves from players with ratings at each rating century mark (2000, 2100, 2200 ... 2800, etc.). These moves were made by players known NOT to be cheating, and are the "baseline" 's' and 'c' parameters for these ratings. He then uses the curve produced by these accurate 's' and 'c' parameters as a model curve, as the "mean" if you will, to compute the standard deviation on a set of moves produced by the test subject. So say a test subject rated 2100 has 100 moves available to analyze. Regan takes each of those 100 moves and finds the error associated with each move, compared to a "true" 2100's curve, and then uses this error to compute the standard deviation. The z-score is just a fancy name for standard deviation. 

I didn't go into this much detail in the paper. Maybe I should have. THis was a very difficult article to write, because of its technical content, and it probably could have been clearer in explaining the z-score computation.

Hope this helps.

Hg

Well, it does look as though I did get the idea after all...

Your article is very well written. I really enjoyed reading it - and I like its message. 

Yes, I hope you did. What is NOT done, and what many people THINK is done, is that Regan DOES NOT compute an IPR and say, "Look, this person's IPR is 500 points higher than his Elo, so he must be cheating." This is NOT done and is completely unscientific. The IPR is a fun number to look at, but a single IPR from a few positions is more or less useless to detect cheating. A z-score is different. It is computed by using very accurate 's' and 'c' parameters garnered from thousands of training positions from players known not to be cheating; and then by comparing the error between the partial credit expected from these "true" Elos and the partial credit actually earned by the test subject's move selections, a z-score (standard deviation) can be computed. Hg

Howard, thank you for the elaborations!

A much "softer" type of computer cheating would be getting the computer evaluation of the current position. No moves would be given, but just knowing the health of your position would be an advantage, and it would be undedetectable.