Borislav Ivanov is BACK!

Sometimes there are more equally good moves (both giving the same evaluation) and Houdini 3 chooses one of them while another engine could pick up another.

I checked B.I's recent game in Trompovsky Attack, all the moves from some point were recommended by Houdini but deviated from Stockfsh 4's choices at some points.

So, there were another ways to play best moves at those points but that would lead to a Stockfish-style game (which is quite different from Houdini-style because different heuristics are implemented in both engines).

If you asked Magnus or another super-GM what he would play move-by-move you would get another answers and probably each of those guys would suggest different plans at some points because all chess players have their own style.

B.I. seems to play in Houdini style - this is nearly impossible for a human to work out this sort of chess (in its roots the moves are based on pure calculations, backed up with some positional heuristics, human players think in terms of plans and they would rather avoid a 4-against-3 endgame which Ivanov won in the discussed game)

So what is the definitive P value for Ivanov's alleged cheating?

Ladies And Gentlemen, The Great Magician, Borislav "Smelly Magic Feet" Ivanov, also known as The Great Shoedini!

Actually, it depends. I already wrote about the subject in the other thread, but here is a summary : basically it is Bayesian inference. You need to do some assumption somewhere.

===========================================

Let's name A = "BI's moves match Houdini 3's at the rate X" (where X is the rate we observed, around 85% if I remember correctly) and B = "BI gets input from Houdini 3 by whatever means" ; no-B = "BI did not cheat". These events are of course correlate ; what we want to know is the probability of B, knowing that A is true, which we denote P(B|A) = P(A&B) / P(A).

-From statistical analysis of BI's games (see LegoPirateSenior's posts in the other forum) taken pre- and after- BI's first strong tournament (Zadar), you would get those results by chance with a probability about 10^-18, being given how much the percentages differ. This gives P(A|no-B) = eps ~ 10^-18.

-It is very obvious that P(A|B) = 1, ie if he cheats then he matches Houdini. As a consequence, P(A&B) = P(B) * P(A|B) = P(B).

-And from the so-called "total probability formula", we must have P(A) = P(A&B)+P(A&no-B).

Now, we need to take a guess about P(B), the probability a priori (ie without the statistical matchup information) that he cheated. We can take a very conservatory guess of 1 in a billion (or 10^-9) (even without the shoe show, it looks fairly conservative to me). You can discuss this, of course, so let's call that P(B) = p ~ 10^-9 to have some idea of how the final result varies with that.

Now, we just fire the math from the data we got so far, and get :

P(B|A) = P(A&B) / P(A) = P(A&B) / (P(A&B)+P(A&no-B)) = 1/(1+r)

where r = P(A&no-B)/P(A&B) =P(A|no-B)*P(no-B) /( P(A|B)*P(B) )

With the previous considerations, this boils down to

r = eps * (1-p)/p

Again : eps is a small number, of the order 10^-18, corresponding to the statistical analysis "by chance" probability, whereas p is a guess on the probability of cheating a priori.

With the numbers I mentioned

r= 10^-18 * (1- 10^-9)/10^-9 ~ 10^-9 hence the probability that BI was cheating, knowing what we know, is about 1/(1+10^-9) ~ 0.999999999.

If you take another estimate for p, you will have another result, but the crucial point is that if you want to get a factor r of the order of 1 (and hence a final probability significantly smaller than 1) you need to take p of the order of eps or smaller, which is completely unreasonable.

Fantastic, Irontiger - thanks for that.

IM Lilov is part of the chess.com community.

IM Lilov lives in Bulgaria. He is Bulgarian. Kind of why he took such an interest in the first place.

I think ivanov was the first case of use of synthetic telepathy/psychotronics to cheat in tournament chess. probably won't be the last. unfortunately. pandora's box has been opened.

Look for Google and read the threads about Ivanov. There is too much to repeat here. There was also a mathematician here who investigated the match between the moves of Ivanov and Houdini3.

@umberto_unity. Lilov has posted several long videos and pieces examining the moves that BI made, and comparing them to Houdini's first choices, and the moves grandmasters 'would make' and why; he shows rhar BI was following houdini's recommendation close to 100 percent of the time, and was making moves that human's just wouldn't make. there was also analysis of the amount of time BI took for his moves; he took the same amount of time for virtually every move; never thinking more or less time; demonstrating that he wasn't thinking, but relying on the transmission of moves from his houdini setup. 

The Bayes calculation above is not very impressive.

1) B = "BI gets input from Houdini 3 by whatever means" is not a random event.  So when we speak about P(B) we are using some subjective probability interpretation.  If we are using some subjective prob, all we get out the back is some subjective probability.

2) The whole calculation is driven by " This gives P(A|no-B) = eps ~ 10^-18." Sure, if we believe at the start that the probability of him matching Houdini is infinitesmal then the subjective probability that he cheated is really high.  Did we need some hand-waving Bayes calculation to get us there?

Probability theory is not really very applicable here because P(BI match Houdini if not cheating) is not really known.  So all that happens is that someone comes up with some new way to estimate that and then puts this pseudo-math framework around it to make it look scientific. 

I think the best evidence here is stuff like Lilov's videos where he discusses moves that computers make that people don't make.  It's obviously hard for me to judge because an IM move often is just as opaque to me as a Houdini move but I found his videos very convincing. 

@chiaroscuro62: The match between the moves of B.I. and Houdini 3 is well known. You can find that out. Then you have two situations:

1. B.I. got his moves from Houdini 3,

2. the match between the moves of B.I. and Houdini 3 is pure coincidence.

You can calculate what the match between the moves of B.I. with Houdini3 should be according to his rating. You can get this number by comparing a lot of games from people with that rating and match those moves with Houdini3. Let's say that that matching percentage is x%. Now can you compare that x% with the match rate between the moves of B.I. in his games in Zadar, being z%. When percentages differ can you calculate how big the chance is that those percentages are the same or not. The chance that x% and z% are the same percentage turns out to be 10^-18.

The chance that they differ is hence 1 - 10^-18. That is of course not a proof that B.I. is cheating. One can only say that a rating with x% does not reflect the matching rate with Houdini. Hence should B.I. have a rating that is much higher. (Higher than Carlsen by the way.)

However, B.I. has after Zadar played a tournament in which he scored significantly worse. (Much worse than you can imagine Carlsen would play on such a tournament.) Even below a rating of 2200. Although it is not calculated, it puts the 10^-18 in a different perspective. How can someone with a rating significantly higher than 2200 score much lower than one can expect from someone with a rating of 2200? The rating difference is by far more than 300 points. It all becomes quite remarkable until one starts to think that B.I. cheated in Zadar, but not in that other tournament.

The analysis of Lilov is very impressive for chess players. For non chess players is the statistical evidence overwhelming.

@Firebrandx: precisely. 

1- yes, "BI is cheating" is a random event in the sense of "something that might have different outcomes, which we want to take some quantitative bet on", that is the perspective here.

Rain is not a random event either, it obeys some atmospheric equations and stuff, neither a throw of dice that is uniquely determined by the initial speed and position of the dice and the elasticity of the dice etc. but if we have very limited information about the causes we can consider them as random. In other contexts, no, we consider it as deterministic.

2-Actually, this comes from the assumption that BI pre-Zadar (BIPZ) and post-Zadar (Shoedini) are the same person, ie the same playing strength. Pay attention : the assumed probability of BI cheating, pre-computer analysis, is p, which is supposed to be small, not big. eps is precisely the chance to have a "miracle" ie a huge match percentage by chance (and surprise, it is small), under some conditions.

(You can skip to the last paragraph if you are in a hurry)

This is a calculation from LPS in the other thread ; the idea is that your move correlation is going to be more or less constant over time. From data of genuine players you can deduce that BIPZ is playing around 2000, Shoedini is playing at 2500 or so ; it might be that in fact Ivanov is around 2250 and had by chance large deviations in both directions, but the probability of that can be computed, and again believing LPS it is around 10^-18. This is eps = P(Ivanov is both BIPZ and Shoedini, and all this by chance), and this comes from real mathematical stuff.

Important note : if Ivanov had been matching Houdini at 100% from the beginning of his career, that argument would not exist. It would still be highly suspicious that he gets to such a gap with all of his other predecessors who peak at 70% for one tournament, but maybe the guy is just really good. Then comes the argument that Houdini is not perfect play etc. and now the math become slippery. The Lilov stuff of thinking time on each move, how natural the moves are, etc. is very convincing, plus it is understandable by the crowd, but it's not quantitative.

Now, to do the Bayes, you do need some assumption of the proba that Ivanov might be cheating, pre move-analysis. This is p in my notations, and this can of course be discussed, it does come from guessing and not from stats.

The lesson of the Bayesian calculation though is : if you want to have reasonable doubt about P(BI is cheating), you need to allow p to be at most of the order of eps. And this is just unreasonable.

If you make Ivanov play naked under laboratory conditions with tons of computer experts, illusionists, etc. watching and he still is Shoedini, then I will have no other choice than to admit the value of p must be adjusted down a lot (he would need to find a way to fool all the observers), and all the strange results were actually obtained by chance. But when he refuses to take off his shoes, the value of p goes up to 10% or so in my book, and in no way it stays under 0.00000001%. The point of the Bayesian stuff is that we know we are in an implausible part of the universe : the huge matchup results are explained either by an implausible cheating or by an implausible miracle, so either must have happened, but which one of those implausible is the least implausible ?

Most of this is not true.

1) The third possibility is that BI has learned to play like Houdini.  That sounds preposterous to me, but chess moves aren't randomly sampled from some distribution.  They are chosen according to some heuristics which are knowable. 

2) " Let's say that that matching percentage is x%. Now can you compare that x% with the match rate between the moves of B.I. in his games in Zadar, being z%. When percentages differ can you calculate how big the chance is that those percentages are the same or not. The chance that x% and z% are the same percentage turns out to be 10^-18."

This is more of the same abuse of mathematics.  There are surely hypothesis tests comparing sample proportions but all it shows here is that BI has a higher match probability.  That doesn't directly address the cheating at all.  BI could simply be using a different way of choosing his moves.

3) "For non chess players is the statistical evidence overwhelming." The statisticaal evidence is not compelling to anybody who knows anything about statistics.

"1- yes, "BI is cheating" is a random event in the sense of "something that might have different outcomes, which we want to take some quantitative bet on", that is the perspective here."

What can I tell you? "BI cheating" is simply not a random event and perhaps you missed the first chapter of every intro stat book ever written.  He is either cheating or he is not and there is nothing random about it. 

"Rain is not a random event either, it obeys some atmospheric equations and stuff, neither a throw of dice"

You have missed the entire point of a random event.  Throws of a dice are random events that come out according to a known distribution.  The issue here is not the deterministic underpinnings of a single throw.  Suggest you pull out an intro stats book and review.

3) "Actually, this comes from the assumption that BI pre-Zadar (BIPZ) and post-Zadar (Shoedini) are the same person, ie the same playing strength."

The aren't the same person.  They are the same chessplayer playing at a different time.  There are thousands fo reasons why his performance could change that have nothing at all to do with randomness.  Among them are: he leaarned to play chess better, he was not preoccupied with breaking up with his girlfriend, he ate better, he wasn't drunk, he didn't take opiates, he fixed his sleeping problem, etc, etc...  Statistics has nothing to say about any of that.

4) "This is eps = P(Ivanov is both BIPZ and Shoedini, and all this by chance), and this comes from real mathematical stuff."

Nah - it's faux math meant to impress the wannabe mathematicians. 

5) "The Lilov stuff of thinking time on each move, how natural the moves are, etc. is very convincing, plus it is understandable by the crowd, but it's not quantitative."

Right - This is real argument.  The statistics stuff is just abuse of statistics piling on a whole bunch of fuzzy concepts like subjective probabilities and assumptions like "He ought to match Houdini the same on one day as he did some other day".  Whenever you see a stats argument, look at the assumptions and see if they are correct.  You have no idea how robust those P-values are to violations of assumptions and the assumption I just gave is surely not completely true for anyone.

6) "Now, to do the Bayes"

Read my explanation again.  Sorry you didn't understand it the first time.  The calculation was completely bogus and merely threw some (completely unsophisticated) math at something to make it look scientific.  It probably impresses the Yekatrinas of the world but isn't real math.

@chiaroscuso62: Irrespective if you in your first sentence refer to your own sentence or to my post, I have to disagree with you. If you, however refer to your own post, then I agree with your first sentence. :-)

ad 1. If you have any knowledge of chess, how chess engines work and how the human brain works, then will you accept this possibility as a fairy tale. Just impossible, not worth any further discussion from my side. The chance that you look outside the window and you see a real unicorn with a fairy on it is bigger than the possibility that a human makes use of the same heuristics of a chess engine.

ad 2. Abuse of mathematics is a big term, for sure when you do not read accurately and you do not draw the proper conclusion yourself:

BI has not a higher match probability, he has a certain match with Houdini3. A higher match probability is not an exact term. Higher match probability than what? If you meant to say that his match probability conforms to a higher rating, then will I agree with you. His match rating conforms to a significant higher rating then he had at the time of the event.

That is still nothing to worry about as it happens to all chess players. Hou Yifan has scored ratings far above her current rating for instance and Ivanchuk far above and under. I know for Ivanchuk that he scored once 300 points lower than his own rating. That is where my 300 points came from. But the difference in ratings of B.I. is even more. Much more if you take his other tournament directly after Zadar into account. The man that could beat a GM above 2600 lost from someone in the 1800's and suddenly could not play in a style that matched the style of Houdini 3 at all.

We all suddenly change of playing style and skill, do we? Uhuh. In one tournament he had a higher match with Houdini 3 than Magnus Carlsen (he thinks similar to Houdini 3), in the next tournament he has lost all of those skills and has a very low match with Houdini 3 (where have his abilities gone? Suddenly disappearing?)

The cheating only becomes the best explanation when you have to explain his worst results too. If you only focus on his good performance, then will you have draw the same conclusion as I did: the rating of that performance is significantly different from the rating he had at the start of the tournament.

ad 3. Consistency is all. If someone does not perform consistently, then can I not conclude that that person has that certain capacity. Statistics is based upon consistency. B.I. is not consistent at all, that makes this statistics so compelling.

I expect you to disagree with my post. I will read your post with respect for you, but I will not respond anymore. Have a nice day.

I am not saying that BI didn't cheat.  I'm saying that your statistics argument is silly, your P-values are sillier, and you have not a clue what you are doing. 

"Abuse of mathematics" is not a very big term and it clearly applies to the silly calculations you have done here.  You did a Bayes calculation that includes a subjective probability.  That means you just threw in some number that seemed okay to you and out the back you got a subjective probability for a non-random event and advertised that as meaning something.  The only thing it means is that if you believe that it is incredibly unlikely that BI matches Houdini you think it is very likely he is cheating.  Big deal.  My Mom understands that and doesn't need to throw bogus Bayes theorem calculations at it.

The rest of your post is just stacking up more assumptions that mean nothing.  If I was BI and you came to me and said that I ought to be as consistent as Carlsen or Hou Yifan or Pfren, I would shrug and say "Why?". 

"Statistics is based upon consistency."

Huh?  Your statistics are based on a whole bunch of clearly made up, unsupported assumptions and really fuzzy thinking about what random means.  There is a notion of consistency in statistics (it means that a statistic converges in some sense to the parameter it is estimating as the sample size increases) but that has nothing to do with your discussion here.

@chiaro: I don't know why you feel the need to insult people. There's really no need for it. 

finally someone is talking some sense

enough of this random internet postings