Controversy over Rybka's disqualification and ban (UPDATE)

Rybka's disqualification and ban from computer chess last summer by the International Computer Games Association (ICGA) is being disputed. In a 31-page article, a computer scientist working at London's Queen Mary University, supported by two chess programmers, argues that "the ICGA's findings were misleading" and the decision to punish Rybka and its programmer Vasik Rajlich "lacked any sense of proportion". Meanwhile, the ICGA has responded with a technical rebuttal.

Disqualification and ban

Last summer the International Computer Games Association (ICGA) disqualified and banned Rybka and its programmer Vasik Rajlich from previous and future World Computer Chess Championships. The ICGA accused Rajlich of plagiarizing two other programs, Crafty and Fruit, and demanded that he returns the trophies and prize money of the World Computer Chess Championships in 2007, 2008, 2009 and 2010.

Our report on the news made the mainstream media: it was picked up by the influential ExtremeTech and then copied by hundreds if not thousands of sites, including the New York Times and Der Spiegel.

Chessbase

It was the biggest computer chess story of recent years, but the editorial team of Chessbase didn't cover it. This is obviously related to the fact that the German company is in fact distributing and selling Rybka. (In their shop, it can be seen that on the Rybka 4 DVD the tag line 'Computer Chess World Champion' has now been traded for 'PC Chess Program by Vas Rajlich'.)

Last week however, Chessbase ended its silence. They published a lengthy article in four parts (you can download it in full in PDF here), dramatically entitled 'A Gross Miscarriage of Justice in Computer Chess' and written by Dr. Søren Riis.

Riis's article

The author, a computer scientist at Queen Mary University in London, makes the case that the ICGA’s allegation of "plagiarism", as well as the points they offered in support of their accusation, are without merit. The author, who is supported by an extensive technical report by Ed Schröder as well as support in the form of unpublished notes from chess programmer Sven Schüle, argues that the ICGA’s charges were based on false premises, tendentious conclusions and manipulated evidence.

Riis speculates that the chief motivating factor behind the persecution of Rajlich was his domination of computer chess programming in 2005-2010, during which time his program Rybka almost invariably annihilated other programs in public tournaments. The computer scientist and mathematician mentions that the people who voted for the Rajlich ban were direct beneficiaries and actually picked up his vacant titles, collateral which they now use to market their own chess engines. Riis further observes that, if anything, it is Rajlich’s program that has been systematically reverse-engineered and plagiarized.

According to Riis justice can only be served if the ICGA publishes a retraction of their accusations and restores Rybka’s world championship titles, concluding his defense by extolling Rajlich as

a great chess programmer, world champion and innocent man.

Biased?

We asked Mr Riis, who is not only a computer scientist in London but also a Rybka Forum moderator and therefore closely connected to the "Rybka family", for some more background. He replied to us:

There where strong arguments for publishing [the article at] ChessVibes as you would be seen as more neutral, while publishing with Chessbase would would open up for the most trivial counter-attack: “Cui Bono”. However, since I am a Rybka Forum moderator it was clear my article would always been seen as biased.

When I first read the ICGA report I thought they put a convincing case but is was only when I investigated the case I began to realise the full extent of the injustice. It was at that point that I decided to write my article which is written as a defense of Vasik Rajlich, and it is as much for him as it is to satify my own sense of justice and fair play.

I have no personal agenda against anyone in ICGA but they handled the Rybka case very poorly. I also don't have a particular personal agenda for Vas apart from the fact that I think he is exceptionally gifted, and I want to see justice done.

ICGA

We also asked the International Computer Games Association to comment. Dr. David Levy, head of the ICGA, sent us (and Chessbase) a lengthy rebuttal entitled 'No Miscarriage of Justice - Just Biased Reporting'. You can download it if full in PDF here but we'll quote from it:

As a historical review of progress in computer chess Riis’s article contains important and interesting information and comments. Unfortunately, however, his thesis lacks objectivity because it circles the core question and attempts to defend Rajlich by attacking the rule he was accused of breaking, attacking the investigative process in various ways and attacking some of those involved in that process.

When a defendant is brought before a court of Law, what is in question is whether or not (s)he broke the Law and not whether the Law itself is appropriate. And so it is with the ICGA rules. In considering the Rybka case the ICGA’s task was to decide the matter on the basis of its Tournament Rule 2, not to question the rule itself.

In his article. Levy tries to point out

the irrelevance to the ICGA rules of some of Riis's key arguments

and

correct some of his erroneous assumptions.

He then gives a number of examples that, according to Levy, point out that Riis's article at Chessbase is a case of biased reporting.

Technical rebuttal

We also received an article by Mark Watkins, a Research Fellow at the School of Mathematics and Statistics at the University of Sydney and member of the Computational Algebra Research Group. He was one of the investigators for the ICGA and his 'A critical analysis of the four parts of Riis' forms a technical rebuttal. You can download it if full in PDF here and we'll quote its summary:

In Part I, Riis suspects he has insights that the Panel ignored, and tries to argue that Rule #2 has become inapplicable. Neither of these assertions is borne out by his arguments in the later parts.

In Part II, among other problems, almost every relevant data point in his graph appears to be erroneous. Nothing in this first section holds water, in particular his intended conclusion that Rule #2 is no longer workable. On plagiarism, he asserts that Rajlich’s failure to cite Fruit in his ICGA entry is their problem (not his), and then returns to railing against Rule #2. He greatly minimizes the magnitude of Fruit/Rybka overlap here, perhaps as a corollary of erroneous conclusions made elsewhere. He then over-values the worth of indirect similarity testing as compared to direct code analysis. He finally mis-dates a quotation from Letouzey on originality, thus applying it to the wrong Fruit version.

In Part III, he gives 10 substantial Rybka/Fruit differences in evaluation, of which at most 4 seem viable, and all were noted by the ICGA. He does not discuss the remaining 20-30 evaluation elements, which were largely found to be substantially similar. He then misconstrues the “algorithmic” nature of the evaluation function, ignoring any creative aspects in its creation. He then monstrously exaggerates the impact of the floating-point zero issue on the ICGA decision, doing similarly with the PST issue in the next part. He fails to address a number of additional Fruit/Rybka congruences that were detailed by the investigation; together these helped to form a much broader picture than the 2 or 3 elements that he presents.

In Part IV, he misinterprets the question of PST copying. He seems to invent his own measure of comparison (raw numbers), while a proper metric would be the number of Fruit code changes necessary to replicate the Rybka output (and whether this number was abnormal). He then eccentrically proposes that one can skirt any “copying” issue by translating to a different programming language. He then suggests a timeline for Rybka development that erroneously states Rajlich went full-time in 2003, rather than in mid-2005. I will not address his defamation of Hyatt and the ICGA.

Riis omits any mention of the fact that Rajlich had previously plagiarized Crafty in private 2004 versions of Rybka, and furthermore that these versions had little internal similarity to the 2005 Rybka. The latter fact played a significant role in the Panel deliberations, strongly implying 2005 Rybka was a re-write, at the least.

Throughout, Riis displays little knowledge of programming, let alone that of computer chess therein. For instance, he claims that a one-time operation that takes less than a millisecond on Fruit startup (or during compile with Rybka) should be made fast for efficiency reasons. Furthermore, he is similarly lacking in any knowledge of the relevant aspects of copyright law, particularly the Abstraction- Filtration-Comparison Test that formed a basis for one part of the ICGA Panel analysis. He appears to apply a minimalist copy/paste standard to what “copying” might mean, ignoring any other creative aspects. Finally, he consistently refuses to apply any inferential capability regarding likely scenarios; combining this with an artificially impossible standard of proof, he is reduced to the pedantry of repeatedly asserting that no one can prove that Rajlich directly copy/pasted Fruit source code, when this was never the issue to start.

For the general chess audience one thing is clear: that this matter is very unclear. We're dealing with highly complicated issues related to programming and copyright. At several forums, heavy debates are ongoing and in fact we received several emails from our readers with a request to write about this subject. No doubt the last and definite word has not yet been spoken.

Update January 8, 2012 22:44 CET: We forgot to mention that last week we also asked Vasik Rajlich to comment on Riis's article. Only after we published today's article, we received an answer from him, in which Vasik refers us to the Rybka Forum, where he posted the following:

I want to give a really big thanks to Ed Schroder, Soren Riis, Chris Whittington, Nelson Hernandez, Nick Carlin, Jeroen Noomen and Alan Sassler for their superb efforts in defending me against the accusations that I have broken ICGA tournament rules. Soren did a great job detailing the shenanigans pulled during the ICGA's investigation, from stacking the jury to premature public accusations to a comprehensive fabrication of evidence.

I also appreciate the tenacity of Chris Whittington and especially Ed Schroder in digging through the mountains of documents and answering them point by point.

Finally, I greatly appreciate the support that I have received from countless others. This support has been touching and uplifting and I really appreciate it. Thanks guys! 

 

In other news, I'm working on Rybka.  Rybka 5 will be ready sometime this year and the remote Rybka renting will be launched sometime after that. We'll make announcements on our web site and in our forum when we have more information. It should be a fun year.

 

Best wishes to everyone for a great 2012!

 

Vas

In the same thread, Rajlich has now responded briefly to the main issue: the possible use of Crafty and Fruit code for creating Rybka. (We left out parts of the thread and only give Rajlich's key answers:)

Of course there is a clear influence of Fruit on Rybka. I haven't tried to quantify this influence or compare it to other engines from Rybka's generation. What I can say is that Rybka is original at the level of source code. In the context of source code, original means that the author either typed his own code or typed the code which generated his own code. For the super-geeks, yes, that can be applied recursively.