Open Source Community Critical Of Chessbase, Fat Fritz 2

The development teams behind the two most successful and influential open-source chess programs, Stockfish and Leela Chess Zero, have issued statements denouncing the commercial program Fat Fritz 2 and the company Chessbase that is selling the program for 99.90 euros.

The statements (Stockfish blog, lichess announcement) assert that the engine in Fat Fritz 2 is Stockfish with minimal changes, that Fat Fritz 2 has violated the GNU General Public License under which Stockfish is released, and that Chessbase's marketing has made false claims about Fat Fritz 2's playing strength.

Last week saw the 25th anniversary of GM Garry Kasparov's victory over IBM's Deep Blue. More famously, the 13th world champion lost the second match a year later. For many, this marks the moment in history when man's superiority over the machine on the chessboard ended.

Kasparov's defeat was artificial intelligence's victory. It was, in a way, the end of a journey that began in 1950 when Alan Turing finished the code for the first-ever written computer program (although Ada Lovelace is seen as his predecessor there).

Turing died in 1954, two years before the term artificial intelligence (A.I.) was coined for the first time by John McCarthy. McCarthy liked to call chess the "drosophila of A.I." because fruit flies made genetics work in the same way that chess in many ways has made A.I. work.

Enter Matt Ginsberg, an American computer scientist, mathematician, and the creator of a computer program that plays bridge and one that solves crosswords. Since he is also a contributor to Stockfish, Chess.com contacted him to discuss the Leela Chess Zero/Stockfish vs. Fat Fritz controversy. Ginsberg, however, rightly pointed out that it's good to emphasize the importance of chess for A.I. and the different ways humans and machines deal with problem-solving.

The strong relationship between chess and A.I. resurfaced in December 2017 when Google's DeepMind introduced AlphaZero, the chess program that shocked the chess world because it defeated Stockfish, at that point the world's strongest chess-playing program, with a highly dynamic playing style based on neural network machine learning. For DeepMind, these early experiments with board games were the basis for groundbreaking research in other fields.

AlphaZero's source code was never released, but the project inspired the computer chess community to develop Leela Chess Zero (Lc0), using the same techniques as AlphaZero. It was released in 2018. In 2020, Stockfish began supporting NNUE, fast neural networks that can run on a single CPU.

The release of Fat Fritz 2, a commercial chess program using a neural network, has provoked strong reactions from the open-source community. What follows is an attempt to untangle a rather complicated story, following a chronological storyline. 

Albert Silver and Deus X

Shortly after DeepMind introduced AlphaZero, Albert Silver started researching neural networks. Silver is a chess and computer enthusiast based in Rio de Janeiro and a regular writer and occasional photographer for Chessbase. The German company has been the market leader in chess database and engine software since the late 1980s.

A neural-network chess program consists of two components: its search functionality (usually called the engine) and the neural network. In his latest article for Chessbase, Silver used an analogy: "In the chess engine world with neural networks, the search is [the] race car and engine, and the neural network is the driver."

Silver introduced a neural-network-based chess program, called Deus X, in 2018, the same year that Leela Chess Zero was released. It is important to note that Silver did not program the engine himself; instead, he used Leela Chess Zero and trained a neural network differently than the Lc0 team had done. 

Instead of having it learn by playing games against itself, he trained Deus X exclusively from human games that he took from a database, with players rated from 2200 and above. "It was a personal idea," says Silver.

Deus X was invited to participate in season 13 of the Top Chess Engine Championship (TCEC), and Silver was interviewed about his new chess program.

In the days ahead of the tournament, some of the other participants raised questions about Silver using the Leela Chess Zero engine as part of his program. Shortly before the tournament started, all the wrinkles were ironed out with a joint statement by the Leela Chess Zero team, the tournament director, and Silver in which it was made clear that Deus X was "powered by" Leela Chess Zero.

Silver estimates that Deus X was about 100 Elo points weaker than Lc0 at the time because it was using a small neural network: "I didn’t have any resources, and I used a very crappy computer."

Despite the Elo difference, Deus X managed to beat Leela Chess Zero in one game. According to Silver, that was the start of his fallout with the open-source community, although the Leela team attributes the loss to a hardware failure and says that no hard feelings resulted.

"This very day that it won that game [August 7, 2018 - PD], they published an attack on me, saying 'we had a change of heart, he’s scum, he’s garbage, he’s been misleading everybody,'" says Silver, referring to this post on the Leela Chess Zero blog. In reality, the blog post does not include inflammatory language and is concerned mostly with whether or not Silver credited the Lc0 work appropriately in describing Deus X.

The Lc0 team wrote that Silver needed "incomparably less effort relative to the effort that he reused from the LCZero project, as the LCZero team had all the implementation work done, including building tools for the supervised training and the engine itself, defining neural network architecture, etc."

After season 13 ended, the TCEC tournament director took the side of Leela Chess Zero, updated its uniqueness criteria, and did not invite Deus X back into subsequent seasons.

Deus X becomes Fat Fritz

Silver began increasing his neural network size and adding engine games in the training process. After getting financial and practical help from Chessbase co-founder Frederic Friedel, he got access to better computer hardware, and in 2019, he mentioned the project to another co-founder, Matthias Wüllenweber.

Wüllenweber was interested in turning Deus X into a commercial chess program but insisted on first getting personal approval from the Leela developers that the Leela engine could be used. Silver contacted Leela developer Alexander Lyashuk on August 15, 2018, apologizing for failing to credit Lc0 properly and asking for permission to host an unchanged Lc0 binary under the name of Lc0, with a weak (<2000 Elo) custom net. The next day Lyashuk emailed back, saying he had talked to other people involved and that there were "no objections."

Silver's program was commercially released in November 2019 under the name Fat Fritz, which refers to Chessbase's long-time engine Fritz that launched back in 1991. It was the market-leading engine throughout the 1990s. 

Fat Fritz was sold together with version 17 of Fritz for 79.90 euros. The package included an interface, a database, and a premium account.

On February 9, 2021, Fat Fritz 2 was released, again as part of a package, now priced at 99.90 euros. Whereas the first edition (which was largely based on Deus X) used Leela Chess Zero for its engine, Fat Fritz 2 uses Stockfish because of its support of NNUE and has a neural network twice the size: where other nets use three layers of 256/32/32 neurons, Silver works with layers of 512/16/16 neurons.

"I spent six months pretty much renting out computers on the cloud to the tune of allowed $10K," says Silver. "I rented these 24/7 developing data from the Fat Fritz neural network that was transformed into data that I could use to train a neural network in a NNUE environment."

Criticism from the open-source community

A day after Fat Fritz 2 was released, an early criticism from the open-source community came from Tord Romstad, one of the first developers of Stockfish. 

I'm so disappointed in @ChessBase for selling FF2. It might be legal, but it's morally questionable and shockingly unoriginal.

— Tord Romstad (@tordr) February 10, 2021

Joost VandeVondele, the current maintainer of the Stockfish project, published a statement on February 15, 2021, where he notes: "ChessBase’s communication on Fat Fritz 2, claiming originality where there is none, has shocked our community."

According to Stockfish developers, there is no significant difference between Fat Fritz 2 and Stockfish, an opinion they also expressed in an article titled "Fat Fritz 2 is a rip-off" posted on lichess. It was co-written by authors of different open-source chess programs.

The article starts by stating that in July 2018, "Silver secretly sent the Leela Chess Zero engine with a custom neural network he had trained to participate in TCEC under the name Deus X." It does not mention the joint statement published before the tournament where Lc0 developers initially acknowledged that Silver had made it clear that Deus X was powered by Lc0.

The developers then move to the first release of Fat Fritz and note that it used the Leela engine with some minor changes: "modifying the name and author strings, and some default parameter values." They point out that in the product description Leela isn't mentioned anywhere, while Chessbase speaks of "an AlphaZero clone."

Silver is unhappy with this description. "I complained about that, I had nothing to do with that," he said.

As for using the official open-source binary of the Leela search, Silver says he wasn't hiding it: "This was made public. In fact, if you install the commercial version of Fat Fritz and you look in the UCI list of engines it says literally: 'Fat Fritz (in Lc0). It can’t get any clearer than that."

The open-source developers presented similar criticism toward Fat Fritz 2, pointing out that only minimal changes were made to the Stockfish engine.

"Even though the Stockfish engine is critical for playing strength, it is mentioned only briefly and the impact of the Fat Fritz 2 neural network over the one used by Stockfish is greatly overstated. The product description says FF2 is 'learning from the surgical precision of Stockfish’s legendary search,' but it isn’t learning from Stockfish, it is Stockfish."

Silver isn't happy either with the suggestion that his program learns from Stockfish. He points out that the neural network learned from the games of Fat Fritz 1 instead. He disagrees with the argument that the impact of his neural network is overstated:

"Both engine and neural network have a very powerful contribution to make," Silver says. "If you take for example the NNUE and removed the neural network from it, it can still play chess because it has the classic engine involved, but you’re going to drop 200 Elo so obviously the neural network is contributing more than just a show of its face, to look pretty and make a nice banner."

Silver says he has never denied that he uses Stockfish in Fat Fritz 2: "The search is absolutely 100 percent Stockfish. Let’s not get any misunderstandings here. I don’t claim otherwise, at all."

Authors list

Nonetheless, the people behind Stockfish are unhappy about one particular change. They claim that in Fat Fritz 2, the list of Stockfish authors has disappeared. (Until now, a total of 180 authors have worked on Stockfish over the years as can be seen in the latest authors file.)

Silver says he doesn't understand the problem there because in recent Stockfish versions the authors are not mentioned either: "It just says Stockfish devs."

It's a little bit more nuanced than that. Whereas Fat Fritz 2 has the note "Stockfish Devs and Albert Silver (neural network)," Stockfish gives "the Stockfish developers (see AUTHORS file)." In other words, Stockfish refers to an authors file, while Fat Fritz 2 does not. For the average user who simply installs Stockfish on his computer, there is no difference because that authors file does not come with the most popular download options.

"This is not a big deal," says Ginsberg. "It just hurts our collective feelings."

A detail like this is typical of how subtle the clash between Silver and the open-source community is. Another example is a phrase used by Silver in his latest article, where he writes about Fat Fritz 2: "On an identical binary to the one Stockfish 12 used, it was coming out ahead in their head-to-head matches."

As Ginsberg points out, this statement cannot be true for the simple reason that Silver used a differently shaped neural network and therefore had to change the engine binary, telling it that the neural network was of a different size.

"I think he’s really nitpicking there," says Silver. "Yes, I modified the Stockfish 12 binary but except for the larger neural network, everything else was unchanged. The search is essentially the same."

Ginsberg disagrees it's nitpicking: "The phrase 'identical binary' is incredibly powerful, it is referring to a specific kind of experiment that he simply did not run."

Chessbase's view on the matter

Chess.com also asked for a comment from Chessbase. Mr Wüllenweber provided the following statement by email:

• Albert has invested a lot of time, innovative ideas and hardware in the development of his neural network. Training a net is easily underestimated, maybe because it doesn't mean to write code. We believe that his different approach to training adds demonstrable value to the engine, especially to its understanding of chess strategy. We have some articles lined up which will illustrate that further in the coming days.

• We regret that the incredible achievements of the people behind the Stockfish project were not acknowledged loud and clear in the first articles when Fat Fritz came out. We always see this in the metaphor "On Shoulders of Giants." Sure, it was an achievement to climb up there. Sure, Albert might see the landscape further from up there, or at least in different colors. But without the Stockfish giant, one wouldn't see anything.

• We learn from the critical feedback not to conduct comparable projects without the close coordination of all the authors involved. For Fat Fritz 1, there was an exchange with the authors of LCZero to describe our intentions. It was a mistake not to initiate the same with Stockfish.

How strong is Fat Fritz 2?

In its marketing, Chessbase used the words "the new number one," stating that in a comparative match of over 1552 games Fat Fritz 2 "clearly beats Stockfish 12" and "outperforms the previous top engine by over 40 Elo points (as of February 2021)."

Although Silver points out that he prefers his own phrase "the best of both worlds" over "the new number one," he does claim that Fat Fritz 2 outperforms Stockfish in his own private tests. 

Initially, Silver tested Fat Fritz 2 against Stockfish 12, which was released September 2, 2020. However, the engine of Fat Fritz 2 is based on a newer version of Stockfish 12—a developer version that is closer to Stockfish 13, which was released on February 19, 10 days after Fat Fritz 2 came out.

Comparing games between two programs that use different search engines seems odd indeed. Why did he do that?

"Stockfish 12 is the only reference most people have, honestly," says Silver. "Yes, there is a Stockfish dev but most people don’t know about it and the problem is that Stockfish dev is a moving target, frankly. Even I don’t test against it in my personal testing."

Another reason for Silver to test against Stockfish 12 was that it was the benchmark on the computer rating lists. Unlike with human chess, for chess programs, there is no single, universally accepted rating list.

"Look, I was asked to provide a test basically to show how it did against the best programs. On the CCRL [the Computer Chess Rating Lists - PD] and the CEGT [the Chess Engines Grand Tournament - PD], Stockfish 12 was on top of the list. So I had to show what they were going to be getting and how it compared to it. And that was the result I produced."

Once Stockfish 13 came out, Silver compared against it instead and published those results in his February 21 article.

The Stockfish team gets different results. On a larger sample, where both engines were compiled from source on the same machine, their results at 60+0.6s were:

On the latest CCRL blitz list (two minutes and a one-second increment), Fat Fritz 2 is the top engine with two Elo points more than Stockfish 13. On the 40/2 list (games with 40 moves in two minutes), Fat Fritz 2 is ahead of Stockfish 13 by six Elo points, although the January 11, 2021 version of Stockfish on its turn is six points ahead of Fat Fritz 2.

The slowest time control CCRL looks at is games at an equivalent of 40 moves in 15 minutes. Testing for Stockfish 13 has only just begun and there are only 24 games so far: a drawn match against Dragon by Komodo. At the top, see again see the January 11 version of Stockfish, although Fat Fritz 2 is as high as the second place.

The well-respected Stefan Pohl, seen as the most independent computer program tester, currently has Stockfish 13 at four Elo points stronger than Fat Fritz 2, with the January 11 Stockfish coming two points higher.

"I’ll be honest with you, I have results all over the place," says Silver as he refers to a wide range of lists. "I don’t know what to say. Everything is going to depend on the openings, the conditions, whatever."

About the question of why the results are so different, Silver says: "Not all lists use the same sample size and there might be different conditions. I did a breakdown of one of the lists that produce the results and I found out that ECO A openings (mostly English, Benoni and irregular openings - PD] represented 60 percent of the games. I thought: what if they had a more balanced distribution of the openings, would it still look like that?"

Ginsberg suggests that Silver's divergent test results are a combination of a relatively old processor and the time control. Silver used an AMD FX-8350 (4GHz), using one thread (each side) with 512MB hash.

"This is a processor that was released in 2012," says Ginsberg. "So a test with games of 90 seconds and a one-second increment is going to be effectively hyperbullet. We haven't done any testing at all at these speeds, and we all use modern CPUs."

As of February 24, 2021, Chessbase has removed the "new number one" claim from its website.

Chessbase has changed their website to remove the claim that "Fat Fritz 2.0" is the "new #1" chess engine. pic.twitter.com/01ia7JQcrU

— NoJoke (@NoJokeChris) February 24, 2021

"We feel that customers buying Fat Fritz 2 get very little added value for money," wrote VandeVondele in his statement. So what is that added value? Why would a customer pay those 99.90 euros?

Silver: "Ultimately I don’t think it’s all about Elo. If it was, it would be really boring, frankly. If you’re talking about a modest difference it shouldn’t be the absolute criterion. Sure, it’s a marketing aspect, there’s no question. But if I am choosing an engine to analyze with, I know super GMs who still analyze with Leela even though its Elo is worse than Stockfish NNUE. Why? Because it provides them with new perspectives, different analysis, different positions, different continuations. They relish that. They don’t care if on list X, Y, and Z it’s 50 Elo behind or ahead. I have done enough tests and enough games to say that FF2 NNUE compared to Stockfish 13 NNUE in the same search, using the same binary, uses different continuations, more than enough to say that they are clearly different."

License issues

Besides Chessbase's claims about Fat Fritz 2's playing strength, another thing the open-source community is upset about is the alleged violation of the GPL license by Chessbase.

In his statement on behalf of the Stockfish developers, VandeVondele wrote:

Selling Stockfish derivatives is possible with the GPLv3 license we grant, but not without requirements. In particular, the license states that if one redistributes a program derived from our work, the corresponding modifications of our sources and all information needed to build that program must be made available. Only after explicitly informing Albert Silver (the author of the net in Fat Fritz 2) of a license violation have matching C++ sources, but not the net weights, been made available. Obviously, we condemn the approach taken.

This part of the clash between Chessbase/Silver and the Stockfish developers seems to be the result of the way Fat Fritz 2 was originally released, in combination with a different interpretation of the GNU General Public License (GPL) under which Stockfish has been released.

In short, the GPL allows for using, sharing, and modifying software, and that includes the right to do so commercially, as long as the derivative work is distributed under the same or equivalent license terms.

When Fat Fritz 2 came out, the online download version consisted of a single installation file that would create a package on the user's computer that included the search engine and the neural network.

"I did it because, one, the neural network is unusable in any other environment, and the second reason is this was what Stockfish was doing, this was standard procedure," explains Silver.

However, because everything was offered as one file, the whole package fell under the GPL. As VandeVondele explained, this meant that everything, including the source code and neural net, had to be made publicly available.

Chessbase stated that it had made the source code of the search engine available, but so far, it hasn't made clear where it can be found. According to Silver, the (link to the) code only has to be provided upon request. In reality, the license says you have to offer the source code "in the same way through the same place at no further charge," whether a link is requested or not. Access to the neural network wasn't provided either.

When confronted with the GPL violation by VandeVondele, Chessbase updated its product three days after its initial release. Since then, the search engine and the neural network are provided separately.

"I was informed that the GPL allows for a breach to be corrected within 30 days so this is just a technicality," says Silver, who isn't planning on giving away the code and other information behind his neural network. He does offer a weaker, earlier neural network for users who want to compile their own copy.

The reason for separating the search engine and the neural network is that according to Silver, once it has been separated from the binary, the neural network is no longer subject to the GPL.

While acknowledging that such matters can only be decided in court, Ginsberg says he disagrees: "You can't un-GPL something. Once it's out there under the GPL, you can't undo that."

Ginsberg noted that he purchased Fat Fritz 2 on February 22 and that his download still consisted of one installation file, saying: "The GPL applies to that file because that file is object code that includes the Stockfish binaries."

It's not the first time that the Stockfish developers have accused a commercial chess program of violating the GPL. A recent example is Robert Houdart's Houdini 6 that, they claim, is a Stockfish 8 derivative without providing the sources on request. This has resulted in the TCEC revoking the championship results of Houdini 6.

"I want to think that they weren’t as careful as they should’ve been," says Ginsberg about what he sees as a mistake made by Chessbase. "I don’t want to say that anybody was evil here but they didn’t act like scientists. I don’t want to complain about them, I just want your audience to know the truth."

Should you buy Fat Fritz 2?

What remains is the question of whether it makes sense for the average chess fan to purchase Fat Fritz 2. According to Silver, it's the end user who needs to decide: 

"Apart from the interface and other perks you pay for, with Fat Fritz 2 you get a large, different neural network, running absolutely in Stockfish, no question about that. It will provide a different evaluation, it will provide a different search. Remember that the search is not the same. It is guided by the neural network itself. Just like you would analyze moves and you might be analyzing at the same speed as somebody else, you’re going to be focusing on different moves because of how you look at chess. It’s the same thing here. Some lines will go deeper, some lines less, and all of that is the neural network that decides this. The speed will be the same but the lines and the moves shown are specific to the neural network itself. Is it gonna play the same? Sometimes yes, sometimes no. But it’s going to provide different ideas. Is this enough value for you? Is that enough to justify buying it? It’s a personal call. I can’t make the call. For me it’s of course it is. For some GMs it’s of course it is. For an average player who doesn’t care about such things, the answer is no. And if the GUI adds nothing to their experience, then really, no. It’s about different opinions, different perspectives, different analysis, different ideas. The value of that he’s going to be up to you to decide."

Ginsberg thinks the race car vs. driver analogy is fair but ultimately doesn't work in Chessbase's favor: "His neural network is worse because it loses Elo. So you’ve got the same car, two drivers, and one guy keeps losing. It’s the driver. Claiming that the net is better is very annoying to the Stockfish developers who have spent so much time making the best net, getting the best driver they could find. So our view is: we have the best car—he doesn’t argue with that—we actually have the best driver, and it’s still our car."

Interestingly, Ginsberg doesn't mind at all that Chessbase is making money from Fat Fritz 2: "I am thrilled that they are making money and I hope they make a lot. Because if they are making money it’s good for chess. I love chess. I want chess to be popular. It doesn’t bother me at all that they’re making money. Not in the slightest."

The conclusion, therefore, could be that in the end, chess wins. Commercial and open-source chess programs will continue to be developed and will get stronger as time passes by, leading not just to higher ratings but to more discoveries, more outstanding games played by computers, and more understanding of chess. Whereas computers need humans to make new steps, humans are helped by computers to appreciate the ancient, royal game even more.