Important questions about chess.com Bots

I'd like to know the following questions about c.c bots:

1- Are they rated according to FIDE or USCF?

2- If their rating is not accurate, what's their error range (+-100 points... +-800 points...).

3- If the rating is more accurate for some bots than others, which bots ratings are more accurate and what is their error range?

4- Is their rating calibrated for classical or blitz time controls (because it's not the same to be 2500 rated in blitz than classical).

5- Are all bots the same Komodo engine configured at different levels? or are they using another engine as a base, which one?

6- How was their rating been determined? Was it by comparing with CCRL list, playing them vs known rated humans... or other methods? which method?

7- What's the difference between bots rating and chess.com rating? is 1100 chess.com rated player expected to win 2000 rated bots as i've seen happen?

8- Are different bots expected to have a different style of play? (so far according to my experience they all play the same style...).

(I don't know if this is the correct forum for this topic. If it's not, i'll change the topic to the appropiate forum, if that's possible).

Thank you in advance

The bot ratings are not based on rated play. Chess.com used to allow computers to play rated games but some people farmed ratings points off lower rated computers, distorting their true ratings.

I don't know how the ratings were set and I've never seen anything from chess.com revealing how they did it. It appears that many are quite far off from actual competitive ratings since low rated players often post about having beaten a high rated bot.

When I've played them, I find that they will play well for a while and then throw out some awful blunders. They aren't completely predictable, however, so a low rated player may beat the Hikaru-bot but a higher rated player may get a drubbing.

Occasionally, they have made programming changes resulting in the bots beating everybody.  People really don't like that.

Hi!

Thank you for your answer.

There are some things you said that baffles me... For example here: "Occasionally, they have made programming changes resulting in the bots beating everybody.  People really don't like that."

By programming changes do you mean changing the strength in the configuration of the engine the bot is using? then it would be configuration changes, not programming, because programming means that chess.com created their own engine (or using a third party open source engine) and are changing the code to determine the strength, which is unnecessary if you just adjust the nodes per second the engine thinks without need of changing the original source code of the engine.

Another thing is this: if people don't like to lose 2700 rated engine, they always have the option to play lower rated personalities, for example 1200 rated bot. I don't think chess.com should fake the ratings just to satisfy the wishes of low 700 rated players who need to believe they are Grand Masters.

"When I've played them, I find that they will play well for a while and then throw out some awful blunders."

For low rated engines (under 1600 more or less), it's difficult for the engine to simulate mistakes. So they gift a piece or do an obvious blunder... That happens with some engines, but not others. It depends how engines simulate the low rating: some engines lower the rating by reducing nodes per second and you don't see a big blunder of a piece gifted. Others do it in a way that looks like an artificial big blunder.

For example, Rodent IV engine configured at 1300 rating mistakes are losing a piece by a forced combination 5 moves deep. However, Chessmaster 11 mistakes are placing a knight in front of your pawn. Usually under 1200 rating Chessmaster 11 almost always sacrifices the knight by taking the pawn at f2 (Nxf2) in the openning, even before you castled, and you take the knight with your king.

For higher rated configurations (above 2200 rating) engines don't play blatant blunders anymore and mistakes are more positional, combinationally forced, etc. Mistakes that are simulated in a more interesting human way.

"It appears that many are quite far off from actual competitive ratings since low rated players often post about having beaten a high rated bot."

I've seen Hikaru Nakamura in youtube videos losing and drawing his own chess.com Hikaru bot. It took him many tries, effort and concentration just to get one won game vs Hikaru-bot (only one won game out of 5 tries). And he's the highest-rated blitz player in the world with a 2900 FIDE blitz rating and the fourth-highest rated rapid player with a 2836 rating.

I've seen also videos of Anand (former World Champion) drawing his bot, and winning with difficulty and effort just one single game. And i've seen GothamChess (International Master) drawing three times his own chess.com bot and he was unable to win a single game, so he gave it up.

This means the strength of the bots here in c.c is Grand Master level, could even be Super Grand Master level, so a player 800 rating saying he beats Hikaru bot (2700 rating), it could be he is just lying and he won using external help with another bot, or it could be he was lucky in one game out of 500 that he lost (or it could be he played a very closed opening with 1 minute time control (no increment) to win taking advantage of internet lagging, which is not a legitimate won game in my opinion).

Usually engines are very consistent, much more than humans (engines don't have a bad day, don't have a headache or feel sad). They always play almost the same strength (+-50 points error). When you test CCRL engines in Arena, the percentage of won/lost games are almost exactly the predicted outcome according to their rating CCRL list.

I've measured in Arena Chessmaster 11 personalitites with different CCRL engines and i've determined the real rating of Chessmaster 11 personalities acording to CCRL. Some personalities are off by 100 points, others by 300 points. But never more than 300 points. Furthermore, an engine claimed to be rated 2500 points could be actually 2200 points (it's off by 300 points) but that engine is always 2200 points (+-50 points error). So, the claimed rating is off, but not the strength, which doesn't change. 

So 800 rated player will not beat 2500 personality in Chessmaster11, unles he is lucky in 1 out of 500 games.

The problem with chess.com bots is that i can't test them in Arena, that's why i would like to know their real rating and how it was measured, if it's reliable... Otherwise, to be honest, instead of playing fake ratings in chess.com i'd rather play CCRL list engines. But i'd like to know how accurate is the rating of chess.com bots because i like the c.c application, the website, the analysis, to play c.c bots in my tablet... etc.

"Chess.com used to allow computers to play rated games but some people farmed ratings points off lower rated computers, distorting their true ratings."

I know, i've seen a youtube video of Hikaru trying to beat his own bot. His own bot was rated 1800 back then, and that bot was beating Hikaru over and over again. He got a draw after 5 tries. That's not 1800 at all, that bot is 2700 minimum.

Fortunately, now i think Hikaru bot rating was fixed, since now it's +2700 rated. But i still don't know the exact rating value.

I know bots are as strong as GMs (and some 800 player saying he won is just lying or lucky in one game), but i'd like to have a confirmation of how reliable are chess.com bots ratings, how much is the error (+-300 points? +-500 points? etc.)

I hope c.c don't fake the engines rating just to please 700 rated players, since 2000 rated players will complain about bots being too easy with fake rating

From time to time I have checked the games played by the engine, and i've seen the engine lost vs 600 rated players and won vs FM and IM masters. And the 600 rated player had 99.9% accuracy, so i know he used external help. So we know c.c bots strength is GM strength and consistent. What we don't know is the exact rating value. For example, Hikaru bot for sure is more than 2500 rating. But is it 2700? 2550? 2890?.... we have no idea and that's what i'd like to know.

Who in chess.com do you think i could ask about the questions i posted in this topic? After 8 days you are the only one who answered, and you said you don't know much about all this...

If we are going to have access to most of the bots for payed version only, these questions are important. Playing bots is the only reason i made premium account, so i'd like to know what am i playing and what am i paying for.

Thank you very much in advance