How are the puzzles rated?

Sort:
JockeQ

Is the rating (difficult level) of the puzzles adaptive and automatically adjusted based on how many people who solves it (taking into account their puzzle rating)? That I think would be the most logical thing to do, similar to the elo-rating when two players meet in a game But I think this can not be the case, because the % solved statistics differs too much.

With an elo function all puzzles would (over time when enogh people have tried it) get pretty much the same %, just small differences because of the time factor and the points you get for partly solving the puzzles. But as it is now, % figures typically varies between 30% and 65% and sometimes even outside this range in rare cases going as low as 20% or even lower. And according to the stats the puzzles has been tried by thousands of people. One puzzle I did had only 7% solving rate, it was an extreme case but I think it shows that it must be some other factor as well other than normal elo-formula and I am curious to know exactly how it's done.

Martin_Stahl
JockeQ wrote:

Is the rating (difficult level) of the puzzles adaptive and automatically adjusted based on how many people who solves it (taking into account their puzzle rating)? That I think would be the most logical thing to do, similar to the elo-rating when two players meet in a game But I think this can not be the case, because the % solved statistics differs too much.

With an elo function all puzzles would (over time when enogh people have tried it) get pretty much the same %, just small differences because of the time factor and the points you get for partly solving the puzzles. But as it is now, % figures typically varies between 30% and 65% and sometimes even outside this range in rare cases going as low as 20% or even lower. And according to the stats the puzzles has been tried by thousands of people. One puzzle I did had only 7% solving rate, it was an extreme case but I think it shows that it must be some other factor as well other than normal elo-formula and I am curious to know exactly how it's done.

 

The puzzle ratings are adaptive. When solved successfully their rating drops and failed it goes up. It's treated as if the played a game against a player with the rating of the solver.

JockeQ

OK this seems very logical to me. Except when looking at the stats, it still doesn't make sense. For example this puzzle:

https://www.chess.com/puzzles/problem/2088424

Rated 1985 but very simple and solved correctly 76%. When solved % is so high, why don't the rating of the puzzle go down? I don't exactly understand how the score is calculated but I don't think that the negative score you get when failing 2s more than 76/24 = >3 more compare to the points you get solving it. 

Martin_Stahl
JockeQ wrote:

OK this seems very logical to me. Except when looking at the stats, it still doesn't make sense. For example this puzzle:

https://www.chess.com/stats/puzzles/jockeq

Rated 1985 but very simple and solved  correctly 76%. When solved % is so high, why don't the rating of the puzzle go down? I don't exactly understand how the score is calculated but I don't think that the negative score you get when failing is more than 4 times of what you get solving it. 

 

A lot of it depends on the ratings of the solvers. For solvers, there's a time element that impacts the rating too, but 15-16 loss is the most common that I've run across and getting that high on a correct puzzle is rare.

 

I think the puzzle rating itself is a straight rating vs rating calculation.

JockeQ

But you never get less than +5p when solving a puzzle. Which means you need to get -16p for a failure for the math to add up. Which might be possible except you dont get in average +5p. 5p is minimum score, I for example got +12p.

 

So it can not be normal elo formula. And you would come to the same conclusion if you looked at the other extreme where %solved ratio is very low (there are some rare cases even below 10%).

This is not very important to me, it is just that my mathematical mind can not make sense of it. But I continue my life with ease without getting an explanation. 🙂

Martin_Stahl

The formula on rating changes for members is not Glicko, though it could be the max possible amount is.  My understanding is the puzzles themselves are rated through Glicko and solve attempt results.

JockeQ

No matter what exact algoritm is used, if it's based on elo principle the average point you (as well as the puzzle) gets must equal 0. Otherwise there would be "inflation".

Martin_Stahl
JockeQ wrote:

No matter what exact algoritm is used, if it's based on elo principle the average point you (as well as the puzzle) gets must equal 0. Otherwise there would be "inflation".

 

My highest puzzle rating was over 3000 ...  There's a lot of inflation happy

JockeQ

OK maybe "inflation" was not the best wording. If the average score for a puzzle just is a small fraction of a point it will it will eventually over time lead to "inflation" played by thousands of players. But if the average points given to a puzzle would be something like one point or more the rating would increase very fast.