Puzzle ratings and success rates

Just noticed a 1200-rated puzzle in Puzzle Rush that has a remarkable 7.2% success rate! How does the system work that allocates ratings to puzzles and delivers puzzles to users in a way that results in such a surprisingly one-sided stat?

Here it is:

https://www.chess.com/tactics/668520

My point is that roughly speaking this problem should have a rating about 350 points higher than its average users on tactics trainer, on the Elo scale. In order for that to be so, something would need to be stopping it being presented it to users in the appropriate rating range to get a normal success rate. Otherwise, its rating is just wrong in any reasonable sense.

Rating problems is not trivial, because there is an element of time as well as the results (and there is the question of whether partial success counts for anything), but none of these issues explains this sort of discrepancy, IMO.

The ideal would be something like all problems being presented to users in a range that causes it to have a normal success rate. This is compatible with players who take their time achieving a higher success rate and those who move quickly getting a lower success rate, as well as with offering problems in a wide range of difficulty to each user.

Everytime I read a post by Elroch, this tune pops up in my head.

I am humbled. I am listening to the whole soundtrack now.

Here is the list of all puzzles:

https://www.chess.com/tactics/problems

Now, sort them by the Pass Rat (lowest), try some of the least passed ones (usually some stalemate or underpromotion thing) to see that they are often really hard... and look at the actual puzzle rating.

Maybe it doesn't fully apply to the puzzle you gave, but there's a "problem" with how rating is calculated, as it takes into account the number of moves played correctly by people and their rating in a too straight way, I suppose?
I.e. even if puzzles are very hard, if the most of the weight lies in single most important move, especially the move that is not the first move, the rating will be lowered because: people did several/most of the moves.

So, people with low or medium rating passing most or part of the puzzle's moves, even if they miss the most important move and fail (lowering puzzle to below 10% pass rate, or even below 5%), will still be used to conclude the puzzle rating by their partial passes, lowering the difficulty rating of the puzzle.

For more information try reading comments from some of these puzzles with lowest pass rate.

https://www.chess.com/tactics/problems?sortBy=pass_percent_lowest

Since the rating is still exactly 1200 (the pass rate has dropped to 7.1%, so people have been doing this problem), I conclude that although 1628 people have done this problem, its rating has remained fixed at 1200 as some sort of initial default. I can't see why this would be necessary or optimal.

I agree this problem is both somewhat technical to calculate properly and relatively difficult to get right by a lucky guess (Qh6 is surely the gambler's move!)

Still rated 1200, success rate 6.4% from 1787 users! Why don't they generate a rating in a sensible way?

That problem has 5.5% from 2074 attempts now!

(Edit: 4.8% from 2422. Still rated 1200. Why??)

I do not know why the rating of this puzzle seems to have gotten stuck at 1200.

I happen to have come across this forum thread, and I think I have been able to remove the puzzle now from the pool of puzzles in use. It currently says 2732 attempts, let us see if it will really be no longer in use now.

At the other extreme,  I have not the slightest clue how this puzzle managed to get a rating of 2635. There are comments on it from when it had half that rating!

Anyhow, here is that problem. Apparently the problem (and the reason for the high rating and the 16% success rate is that most people tend to pick a different third move to me, with a mate that is one move slower. This really should be no worse than solving it slowly: mate is mate!

https://www.chess.com/puzzles/problem/178473

I have delisted it for now.

perhaps this is a better example

https://www.chess.com/puzzles/problem/583608

update : Now it only has 4% pass rate with 2768 attempts!

Here is a 900-rated puzzle that (IMHO) doesn't even qualify as a tactic! What I mean is that the opening move achieves a winning advantage, but there is no quick finish (I really haven't looked deep enough to see when white wins material). The second move in the puzzle is a tactic, but a more sensible response like R5e7 leaves the resolution far from clear.

https://www.chess.com/puzzles/problem/978320/practice

it now has 4% success rate lol

It tries to lure people away from mate by giving away the queen

Puzzles like those give people trying to get a new puzzle rush highscore a hard time because you'd just get stuck on an 1100 puzzle and get it wrong =)

I don’t think anyone is attempting that one anymore