I've gotten a number of puzzles lately that have absurdly low pass rates for their rating, and it can not be chalked up to low sample size. The number of attempts is large enough that any functioning elo-like system for rating puzzles should have stabilized to its rating and a near-50% pass rate. Here is an example puzzle (996888) with a screenshot of the rating as of Nov 29, 2023
In the comments of the puzzle, another user says that the rating was 2978 at 1533 attempts on October 6, 2022. That means that the 1669 attempts between 10/6/22 and 11/29/23 somehow brought the rating down (and presumably, but not necessarily, the pass rate up, but still only to 6%).
What is the reason for this? Are some more-recent, low-rated users solving this puzzle easily and driving the rating down, or is there some flaw in the algorithm that calculates the rating?
It is interesting to follow the comments made in older puzzles to try and understand the ratings of puzzles.
I think this one has an easy first move and a hard second move. I would guess that 90+% of people get the first move and only a very low percentage get the second move.
It's entirely possible that the rating reflects a manual readjustment by Chess.com and not one brought about by algorithm adjustments.
https://support.chess.com/article/2016-how-do-puzzle-ratings-work
How is the rating of each puzzle decided?
You may notice that when you get a puzzle wrong or right, your Puzzle Rating changes, but the rating of the puzzle does not. This is because the rating of each puzzle is locked after a certain point.
When a puzzle is newly added, its rating is determined by who is able to solve it, but after a set amount of time, the puzzle rating becomes locked in place and does not change anymore.
In some cases, if a puzzle rating is found to be too high or too low, Chess.com may adjust the rating to be more accurate.
I've gotten a number of puzzles lately that have absurdly low pass rates for their rating, and it can not be chalked up to low sample size. The number of attempts is large enough that any functioning elo-like system for rating puzzles should have stabilized to its rating and a near-50% pass rate. Here is an example puzzle (996888) with a screenshot of the rating as of Nov 29, 2023
In the comments of the puzzle, another user says that the rating was 2978 at 1533 attempts on October 6, 2022. That means that the 1669 attempts between 10/6/22 and 11/29/23 somehow brought the rating down (and presumably, but not necessarily, the pass rate up, but still only to 6%).
What is the reason for this? Are some more-recent, low-rated users solving this puzzle easily and driving the rating down, or is there some flaw in the algorithm that calculates the rating?