Chess Mentor Ratings need revamping...

First off, I enjoy Chess Mentor quite a bit.  Silman is one of my favorite authors, and it's a joy to go through all of the different problems. 

But I believe Chess Mentor loses some of its value due to what I feel are the wildly inaccurate ratings it gives you.  First off, a little background.  I'm a 1534 rated USCF member.  Quite average.  

While I do believe that Chess Mentor has helped me, it is giving me a current rating of.....drum roll please......2513!!!

I know myself.  I know my relative strengths and weaknesses...and while I'd surely love to break the 2000 barrier at some point in my lifetime, I'm humble enough to admit that there's no way on this earth that I am remotely close to a 2500 rating.  On a good day, I can say that my playing strength might be around 1800.  On a bad day, in the 1300-1400 range.  So 1534 USCF seems about right to me.

I think I have a good enough sample size of problems I've answered.  A few hundred problems, with a good 150-200 problems in the 1900-2100 range.  It's terribly easy to increase your rating by going through some of the easier problems as well (in the 1400 range). 

So here are a few suggestions:

*When you achieve a certain rating over a period of X number of problems answered, correctly answering a very easy problem (let's say a problem with difficulty of 500 or more points lower than your current rating) should yield no more than 1/4 rating point.  It's too easy to answer 50 problems in a row with 100% accuracy to increase your rating.

*For the medium difficulty problems (let's say those with a problem difficulty of 1600-1900), I believe their difficulty levels all need to be re-evaluated...and lowered by around 200 or more points.  Some of the 1700-1800 rated problems can be solved by OTB 1400 players without too much difficulty.

Maybe...just MAYBE my true playing strength is slightly higher than my actual USCF rating.  But I emphasize the word slightly.

I don't know what the right to change things will be.  I certainly would not expect an online Chess mentoring program to give you a perfectly accurate assessment of one's playing strength...and I believe a delta of +/- 250 points would have to be acceptable.  But as it stands, for me, it's at a delta of 1000 rating points. 

Again, I'll reiterate that I really do enjoy the Chess Mentor program.  But it definitely needs some tweaks.

Carey

My chess mentor rating is up to 2652 now.  :)  I am trying some of the 2300+ rated problems and scoring well.  But my turn-based chess rating is 300-400 points lower than that.  :)

Thanks for the post, Einstein. 

Ok, at least you're basically at master strength OTB.  Given Chess Mentor's inflated ratings, it would make sense that you're scoring at a 2600+ clip. 

I'm nowhere near master strength.  But the point is clear...Chess Mentor's ratings are definitely overrated.

ah... this is just a hard issue. we have tried to please everyone with how chess mentor ratings work, and we can't. some said it was too hard. others said too easy. other said unfair. others said too fair (just kidding).

anyway, we'll need to make some adjustments eventually. but remember that it is just a relative measurement. :)

I  concur. With hints (which aren't penalized too heavily) I could get to 2650+ without much difficulty. Moreover, some problems should not be rated. For example, opening problems where the author wants a specific move despite other sensible alternatives. Even the TacticsTrainer seems a bit inflated. My rating was 2100+ but just like ChessMentor, it is in a vacuum. It does not correlate well with Chess.com's game ratings or my OTB rating. They should be adjusted.

I just figure consider it a compliment :P.  If a computer thinks you play like a 2600... then just agree! :)

Hi Erik,

Thank you for the honest feedback.  I agree, that it can be a beast to figure it out, and certainly an ongoing challenge.  But as a business owner, I suspect that Chess Mentor will be one of the main revenue sources for chess.com.  So it behooves CC to put in the effort to make it as accurate as possible. 

I think the feedback of actual OTB-rated players would be invaluable.  It's too difficult for people who only have CC ratings (but no true OTB tourney ratings) to accurately judge Chess Mentor.  (I think it's entirely possible for people with no actual tournament training to score well into the 1500 online chess rating range...)

I understand some of the difficulty.  The Chess Mentor problem makers need to constantly come up with new problems and may not have the time to poll actual OTB-rated players for their opinion on certain problems. 

Perhaps adding an optional poll after every problem can help your team determine the relative accuracy of the problems over a long enough period of time. 

The poll could be EXTREMELY simple. 

Example: This problem was rated at 1500 difficulty.  Is this rating level...

a) too high.  it's more difficult than 1500

b) too low.  it's far easier than 1500

c) just right

Just a thought.

Regards,

Carey

Hahah...yeah, it's certainly good for an ego boost!  LOL.

The ratings in chess mentor are some numbers. Don't count on them too much and do not select lessons based on these numbers. You will find lessons rated ~1600 which might prove hard even for players >2000, for instance... Try to do the problems without hints or other sources of help.

Also, if you play well a short sequence of moves as in mentor, it does not mean that you will be able to play an entire game at the same level. That's why the ratings you get in mentor cannot be compared directly with the OTB ratings.

What is the point of having ratings if they are not at least reasonably accurate and cannot be roughly compared to your OTB ranking?

The whole point of having these ratings is to give you a barometer for your Chess progress. 

For the record, I haven't seen any puzzles in the 1600 range that would be challenging for a player >2000.  But I've seen MANY problems in the 1600-1700 range that a 1300-1400 OTB player could answer.

First, an OTB rating is obtained by playing long games with humans. There is no substitute for that. Playing a whole game is very different from spotting some correct moves.

Second, mentor is a learning tool. That's why it has hints. To get a rating close to OTB then there must be no hints and, once you make a move, that's it, you should not have the chance to go back. Is that what you would like?

"For the record, I haven't seen any puzzles in the 1600 range that would be challenging for a player >2000.  But I've seen MANY problems in the 1600-1700 range that a 1300-1400 OTB player could answer."

Very very hard to believe. Check out for instance this puzzle.

http://www.chess.com/chessmentor/view_lesson.html?id=5634

My computer rated ~2800 can't solve it. Can you solve it? But give a full line, not just a single move. And, of course, without looking at the hints or other available help.

Just as you can't compare the actual numbers of a USCF OTB rating, an ICC blitz rating, a Chess.com Turn-Based rating, or an ICCF CC rating, I think there's not much point in comparing a Chess.com Turn-Based rating to a Chess Mentor rating or a Tactics Trainer rating. Just treat them as individual rating systems the same as you would have to do if Chess Mentor ratings ran from 1-100 and Tactics Trainer ratings ran from A to Z.

 You know, it might not be a bad idea to give chess mentor and tactics trainer it's own ratings systems (like stars, or letters).  As someone mentioned earlier, I also feel that these two training system's ratings should be seen as individual ratings on their own...

I think you're not understanding my point.  Yes, I clearly understand that it's a learning tool.  (Why else would I subscribe to it?)

My point is, if it's not even accurate to within 1000....let me say that again....ONE-THOUSAND rating points, then it fails at one key aspect: giving me an idea of what my actual progress is.  Does solving 2100 problems tell me that I'm making progress in my Chess thinking, or am I simply running with the rest of the OTB 1600s out there?  How are other people doing?

For the record, the puzzle you listed there was pretty darn easy to solve.  It's such a common theme with Sicilian dragons.  Rip open the h-file, and as Fischer put it, "sac, sac mate." 

It is NOT a tactical puzzle (which probably explains why your 2800 computer couldn't solve it).  But h4 was a pretty natural move, and something that a 1500 club player could have come up with relative ease.  Almost any player who has ever played the Sicilian dragon (on either side) with any sort of regularity could solve that problem.

This puzzle was not difficult at all.  It's a very common positional

Fair enough, nimble.  It's clear that there will always be disparity...and that you can't always expect to have the same range.  But at LEAST the other Chess ratings are somewhat accurate with what I believe my actual playing strength is (+/-300 or so, I'd estimate).  

But as I mentioned before, the ratings should give you SOME indication of what your progress is.  How are other 1500-1600 OTB players doing?

Solving problems and having Chess Mentor tell me that I'm approaching 2600 gives me a hugely false sense of progress.  One of the huge perceived values of Chess Mentor is being rewarded (and sometimes penalized) with tangible rating points.  And I believe Chess Mentor loses a lot of its value with out-of-this world ratings estimates.

All rating systems are flawed (in practice). As long as Chess Mentor provides a fair estimate of the trend in my improvement, I am happy!

careyfan: if that puzzle was soo easy, why don't you give the continuation? Even a monkey can make a simple one pawn push. What happens after that? Why h4 is good but pushing the g pawn (which is also common) is wrong? I found that puzzle extremely difficult, I spent a lot of time on it and I could not solve it. Just pushing a pawn and hoping that this is OK ... well, this is not chess.

Awesome avatar, btw.  I'm a HUGE celtics fan.

It's not a fair estimate of the trend for my improvement.  That's my whole gripe.

Whoah there, Costelus...relax there, buddy.  I'm not here to say that I am god's gift to Chess...and that all of the puzzles are too easy.  I simply want the ratings to be modified...and would really like for Chess Mentor to give me a better indicator of my progress.  Do you really believe that Chess Mentor's rating system cannot be improved?

Honestly, the puzzle was not THAT difficult.  The puzzle asks for you to give the best move in the situation and gives you a score based on ONE move. 

g4 makes sense, but there's a simple reason that h4 is better-- it's faster and forces the file open immediately.  The Sicilian dragon used to be my favorite opening, so I'm very familiar with this concept.

The puzzle didn't ask for a further continuation. Perhaps it would've been an appropriate level of difficulty IF they asked for more moves. 

Anyhow, my point remains...I'd like for the Chess Mentor puzzle writers to come up with a more accurate way of measuring one's progress and/or applying more accurate difficulty ratings to them.  I've offered up a few suggestions.

The key word here (that you seem to be missing) is trend, or relative.  It doesn't matter if your Chess Mentor rating is 7, 290, 1700, or 589281!  If your rating used to be lower than it is now, you are improving.  If it used to be higher, you need to work harder.  It's as simple as that!