Chess Mentor Ratings need revamping...

I really don't think it's THAT simple.  It's obvious that the makers of Chess Mentor were trying to have the ratings coincide with people's actual OTB strength. 

But it seems like they simply used judgment calls in determining a puzzle's difficulty.

I'd like to know where I stand relative to other players in a similar skill range.  I've offered a few ways that it could be improved.  I know that pragmatically, it's too difficult to poll thousands of players on before posting the problems.  But at least they can start polling the people who ARE answering them.  Implementing a polling system is a good (and low-pain way) to start helping them to improve Chess Mentor overall.

I don't think its obvious they tried to conincide with OTB ratings at all, actually.

And what do you mean where you stand relative to other's of similar strength?  You are about as equally skilled at chess as others of similar strength to you.  If two people study with Chess Mentor, the one with the higher rating is likely better at chess (or whatever aspect you are studying).  You are asking for a feature that is akin to saying to your chess tutor "How good am I relative to the people that other guy is tutoring?", when the two tutors have never met or shared students.

Thanks for the feedback.

I'm not asking for the world.  Take the Tactics trainer, for example.  At least it gives you things like:

*% of people who answered correctly

*Average length of time

*Rating

I at least have a pretty good idea of where I stand against other people with Tactics Trainer.  Is it not possible to implement similar type tools with Chess Mentor?

Can I gauge whether or not I'm making progress with Chess Mentor?  Certainly I can tell that if my rating has plateaued, I'm not making any measurable progress.  But I would like to see how I stack up compared to other Chess Mentor users, what their average ratings might be, etc.  I don't think this is a difficult thing to ask for.

From what I have seen of the Chess Mentor, I would do away with asigning a rating result completely.  It overrates me significantly every time, and I don't think that the appication lends itself to providing an accurate 'result rating' at all.  I would assign a solid, reliable rating to the problem itself, then leave the user to determine for themselves how they fared.

The tactics trainer does pretty well I think, though the penalty for a missed problem is VERY high.  But, if you take too long to answer correctly you don't get much and can even lose points if too slow against the average time, so the pressure to risk a wrong answer is high, though the price of being wrong is huge.

Hi D_Black,

If they don't make any changes at all, then I would tend to agree with you-- do away with the ratings altogether...at least until they improve it.

Overall, I'm pleased with the types of problems Chess Mentor gives me, just not the type of unrealistic feedback I get from it.  Tactics Trainer, despite its flaws, does a much better job. 

You're right about the harsh punishment for picking the wrong move...although I don't necessarily agree that it's a bad thing.  In those positions, there are definite winning moves...and if you don't spot them, you give up the chance or even give the advantage back to your opponent.  So it's kind of justified, in a sense.

Coming from someone who has spent literally hundreds of dollars on various chess software, books, videos, coaches, etc. Chess Mentor (along with CT Art) by far offers the best chess instruction all the way to the intermediate player. The rating is rather pointless, and nobody should worry about it at all. As long as you are learning from the material in question, there should not really be a problem.

Oh, and for the ones complaining that their chess mentor rating is too high...it is probably because of the courses they have taken. I would challenge anybody to do the Roots of positional understanding course not lose points everytime.

Coming from someone like yourself (I too have spent hundreds of dollars on books, videos, etc.), I also believe that the lessons are good.  I never questioned the quality of the lessons...only the ratings.

But that doesn't mean that the rating system can't be improved.  We are paying a monthly service fee for it, so it is our right to provide feedback and ask for improvements whereever possible.  Again, I'm not asking for the world...only to improve the rating system so that we can more accurately measure our progress.  If the rating system is pointless, as you say, then why implement it at all?

Ok, I grant you the fact that the roots of positional understanding lessons are pretty challenging...and I feel that the difficulty ratings of the problems is more in line with what they actually should be. 

I think the point is that the rating is useful within itself. I mean that if you work your CM rating up to 2400, and then take a course that drops you down to 2150, you need some more work with similar course material. It doesn't matter that your CM rating isn't similar to your Chess.com turn-based rating.

Suppose Erik had set up the CM ratings so they were letter grades from A-Z, with Z being worst. And you worked your CM rating up to C. Would you argue that such a CM rating was no good because it didn't correspond to your Chess.com drating of 1500? It's actually okay if they're apples and oranges.

Right now, your OTB rating is probably quite different from your CC rating, and both are probably quite different from your Blitz rating. So your CM rating is also different. So, what? Those four ratings really are apples and oranges because they rate different things. So what if their ratings are different, too? 

That would be true if the ratings were internally consistent. However, it hasn't been my experience. I've seen several problems of the same rating level but clearly not the same difficulty level. Thus, going up or down doesn't necessarily mean you're stronger or weaker but rather that the problem set is not graded the same. The ratings appear to be assigned by chess.com staff instead of being calculated from the player pool and therein lies the problem.

Bingo!  Finally someone who gets it!  Yes, I believe this is a big part of the problem.

Wow, if your complaint is that the hundreds and hundreds of Chess Mentor lessons aren't graded consistenly from one to the other . . . whew!  Anyone who has graded a hundred essay exams knows how dauntingly impossible that task is. First, this is a very subjective thing. Second, being subjective, it shifts in response to changing points of view as more and more problems are graded--even within the same grader. Third, there may be multiple graders.

"Calculated from the player pool"? These are not competitions we're talking about here. They're chess problems with explanations designed to provide a learning experience. They do that wonderfully. Take the learning and run.

I haven't the foggiest notion what my CM rating number is. Since it has nothing to do with any other of my chess ratings (or non-ratings)--Chess.com, USCF, ICC, other websites, etc.--I don't care what that number is. I don't even care if Chess.com does away with players' CM ratings entirely. I can tell just by using CM where my weaknesses are without seeing a number. Actually, it may have taken this discussion to make me see that clearly. I feel like I'm even less interested in my CM rating number now than I was when this discussion began. So, I'll leave the discussion to those who do care about their CM rating numbers.

they WERE calculated by the ratings pool, but then we changed it due to other complaints :) ugh. this has been a big ugly circle... first people said they lost too many ratings points for things. so we adjusted. then that caused the problem ratings to deflate. that caused people to feel like chess mentor didn't have high-enough rated problems. so we fixed the ratings where they were. and now this. frustrating! maybe eventually we'll add back in the ratings, add back in the right balance of penalties, etc. we'll see...

Hi Nimble,

Your points are well taken.  I certainly do NOT expect the CM management to "grade" every single problem.  I understand that it's logistically too complicated.  They did enough by giving it their subjective baseline ratings...but I believe that they can do more.  And who knows...they probably ARE doing more from behind the scenes.

I suggested that the problems be evaluated from a pool of Chess players, not too much unlike Tactics Trainer is currently done.  Logistically, I do NOT believe that this is too difficult to achieve.  The Chess.com staff has proven that they have the intelligence, talent and means to implement a wide variety of measuring/analytical tools, which is part of what makes chess.com so much more valuable to me than ICC (where I was a long time member).  I actually cancelled my ICC membership because I was able to glean a lot more valuable information out of Chess.com.  And I'm fairly sure that one of their peripheral goals is to wrest away some of the ICC membership....

If they can come up with a better way to more objectively evaluate the problems (and hence increase the accuracy of the ratings), then go for it! 

Chess.com is young and has captured a pretty strong captive audience based on the fact that they have a good product.  But they are still a work in progress, and they certainly have lots of room to improve.  They need ways to keep their services sticky, especially for paying members.  And I'm merely suggesting ways that they could improve the service for people like me.

Having a more objective means to measure the difficulty of the CM problems is great not only for the player, but also to the CM folks as well.  Plus, it's a relatively cheap and easy way to grab data from their customers. 

Nimble, If you personally get enough value out of the problems alone without any tools to measure your progress, then that's great.  I have no problem with that.  Again, I'm only suggesting (as have others), that the rating tool can be expanded and improved.  They already have many great tools (games explorer, tactics trainer numbers, statistics, etc.) and have successfully differentiated themselves from ICC and other game sites by doing so.  Improving one aspect of their CM product would certainly seem in line with the types of offerings they currently have.

Erik,

Thanks for your feedback.  Frustrating as it may be, as you know, continually going back to improve the product is a hallmark of all successful web-based service companies.  And I'm very happy to see that the CC staff is continually looking for ways to improve.  =)

ICC got stale for me after a while.  But CC has rejuvenated my love for Chess.  So I believe you're on to something.

To add to your note, yes...this is precisely what's needed! 

Not to look too far into the future, but perhaps you can even apply ratings to specific aspects of a Chess player's game, i.e.- Openings: 1750, Strategy: 1900, Tactics: 1445, Overall: 1695.  

Thanks for taking the time to read this, Erik.

Careyfan-

Well, I certainly can't disagree with you that better is better, of course. And if better ratings would help some people, that's great, too. I'm not a programmer, so I don't know how much time and trouble we're talking about here. I can't evaluate the cost-benefit, or where it should place on the list of priorities regarding this website. It could well be that a CM rating is more important to many others than to me. So . . . I hope it all works out well for them, too.

Chess Mentor is a teaching tool.  It isn't clear to me why it needs ratings at all.  What you're doing is studying lessons in an interactive way and you're either learning something or you aren't, mostly depending on the effort you're putting into understanding the material.

The way you measure progress on your chess traing program is your various CC and/or OTB ratings against real opponents...   

Thanks for the feedback, Nimble.  Although I come from a semi technical line of work, I too cannot evaluate the cost-benefit analysis or where it should be placed on the list of Chess.com priorities. Only the CC staff can do that!

I think ratings and/or some other measuring tools can be of immense benefit to people like me.  I'm very encouraged by the fact that CC staff members actually take the time to read (and occasionally respond) to posts from people like us.

I have plenty of other feedback for them on other aspects of their site, but since CM is one aspect that I focus pretty heavily on, I'm selfishly raising points about it.  :)