Does Tactics Trainer make you better? A statistical study

Does Tactics Trainer make you better? A statistical study

Feb 27, 2011, 1:12 PM |

I've been using Tactics Trainer as my main training method for maybe a month now, and I feel like I've been improving. I had also read Blunderprone's discussion of the "Michael De La Maza Method" of chess improvement--doing huge numbers of tactics problems on a specific schedule. For low and mid-strength players, this is supposed to be a great way to improve--some people reported ratings jumps of a few hundred points. This has made me even more enthusiastic about Tactics Trainer. However, I wanted to make sure that doing tons of tactics problems would be an effective way to improve, so I decided to do a little statistical analysis. 

There are a few people who use the Tactics Trainer a whole lot--some have done more than 50,000! I took the 100 heaviest users (all but a few of whom had done more than 20,000 problems) and used the PASW/SPSS statistics software to look for correlations. Here's what I found:

-These players' Online Chess rating is weakly correlated (r=0.183) with the number of problems they've done. In other words, there's a limited but positive relationship between having done more problems and having a higher rating. Here's a graph to make the minimal pattern more visible:

-There is a strong relationship (r=0.671) between Tactics Trainer rating and Online Chess rating. People that are good at Tactics Trainer also tend to be good at Online Chess. Here's another chart:

-Your online rating (r=-0.299) and your Tactics Trainer rating (r=-0.553) are typically higher if you fail a higher percentage of the problems you try. 

-Unsurprisingly, people who do more tactics have a slightly lower success rate (r=-0.187). 

-Doing lots of tactics makes you a bit better at Tactics Trainer (r=0.174). Here's a graph of that relationship:

A few notes and caveats:

-Correlation isn't causation, so I was being colloquial when I said things like "doing a lot of tactics makes you a bit better."

-These numbers may not reflect the experiences of more typical Tactics Trainer users. These people are all extremely high users--people who, like me, have only done a few thousand problems might see a stronger relationship.

-Not all people use Tactics Trainer as a training tool. Some of these users are merely using it for fun and do not necessarily concentrate on getting every problem right.

-The data does not fully reflect the fact that getting part of a tactics problem right gives you partial credit. Your tactics rating might thus be a slightly better measure of how successful you are than your pure success percentage.

-Many of the top users do not play online chess at all. Thus, the final data only reflects the 52 people in the top 100 who had played at least 10 games. 

-The Tactics Trainer is not typically used for the Michael De La Maza Method, which usually allows players unlimited thinking time and uses a smaller set of problems that players repeat. Only the top 2 Tactics Trainer users have more attempts than there are problems in the Tactics Trainer database, so there likely has not been much repetition of problems by individual users (though it is probably higher than the data suggest, since the problems an individual user sees are selected by that user's current Tactics Trainer rating). In sum, then, this was not a study of the MDLM method.

-All of the users studied had very high rates of Tactics Trainer usage, so this analysis only reflects the improvement one could expect from using it a whole, whole lot versus just a lot. In the future, I might add in some mid-level users, with only a few thousand problems (or even a few hundred problems) under their belts, and see if the relationship between Tactics Trainer use and online chess rating is still weak. I suspect it won't be. The mean online chess rating of these high-level users was 1691 (standard deviation 369 pts); the mean online chess rating for the whole site is currently 1339 (standard deviation unknown). I don't know how the site-wide rating compensates for all the people who have only played a few games (they're not in my data set at all), so this may not be a useful comparison.

-Restricting the data field lowers the correlation coefficient, so the correlation between attempts and online rating may again be higher.

-Tactics Trainer and Online Chess are both currently having their ratings systems re-adjusted. I did not make allowances for this.

-Most Tactics Trainer problems are about capitalizing properly on a blunder; this is not the only decisive factor in games. Thus, there may be limits to how much you can improve by studying tactics.

-Message me if you'd like a copy of the data file.