Chess.com Lessons vs Improvement: The Stats

Background

On January 1, 2018, Chess.com announced a new lessons feature.

The Chess.com lessons originally ranged in level from players new to chess (0-400 rating) through advanced (1000-1600 rating). In May of 2018, the site introduced the mastery series for players pushing beyond the advanced level.

Traditionally, chess players have improved by practicing, reading books and analyzing with stronger players. Now some new methods are being developed for chess improvement. A new field is emerging for self-taught chess players, and the Chess.com lessons are an exciting platform for chess improvement.

Chess.com wanted to evaluate the effectiveness of the new lessons platform for players rated below 1000 and who did not have previous experience on the old lessons platform. We hypothesized there are a few other important factors to be analyzed. It was hypothesized that younger age, lower initial rating and number of games played may all be associated with rating gain.

This study will evaluate the effectiveness of Chess.com lessons on six month rating change, with and without adjustment for other important factors.

Methods

Study Population:

The study participants were Chess.com members for at least six months and rated below 1000 in one of the three main rating categories: blitz, bullet, and rapid. Participants were also required to have at least 300 games in a category with their rating below 1000. The last requirement was to have completed no prior lessons. Some exceptions were made for players with a couple lessons if all other criteria were met. The purpose of these criteria was to create a clean set of members who were at a stable rating, in a rating range that the lessons will be helpful, who had little-to-no experience using Chess.com lessons.

The goal was to select 300 members and assign them into three randomized groups: 

A. Members who complete exactly one Chess.com lesson per day

B. Members who complete as many lessons as they want every day.

C. Members who are not selected to be in the study

Of the 1,229 members who signed up to participate in the study, 210 met the criteria. Six of the 210 participants were excluded from analysis due to account closures or desire to exit the study. Groups A and B were initially assigned 100 members each, with the remaining 10 members assigned to group C for analysis.

Statistical Methods:

The baseline characteristics were measured using number and proportion for categorical variables and median and interquartile range for continuous variables. To determine the six-month blitz rating change, the difference between November 1, 2019 and May 1, 2019 ratings was calculated for players with at least 300 initial blitz game and 20+ games played. Similarly, rating changes were calculated for bullet and rapid categories. Baseline variables consisted of age, initial ratings in the three categories, initial games played and if the participants had a chess coach. Games played over six months in the three categories were tracked in addition to lessons completed.

Univariate linear regression was performed on six-month rating change for each of the three categories to assess the associations between rating change and age, initial rating, initial games played, games played over six months and lessons completed over six months. Multivariate linear regression was also performed using backwards selection and keeping variables with p < 0.10. Results were analyzed using Microsoft Excel and RStudio.

Results

The baseline characteristics between the three groups were very similar. The median age was 25 or 26 years old. Initial ratings were within 100 points for all three groups. Games played at baseline were similar but with wide interquartile ranges (IQR). A very small number of participants had coaches so that variable was not analyzed. Blitz was the most popular rating category, with over 300 games at baseline with 72.6 percent of Group A and 65.6 percent of Group B meeting the criteria. 

The rating change in each group among players with at least 300 baseline games and 20 games played in category of interest varied greatly among the participants. The IQR was over 100 for all groups in blitz and bullet. Groups A and B were very similar in terms of lessons completed, with median lessons completed 56 for Group A and 51 for Group B. Group C had a median of two lessons completed, as those participants were not asked to follow a specific lesson plan.

The three groups did not have a significant difference in the Table 3 models, and were combined for overall rating change analyses.

Modeling six-month blitz rating change, blitz games played had a highly significant association (p < 0.001). This suggests that the more blitz chess one plays, the higher the rating increases on average. As hypothesized, the data also suggested that younger players and players with lower ratings will tend to gain more points on average (p = 0.09 for both variables).

Prior blitz games did not seem to matter (p = 0.48), which could be due to the study criteria requiring at least 300 prior blitz games. Lessons completed was not significant in the univariate models (p = 0.22), but there was a positive association with rating change. Putting these variables into the multivariate model, only previous blitz games dropped out using backward selection.

The remaining variables predict that a 25-year-old rated 900 could expect to gain 65.5 points over six months, plus an additional 3.8 points per 60 blitz games played and 14.1 points per 60 lessons completed.

Bullet rating change over six months had less power than the blitz models, and no variables were predictive in the multivariate model. There is some suggestion that over a six-month period, playing bullet games is the most important factor to increase one's bullet rating. Lessons completed had virtually no effect on rating change (p = 0.99).

Six-month Rapid rating change was the smallest sample size of the three models, but still showed very strong improvement for those who completed the most lessons. Lessons completed was the most important univariate predictor (p = 0.003) and the only significant multivariate predictor. The model predicted that every 60 lessons completed resulted in an additional 15.5 rating points over a six-month stretch.

Rapid games played was the second strongest univariate predictor, suggesting that more rapid games played leads to larger gains in rapid rating. The variable was not significant, though, once we modeled it with lessons completed.

The results of the study showed that lessons completed had a significant positive association with rating change in both blitz and rapid over a six-month period.

Completing lessons also appeared to have a stronger association with rating gain as the time controls became slower.

In statistical psychology, it is hypothesized that the brain has an intuitive component and an analytical component. It would be interesting to explore this in more detail with regards to chess. As a chess player learns new information, say from lessons, does that information first get recalled at an analytical level only later to be incorporated into intuitive thinking?

A new year is approaching and new resolutions are about to be made. One resolution that many chess players make is to gain some rating points to hit a new rating milestone. I suggest that you track your training and see which methods work best for you.

A combination of games, lessons, and tactics will go a long way for the self-learners out there!