Crowd Chess

Hello users of Chess.com,

I am a researcher from University of California, Irvine studying social aspects of technology. As part of our research, we are looking to test whether groups of players would be able to beat a computer at chess. Although we know that computers are commonly able to defeat human players, we wish to test whether the combined wisdom of groups of competent but diverse people would be better than an individual player and thus able to beat computers. We were thus wondering if perhaps members of Chess.com would like to help us test this hypothesis by anonymously picking multiple chess moves that can be averaged and then played against the computer. If you are interested, the game will follow a one move per day play style, and will begin Monday, January 11th. You may sign up at http://lotus.calit2.uci.edu/joel/chess/index.php . If you have any questions, please post them below.

Regards,

Robert Beckman

Generally groups of people playing chess do better than any one of them playing alone, but even groups of people can be beaten by a single grandmaster (see: Kasparov vs The World). And since computers can beat grandmasters with relative ease now...

That's not an entirely accurate comparison, it's true the the best computers can beat GMs, but that's at OTB chess, correspondence games like this would favor "the best GMs" (even if the GMs couldn't use computers to help check).  For example, one of the top 10 GMs by themselves in OTB play easily have 50% of their moves agree with Rybka already (obviously OTB moves are made only under their own power). I would guess a team of GMs on top of time to reseach moves could easily beat a lone computer.

It is interesting though, like you said, groups usually do better than a lone person.  Even outside of chess, like guessing weight or number of m&ms in a car, interesting stuff.

The theory behind this idea can be found in economist James Surowiecki's "The Wisdom of Crowds." For a little light reading it can be summed up at http://en.wikipedia.org/wiki/The_wisdom_of_crowds . Basically, games such as Kasparov vs The World are not quite applicable to the theory for a couple of reasons:
-The game was not decentralized. The crowd knowledge idea requires everyone to think independently, otherwise information cascades could occur and skew the input. The simple voting scheme as well as the discussions players had over moves most likely created some of these cascades.
-The game relied on a plurality vote. This assumes that everyone's knowledge of the board can be summed up into one single move, which isn't the case. Although strategy and tactics cannot be fully integrated into a web based system, one can get a more accurate representation of a players knowledge of the board by asking them their best move, followed by their second best, and so on, and then averaging by a Borda Count or a Modified Borda Count. The system we've developed asks for a players four best moves (through trial and error, four seems to be the best way to get enough relevant knowledge while not over burdening the player).
You will also note, however, that despite these flaws in the system, Kasparov noted that he "spent more time analyzing this than any other game," implying that the system was possibly on the right track.

i can think of positions in which there are not four moves to do, especially four good moves.

In such situations where there are not four moves, the system will truncate your choices. In situations where there are not four good moves, one would hope that the averaged move is a good move.

Will you make an announcement on the result of the game here at chess.com? I hope so. At least the game score, though it would be even better to see more details about the votes.

i really think that the computer will win unless you have a lot of GMs playing.

tonydal with the zinger!

So in essence, the idea is to have a group of cc players entering their first, second, third, etc choices anonymously and then tabulating the totals of all players participating and playing the move with the highest overall score vs the computer?

what happens if there is a tie in the move tally? it's definitely possible.

Although the Borda Count is a good method, there can be some surprising and unforeseen inconsistencies in this system, which could be potentially disastrous for a chess game, especially against a computer. I'm with Arrow on this one, this experiment can not work perfectly, but a computer always will.

Anyways, sounds like fun: I'm in

Basically right. See this website for more info on the system. http://en.wikipedia.org/wiki/Borda_count

what would be more interesting would be to have the crowd play another crowd ... like vote chess. or if you want to be truly evil have a crowd play against itself (unknowingly).

but i think you're interested in the "technology" aspect, so this would be outside the scope of your study.

most people agree that the computer will likely win. a group of average people (or even relatively skilled players) probably cannot come up with the moves to beat a single GM, let alone a calculating computer.

however, i've joined up, so i'll play my part (for what it's worth!) ...

I don't see the logic in this set up at all.  If you take a powerful computer, and play it against many smaller weaker computers (us) operating independently, of course they will not win, they would only be as good as the average ELO strength of the group.  Better, would be some sort of "cloud computing" method in which the humans operate together, spreading the load so as to maximize their effectiveness.  For instance, splitting the overall group into 5 groups perhaps, one only to choose candidate moves, and each group responsible only for analyzing a single move as deeply and accurately as possible.  Organized thought will always prevail over a random vote.

I think if we've got to wear uniforms, they should all be different.

I remember many years ago, GM Quinteros played chess against South Africa, live on TV with people phoning in votes.

He mated a whole country in less than 10 moves.

The sum of mediocrity can never equal genius, IMO

I like rooperi's comment!

All of the evidence mentioned in this thread is anecdotal. Kudos to the OP for putting together a somewhat controlled study of the phenomenon.

I have personal opinions about the ability of a group to play chess together, but I'll hold them because they're really just guesses.

Be that as it may, this system, for reasons already nicely explained, is entirely different. In any case, this is just a fun game theoretic experiment, and there is much validity in the wisdom of crowds, no matter how unlikely it may sound. You'll be surprised by the results of this study I think...

Come now, do you honestly think this can end in anything other than "we get our asses handed to us by the computer"? Unless the engine is, say, Rybka playing at 1-second-per-move, the odds of us beating it are a million to one. No wait, screw that, the odds don't exist. Unless, of course, enough of the human players cheat with engine analysis to have it basically be Rybka vs Rybka. Which, now that I think about it, is probably not actually that implausible. So maybe I will be surprised after all.

In any case, collective intelligence is a term that has been bandied around like it's some sort of cure-all. While it certainly is true that in some studies the "average answer" turns out to be highly accurate, this does not apply to everything, and cannot do the impossible. Beating a full strength engine using any sort of entirely human-based mental power (i.e. no engine help) is pretty much in the realm of "impossible".

(A possible illustration of the failure of collective intelligence is the classic "guess 2/3 of the average" game. Theoretically if all the players were perfectly logical they'd all pick the same value and hence all win. But what are the odds of that? Admittedly the example is not fully applicable in this context though, because it's not really an example of collective intelligence.)