State of the Bug Bounties - Feb. 25, 2021

Hello! Holá! Hallo! Bonjour! 你好！

Ever since Chess.com published our initial Bug Bounty Policy, I have been involved with the bug bounty program. Eventually, it was decided I would be the security liaison for the program, and, boy, what a wild ride it has been since then. There have been a lot of changes to the site, the submitted bounties, and life in general since then.

I wanted to write a post addressing a few things, as well as making a few personal comments. That's what this post is about! It is possible that this will be something I do regularly. It is possible that this will be something that I maybe do once or twice more at some point in the future. It is possible that this will be the one and only State of the Bug Bounties post. Only time will tell!

Before anything else, I'd like to make clear what these posts will not contain and will not be used for.

• I will not post any information regarding bounty hunters, security researchers, etc. This includes no names, no amounts awarded, no numbers of submissions (neither submitted nor completed), and such.
• I will not use this to discuss reports with reporters. If you submit a report, please send it to bounties@chesscom.atlassian.net

With that said, let's get on with the State of the Bug Bounties!

Response Times

I'd be remiss if I didn't start off with the elephant in the room: wait times. First, let me apologize once again for the length of time some bounties are taking, even for initial responses. Even though the policy states a 5-day initial response time, I do my best to respond sooner than that. However, that doesn't always happen, for various reasons.

First and foremost, the amount of bounty submissions has EXPLODED. I won't give any exact details here, but suffice to say that, initially, we were only getting a one or two a week. This was easily manageable without a dedicated person focusing solely on them. As time went on, after a few months, we started seeing several a week each week. Quite a few were duplicates, though, so the actual amount of work was less than that, keeping it still manageable.

Then, something happen. Well, lots of things happened. A worldwide pandemic kept people at home, which sent a flood of new traffic to our site. "The Queen's Gambit" was released, causing a huge worldwide explosion in interest in chess, and another flood of new users hit our site. Media and news outlets starting reporting on chess more, and Chess.com got mentioned... a LOT. All this caused our site to explode with new users, which is great! Even greater was our team responsible for keeping servers up and running did an excellent job doing so during some of the busiest times in the history of the site.

It wasn't just players who found new interest in Chess.com, though. Security researchers, bug bounty hunters, and other similar folks, seemingly suddenly, had a new target. Those few reports a week, including duplicates, because more than a few reports each week with less duplicates. Then multiple new reports every day.

This was much less manageable given the time and resources we had allocated for bounty work. Bounties backed up. Reports went days, sometimes weeks, without response. All the while, I was struggling to keep up... and failing, honestly. Something needed to change.

After discussing this with my team, and making known both my frustrations and the frustrations of our thankfully patient reporters, a decision was made. Even though, during this busy time, we need as many hands on deck as possible in terms of active developers (cough https://www.chess.com/jobs cough), we thought it best to have at least at least one dedicated full-time employee whose sole job (at least for now) is to handle incoming bounty submissions.

So, hi! Hello! Here I am. Bounty submissions are my primary, nigh only, focus for the foreseeable future as we get caught up. Response times are still not where I'd like them to be, but we're getting there. All I ask is that you all please continue to be patient with us. I promise, if you email chesscom.atlassian.net,">bounties@chesscom.atlassian.net, we will see your message and we will respond when we can.

Issues with Response Delivery

Speaking of responses, I'd like to take a second to address something that seems to be happening more often. I've now had multiple reporters seemingly not receive replies I have sent. I don't know why this has suddenly started happening, and thankfully the impact as been limited. However, if you send a report and haven't received a response after a few days, please check your spam folders! We've had cases of our responses going to spam and not being seen by reporters. It is frustrating on both ends, I'm sure. I send a reply, then get a response back saying "I haven't heard back from you". The worst part is, if I try to respond again, those messages also likely go to spam.

To help combat this, I'd ask that you mark bounties@chesscom.atlassian.net as not spam in whatever way your email provider allows.

What Is (and Isn't) a Security Liaison?

Now, let's talk about me or, more specifically, my role in all this. What is a security liaison? Put simply, I'm the middleman between reporters and our team. I read every single report we receive and decide how to proceed. Sometimes, it's as simple as closing duplicates. Sometimes it's creating a ticket for our team to handle, determining severity and award amount, and shepherding the bounty from submission to getting fixed to verifying the fix to payment. Sometimes it's reproducing reports to understand them better, using tools like Burp Suite, to determine the root issues in order to even know what the problem is in order to ticket it for our team.

Suffice to say, it's a lot. More importantly, it's a continual learning process. Before the Chess.com bug bounty program, I had never even heard of Burp Suite! Can you imagine?!

I get help from our team, when appropriate. I get help from reporters when appropriate. The key is working together, and I'm thankful for all the reporters who have been willing to work with me.

The most important thing, though, is that I'm (mostly) a middleman. I occasionally will handle fixes myself, if they are quick and easy enough to not take much time. I do so in order to put less burden on our already incredibly busy developers. Ironically, this means that the tickets that do go to our already incredibly busy developers are the ones that require more time and attention.

We're trying to prioritize bounties more lately, which should help with responses, but keep this in mind when wondering why a fix it taking a long time. If the bounty ticket is deemed lower priority than other work, for any number of reasons, it is likely to take a little longer to get done. That said, since bounties aren't just regular work and have the added situation of bounty hunters waiting on us to get them done, as I said above, we're trying to improve our prioritization.

I've digressed a bit, so let's get back on track. We've talked about what a security liaison is, but what isn't it? I pretty much already touched on some of it. The security liaison is not always (in fact, not usually) the developer who actually works on fixes. Relatedly, as security liaison, I'm not always the proper expert or authority on different parts of the site, different systems in place, etc. That's another reason I act as middleman. Our team of developers have their expertise across the site, and it's my job to determine who is best to ask about and/or assign tickets to.

Finally, not only am I not usually the technical authority, sometimes I'm not the security authority either. That might sound strange for someone who is supposed to be a security liaison. However, I'm constantly learning as I go, and I'm trying to keep as knowledgeable as I need to be in order to understand reports and be able to translate them into developer tickets.

Current State of the Program

So, what is the current state of the bounty program? Well, honestly, things are looking up! Unfortunately, last week, I was affected by the power outages in Texas, so I was unable to work for several days. That caused another mini-backup of reports in the queue. This week, I've done my best to at least respond to any initial submissions that were 5-days old or older as well as shepherd a number of in progress reports on their way, including completing several.

Many reports have not gotten much more than an initial response, and I'm working on rectifying that. I usually prioritize my report handling and responses based on two factors: date of the original submission (or latest response) and severity of the reported issue. If you have submitted a report more recently than others, or if the report you submit is deemed a lower priority than others, then that report is lower in the queue and takes more time to get to. Again, I'm working on this, and slowly getting caught up, but, ask I've asked before and will continue to ask, please be patient with me and with us as we work our way through the queue.

To give you an idea of the queue, as I write this, I have more than 50 reports in various stages of "not complete", be they initial reports, waiting for triage, waiting for fix development, etc. That's a lot of reports. Some will be easy. Some will take time. Some will be "FIX NOW!" important, others will be queued.

Final Thoughts and Comments

As we move into the future, continue catching up with the queue, and continue improving the program and the process, I think that the bounty program will become more of a source of pride than stress and anxiety on our end. Relatedly, I think it will be come a lot smoother and more enjoyable for our reporters. That's the goal, anyway!

Speaking of our reporters, I'd like to say that I've enjoyed getting to know some of them, especially repeat reporters. One of my main goals as security liaison, personally, is to foster good relationships between reports and not only myself but Chess.com in general. It genuinely is, and has been, a pleasure working with our reporters. I'm thankful for all I've learned, for all they have taught me, and, as always, for their incredible patience.

So, with all that said, it's time to end this post. For those of you who stuck through and read to the end, thanks! Avenues of communication can be limited, in both scope and simple existence. We have reports, themselves, which are focused around the specifically reported issue. We have the comments on the policy page, but those aren't necessarily the best place for discussions and posts like this one. We have the policy page itself, but that is for defining the policy, expectations, etc. When I found myself wanting to give a big, public, general update about the program, my own personal Chess.com blog seemed like as good of a home as any.

Hopefully this was informational and entertaining! Thank you all for reading!