Coming down from !!Con, there are a lot of things I want to talk about. One of the things that surprised and delighted me the most was how well anonymous talk review worked. I would like to tell you about this and how we structured our talk review generally.

We decided to review all the talks anonymously, to try to reduce our bias. This was a ton of work. Alex Clemmer personally anonymized all the talk proposals and then sat out of the review process because he knew the identities of the proposers. He did an amazing job (including watching attached videos and summarizing them for the reviewers!).

I was honestly a bit scared by this – how were we going to make sure people were good at public speaking if we didn’t even know who they were?! But it turned out my fears were totally unfounded. We ended up with

• a diverse range of talks (about art! compilers! type theory! assembly! games!)
• by a good mix of experienced and new speakers
• which were very well presented. The quality was even higher than I expected, and I’d worked on reviewing the proposals!

Several people told us that they liked the anonymous review because they were confident they’d be judged fairly on the merits of their proposal.

One of our accepted speakers, Katherine Ye, told us:

Thank you so much for [anonymizing everything]! It’s a relief to know that I wasn’t picked for gender, race, age, or anything like that.

I would happily do anonymous review again. It was great.

Outreach to women.

This is going to be a whole separate post, since we worked really hard on this. Basically: about 30-40% of our talk submissions were from women, and 40% of the talks we accepted were by women. We didn’t measure other axes of diversity (like race/class/other things). There’s a lot of room for improvement.

168 proposals!!!

We were blown away by how many people submitted talks. And we only had space for 24 talks! We had to make super hard decisions, and we didn’t have much time.

Talk review: Letter grades

Lindsey suggested this way of managing review after having seen it at academic conferences. Each of us assigned a proposal a letter grade (A-D), from “I will fight for this talk”, to “I will fight for this talk to be rejected”. We then looked at the highest & lowest grade for each talk and binned them into AA, AB, AC, AD, BB, BC, BD, CC, CD, DD. We rejected any talk that didn’t have an A.

If I knew who a proposal was by, I abstained from reviewing it (so hard!).

This left us with maybe 80 talks that at least one person thought was excellent-sounding. Still too many!

GitHub issues for discussion

We then used GitHub to discuss talk proposals. We put each talk into an GitHub issue, and tagged them. Here are the tags we used.

• AA - AD Letter grade ranking
• accept - Accepted
• reject - Rejected
• has-advocate - Someone has advocated this talk in the comments
• reject-advocate - Someone has argued for rejecting this talk in the comments
• a category: (work/fun/science/product/art/proglang/other)

To be accepted, a talk needed an advocate – someone who would argue for it to be accepted. Once it had an advocate, we would discuss its merits in a video chat and decide whether to accept or reject it.

We did that, and after much sadness (what do you mean this is still too many talks! We can’t accept three talks about the Curry-Howard isomorphism!), we had a program!.

Super-personal responses

Alex, our Chief Anonymizer, did not only do an amazing job of anonymizing! Not at all. We got a ton of submissions that we would have loved to accept if we’d had more space.

So Alex sent personal emails to people whose talks we rejected saying how interesting he thought their talks were and which parts specifically he liked.

If your talk got rejected, you should consider submitting it to another conference! We probably loved it.

Amazing results

My favorite things about anonymous talk review were

• it gave the speakers more confidence that they’d be judged fairly
• it was impossible to accept less-interesting-sounding talks just because they were “by someone famous”. There were amazing talks by people who had never spoken at a conference before.
• it was impossible to give preferential treatment to our friends! This was a bit sad (some of my friends submitted talks that I knew were great, and were rejected, and I couldn’t do anything), but it made the conference better.

Here’s the final program. There was weaving! Assembly! How serial protocols work! Logic programming! Hacking poetry! Robots dancing! Python’s 1500-line switch statement! All of the speakers did an amazing job – I think for every single talk I heard someone say “Wow, that was my favorite talk!”

None of this is to say that anonymous review is the best way to do talk review. PyCon does not do anonymous review at all, and I think they do a great job. But it worked very well for us.