Management
Interviews are stressful no matter if you’re the candidate or interviewer. This is how we used a NERF gun to relieve that stress and conduct better interviews.
No matter which side of the table you’re sitting on, the interview process puts a lot of pressure on you.
If you’re the one conducting the interview, you’re expected to ask brilliant questions – letting you make a big decision about someone’s future in a ridiculously short amount of time.
And if you’re the one being interviewed, you feel an intense pressure to appear confident (but not cocky), professional (but not stuffy), thorough (but not wordy), driven (but not too ambitious), honest (but not about anything negative), etc. etc. etc.
Unfortunately, all that stress can create situations where fantastic candidates appear unfit for a job – or worse – where poor culture fits seem like great candidates.
Back when I was running Hulu’s Customer Support department, we saw this type of thing all the time. But we managed to find a solution by incorporating a NERF gun into our interview process.
First, some background…
At the time, we were in a constant state of “we need more people… YESTERDAY” trying to keep pace with the growth of our subscription business. So, we did a ton of interviews – sometimes as many as five or six full onsite loops in a single week. That may not sound like a lot, but when you include the prep and debrief time… trust me, it adds up.
In order to operate as efficiently as possible, we worked really hard at weeding out unqualified candidates before investing all the time in an onsite. So, we’d have all candidates do things like write Yelp reviews and respond to fake customer emails, as well as performing at least two rounds of phone screens before deciding if we wanted to bring them in.
Once we did, we’d have the candidates meet with members of our team in a series of four separate 30-minute interviews. During each one, the interviewer was responsible for evaluating the candidate in a few different skills that were important to the job, ultimately coming to a Yes/No vote. To back up that vote, the interviewer was responsible for gathering evidence during their time with the candidate, and writing it all up afterwards.
Then, later that day, the whole interview team would get together to debrief and discuss our impressions of the candidates. (This was a little time-consuming, but totally worth it)
The “cold start” problem
A lot of people treat the entire hiring process like a necessary evil, and just do the bare minimum to get people in the door – but not us. We treated it like one of our products, and used our sheer volume of candidates to experiment, iterate, and improve the quality of our hires.
To do so, we tracked a lot of data. And, after a while, we noticed an unusual pattern across a bunch of candidates. Can you spot it?
If we had a situation where our votes were three-to-one, the odd-one-out was often the first interview of the day. Now, of course, you will have some level of disagreement when you have human beings interviewing other human beings – that’s not the point here.
This wasn’t just one person having valid reasons for voting differently than the rest of us. This was specifically the first person of the day coming to a different conclusion than everyone that came after them. And often being really surprised to hear how the other interviewers described the same candidate.
Which was weird.
So we dove into the issue to try and understand what was happening. We looked at the employees conducting the interviews in those first slots, the times of day of the interviews took place, the demographics of the candidates, etc. etc. etc.
The insight came in one of our debriefs, when an interviewer said “it’s like they don’t show us their true personality at first” and we realized that was really the core of it. We had a cold start problem.
A lot of candidates were used to, and prepared for, the stuffy interviews other companies conduct. But we talked to them like human beings, and handled ourselves a bit differently (see the Ryan Gosling article linked at the end of this one).
For overall “good” candidates, they would sometimes come across as a little unfriendly in their first interview – as they were so focused on trying to figure us out. Conversely, “bad” candidates could appear happy and positive at first, as that was what they expected us to want. In both cases, over the course of the day, their true personalities would come through as they became more comfortable, or their facade wore down.
We could have just accepted this cold start as a byproduct of our quirkiness, but that didn’t sit well with me. It introduced some uncertainty into our decision making process, and I wanted to reduce as much of that as possible. So, we found a way to start off more warmly.
Our NERF gun solution
Given his knack for approaching problems in unique ways, I asked Ryan Bender to see if he could figure out a solution for this. And he hit a bullseye (shooting-gallery pun… intended).
Here’s the gist of it.
First, we started putting together a “what to expect” document to better prepare candidates prior to their onsite. Secondly, we decided to add something to our interviews to be a little more fun. And a lot more “us.”
In the first interview of the day, the interviewer would walk in with a tiny NERF gun, three darts, and a stack of six Dixie cups. Then, they’d start the interview by challenging the candidate to a shooting competition.
“Hi, I’m Ryan! Interviews can always feel awkward at first, so I thought we could begin by playing a quick game to loosen up. How does that sound? We each get three darts to knock over as many cups as possible.”
The interviewer would proceed to set the Dixie cups on a ledge in the conference room, while asking the candidate to stand up and keep score on a white board.
Then, once everything was ready, the interviewer would be the first to take a turn shooting the cups. Whenever possible, the interviewer wouldn’t try toooo hard, allowing the candidate to win (not by overtly missing or anything blatant, but by aiming carefully to only score one or two – while the candidate could easily get three or four).
After a couple rounds of this, the interviewer would happily congratulate the candidate on their win, and move on with their interview.
When the fourth interview slot of the day came around (which was always when I would go), I would ask the candidate about it.
“Wow, looks like you beat Ryan pretty badly. What did you think of that challenge?”
After hearing their explanation, I’d follow up with something like:
“I’d love a rematch to win back our team’s honor. How about we make it a little more interesting. This time around, you can set up the cups on this table in literally any way you’d like – your only goal is to make it hard for me to knock any of them off the table.”
I’d then hand them the stack of cups, and give them free reign of the table. Interestingly, even though I left it open for anything, every single candidate either clustered the cups next to each other in the center of the table, or spread them out as far as possible. And I’d still be able to knock off at least two.
But then it would be my turn to create a target.
I’d look around the table thoughtfully, get an inspired look on my face, carefully put all the cups inside each other… and then plop them in the dead center of the table. Upside down.
If it’s been a while since you’ve played with NERF darts, just know that they are incredibly light – but a stack of six Dixie cups is not. So, even with three well-placed hits, there was literally no way that super stack was getting knocked off the table. (This was done carefully, intentionally, and for reasons I’ll get to in a minute)
We’d talk about it for a minute, I’d playfully thank them for letting us win our honor back, and we’d move on with the rest of the interview.
What we got out of this
Well, first of all, once we started doing this NERF gun thing, we immediately rid ourselves of the cold start problem and started having more productive interviews. I think there were a couple reasons for this.
We released the tension
Most of our candidates were pretty new to the professional workforce, so they were understandably nervous going into their interviews. For many, this was the first time they’d interviewed for a corporate job of any kind.
If you think back to what it was like when you went on your last big interview, you were probably pretty nervous too. But, by having the candidates physically stand up, move around, and concentrate on something innocuous and fun, we gave them an opportunity to release their tension. (I think it did something similar for our interviewer team too, BTW)
We boosted their confidence
This may sound a little ridiculous, but bear with me. By giving candidates a win in the first couple minutes of their first interview, we had given them something you don’t normally get in an interview.
In most interviews, you’re left wondering if you’ve done a good job – and usually don’t get any kind of feedback from your interviewers. But here, our candidates were put in a situation where they could easily do well, and even get high fives throughout the day as other interviewers saw the scores.
Add this to the release of tension, and I think it helped candidates feel comfortable dropping their guard – letting their true personalities through. (The merits of a false win are certainly debatable, and I probably wouldn’t do it exactly the same way again – but it seemed to work at the time)
We gathered evidence for our evaluations
We already knew that writing, verbal, and tech skills were important, so we had various ways of gathering evidence for those evaluations. But culture-fit is a much more difficult thing to judge. In the startup world, it’s often reduced to the idea of the “beer test” – where you answer the question “would I want to get a beer with this person?”
Maybe it’s because I hate beer, but I was never fond of that particular test. Plus, it’s overly reductive of candidates, is open to some serious unconscious-biases, and, frankly, is not that representative of what it’s like to actually work alongside someone.
Around this same time, we had been actively working on defining and reinforcing the characteristics that made someone a good culture fit on our team. This was both so we could give our team members fair and comprehensive performance reviews, but also so we could more accurately hire people who possessed those same skills and qualities.
Playfulness and resilience were key characteristics for us, and this NERF challenge gave us direct evidence for evaluating candidates in those areas.
For instance, some responded really positively to the challenge – having fun and joking with their interviewers about it. Others kind of shrugged it off like it was a waste of their time. Others got a little cocky about winning.
I found it particularly interesting to see how they responded to defeat in the fourth interview with me. When I placed the super stack of cups in the center of the table, some people immediately recognized there was no way to win, and responded with some variation of “daaaaammmm, well done!” followed by a high five. Others kept trying until they were out of darts – determined to find a way. But a couple were clearly annoyed at being defeated – one even got huffy about it.
No matter what, their reactions told us something about them.
The moral of the story
I’m sure some of you reading this have had a reaction like “that’s dumb, and a complete waste of time.” And, well, you’re certainly entitled to think so. But you probably wouldn’t have been a very good fit for our team.
And that’s both totally fine, and kind of the point. This worked for us – because it was “us” at the time.
I think of interviews as 50% figuring out if you want to ask a candidate to join your team, and 50% giving them an opportunity to decide whether they’ll actually want to.
Put another way, it’s not only your responsibility to evaluate a candidate’s fit, but to also give them an accurate picture of what it’s like to work with you. So, in an office where NERF battles broke out in the lulls between calls, giving candidates a NERF challenge helped us do both.
Plus, when the time comes to choose teams for a NERF war… it’s always nice to know what your team is working with.
My challenge for you
I bet there’s something in your interview process that could be improved. Do you know what it is?
Are there problematic patterns you can see across multiple candidates? Are there areas of your process where you’re left guessing? Do you often have major disagreements among your interview team because they rated candidates differently? Have you had people leave your company because they didn’t fit the culture? Or are your candidates just generally pretty nervous when they walk in the door?
If so, there’s probably an opportunity to do something differently. A NERF challenge might be it – it might not. But you won’t know until you try… and man, can that be fun.
More silliness
If you got a kick out of this, you may also find it interesting to see how we used the idea of Ryan Gosling to fight culture vampires.
Final note:
An obvious concern with a unique hiring practice like this is whether it can be construed as discriminatory. We were hyper aware of that at the time, and did our best to make sure it wasn’t. How well it would have worked in the long run is an open question — I moved to a different department not terribly long after we started doing it.