Imperfect #3
In defense of the take-home coding challenge
New year, new issue! It’s hard to hire these days, so today we’re taking a fresh look at the hidden benefits of a poorly loved tool in our hiring belt: the take-home coding challenge.
The Pandemic hiring squeeze
As I write this in January 2022, the software industry is going through a huge spike of demand for developers.
VC money is pouring into the startup ecosystem, there are more pre-IPO unicorns than ever, and developers are spoilt for choice. Salaries in the industry are reported to have risen by as much as 30% in 2021, making it hard for poorly capitalised startups to compete for talent against better funded rivals. Even amongst well funded companies, competition is fierce for the most experienced developers.
The COVID pandemic brought remote work to the mainstream, with even remote-averse companies having to comply with legal mandates and allow employees to work from home when possible. This opened up the job market globally, as developers gained even more access to jobs in regions with higher salaries. Entire companies sprung up to make this possible on a bureaucratic level.
As we’re all competing for scarce talent at a global level, we don’t want to lose the right people over a bad hiring process. Hiring processes that throw off candidates, like technical interviews and take-home tests, were already under scrutiny before COVID. They are now positively frowned upon by many.
The take-home coding challenge
I saw a need to change how I hire as early as last Summer, endeavouring to make the process as quick and easy as possible for my candidates. Here in Berlin, tech startups usually issue a take-home coding challenge as part of the process, and that’s a big issue now.
The take-home test is usually given after a screening interview. There’s a certain impression that the candidate is credible, but before committing to a full technical interview, companies like to see a sample of work that is a) relevant and b) comparable to other people’s. Asking all candidates to solve the same technical problem is an obvious way to evaluate their skills and compare their performance to others. Automated testing tools like leetcode are popular alternatives, but they’re impersonal and allow no nuance. We’ll discuss them another time.
The problem with take-home tests is that they take time. Candidates have to either solve them at home, which is at best an imposition and at worst impossible, or sneak them into their working hours which isn’t legal to begin with. If we’re trying to lure developers out of their current jobs and come work for us, the onus is on us to demonstrate to them why they should want to do that. Throwing a multi-hour coding challenge into their lives puts the onus back on them, and more and more decide against it.
Hiring developers is an arms race. Successful tactics have a short half-life, and the take-home test is buckling under the strain of so many companies competing for the same talent. Once a desirable employer starts doing something else, the pressure is on for everyone else to follow. So, many companies are not issuing take-home tests anymore.
This is a logical, but risky move in my opinion.
How we 10x-ed Hellofresh Engineering
When we grew the Engineering arm of Hellofresh by 10x in two years, the market was nowhere near as hot as it is today, but we still needed to move fast. Not only that, we also needed our batting average to be very high. We needed engineers who could come in, onboard fast and make a difference. That meant we were very demanding of who we extended offers to. The take-home coding challenge was the bread and butter of our technical evaluation.
How we got there is a story for another day, but by the time our technical hiring pipeline was fully operational, it was a well-oiled machine. We were able to move candidates between stages within hours of them submitting their application or coding challenge.
We mobilised almost the entire engineering organization to this effort. Everyone who had fully onboarded could be called upon to review a candidate’s submission. This meant there was always someone who could be called upon on short notice.
Submissions were reviewed by two people, never just one. This made sure there would always be two different perspectives into findings.
Recruiters were allowed to ask developers directly to review a submission. Developers were empowered to drop other tasks to do so.
Crucially, our developers were instructed to treat reviews as if they were internal. In other words, to review the work of these candidates as if they were already colleagues, and as if the code was meant for production. This meant being polite, constructive, and thorough.
A few good things came out of that last one:
1. Scrutiny over code reviews of candidate submissions was high. I personally kept tabs on them, and routinely participated and shared feedback with my developers on what they had done well and what they could have done better. Tone and constructiveness were especially important. Our teams knew, in no uncertain terms, that when reviewing submissions they were acting as ambassadors of the company. Their kindness or harshness would determine how many people would want to join us. Their attention to detail would speak to the quality of our entire engineering organisation. Their constructiveness would speak to our ability to grow together. So there was significant incentive to get code reviews right. That experienced filtered in, and made us have better code reviews for our own code.
2. Even candidates we rejected would routinely leave positive feedback, as they invariably learned something from having their code reviewed. That meant that their time was never wasted. They would move ahead in the interview process or they would learn something useful. Many of them re-applied later.
3. As candidates were encouraged to participate in the review, challenging our comments, asking for clarification and generally exchanging ideas, we could pick up on cues about what it would be like to work with them in terms of tone, defensiveness or openness, communication skills, and overall thinking.
The first one helped us as an organization be better at delivering quality software. The second garnered good will among the developer community, and the third helped hire people who are not only good programmers, but good communicators and constructive team mates.
Our take-home challenges were also pretty easy. There was an assumption that the candidate would succeed in delivering a working solution, the question was how. We paid attention to architecture, systems design, security and performance considerations, choice of libraries, approach to testing, code cleanliness, documentation, and whether or not the code ran in the first place. Getting those things right, and communicating well in the pull request on Github, was the real challenge. Because that’s what real work looks like. And our goal was to find people who would work well with us.
Caveats
Everything I said about how great coding challenges are depends on how well we can mobilise our organisations to evaluate them properly. I’ve definitely been in a position where as much as I’d like to spare the time, my teams and I were underwater with work to be delivered and deadlines to meet, and support for dedicating so much time to hiring was thin on the ground. Not a great position to be in, as sacrificing on our hiring standards is a sure way to jeopardise our future - either by not hiring, or by not hiring well. So I clawed my way out of that position and you should too.
If you find yourself in the same position, my advice is to be firm. Treat hiring as its own project. Make it an epic in JIRA and set aside time for it. Work it into your sprint planning. Give it visibility, and be prepared to argue for it. Your stakeholders have an interest in the growth of your organization, let them co-own the responsibility by supporting or rejecting the allocation of this time.
Getting signal without the test
These days, smaller, less well known companies may not have the employer branding to demand their candidates do time consuming coding challenges at home. But it’s still possible to make up for that and get equivalent signals.
Asking a candidate to review some code, for example. It can be done in half an hour and reveal a lot about what they know and what they look for.
Also asking a candidate to bring into an interview some code that they wrote (and isn’t protected intellectual property), and then asking them to walk us through it, explaining it to us as if onboarding us into the codebase. It’s possible to get lots of insight into how someone thinks about software development, how they structure their code, how they pick libraries, which considerations they give to which aspects of production, whether or not they instrument and make their code observable.
But if at all possible, I find that there is nothing like the collaboration between the candidate and our developers while co-reviewing a change, to get the strongest signal for whether us and that person would work well together in the future.
Conclusion
Take-home tests are onerous and can throw off developers who are spoilt for choice. But they’re great opportunities to involve our teams in the hiring process, get more feedback, evaluate the candidate’s communication skills, and turn the process into a positive experience. As more and more companies do away with that practice in the name of shorter hiring processes and being more competitive, much of this signal is lost.
I worry that missing out on all this signal is going to lead to many companies adopting a hire fast, fire fast approach, especially as Hastings and Meyer’s book No Rules Rules spreads through the boardrooms of the tech scene. That would be terrible for productivity and for people’s satisfaction with their career.
So, as we all search for the faster, easiest, least onerous way to find and evaluate people to join our teams, it’s important to remember that there are ways to search for that signal even without a coding challenge, including live code reviews and systems design exercises.
After all, we’re looking for people who are not just great coders but also good communicators. Because coding is communicating.
This edition, my recommended reading is Gergely Orosz’s excellent Pragmatic Engineer newsletter, which lately has focused a lot on the hiring market situation and what we as hiring managers can do to navigate it successfully. Please enjoy:
Don’t forget to subscribe to Imperfect Leader for more updates in 2022:
Follow me on twitter for everything that doesn’t fit the newsletter. I often tweet about engineering, unsurprisingly: @pedrotypes
I hope you got something valuable out of this issue!
If you did, please consider sharing with your friends. I’m sure someone you know will also enjoy it.
Thanks for reading and until next time!