A take-home assessment senior engineers would actually do?

Take-home assessments carry a bad rep, I often hear “seniors don’t do take-homes”.

In theory, take-home assessments shouldn't get such a bad rap, should they? I've found a few reasons to actually prefer them - and I'm not alone; other seniors I've talked to feel the same. These assessments can give a better sense of what real on-the-job work is like, without the stress of someone watching over your shoulder while you code and tackle problems like in a live interview.

So why the negative perception? I'm convinced it mainly comes down to a lot of companies not quite getting it right. This mishandling creates a bad reputation for the whole practice. Here are a few blunders companies often make with take-home assignments that could make any engineer want to run away:

1. Take-homes are often too long. This usually stems from instructions that are overly broad, undefined, and ambiguous. When the instructions are structured in this way, candidates don’t know how long they should spend. Even worse, it is unclear what is the evaluation criteria. What's even more frustrating? The notion that the more time you pour into it, the better your score. This is a downright flawed concept, particularly for senior engineers who can't afford to spend as much time as junior engineers immersed in the job hunt.
2. Secondly, take-home assessments often focus on the wrong skills. They usually task candidates with CRUD tasks or writing boilerplate code, which feels like a real time-drain. Often, this arises from companies not investing enough thought or effort into designing a meaningful take-home assignment, leading to candidates spending an inordinate amount of time just setting up the project before they can even dive in.
3. Thirdly, companies often use take-home assessments as a time-saver, which ends up making the entire process feel impersonal. Some companies send the assessment as the first step and don’t provide any feedback to candidates. Expecting senior engineers to invest time in a take-home assessment before they've even had a conversation with someone from the company is a dreadful idea. Instead, companies should dedicate time before assigning a take-home assessment to generate enthusiasm about the role among candidates, and they should offer some form of compensation to those who have completed the assessment. Compensation doesn't necessarily have to be monetary; it could be promised feedback or a follow-up discussion about the product. From our experience, properly executing a take-home assessment doesn't actually save the company time. Its purpose is to yield a more accurate evaluation and ultimately improve the candidate experience.
4. Lastly, some of the worst companies exploit take-home assessments as a source of free labor. While most companies don't engage in this practice, the few that do have contributed to a negative stigma around take-homes. Some assessments are devised around features that the company genuinely plans to develop, effectively using the assessment as a means to garner ideas for new implementations.

Having collaborated with companies to construct take-home assessments for senior engineering hiring pipelines, I am confident that it's possible to create an assessment that senior engineers would not only be willing to do, but would actually find enjoyable. If you're curious about crafting a universally appealing take-home assessment that senior engineers (and others!) would enjoy, check out our other post here.

In this particular post, however, I delve into a specific type of assessment that we've observed not only sparks excitement among senior engineers, but also provides a realistic measure for collecting strong indicators of engineering talent.

The Code Review Assessment

I'm convinced that a code review assessment can be an excellent method for evaluating senior engineers. Here's how it works:

1. The applicant receives a pull request on a GitHub repo with some requested changes for a specific feature.
2. The candidate conducts a code review of this pull request as part of the assessment.
3. You, as the interviewer, can play the role of the author of the pull request and respond to queries about the scope of the pull request via Slack or email. This lends a more personal touch to the process.
4. The next step of the process can be a live discussion about the pull request. This conversation can extend the review, or even prompt candidates to implement some of the fixes they suggested.

Here are some of the benefits of this process:

1. A pull request review can be confined to an activity requiring less than an hour. With more context and a better-defined task, candidates aren't enticed to spend excessive hours striving for a better grade.
2. It is representative of the job. Code review is a common responsibility for a senior engineer. It measures not only coding skills but also evaluates communication and other crucial soft skills.
3. The interview feels more personal and well-considered. Candidates don't feel they're performing futile tasks, as companies have clearly invested thought into the interview process and the development of engaging interview questions.
4. Lastly, the assessment can feed into subsequent stages of the interview process, making the entire process seem more cohesive and synchronized, rather than a series of disjointed steps that don't connect well.

So, why aren’t companies doing this more often?

The reality is that many companies don't adopt this approach because it's quite challenging to implement. Here are a few obstacles associated with running a code review assessment:

• Administering the assessment can be manual and tedious. Here's how one company tried to do the process:
• They create a base repo in their GitHub organization.
• For each new candidate, they fork the repo into a personal account (forking within the organization isn't feasible due to cost).
• They manually create two issues (one explaining the assignment, the other resembling a JIRA ticket with the requirements).
• They manually generate a pull request.
• They inquire about the candidates' availability for the review, then send them the repo invitation at the agreed time, allowing 45 minutes for the review.
• Finally, they save a snapshot of the page and delete the repo (as they can't have multiple forks of the same repository on their account).
• Creating the assessment is challenging. You need to write actual code for candidates to review, and you also need to test the assessment to ensure that the pull request has a variety of issues for candidates to spot. There shouldn't be so many minor issues that distract candidates, and there should be enough substantial issues in the assessment to make it relevant. Lastly, you should test the assessment using tools like ChatGPT to ensure that reviewing the code is not a trivial task for this pull request.
• Lastly, if you wish to support multiple languages for a read code assessment, you need to translate the assessment into several languages, which can be a tedious and error-prone task.

However, now is a good time to shout out some companies where we have seen them do this well. GitLab, for instance, does a 1-hour take-home assessment that involves reviewing a merge request of a previous feature the company has worked on. This is especially relevant given that GitLab is an open-source project. We've also observed other companies like Fastly and a few small YC startups gradually incorporating this into their processes.

In light of these challenges and the benefits of this interview process, we're developing a product that can simplify the process. At Hatchways, you can create a code review assessment based on any GitHub repository and send them to candidates privately with a single click. We also assist your team in creating the best assessment for you and in supporting multiple languages so you don't have to expend tedious time setting this up. If you're interested in this or just seeking advice on your take-home interview process, set up a call with us here.