People: How do you provide a good candidate experience during the technical assessment?

Tips on providing a positive candidate experience for software engineers undertaking a technical assessment, for busy managers and founders of growing technology companies – from the 2021 series of Threads discussions.

Discussion host: Adam Dangoor, senior engineer, technical interviewer and interview coach.

Context

The best software engineers will often have a choice of which company to join. The technical assessment and interview process is an opportunity to demonstrate the positives and the reality of your firm/project so that the right people choose to work for you. 

You only have three-five hours to play with, so ensure the process explores all that you and the applicant needs, in order to both reach a reliable decision.

You wouldn’t assess a fish by gauging its ability to climb a tree, so don’t assess an engineer’s ability on aspects unimportant for the role and its deliverables.

These notes explore good practices, and how these can be applied to small and mid sized operations, while at all times remaining fair, representative and repeatable.

Constructing the technical assessment

The assessment must measure what you actually need to evidence. The instructions should make it clear what you are assessing for; ability to implement a working MVP, an algorithm optimised for performance, good test coverage and documentation etc.

The assessment should also give some scope for applicants to demonstrate their individual strengths and style, provided that’s what you’re able to hire for.

For example if you use CI/CD/TDD/BDD and don’t necessitate prior experience in the job advert, then don’t accidentally bake in this as a must-have aspect of the assessment.

You may at some stage need to hire someone with greater expertise than is currently held on the team. You could engage an expert freelance consultant to help conduct technical interviews. Similarly, the consultant could help the applicant interview the firm too, to help them validate whether the firm has ability to deliver upon its aims.

Another approach is to set a joint project exploring a domain/tech topic well-known to the applicant. E.g. you both prepare an analysis and then discuss approaches and solutions together, as a practical exercise in seeing how they think and collaborate. Good applicants should teach you something, and they should gain a true sense of the firm and its people. Caution: don’t pick something that could be misconstrued as seeking ‘free advice’, pick something abstractly relevant.

If you have the resource; you could offer a pick list of methods by which the candidate can choose to be assessed, balancing their personal strengths with time pressures. You’ll need to weigh up the time to create reliable and comparable methods, and a review process for each, which may be a challenge for small teams.

For example, options such as;

• Compact & succinct (for someone under time pressure, where the burden of evidence falls more heavily on the applicant’s ability to convey skills); technical Q&A directly relevant to the engineering at hand, usually 90mins with a lead engineer and a fail/pass decision, before proceeding to a final 90 mins with hiring manager for validation before the hire/reject decision. This puts more onus on a clear case being presented by the applicant, with less wiggle room, but it should save valuable time if things are moving quickly.
• Paced & methodical (for someone passively looking, with the luxury of time): any combination of; relaxed scene setting chit-chat with an engineer or hiring manager (45 mins), then a coding exercise (two-three hours, plus code review), discussion of the approach/solution (45 mins), deeper technical discussion Q&A (90 mins), then a broader cultural and skills mapping exercise with hiring manager or department head (90 mins). Here there’s more scope to delve into things and to circle back to weaker aspects.

If part of the technical assessment relies upon understanding a particular spec – such as http rpc spec, or a complex api – flag this upfront and make an allowance for this in the allotted time (e.g. 20 mins to read the spec).

If you’ve ended up with a sprawling exercise – well over four hours (aim for two-three max.) – consider radically reducing the scope by giving applicants a really small code base to add a meaningful feature too, rather than building from scratch.

For assessing a recent graduate, you could invest an hour learning about the topic studied – enough for a structured discussion – and then have a discussion around that topic, exploring how they think, how they approached their work, and their reflections.

You’ll benefit from standardising the interviews with a crib sheet for each interviewer to apply structure (some deviation is fine, if relevant), to easily record what they evidenced, and to give a reliable outcome.

Ask your recently-hired people to critique your process and instructions, and to help improve it. They’ll likely have encountered other methods too, and this will be fresh in their minds.

A good candidate experience for software engineers

Be understanding and respectful of the person in front of you. Everyone has their own ‘stuff’ going on with circumstances unique to them. They are likely coming at things from a different – and usually valid – perspective. The least you can do is have empathy with the person who’s doing their best to portray what they bring to your team.

Each interviewer should know the job spec and applicant’s CV equally well. Each interviewer should, in advance, coordinate with the other interviewers which aspects to focus on. This gives a thorough approach with minimal duplication, except where cross examination is merited.

You should allow a healthy amount of time for the candidate to interview you too. Much of this can be covered during an exploratory first discussion, or at first interview. They should know whether, and why, they wish to proceed before any technical assessment begins.

When inviting someone to their second interview, let them know what you’re assessing for. The depth of insight uncovered will outweigh any temptation to give model answers. It’s also ok to say “You should already know all that we need to explore, so please don’t sweat the small stuff in your preparation”.

Supporting the candidate through the process

During a first exploratory chat – to help the applicant get orientated – be sure to ask what they want from the discussion. Be authentic, share the story of the company/project (people remember stories better than metrics). Treat it as an open discussion and a transparent process. This person may know another ideal person, a lasting positive impression travels! 

Be transparent about what it’s like to work on your code base in the environment and culture that you’re building. Be honest about the traits and skills you’re looking for, and how you can build upon these together. 

Let the applicant know the steps in the process and anticipated timeline. Ask if they need things to move more or less quickly for any reason. 

Nominate a single point of contact to support the applicant through the process – and it’s best if that person isn’t a hiring gatekeeper, as they may feel more comfortable divulging feedback and interests to a neutral party.

Set expectations for time and apply structure to help you stick to it. If you need more time, check that’s ok, and flag it early to help the applicant pace their answers.

Make someone in the hiring group available outside of the working day, within reason, to accommodate applicants who can’t easily find time during the day.

No one likes being left dangling, awaiting an update. It’s much better to say “Please bear with us, we may need up to x days” than to incrementally delay giving an update ’til you have complete information.

Any final discussion with an exec/C-level person should not bring any major surprises for either party. It’s mainly to show conviction of leadership, and to rubber stamp. Similarly, such a person should be able to illustrate the vision of the firm/project and ought not miss opportunity to impress. This person should also be briefed in advance (ideally within 48hrs), to avoid duplicated discussion.

Conducting the technical assessment

Dress code arises more than you might think, especially for female applicants, so state what the company culture/policy is.

If remote, all attendees should have their camera on. 

Don’t show up late. Book at least 10 minutes before to refresh yourself with their CV and what you need to cover, and ask if the applicant has a hard stop (think: child care, other commitments etc.).

Very few engineering positions – pre-sales, to some extent – require giving the ‘correct’ answer on the spot. Therefore, assess for the person’s ability to uncover and validate requirements, and to take a suitable approach to reaching an acceptable answer.

It isn’t normal to be observed while coding. You could reserve a pairing exercise for when it’s especially helpful to an applicant (e.g. someone moving into a new programming methodology), and/or if it’s an important aspect of your usual engineering approach. It must be performed in a relaxed and supportive way.

Live coding tests (done on-site or via screen-share) do offer some value, but are regarded as unpleasant by many, because of the pressure to ‘ace it’ live. Also biases towards people who spend time practicing interview-specific coding problems and learning algorithms.

Consider the feedback and encouragement that you offer during the assessment process. It’s ok to offer a hint to get someone ‘unstuck’, provided it’s consistent with other applicants and interviewers.

Consider the signal you get, and lose, when the candidate is stuck. If you help them too early, you do not know if they were just thinking and would have fixed the issue themselves. If you spend 10 minutes watching them struggle on a less significant aspect, you lose the information on whether they could have done the next part.

Be aware however of staying within the candidate’s comfort zone, if that is not enough signal alone to make a reliable hiring decision.

Standardise how much help is reasonable to offer across all interviewers, avoiding favouritism. If an interviewer gets someone unstuck, they should record how much they intervened, and at what point, in the notes.

Requiring hand holding on the understanding of the problem (provided it’s well described and has been guinea pig tested) should be seen as more of a negative than hand holding on solving the coding problem.

Structure the interview questions in a way that gives the applicant some early wins to boost morale. This can later be revisited and expanded upon through scenario-based questioning. This helps them tune in, and to progress through increasingly-challenging questions more comfortably.

Give positive feedback, verbally, throughout. Someone who has failed the interview should leave feeling listened to, respected. If rejected, they should see that this decision was based on what was evidenced, not a vague impression that wasn’t explored in the process. “Sorry, you did well but you’re not quite what we’re looking for” isn’t acceptable feedback.

If you encourage the candidate this can make the situation more friendly while not detracting from the signal you get. Encouragement might be “Nice work fixing that issue” or “I’m glad to see such clear variable names”.

You can teach someone how to code, but you’ll find it much harder to teach them how not to be an asshole. You may be able to (just about) tolerate one asshole within a large team, but only if their output justifies the impact and only if there’s a shared consensus that this person is worth the hassle at this stage. Note how the candidate makes you feel throughout the process.

Reviewing coding exercise solutions

You will need an agreed code review spec for take-home exercises. Each reviewer needs to know what they are checking for, be that a working solution, an optimal algorithm, high performing code, test coverage, production-quality output etc. 

Some people cheat on take-home exercises, so use follow-up interviews to validate how well they can explain their rationale. They should be able to explain why they did/didn’t take certain approaches, and have a view on what they’d improve given more time. How they’d scale it for production. How they were confident the code works.

Let the applicant know what qualities you’ll be assessing for in the take-home exercise. This could be: simply meets the basic spec; is fast; is secure; MVP where quality-of-code comes later; whether/not to worry about algorithm performance.

If “How did you know if your code worked?” gets “it seemed to work as expected” isn’t nearly so strong as I wrote unit/functional/user tests and here’s the metrics to show that.

Or, if you’ve given an intentionally loosely defined exercise as the applicant to make a note of what they’ve prioritised and why. You could give some business context in the doc and see if the applicant picks up on the ‘right’ things.

Encouraging the candidate through the interview questions

At the early stages an applicant may not have had a chance to research your company beyond the job advert. After all, if you’d met this person by chance in a cafe and got chatting about shared interests, it’s unlikely they’d know your firm. Don’t put them on the spot with “So, what do you know about our company?” as it causes a power imbalance.

It helps to set an encouraging tone “Here’s five discussion points relevant to the role and, don’t worry, we don’t expect you to ace each one” to help the candidate feel appreciated and comfortable.

Avoid relying on questions that encourage performed answers which may cause performance anxiety (and mainly measure memory recall) “Why do you want to work here?” isn’t as insightful as “What new skills do you think you’ll build in this role?”.

To help you think along the right lines; 

• “Tell me about a time when you challenged a spec that saved lots of wasted effort”
• “Talk me through how you improved a methodology for the team”
• “Tell me about the bug you fixed in the workplace that you’re most proud of”
• “How do you strike the balance between automation vs manual fixes?”

A good starting point for assessing lead engineers is “Tell me about a time when you faced a problem that required deep technical thought” from which you can dig into how and why they structured their approach, what they decided to not to do and why not.