Tips on interviewing software engineers/developers for the required level of engineering rigour, for busy managers and founders of growing technology companies – from the 2021 series of Threads discussions.
Discussion host: Adam Dangoor, senior engineer, technical interviewer and interview coach.
Context
Many interview processes assess familiarity with a technical stack, but this only tells part of the story. Interviews may touch on engineering rigour, but under-prepared questioning often relies on the interviewer’s intuition or whether they ‘like’ the applicant – leading to unstructured outcomes.
Determining software engineering rigour required
Engineering rigour means different things to different people. Rigour at NASA differs from that of a bootstrapped start-up building a MVP, where its own flavour of rigour may be achieved later. Most engineering approaches lie somewhere in between, reflecting the use case.
Forms of rigour may include: testing methods that gain high confidence the code is working; architecture and design practices suited to the problem space; coding behaviours such as not patching over a bug; avoiding unnecessary over-engineering; clear documentation with comments on pull requests; and neat, readable, and maintainable code.
Reflect on the level of rigour you need. Assess what you currently have across the teams and their methods. Aim to strike the right balance between today’s demands and the upcoming needs.
As a whole, an engineering function may comprise distinct levels of rigour across different teams, provided the code shipped meets the standard required.
Some use cases merit a ‘fast and fluid’ approach, such as for exploration, PoCs, MVPs, new low-impact features. Others warrant ‘slow and steady’, such as for critical applications, or a large or high-value code base. There are many degrees and flavours in between. Consider what you have vs what you need, and the people and approaches that can help you get there.
Engineers/developers will seek to apply the ‘right’ amount and flavour of rigour, based on their interpretation of the company’s defined goals and the project needs. What’s considered ‘right’ may vary between engineers/developers; each may have a range of personal styles, strengths and approaches in their toolkit.
You aren’t just assessing for a binary ‘too much’ or ‘too little’ rigour, but also the ability to adapt and apply suitable rigour in the appropriate way.
For example the person who questions the reasons behind a requirement, or who ensures there are no holes in the understanding, is displaying intentions of achieving high rigour. Such a person may ‘fight back’ a bit, but in a good way. Such a person may also over-engineer at the wrong time if the actual need isn’t well understood.
Engineering rigour is important from your early hires to ensure you don’t hit a productivity wall later on.
You can’t ‘just do what Google does’. At the extreme, Google’s approach to validating a new feature which has tens of millions of users may be disproportionate in your case.
Interview preparation before hiring software engineers/developers
Open up a structured discussion with the current team about the level of rigour being achieved, and that which is required. Communicate the agreed standards across the teams, and uphold the methods needed to achieve this.
From this renewed understanding you can now define the attributes and abilities sought from new hires. This avoids costly mismatches that may only be discovered later.
Describe these attributes and abilities in the job description and job advert. It will help the applicant self-identify whether they share your outlook and methods.
It’s for the hiring process – interviews and technical assessment – to evidence the applicant’s approach and ability.
Gauging rigour from take-home coding exercises
To some extent your interview and assessment process should give the applicant a feel for your culture and domain, helping them self-identify whether there’s a good fit. It’s a two way process.
Abstract logical reasoning exercises – for example – are biased against those who aren’t used to thinking in that particular way. This can preclude people who have been out of university for longer, but may better suit an algorithm designer. One size does not fit all.
Be explicit about what you’re measuring in any technical exercise given, so applicants know where to expend their attention. E.g. for a web API design exercise state in the instructions; “use your skills as if this was a serious production service”, or “the main goal is to have something working”, or “data integrity is key”.
Employers tend to underestimate the time required to ‘do things properly’ on take-home exercises by half. Three to four hours is considered a reasonable investment for a take-home exercise. Thorough applicants will likely be doubling that, but may not feel comfortable disclosing this.
Benchmarking with ‘blind’ guinea pig applicants (e.g. from another team) gauges the actual time needed to complete things accordingly and gains feedback on the instructions offered.
Also allow for any domain-accrued knowledge, so you don’t accidentally preclude people who don’t yet know your problem space.
You will need an agreed code review spec for take-home exercises. Each reviewer needs to know what they are checking for, be that a working solution, an optimal algorithm, high performing code, test coverage, production-quality output etc.
Consider giving the applicant’s solution to its reviewers ‘blind’ so they don’t apply bias.
Indications of rigour
Engineering rigour isn’t reliably indicated by someone “knowing the technology stack well” (that’s an ability to understand technical concepts, applying concepts appropriately is the differentiator). Nor is it “this person seems well-spoken” (that’s elocution). Nor “they seem like me/the team” (homogeneity, bias). And nor “I’ve heard of their employer” (again, bias).
Similarly, public code repo’s only tell part of the story. An impressive repo doesn’t guarantee rigour in the workplace.
Placing most weight on an applicant’s contributions to open source projects is biased against those who aren’t able to – or drawn to – contribute to open source projects.
Lack of evidence of public code examples isn’t an evidence of lack. Sure, you can see some code, but not the context or constraints within which it happened.
The applicant with an impressive, solid, repo may work well without any external pressures (e.g. lives at home with parents, is single, or has few outside interests) but may crumble in the face of conflicting priorities. Such an employee may need shielding from distractions.
Someone who has a repo full of shiny yet unfinished projects may quickly lose interest in a large, mature and critical code base. Such an employee may be better suited to ‘fast and fluid’ projects, or may now feel ready to work on a long term code base.
Thinking “this person doesn’t feel like a start-up person to me” is a lazy presumption if you haven’t evidenced whether they actually have the right temperament and approach.
Similarly thinking “this person has too much big company experience for us” may miss a great person who’s frustrated by inertia, or who’s used to internal start-up or spin-out approaches.
Amazon, for example, values frugality from its engineering hires and will assess for this. Applicants making presumptions about ‘generous time and resources’ wouldn’t get far.
Example engineer/developer interview questions
In this section of the technical interview you’re aiming to evidence whether the person has, and desires, a compatible level of engineering rigour, be that constant or variable.
You’ll benefit from a templated and repeatable set of questions that each interviewer can conduct and score. This reduces bias and variability, arriving at an evidenced conclusion to inform a hiring decision. Iterate the approach as you go.
To help you think along the right lines (include your engineers in designing the questions):
- You could begin with “tell me about a recent project you’re pleased with” and then “how did you choose your approach?” and then “if you had one more month, with the goal of scale/security/xyz in mind, what would you do next?”.
- Or, if that person has ‘big company’ experience and you’re interested if they’re start-up compatible “what if you had one less month….”.
- Code quality; “How do you know your code worked?”. Did the applicant revisit the spec, write tests before coding, pair on critical aspects, complete integration and acceptance testing etc.?
- Code performance; “How would you determine your code’s performance?”. Striving for outright speed unnecessarily may not be appropriate if at the cost of other things.
- Documentation; “Tell me about a pull/merge where you documented; how did you decide to document and why?”. Did they consider others’ needs and future aims?
- Decision making; engineers tend to know what the right thing is to do. They will also face complex situations requiring good decisions on what not to do and be able to explain why. Dig around what they evidenced and ruled out, how and why, and whether they understood the ramifications.
- “Tell me about something you failed at, or when something went wrong”. Watch for the honesty of the reply, are they open to critique, did they escalate appropriately or patch and move on, did they understand the impact/gravity of the situation?
Using the STAR technique can be helpful for interviewers who are trying to assess a candidate’s project. STAR stands for Situation, Task, Action, Result.
Once the interviewer understands the candidate’s situation and task, they can have a reasonable understanding of what rigour would be suitable. Then, questions on the user’s ‘Action’s can be focused on the decision making. ‘Result’ is key for understanding rigour – did the candidate appropriately measure the success of their deliverable – whether that was with performance metrics or something else.
Conducting the interview
Whether someone answers interview questions with “I” versus “we” is only partly indicative of their attitude and approach. It’s up to you to dig into the responses.
Someone who always replies with “we” may be hoping to display a ‘team player’ approach, but they may also be revealing that they were a smaller part of the project. Someone who always replies with “I” may accidentally appear a self-centred ‘brogrammer’ so do also explore team fit. Dig around to avoid any presumptions.
Sooner or later you will run into ‘model’ answers. It’s up to you to get beneath these with good questioning, exploring whether the applicant understands the why’s and wherefores.
A good question can be “What’s the hardest bug you’ve ever faced?”. Then, after exploring the measures taken, “What would happen if someone re-introduced that bug?” (this ought not be possible if correct documentation was in place).
Similarly; “What was the one bug you never fixed?”. Can they talk about its ramifications, how far they got in the process, and that they learned?
Determining lead vs principal hires
Rigour is often what sets apart the top engineer on your engineering (non-management) track, and someone at the level below. When determining whether someone should be staff engineer, lead engineer or principal engineer, you may find it valuable to have a team discussion about how this candidate compares to your internal levels.
For example “I found this candidate has a great understanding of testing and a determination to increase the quality of our codebase. I believe that this will improve the testing skill in the company and that is one part of why I think we should hire this person as a principal engineer”. That’s a much more useful discussion point than “This person has 15 years’ experience and worked at BigCo”.
You must log in to post a comment.