At work, I have been responsible for conducting the coding tests during the interview process, and have been doing that for over a few years now. Over time I have made some mistakes, learned some things, and this post is a summary of what one should consider when creating a coding test.
Coding tests are generally the first round of an interview; it is either online or on campus, depends, but a coding round is considered the first gate a candidate has to cross. How you conduct it has some implications. For example, if you asked the candidate to submit code online, you may want to verify that the code was written by the candidate, they understand it, and that they did it in the stipulated time. You can do that with an additional pair programming round on campus. If you conduct the test on-campus there is little room for such doubt. But then you need to have things like a dedicated desk or meeting room for the test, a machine, and maybe even food and coffee. But there are some considerations integral to coding tests themselves, these are the ones we shall be looking at today.
Before we begin, let me call out that there are those who believe that interview is not the best process to understand the suitability of a candidate, and I would not disagree. But I believe that if done right, coding tests are the best way to understand a person’s approach to problem-solving and their expertise and command of programming. How to go from there is your choice.
We shall see the points I have learned to consider when creating a test problem, and we shall also discuss my thoughts on them and the reasoning behind them.
What to Consider
Skills: What Are You Looking For?
First, you need to identify the skill set you need, i.e. the knowledge that matter to you for the role you are looking to fill. People consider knowledge of C as a basic expectation in computer science, but if a developer is not expected to work on C it is pointless to ask a question with C in mind. Similarly, questions on shortest path algorithms, sorting, and searching algorithms may be pointless, depending on the position. If there was ever a need to use them, I would not expect any employee to implement them anyway, why expect them to implement them in an interview. All we need to know is that they understand how those algorithms work. In an interview, we need to better target the skills that matter to us. Yet, it is better to skip any frameworks that you might use, frameworks can be taught and learned, teaching problem-solving approaches takes longer. I am also of the opinion that the programming language used to solve the problem should be the interviewee’s choice, but that is not always feasible since the interviewer should have enough skill in that language to review the code they submit!
Prioritize: What Is the Value of Each Practice/Skill to You?
Prioritize the skills into these categories: ‘must have’; ‘should have’; ‘good to have’; and ‘cool to have.’ Model the problem in a way that targets the ‘must have’ skills and probably touches on some ‘should have’ skills. Skip anything under the ‘cool’ category. This is not to say that we should hire a person who can only do the job we require them to do at the time of hiring. It is to say that the person should be able to do at least that much. You have the rest of the interview process to assess the person’s ability to learn or solve problems creatively; the coding test is the gating criteria.
Time: How Much Time Can You Allot for the Test?
This matters a little bit in an online test but is a huge consideration in an on-campus test. For on-campus tests, we need to have provisions for machines and a meeting room or desk. If it is an online test, how much time to provide the candidate to solve the problem is the biggest consideration. Either way, how much time to demand from the candidate is the main question and we need to decide on a problem that can be solved in a reasonable amount of time.
It is also important to accommodate for the pressure the candidate may be under during the test. This matters even more for on-campus tests; it can put your time calculations off considerably. I try to make the problem statement very clear and as definitive as possible. Being explicit about what is expected and what is not helps because an unclear question leaves room for random implementations or shortcuts where I least expect them. That makes it difficult to evaluate the solutions on equal grounds. On the other hand, unclear questions leave room for creativity, but I have seen people miss the point with these.
Branding: Showcase Your Company's Culture
The interview is a window; a window for the company into the candidate's capabilities and a window for a candidate into company’s culture. Just as the company needs to like the candidate, the candidate needs to like the company. And the problem statement is the first impression of the company! It is important to make the problem ‘fun’ to work on. I prefer to use casual ways of describing the problem and try to make it enjoyable to read and to solve. Your company should feel like a fun place to work, right from the start of the interview process. You should choose a style that best describes your organization. As a side note, nerd jokes or references to Hitchhiker's Guide to the Galaxy or Star Wars do not always work.
Creating the Problem Statement
Over time, I've established a process for creating the problem statement and enabling a predictable estimation of time to find a solution. Over time you can choose to skip some of the steps, but to begin with here is the process:
Once the problem is identified, create the write-up describing it. Instructions, guidelines, and rules about time, how to submit the code, languages allowed, etc. should be mentioned at the top.
Once the problem statement is formulated, ask a peer to read and explain it to you. Ensure it means what you expect it to mean.
Solve the problem. Record the time.
Ask a peer to solve the problem, record the time taken. The peer here should be from the target experience range and have the proper skill set.
Anything you learn during these discussions, or while solving the problem, should be converted into instructions and added to the top of the test prompt.
Fix the scope of the problem to fit the time.
Solve it again when you are in a different mindset and record the time.
This process, although tedious, can help to properly set your expectations. Consider the quality of your solution in the time given and how acceptable you find your own mistakes. Consider that you already knew the problem and had (unknowingly/knowingly) designed a solution in your mind. That should help set expectations from an interviewee ‘under pressure, under a time limit, likely less experienced than yourself, and who was groomed in a different company culture with different practices than yours.’ Sounds difficult, right? It is not to mean that the evaluation should be lenient though, it is only to identify what is acceptable.
While evaluating, one should remember that you are not looking for candidates who solve the question the same way you do. In other words, you are not looking for candidates who think just like you. Sometimes, different is better. It is always surprising how many different ways you find to solve a problem!
That’s all from me, all the best for finding the right candidates!