5 Reasons I Distrust Coding Tests

Important considerations that may not be obvious

Doug Wilson
8 min readNov 23, 2024
Photo by Glenn Carstens-Peters on Unsplash

OK, we’d like to move forward in the interview process by inviting you to take an online coding test!”, wrote the hiring manager.

“Oh, bother,” thought Pooh.

Which got me thinking. In hiring roles, I’ve used coding tests myself to separate the best candidates from the … well, not so best. Why does it bother me so much, when I’m asked to take one myself? And as with most things, the answer is that I think about and approach them differently than most people.

Please allow to explain. I will also offer practical suggestions with examples and alternatives. So, let’s forge ahead, shall we?

  1. Coding tests can require significant time and effort and often (usually?) result in zero reward. After all, there can be only one! ;)
    When combined with several stages of interviews, white boarding sessions, group panels, etc, it’s not unusual to invest 2–3 hours in pursuit of a single role that several dozen (or even hundred) people are also pursuing. That’s a lot of time for the candidate and a LOT of time for those participating from the hiring side, who have to interview each candidate and evaluate the test results.
    The total cost can be significant, so as a candidate, I need to be certain that the role I’m pursuing is worth my time. Maybe that’s an intentional part of the calculus of requiring them. But as part of the hiring team, I would be wise to make the test as brief and its results as clear as possible, which means thinking carefully about what we hope to discover and how we’ll know it when we see it.
    In the world of education, this is often called a “rubric” — a guide or framework (often in tabular form), which is used to evaluate the relative quality of various test results and which serves as a set of objective criteria for ranking them.
    In tests I design and administer, this includes:
    Well-organized import section, e.g. import org.springframework.http.HttpStatus — alphabetized, logically grouped using blank line white space, etc for readability and maintenance,
    Method and variable naming and appropriate data typing, e.g. void CacheEntityData(), CorrelationUuid UUID, CreatedAtDateTimeUtc Timestamp, etc, for unambiguous, self documenting code,
    Set up code section, e.g. CorrelationUuid = null, CreatedAtDateTimeUtc = UUID.valueOf(“9999–12–31 23:59:59.999”) aka the end of time, etc, for explicitly establishing consistent, known, testable pre-conditions,
    • Effective use of try/catch blocks to contain any activity that can cause runtime exceptions, extra points for a finally block,
    Meaningful log statements with appropriate logging levels, e.g. logger.info(“Attempting to CacheEntityData()”), logger.info(entityTypeDefinitions.size() + “ EntityTypeDefinitions cached locally”), logger.error(“CacheEntityData() failed due to: “ + e), logger.info(“CacheEntityData() succeeded”), etc, to enable quick, easy debugging, alerting, etc,
    Clear, thoughtful, and useful inline comments, e.g. #NOTE: JWT shared secret values will be created and managed by the scripted infrastructure using a secret management system., //TODO: Consider converting this method to a BPMN process when possible, etc, for clarity, maintainability, and important in-the-moment idea capture, and
    Attention to “little” details, e.g. HttpStatus.CREATED (201) not HttpStatus.OK (200) for HTTP POST operations, Copyright © 2024 MyCorp Inc notices at the top of files, etc, that indicate that this candidate understands and values more than just technical considerations, i.e. is not struggling just to get something to work but is aware of the important business and technical contexts where the code must ultimately succeed.
    Anything I missed?
  2. Coding tests are, strictly speaking, unnecessary, if you want to see how I code (especially when “no one is watching”). Just see my repository(s).
    What I look for is structure and attention to detail, e.g. meaningful (to the developer’s audience, not necessarily to the developer) repository name, brief, clear About description, detailed README file with Getting Started instructions, clear, hierarchical repository structure, License information, .gitignore file, etc.
    Your priorities may differ, but be clear about what your priorities are and why. Write this down and have it handy as you review. Take specific notes on what you find.
  3. One size does not fit all aka I’ve been coding for longer than most hiring managers have been alive.
    Giving the same coding test to an engineer vs an architect role candidate, a beginner vs a veteran coder, etc and/or evaluating the results with the same rubric will not produce meaningful results. Timed tests will yield different results, based on the candidate’s personal priorities and understanding (or assumptions) about how the test will be scored. Untimed tests may consume more time than candidates want to invest and/or may produce fear and frustration, but they may be reluctant to stop, especially given the pressure of recent unemployment, etc.
  4. Many coding tests are conducted in an “alien” web-based development environment, rather than with the tools with which the candidate is accustomed. Keystrokes that have become “muscle memory” from repetition may be different or unavailable, resulting in wasted time and frustration with the clock running. Color coding and auto-completion may also be different or absent altogether. And as artificial intelligence (AI)-assisted coding becomes more ubiquitous, its absence will be even more keenly felt.
  5. Finally, many coding tests, in my opinion, look for the wrong things, e.g. solving a specific, often challenging problem in the allotted time vs demonstrating the candidate’s general thinking and coding processes, and therefore deliver relatively little value.
    A great candidate might, for example, almost reflexively set up a great foundational structure, including some or all of the examples I mentioned in the bullet points under #1 above:
    Well-organized import section, to include the libraries that will almost certainly be needed and to create a neatly organized structure for those that may be discovered along the way,
    Method and variable naming and appropriate data typing, to carefully consider and organize the “raw materials” that may be needed in each step and how to make the purpose of each clear to current team mates and future maintainers (Note: If they skip this while “under time pressure” from the test, they will almost certainly skip it while under similar time pressure in sprint after sprint after sprint, creating technical debt every time they start to code.),
    Setup code section to set up safe, meaningful initial values that can be tested later,
    Meaningful log statements with appropriate logging levels, e.g “Attempting to … ” at the top of the method, “ … failed due to … ” in the catch block, and “ … succeeded” just before returning, etc,
    Clear, thoughtful, and useful inline comments, which can be used here to outline the steps in the candidate’s initial solution logic, and
    Attention to “little” details like the appropriate return code, value type, or data structure, which could easily be overlooked once in the “heat of battle” of actually solving the given problem,
    And all while thinking deeply about the best way(s) to solve the problem instead of just stampeding into coding. Note: If the test is being conducted in an “alien” environment, a great candidate may not have access to a macro that s/he has created to do all or most of this automatically, i.e. may be penalized for actually being great.
    My point is that instead of looking, as most organizations do, for candidates who can quickly sling some code that sort of works, I believe that coding tests should be an opportunity for developers to demonstrate:
    • Here are the things I always do,
    • Here is my process, test or no test, and
    • This is what you can expect from me every time I sit down at the keyboard. The way I work has been forged and refined over months, years, or decades, and it is as predictable and reliable as the code’s performance and the output it always, almost infallibly produces.

The valuable structure that this pre-work produces might look something like this but might consume a significant portion of the allotted test time.

import java.sql.*;

public class BusinessEntityController
{
private Connection connection = null;
private PreparedStatement preparedStatement = null;
private ResultSet resultSet = null;

private ArrayList<EntityTypeDefinition> entityTypeDefinitions = null;

//NOTE: Caches business entity data locally for performant referencing
private void CacheEntityData()
{
try
{
logger.info("Attempting to CacheEntityData()");

//NOTE: Get database connection parameter data

//NOTE: Set up and call PreparedStatement

//NOTE: Read ResultSet data
entityTypeDefinitions = new ArrayList<EntityTypeDefinition>();

}
catch (Exception e)
{
logger.error("CacheEntityData() failed due to: " + e);
}

logger.info("CacheEntityData() succeeded");
}
}

OK, so what can we do to make candidate evaluation more effective, less stressful, and less expensive for all involved?

  • Limit the coding test complexity to keep the time required to a minimum. Coding tests may not be the best tool to evaluate a candidate’s problem solving ability, i.e. did they choose the appropriate sorting algorithm and implement it correctly … in the 30 minutes allotted. These tests do offer valuable insight into the candidate’s coding habits — good and bad.
  • Develop, maintain, and utilize multiple coding tests and evaluation rubrics, appropriate for different roles and experience levels.
  • Augment coding tests with repository walk-throughs or inspections, as well as with technical design diagramming, e.g. entity-relationship (ERD) diagrams, C4 diagrams, process diagrams, etc, and writing, e.g. technical requirements, exercises.
  • Consider “take home” coding tests that enable the candidate to use the tools with which they are most familiar and effective. Ask what tools they used, and refer to that valuable list when considering team tooling changes. “Hey, this one comes up a lot. Maybe we should try it.”
  • Ensure that your coding tests are designed to evaluate the things that are most important to your team.

I hope that you’ve found this useful. I welcome your constructive feedback and suggestions.

Please see my other articles on “Technology in the Service of Business”, and check out my premier digital product below!

See Good Data Analysis & Design in Action!

I’ve launched an Open Source product that encapsulates and simplifies all of these Data Analysis & Design best practices and more.

Free Open Source AAF Data Product

Subscribe to get the latest on new features and releases here.

Read about the Adaptív Application Foundation data layer (AAF Data) product — business entity data modeling and persistence — on my Deceptively Simple Technologies digital product site here.

AAF Data includes:

  • Basic business entity models,
  • Database creation scripts for roles, schemas, tables, functions,
  • Scripted lookup/reference data,
  • A Dockerized, RESTful business entity microservice (BEM) providing
    basic CRUD operations, and
  • SwaggerDoc OpenAPI application programming interface (API) documentation

Check out and contribute to the code repository here. Join the revolution!

Learn about AAF Data implementation, customization, training, and support services from my Cygnus Technology Services site here.

About the Author

Doug Wilson is a mission-focused technologist, software development leader, and trusted advisor with 25+ years of proven innovation and problem solving experience, putting technology to work in the service of business.

Today, he advises select organizations on how to increase business agility while avoiding serious business and technology risks. He believes strongly that our systems should encourage not penalize us for learning and improving.

Learn more about the services he provides through his Cygnus Technology Services consulting organization, or schedule a free consultation to learn how to put his experience and unique point of view to work for you.

--

--

Doug Wilson
Doug Wilson

Written by Doug Wilson

Doug Wilson is an experienced software application architect, music lover, problem solver, former film/video editor, philologist, and father of four.

No responses yet