How to identify a skilled programmer during an interview

How does one identify a skilled programmer?  No company that has interviewed me could tell the difference between myself and other programmers they'd interview.  The interview process is truly a game of luck in this industry--on both sides.  Both the programmer and the company are basing their actions entirely on luck.

Companies have come up with numerous methods to attempt to discern a good programmer from a bad one.  The best tricks they have include a series of math problems, algorithms, problem solving technique tests, and even obscure programming questions, some without definitive answers.  As an example: Is there an authoritative source of information on the core principles that define object oriented programming?  I've heard everywhere from 3 to 7.  In a field of research about a synthetic concept, an authoritative answer is almost impossible to obtain.

Programmers were then forced to study to the interview.  Careercup is one of my favorite sites for this.  This almost certainly detracted from them growing in knowledge about their field, but also resulted in an interesting outcome.  The people selected as the best programmers were the best interviewers with the ability to memorize large amounts of obscure bits of information.  While this skill may benefit a programmer, it certainly isn't the ideal way to determine the quality of that programmer.

Of course, when companies saw their attempts not working, they started getting more creative, which really just means they got more obscure in their questioning.  It may not be news to you, but it is unbelievably silly to me.  There are programmers who deliberately train in order to write code that compiles on a whiteboard!

Seriously, there is never a case where writing code on a whiteboard has benefited me.  Whiteboards are certainly important for explanation and diagramming code concepts, but never for actual code.  Pseudo-code at best.  And even then, an IM (at the very least) would be a faster and easier form of communication.

If you're with me this far, you must agree that the interview process for programmers is silly.  So how do we fix it?  For one thing, we need to ask questions relevant to programming.  Some companies are doing just this.  The interview process is long and tedious, but effective.  They send programming problems that are custom created to their interviewees.  Of course, there are flaws here too, some of this is preempted by (can you believe it) over the phone coding samples.  Additionally, the companies who create these interview questions are more concerned with the result than the process, which is contradictory to the way programming works.

I can provide such an example from my own experience.  Once I interviewed for a position and they told me that they wanted me to be tested on a new technology which I had never worked with before.  When I explained that it was new to me they said "Great, then we'll really get to see how you do" which is precisely the right answer.  Unfortunately, it was all talk.  The test they gave me was ultimately broken.  They hadn't even attempted to compile their own code.  In the extremely limited time frame (2 hours) I was only able to debug it enough to get it to compile correctly.  That's right, in 2 hours I figured out a brand new technology and fixed broken code.  But that's not the answer they wanted.  They wanted a complete program built around this technology.  I was surprised, but I shouldn't have been.  Good ideas, failed in the end.  They said they wanted the tech to be new to me (it was not in their job requirements at all).  This may have been true, but then they were hiring based on the resulting code wrapped in that technology.  Of course, this turned the interview game back into a game of luck.

So companies are getting the right ideas but they're not executing them well.  I mean, Amazon.com still asks questions like "estimate the number of jelly beans in this jar".  So the followup question that anyone who's reading this probably really cares about is "what are the right questions?".  I am going to list them, but I'm going to ask for patience here, because the questions don't make sense unless you understand where I'm coming from when I write them.

First, the interviewer needs to understand that programming is not like factory work.  This is a brain oriented business, not a wrench-turning mentality.  Therefore, being able to recite techniques off the top of his or her head is less beneficial than basic problem solving and research skills.  This would lead programming interviews to be more like academic or scientific interviews.

Programming isn't like academic or scientific work either.  In fact, in a world where the technology to learn is so readily available as it is in programming, and the only barrier to that knowledge is the person.  Self-taught programmers benefit from a drive that most scholarly programmers don't have.  In fact, in programming, those with degrees tend to be less skilled than those without, all else being equal.  Discovering that, most hiring managers tend to assume that they should then base a programmer's interview on their own.

Programming is most certainly not like management.  Programmers benefit from a lack of social interaction, which is of course the opposite of what most managers benefit from.  I've literally had companies turn me down as a candidate for the position when I said "I'm not great with people".  While a level of courtesy is demanded by anyone working with others, to be excellent at dealing with other people certainly does not fit my job description (and I don't want to work for a company who thinks it does).

Further, anyone who's done any research on interview techniques knows that it is a bad idea to ask simple questions.  This is more important in a field as diverse and complex as programming.  Therefore, any silly trick questions, or questions with one sentence answers are a bad idea.  Every question I've written is designed to provoke a discussion, so expect that to happen during the interview.

Programmer's are a unique type of people, and they therefore demand a unique sort of interview.  So here are a few simple questions I've come up with as a starting ground for the way that you might interview a programmer.  If this post becomes popular, you certainly will want to deviate from these questions.  But the core ideas are important, and without it, you will gain no real benefit from this post.

1. If another developer is overworked and one of his projects is assigned to you, what do you do when you have difficulty understanding his code?
1. This questions is a qualifier.  A junior developer will answer that he will ask the other programmer for information or if he's overzealous preclude that he would figure out all the code on his own.  These answers are distinct and do not mix in junior developers.  Make no mistake, both of these can blossom into great developers, but if they answer your question this way, they are a junior developer in mentality.
2. A mid-to-senior developer will go along with the idea of asking for help from the previous programmer only after he has exhausted all other avenues of research (including Google), and this period of research is sure to come up in the discussion.  In fact, if you spend enough time, this question alone might let you define the difference between mid and senior as well, but this is all about a discussion, not a test.
3. A non-developer would say something like ask the Project Manager or read through the documentation.  These sorts of things aren't even discussed by developers as they're assumed in the question and by the time the programmer has begun answering, he has likely assumed that these are useless.  Again, remember that these questions are meant to provoke a discussion.  A skilled programmer might mention briefly about documentation, but the core of the discussion would revolve around 1 or 2.  If you struggle to get him to discuss, you might want to throw out examples of where they got stuck in the code.  If you are not knowledgeable enough to do so, bring a programmer in on the interview.
2. You are taking over a project filled with legacy code.  Where do you begin?
1. This question is relevant to your organization.  If you have legacy code and you absolutely do not want to upgrade, then the last thing you want him to answer with is "A complete rewrite" (but many programmers will laughingly suggest this first even if they don't mean it).  Alternatively, if you're trying to learn to be a more agile organization, you might want him to consider things like re-architecting the code.
2. Regardless of his answer here, which should help you determine how he'll fit in your organization, there are important key ideas to listen to in the discussion.  You want him to talk about refactoring, even if he doesn't know the exact word to define the concept.  You want him to talk to you about testing the code to verify it works the same as it did before (if he goes into detail about testing, he's more senior developer level than junior).  You want to hear the programmer expound his favorite software concepts.  Keep the discussion going as long as you can and you can really grasp stuff here.  It is doubtful that a programmer would run out of things to say here, so if your interviewee does, then there's reason to suspect he may not be a real programmer.
3. Describe a large scale project you worked on by yourself.
1. Not all programmers have worked on large scale projects on their own, but the good ones have.  I've got at least half a dozen websites, from e-commerce to D&D character sheet web-apps that I've built on my own, and that's just in my free time.  Don't set an explicit standard (and definitely don't base it on me), but the more large scale projects they've built on their own, the more capable they are as a programmer.
2. While any good programmer is going to be very proud of his or her work on his own, this will begin the process of seeing how they work with other programmers.  Sometimes it's purely a matter of the jobs they've had that define whether they've worked with other programmers enough to have had a good team.  But you can start to sense whether they have an intimate dislike of working with others from this discussion.  This is the first question on this list that really will show you what type of programmer they are, instead of simply if they're a programmer at all.
3. Tell the interviewee to use a whiteboard.  From his use of it you can also learn about how he explains things.  Bring another developer in for this question.  See if the other developer can understand the concepts after the discussion.  A senior developer should be able to effectively articulate the architecture to the other developer, and it should be interesting enough to get that developer involved and asking questions.  A junior developer will struggle more here, and he may ultimately be describing a mess of spaghetti code, but it's all his, and the developer in the room should be able to help you define the developer's skill level.
4. This question does not imply it's opposite.  It is generally a bad idea to ask about a large scale project that programmers have worked with a team on because their role is dependent largely on the organization, and not themselves.  They may not have been involved in the architectural process, or even be permitted to know what their team members were doing (security and all).  And of course, they may not be permitted to talk about that project at all (NDA).
4. When storing user information in a database, programmers encrypt or hash passwords.  Why?
1. This is another good qualifier.  It's not about knowing a particular concept.  If you have to explain to the interviewee about databases and encryption, you might have an issue, but most of the concepts, hell even hashsets can be completely alien to a highly skilled developer.  This question instead asks the more broad question of  "Why"
2. The answer of course is security, but don't let the discussion end there.  After all, the data is stored on your servers, how could there be concerns about the data being accessed from the outside.  Concepts like SQL Injection and even SSL & Session storage can easily mix into this discussion.  We're not eliminating programmers based on a specific set of arbitrary terms here, but based on concepts and large-scale understanding about security over the internet.
5. What language are you most confident in?
1. I am honestly surprised to tell you I've never once been asked this question.  Not one company that has hired me cared if I was most confident in their preferred language.  And, if my company told me they were going to ask this question in the interview, I would advise them against it.  Because they aren't in the mindset that I've emphasized throughout this blog post.
2. When you think about programmers like researchers, you can think about this question differently.  It's not about the answer, so much as how quickly they answer.  A programmer who has not explored a lot of languages thoroughly will answer immediately, and a non-programmer will answer immediately with the language listed on your job requirements.  But a programmer with a breadth of skill across multiple languages, someone who dedicates his life to the pursuit of programming knowledge will pause, and list a few of his favorites.
3. The languages that are mentioned matter too.  If he is a web-developer he will list among his top 3 languages JavaScript and some back-end language like C# or PHP.  If he is a windows developer, he'd better mention .NET.  If he's a linux guy he'll likely mention Perl.
4. This question should spark others, like "How did you get into that one?"  And "What was your first language?"  Steer this discussion correctly and you can get a programmer to describe his entire career.

Well, it's just a few, but it's a start.  I hope that you benefit from these questions and that the programming community as a whole can start better hiring practices, which create better code in organizations, which creates better programmers competing for those jobs and helps the community grow stronger.

One last word of warning.  I mention bringing programmers in on the discussion, but for many organizations this is a double edged sword.  Some organizations are acutely unaware that they have hired incompetent programmers.  What's worse, some programmer's have taken to protecting their jobs by ensuring the organization hires people less knowledgeable than themselves.  If you bring a programmer in on an interview, there are a few key things to look for to ensure that he is a participant, and not a barrier.  First, if he bogs your interviewee down in the details of code implementation, he's not being helpful.  If he starts trying to ask questions on his own that are outside of the existing discussion, it's important to ensure that he's doing so in good faith.  Finally, if your interviewee fits your qualifiers as a skilled senior developer and the developer you involved in the discussion suggests not hiring him, it may be a personality conflict, or it may be something more.  Put simply, more communication is better than less here.

Alright fine, one last one.  This is a gift.  Any modern day programmer should know the answer to this question.  If they don't, they're not a real programmer.  If you're reading this post in the distant future, this question may no longer be relevant, but at the time of posting it is extremely relevant.

What is StackOverflow.com?