Any time you want to know if you succeeded at some task, then you have to compare your performance against your goals. In this case, you hired two people at a cost of $X. If your goal was to hire two people then you succeeded. If your goal was to hire four people, then maybe you failed. Since you can just repeat the exercise N times until you hire two more people, then it just depends on whether $N*X fits in your budget.

As for the questions themselves, I'd say that they are too finely graduated on the low end, and not finely graduated enough on the high end. This means that you ended up in a situation where you can't tell if one of your candidates is better than the other(s), while you can tell a lot about the relative skill levels of the people you didn't want to hire. You want more discrimination at the high end so that you know which of your candidates you would like to hire, and which you must hire.

The other thing that you could do is just be willing to hire junior engineers who need training, alongside the senior engineers who can train them. Anyone who got question 8 correct, for instance, but not the rest, could have been a good choice for this role. They would likely have learned a lot over the next year or two, and then you'd have a lot of capable engineers.

Another perspective would be to try to model the costs of the hiring process, and how the overall cost per hire might change if you could adjust the survival rates for each stage of the funnel (or other parameters, such as what the budget is for the role, or flexible working conditions, or how long you're willing to train people, or whatever)

e.g. i imagine the largest cost is going to be onboarding the people that you've decided to hire, not the stages in the preceeding application/scheduleinterview/interview/techinterview/offer steps, so however you fiddle with the funnel it's probably not going to make the process 2x as cost efficient, but there is some risk that the funnel is perhaps selecting for the wrong people .

Agree, the post-hire costs seem easier to quantify.

One of the things I struggle with is what you mentioned above -- what's the opportunity cost of losing people who have a higher ceiling / learning velocity but less immediately applicable experience.

I don't know how to measure that before hiring someone though. Is there a good way to assess that?

That's a different budget category though. I was only referring to the cost of the time used in the interview process. If you've budgeted a month to hire four people and only got two, then you're way behind. But I agree that this may be a small thing compared to the six months you might be expecting a new hire to take before they've become fully productive.

A talented software guy who doesn't know javascript could well make a better long term hire than a less talented guy who does. On the other hand any decent degree holder ought to be able to do this mostly trivial test given enough time but if this is a sit down test then all you've discovered is who is most up to date with their javascript and not who would be the better hire long term.

It ought to give you a bit of self reflection if you realize that your solution to the problem you faced has been pretty sub par.

Reasoning behind the questions:

1-7 - basic HTML/CSS/JS proficiency (and ability to effectively Google for help if they didn't know about CSS' nth-child or whatnot)

8 - get them familiar with the jQuery animate function (which I wanted to use later as a visual proxy for any async function) -- I was more than happy to link them to the jQuery docs or explain the parameters here if they weren't familiar with it.

9 - testing understanding of async functions and how to use callbacks. Ideally they used recursion but I would have accepted a calculated timing-based solution using setTimeouts as well.

10 - wanted to see how they'd go about learning something new given the documentation -- async.js' docs don't have great examples included so I was hoping to see them search for them on Stack Overflow if they didn't quite understand the callback formats.

11 - algorithm design and recursion. But mostly I thought people would get stuck here and I wanted to see how they dealt with that and what they did to get themselves un-stuck.

Take home would make more sense for these... Researching/Googling/SO-ing under 1h limit and when someone watches you, breathing behind the neck, should be uneasy.

This. The timeline is a bit tight for a live interview. They have about 5 mins per task. I think you're probably filtering too hard on people with previous experience using jQuery and don't have interview jitters.

My suggestion would be offload some of those basic questions to an automated at home code test. It'll filter out candidates who can't FizzBuzz, so you're left with a more reasonable 1 hour technical interview with less pointless candidates.

The questions you asked and the reasoning behind asking them was quite good, and I would’ve liked this interview.

The timing though was a little ridiculous. You could, and perhaps SHOULD, easily spend an hour on 10 and an hour on 11.

IME, this is the biggest problem with the current state of the interview process - what I call Hollywood hacker testing. Unrealistic timing expectations based on what’s seen in Hollywood movies.

The take home test is worse for this! At least this wasn’t part of your process...you don’t want have a hiring process where the only person who can put in 60 hours wins.

Be realistic, if you really care about improving the hiring process and not being another dumping ground on Glassdoor for interviewees you’ve angered.

No your questions aren't bad they're actually quite good for what you're testing. Good reasoning to back them up too.

Your funnel also looks pretty normal. I wouldn't be surprised if all 8 fail the test for front-end, welcome to hiring in the real world.

You shouldn’t be hiring if you’re completely oblivious as to why all 8 would fail.

this might be a good way to filter for people who are fluent in the stack & dev environment you are measuring with, and have an intermediate or better level of base programming ability. this isn't a way to measure for other skills you might want people to have in a dev team (e.g. structuring code sensibly, controlling technical debt, experience with production deploys or whatever), and you're potentially going to miss highly capable devs who simply aren't fluent in what you're measuring, yet might be able to ramp up to be highly useful in a month or two.

i'm not a frontend dev , but having an 11 step technical question in only 1 hour seems a bit excessive. i'm more used to having a single technical question in one hour - with a few facets and some asymptotics, with an optional harder extension question for the less common person who can get a good solution to the first problem rapidly.

it kind of depends on what you want for the role --- do you have a bunch of boxes in the org chart where you want people who are intermediate-ish and can hit the ground running cranking out new features? or do you want some people with really good potential and are prepared to invest a bit in training them?

instead of focusing on the technical interview stage of the pipeline, another option would be to try adjusting the earlier stages. What evidence do you have that the filtering from 150 applicants down to 8 finalists is not throwing away a bunch of potentially good hires? e.g. What would happen if you tried to run all 150 applicants through the technical interview first?

We ordered our steps in the funnel that way because we wanted to prioritize people who wanted to work for _us_ (as opposed to getting any job or just the top paying job) and who had good communication skills and seemed like they would make a good fit with the team.

The technical skills are important, for sure, but we'd rather have someone less experienced who was motivated to learn because this was their dream job than someone with tons of experience that didn't really care about having _this_ job.

That's a good point about the timing; I did allow people to go over the time limit if they seemed like they were still making progress and needed it. Prior to the interviews I thought steps 1-8 (which were setting up for 9-11) would take about 15 minutes and we'd have the remaining 45 for the meat of the interview.

One rule of thumb I've found when constructing interviews is take how long it would take you (or any other member on your team) to do the problem cold and then multiply that by 2 to 3x.

This gives two main advantages. The first is you have plenty of time for a conversation. Even assuming ideal conditions it takes between 2 and 3 times longer to walk someone through a solution as you implement it than just to incept an idea. Second is the interview platform is all ready stressful and deeply unnatural so building in some buffer for that necessary context switch helps you extract more signal.

That all being said, if you beat a 2% rate on your total funnel you're doing pretty well. Most people just aren't going to be a good fit; that's just the reality of this industry with huge choice for the candidate (you almost certainly will not be anyone's first pick; the numbers are against you), near limitless specializations, and no real qualification body.

This endless interview question is really stupid. If you’re in an interview saying “OK, NOW!” then you’re a huge waste of time.

Ask specific, probing questions to assess knowledge in your ideal areas, in some combination with an open ended problem solving question.

What worries me is that I had done a bit of that previously in the first interview. Some of the people who seemed perfectly able to talk about things in theory wouldn't have passed FizzBuzz.

For this position we need people who can actually write code so testing coding ability seemed relevant. Hiring someone who can talk about code but not write a for loop or if statement wouldn't have been a good hire.

The open-ended problem solving question is a coding question.

You wasted hours of people's life for no result. You've probably misjudged the good candidates and excluded them for small details.

You should look at a people deeper. Listen to them more carefully. Try to understand what they are interested in. You don't need their resume nor tests for that. If you feel one is interested by programming, hire him immediatly and try to work with him. If you were wrong, assume your responsability.

If you can't do that, send me your candidates. I'll tell you who are the goods.

All talk no action. That's how horrible technical interviews exist to this day. We should all turn away from such discussion since it just wastes hours of people's life for no results as we soon realize that 'interviewers keep asking these questions because they can'

I'm not sure what that first sentence means, but that seems like a bit much of a response for an interview...

A lot of people are "interested in programming" that doesn't make them a good hire, or that they'll ever do the job. I think the interviewer wants folks who can do the job, now.

As the comment above explains, recruiters can do that because we let them do that. I advise people to ask for money to pass a test. The test is the problem's recruiter, not the candidate.

A lot of people say they are interested in programming but very few are. Programming is difficult. More and more people do that for the social position. They should do something they are good at. Most of them who persevere in computers become managers.

If a recruiter isn't able to evaluate someone by talking, he shouldn't recruit people.

I'm not a recruiter. I'm a startup founder. And I'm working on hiring people I'm going to work with every day.

As much as I'd like to hire every person interested in coding, unfortunately that would be detrimental to our core mission of building a product. And in the medium-run it'd be bad for everyone anyway because we'd just go out of business.

I definitely learned I can't evaluate people's technical ability only by talking... which worries me. Because prior to this I expected that I would have been able to predict who would be able to code and who wouldn't. Some of my "favorite" candidates before the technical interview wouldn't have been able to complete FizzBuzz.

You should only hire those who are interested by what they do. The others are bad and won't be productive.

If you can't evaluate if you can trust someone (for something), how can your business be successful? Relationships are everything in trade.

Embrace the risk and pick someone. The one you trust the more.

Recruiters can do ... what?

I'm also not sure what you mean by "ask for money to pass a test".

Pay a wage for passing the tests.

I think I would pass your interview but it seems like a lot of busy work. IMO asking devs to talk through these challenges, instead of actually do them, would be better bang for the buck and more respectful of the candidates' time.

I'd say drop the first 7 questions. You have no reason to doubt anyone who can do the last 4 can do the first 7, and you have no reason to care whether anyone who can't do the last 4 can do the first 7.

You basically just said "eliminate core CSS / HTML questions and only ask about jQuery."

I agree that you'd probably want to reduce the number of tasks. 5-6 would be fine. As someone who does a bit of JS as well as maybe 10-20 other skills I'd be happy doing most of these, but I would need to be able to look up the syntax for a bit of this stuff. I think having a small number of tasks and then talking around is better than having a prescriptive pass/fail. Almost anyone could work this kind of stuff out with the internet and Stackoverflow but would that make them a good fit for your role?

These are not good questions for JS developer interview. Most of these questions can be asked in a phone interview/online test. You want to have people that know how to build stuff. If you are dragging someone for an in-person interview then spend a bit more time on pair programming interview.

Good people do not need to apply anywhere to get well-paid jobs. You want to poach people, find people from open-source contributions or your network recommendations.

1/2/3 - Shows a knowledge of basic, modern CSS.

4 - Somewhat advanced IMHO, and how often do you need this?

5/6/7 - Tedious.

8 - Okay, showing some knowledge of JavaScript. 9 - Showing the ability to write a control structure.

10/11 - Showing the ability to interact with the DOM and modify someone else's code.

Why would you give this code test after the interview? Why not give it to the potentially viable applicants first?

What kind of salary are you looking at? Anyone over 2-3 years of experience should be able to answer most/all of these questions.

We gave the non-technical interviews first because we wanted to prioritize the people who wanted _this_ job over the ones who seemed to be applying for _every_ job.

If we're going to spend a ton of time onboarding and training someone we want them to be happy with what they're working on so they're more likely to stick with us longer term.

It was surprising how many naked resumes we received with no cover letter and no indication that the candidate even knew what company they were applying to.

Just as someone learning / working on a career pivot into the web dev field. You said they get to google... that's the key for me and makes the questions quite reasonable in my mind.

I probabbly couldn't do many of those (well I could some) off the top of my head, but if I can google, yeah I can do that. I know enough to get most of those done.

Heck I'll try them on my own sometime.

I don't do frontend but they all sound fairly easy to me, although 9 is odd (I'm assuming you want people to compute timeouts? To prove they're decent at arithmetic? In which order should the balls animate, row-first or column-first?).

10 is filtering out people who have anxiety about doing things in an interview that they don't yet know how to do -- if your work environment doesn't require learning something brand new while a manager looks over your shoulder, you may be better off having it as a homework assignment.

11 is fine but the people who froze on 9 and 10 because they didn't understand 9, don't like math, had their brain freeze up on them at the "I don't know what async.js is!!!" step, or something else don't get to it.

In addition, while I would be able to do this in an hour, the fact that there are 11 points may be pre-emptively stressing some people (some people get anxious at long lists of tasks, time-limited, that don't entirely fit in their working memory), and things like reading through documentation can take a bunch more time if someone is more thorough or less efficient at searching (prefers tutorials to documentation, has trouble googling "async.js" because of e.g. blog posts on ajax hogging up their search results page), if someone's eyes are tired (at this point they have been coding at least half an hour), and so on.

My suggestions would be, present the questions 1 by 1 not 11 at once, verbally in addition to having the current question in writing. At the looking up docs steps, have the link to the docs ready, or otherwise give a heads-up to the interviewees before the interview ("you may want to read up on async.js as preparation"). Otherwise, give the same exercise as homework.

After questions 1-8, if the candidate is satisfactory, say "great job! I only have a few more questions", thank them for their patience, generally validate their good answers. Make them feel like you're asking for their help vs. just evaluating them.

Sorry, I probably wasn't as clear on the format as I should have been. I didn't present them all 11 tasks up front, I walked them through them 1 by 1 verbally.

The questions flow pretty naturally from one to the next since it's mostly building on the previous step each time.

And yeah, I tried to make them feel comfortable and less like this was a test and more like a collaborative coding exercise. Wanted to try to make it as low stress as possible but I know that's hard given the interview environment.

Re 9 - I was expecting people to use callbacks to do it rather than computing timeouts (but a couple people did try to compute the timeouts and that was a fine solution as well -- asking people about why they chose what they chose led to helpful discussions). Something like this is what I would have done:

    var circles = $('.circle').toArray();
    var current = 0;
    var animateNext = function() {
        if(current >= circles.length) return;
        $(circles[current++]).animate({backgroundColor: 'black'}, animateNext);
    };
    animateNext();

It's not how I would code it but it sounds straightforward for someone who's done any jQuery, frankly. How did your candidates fail on this one? Did they just freeze up and not try?

They seemed to just not be able to grasp the asynchronous nature.

Mostly confusion between calling a function and passing the result as a parameter and passing a reference to that function. But also struggling with the recursive nature of calling a function from inside of itself.

I should also say it wasn't the syntax or JS/jQuery-specific aspects of it that were the problem. Most couldn't explain in English/pseudocode what their strategy for solving the problem would be.

You may be filtering your pipeline according to the wrong criteria, and rejecting candidates strong in algorithms but weak(er) in JS specific or framework-specific stuff at a stage that's too early. Or those candidates may be unicorns -- usually the early-career people who like thinking about recursion don't jump into applying to frontend roles, which they see (wrongly IMO) as a lower tech, design-oriented kind of role.

If it's an entry-level role and you don't have good signals for the steps prior to the interview, maybe consider a pre-interview assignment that includes some asynchronous concepts, and administer it much earlier in the pipeline. That's a slightly lesser investment but should be as good a signal.

If it's a senior role, then your questions are way too easy, but also you should go on Linkedin and ask individual people with good experience to apply, that's what everyone else is doing.

Edited to add: Well, I saw in your other answers that you are filtering candidates based on "really wanting" the job very early in the process... I work in a region that is not the hottest job market wise and even here no one could afford to do this. Software devs are just too in demand... you're going to get pretty mercenary candidates unless you go with personal referrals. If the non-technical aspects of the candidates are more important, which with your ordering of the hiring pipeline it sounds like they are, then you need to consider hiring fewer people and growing less fast, pr hiring for fit first and training the recursion skills later.

> Most candidates were struggling by task 9.

Make the test 5 tasks long.

Good advice. The steps were kind of tedious to read. Maybe just trim it to steps 6-11, which still tests HTML/CSS knowledge.

- make a 10x10 grid of circles, center it in the page horizontally and vertically

- alternate the colors of the circle

- have them switch direction

- animate, etc.

Sorry, I should have explained. I didn't give them a list of tasks at the start; the format was more conversational and I verbally walked them through the list.

The reason I did it that way was to give several easy "warm up" questions at the beginning to get them comfortable.

I like your idea of combining several of the first steps together though -- I may try that and see if that's more or less stressful for the candidate.

I'm honestly amazed anyone can keep focus on menial questions past the third one. Is this really a normal practice? None of the hiring interviews I've taken part in had these. They had maybe one practical problem (that was abstract but not especially difficult) that you could solve in ten minutes of back and forth.

Glad to hear you actually hired those that performed best on the test! To often companies serve out tests that are so hard that none of their current developers can solve them and they still don't hire those that can. Yes the test is really hard. I would have trouble solving it especially if someone was staring at me while doing it. Those you are hired are no doubt stellar developers!

They are not hard for the role

It sounds like you're hiring for front-end developers/designers to sling html, css and a bit of JavaScript rather than "JS developers" which these days could mean someone that works exclusively on the back end. If it's those back end guys making it through the earlier filters then these results would make sense.

It might not be the questions but the rest of the process.

We need people who can do both of those things. Unfortunately there's not much Node-expertise around here so we didn't really come across any back-end Node people.

I figure I can teach someone the Node-specific stuff if they have a solid foundation. Similarly, I think I can teach almost anyone HTML/CSS so I wasn't too concerned about those parts either; they were mostly just necessary as setup for the later questions.

The things some people tend not to ever be able to fully grasp are recursion and async programming which is why I wanted to make sure to include those.

I would succeed in this interview but question the role. Critical thinking and abstract thought seem discouraged with all these "do it or fail" questions.

If this is really a JS role, a Node developer could potentially be an excellent fit with a little CSS and jQuery refresher but would fail the interview miserably.

Sorry, I shouldn't have made it sound like "do it or fail" -- I was limited to 2000 characters here so couldn't do much more explanation in the post itself.

Discussing how to approach the more difficult questions was a big part of the interview and the thought process behind how they approached it was more important to me than coming up with the "right" answer.

We didn't have any exclusively-Node developers in our finalists (I would have loved that though; Node is my jam) but for those that had less JS-specific experience we gave them a couple of weeks to brush up on HTML/CSS and pointed them towards refreshing their memory on relevant JS topics (even going as far as buying a few of them the JavaScript the Good Parts book to read before the technical interview).

You're making the common mistake of not thinking through what it's like for the interviewee in that situation. People are notoriously bad at designing technical interviews because they think of it in the lens of what it looks like for them and not what it looks like for the interviewee

- You said it's a 1 hour interview. It's really more like a 30 minute interview, because the interviewee will likely spend at least half of that time (probably more) thinking about how you're judging their behavior rather than focusing on the tasks. A good interviewer can prevent this sort of thing from spiraling out of control, but it's never eliminated completely.

- #2-#4 are redundant. If the interviewee can create a red circle, they can create more than one of them with a different color. Interviewees will underestimate their time management, so they'll take longer on these trivial tasks. They might even overanalyze what you're trying to do here, thinking that there's more to what you're looking for and spend longer on something trivial

- Questions like #3, #5, #9 are the kinds of questions interviewees will overanalyze. They might wonder what you mean, or whether there's more to what you're looking for. They can ask questions, sure, but you expect X and you won't tell them that you expect X, and they'll spend stupid amounts of time wondering what your X is. It's not an exaggeration either. Sometimes interviewers implicitly expect certain things without making them explicit. And you might only find out later (maybe even never), when they tell you that they wanted something that you definitely could've provided but the interviewer never specified. Or worse, you might spend additional time doing X and then they actually wanted Y. Interviewees will try to think about and tease out these things, which takes away substantial time from solving the actual problem

- Get rid of delay of 1 second for #8, this just slows down interviewees and doesn't give you as an interviewer any special knowledge about the interviewee

- #9 and #10 are basically the same question, get rid of one of them

- Use common sense. You probably expected that most of those 20 people should breeze through that interview. If they get something wrong or get confused about something, don't necessarily filter them out -- Think about the weaknesses in your own evaluation before doing that. Interviews are guidelines not hard rules, don't make the process kafkaesque

- I'm glad you're asking this sort of thing, because people should absolutely care a lot about false negatives. The 4 that made it might be the best ones, they might be mediocre candidates that were just good at your particular brand of interviewing. You want the best people on the team, not the people that are best at passing some weird interview that has little to do with their job

This is good feedback, thank you.

to be quite honest I would say this represents a lot of aspects of what's wrong with tech hiring. it's basically an abstract quiz, you're asking people to do random semi-related challenges while a clock ticks down and someone watches over their shoulder. this is basically zero percent representative of what actual coding is like and likely filtered out a few people who would have done well at whatever the actual job you're offering is.

it's not that the questions are too hard, it's that the whole challenge is bogus. if you have to ask these questions, make it a take-home challenge and discuss the implementation with people when they come to the in-person interview. being able to explain the hows and whys of already-written code is much more valuable than being able to write code under pressure, and much closer to what someone would actually be doing at work.

even better would be to figure out what exactly you're looking for in candidates and design the interviews to reflect that. right now it seems like you're looking for people who are good at coding under pressure... is that a normal part of the job? it very well might be, but i doubt it. think more about what you really want from people on a day-to-day basis.

You can merge some of the questions.

hire people, not test results*

*test results in a foreign, artificial, high-stress environment, using an IDE no one uses for normal work.