>From my admittedly anecdotal and narrow point of view, once you clear a baseline of competence, your willingness to do the work takes you much further than knowing a bit more.
From my own, 15+ years of experience focusing entirely on this problem point of view, I can't emphasise enough how much I agree with this principle. Finding people who are technically capable of doing the job isn't hard. Finding people who can do the job well, that actively make doing the job more rewarding is much more difficult and no technical assessment on earth can screen for that.
>Nobody should burn tokens for a company that hasn’t had a proper conversation with them yet.
PREACH! This same sentiment applies to all tasks that you expect candidates to do during an interview process. If you want them to put in the work, get buy in first. You get that buy in by having conversations with them and helping them understand why they should care about your company and your opportunity in the first place.
I honestly have no notes on this post other than to encourage you to hold on to your current way of approaching the problem because I promise you, you are absolutely on the right track!
> They can’t get a visa. They live in another city. The contract terms don’t work for them.
I have to share my stories. For a position I’m a really good fit for at a company I’m familiar with, I asked a guy to ask a guy, and they said they would only hire in the US because the team is in the US.
For another position, they reached out to me. They can’t hire in Japan but offered relocation.
In both cases, I’m super grateful that people are willing to help, but I could sense that times have changed since 2023. Globalization is no longer that strong; many, many jobs are restricted to certain locations, such as certain cities in the US or India. I don’t know about the EU, though.
This gives me some hope that if I’m willing to leave Japan, I can get a good job.
> Globalization is no longer that strong; many, many jobs are restricted to certain locations
It's not entirely clearly what time periods you're referring to, but when I entered the software development industry (Europe), almost 0% were remote, and it was virtually unheard of. Skype was just beginning to become mainstream around this time, and slowly more remote positions appeared, which culminated with COVID, and then slowly disappeared over time.
But today there is still a lot more remote jobs than existed even a decade ago, and a huge amount of remote jobs available today compared to two decades ago. Globalization seems to be going up and down but nowhere near as isolated as it used to be. But again, I think the perspective changes greatly depending on your reference here.
I appreciate that I'm out of touch here, since I have been several years at the same job, but I can't really see how the LLM should have any role in the interview process at all - indeed, other places are going to increasing lengths to keep them out! You're not interviewing the LLM, after all.
I suspect LLMs make the "take home coding challenge" even more useless than it already was. If you want to make it part of the candidate interview, then I guess you've got to sit them down with a laptop.
I don’t understand why giving someone an api key for their limited use while doing an interview or assignment was thrown out. This is an obvious answer and other companies are doing this. For example, C1 talked about doing this on a recent episode of the Cup o’ Go podcast.
-If the AI part is so important, why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter? That way you could even level the playing field by giving everyone the same access to the same models.
-I believe some of this is temporary. Just about everyone is now constantly using AI, so I believe we'll soon care more about someone actually being able to explain how they'd go about something and why rather than just "can you produce code that does this"
-The idea of "applications are a numbers game" and spewing hundreds of resumes everywhere is dying. I know extremely highly qualified people whose resumes never get looked at and recruiters who never look at applications. Everyone is making application materials with AI and HR teams review them with AI. I believe we'll enter an era that's much more manual and high touch.
What's the risk in giving someone a deepseek api key that is loaded with $5? Five dollars in deepseek tokens is enough for like a week of constant use.
Yeah I don’t see why the idea was thrown out so flimsily. Surely you can set up a sandbox possibly a wholly separate account from the corporate one and give that to the candidates
- I'd hire someone who keeps building even when jobless.
What you build is your CV/Cover, and your learning opportunity. These days, you have to know LLMs inside and out, which means using them heavily to learn.
- The subscription is probably below or around the daily rate, so it pays off quickly.
I'd argue that it's an investment in your education - not only learning to work with LLMs, but also using LLMs to teach you almost anything (even beyond tech).
I'm surprised companies are doing take homes in an era of LLMs.
I suppose it can save valuable paid time by forcing the candidate to waste theirs instead, but surely you get a lot more signal from actually talking to the candidate and seeing how they think and understand what they're doing.
You can actually talk to the candidate about their solution: why they picked one approach over another, why X wouldn't work, how well they use LLMs (e.g. 10-line comments for a single line of simple code? do they reuse functions or create four different clones?), etc. With LLMs, assignments can be larger/more complex so you can see more.
I'm not saying it's better or worse, but I don't think assignments are inherently bad. It works the same way in education from what I've heard - written assignments may not be very valuable on their own anymore, but you can ask students questions about specific things irl to see if they actually know their stuff.
Very interesting post about hiring. I am no hiring professional, but after reading to the end this stuck out the most: Do it together.
If you expect LLM usage proficiency you can easily check that on the spot, no need to bother the applicant for private token usage.
> How you behave, how you relate to others, how you approach problems and the energy you bring to a team matter as much as your technical ability, and often more.
As much as I agree with this, I don't think that the whole rest of the post (about how best to examine LLM usage in an interview loop) coheres with this.
You cannot conclusively test for common decency in either interviews or recommendations. People are on their best behavior in interviews, and it's rare for someone to pass along a recommendation that will criticize them (or their behavior). Length of tenure at prior organizations is not proof, since too many will tolerate poor behavior. Accept that this is a risk involved with any hire, have clear, written business conduct policy, and be willing to show the door to people who cannot control their behavior after written warnings.
Asking how people approach problems with LLMs need not be more than a conversation. Architectural interviews need little more than a whiteboard. If you wouldn't hover over the shoulder of an employee while they work, why do you feel the need to do so in an interview?
I get it - bad hires are expensive in more ways than one. But screen-to-offer should not take longer than two weeks in small companies. Interviews help you derisk, but do not ever fool yourself into thinking that an interview process will ever reduce your hiring risk to zero. Getting a handful of dates before a shotgun wedding will always be risky. Accept it and learn to be more comfortable with risk.
> the energy you bring to a team matter
This is all the more true for hiring managers. Bad management will take strong, high-energy, high-ownership performers and handcuff them. Good management can, sometimes, take people with weaker skills, train them, set them up for success, and inspire them into becoming great performers. Someone who is interviewing, who might accept your job offer, is also taking a risk on whether you are the bad kind or the good kind of manager. The only difference is that job seekers often have more appetite for that risk than job offerers.
I honestly don't get the point of asking the candidate to use a LLM. If you think that they can replace technical skills (which is fair), why testing these skills anyway ?
Why not put a proxy in front of the model, give each candidate a key with a hard budget, and revoke it after? If you require AI for the interview, you should pay for it
I like the idea to consider the token cost for the candidate, good.
But why not keep it "on paper", asking how the candidate would use an LLM or agent and why?
Actually using an agent means asking the candidate to deliver something that you must validate. And you then need to evaluate the candidate by looking at the output of the LLM indirectly.
Isn't it better to directly ask the candidate to explain how an LLM could be used (or if/when it should be used) for your use cases?
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.
I think you know how to do this. You are head of engineering at an AI startup. You know how to create and revoke API keys and how to share them with your interview candidates.
I thought about this but it's also a liability and puts too much strain on me.
1. Whatever limit I put needs to be completely arbitrary. 5$? 10$?
2. I cannot guarantee they will use it correctly. They might use it for the challenge. They might use it to generate harmful content.
3. It requires me to manually create and revoke keys. Candidates might want more credits. They might get leaked.
There are solutions. I just can't think of good ones.
virtual card. some of the startup finance/banking tools can issue one for a fixed dollar amount. candidate uses it to bump their own sub for a month or top up credits.
It’s not all candidates, only candidates who progress to a certain gate in the process.
I think “we offer a $50 Visa/Amex gift card as part of this interview step” is easier to admin, might skate by finance more easily, and avoids all the tech support angles of “here’s how to use our specific tooling”.
Setup a tcp/http proxy behind auth, ask candidates to use that endpoint, set that proxy to add whatever you need for properly authed LLM inference. You get both instrospection + reuse whatever subscriptions you have. Mention to the candidate that the requests go through your infra though, so you set the right privacy expectations.
> Giving candidates a company account for the challenge is off the table: it’s a liability, and I can’t guarantee how it will be used. Reimbursing what they spend doesn’t work either. It’s hard to justify as a business expense, and it quickly drifts into a financial gray area.
If you can't give qualified candidates who request it an OpenRouter key with $50 on it how are you affording to pay salaries? How little do you value the interviewing and CV review time of your current engineers? What's the liability you're worried about incurring?
If you're looking for a job without a stable source of income every expenditure matters. You need to cover essentials -- rent/mortgage, clothing, food, utilities, etc. Those give you a baseline burnthough rate: how long you can survive on just your savings. Any additional expense decreases that and makes it harder to build back up if doing things like saving for a house.
I've been a broke student with loans where I had to decide between gas to go to an interview or eating that day. It's not about 20$, is about how morally ethic it is to ask people to use their own personal money to just have a chance to get hired and how to try and minimise that.
The relevant question is “how incidental/baseline to life is this expense?”
We require clothes for on-site interviews. We require phone service for phone screens and electricity and internet service for video interviews. We don’t pay gas/mileage for on-site interviews for local candidates. People think of those as incidental/de minimus expenses and we ignore them.
I think LLM subs are going to be just as baseline as mobile phone service very soon (far, far faster than mobile phone adoption was in the US or Europe).
Ok how about this. Employers create an automated interview that requires you to burn through a $20/month subscription for every job application. When you run out of tokens, you can't apply to another job, so now you need $200/month subscriptions during the job hunting phase.
If you think $200/month is still too cheap for a job, then let's say each individual job application requires a $200/month subscription.
>From my admittedly anecdotal and narrow point of view, once you clear a baseline of competence, your willingness to do the work takes you much further than knowing a bit more.
From my own, 15+ years of experience focusing entirely on this problem point of view, I can't emphasise enough how much I agree with this principle. Finding people who are technically capable of doing the job isn't hard. Finding people who can do the job well, that actively make doing the job more rewarding is much more difficult and no technical assessment on earth can screen for that.
>Nobody should burn tokens for a company that hasn’t had a proper conversation with them yet.
PREACH! This same sentiment applies to all tasks that you expect candidates to do during an interview process. If you want them to put in the work, get buy in first. You get that buy in by having conversations with them and helping them understand why they should care about your company and your opportunity in the first place.
I honestly have no notes on this post other than to encourage you to hold on to your current way of approaching the problem because I promise you, you are absolutely on the right track!
I have to share my stories. For a position I’m a really good fit for at a company I’m familiar with, I asked a guy to ask a guy, and they said they would only hire in the US because the team is in the US.
For another position, they reached out to me. They can’t hire in Japan but offered relocation.
In both cases, I’m super grateful that people are willing to help, but I could sense that times have changed since 2023. Globalization is no longer that strong; many, many jobs are restricted to certain locations, such as certain cities in the US or India. I don’t know about the EU, though.
This gives me some hope that if I’m willing to leave Japan, I can get a good job.
It's not entirely clearly what time periods you're referring to, but when I entered the software development industry (Europe), almost 0% were remote, and it was virtually unheard of. Skype was just beginning to become mainstream around this time, and slowly more remote positions appeared, which culminated with COVID, and then slowly disappeared over time.
But today there is still a lot more remote jobs than existed even a decade ago, and a huge amount of remote jobs available today compared to two decades ago. Globalization seems to be going up and down but nowhere near as isolated as it used to be. But again, I think the perspective changes greatly depending on your reference here.
I suspect LLMs make the "take home coding challenge" even more useless than it already was. If you want to make it part of the candidate interview, then I guess you've got to sit them down with a laptop.
-If the AI part is so important, why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter? That way you could even level the playing field by giving everyone the same access to the same models.
-I believe some of this is temporary. Just about everyone is now constantly using AI, so I believe we'll soon care more about someone actually being able to explain how they'd go about something and why rather than just "can you produce code that does this"
-The idea of "applications are a numbers game" and spewing hundreds of resumes everywhere is dying. I know extremely highly qualified people whose resumes never get looked at and recruiters who never look at applications. Everyone is making application materials with AI and HR teams review them with AI. I believe we'll enter an era that's much more manual and high touch.
Because 90% of IT departments' eyes just glazed over. Assuming you're interviewing at a company big enough to have one.
- I'd hire someone who keeps building even when jobless.
What you build is your CV/Cover, and your learning opportunity. These days, you have to know LLMs inside and out, which means using them heavily to learn.
- The subscription is probably below or around the daily rate, so it pays off quickly.
I'd argue that it's an investment in your education - not only learning to work with LLMs, but also using LLMs to teach you almost anything (even beyond tech).
I'm not saying it's better or worse, but I don't think assignments are inherently bad. It works the same way in education from what I've heard - written assignments may not be very valuable on their own anymore, but you can ask students questions about specific things irl to see if they actually know their stuff.
As much as I agree with this, I don't think that the whole rest of the post (about how best to examine LLM usage in an interview loop) coheres with this.
You cannot conclusively test for common decency in either interviews or recommendations. People are on their best behavior in interviews, and it's rare for someone to pass along a recommendation that will criticize them (or their behavior). Length of tenure at prior organizations is not proof, since too many will tolerate poor behavior. Accept that this is a risk involved with any hire, have clear, written business conduct policy, and be willing to show the door to people who cannot control their behavior after written warnings.
Asking how people approach problems with LLMs need not be more than a conversation. Architectural interviews need little more than a whiteboard. If you wouldn't hover over the shoulder of an employee while they work, why do you feel the need to do so in an interview?
I get it - bad hires are expensive in more ways than one. But screen-to-offer should not take longer than two weeks in small companies. Interviews help you derisk, but do not ever fool yourself into thinking that an interview process will ever reduce your hiring risk to zero. Getting a handful of dates before a shotgun wedding will always be risky. Accept it and learn to be more comfortable with risk.
> the energy you bring to a team matter
This is all the more true for hiring managers. Bad management will take strong, high-energy, high-ownership performers and handcuff them. Good management can, sometimes, take people with weaker skills, train them, set them up for success, and inspire them into becoming great performers. Someone who is interviewing, who might accept your job offer, is also taking a risk on whether you are the bad kind or the good kind of manager. The only difference is that job seekers often have more appetite for that risk than job offerers.
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.
1. Whatever limit I put needs to be completely arbitrary. 5$? 10$? 2. I cannot guarantee they will use it correctly. They might use it for the challenge. They might use it to generate harmful content. 3. It requires me to manually create and revoke keys. Candidates might want more credits. They might get leaked.
There are solutions. I just can't think of good ones.
I think “we offer a $50 Visa/Amex gift card as part of this interview step” is easier to admin, might skate by finance more easily, and avoids all the tech support angles of “here’s how to use our specific tooling”.
If you can't give qualified candidates who request it an OpenRouter key with $50 on it how are you affording to pay salaries? How little do you value the interviewing and CV review time of your current engineers? What's the liability you're worried about incurring?
linkedin-drivel.
We require clothes for on-site interviews. We require phone service for phone screens and electricity and internet service for video interviews. We don’t pay gas/mileage for on-site interviews for local candidates. People think of those as incidental/de minimus expenses and we ignore them.
I think LLM subs are going to be just as baseline as mobile phone service very soon (far, far faster than mobile phone adoption was in the US or Europe).
If you think $200/month is still too cheap for a job, then let's say each individual job application requires a $200/month subscription.