alt Hacker Newsso, i've gone through a bunch of iterations on how to interview in days of LLMs and i came up with a process that is quite good:
first, a coding problem that is solved live (right now, online) -- it's not a hard one and most LLMs can easily one shot it.
BUT: if you are not a senior developer, you need to do it without LLM, and if you are a senior+ and do it via LLM, the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
the main component here is to know if the candidate understands coding in general, understands problem solving and how to dive into an existing (although quite small) codebase. solving the problem is not really the goal -- in fact, most candidates we hired did not solve it 100%, but were able to understand and explain everything that they put it on code. this usually takes 1h30.
(we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
the second is an architecture interview, where we build a system together BUT instead of doing the whole spec, miro board, whatever, we do it in more of a free way: we start with a really simple, MVP level, spec and we ask how the candidate would build it. then, we add another feature, and we ask how they would go for it. during this whole thing, i ask random questions like "why would you use x instead of y?" or "why do you prefer this approach?".
one important thing: we allow the candidate to say "i don't know" and we tell the candidate that there's no gotcha questions (which is true, i hate these kind of questions). this lets the candidate be super comfortable (feedback we had from real people).
in general, this flow helped us a ton and i really really like running it :)
(edit: spelling)
Replies
we also ask the candidate to open a PR against the repo just to see if they know the basics of git
A candidate may be able to open a PR and not be able to make a commit without an IDE.
Another candidate might know plenty of git, but haven't worked with Github.
Git is a simple tool. I expect any engineer can learn to use it. While git is common, I would not expect an engineer who worked at a company using a different version control system to know how to use git (which is a really awful - I still miss mercurial). I expect you to know the general concept of version control, but the specific on is a tool I expect you can learn quickly.
I run a live coding challenge on a rather benign coding problem with the explicit expectation that candidates will use an LLM.
Interestingly, an LLM involved feels like it’s made it easier to assess candidates. Senior candidates know exactly what they’re doing and strongly driving the LLM. They pre-communicating design decisions, gotchas, and requirements. They also ask a lot of clarifying questions and dial in the scope (after all not building something is better than building it).
Less experienced candidates tend to jump right in, build way too much, and stumble because they delegated all of the critical thinking to the agent. They often end up in a place where the agent doesn’t align with their vision and they can’t really reign it back in.
> (we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
How does that work? Git has no concept of pull requests.
> we also ask the candidate to open a PR against the repo just to see if they know the basics of git
I've heard this multiple times now and it still boggles my mind that git is something you have to test for.
how do they grade a right answer, suggested by the agent itself??
What I though of was to give candidates a not-so-smart LLM, one that is almost guaranteed to make mistakes.
The idea is to see how the candidates deals with it. Maybe he will just say "fuck it" and do it by hand, or patiently point out every mistake so that the LLM gets it right. Both indicate competence, what you don't want is someone who doesn't see the mistakes and hand you slop.
> we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it
> (we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
Worked with a "super senior", lead architect. Management was raving about the guy before he joined. Insane resume. Skipped the regular interview process because of this.
Had to show him how to open a PR and setup his git config so his commits are linked to his account. And then his PR's failed the automated ci checks for descriptions and titles, and he came to me to ask how to fix it. The failing gh action literally says what was broken.
This is all with Claude available btw, so one could literally just ask.
I guess it's a bit of a modern fizzbuzz. Where it's so simple but still manages to stump a lot of candidates.
How would you prepare for the coding problem as a senior? What are the assumptions that underpin this?
> we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it
Will you allow jj instead. I switched to jj 1.5 years ago (as many others have), and my git is now rusty.
I mean sure, I can do add, commit, basic diff, etc. But rebase/merge - oh dear, I forget which one is source/target (jj made the infinite design choice that the ordering doesn't matter - you have two parents making a child - it's symmetric).
>>> the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
You have missed the point entirely.
Something similar for senior devs: code reviews. Reading and understanding code has always been important. I find offering simple code reviews very telling. In a way, your LLM approach matches that. It's a way to do a code review. It's a test of understanding, and not just memorization.
[dead]
I feel like git operations are one of the few things that I truly don't care at all about whether someone uses an LLM for and doesn't tell me. PR descriptions and commit messages should be written by humans for humans, but like, if someone doesn't know how to manually configure their git remotes and what exact command to use to amend/squash their commits into the shape needed and instead has an agent prepare all that and use the GitHub CLI to open the PR, why should I care? Beyond the code itself, I care about the things that will continue existing in the long run fitting my team's conventions (the number of commits, the commit messages, the PR description and comments, and then whether there's a merge commit at the end), but the process used to create those things doesn't matter to me.