What I though of was to give candidates a not-so-smart LLM, one that is almost guaranteed to make mistakes.

The idea is to see how the candidates deals with it. Maybe he will just say "fuck it" and do it by hand, or patiently point out every mistake so that the LLM gets it right. Both indicate competence, what you don't want is someone who doesn't see the mistakes and hand you slop.