A fine-tune will generally outperform just about any other method on a closed domain, non generative problem.

LLMs are great because they can handle open domain problems in part because they are generative.