A small transformer like BERT or variants is a better fit. It only takes a few examples, which can be generated synthetically using an LLM.

Trains quickly and classifies speedily on modern hardware.

Had a lot of fun doing stuff like this years ago, before LLMs were a thing.