I've seen a few people comment along the lines of "yes but LLMs dont need lot of data nor competence". So with our classifier "Linnaeus", the data came from JIRA tickets plus additional signals. To get up and running we didn't use a lot of data (a few dozen tickets across approximately 20 classes to begin with - eventually expanded to 120+ classes). There was very little to no manual data labelling - mostly it was devs selecting "gold standard" JIRA tickets for us to ingest. The model was retrained daily, and we had extra signals like thumbs up/down with comments that would be used for re-training.