I don't think they fit in as a layer of abstraction, but instead are outside of it. An abstraction simplifies away the inner workings of what is being abstracted. The LLM exists outside of your code. It is not part of it, thus, it is not abstracting it away. If this were the case, a coworker would be an abstraction to code they own (you could argue this, but I think it erodes the meaning of abstraction). LLMs behave like program synthesizers rather than another layer of abstraction. They take natural language as input, and using fancy math produce a (hopefully) relevant and useful output based on that input. They can produce layers of abstraction, but are not part of a program's abstraction stack.

However, they can abstract away the need to understand implementation, similar to a coworker. They can summarize behavior, be queried for questions, etc, so you don't have to actually understand the inner workings of what is going on. This is a different form of abstraction than the typical abstraction stack of a program.