Depends on whether you define "the LLMs we use" as the collection of weights or if your definition contains the software stack that runs it

Technologically the LLMs we use today don't implement this behavior, but you could take the weights of Sol and add a couple (very large) patches to vllm (or whatever OpenAI has today) and have a version of Sol that does have "memory"