alt Hacker NewsI've tried something like this and ran into a problem which I'm not seeing a fix for. Generally, to really apply this method, you need to understand the person asking the question.
>Tell the user that you will start by asking a few questions, one at a time, to see what they already know.
This puts it into the session each time. Do this once and it seems to work. By the 5th time, you are getting tired of reconfirming your mastery. This ends up depending upon the user to have a pretty solid foundation of what they know and then use this skill to expand the boundary a bit, and even then the AI can end up making so many assumptions that a person has trouble following it along.
I was thinking some sort of concept mapping kept long term might help, but such a mapping can become really difficult to build. Which should be expected when we consider that teaching isn't just its own discipline, but has many sub-disciplines targeted to specific areas (even if some of it, like classroom management, doesn't apply to this scenario).
I made a few attempts at a knowledge tracking tool for personal use to track how much I knew Japanese beyond a simple JLPT level or similar with the plan to tie it into something like this, but it wasn't great. I could 'cheat' and just tap into my Wanikani API to see my progress, but that still skips so many other areas of Japanese, and even when WaniKani says something is burned, I really want AI to still use it until I truly internalize it like I do words in English. I suspect other areas of knowledge are similar.
Yes, lots of LLM Skills quickly turn into applications. The skill is the program and the LLM is the compute, and that leaves the harness to manage the state. I’ve found that these skills need explicit instructions on how to manage state across sessions.
In my work, this typically means an opinionated local folder structure and a dedicated /init skill that implements it. Immediately this means we have more than one skill so we’re talking about a plugin now.
Then I think about meta-state. Not just “what is on disk” but “where are we in this application’s multi-session lifecycle?” For Socratic tutoring, that means checking to see if we’ve completed an initial intake, then what skill levels the user has demonstrated, and finally what’s next on the menu.
For this specific skill, I’m surprised it doesn’t say anything about curriculum. It just assumes the model knows the topic and can improvise both a curriculum and credible assessments of progress in that curriculum. That can’t be true for all things people want to learn, so we need a /build-curriculum skill. And probably a /validate-curriculum skill for actual experts to use before launching a given curriculum.
Anyway, I’ve been thinking a lot about this kind of thing as I roll out skills to my firm.