alt Hacker NewsI built a low-latency AI companion that plays Skyrim with me
Comments
>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it)
I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old.
unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?
Very promising project! The latency is great and looks pretty polished for solo work.
> ALE is designed to be largely invariant to phrasing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it doesn't matter, it will still understand you > It creates embeddings from the full text as well as its extracted structure
I get how you utilize embeddings, but most interesting part for me is how you decompose multiple commands? You decompose sentences before embedding?
I like the approach here. It’s cleverer than just “dump context to LLM”. I couldn’t quite figure out if it genuinely runs purely locally though.
If nothing else - this is how NPCs should work in games moving forward!
This is the sort of thing that the GPT-Live model would excel at when it launches. I'm not sure you'd need the ALE model at all if they launch Live in the API with acceptable pricing as it solves the latency issues with voice entirely. It's an impressive piece of engineering that hasn't been discussed much yet, I guess because it's only a ChatGPT feature and not accessible to developers. The model can dispatch tools asynchronously while speaking which would be ideal for a game.
What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.
Love the concept of void mode. A companion that follows you across games, that's cool.
This is a fantastic application of an LLM. What’s great is that the LLMisms fold neatly into “dumb but lovable sidekick”.
So even when it chokes or stumbles on a command, the kind of frustration the user expresses when correcting it feels natural and part of the game even.
An 80% good enough solution would be to have it periodically say "Hey, you. You're finally awake."
Wow, what a great demo video. And this is the worst this tech will ever be. Though maybe nobody will ever beat the dog's hilarious persona.
Great concept and presentation. How much knowledge about the world does the agent have? Does Skyrim accurately model what their in game character can actually see and hear?
>It creates embeddings from the full text as well as its extracted structure, then semantically combines and compares it with action prototypes. A separate classifier estimates whether the turn is a command, question, chat, clarification or complex request. Everything gets merged together.
Does this mean the prototypes and classifications need to follow what you can actually do in the game? Were they all hand-coded or generated somehow?
Some variant of this idea has played in the back of my mind over the last few years. I'd love to play Skyrim, but where each character builds their own context for their history, their personality, their goals, and their interactions with you. The world would feel incredibly alive.
Nice demo, I'm always impressed by making these things low latency enough to be actually interactive. Were you inspired at all by the autonomous vtuber Neuro-sama recently getting some in-game integrations to allow her to play Skyrim on her own? (https://www.youtube.com/watch?v=3o7ORYqLrRw shows the highlights from the first stream.)
Does it have any agency of its own, or is it entirely reliant on you commanding it?
> "Maybe it's because Varkos is a dog, and who doesn't like dogs"
Me
The voice reminds me of Marcus the worm from VRChat
Wow that demo video is way better than expected. Well done
I imagine playing fallout with this and.. please keep at it!
This is excellent, I really want to connect this with fallout community edition.
How is the personality evolution saved? Plain text?
Pretty neat but a talking dog is so weird, especially one that sounds like Stellan Skaarsgard
Would be better if he just barked
Have you seen Mantella yet? https://art-from-the-machine.github.io/Mantella/
tl;dr: let's you talk to all NPCs via LLM. Last time I tried it, latency was too much for it to be enjoyable, your approach seems to be much faster.
awesome idea and implementation. feels like a game feature that you didn't know that you wanted.
I guess this means you can fus ro dah the LLM off a mountain, so there is that.
I expected slop, but this is really good! (both demo and write-up)
hahahaha that's awesome, good job man
Next up: A smart glasses companion that follows you around in the game known as life.
Ive seen something like this in sci fi films.
[dead]
[flagged]
Video game seem like the perfect fit for llm use and I don't understand why it hasn't happened already.
[dead]
[flagged]
I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game.
The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.