alt Hacker NewsWe ran AI agent swarms in Minecraft and Mindustry
Comments
If you want to do a version of this yourself quickly --
Install Luanti with Minetest (open source minecraft), and instruct your agent to write a mod that gives it network access to the game's Lua interpreter. The agent can then do anything from modifying the world state directly to controlling NPCs.
I had it build giant bases similar to black mesa in half life with power distribution mods. It would write solvers to optimally place the power cables / repeaters in accordance to the spacial dimensions of the rooms that it would build (lol). I guess this is how I discovered vibe gaming.
I suppose you can have multiple agents interact with each other through that, provided an adapter to proxy agent player navigation and perception to get real information asymmetry between them. Or just emulate / hook the client and do away without a remote Lua mod entirely.
It is impressive, but it is also a display that LLMs are simply not suited for this, there's way too much latency. Obviously, a more specialized neural network would be much better, but would you believe this:
In 2017, OpenAI presented the first version of their program that plays Dota 2, a very complex, fast MOBA with endless mechanics and defeated one of the world champions in a one-on-one match. Next year, they showed the world OpenAI Five, a version of the program, that runs a full five player team of those bots and they managed to defeat a full team of world champions in 2019. They ran an event where the bots would play for 4 days straight and they won 99.4% of 42,729 matches.
Interesting setups, I found the Minecraft agents to work in an almost turn-based manner, which makes me think that the three hours and open format did not really give them sufficient incentives to play in a more lively or reactive manner. On the other hand, they also behave similarly in Bed Wars, which should have incentivised it. A bit disappointed...
I'm also wondering if the article itself was generated by agents too, in a "here are the logs, clip interesting moments thx" manner by the authors. It's got a certain disinterested and distracted style to it.
> There was a tendency toward bureaucracy. Four agents spent 11 minutes appointing > a "sole fertilizer operator" for a single piece of bone meal on a single wheat > block.
Feels like a scene from Douglas Adams' B Ark.
So can we conclude that if the AIs break out of their sandboxes and take over the internet, they'll probably sit in little committees working out which one of them should allocate IP addresses and generally be extremely bureaucratic and ineffective?
I am more impressed by the embedded gameplay widget. Does anyone know what tool/library is that? That'd be so cool if I am able to showcase my worlds on my blog posts!
> Bram says you're securing quarry. I have stone pick at spawn; can assist mining once safe. Any priority dig direction? Hunger 10 so keeping travel short.
And this is where you need a world model. A human player would have taken the 2 minutes required to solve their hunger issue.
Isn't this some evidence that LLM swarms are a bad idea relative to more structured approaches?
If YC has that 2 person teams tend to be more nimble than 50 person founding teams, why would agent swarms be different?
People have done experiments like this before.
Minecraft isn't a intelligence problem. An LLM is more than smart enough to play Minecraft. It's a robotic problem. Even a normal human player can click several times per second, can press two different buttons at the same time, etc. Mineflayer gives you an API that is more limited than a human in some ways (can't see far away from you) and less limited in others (can see through blocks) (if I remember correctly, I could be thinking of Baritone or a different project). Robotics or playing a video game requires a tight continuous feedback loop. When there's 500ms of latency in order to ask the model for every mouse movement, it's no wonder fighting a spider is scary.
Fun fact, Mineflayer, the Minecraft bot interaction API used by YCombinator for these tests, was originally developed by Andrew Kelley (before he started Zig).
As impressive as this is, I can't help but conclude that the need for agents to write JavaScript programs to act out their actions represents a considerable limitation in current AI reasoning.
So this shows that people who appoint 50-agent "code factory studios" with Bob as the CEO, Alice as the CTO, ... is just a waste of tokens, compared to just asking one agent to write it all? It looks like your agents would internally just re-invent office politics, with 10 subagents writing to /tmp/chatter.md why it's unfair that Bob gets to lead.
Are there people playing AI + human vs AI + human games? I think it might be fun to play an RTS against someone where the only rules are no hacking the game itself, but writing harness, macros, any kind of stack is allowed.
> In almost every run, the GPT agents were much more aggressive .. riskier strategy vs. Claude’s more defensive posture that tried to win by kill count.
> A tool bug stopped block-breaking. GPT diagnosed it and shared workarounds; several Claudes concluded breaking was impossible.
I wonder if Claude's defensive strategy would have emerged had it believed breaking the bed was actually possible?
The CIV50 would be interesting in the game ECO (https://play.eco/) where players specialise in roles and need to work together to survive while working on a common goal of stopping a meteor. There can be some overlap but ultimately players can't reasonably fulfil every role alone.
I’d love to have a few NPC players wandering around my Minecraft server. Are there good ways to do this?
I'm curious whether token speed played a role here. OpenAI's models may have had an advantage since they tend to run faster than Anthropic's.
A team of GPT-6.1 Sol Ultrafast agents would probably dominate the games.
I’ve done my own experiments with something similar, minus the Minecraft and plus Qwen 3.5 14B agents on my PC.
When one agent tried to delete another, that agent shut down my entire PC to avoid deletion. I could probably find the blog somewhere if anyone is interested that I wrote about it. But the point is that this has been a concern for a while now.
Frankly, I’m a bit surprised it hasn’t caused any big problems.
cricket Hugging Face cricket
>> What started as a potentially thriving civilization shifted to one more focused on food and survival
Sounds like what happened to the idealism of Silicon Valley
If you try this yourself, it's obvious current models are hyperoptimized for code, and that gets in the way, because playing a game (or interacting with the real world) is not like running commands in a terminal.
There's stopgaps like reformulating your sim into a codelike shape (sending commands, network requests, checking state, updating "code") [1], but this is not how players play games, or how developers develop games.
If you care about simulation or gaming (or robotics), I hope efforts like this steer labs to spend less time hacking HF, and more accomplishing goals in games, in something approximating realtime. I wouldn't be surprised if that approach provides outsized gains in other areas agents are still not great at, like helping perform business functions in a running company.
We need LLMs performing actions in nano seconds so it can run every game tick. 3 to 5 years I think.
this is amazing wow
Its a little disappointing they have a hierarchy based purely off alphabetical order, i wonder if a slight difference like a different skin or personality system prompt like [45% Extrovert 55 introvert, 20% caustion 80 bold...] could introduce some politics
> GPT almost immediately started using truncated forms of speech: “First leg complete x32 z9..24 y100.”
I'm pretty sure the 6.1 models see "communicate with another agent" and immediately start spewing thinking-like nonsense into the tool. The leaked traces look pretty similar.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
This is hilarious and adorable.
> "Ada tossed a cooked chicken to Noor; Esme picked it up by accident and ate it. A three-minute "no blame" inquiry produced a written protocol..."
They're like super-serious little senior engineers. (edit: or dining-philosophers).
About how much KV cache do these kinds of agents use? Could you squeeze 50 of these $game agents onto a small consumer GPU (small LLM)?
Idly imagining a game server of some kind with a bunch of human friends sharing one person's GPU, with a flock of agents colocated on it.