alt Hacker NewsIn case it's not obvious, this isn't a model, this is python scripts that call out to models (code not available).
It's mostly "did you know you can attach an LLM to a PCG system?", but there is one idea here you don't see much of: an image model performs the composition (which image models are really good at), and then you extract the objects into 3d via things like SAM3D before placing them in the world, which is pretty interesting.
The rest is standard stuff you'll find in your favorite PCG system/game engine. Still, think most people don't realize how good LLMs are at 3D these days, especially Fable 5 (which this project predates).
Replies
That’s exactly how I’ve used tools like Tripo and Meshy.
Use any image generation tool (or any image) and have a multimodal LLM like Gemini extract individual parts of the image “isolate with a transparent background” - then you can use those images for image-to-3d in those tools.
Don’t forget to use low or “smart” poly features otherwise you get too many vertices to the point you can’t performantly raycast etc.
But yeah - it’s there.
> Still, think most people don't realize how good LLMs are at 3D these days, especially Fable 5
Can you explain more? Can I use Fable 5 to generate a 3d model from a couple of photographs?
LLMs are a PCG system. This is one of my pet peeves, I've been working on a procedural generation constraint system for quite a while (way before LLMs) as a hobby, that came with a domain specific language, where you could specify constraints.
Things like the red key must be reachable before the red door, and there should be no path that bypasses the red door, etc. written as a set of constraints.
I was working on is because I realized, constraints tend to be similar in nature (graph cut, minimum/maximum shortest path etc.) algorithmically, but its usually quite hard to control designer intent.
The reason I shelved the project, was I asked an LLM a couple years ago, to generate a dungeon room graph, following a similar set of constraints, but stated in plain English, and it did a perfect job.
Which brings me to my final point. LLMs are amazing and flexible constraint solvers, and are woefully underutilized in scenarios like this.