alt Hacker NewsAll of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.
What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.
You can find the project here: https://github.com/ory/lumen
Replies
I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)
This sounds quite similar to dirac which made a stir a few months ago:
https://github.com/dirac-run/dirac
I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.
If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.
A bit different but in the same domain GitNexus has been really good for me. Symbol based search and impact analysis. Once I added git hooks to auto index and added some explicit instructions it saved quite a bit for me. I have no affiliation just wanted to add that as some thing to try out.
You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)
When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.
Isn't that what everyone was doing in the beginning of harnesses(cursor) and learned not to do for various reasons? https://news.ycombinator.com/item?id=43164253
skills have a pretty straightforward token win - you remove the decision making from the agent so it does specific things in a row. at some point youre better off inverting control to a script that only occasionally calls the agent for a small decision
How does lumen compare to semble?
Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.
> One of: Claude Code, Cursor, Codex, or OpenCode
What makes it incompatible with Pi, Zed or any other harness?
TBF, what difference does it make when the very harnesses you're using these "hacks" in are themselves vibe-coded?
Lumen is pretty cool, it's just local RAG, but it's a realistic approach at RAG. I would still keep tool-calling in some places though.
How does this compare to ripgrep and fzf?
> Whether it's caveman
the kinda guy who honestly thinks caveman.md reduces costs, actually adds it to his system prompt and is painstakingly reading the terse output
So we replace all those "hacks" and snake-oil with even more pseudoscientific hacks and snake-oil?
[flagged]
I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?
Why should I trust that what you're peddling isn't snakeoil?