logoalt Hacker News

freakynittoday at 3:56 AM1 replyview on HN

What I have personally observed with such tools is that they make the AI's dumb, similar to how it makes coders dumb when relying more on AI tools.

These agentic AI's are already smart enough to figure out a highly optimized path to code exploration or search. But, with these tools, they just go very aggressive, partly because the search results from these tools almost in 100% of the cases do not furnish full details, but, just the pointers.

To confirm this behaviour, I did a small test run. This is in no way conclusive, but, the results do align with what I been observing:

---

Task: trace full ingestion and search paths in some okayish complex project. Harness is Pi.

1. With "codebase-memory-mcp": 85k/4.4k (input/output tokens).

2. With my own regular setup: 67k/3.2k.

3. Without any of these: 80k/3.2k.

As we see, such a tool made it worse (not by much, but, still). The outputs were same in quality and informational content.

---

Now, what my "regular setup" mentioned above is?:

Just one line in AGENTS.md and CLAUDE.md: "Start by reading PROJECT.md" .

And PROJECT.md contains just following: 2-3 line description of the project, all relevant files and their one-line description, any nuiances, and finally, ends with this line:

    ## To LLM
    Update this file if the changes you have done are worth updating here. The intent of this file is to give you a rough idea of the project, from where you can explore further, if needed.

Replies

Bibabomastoday at 4:44 AM

Hey, codebase-memory-mcp and semble are not exactly the same, but it's an interesting comparison, I'll put it on the todolist to check that out and add it to our benchmarks if feasible. If you ever get a chance to do this same comparison with semble it would be super useful feedback since these "real" scenarios are hard to benchmark/replicate.

show 1 reply