alt Hacker NewsHey, codebase-memory-mcp and semble are not exactly the same, but it's an interesting comparison, I'll put it on the todolist to check that out and add it to our benchmarks if feasible. If you ever get a chance to do this same comparison with semble it would be super useful feedback since these "real" scenarios are hard to benchmark/replicate.
So, I just tested with semble. Your MCP integration did not work, and kept throwing error (Failed to connect to "semble": MCP error -32000: Connection closed) though I installed using documented manner (tried both: pip and ux methods).
Anyways, I made it work by making it generate relevant doc (using semble init), and then copying this into AGENTS.md, and then prompting it with this line:
""" Start by reading AGENTS.md in current folder. Now, the task::: `Explore the ingestion and search paths. Do not read README.md at all`. Prefer to use `semble` search for code search. Do not do new installation. semble is already available at `/Users/nitinbansal/.local/bin/semble` . """
The results are much better. Even better than my own setup, but, vary a lot. I did 4 runs:
95k/2.9k
25k/2.7k
71k/2.9k
37k/4.0k