alt Hacker NewsIf you don't run your own LLM proxy, how do you know whats going on?
Comments
I'd love to do something like this, or use API keys, but I still don't understand how people justify the cost of using API keys vs. subscription.
I tested it once, I'm in on the 20x claude max ($200/month) and I either max my weekly limit every week or come really close. But it only took 2-3 days to hit $200 spend with api keys.
Proxies are good if you pay API prices.
For subscriptions, it is easier to collect the JSONL files from ~/.claude/projects/ and similar directories. They are better organized (e.g., subagents) than what the proxy sees.
However, the best value comes from collecting agent sessions across the whole organization. You'll need to scrub credentials and handle security.
Tailscale makes Aperture available for free, and while it's not something you own, it is in your tailnet, so at least you have a certain amount of control over how things access it. It gives a nice view of what agents are doing, with lots of configurability and extensions, and the team behind it has been updating pretty steadily.
I happen to be working on an extension to Aperture, so I have my own reason to promote it. But I do think it's a nice easy solution, and I promoted it to people before I was extending it.
One man’s fun is another man’s “idk what any of these words mean, I pay $20/mo for some company to abstract this complexity away”
For clarity, I do like blog posts like these.
I’m not sure what people are doing with these subscriptions to feel the need to monitor what’s going on this closely. A $100 OpenAI subscription feels effectively infinite for my usage at home
Yup. Setting up liteLLMProxy on some servers as we speak and will start playing around.
I can't orchestrate agents in sandboxes without it.
1. It can issue virtual keys, so provider API keys never go near the sandbox
2. I can see everything
3. Guardrails
And much more
Looking forward to it.
Yeah you should run an LLM proxy, but imagine a world where LiteLLM and all the other LLM shovels weren't buggy vibecoded messes. And with the velocity of updates I feel there has to be vulnerabilities there. Or am I just holding it wrong? Anyone got a good alternative?
Posting in case it is useful to others - I've built a OSS Mac App that handles this across various providers and lets you examine usage in exacting detail.
The App: https://github.com/Pythia-Software/pharos Blog Post: https://grady.dev/projects/pharos/
OpenRouter does all this for you. It's pretty nice. Although for apps I use another server to lock down the requests even more and prevent the client from seeing my API key.
Want to shamelessly self promote my self hosted proxy, supports subs and API usage.
Primaily designed as a local LLM ops project, but will definitely do proxying for claude-code, opencode, codex, etc
We've tackled the core problem of agent telemetry normalization with our open-source project Beacon! https://github.com/Asymptote-Labs/agent-beacon
Beacon is a harness-agnostic, environment-agnostic framework for capturing, normalizing, and storing the full session history of AI agents. We extend the OpenTelemetry GenAI standard and have done the heavy lifting of integrating with 25+ agent harnesses through a combination of OTel instrumentation, hooks, and local session database capture.
This gives you a unified view of agent activity across environments, including local agents, cloud agents, browser-based chat, and agent SDKs, with the ability to track and retain complete session histories across all of them.
Would love your feedback!
Toyed around with it back when claude code showed up, but never really got any useful actionable insights from it so stopped
(Aside from ditching CC & it's bloated sys prompts that is)
I don't do this but I do have a system that imports all my conversations with AI (claude code, chatgpt, claude app, etc.) to a storage system I call Chatacombs: a unified storage of most conversations I've had with AI, as detailed as I can easily get.
My org is using https://visiqlabs.com/ for this. not a proxy, agent integration with rules engine
Until they quit subsidizing my personal tokens I don't think I want this. I do love having insight when I run qwen locally through Claude Code Router so I do agree it's so valuable to have, just not valuable enough to pay API rates lol
I like the content but I have to say the Mark Manson writing style is off putting.
I agree its so important to proxy and capture all the data. I've been very happy with bifrost as a proxy. It's a quality piece of software, unlike LiteLLM. You also get this one interface to access all your models, across all services, including self-hosted, STT models, everything. There are ways to use subscriptions with proxies too, but they are discouraged because of grey-hat uses..
https://github.com/gumieri/nenya
my own ai gateway/proxy
This is the best one I found https://github.com/kenn-io/agentsview
It is a not proxy. It can clearly show whats going on across all agents. The proxy's are generally inconvenient and yet another point of failure. I think the author got to the resort option quickly
I do know checking both agent session and logging at the inference side...
As long the harness is not open source a proxy won't help, it's easy to phone home on a different subdomain also.
Well that was one of the reasons why i build my own agent/harness - to have actual full observability of what it is doing and why it is doing what it is doing.
Don't really understand why other agents to give those insights but well found my solution.
Just one info for many: if you’re a big company and have enterprise needs, subscriptions aren’t a solution. OpenAI, Anthropic, Google (ain’t sure about them tbh), GitHub Copilot aren’t offering subsidized subscription prices in their enterprise plans; discounts are calculated differently and these services are much more expensive.
For example, Github Copilot has these AI credits that are insane in terms of costs with the usual token-based pricing.
Feel free to prove me wrong, I really wish I was.
Seeing some good stuff in the comments, but how are people doing this?
Some don't need to. Not yet anyway. There's config, there's control, and then there's setup-theatre as a hobby.
what proxy is he using? i support not trying to support a specific product in a post but not even mentioning what you use after writing this seems pretty odd.
It's LLM, of course they don't know what's going on.
By running my own agent harness that provides detailed API metrics in the GUI and logs very verbosely.
Umm, observability ? I think the article means to say how do you know the agents arent talking nonsense to each other if you use a multi-agent system. Here again a thin layer of o11y would do the trick and also hooking into the wider ecosystem. Thats without having to bother with a proxy.
Agents can log traces, which can be super useful for RL, prompt optimization, tool improvements, etc.
Check out Calibra for example: https://calibra.swival.dev which turns traces from Swival into nice web pages, allows diff'ing traces, etc.
What is the use case for these sorts of things?
Anyone know an easy, out of the box method of doing this inspection on DGXOS (the OS for DGX Spark devices)?
I'm a little miffed that this isn't just an immediate OOBE, but I'm prepared to do some nc/ssh tunnel magic if that'll work properly .. just wondering what other DGX users have vis a vis best practices ..
> how do you know what's going on
there's the kind of people who care and the kind who doesn't ...
your local politicians and landlords do all kinds of normal, non-funky stuff you don't know about ... as do your local ISP-techies and hobbyists with access to supply chains and sub-distributing units ...
or your local school teachers and some parents and their widdle vintage schemes and circumambient tales they tell some kids but not others ... Lord, some guy even caught teachers and parents running schemes on teens to prep them for gay homo fag stuff, despair, frustration ... running for years ...
and you wonder how peeps don't what is going on xD?<
even those hackies and hackers and programmers and brogrammers and AI superusers who think they know and care and wield some power really really don't
but there you go, AI augmented diluted delusions
Does anyone here use LiteLLM for this? How well does it do? Can I use it with my Claude subscription?
All that posturing for the purpose to be manually monitoring cache hits and misses. This can't be serious. Buy a larger plan or reduce some instructions and you don't have to deal with this waste of time.
> Own your infrastructure
...proceeds to use codex.
With just a 12gb VRAM GPU + 64gb of CPU RAM, and you can actually own your own infrastructure. I can't wait for hardware prices to fall so that becomes even more accessible.
vibes
I am boggled how the first step of an llm after you make an orchestrator isn't some kind of proxy that records your personal internet surfing to be used by your personal AI later. Every college should be running in house AI with their own in house search engines and in house any service. Don't go to any college that wants you to use chatgpt or other corporate nonsense. If they can't make local LLMs work, they are not a real college.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
You don't need a proxy to do this. I would argue you should not use a proxy to do this because the end result is a single point of failure direct in the line of your productivity.
Both Codex[0] and Claude Code[1] output OTEL traces that you can ingest and process.
Other harnesses like OpenCode, Pi, etc. you can hook into their plugin architecture and implement your own OTEL emission from a plugin (or use an off the shelf plugin[2]).
Host an OpenTelemetry collector[3][4] or rely on your existing collector infra (e.g. Datadog (no affiliation), AppInsights, etc.)
The OTEL traces will have exactly what you need. Tool calls, full prompts, token usage/cost, etc. It is less intrusive and does not become a single-point-of failure since the OTEL emission and collection is best-effort whereas a failing/overloaded proxy is going to strand your team until resolved. It's OK if you lose some OTEL traces; no productivity lost.
The configuration to enable this for Codex and CC is very straightforward (see [0][1] + [5])
--
[0] https://learn.chatgpt.com/docs/config-file/config-advanced#o...
[1] https://code.claude.com/docs/en/monitoring-usage
[2] https://github.com/DEVtheOPS/opencode-plugin-otel
[3] Open source collector: https://opentelemetry.io/docs/collector/
[4] Sample configuration for filtering noise: https://github.com/zeeq-ai/zeeq-app/blob/main/build/otel-col...
[5] Concrete examples of .codex/config.toml and .claude/settings.json: https://zeeq.ai/docs/key-features/session-telemetry#configur...