alt Hacker NewsI like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.
It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.
I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.
I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.
Replies
Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for.
As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several pieces and how they work together (hint: convoluted and horrifically inefficiently) to modify any piece, and inevitably you learn that to get what you want, you need to swap out a major portion of it... but doing that while maintaining the auto-registry nonsense is a gigantic effort. If it's even possible.
It is the new poster-child for "design by committee". It's horrific. Unfortunately it's also usually the best option in large setups. I greatly approve of the high level goal, but omfg
Totally agree. However I am hopeful. We started the first full instrumented project a few years back. It took us a long time to do the whole work including understanding the SDK, mapping the dimensions and getting everything right. Our last project we did the whole thing with agents and they really took away a lot of the pain from the implementation part. We also use Axiom MCP so when we need some trace or event in the logs the agents look for it and if they don’t find it they’ll add it for the next time. It’s really been a different experience.
Ran into the same issue and didn't find any willingness in the OTEL gods to close this gap.
I tried to emit metrics from a python app using otel once. Gave up and switched to prometheus. What a nightmare.
OpenTelemetry reminds me a lot of the bad old days when Java/XML maximalism was fashionable.
Even just basic wire protocol is ass that's PITA to parse, like list of attibutes (which have to be unique) isn't a map but array of maps with some weird way to encode key and type. The whole project is industrial scale mediocrity
The article assumes the issue with OTel is slow feature development, which isn't my experience at all. The issue I've had is that the SDKs have terrible performance overhead for instrumentation and are, as you say, highly resistant to integrating the output of better performing (or just preexisting) instrumentation. In Python and Ruby, at least, the CPU cost of all the mandatory abstraction is way too high.