OTel isn’t going well

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.

It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.

I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.

I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.

For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.

I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.

OpenTelemtry is the perfect example of an overengineered mess.

While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped.

A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get.

I think OpenTelemetry turned metrics into a farce. In many situations even self-rolled telemetry works better even with the added stuff. The annoying thing is that OpenTelemtry is that big standard now one kind of has to to add compatibility. So please, if you write software, make sure you don't lock yourself into OTel.

I've never found instrumentation to be a huge issue. Sure it takes more effort but you get a lot more value once you understand _business_ events.

I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasant

Sounds a lot like K8s. It's not a framework you use, it's a framework to build a framework on top of.

I wish the observability vendors would move to using it under the covers so it's easier to mix and match.

I wish the otel support wasn't super buggy in most of the frameworks and backends.

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host.

Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today:

1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.

2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.

3. Serverless runtimes pay a heavy penalty for cold starts with OTel.

4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.

5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.

6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?

Its a shame that the various implementations are pretty horrible. Global state, static methods etc etc.

If you get rid of that, and just pass dependencies around, create some appropriate local abstraction around them.. the tooling, be it datadog or honeycomb does a great job making it useful. Can't really say the same for grafana, but ymmv - depending on budget

I think the industry would benefit from some general evangelism for observability. Being able to do distributed tracing was both a "well, duh" and mindblown experience when I first learned about it a decade ago. It made supporting software so much better.

OTel is a fine system for learning observability; it does an okay job of exposing capabilities given how diverse the vendor ecosystem is.

> However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on.

This is just plain wrong, binaries of the collector are shipped which are available to use straight away. You can use the builder if you want to create your own version with a selected set of components but it is no way a hard requirement.

It feels like OTel tried standardizing before the correct design was anywhere close to being settled. It's only time to standardize once there's consensus on all the important points, and what's left is minor details that don't matter for anything other than compatibility.

I disagree. I'm an observability geek, and OTel is... fine.

It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:

1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.

2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).

I know sadly very little about otel, it feels “heavy” in a way I am not used to, I am used to simple systems - configured and composed in a way that makes a larger system.

20 years ago, we were doing (what I think) OTel is doing: with “hit IDs” (half way between a session and a request) that were consistently applied when logging the cause a request being fired; along centralised logging and really good timekeeping. Essentially a unique identifier as a tag that followed the request as it passed through the system.

This was enough to debug basically any problem.

We could even measure the distance between requests of the same “hit” and the total wall-time before it managed to return through the load balancer, so we could track our p99 easily.

Though truthfully we didn't make pretty graphs.

I sometimes wonder what OTel gives me more than this, but I work in games now and lots of these things that work well in webdev do not apply at all to our problems.

OTel is very complicated while yeah for example datadog is just dropin. And Graylog support for OTel makes it a second class citizen in the logs (all attributes are prepended with otel_attributes_ which makes searching difficult).

Using is hard, vendors are hostile, it seems like no-one want it to be a first class citizen...

I've found their django instrumentation to be kinda useless for larger apps.

The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation.

There is no in-between where I can inject the functionality required in a way that is compatible with the application.

The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage.

At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!

Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.

Premature instrumentation is the root of all evil. And the source of a significant part of AWS revenue. It should not cost more to monitor an app then run it.

I have always been turned off to attempt to use OTel by the feeling that it is a little bit too over-engineered a that it might be very bad in term of performance/wasted network traffic when you see the data structure that it is using.

(TFA author is not me.)

I wish otel was never there. It is badly designed abstraction and due to otel the code gets very very messy and bad.

[flagged]

[dead]

It is crazy to me how often people don't grok how to design software well.

1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?

2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?

If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.