logoalt Hacker News

jandrewrogersyesterday at 3:31 PM1 replyview on HN

This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction.

Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance.

In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure.

"Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems.

Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.


Replies

skrtskrtyesterday at 4:14 PM

You are conflating the challenges of ingesting and querying at large scale with the what the original comment is about, which is emitting them more easily.

show 1 reply