alt Hacker NewsLogs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging.
All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0".
The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.
Replies
> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)
A metric is not event based.
You don't have a metric 'person logged in' because you would need to scrape the metric at the moment a person logged in.
You have a metric called 'overall people have logged in so far' and you do math on it.
The 'person logged in' is an event you log out.
This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction.
Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance.
In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure.
"Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems.
Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.