logoalt Hacker News

thorian1828i03today at 2:50 AM2 repliesview on HN

One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...


Replies

EdSchoutentoday at 2:54 AM

If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load.

What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.

The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).

show 3 replies
PunchyHamstertoday at 3:15 PM

It is very easy way to have your tracing infrastructure cost more than actual infrastructure.