Is anyone grouping error logs by trace instead of by message? Looking for opinions on an idea
retained text
Not selling anything and no link here, I’m just looking for opinions before I go further. First question: does Sentry, Datadog or a Grafana alert already cover this well enough that I’m reinventing the wheel? If so, tell me what it does better and I’ll stop. The problem: in my setup, an error shows up as thousands of log lines that are really a handful of distinct bugs. Finding the cause means grouping them, then opening the trace for one occurrence and reading a very large OTLP JSON. The idea: 1. Pull error logs from Loki 2. Group them by a fingerprint of the normalized message (IDs, numbers and quoted values replaced by placeholders), so one bug = one group 3. Take the `trace_id` that OTel puts on the log and fetch the trace from Tempo 4. Condense the OTLP JSON into a short summary: spans in order, failed spans marked, exception messages, only the useful attributes 5. Open one issue per group with count, first/last seen and that trace summary 6. Later: have an agent read the group and trace summary and suggest a fix It’s early and only partly built. Not done yet: avoiding duplicate issues across runs, redacting sensitive data, and the fix agent. Other things I’d like to know: * Do your logs reliably carry `trace_id`/`span_id`? The whole approach depends on it. * Is grouping by normalized message good enough, or would you group by exception type, stack or failing span? * If an agent suggested fixes from a trace summary, what would you need to see before trusting it? Happy to share the repo if anyone is curious.
document fetched 2026-11-20 00:25 · canonical https://reddit.com/r/devops/comments/1xzw8p8/is_anyone_grouping_error_logs_by_trace_instead_of