Charts/Core charts
Traces
<Traces /> · 14 examples
One request, drawn as nested bars across every service it touched.
Overview
A user clicks Pay. Forty milliseconds later a spinner is still spinning, and somewhere between the gateway, the cart service, the fraud check and a database that everyone has agreed not to discuss, the time went missing. A log file can tell you what each service did. It cannot tell you what they did in order, or while waiting for whom.
A distributed trace fixes that by recording each unit of work as a span: a service name, an operation, a start time, a duration, and the id of the span that caused it. Put the spans on a shared clock, indent each under its parent, and you get the picture this chart draws. It descends from Google's Dapper paper, and from Zipkin and Jaeger, and today from OpenTelemetry. The visual idiom, the waterfall of bars with a tree on the left, has barely changed, because it works.
The chart reads like a Gantt chart that has had a bad day. Long bars are slow things. Bars that start after another ends are things that waited. A gap between a parent and its first child is time spent doing nothing in particular, which is the most expensive kind.
Props
API
<Traces> shows one distributed trace: every span of a request as it passed through your services, on a shared timeline. It's a port of Grafana's Traces panel (Grafana's trace view); see Coming from Grafana.
use zopra_charts::*;The buttons use gpui-kit's built-in icons, so register its assets once when you start the app: gpui_kit::application().with_assets(gpui_kit::assets::Assets).
<Traces>TraceSpanRowSpanFilters- What's on screen
- Custom actions and share links
- Enums and types
- Coming from Grafana
Not sure which option fits? See the guide.
<Traces>
| Prop | Type | Default |
|---|---|---|
data | Trace or Signal<Trace> | required |
filters | SpanFilters | none |
focused_span | span ID | none |
hide_header_details | bool | false |
share_link | Fn(&str, &str) -> SharedString: trace ID and span ID to a URL | none: no Share button |
actions | &'static [SpanAction] | none |
bar_label | SpanBarLabel | Duration |
name_column | f32, share of the width | 0.4 (draggable between 0.2 and 0.85) |
class | Tailwind classes | fills its parent |
key | anything Into<ElementId> | – |
filterssets the search the view starts with. After that, people change it in the filter bar.focused_spanexpands the span's parents, opens its details and scrolls to it. Change it to jump to another span.- With
hide_header_details, only the title row stays above the timeline: no banner, details, overview or filter bar.
An empty trace shows "No data found in response".
Trace
| Constructor | Use it for |
|---|---|
Trace::from_spans(rows) | spans you build yourself with SpanRow |
Trace::from_jaeger_json(text) | a Jaeger API response ({"data": [{ "traceID", "spans", "processes" }]}), or a single trace object |
Trace::from_otlp_json(text) | OpenTelemetry's JSON export ({"resourceSpans": …}) |
The JSON constructors return Result<Trace, serde_json::Error>; parts they don't recognise are skipped.
| Method | Returns |
|---|---|
id() | the trace ID |
name() | "service: operation" of the root span |
duration() | µs from the first span's start to the last span's end |
len() / is_empty() | the number of spans |
How spans are arranged.
- Each span sits under its parent. Without a parent, it sits under the span its first reference points to.
- Children are ordered by start time. Spans whose parent isn't in the trace go at the top level.
- Spans with a start time of 0 are dropped.
- A repeated span ID gets
_1,_2, … added. - A tag repeated with the same value is kept once, and the span gets a warning about it.
Keep the trace in a signal (use_signal(Trace::from_spans(…))). A Trace passed directly is arranged again on every render of your component.
SpanRow
SpanRow::new("b7ad6b71", "checkout", "PlaceOrder")
.parent("53995c3f")
.trace_id("4bf92f3577b34da6")
.start(1_729_000_000_000_000)
.duration(812_000)
.kind(SpanKind::Server)
.tag("http.request.method", "POST")
.tag("http.response.status_code", 500)
.service_tag("k8s.pod.name", "checkout-7d9f")
.log(SpanLog::new(1_729_000_000_300_000).name("retry").field("attempt", 1))
.status(SpanStatus::Error)| Method | Argument |
|---|---|
new(span_id, service, operation) | text |
parent | parent span ID |
trace_id | text |
service_namespace | text; shown as namespace/service |
start | µs since the Unix epoch (u64) |
duration | µs (u64) |
tag(key, value) | span attribute |
service_tag(key, value) | resource attribute |
log(SpanLog) | an event: SpanLog::new(µs).name(…).field(key, value) |
reference(SpanReference) | a link to another span: SpanReference::new(trace_id, span_id).tag(…) |
kind | SpanKind |
status / status_message | SpanStatus / text |
library(name, version) | instrumentation library |
trace_state | W3C trace state |
warning / stack_trace | text; repeatable |
error_color | colour of this span's error icon |
Tag values can be text, whole numbers, decimals or bool. A span counts as an error when its status is Error or it has the tag error = true.
SpanFilters
SpanFilters::new()
.service(FilterOp::Eq, "checkout")
.duration(FilterOp::Ge, "100ms")
.tag("http.response.status_code", FilterOp::Eq, "500")
.text("timeout")
.matches_only(true)| Method | Matches spans where |
|---|---|
service(op, value) | the service name =, !=, contains (=~) or doesn't contain (!~) the value |
span_name(op, value) | the same, for the operation name |
duration(op, value) | the duration is =, >=, <=, > or < the value: "250" (µs), "500ns", "120us", "120µs", "80ms", "2s", "1m", "1h" |
tag(key, op, value) | a span, resource or event attribute matches. Also understands kind, status, status.message, library.name, library.version, trace.state and id |
text(query) | any word of the query appears in the span's names, attributes, events, kind, status or library (ignoring case), or a word is the span ID |
critical_path_only(bool) | the span is on the critical path |
matches_only(bool) | – : hides every span that doesn't match, instead of tinting the ones that do |
All filters must match. With Contains / NotContains, a tag filter matches keys and values that contain the text.
What's on screen
Header. The root service and operation, the HTTP method and status, and a ⋮ menu with Copy link (when you pass share_link) and Export as JSON. Below that:
- A banner for the most telling error span (deepest, then longest), or a 4xx span, with its share of the trace and a Go to span button.
- Trace ID (click to copy), start time, duration, number of services and spans, and the route.
- Overview: a minimap of the whole trace. Drag across it to zoom the timeline, drag the edges to adjust the zoom, and use Reset selection to zoom out.
- Filter bar:
- Pick a key, an operator and a value, then press Add or Enter.
- The chips show active filters.
- The Critical path, Errors and High latency (slowest 10%) pills toggle common filters.
- The arrows step through matches, showing "i/N matches".
- Show all spans switches matches-only on and off.
Timeline.
- The name column shows a span's service only where it changes from the row above.
- The ⌄ / › chevron expands and collapses a span. The header buttons, or the
op[]keys after clicking the timeline, expand one level, collapse one level, expand everything and collapse everything. - Indent guides highlight every row that shares the ancestor under the pointer.
- A collapsed client span shows the server it called. A client span with a
peer.servicetag shows the peer. - Bars use one colour per service. Thin dark segments mark the critical path, and ticks mark events; hover a tick to read them.
- Hover a bar for
service::operation | duration. - Drag across the timeline header to zoom. Drag the column divider to resize the name column.
- Rows are tinted for errors, search matches and the focused span.
Span details (click a row).
- Service, duration, start time, child count, kind, status, library and trace state.
- An exception box when the span recorded one.
- Span and resource attributes grouped by category (HTTP, Database, Kubernetes, Cloud, …), with a copy button on every value.
- Events, warnings, stack traces and references (click one to go to that span).
- Your actions and Share, and the span ID (click to copy).
Custom actions and share links
fn open_logs(span: &SelectedSpan<'_>, window: &mut Window, cx: &mut App) {
println!("logs for {} in {}", span.span_id, span.service);
}
const ACTIONS: &[SpanAction] = &[SpanAction { label: "Logs", run: open_logs }];
<Traces
data={trace}
actions={ACTIONS}
share_link={|trace: &str, span: &str| format!("https://traces.example.com/{trace}?span={span}").into()}
/>SelectedSpan gives trace_id, span_id, service, operation, start and duration. The Share button copies the link for the open span; Copy link in the ⋮ menu copies it with an empty span ID.
Enums and types
| Type | Values |
|---|---|
SpanKind | Internal, Server, Client, Producer, Consumer |
SpanStatus | Unset (default), Ok, Error |
SpanBarLabel | Duration (default), None, Tag(&str): the label after each name |
FilterOp | Eq, NotEq, Contains, NotContains, Gt, Ge, Lt, Le |
FilterKey | Service, SpanName, Duration, Text, Tag(key) |
TagValue | Text, Int, Float, Bool; made from &str, String, numbers and bool |
SpanAction | { label: &'static str, run: fn(&SelectedSpan, &mut Window, &mut App) } |
Durations are shown the way Grafana shows them: 850μs, 1.23ms, 1.5s, 1h 5m.
Coming from Grafana
| Grafana | Here |
|---|---|
data frame with traceID, spanID, parentSpanID, operationName, serviceName, startTime, duration, tags, logs… | SpanRow, or Trace::from_jaeger_json / from_otlp_json |
| Span filters, Show matches only, Select critical path | filters |
focusedSpanId | focused_span |
hideHeaderDetails | hide_header_details |
createFocusSpanLink | share_link |
createSpanLink (logs, metrics, profiles links) | actions |
| Span bar label setting | bar_label |
Not available here, because they need a Grafana server: data source links to logs, metrics and profiles, the Pyroscope span flame graph, plugin extensions, and the adaptive traces banner. The span colours come from your theme's palette rather than Grafana's.
Run them
14 examples
Press Run to load an example into the live render at the top of the page. Open one to read its code.
How to read it
- 01Read the header: the root operation, total duration, the number of services and spans, and the banner pointing at the most telling error.
- 02Scan the bars top to bottom. Indentation shows who called whom, and bar position shows when. A staircase of bars is a chain of sequential calls. Bars stacked at the same time are parallel.
- 03Turn on the Critical path pill. Whatever stays highlighted is what the request was actually waiting for.
- 04Click a span to open its details: tags, process attributes, logs, warnings and stack trace. Drag across the overview to zoom, and use the filter bar (service, duration, tags, text) to hide what you do not care about.
When to use it, and when not to
Reach for it when
- You are chasing latency in a system with more than two services and need to know where the time went in one particular request.
- You have a trace export from Jaeger or OpenTelemetry and want to look at it without standing up a backend.
Trace::from_jaeger_jsonandfrom_otlp_jsonread both. - You want to separate work that is slow from work that is merely waiting, using the critical path filter.
- You are debugging errors and want to see which span failed first, with its tags, logs and stack trace.
Skip it when
- You want to know how latency is distributed across thousands of requests. One trace is one anecdote. Use a Histogram or a Heatmap.
- You want to see which services call which, regardless of any one request. That is a Node Graph.
- You want to see where CPU time goes inside one process. That is a Flame Graph, built from samples rather than spans.
- The trace has tens of thousands of spans and you are hoping to see all of them at once. Filter first, then read.
The math behind it
Spans, parents and the tree
Each span has a start in microseconds and a duration , so it covers . Spans with a start of 0 are dropped. A span's parent is the span named by its parent field; if there is none, the span under its first reference is used (a "follows from" link). Spans whose parent is not in the trace are placed at the top level.
Children are ordered by start time and the tree is walked depth first, which gives the row order and the indentation depth. The trace as a whole runs from the earliest start to the latest end.
From time to pixels
The timeline column is a linear scale. If the visible window is (the whole trace until you drag across the overview to zoom) and the bar area is pixels wide, a span is drawn at the position and width below. The label on the bar is its duration by default, or its percentage of the trace, or none.
Critical path
The critical path answers the question "if I made only this one thing faster, would the request be faster?" It is found by walking backwards from the end of the root span. At each span, take the child that finishes last before the current point. The time between that child's end and the current point belongs to the parent, so it is added as a section of the parent's critical path. Then continue into that child. When a span has no such child left, its own remaining time from its start is added and the walk returns to its parent, resuming from where the child began.
Two details keep this honest. A child is first clipped to its parent's interval, and a child that lies entirely outside it is ignored. Spans linked only by a reference (not a parent) are never on the path. The result is a set of time sections, not whole spans: a span can be partly critical, and the chart shades exactly those parts.
Real life usage
The 900 milliseconds that were 40
Ines was the on-call engineer for an online ticketing site, a job that mostly consisted of being awake on the day a band announced a tour. The checkout endpoint had a p95 of 1.4 seconds. The product team wanted it under 600 ms by the next on-sale, which was eleven days away and, in the way of these things, already sold out in people's imaginations.
Everyone had a theory. The database was slow. The payment provider was slow. The seat-hold service was written in a hurry and, said one colleague, had it coming. Each theory had a graph to support it, and the graphs were all from different services and different hours.
What Ines needed was not another graph. She needed one slow request, taken apart, so she could see what the checkout was actually waiting for and for how long.
She exported a trace of a 1.3 second checkout from her tracing backend, loaded it into a small desktop tool with Trace::from_otlp_json, and put it on screen with <Traces>. The tree had 61 spans across seven services. The database spans, the prime suspect, were short thin slivers.
She switched on the critical path. Most of the bars dimmed. What remained lit was a long block in the seat-hold service, and inside it a run of four calls to the pricing service, one after another, each about 180 ms. They had no data dependency. Somebody had written them in a loop.
The fix was to send the four requests together. The checkout dropped from 1.3 seconds to 480 ms in the test environment, p95 landed at 540 ms after release, and the loop's author, to their credit, framed the trace and hung it above their desk.
The database was never mentioned in the post-mortem again. It was, in the end, merely very quiet.
Pitfalls
- Clock skew. Spans from different machines are placed by their own clocks. If a child appears to start before its parent, the machines disagree. The chart draws what it was told.
- Missing parents. A span whose parent was never recorded shows up at the top level, not under the right caller. That is a sampling or instrumentation gap, not a chart bug.
- One trace is not the system. A trace you picked because it was slow tells you why that request was slow. Check that the pattern repeats before changing the architecture.
- Critical path is about time, not cost. A short span on the path that is called ten thousand times a day may matter more than a long one that happens once.
- Zero start times. Spans with a start of 0 are dropped, so an exporter that leaves the field empty will produce an empty chart.
From around the web