Flame Graph

<FlameGraph /> · 16 examples

A profile drawn as stacked boxes: width is time spent, height is call depth, and the wide ones are your problem.

Overview

A profiler produces a lot of stack traces. Thousands per second, each one a list of functions that were running at that instant, and a text file of them is roughly as readable as the phone book of a city you have never visited. The question you actually have is short: where is the time going?

Brendan Gregg invented the flame graph around 2011 while chasing a performance problem at Joyent, and the idea is almost embarrassingly simple. Merge the stacks that share a prefix. Draw the root at the bottom, each callee on top of its caller, and make every box as wide as the number of samples it appears in. Hot code becomes wide. Deep code becomes tall. Code that does nothing in particular becomes invisible, which is the best thing that can happen to code.

The name comes from the shape, a skyline with flickering tops, and does not mean what you hope it means. A wide flame is not a fire you have to put out. It is somewhere you could look. Whether it is a bug or just the part of the program that does the work is a judgement the chart cannot make for you, and the most common mistake is to optimise the widest box in the graph, which is usually called main.

Props

API

<FlameGraph> shows a CPU, memory or time profile three ways: as a flame graph, as a top table of the heaviest functions, and as an expandable call tree. It can compare two profiles too. It's a port of Grafana's Flame Graph; see Coming from Grafana.

use zopra_charts::*;

The buttons use gpui-kit's built-in icons, so register its assets once when you start the app: gpui_kit::application().with_assets(gpui_kit::assets::Assets).

<FlameGraph>
Profile
Panes and layout
Interacting with the graph
Colours
Values and units
Custom menu actions
Enums and types
Coming from Grafana

Not sure which option fits? See the guide.

`<FlameGraph>`

Prop	Type	Default
`data`	`Profile` or `Signal<Profile>`	required
`layout`	`PaneLayout`	top table and flame graph side by side
`header`	`bool`	`true`
`unit`	`SampleUnit`	`Count`
`format`	`Fn(f64) -> SharedString`	none
`decimals`	`usize`	chosen by the unit
`color_scheme`	`FlameColors`	`ByPackage`, or `Diff` for a diff profile
`text_align`	`LabelAlign`	`Left`
`collapsing`	`bool`	`true`
`vertical`	`bool`	`false`
`keep_focus_on_data_change`	`bool`	`false`
`actions`	`&'static [FrameAction]`	none
`class`	Tailwind classes	fills its parent
`key`	anything `Into<ElementId>`	–

layout, color_scheme and text_align set the starting state. After that, the header and toolbar controls change them.

`Profile`

A profile is a call tree: each frame (a function in a call stack) has a total value, which includes everything it called, and a self value, which is only its own work.

Constructor	Use it for
`Profile::from_folded(text)`	Folded stacks: one line per stack, frames separated by `;`, then a count, e.g. `main;parse;lex 42`. This is what `perf` + `stackcollapse`, `cargo flamegraph` / `inferno`, `py-spy --format raw` and `async-profiler -o collapsed` produce. A `total` root frame is added on top.
`Profile::from_nested_set(rows)`	`(level, label, value, self)` rows in depth-first order, Grafana's profile format. Level 0 is the root.
`Profile::diff(&baseline, &comparison)`	Two profiles merged into a diff: matching stacks line up, and each frame keeps both values.
`Profile::diff_from_nested_set(rows)`	`(level, label, value, self, value_right, self_right)` rows that are already a diff.

Method	Returns
`is_diff()`	whether the profile compares two profiles
`len()` / `is_empty()`	the number of frames

Labels are stored once per distinct name, however many frames use them.

Panes and layout

Pane	Shows
`Pane::FlameGraph`	the flame graph
`Pane::TopTable`	one row per function: Self and Total (or Baseline, Comparison and Diff), sortable by any column
`Pane::CallTree`	the call tree as an expandable table, with its own search

`PaneLayout`	Meaning
`PaneLayout::Single(pane)`	one pane
`PaneLayout::Split(left, right)`	two panes with a draggable divider; stacked top and bottom when `vertical` is set

A split layout shows a single pane when the chart is narrower than 800px (unless vertical is set). With header={false} the search box and pane pickers are hidden, so whatever layout you pass is fixed.

Interacting with the graph

Action	Result
hover a frame	tooltip with the total, self and sample count, as percentages too; Baseline / Comparison / Diff for a diff profile
click a frame	menu: Focus block, Copy function name, Sandwich view, your actions, and group expand / collapse
Focus block	zooms to that frame; frames above it are drawn lighter. A pill shows its share and clears the focus.
Sandwich view	shows that function's callers above and its callees below, merged across every place it was called
search box	dims every frame that doesn't match and filters the top table. Separate several terms with commas. Each term is tried as a regular expression first, then as a fuzzy match.
top table search / sandwich buttons	search for exactly that function / open its sandwich view
call tree up / down arrow buttons	show that function's callers / callees as the tree
call tree search	finds functions (case-insensitive; regular expressions allowed), shows "1 of N", and the arrows step through matches, expanding and scrolling to each
Reset	clears search, focus and sandwich

Collapsing. When a frame is the only child of its parent and holds more than 99% of the parent's value, the two are grouped onto one row and labelled (N) name. A strip on the left marks groups. Expand or collapse them from the frame menu, or all at once with the toolbar buttons. Set collapsing={false} to give every frame its own row.

Changing data. When data changes, focus, sandwich and grouping reset, unless you set keep_focus_on_data_change.

Colours

`FlameColors`	Colour of each frame
`ByPackage`	by package or module name, so related functions share a colour
`ByValue`	yellow for small frames to red for large ones, relative to the current zoom
`Diff`	for diff profiles: your theme's good colour where a frame shrank, its neutral colour where it stayed the same, and its bad colour where it grew
`DiffColorBlind`	the same, in blue, grey and red

A diff profile shows a legend from −100% (removed) to +100% (added). Frames too narrow for a label are drawn grey. Set the good, neutral and bad colours in your theme.

Values and units

`SampleUnit`	Values shown as	Tooltip title
`Count`	1.2 K, 3.4 Mil	Count
`Nanoseconds`	ns, µs, ms, s, min	Time
`Bytes`	B, KiB, MiB, GiB	RAM

format replaces the unit's formatting everywhere, and decimals fixes the number of decimals. Sample counts in tooltips are always shown in full, with thousands separators.

fn open_source(frame: &Selected<'_>, window: &mut Window, cx: &mut App) {
    println!("open {}", frame.label);
}

const ACTIONS: &[FrameAction] = &[FrameAction { label: "Open source", run: open_source }];

<FlameGraph data={profile} actions={ACTIONS} />

Selected gives the clicked frame's label, total, self_value and level.

Enums and types

Type	Values
`PaneLayout`	`Single(Pane)`, `Split(Pane, Pane)`
`Pane`	`TopTable`, `FlameGraph`, `CallTree`
`FlameColors`	`ByPackage`, `ByValue`, `Diff`, `DiffColorBlind`
`LabelAlign`	`Left` (default), `Right`: with `Right`, labels that don't fit keep the end of the function name
`SampleUnit`	`Count` (default), `Nanoseconds`, `Bytes`
`FrameAction`	`{ label: &'static str, run: fn(&Selected, &mut Window, &mut App) }`
`Selected`	`{ label, total, self_value, level }`

Coming from Grafana

Grafana	Here
data frame with `level`, `label`, `value`, `self`	`Profile::from_nested_set`
`valueRight`, `selfRight` (diff)	`Profile::diff_from_nested_set`, or `Profile::diff`
Show flame graph only	`header={false}` with `layout={PaneLayout::Single(Pane::FlameGraph)}`
view mode, pane selectors	`layout`
color scheme	`color_scheme`
text align	`text_align`
disable collapsing	`collapsing={false}`
vertical	`vertical`
keep focus on data change	`keep_focus_on_data_change`
extra context menu buttons	`actions`
Unit, Decimals	`unit`, `format`, `decimals`

Not available here: the "Analyze with Assistant" button and Grafana's usage analytics.

Run them

16 examples

Press Run to load an example into the live render at the top of the page. Open one to read its code.

How to read it

01Read width first. It is the share of the profile that includes this function and its callees. Height is depth, not cost.
02Look for wide boxes with nothing on top of them. That is self time, the work done in that function itself, and usually the real hot spot.
03Follow a wide tower down from main to find the path that leads to it. Colour groups related code by package by default, so you can see which library the time is in.
04Click a frame to focus on it and zoom in, and use the menu for the sandwich view, which shows all callers and callees of a function across the whole profile. Search to highlight matches.

When to use it, and when not to

Reach for it when

You need to find where a program spends CPU time, wall time, memory or allocations, and you have stack samples.
You want to compare two builds. A diff profile colours what got relatively slower or faster.
The profile is huge. Hundreds of thousands of samples reduce to a few dozen visible boxes, and you can zoom into any of them.
A flat top table is not enough: you need to know who called the expensive function and what it called.

Skip it when

You have measurements over time rather than stack samples. A flame graph merges away the time axis. Use Trend or Time Series.
The hierarchy is not a call tree but a set of categories with sizes. An icicle or sunburst is a closer fit.
You want exact counts for a short list of functions. The top table pane does that better, and is built in.
The profile has very few samples. A flame graph of 40 samples is a drawing of noise.

The math behind it

Layout from samples

A profile is a list of nodes in depth-first order, each with a level and a value (the number of samples, nanoseconds or bytes spent in that frame including its callees). The chart lays them out left to right: a node starts where its previous sibling ends, or at its parent's start if it is the first child. With $T$ the value of the root and $v_n$ , $a_n$ the value and start of node $n$ :

The width of a node is therefore its share of the whole profile, and widths of children never exceed the width of their parent. The remainder of a parent that is not covered by children is its self time, the part of the box that has nothing above it.

a_n = \begin{cases} a_{\text{parent}} & \text{first child} \\ a_{\text{prev}} + v_{\text{prev}} & \text{otherwise} \end{cases} \qquad \text{self}_n = v_n - \sum_{c \in \text{children}(n)} v_c

From share to pixels

If the view shows the fraction $[z_0, z_1]$ of the profile (full width until you zoom) in $W$ pixels, one unit of value is worth $W / \bigl(T\,(z_1 - z_0)\bigr)$ pixels, and a box is drawn $v_n$ times that wide. Boxes under half a pixel are not drawn, boxes of 10 pixels or less lose their border and are muted, and a label only appears when the box is at least 20 pixels wide.

A chain of single calls is a common source of tall, thin skinny towers. By default, a node that is the only child of its parent and carries more than 99% of the parent's value is grouped with it onto one row, so the graph shows the call that matters and not the six wrappers around it. collapsing={false} shows every frame.

w_n = v_n \cdot \frac{W}{T\,(z_1 - z_0)}

Differential profiles

Profile::diff(&before, &after) merges two profiles on the same call paths. Because the two runs rarely have the same length, the chart compares shares, not raw values: for a frame with values $l$ and $r$ in profiles of total $L$ and $R$ , it takes percentages rounded to two decimals, $p_l = 100\,l/L$ and $p_r = 100\,r/R$ , and computes the relative change, clamped to $\pm 100$ .

Zero change gets the neutral colour, and the colour mixes towards the "better" end for negative change and towards the "worse" end for positive change in proportion to $|\Delta|/100$ . A frame that exists only after has an infinite relative change and is clamped to full intensity. If either total is zero the neutral colour is used.

\Delta = \mathrm{clamp}\!\left(100\,\frac{p_r - p_l}{p_l},\ -100,\ 100\right)

Real life usage

Why the nightly export took four hours

Lucia maintained the invoice export at a logistics software company in Rotterdam, a Rust service that every night turned a million shipments into a pile of PDFs and a spreadsheet nobody read. For two years the job finished in about forty minutes. In March it started taking four hours, and the finance team started describing it, at length, as the slow thing.

Her first guess was the database, because it is always the first guess. Her second guess was the new PDF library, because it was the only thing that had changed. She had reasons to believe both, and no way of choosing between them without a lot of arguing.

She ran the job under perf for ten minutes on a copy of the production data and got a file of folded stacks, one line per distinct call path, with a count at the end. It was 61,000 lines long. She did not read it.

She loaded it into zopra-charts with Profile::from_folded and showed it in a <FlameGraph>, then did the same for last year's build and asked for a Profile::diff. The diff coloured every frame by how much its share of the run had changed, and nearly the whole graph came out a pale, uninteresting grey.

One tower came out red. It sat under a function called normalize_address, and it was wide: 71 per cent of the profile, where a year ago it had been 4. The database was fine. The PDF library was fine. A well-meaning change had added a regular expression, compiled inside the loop, to every one of a million addresses.

Moving the compile out of the loop took one line. The job went back to thirty-eight minutes, finance stopped talking about the slow thing, and the commit message, per tradition, said fix perf.

Lucia keeps the red tower as her desktop background. The regular expression, she maintains, was always going to be the problem. Nobody in the room was able to say the same about the database, so everybody was forgiven.

Pitfalls

Width is not self time. A wide box can be wide only because its callees are. Check what sits on top of it.
Sampling is statistical. Very short boxes are noisy and a function that runs for 1 ms once may never appear. Capture long enough for the question you are asking.
Left to right is not time. Siblings are placed in sample order or alphabetically, depending on the profiler, so the x axis is not a timeline. Do not read a sequence into it.
Inlined and missing frames. Optimised builds inline functions away and omit frame pointers, and the graph then shows what the profiler could recover, not what you wrote.
Passing a Profile directly. It is processed again on every render. Build it once and keep it in a signal.

From around the web

Watch and read

LISA13 - Blazing Performance with Flame GraphsUSENIXThe inventor's own plenary talk on what flame graphs show, how they are built, and how to read them.

Charts/Base

Flame Graph

<FlameGraph /> · 16 examples

A profile drawn as stacked boxes: width is time spent, height is call depth, and the wide ones are your problem.

Overview

Props

API

use zopra_charts::*;

The buttons use gpui-kit's built-in icons, so register its assets once when you start the app: gpui_kit::application().with_assets(gpui_kit::assets::Assets).

<FlameGraph>
Profile
Panes and layout
Interacting with the graph
Colours
Values and units
Custom menu actions
Enums and types
Coming from Grafana

Not sure which option fits? See the guide.

`<FlameGraph>`

Prop	Type	Default
`data`	`Profile` or `Signal<Profile>`	required
`layout`	`PaneLayout`	top table and flame graph side by side
`header`	`bool`	`true`
`unit`	`SampleUnit`	`Count`
`format`	`Fn(f64) -> SharedString`	none
`decimals`	`usize`	chosen by the unit
`color_scheme`	`FlameColors`	`ByPackage`, or `Diff` for a diff profile
`text_align`	`LabelAlign`	`Left`
`collapsing`	`bool`	`true`
`vertical`	`bool`	`false`
`keep_focus_on_data_change`	`bool`	`false`
`actions`	`&'static [FrameAction]`	none
`class`	Tailwind classes	fills its parent
`key`	anything `Into<ElementId>`	–

layout, color_scheme and text_align set the starting state. After that, the header and toolbar controls change them.

`Profile`

A profile is a call tree: each frame (a function in a call stack) has a total value, which includes everything it called, and a self value, which is only its own work.

Constructor	Use it for
`Profile::from_folded(text)`	Folded stacks: one line per stack, frames separated by `;`, then a count, e.g. `main;parse;lex 42`. This is what `perf` + `stackcollapse`, `cargo flamegraph` / `inferno`, `py-spy --format raw` and `async-profiler -o collapsed` produce. A `total` root frame is added on top.
`Profile::from_nested_set(rows)`	`(level, label, value, self)` rows in depth-first order, Grafana's profile format. Level 0 is the root.
`Profile::diff(&baseline, &comparison)`	Two profiles merged into a diff: matching stacks line up, and each frame keeps both values.
`Profile::diff_from_nested_set(rows)`	`(level, label, value, self, value_right, self_right)` rows that are already a diff.

Method	Returns
`is_diff()`	whether the profile compares two profiles
`len()` / `is_empty()`	the number of frames

Labels are stored once per distinct name, however many frames use them.

Panes and layout

Pane	Shows
`Pane::FlameGraph`	the flame graph
`Pane::TopTable`	one row per function: Self and Total (or Baseline, Comparison and Diff), sortable by any column
`Pane::CallTree`	the call tree as an expandable table, with its own search

`PaneLayout`	Meaning
`PaneLayout::Single(pane)`	one pane
`PaneLayout::Split(left, right)`	two panes with a draggable divider; stacked top and bottom when `vertical` is set

Interacting with the graph

Action	Result
hover a frame	tooltip with the total, self and sample count, as percentages too; Baseline / Comparison / Diff for a diff profile
click a frame	menu: Focus block, Copy function name, Sandwich view, your actions, and group expand / collapse
Focus block	zooms to that frame; frames above it are drawn lighter. A pill shows its share and clears the focus.
Sandwich view	shows that function's callers above and its callees below, merged across every place it was called
search box	dims every frame that doesn't match and filters the top table. Separate several terms with commas. Each term is tried as a regular expression first, then as a fuzzy match.
top table search / sandwich buttons	search for exactly that function / open its sandwich view
call tree up / down arrow buttons	show that function's callers / callees as the tree
call tree search	finds functions (case-insensitive; regular expressions allowed), shows "1 of N", and the arrows step through matches, expanding and scrolling to each
Reset	clears search, focus and sandwich

Changing data. When data changes, focus, sandwich and grouping reset, unless you set keep_focus_on_data_change.

Colours

`FlameColors`	Colour of each frame
`ByPackage`	by package or module name, so related functions share a colour
`ByValue`	yellow for small frames to red for large ones, relative to the current zoom
`Diff`	for diff profiles: your theme's good colour where a frame shrank, its neutral colour where it stayed the same, and its bad colour where it grew
`DiffColorBlind`	the same, in blue, grey and red

A diff profile shows a legend from −100% (removed) to +100% (added). Frames too narrow for a label are drawn grey. Set the good, neutral and bad colours in your theme.

Values and units

`SampleUnit`	Values shown as	Tooltip title
`Count`	1.2 K, 3.4 Mil	Count
`Nanoseconds`	ns, µs, ms, s, min	Time
`Bytes`	B, KiB, MiB, GiB	RAM

format replaces the unit's formatting everywhere, and decimals fixes the number of decimals. Sample counts in tooltips are always shown in full, with thousands separators.

fn open_source(frame: &Selected<'_>, window: &mut Window, cx: &mut App) {
    println!("open {}", frame.label);
}

const ACTIONS: &[FrameAction] = &[FrameAction { label: "Open source", run: open_source }];

<FlameGraph data={profile} actions={ACTIONS} />

Selected gives the clicked frame's label, total, self_value and level.

Enums and types

Type	Values
`PaneLayout`	`Single(Pane)`, `Split(Pane, Pane)`
`Pane`	`TopTable`, `FlameGraph`, `CallTree`
`FlameColors`	`ByPackage`, `ByValue`, `Diff`, `DiffColorBlind`
`LabelAlign`	`Left` (default), `Right`: with `Right`, labels that don't fit keep the end of the function name
`SampleUnit`	`Count` (default), `Nanoseconds`, `Bytes`
`FrameAction`	`{ label: &'static str, run: fn(&Selected, &mut Window, &mut App) }`
`Selected`	`{ label, total, self_value, level }`

Coming from Grafana

Grafana	Here
data frame with `level`, `label`, `value`, `self`	`Profile::from_nested_set`
`valueRight`, `selfRight` (diff)	`Profile::diff_from_nested_set`, or `Profile::diff`
Show flame graph only	`header={false}` with `layout={PaneLayout::Single(Pane::FlameGraph)}`
view mode, pane selectors	`layout`
color scheme	`color_scheme`
text align	`text_align`
disable collapsing	`collapsing={false}`
vertical	`vertical`
keep focus on data change	`keep_focus_on_data_change`
extra context menu buttons	`actions`
Unit, Decimals	`unit`, `format`, `decimals`

Not available here: the "Analyze with Assistant" button and Grafana's usage analytics.

Run them

16 examples

Press Run to load an example into the live render at the top of the page. Open one to read its code.

How to read it

01Read width first. It is the share of the profile that includes this function and its callees. Height is depth, not cost.
02Look for wide boxes with nothing on top of them. That is self time, the work done in that function itself, and usually the real hot spot.
03Follow a wide tower down from main to find the path that leads to it. Colour groups related code by package by default, so you can see which library the time is in.
04Click a frame to focus on it and zoom in, and use the menu for the sandwich view, which shows all callers and callees of a function across the whole profile. Search to highlight matches.

When to use it, and when not to

Reach for it when

You need to find where a program spends CPU time, wall time, memory or allocations, and you have stack samples.
You want to compare two builds. A diff profile colours what got relatively slower or faster.
The profile is huge. Hundreds of thousands of samples reduce to a few dozen visible boxes, and you can zoom into any of them.
A flat top table is not enough: you need to know who called the expensive function and what it called.

Skip it when

You have measurements over time rather than stack samples. A flame graph merges away the time axis. Use Trend or Time Series.
The hierarchy is not a call tree but a set of categories with sizes. An icicle or sunburst is a closer fit.
You want exact counts for a short list of functions. The top table pane does that better, and is built in.
The profile has very few samples. A flame graph of 40 samples is a drawing of noise.

The math behind it

Layout from samples

a_n = \begin{cases} a_{\text{parent}} & \text{first child} \\ a_{\text{prev}} + v_{\text{prev}} & \text{otherwise} \end{cases} \qquad \text{self}_n = v_n - \sum_{c \in \text{children}(n)} v_c

From share to pixels

w_n = v_n \cdot \frac{W}{T\,(z_1 - z_0)}

Differential profiles

\Delta = \mathrm{clamp}\!\left(100\,\frac{p_r - p_l}{p_l},\ -100,\ 100\right)

Real life usage

Why the nightly export took four hours

Moving the compile out of the loop took one line. The job went back to thirty-eight minutes, finance stopped talking about the slow thing, and the commit message, per tradition, said fix perf.

Pitfalls

Width is not self time. A wide box can be wide only because its callees are. Check what sits on top of it.
Sampling is statistical. Very short boxes are noisy and a function that runs for 1 ms once may never appear. Capture long enough for the question you are asking.
Left to right is not time. Siblings are placed in sample order or alphabetically, depending on the profiler, so the x axis is not a timeline. Do not read a sequence into it.
Inlined and missing frames. Optimised builds inline functions away and omit frame pointers, and the graph then shows what the profiler could recover, not what you wrote.
Passing a Profile directly. It is processed again on every render. Build it once and keep it in a signal.

From around the web

Overview

API

<FlameGraph>

Profile

Panes and layout

Interacting with the graph

Colours

Values and units

Custom menu actions

Enums and types

Coming from Grafana

16 examples

How to read it

When to use it, and when not to

Reach for it when

Skip it when

The math behind it

Layout from samples

From share to pixels

Differential profiles

Why the nightly export took four hours

Pitfalls

Watch and read

Related charts

Overview

API

<FlameGraph>

Profile

Panes and layout

Interacting with the graph

Colours

Values and units

Custom menu actions

Enums and types

Coming from Grafana

16 examples

How to read it

When to use it, and when not to

Reach for it when

Skip it when

The math behind it

Layout from samples

From share to pixels

Differential profiles

Why the nightly export took four hours

Pitfalls

Watch and read

Related charts

`<FlameGraph>`

`Profile`

`<FlameGraph>`

`Profile`