Charts/Diagram charts
UpSet
<UpSet /> · 10 examples
A bar chart of set intersections, with a dot matrix underneath saying which sets each bar belongs to.
Overview
Put five circles in a Venn diagram and you get something a florist would be proud of and a statistician would send back. There are 31 ways to belong to some of five sets, the circles can only draw a handful of them well, and the labels end up wedged into slivers the width of a hair. The Venn diagram is a lovely idea that scales to about three.
UpSet, introduced by Alexander Lex, Nils Gehlenborg and colleagues in a 2014 paper, throws out the circles and keeps the idea. Every combination of sets that actually occurs becomes a column. The bar on top says how many items are in exactly that combination. Under it, a column of dots says which sets those are: filled for yes, grey for no, a line joining the filled ones. It looks like a bar chart that is wearing a small spreadsheet, and it works far better than it sounds.
The result is that you can read off the biggest intersections in order, instead of squinting at overlapping lenses. The price is that it is a plot you have to be taught to read once. After that it is hard to go back.
Props
API
<UpSet data={items} sets={|i: &Item| i.sets} />| Prop | Type | Default |
|---|---|---|
data, sets, value, separator | as on <Venn> | |
max_intersections | usize: combination bars shown, largest first | 20 |
min_size | f64: hide smaller combinations | 0. |
Bars at the top are exact combinations, the dot matrix below shows which sets each one contains, and the bars on the left are set sizes. Use it when there are too many sets for a Venn diagram.
Run them
10 examples
Press Run to load an example into the live render at the top of the page. Open one to read its code.
How to read it
- 01Start at the top bars and read left to right: columns are sorted from the biggest combination to the smallest.
- 02For any bar, look straight down to the dots. The filled dots, joined by a line, are the sets whose intersection this bar counts, and nothing else.
- 03Use the bars on the left to see which sets are big. A big set whose members mostly sit in single-set columns is a set that stands on its own.
- 04Hover a column for the exact size and its share of all items. Hover a left bar for the full size of that set.
When to use it, and when not to
Reach for it when
- You have four or more sets and you want to know which combinations are common and which are rare.
- You care about exact sizes. Every bar has a number and a common baseline, so comparing 412 to 389 is something the eye can actually do.
- The data is genomic, survey, tag or feature-flag data, where items belong to many categories at once and the interesting question is which categories travel together.
- You want to see that most items belong to just one set (a very tall first few bars, then a long tail) or the opposite (a lot of items in many sets at once).
Skip it when
- You have two or three sets and the audience has never seen an UpSet plot. Venn answers the same question with no tutorial.
- Hundreds of combinations all matter. You will only see the top 20 by default and it is easy to make the picture worse by raising the limit.
- You need to see containment (every item in A is also in B). A sunburst or tree shows nesting better than dots do.
The math behind it
Combinations, not circles
Each row of the data lists the sets one item belongs to, joined by a separator (& by default), and optionally a value for how many items the row represents. Without one, a row counts as 1. Non-positive or non-finite values and rows with no set names are discarded. The rows are collapsed to exact combinations: a bitmask of memberships and a total .
Intersection sizes in UpSet are exclusive. The bar for counts only the items that are in and and in nothing else. That is why the bars add up to the whole data set, and why they differ from the pairwise overlap a Venn diagram would show.
The set bars
The horizontal bars on the left show the size of each set. Unlike the columns, a set size counts everything in the set, including the items it shares with others.
The longest set bar takes the full width of that panel and the rest are drawn relative to it.
Which columns are drawn
The columns are sorted by size, largest first. Combinations smaller than min_size (default 0) are removed first, then only the top max_intersections (default 20) survive. A plot with 5 sets can have 31 columns, and past about 20 the dots become too small to read. The cut-off is a readability setting, not a statistical one.
The matrix has one row per set in order of first appearance, with alternating stripes. Each column draws a line from its first to its last filled dot, so combinations that skip a row are still easy to follow. At most 64 sets are supported.
Real life usage
Seven flags and a bad Tuesday
Marcus was a platform engineer at a payments company called Ledgerly, and he was in charge of feature flags. There were seven of them guarding the new checkout: express-pay, saved-cards, a currency thing, three experiments nobody remembered starting, and one called temp_do_not_remove that had been there for three years.
On a Tuesday, checkout errors rose to four times the normal rate for forty minutes. By the time anybody looked, it had fixed itself, which is the least helpful way for an incident to end.
The question for the post-mortem was simple to ask and annoying to answer: which combinations of flags had the failing users actually had on? Seven flags have 127 possible combinations, and nobody wanted to open 127 dashboards. A Venn diagram of seven circles was never going to survive the meeting.
Marcus exported the failed sessions with the flags they had, one row per session and the flags joined by an ampersand, and passed it to <UpSet> in zopra-charts. He left every option at its default but the title.
The tall bars on the left were the ones he expected: express-pay alone, saved-cards alone, the two together. But the third from the right was a short column of 61 sessions with the dots lit in rows two, five and seven, and the line between them was the first thing anyone had pointed at all afternoon. Two of the experiments and the flag nobody dared remove, all on together.
That combination existed on exactly one day of the year: the day the three experiments' schedules overlapped. They had never been tested together because they had never overlapped before. One of them was turned off that afternoon. The error rate was back to normal by five, and the other two experiments were rescheduled to avoid each other.
Marcus has since added a column to the rollout checklist: the UpSet plot of who is in what. He says it is the only chart in the company where the most useful part is the part with the least ink.
Pitfalls
- Exclusive versus total. A column counts items in exactly that combination. If you add the "A and B" bar to the "A, B and C" bar you get the pairwise overlap, but no bar shows that on its own.
- The 20-column default. Combinations beyond
max_intersectionsare not drawn, so the visible bars may not sum to the total. Raise the limit or filter withmin_sizedeliberately. - Sets with the separator in the name.
&is the default separator, so a set calledR&Dis read as two sets. Passseparator. - Counting items versus counting rows. With no
value, each row counts as 1. If your rows are already aggregated (one row per combination with a count), pass the count asvalueor every bar will be 1. - Row order is not meaningful. Sets appear in the order they were first seen in the data, not by size. If you want the biggest set on top, sort your input.
From around the web