01 / The idea
Turn matching pairs into groups.
A detector compares a weekend’s photos and reports pairs that look the same: 0412 with 0413, 0413 with 0414, and so on. The library needs groups, not pairs. If 0412 matches 0413 and 0413 matches 0414, all three are one stack in the duplicates view, with one photo to keep.
Union-find keeps those groups as a forest. Every photo points at a parent, and the photo at the top of each tree points at itself: the root, which names the group. Find follows parents up to the root. Union finds two roots and, if they differ, points one at the other. Two photos are in the same group exactly when their roots match.
Apple’s Photos app “identifies duplicate photos and videos in your photo library in the
Duplicates collection.” Python’s networkx ships a UnionFind class. LLVM’s EquivalenceClasses has a find that “does the path-compression part that makes union-find ‘union findy’.” And the
Rust compiler depends on ena, a crate of “union-find, congruence closure, and other unification code.”
02 / Name the rule
Join roots, not photos.
Merging two groups never touches their members. It changes one link: the root of one tree now points at the root of the other. Once the roots are found, merging a group of six costs the same as merging two single photos.
The work is in finding roots, and that depends on how tall the trees grow. Two rules keep them short. Union by size puts the smaller tree under the larger, so a photo moves one link further from its root only when its group at least doubles; no tree of n photos is more than log₂ n links tall. Path compression repoints every photo a find passes straight at the root, so the next find from any of them takes one step.
With both, m operations on n photos cost O(m α(n)) in total. α is an inverse of the Ackermann function, which grows so slowly that it is a small constant for any number of photos you could store.
Why can’t a union-find group be split?Links forget where they came from
A link records that one root went under another, not which pair caused it. After the pair 0414 and 0417, the link from 0415 to 0412 exists because of 0414, but nothing in the forest says so. Taking 0414 out cannot tell which links to undo, and path compression may already have repointed photos past links that belong to no pair.
So splitting means rebuilding from the source of truth, the list of pairs. That is quick:
rebuilding from p pairs is p near-constant unions. Structures that can undo exist, but
they keep more than parent links. The ena crate, for one, can take a snapshot and
later roll changes back to it.
03 / Follow one operation
Ten photos, seven pairs.
The weekend has two bursts of the same scene, 0412 to 0414 and 0415 to 0417, and two near-identical shots, 0418 and 0419. The detector reports seven pairs. One of them, 0414 with 0417, says the two bursts are the same scene.
Before you watch, predict how many links change when that pair merges two groups of three, and what checking 0412 against 0416 does to the forest. The animation replays what the TypeScript example recorded. Try it lets you mark and check pairs of your own.
Find both roots. Join them, or stop.
Forest · each photo points to a parent; a root points to itself
- 0412root
- 0413root
- 0414root
- 0415root
- 0416root
- 0417root
- 0418root
- 0419root
- 0420root
- 0421root
Groups No duplicates yet
Sets 10 Links followed 0 Repointed 0 Roots joined 0
Every photo starts as a set of its own.
Each of the ten photos points at itself: it is a root, and a set of one. A detector has reported seven pairs of photos that look the same, and the library merges sets as each pair arrives.
Reduced motion: choose a scene to see its completed state.
Read this scene
Each of the ten photos points at itself: it is a root, and a set of one. A detector has reported seven pairs of photos that look the same, and the library merges sets as each pair arrives.
Each of the ten photos points at itself: it is a root, and a set of one. A detector has reported seven pairs of photos that look the same, and the library merges sets as each pair arrives.
Links followed so far: 0. Groups: 0.
Watch restarts when you return. Step through keeps your selected step. Try it starts from ten separate photos each time you open it.
04 / Read the shape
One forest, one library.
Basic form is DisjointSets: add, find, union, connected, and
groups, with union by size and path compression each able to be switched off so the lesson
can measure them. In the wild wraps it in PhotoLibrary, which checks photo ids and sizes, records every pair, keeps the
largest file of each group, reports the bytes the rest take, and takes a photo out by
rebuilding.
Disjoint sets as a forest of parent links. Union by size and path compression can each be switched off, so their effect can be measured, and every link read, repointed, or joined can be recorded.
export type Step = {
/** up: read one parent link. point: repoint a node straight at its root. link: put one root under another. */
kind: 'up' | 'point' | 'link';
node: string;
/** The parent that was read, the root pointed at, or the root linked under. */
to: string;
/** For link: the size of the merged set. */
size: number;
};
export type Options = { bySize?: boolean; compress?: boolean };
// Disjoint sets as a forest. Every item points at a parent, and a root points at itself; two
// items are in the same set when their roots match. Union by size puts the smaller tree under
// the larger, so no tree grows deeper than log2 of its size. Path compression repoints every
// item on a find straight at the root, so the next find from there is one step.
export class DisjointSets {
readonly bySize: boolean;
readonly compress: boolean;
#parent = new Map<string, string>();
#size = new Map<string, number>();
#sets = 0;
constructor({ bySize = true, compress = true }: Options = {}) {
this.bySize = bySize;
this.compress = compress;
}
has(id: string): boolean {
return this.#parent.has(id);
}
add(id: string): void {
if (this.#parent.has(id)) throw new RangeError(`${id} is already in a set`);
this.#parent.set(id, id);
this.#size.set(id, 1);
this.#sets++;
}
find(id: string, steps?: Step[]): string {
this.#need(id);
return this.#root(id, steps);
}
// Returns false when the two were already in one set.
union(a: string, b: string, steps?: Step[]): boolean {
this.#need(a, b);
let big = this.#root(a, steps);
let small = this.#root(b, steps);
if (big === small) return false;
if (this.bySize && this.#size.get(big)! < this.#size.get(small)!) [big, small] = [small, big];
const size = this.#size.get(big)! + this.#size.get(small)!;
this.#parent.set(small, big);
this.#size.set(big, size);
this.#sets--;
steps?.push({ kind: 'link', node: small, to: big, size });
return true;
}
connected(a: string, b: string, steps?: Step[]): boolean {
this.#need(a, b);
return this.#root(a, steps) === this.#root(b, steps);
}
sizeOf(id: string): number {
this.#need(id);
return this.#size.get(this.#root(id))!;
}
parentOf(id: string): string {
this.#need(id);
return this.#parent.get(id)!;
}
/** Every set, each in the order its items were added, sets in the order of their first item. */
groups(): string[][] {
const at = new Map<string, string[]>();
for (const id of this.#parent.keys()) {
const root = this.#root(id);
if (!at.has(root)) at.set(root, []);
at.get(root)!.push(id);
}
return [...at.values()];
}
items(): string[] {
return [...this.#parent.keys()];
}
get count(): number {
return this.#sets;
}
#need(...ids: string[]): void {
for (const id of ids) if (!this.#parent.has(id)) throw new RangeError(`unknown item ${id}`);
}
// Follow parents to the root, then, with compression, repoint the path at it.
#root(id: string, steps?: Step[]): string {
let root = id;
for (let up = this.#parent.get(root)!; up !== root; up = this.#parent.get(root)!) {
steps?.push({ kind: 'up', node: root, to: up, size: 0 });
root = up;
}
if (this.compress)
for (let node = id; this.#parent.get(node) !== root;) {
const next = this.#parent.get(node)!;
this.#parent.set(node, root);
steps?.push({ kind: 'point', node, to: root, size: 0 });
node = next;
}
return root;
}
} // Step is one move. Kind is "up" (read one parent link), "point" (repoint a node straight
// at its root), or "link" (put one root under another). To is the parent read, the root
// pointed at, or the root linked under; Size is the merged set's size for a link.
type Step struct {
Kind string `json:"kind"`
Node string `json:"node"`
To string `json:"to"`
Size int `json:"size"`
}
// DisjointSets is a forest. Every item points at a parent, and a root points at itself;
// two items are in the same set when their roots match. Union by size puts the smaller
// tree under the larger, so no tree grows deeper than log2 of its size. Path compression
// repoints every item on a find straight at the root, so the next find from there is one step.
type DisjointSets struct {
BySize bool
Compress bool
parent map[string]string
size map[string]int
order []string
sets int
}
func NewDisjointSets(bySize, compress bool) *DisjointSets {
return &DisjointSets{BySize: bySize, Compress: compress, parent: map[string]string{}, size: map[string]int{}}
}
func (d *DisjointSets) Has(id string) bool {
_, ok := d.parent[id]
return ok
}
func (d *DisjointSets) Add(id string) error {
if d.Has(id) {
return fmt.Errorf("%s is already in a set", id)
}
d.parent[id] = id
d.size[id] = 1
d.order = append(d.order, id)
d.sets++
return nil
}
func (d *DisjointSets) Find(id string, steps *[]Step) (string, error) {
if err := d.need(id); err != nil {
return "", err
}
return d.root(id, steps), nil
}
// Union returns false when the two were already in one set.
func (d *DisjointSets) Union(a, b string, steps *[]Step) (bool, error) {
if err := d.need(a, b); err != nil {
return false, err
}
big, small := d.root(a, steps), d.root(b, steps)
if big == small {
return false, nil
}
if d.BySize && d.size[big] < d.size[small] {
big, small = small, big
}
d.parent[small] = big
d.size[big] += d.size[small]
d.sets--
record(steps, "link", small, big, d.size[big])
return true, nil
}
func (d *DisjointSets) Connected(a, b string, steps *[]Step) (bool, error) {
if err := d.need(a, b); err != nil {
return false, err
}
return d.root(a, steps) == d.root(b, steps), nil
}
func (d *DisjointSets) SizeOf(id string) (int, error) {
if err := d.need(id); err != nil {
return 0, err
}
return d.size[d.root(id, nil)], nil
}
func (d *DisjointSets) ParentOf(id string) (string, error) {
if err := d.need(id); err != nil {
return "", err
}
return d.parent[id], nil
}
// Groups returns every set, each in the order its items were added, sets in the order of
// their first item.
func (d *DisjointSets) Groups() [][]string {
at := map[string]int{}
groups := [][]string{}
for _, id := range d.order {
root := d.root(id, nil)
i, seen := at[root]
if !seen {
i = len(groups)
at[root] = i
groups = append(groups, nil)
}
groups[i] = append(groups[i], id)
}
return groups
}
func (d *DisjointSets) Items() []string { return slices.Clone(d.order) }
func (d *DisjointSets) Count() int { return d.sets }
func (d *DisjointSets) need(ids ...string) error {
for _, id := range ids {
if !d.Has(id) {
return fmt.Errorf("unknown item %s", id)
}
}
return nil
}
// root follows parents to the root, then, with compression, repoints the path at it.
func (d *DisjointSets) root(id string, steps *[]Step) string {
root := id
for d.parent[root] != root {
record(steps, "up", root, d.parent[root], 0)
root = d.parent[root]
}
if d.Compress {
for node := id; d.parent[node] != root; {
next := d.parent[node]
d.parent[node] = root
record(steps, "point", node, root, 0)
node = next
}
}
return root
}
func record(steps *[]Step, kind, node, to string, size int) {
if steps != nil {
*steps = append(*steps, Step{kind, node, to, size})
}
} Reading the TypeScriptTwo maps and a swap
#parent and #size are Maps from photo id; a size only means
something at a root. union finds both roots and swaps them with [big, small] = [small, big] when the second group is larger, so the rest of the
method has one case.
#root walks up once to find the root, then walks the same path again,
repointing each photo. groups() calls it for every photo, so listing the groups
also flattens the forest.
Reading the GoAn order slice and array keys
DisjointSets keeps an order slice beside its maps, so groups
come out in the order photos were added, not in map order. PhotoLibrary stores pairs as [2]string arrays, which compare
with ==, so looking for a recorded pair needs no key type.
Separate drops pairs with slices.DeleteFunc, then builds a
fresh DisjointSets and replays the pairs that are left.
What would I normally use in application code?A few lines, or a batch job
Neither TypeScript nor Go ships union-find. With both improvements it is about thirty
lines, and application code usually writes it inline: an array of parents indexed by
number, and a find with compression. Python’s networkx has UnionFind.
If the groups live in a database, store the pairs and recompute groups in a batch job, or keep a group id column and merge ids in a transaction. Union-find is for working in memory on a set of pairs that only grows.
05 / Try a decision
Not a duplicate.
Someone disagrees with the detector about one photo. Decide what the library has to do before the feedback tells you.
06 / Follow the cost
4,095 links, or one.
Here is every operation at a glance, with n photos and p recorded pairs. The rest of this section measures what the two improvements buy.
| Operation | Time | Extra space | What it assumes |
|---|---|---|---|
| Find a group’s root | O(α(n)) amortized | O(1) | With union by size and path compression. With union by size alone, at most log₂ n links. |
| Merge two groups | O(α(n)) amortized | O(1) | Two finds, then one link changes. No other photo in either group moves. |
| Check two photos | O(α(n)) amortized | O(1) | Two finds, then compare the roots. |
| List every group | O(n α(n)) | O(n) | One find per photo. The finds flatten the forest as they go. |
| Take a photo out | O(n + p α(n)) | O(n + p) | Rebuild from the p pairs that remain, because a merged set cannot be split. |
| Neither improvement | O(n) per find | O(1) | Pairs in an unlucky order build a chain as tall as the group. |
| Hold n photos and p pairs | — | O(n + p) | A parent and a size per photo, plus every recorded pair and its lookup key, kept so the groups can be rebuilt after a photo is taken out. |
In the animation, the bridging pair followed two links and changed one, joining two groups of three. Every setting, with or without either improvement, gives the same groups; the improvements only change how far a find walks. With union by size, no photo is ever more than log₂ of its group’s size links from its root.
The worst case is a realistic one: a detector that reports a burst in shooting order, each new photo matched with the one before. Merging 4,096 photos that way with neither improvement built one chain, and looking up every photo once followed 8,386,560 links, 4,095 of them for the oldest photo. With union by size, every photo went straight under one root, and each lookup took one link. With path compression alone, the first lookup paid 4,095 links and flattened the chain on the way; every later lookup took one.
Random pairs are gentler. With 100,000 photos, 100,000 random pairs, and 100,000 random checks, the longest single call followed 7 links with neither improvement and 3 with both, and the total fell from 237,409 links to 210,675. The improvements are insurance against the order your data happens to arrive in. These are counts from the lesson’s TypeScript example, not timings.
07 / Give it a real job
Keep the pairs, not just the forest.
The pairs are the record, and the forest is a fast index over them. PhotoLibrary stores every distinct pair it is given, so it can rebuild after a photo
is taken out.
Real review adds two decisions. A person’s “not a duplicate” should outlive the next detector run, so store it and ignore later pairs that name that photo. And keeping the largest file is a policy, not a fact: an app might prefer the highest resolution or the copy someone edited. This library keeps the largest and says so.
Build UIs?See where grouping by pairs already shows up, and the day you merge an imported address book.
Where it already is in your components
The Duplicates collection in Photos is this job done for you. So is any screen where people mark two things as the same, such as two contacts, two tags, or two accounts, and see them gathered into one: pairs go in, groups come out, and a chain of pairs joins things nobody compared directly.
When you have to own it
You are building the import step of an address book. Someone brings a file from their old
phone and another from their work email, and Sam turns up three times: once with sam@okafor.dev, once with Sam@Okafor.dev and a mobile number, and
once with only that number, written with spaces. Two rows are the same person when a normalized
email or phone number matches, so the first row joins the second by email, the second joins
the third by phone, and all three become one contact, though the first and third share nothing.
Keep a map from each normalized email and number to the first row that had it. Every later row
that hits the map is a pair: union the two, then draw one card per group.
Then it gets real: rows arrive in batches while the person is already reviewing cards. Merge each batch as it lands, because that costs a few lookups per row, but redraw at most once per frame, and key each card by its earliest row so a card that grows keeps focus. A union cannot be undone, so when someone taps Not the same person, remember that those rows are apart, rebuild from the rows, and ignore matches between them from then on.
Merged contacts from an import. A map from each normalized email and phone number finds the pairs, a small union-find joins them, and each group becomes one card.
import { useMemo } from 'react';
export type ImportedRow = { id: string; name: string; email?: string; phone?: string };
export type Contact = { key: string; name: string; details: string[]; rows: number };
// Two rows are the same person when a normalized email or phone number matches. Case and
// spaces do not make a different address; brackets, dashes, and spaces do not make a
// different number. Each key comes with the text as the row wrote it, for the card.
function matchKeys(row: ImportedRow): [key: string, written: string][] {
const keys: [string, string][] = [];
const email = row.email?.trim().toLowerCase();
const phone = row.phone?.replace(/\D/g, '');
if (email) keys.push([`email:${email}`, row.email!.trim()]);
if (phone) keys.push([`phone:${phone}`, row.phone!.trim()]);
return keys;
}
export function mergeContacts(rows: ImportedRow[]): Contact[] {
// A small union-find over row positions. A row that is its own parent is a group's root.
const parent = rows.map((_, i) => i);
const size = rows.map(() => 1);
const find = (i: number): number => (parent[i] === i ? i : (parent[i] = find(parent[i])));
const union = (a: number, b: number) => {
let [big, small] = [find(a), find(b)];
if (big === small) return;
if (size[big] < size[small]) [big, small] = [small, big];
parent[small] = big;
size[big] += size[small];
};
// The first row seen with each email or number. A later row with the same one makes a
// pair, and pairs chain: A matches B by email, B matches C by phone, and all three are
// one person, though nothing compared A with C.
const firstWith = new Map<string, number>();
rows.forEach((row, i) => {
for (const [key] of matchKeys(row)) {
const first = firstWith.get(key);
if (first === undefined) firstWith.set(key, i);
else union(first, i);
}
});
// One card per group, in the order of each group's first row. Every row with a key is
// in one group, so a key seen once is already on its card.
const cards = new Map<number, Contact>();
const shown = new Set<string>();
rows.forEach((row, i) => {
const root = find(i);
let card = cards.get(root);
if (!card) cards.set(root, (card = { key: row.id, name: row.name, details: [], rows: 0 }));
card.rows++;
for (const [key, written] of matchKeys(row)) {
if (shown.has(key)) continue;
shown.add(key);
card.details.push(written);
}
});
return [...cards.values()];
}
export function MergedContacts({ rows }: { rows: ImportedRow[] }) {
// Grouping is one pass over the rows, so redo it only when the import changes.
const contacts = useMemo(() => mergeContacts(rows), [rows]);
return (
<section>
<h2>
{rows.length} rows · {contacts.length} contacts
</h2>
{contacts.map((contact) => (
<article key={contact.key}>
<h3>{contact.name}</h3>
<ul>
{contact.details.map((detail) => (
<li key={detail}>{detail}</li>
))}
</ul>
{contact.rows > 1 && <p>Merged from {contact.rows} rows</p>}
</article>
))}
</section>
);
}
08 / Make the call
Ask whether groups only grow.
Reach for union-find when things join and never leave, and the question is whether two of them are together: duplicates, linked records, pixels of one region, or points already connected by cable. It answers in near-constant time with two maps or two arrays.
Look elsewhere when the question changes. If you need the way two photos are connected, which pairs lead from one to the other, keep an adjacency list and search it breadth first. If groups must split often, rebuild from the pairs each time, or use a structure that can undo. If you want the cheapest set of links that connects everything, Kruskal’s algorithm, in Applied algorithms, is sorting plus union-find. And if you only need to know whether something is in one fixed set, a hash set is simpler.
09 / Take the idea with you
Explain it without saying “union-find.”
“Every photo points to another photo in its group, and one photo in each group points to itself and names the group. To ask whether two photos are together, I follow both up to their names. To join two groups, I point one name at the other, the smaller group under the larger, and I point everything I pass straight at the top. I cannot split a group, so I keep the pairs and rebuild.” That describes the mechanism. The name is what you call it in a review.
Before moving on, explain three things without the name: why merging a group of six changes one link, why the oldest photo in a chain took 4,095 links with neither improvement, and why taking a photo out means rebuilding. Then find a place in your code that keeps merging lists of related ids, and see whether it is union-find in disguise.
Connections to follow nextRelated lessons
- Graph overview names nodes, edges, direction, and weight, and sends each question to the lesson that answers it.
- Adjacency list keeps the pairs themselves, when you need to know how two things are connected.
- Breadth-first and depth-first search find connected groups by walking those pairs instead.
- Hash map is what the parent and size tables are here, keyed by photo id.
- Kruskal’s minimum spanning tree, in Applied algorithms, is built on union-find.