← Applied algorithms
Smaller data, steadier systems From repeated bytes to pointers back

LZ77 compression

Say it once. Point back after that.

Open Chrome DevTools, turn on big request rows in the Network panel, and a response’s Size column shows two numbers: what came over the network, headers included, and underneath it the uncompressed size. When the server answered with Content-Encoding: gzip or br, most of the gap on a large response is a format built on LZ77. Both start from the same trick: bytes that already appeared are sent as a pointer back to them.

We’ll build that half for a small JSON response, count what it saves, and see why some bytes never get smaller.

TypeScriptGoOne API response in each language

01 / The idea

The second order repeats the first.

Here is a small response from a store’s orders endpoint, 91 bytes of JSON: [{"id":101,"status":"shipped"},{"id":102,"status":"shipped"},{"id":103,"status":"packing"}]. Most of the second and third orders are bytes you already sent: {"id":10, ,"status":", shipped. JSON repeats like this whenever every object carries the same keys.

LZ77 replaces bytes that already appeared with a copy: how far back they start, and how many to take. Everything else goes out as a literal byte.

Watch the encoder walk the response. It keeps the bytes it has already covered in view, its window, and at each position asks one question: does what comes next already appear back there?

LZ77

Say it once. Point back after that.

ORDERS RESPONSE · 91 BYTES 0 bits · 0 raw

Nothing written yet.

01/ 03
Nothing to point back to

The first order is new, so every byte is a literal.

Nothing has been written yet, so there is nowhere to point. The first 31 bytes, the opening bracket, the whole first order and the comma after it, go out as literals at 9 bits each.

Reduced motion: choose a scene to see its completed state.

Read this scene

Nothing has been written yet, so there is nowhere to point. The first 31 bytes, the opening bracket, the whole first order and the comma after it, go out as literals at 9 bits each.

0 of 46 tokens cover 0 of 91 bytes, using 0 bits against 0 raw. No tokens yet.

Watch and Step through replay the encoder on the orders response. Try it runs the same TypeScript on your own text, one token at a time, and starts fresh each time you open it.

The decoder never searches. It reads a token, writes a byte or copies from its own output, and moves on. All the searching happens once, on the side that compresses.

02 / Name the rule

Longest match, nearest first, three bytes or more.

At each position, the encoder tries every distance back into the window and measures how many bytes match. It takes the longest. When two distances match equally far, it keeps the nearest, so the same input always gives the same tokens.

Window

The last 4,096 bytes

Everything a copy may point into. DEFLATE’s reaches 32,768.

Literal

One byte · 9 bits

A flag bit, then the byte itself.

Copy

Distance, length · 24 bits

A flag bit, 15 bits of distance, 8 bits of length.

Those bit counts are this lesson’s own fixed-width format, chosen so every size can be counted by hand. They decide the shortest copy worth making. Two literals cost 18 bits, less than one 24-bit copy. Three cost 27, more. So a match shorter than three bytes goes out as literals.

One rule holds after every token: everything before the cursor is exactly what the decoder has written, and a copy only points into that. The copy’s length may run past the cursor, as the next section shows. Its start never can.

The limits come from DEFLATE. RFC 1951 says its representation “limits distances to 32K bytes and lengths to 258 bytes,” and a copy there is at least 3 bytes long. Our window stops at 4,096 so the search stays small enough to follow.

Why nearest first?A tie rule here, a size win in DEFLATE

In our format a distance always costs 15 bits, so nearest-first only makes the output repeatable. DEFLATE has a reason to care: its Huffman stage gives common values shorter codes. RFC 1951’s description of a compressor says it “searches the hash chains starting with the most recent strings, to favor small distances and thus take advantage of the Huffman encoding.”

03 / Read the shape

Encode searches. Decode only copies.

Basic form is the whole algorithm: encode, decode, and sizeInBits. In the wild is a server’s per-response decision. At the call site runs three bodies. Both languages print the same four lines and produce the same tokens for every shared case.

Encode takes the longest match within the window, nearest first, and falls back to a literal below three bytes. Decode copies one byte at a time. sizeInBits counts the lesson’s fixed-width format.

TypeScriptReading
lz77.ts
// LZ77 over bytes. Each token is either one literal byte or a copy: go back `distance`
// bytes in what has already been written and copy `length` bytes from there.
export const MIN_MATCH = 3; // shorter copies cost more than the literals they replace
export const MAX_MATCH = 258; // DEFLATE's longest copy (RFC 1951)
export const MAX_WINDOW = 4096; // a teaching bound; DEFLATE looks back up to 32,768 bytes
export const MAX_INPUT = 4096;

export type Literal = { byte: number };
export type Copy = { distance: number; length: number };
export type Token = Literal | Copy;

export type Lz77ErrorCode = 'bad-window' | 'too-long' | 'bad-token';

export class Lz77Error extends Error {
	readonly code: Lz77ErrorCode;
	constructor(code: Lz77ErrorCode, message: string) {
		super(message);
		this.name = 'Lz77Error';
		this.code = code;
	}
}

export function encode(input: Uint8Array, window = MAX_WINDOW): Token[] {
	if (!Number.isInteger(window) || window < 1 || window > MAX_WINDOW)
		throw new Lz77Error('bad-window', `window must be 1 to ${MAX_WINDOW} bytes`);
	if (input.length > MAX_INPUT)
		throw new Lz77Error('too-long', `input is limited to ${MAX_INPUT} bytes`);
	const tokens: Token[] = [];
	let cursor = 0;
	while (cursor < input.length) {
		const longest = Math.min(MAX_MATCH, input.length - cursor);
		let best = 0;
		let bestDistance = 0;
		// Nearest first, and only a strictly longer match replaces the best: ties keep the nearest.
		for (let distance = 1; distance <= Math.min(window, cursor); distance++) {
			let length = 0;
			// The copy may run past the cursor. The decoder will have written those bytes by then.
			while (length < longest && input[cursor - distance + length] === input[cursor + length])
				length++;
			if (length > best) {
				best = length;
				bestDistance = distance;
				if (best === longest) break;
			}
		}
		if (best >= MIN_MATCH) {
			tokens.push({ distance: bestDistance, length: best });
			cursor += best;
		} else {
			tokens.push({ byte: input[cursor] });
			cursor += 1;
		}
	}
	return tokens;
}

export function decode(tokens: readonly Token[]): Uint8Array {
	const output: number[] = [];
	for (const token of tokens) {
		if ('byte' in token) {
			if (!Number.isInteger(token.byte) || token.byte < 0 || token.byte > 255)
				throw new Lz77Error('bad-token', 'a literal is one byte, 0 to 255');
			output.push(token.byte);
		} else {
			const { distance, length } = token;
			if (!Number.isInteger(distance) || distance < 1 || distance > output.length)
				throw new Lz77Error('bad-token', 'a copy can only point back into bytes already written');
			if (!Number.isInteger(length) || length < MIN_MATCH || length > MAX_MATCH)
				throw new Lz77Error('bad-token', `a copy is ${MIN_MATCH} to ${MAX_MATCH} bytes long`);
			// One byte at a time, so a copy can read bytes it wrote a moment ago.
			for (let i = 0; i < length; i++) output.push(output[output.length - distance]);
		}
		if (output.length > MAX_INPUT)
			throw new Lz77Error('too-long', `output is limited to ${MAX_INPUT} bytes`);
	}
	return Uint8Array.from(output);
}

// This lesson's own fixed-width format, not DEFLATE's: one flag bit, then 8 bits for a
// literal, or 15 bits of distance and 8 bits of length for a copy.
export const LITERAL_BITS = 9;
export const COPY_BITS = 24;

export function sizeInBits(tokens: readonly Token[]): number {
	return tokens.reduce((sum, token) => sum + ('byte' in token ? LITERAL_BITS : COPY_BITS), 0);
}
GoAlongside
lz77.go
// LZ77 over bytes. Each token is either one literal byte or a copy: go back Distance
// bytes in what has already been written and copy Length bytes from there.
const (
	MinMatch  = 3    // shorter copies cost more than the literals they replace
	MaxMatch  = 258  // DEFLATE's longest copy (RFC 1951)
	MaxWindow = 4096 // a teaching bound; DEFLATE looks back up to 32,768 bytes
	MaxInput  = 4096
)

// Token is a literal when Length is 0, and a copy otherwise.
type Token struct {
	Byte     byte
	Distance int
	Length   int
}

func (t Token) IsCopy() bool { return t.Length > 0 }

type Lz77Error struct {
	Code    string // "bad-window", "too-long", or "bad-token"
	Message string
}

func (e *Lz77Error) Error() string { return e.Code + ": " + e.Message }

func Encode(input []byte, window int) ([]Token, error) {
	if window < 1 || window > MaxWindow {
		return nil, &Lz77Error{"bad-window", fmt.Sprintf("window must be 1 to %d bytes", MaxWindow)}
	}
	if len(input) > MaxInput {
		return nil, &Lz77Error{"too-long", fmt.Sprintf("input is limited to %d bytes", MaxInput)}
	}
	tokens := []Token{}
	cursor := 0
	for cursor < len(input) {
		longest := min(MaxMatch, len(input)-cursor)
		best, bestDistance := 0, 0
		// Nearest first, and only a strictly longer match replaces the best: ties keep the nearest.
		for distance := 1; distance <= min(window, cursor); distance++ {
			length := 0
			// The copy may run past the cursor. The decoder will have written those bytes by then.
			for length < longest && input[cursor-distance+length] == input[cursor+length] {
				length++
			}
			if length > best {
				best, bestDistance = length, distance
				if best == longest {
					break
				}
			}
		}
		if best >= MinMatch {
			tokens = append(tokens, Token{Distance: bestDistance, Length: best})
			cursor += best
		} else {
			tokens = append(tokens, Token{Byte: input[cursor]})
			cursor++
		}
	}
	return tokens, nil
}

func Decode(tokens []Token) ([]byte, error) {
	output := []byte{}
	for _, token := range tokens {
		if !token.IsCopy() {
			output = append(output, token.Byte)
		} else {
			if token.Distance < 1 || token.Distance > len(output) {
				return nil, &Lz77Error{"bad-token", "a copy can only point back into bytes already written"}
			}
			if token.Length < MinMatch || token.Length > MaxMatch {
				return nil, &Lz77Error{"bad-token", fmt.Sprintf("a copy is %d to %d bytes long", MinMatch, MaxMatch)}
			}
			// One byte at a time, so a copy can read bytes it wrote a moment ago.
			for i := 0; i < token.Length; i++ {
				output = append(output, output[len(output)-token.Distance])
			}
		}
		if len(output) > MaxInput {
			return nil, &Lz77Error{"too-long", fmt.Sprintf("output is limited to %d bytes", MaxInput)}
		}
	}
	return output, nil
}

// This lesson's own fixed-width format, not DEFLATE's: one flag bit, then 8 bits for a
// literal, or 15 bits of distance and 8 bits of length for a copy.
const (
	LiteralBits = 9
	CopyBits    = 24
)

func SizeInBits(tokens []Token) int {
	sum := 0
	for _, token := range tokens {
		if token.IsCopy() {
			sum += CopyBits
		} else {
			sum += LiteralBits
		}
	}
	return sum
}
Reading the TypeScriptBytes, not characters

Input is a Uint8Array. planResponse turns text into bytes with TextEncoder, which produces UTF-8, so é is two bytes. Indexing the string itself would count UTF-16 code units instead, and the tokens would not match what goes over the network.

A token is { byte } or { distance, length }, and 'byte' in token tells them apart. Errors are an Lz77Error whose code matches the Go version.

Reading the Go[]byte, and a zero length for literals

Input is a []byte, and []byte(body) is already UTF-8. A Token is a literal when Length is 0. Encode and Decode return (value, error), with a *Lz77Error carrying the same codes as TypeScript.

What is refusedWindows, sizes, and tokens that don’t fit

A window outside 1 to 4,096 bytes is bad-window. Input or decoded output over 4,096 bytes is too-long. When decoding, a copy that points before the start of the output, a distance of 0, or a length outside 3 to 258 is bad-token.

Both languages check that decoding every encoding gives back the input, and that each token really is the longest match and the nearest one, on the shared cases and on 300 generated inputs.

04 / Try a decision

A copy longer than what it copies.

At the call site, and in the lab’s “A copy that overlaps” preset, “ha ha ha ha ha!” became h a \x20 <3,11> ! (the lab draws that space as ␠): a copy of 11 bytes from only 3 back. Before you trust that, decode a smaller one yourself.

The decoder has written ab. The next token is <2,6>: go back 2 bytes and copy 6. What is the output now?

This is written into DEFLATE itself. RFC 1951: “if the last 2 bytes decoded have values X and Y, a string reference with <length = 5, distance = 2> adds X,Y,X,Y,X to the output stream.” It is also why a decoder must behave as if it copies one byte at a time whenever a copy overlaps: a bulk copy of the source range would read bytes that haven’t been written yet.

05 / Follow the cost

Repetition is the whole budget.

LZ77: time and extra space per operation
OperationTimeExtra spaceWhat it assumes
Choose one tokenO(w · m)O(1)Tries every distance in a window of w bytes, comparing up to m = 258 bytes at each. A comparison stops at the first byte that differs.
Encode n bytesO(n · w · m)O(t)At worst. Holds the t tokens it emits; t is at most n.
DecodeO(n)O(n)One step per byte written, into the output it is building. No searching.
Count the sizeO(t)O(1)9 bits per literal and 24 per copy, in this lesson’s fixed-width format.

The work is lopsided on purpose. Compressing searches; decompressing copies. A response is compressed once on the server and decompressed on every device that reads it.

The size depends mostly on how much repeats. We measured real gzip on three bodies next to our format. These are sizes from real codecs, not our teaching format pretending to be gzip:

Bytes. gzip at its highest level, measured once on 13 September 2026 with Node 22.21.1’s zlib and Go 1.27.1’s compress/gzip. The random bytes are new on every run, but their gzip size is not: five runs each gave 1,047 in Node and 1,049 in Go (Go’s figure added 22 September 2026).
BodyRawLesson formatgzip (Node)gzip (Go)
The orders response (3 orders)91586970
The same shape with 40 orders1,227267181183
1,024 random bytes1,0241,1521,0471,049

Three things stand out. On the tiny response our format wins only because a gzip file always carries a header of at least 10 bytes and an 8-byte trailer (RFC 1952). The DEFLATE data inside it is 51 bytes, smaller than our 58; those 18 bytes of framing take it to 69. On 40 orders gzip wins clearly. It doesn’t spend a fixed 9 bits on every literal and 24 on every copy: its Huffman stage gives common values shorter codes. And random bytes get bigger in both. RFC 1951 says it plainly: “no lossless compression algorithm can compress every possible input data set.”

What does the window size change?Reach against search time

A bigger window finds repeats further back and costs more searching. Shrink ours to 16 bytes and the orders response loses every copy: the repeats are 30 bytes back, so all 91 bytes go out as literals, 819 bits. Try it in the lab with a 16-byte window.

Real encoders don’t try every distance. RFC 1951 describes a compressor that “uses a chained hash table to find duplicated strings, using a hash function that operates on 3-byte sequences,” and truncates very long chains to avoid a worst case. The format doesn’t require that strategy; any encoder whose tokens decode correctly is allowed.

06 / Give it a real job

Compress a response only when it pays.

A server that compresses responses decides for each one. planResponse in In the wild encodes the body’s UTF-8 bytes, counts the whole bytes its tokens would fill at 9 and 24 bits, and chooses them only if that is smaller than the body itself. The orders response goes from 91 bytes to 58. The laugh goes from 15 to 8. A 16-byte API key with no repeats would grow to 18, so it goes out as it is.

In production that encoder is gzip or Brotli, not our format, and nothing but this lesson can read our tokens. The decision around it is the same one: repetitive text is worth compressing, and a short body with nothing to point back to is not.

Build UIs?Every fetch you make already negotiates compression. Uploads are where it becomes your call.

Where it already is in your components

Every fetch takes part. The browser adds Accept-Encoding to the request itself: the Fetch standard lists it among the forbidden request headers, “forbidden so the user agent remains in full control over them.” When the response comes back with Content-Encoding: gzip, fetch decodes it before your code reads a byte; the standard’s steps handle content codings and track the encoded and decoded sizes separately.

That pair is what DevTools shows. With big request rows on, Chrome’s documentation says the bottom value in the Size column “is the uncompressed size of a request.” MDN describes gzip as “a format using the Lempel-Ziv coding (LZ77), with a 32-bit CRC.”

When you have to own it

Uploads go the other way. fetch sends the body you hand it, and Accept-Encoding only describes what the browser accepts back. So when a support tool uploads a few megabytes of exported JSON, compressing it is the page’s decision.

CompressionStream is the browser’s own gzip, Baseline since May 2023 on MDN and a stable global in Node from 22.15, which is how the snippet’s tests run. It skips bodies under a kilobyte and formats that are already compressed, judged by the type the Blob or File carries. In those tests, a 16-byte body with no repeats came out of gzip bigger than it went in.

Two conditions sit outside the snippet. The server has to expect Content-Encoding: gzip on a request and decompress it; nothing negotiates that for an upload. And Content-Encoding is not one of the Fetch standard’s CORS-safelisted request headers, so a cross-origin upload needs the server to allow it.

upload.ts
// Shrink a large export before uploading it. CompressionStream is the browser's own gzip,
// with DEFLATE's LZ77 and Huffman stages, not this lesson's teaching format.
export const MIN_BYTES = 1024; // a policy: small bodies save little, and gzip adds its own header

// Already-compressed formats have little repetition left to find.
const COMPRESSED = /^(image\/(?!svg\+xml)|video\/|audio\/|font\/woff2?$|application\/(zip|gzip)$)/;

export function worthCompressing(type: string, bytes: number): boolean {
	return bytes >= MIN_BYTES && !COMPRESSED.test(type);
}

export function gzip(body: Blob): Promise<Blob> {
	return new Response(body.stream().pipeThrough(new CompressionStream('gzip'))).blob();
}

// Takes a Blob or a File, so the type it was made with decides whether to compress.
export async function uploadExport(
	url: string,
	body: Blob,
	send: typeof fetch = fetch
): Promise<Response> {
	if (!worthCompressing(body.type, body.size))
		return send(url, { method: 'POST', headers: { 'Content-Type': body.type }, body });
	return send(url, {
		method: 'POST',
		// The server has to expect this and decompress it. Nothing negotiates it for an upload.
		headers: { 'Content-Type': body.type, 'Content-Encoding': 'gzip' },
		body: await gzip(body)
	});
}

React and Svelte don’t change any of this. Call uploadExport from the export button’s handler in either one, with new Blob([JSON.stringify(rows)], { type: 'application/json' }), or with a File from a file input as it is.

07 / Make the call

Ship gzip. Keep the idea.

For real data, use a real codec. In Go, compress/gzip and compress/flate implement RFC 1952 and RFC 1951; in the browser, CompressionStream. Their output reads anywhere gzip does. This lesson’s tokens don’t: no Huffman stage, no bit packing, no header, and nothing else understands them.

Keep the idea for the decisions around the codec. Compress text that repeats: JSON, logs, HTML, CSV. Skip formats that are already compressed. Don’t expect a tiny body to shrink. And remember the window: a repeat further back than the encoder looks is invisible to it.

SourcesStandards, packages, and docs, checked 13 September 2026
  • RFC 1951, DEFLATE: “a combination of the LZ77 algorithm and Huffman coding”; distances and lengths; the overlapping copy; the hash-chain compressor in section 4.
  • RFC 1952, gzip: compression method 8 is “deflate”; the member header and CRC32 and ISIZE trailer.
  • RFC 7932, Brotli: also “a combination of the LZ77 algorithm and Huffman coding.”
  • Fetch standard: forbidden and CORS-safelisted request headers, and handling content codings.
  • MDN on Content-Encoding and CompressionStream; Chrome DevTools’ network reference; Node’s globals.
  • Go 1.27.1 source, the version the sizes above were measured with: compress/flate “implements the DEFLATE compressed data format, described in RFC 1951”; compress/gzip follows RFC 1952.

08 / Take the idea with you

Explain a gzip response without saying “LZ77.”

“The server sends each new run of bytes once. When a run shows up again, it sends how far back to look and how many bytes to copy, and your browser rebuilds the rest before your code sees it.”

Before moving on, open DevTools on your own app, turn on big request rows, and find the response with the biggest gap between its two sizes. Then look at how much of its body repeats.

Connections to follow nextRelated lessons
  • Ring buffer is how a streaming decoder can keep a window of recent bytes without growing.
  • Huffman coding is DEFLATE’s other half: shorter codes for the tokens that appear most.
  • Aho–Corasick matching finds many known strings in one pass, where LZ77 looks back for any string that repeats.

Copy the complete example, add a fourth order with a new status, and predict which of its bytes become copies before you run it.

Back to applied algorithms →