01 / The idea
The second order repeats the first.
Here is a small response from a store’s orders endpoint, 91 bytes of JSON: [{"id":101,"status":"shipped"},{"id":102,"status":"shipped"},{"id":103,"status":"packing"}]. Most of the second and third orders are bytes you already sent: {"id":10, ,"status":", shipped. JSON repeats
like this whenever every object carries the same keys.
LZ77 replaces bytes that already appeared with a copy: how far back they start, and how many to take. Everything else goes out as a literal byte.
Watch the encoder walk the response. It keeps the bytes it has already covered in view, its window, and at each position asks one question: does what comes next already appear back there?
Say it once. Point back after that.
Nothing written yet.
The first order is new, so every byte is a literal.
Nothing has been written yet, so there is nowhere to point. The first 31 bytes, the opening bracket, the whole first order and the comma after it, go out as literals at 9 bits each.
Reduced motion: choose a scene to see its completed state.
Read this scene
Nothing has been written yet, so there is nowhere to point. The first 31 bytes, the opening bracket, the whole first order and the comma after it, go out as literals at 9 bits each.
0 of 46 tokens cover 0 of 91 bytes, using 0 bits against 0 raw. No tokens yet.
Watch and Step through replay the encoder on the orders response. Try it runs the same TypeScript on your own text, one token at a time, and starts fresh each time you open it.
The decoder never searches. It reads a token, writes a byte or copies from its own output, and moves on. All the searching happens once, on the side that compresses.
02 / Name the rule
Longest match, nearest first, three bytes or more.
At each position, the encoder tries every distance back into the window and measures how many bytes match. It takes the longest. When two distances match equally far, it keeps the nearest, so the same input always gives the same tokens.
The last 4,096 bytes
Everything a copy may point into. DEFLATE’s reaches 32,768.
One byte · 9 bits
A flag bit, then the byte itself.
Distance, length · 24 bits
A flag bit, 15 bits of distance, 8 bits of length.
Those bit counts are this lesson’s own fixed-width format, chosen so every size can be counted by hand. They decide the shortest copy worth making. Two literals cost 18 bits, less than one 24-bit copy. Three cost 27, more. So a match shorter than three bytes goes out as literals.
One rule holds after every token: everything before the cursor is exactly what the decoder has written, and a copy only points into that. The copy’s length may run past the cursor, as the next section shows. Its start never can.
The limits come from DEFLATE. RFC 1951 says its representation “limits distances to 32K bytes and lengths to 258 bytes,” and a copy there is at least 3 bytes long. Our window stops at 4,096 so the search stays small enough to follow.
Why nearest first?A tie rule here, a size win in DEFLATE
In our format a distance always costs 15 bits, so nearest-first only makes the output repeatable. DEFLATE has a reason to care: its Huffman stage gives common values shorter codes. RFC 1951’s description of a compressor says it “searches the hash chains starting with the most recent strings, to favor small distances and thus take advantage of the Huffman encoding.”
03 / Read the shape
Encode searches. Decode only copies.
Basic form is the whole algorithm: encode, decode, and sizeInBits. In the wild is a server’s
per-response decision. At the call site runs three bodies. Both languages print
the same four lines and produce the same tokens for every shared case.
Encode takes the longest match within the window, nearest first, and falls back to a literal below three bytes. Decode copies one byte at a time. sizeInBits counts the lesson’s fixed-width format.
// LZ77 over bytes. Each token is either one literal byte or a copy: go back `distance`
// bytes in what has already been written and copy `length` bytes from there.
export const MIN_MATCH = 3; // shorter copies cost more than the literals they replace
export const MAX_MATCH = 258; // DEFLATE's longest copy (RFC 1951)
export const MAX_WINDOW = 4096; // a teaching bound; DEFLATE looks back up to 32,768 bytes
export const MAX_INPUT = 4096;
export type Literal = { byte: number };
export type Copy = { distance: number; length: number };
export type Token = Literal | Copy;
export type Lz77ErrorCode = 'bad-window' | 'too-long' | 'bad-token';
export class Lz77Error extends Error {
readonly code: Lz77ErrorCode;
constructor(code: Lz77ErrorCode, message: string) {
super(message);
this.name = 'Lz77Error';
this.code = code;
}
}
export function encode(input: Uint8Array, window = MAX_WINDOW): Token[] {
if (!Number.isInteger(window) || window < 1 || window > MAX_WINDOW)
throw new Lz77Error('bad-window', `window must be 1 to ${MAX_WINDOW} bytes`);
if (input.length > MAX_INPUT)
throw new Lz77Error('too-long', `input is limited to ${MAX_INPUT} bytes`);
const tokens: Token[] = [];
let cursor = 0;
while (cursor < input.length) {
const longest = Math.min(MAX_MATCH, input.length - cursor);
let best = 0;
let bestDistance = 0;
// Nearest first, and only a strictly longer match replaces the best: ties keep the nearest.
for (let distance = 1; distance <= Math.min(window, cursor); distance++) {
let length = 0;
// The copy may run past the cursor. The decoder will have written those bytes by then.
while (length < longest && input[cursor - distance + length] === input[cursor + length])
length++;
if (length > best) {
best = length;
bestDistance = distance;
if (best === longest) break;
}
}
if (best >= MIN_MATCH) {
tokens.push({ distance: bestDistance, length: best });
cursor += best;
} else {
tokens.push({ byte: input[cursor] });
cursor += 1;
}
}
return tokens;
}
export function decode(tokens: readonly Token[]): Uint8Array {
const output: number[] = [];
for (const token of tokens) {
if ('byte' in token) {
if (!Number.isInteger(token.byte) || token.byte < 0 || token.byte > 255)
throw new Lz77Error('bad-token', 'a literal is one byte, 0 to 255');
output.push(token.byte);
} else {
const { distance, length } = token;
if (!Number.isInteger(distance) || distance < 1 || distance > output.length)
throw new Lz77Error('bad-token', 'a copy can only point back into bytes already written');
if (!Number.isInteger(length) || length < MIN_MATCH || length > MAX_MATCH)
throw new Lz77Error('bad-token', `a copy is ${MIN_MATCH} to ${MAX_MATCH} bytes long`);
// One byte at a time, so a copy can read bytes it wrote a moment ago.
for (let i = 0; i < length; i++) output.push(output[output.length - distance]);
}
if (output.length > MAX_INPUT)
throw new Lz77Error('too-long', `output is limited to ${MAX_INPUT} bytes`);
}
return Uint8Array.from(output);
}
// This lesson's own fixed-width format, not DEFLATE's: one flag bit, then 8 bits for a
// literal, or 15 bits of distance and 8 bits of length for a copy.
export const LITERAL_BITS = 9;
export const COPY_BITS = 24;
export function sizeInBits(tokens: readonly Token[]): number {
return tokens.reduce((sum, token) => sum + ('byte' in token ? LITERAL_BITS : COPY_BITS), 0);
} // LZ77 over bytes. Each token is either one literal byte or a copy: go back Distance
// bytes in what has already been written and copy Length bytes from there.
const (
MinMatch = 3 // shorter copies cost more than the literals they replace
MaxMatch = 258 // DEFLATE's longest copy (RFC 1951)
MaxWindow = 4096 // a teaching bound; DEFLATE looks back up to 32,768 bytes
MaxInput = 4096
)
// Token is a literal when Length is 0, and a copy otherwise.
type Token struct {
Byte byte
Distance int
Length int
}
func (t Token) IsCopy() bool { return t.Length > 0 }
type Lz77Error struct {
Code string // "bad-window", "too-long", or "bad-token"
Message string
}
func (e *Lz77Error) Error() string { return e.Code + ": " + e.Message }
func Encode(input []byte, window int) ([]Token, error) {
if window < 1 || window > MaxWindow {
return nil, &Lz77Error{"bad-window", fmt.Sprintf("window must be 1 to %d bytes", MaxWindow)}
}
if len(input) > MaxInput {
return nil, &Lz77Error{"too-long", fmt.Sprintf("input is limited to %d bytes", MaxInput)}
}
tokens := []Token{}
cursor := 0
for cursor < len(input) {
longest := min(MaxMatch, len(input)-cursor)
best, bestDistance := 0, 0
// Nearest first, and only a strictly longer match replaces the best: ties keep the nearest.
for distance := 1; distance <= min(window, cursor); distance++ {
length := 0
// The copy may run past the cursor. The decoder will have written those bytes by then.
for length < longest && input[cursor-distance+length] == input[cursor+length] {
length++
}
if length > best {
best, bestDistance = length, distance
if best == longest {
break
}
}
}
if best >= MinMatch {
tokens = append(tokens, Token{Distance: bestDistance, Length: best})
cursor += best
} else {
tokens = append(tokens, Token{Byte: input[cursor]})
cursor++
}
}
return tokens, nil
}
func Decode(tokens []Token) ([]byte, error) {
output := []byte{}
for _, token := range tokens {
if !token.IsCopy() {
output = append(output, token.Byte)
} else {
if token.Distance < 1 || token.Distance > len(output) {
return nil, &Lz77Error{"bad-token", "a copy can only point back into bytes already written"}
}
if token.Length < MinMatch || token.Length > MaxMatch {
return nil, &Lz77Error{"bad-token", fmt.Sprintf("a copy is %d to %d bytes long", MinMatch, MaxMatch)}
}
// One byte at a time, so a copy can read bytes it wrote a moment ago.
for i := 0; i < token.Length; i++ {
output = append(output, output[len(output)-token.Distance])
}
}
if len(output) > MaxInput {
return nil, &Lz77Error{"too-long", fmt.Sprintf("output is limited to %d bytes", MaxInput)}
}
}
return output, nil
}
// This lesson's own fixed-width format, not DEFLATE's: one flag bit, then 8 bits for a
// literal, or 15 bits of distance and 8 bits of length for a copy.
const (
LiteralBits = 9
CopyBits = 24
)
func SizeInBits(tokens []Token) int {
sum := 0
for _, token := range tokens {
if token.IsCopy() {
sum += CopyBits
} else {
sum += LiteralBits
}
}
return sum
} Reading the TypeScriptBytes, not characters
Input is a Uint8Array. planResponse turns text into bytes with TextEncoder, which produces UTF-8, so é is two bytes. Indexing the
string itself would count UTF-16 code units instead, and the tokens would not match what goes
over the network.
A token is { byte } or { distance, length }, and 'byte' in token tells them
apart. Errors are an Lz77Error whose code matches the Go version.
Reading the Go[]byte, and a zero length for literals
Input is a []byte, and []byte(body) is already UTF-8. A Token is a literal when Length is 0. Encode and Decode return (value, error), with a *Lz77Error carrying the same codes as TypeScript.
What is refusedWindows, sizes, and tokens that don’t fit
A window outside 1 to 4,096 bytes is bad-window. Input or decoded output
over 4,096 bytes is too-long. When decoding, a copy that points before the
start of the output, a distance of 0, or a length outside 3 to 258 is bad-token.
Both languages check that decoding every encoding gives back the input, and that each token really is the longest match and the nearest one, on the shared cases and on 300 generated inputs.
04 / Try a decision
A copy longer than what it copies.
At the call site, and in the lab’s “A copy that overlaps” preset, “ha ha ha ha ha!” became h a \x20 <3,11> ! (the lab draws that space as ␠): a copy of 11 bytes from only 3 back. Before you trust that,
decode a smaller one yourself.
This is written into DEFLATE itself. RFC 1951: “if the last 2 bytes decoded have values X and Y, a string reference with <length = 5, distance = 2> adds X,Y,X,Y,X to the output stream.” It is also why a decoder must behave as if it copies one byte at a time whenever a copy overlaps: a bulk copy of the source range would read bytes that haven’t been written yet.
05 / Follow the cost
Repetition is the whole budget.
| Operation | Time | Extra space | What it assumes |
|---|---|---|---|
| Choose one token | O(w · m) | O(1) | Tries every distance in a window of w bytes, comparing up to m = 258 bytes at each. A comparison stops at the first byte that differs. |
| Encode n bytes | O(n · w · m) | O(t) | At worst. Holds the t tokens it emits; t is at most n. |
| Decode | O(n) | O(n) | One step per byte written, into the output it is building. No searching. |
| Count the size | O(t) | O(1) | 9 bits per literal and 24 per copy, in this lesson’s fixed-width format. |
The work is lopsided on purpose. Compressing searches; decompressing copies. A response is compressed once on the server and decompressed on every device that reads it.
The size depends mostly on how much repeats. We measured real gzip on three bodies next to our format. These are sizes from real codecs, not our teaching format pretending to be gzip:
| Body | Raw | Lesson format | gzip (Node) | gzip (Go) |
|---|---|---|---|---|
| The orders response (3 orders) | 91 | 58 | 69 | 70 |
| The same shape with 40 orders | 1,227 | 267 | 181 | 183 |
| 1,024 random bytes | 1,024 | 1,152 | 1,047 | 1,049 |
Three things stand out. On the tiny response our format wins only because a gzip file always carries a header of at least 10 bytes and an 8-byte trailer (RFC 1952). The DEFLATE data inside it is 51 bytes, smaller than our 58; those 18 bytes of framing take it to 69. On 40 orders gzip wins clearly. It doesn’t spend a fixed 9 bits on every literal and 24 on every copy: its Huffman stage gives common values shorter codes. And random bytes get bigger in both. RFC 1951 says it plainly: “no lossless compression algorithm can compress every possible input data set.”
What does the window size change?Reach against search time
A bigger window finds repeats further back and costs more searching. Shrink ours to 16 bytes and the orders response loses every copy: the repeats are 30 bytes back, so all 91 bytes go out as literals, 819 bits. Try it in the lab with a 16-byte window.
Real encoders don’t try every distance. RFC 1951 describes a compressor that “uses a chained hash table to find duplicated strings, using a hash function that operates on 3-byte sequences,” and truncates very long chains to avoid a worst case. The format doesn’t require that strategy; any encoder whose tokens decode correctly is allowed.
06 / Give it a real job
Compress a response only when it pays.
A server that compresses responses decides for each one. planResponse in In the wild encodes the body’s UTF-8 bytes, counts the whole bytes its tokens
would fill at 9 and 24 bits, and chooses them only if that is smaller than the body itself. The
orders response goes from 91 bytes to 58. The laugh goes from 15 to 8. A 16-byte API key with
no repeats would grow to 18, so it goes out as it is.
In production that encoder is gzip or Brotli, not our format, and nothing but this lesson can read our tokens. The decision around it is the same one: repetitive text is worth compressing, and a short body with nothing to point back to is not.
Build UIs?Every fetch you make already negotiates compression. Uploads are where it becomes your call.
Where it already is in your components
Every fetch takes part. The browser adds Accept-Encoding to the
request itself: the Fetch standard lists it among the forbidden request headers, “forbidden so the user agent remains in full control
over them.” When the response comes back with Content-Encoding: gzip, fetch decodes it before your code reads a byte; the
standard’s steps handle content codings and
track the encoded and decoded sizes separately.
That pair is what DevTools shows. With big request rows on, Chrome’s documentation says the bottom value in the Size column “is the uncompressed size of a request.” MDN describes gzip as “a format using the Lempel-Ziv coding (LZ77), with a 32-bit CRC.”
When you have to own it
Uploads go the other way. fetch sends the body you hand it, and Accept-Encoding only describes what the browser accepts back. So when a support
tool uploads a few megabytes of exported JSON, compressing it is the page’s decision.
CompressionStream is the browser’s own gzip, Baseline since May 2023 on MDN and a stable global in Node from 22.15, which
is how the snippet’s tests run. It skips bodies under a kilobyte and formats that are already
compressed, judged by the type the Blob or File carries. In those tests, a 16-byte body with
no repeats came out of gzip bigger than it went in.
Two conditions sit outside the snippet. The server has to expect Content-Encoding: gzip on a request and decompress it; nothing negotiates
that for an upload. And Content-Encoding is not one of the Fetch standard’s CORS-safelisted request headers, so a cross-origin upload needs the server to allow it.
// Shrink a large export before uploading it. CompressionStream is the browser's own gzip,
// with DEFLATE's LZ77 and Huffman stages, not this lesson's teaching format.
export const MIN_BYTES = 1024; // a policy: small bodies save little, and gzip adds its own header
// Already-compressed formats have little repetition left to find.
const COMPRESSED = /^(image\/(?!svg\+xml)|video\/|audio\/|font\/woff2?$|application\/(zip|gzip)$)/;
export function worthCompressing(type: string, bytes: number): boolean {
return bytes >= MIN_BYTES && !COMPRESSED.test(type);
}
export function gzip(body: Blob): Promise<Blob> {
return new Response(body.stream().pipeThrough(new CompressionStream('gzip'))).blob();
}
// Takes a Blob or a File, so the type it was made with decides whether to compress.
export async function uploadExport(
url: string,
body: Blob,
send: typeof fetch = fetch
): Promise<Response> {
if (!worthCompressing(body.type, body.size))
return send(url, { method: 'POST', headers: { 'Content-Type': body.type }, body });
return send(url, {
method: 'POST',
// The server has to expect this and decompress it. Nothing negotiates it for an upload.
headers: { 'Content-Type': body.type, 'Content-Encoding': 'gzip' },
body: await gzip(body)
});
}
React and Svelte don’t change any of this. Call uploadExport from the export
button’s handler in either one, with new Blob([JSON.stringify(rows)], { type: 'application/json' }), or
with a File from a file input as it is.
07 / Make the call
Ship gzip. Keep the idea.
For real data, use a real codec. In Go, compress/gzip and compress/flate implement RFC 1952 and RFC 1951; in the browser, CompressionStream. Their
output reads anywhere gzip does. This lesson’s tokens don’t: no Huffman stage, no bit
packing, no header, and nothing else understands them.
Keep the idea for the decisions around the codec. Compress text that repeats: JSON, logs, HTML, CSV. Skip formats that are already compressed. Don’t expect a tiny body to shrink. And remember the window: a repeat further back than the encoder looks is invisible to it.
SourcesStandards, packages, and docs, checked 13 September 2026
- RFC 1951, DEFLATE: “a combination of the LZ77 algorithm and Huffman coding”; distances and lengths; the overlapping copy; the hash-chain compressor in section 4.
- RFC 1952, gzip: compression method 8 is “deflate”; the member header and CRC32 and ISIZE trailer.
- RFC 7932, Brotli: also “a combination of the LZ77 algorithm and Huffman coding.”
- Fetch standard: forbidden and CORS-safelisted request headers, and handling content codings.
- MDN on
Content-EncodingandCompressionStream; Chrome DevTools’ network reference; Node’s globals. - Go 1.27.1 source, the version the sizes above were measured with:
compress/flate“implements the DEFLATE compressed data format, described in RFC 1951”;compress/gzipfollows RFC 1952.
08 / Take the idea with you
Explain a gzip response without saying “LZ77.”
“The server sends each new run of bytes once. When a run shows up again, it sends how far back to look and how many bytes to copy, and your browser rebuilds the rest before your code sees it.”
Before moving on, open DevTools on your own app, turn on big request rows, and find the response with the biggest gap between its two sizes. Then look at how much of its body repeats.
Connections to follow nextRelated lessons
- Ring buffer is how a streaming decoder can keep a window of recent bytes without growing.
- Huffman coding is DEFLATE’s other half: shorter codes for the tokens that appear most.
- Aho–Corasick matching finds many known strings in one pass, where LZ77 looks back for any string that repeats.