← Math in Practice
Concept Quantities and representation

Floating point and rounding

A displayed decimal is a useful description, not always the value stored in memory.

A checkout total is calculated from a price and quantity. The screen shows $0.30, but a test comparing the computed number to 0.3 fails. Is the arithmetic wrong, is the display hiding a difference, or has a later rounding step changed the value?

The judgment to keep

Trace what the program stores, what each operation rounds, and what the product means by “equal.” Choose a comparison tolerance from the domain and units; represent money in integer minor units only when the currency and rounding policy support that model.

TypeScriptGo Binary fractions · rounding · floating comparisons · integer minor units
01 / Read the surprising sum

The cents look right on screen. The equality check still fails.

Imagine a checkout preview that adds two tax or discount components. One test says the result must equal exactly 0.3, but the runtime reports 0.1 + 0.2 = 0.30000000000000004. That tiny gap is not evidence of a faulty addition instruction. The inputs were converted from decimal text to binary floating-point values before the addition happened.

This is a deliberately small reproduction, not a real checkout incident. It isolates a clue: the displayed decimal, the stored binary value, and the application's equality rule are three different things. A formatter may print a short decimal that round-trips to the same binary value, so a neat display does not prove that the stored value is exactly that decimal fraction.

Requested decimal arithmetic0.1 + 0.2 = 0.3
Common binary64 result0.1 + 0.2 = 0.30000000000000004
Absolute difference0.30000000000000004 − 0.3 ≈ 5.55 × 10⁻¹⁷
02 / Inspect the representation

Binary fractions repeat too, just like one-third in decimal.

In base ten, 1/3 = 0.3333… cannot fit in a finite number of decimal places. The fraction 1/10 has a finite decimal spelling, but its binary expansion repeats: 0.0001100110011…₂. A typical JavaScript number and Go float64 use IEEE 754 binary64: a finite set of significant binary digits with a wide exponent range. Most decimal fractions therefore map to the nearest representable binary value, not their exact rational value.

The addition then operates on those nearby values and rounds its result to the nearest representable value under the format's rounding rule. The actual decimal output depends on the language's formatting algorithm; 0.30000000000000004 is the familiar shortest round-tripping display for this binary64 result. It is a traceable consequence of the encoding, not random noise.

Input conversiondecimal text → nearest binary value

0.1 and 0.2 are not exact binary fractions.

Operationstored value + stored value

The exact real sum may need another representable-value rounding.

Observationformat → decimal text

Display precision can conceal or expose the residual difference.

03 / Name the failure mode

Representation error, rounding, and integer precision are related but distinct.

Representation errorInput not exactly encodable

0.1 is approximated when represented in binary. The error exists before an explicit rounding call.

RoundingChoose a nearby value or display

Calling round or formatting to two decimal places applies a rule at a chosen step. Rounding too early can bias a total.

Integer precisionAdjacent integers stop being distinct

Binary64 represents every integer exactly only through 2⁵³ − 1. Above that, n and n + 1 may map to the same Number.

These call for different investigations. If two decimal inputs are slightly inexact, inspect the fraction and operation. If a result changes after formatting or a call to round, inspect the rounding mode, decimal scale, and when the rule is applied. If an identifier or counter above 9,007,199,254,740,991 changes by one but appears unchanged, inspect whether it passed through JavaScript number, JSON number parsing, or a binary64 API.

A check for Number.isSafeInteger can identify a Number that is not an exact safe integer. It cannot recover digits already lost while parsing. Large integer identifiers should usually cross JSON boundaries as strings, or use a deliberately supported arbitrary-precision format end to end.

04 / Choose a comparison rule

“Close enough” needs a unit, magnitude, and consequence.

Exact equality remains right for exact values and deliberate sentinels. For computed measurements, a common rule accepts a difference smaller than either an absolute tolerance or a relative tolerance scaled by the magnitude:

Differenced = |a − b|
Allowed errormax(absTol, relTol × max(|a|, |b|))
Decisionclose when d ≤ allowed error

Absolute tolerance protects comparisons near zero; relative tolerance scales for larger values. For 0.3 versus the computed sum, an absolute tolerance of 10⁻¹² accepts the roughly 5.55 × 10⁻¹⁷ gap. That value is only an illustration. There is no universal epsilon: comparing a distance in meters, a temperature reading, a normalized probability, and an account balance calls for different tolerances and different consequences when a comparison passes.

For a safety boundary, do not replace a required inequality with approximate equality. If the decision is “is this value at least the limit?”, compare against the domain threshold, define the boundary convention, and account for measurement uncertainty separately. Approximate equality is useful for expected-vs-measured checks, not as a universal substitute for ordering.

Compare two readings
Decision under this rule Within tolerance
Absolute difference5.5511e-17
Allowed difference1.0000e-12

This rule is a teaching aid. Decide the tolerance from measurement resolution, units, scale, and the cost of a false match; document that choice at the call site.

05 / Represent money deliberately

Integer cents avoid binary fractions, but they do not decide the accounting rule.

If an amount is denominated in a currency with two minor decimal places, $12.34 can be represented as the integer 1,234 cents. Addition and subtraction of those minor units are exact as long as the integer type does not overflow. In TypeScript, use bigint or a vetted decimal-money library for values beyond safe integer range; in Go, a bounded int64 may be sufficient after explicit range checks, while math/big handles larger quantities.

Multiplication and division bring policy back. A tax rate or unit price can produce a fraction of a cent. The system must specify whether to round per line or per invoice, which rounding mode applies, how negative refunds are handled, and when a currency has a different minor-unit scale. Integer cents also have a maximum range and can overflow; exact representation is not permission to skip validation.

Exact parse"19.99" → 1,999 minor units
Multiply by 31,999 × 3 = 5,997 → "$59.97"
Divide among 75,997 ÷ 7 = 856 remainder 5
06 / Practice in code

Keep the tolerance and the money scale visible in the code.

The same snippets in TypeScript and Go compare finite measurements with nonnegative absolute and relative tolerances. They also parse a decimal currency string into integer cents without passing the amount through binary floating point. The money parser deliberately accepts at most two fractional digits; that is a sample policy, not a universal currency rule. Production code should use currency-aware scales and enforce storage bounds and business rounding policy.

Compare the same choices in TypeScript and Go.

Tolerance is supplied by the caller; decimal currency text is converted exactly to a minor-unit integer.

TypeScriptTolerance-aware measurement comparison and exact minor-unit parsing
floating-point.ts
/** Compare measured values using a domain-sized absolute tolerance and relative tolerance. */
export function approximatelyEqual(
	a: number,
	b: number,
	absTolerance: number,
	relTolerance: number
): boolean {
	if (![a, b, absTolerance, relTolerance].every(Number.isFinite)) return false;
	if (absTolerance < 0 || relTolerance < 0) return false;
	const difference = Math.abs(a - b);
	if (difference <= absTolerance) return true;
	const scale = Math.max(Math.abs(a), Math.abs(b));
	return scale > 0 && Math.abs(a / scale - b / scale) <= relTolerance;
}

/** Parse a decimal currency amount with at most two fractional digits into cents. */
export function parseCents(amount: string): bigint {
	const match = /^(-?)(0|[1-9]\d*)(?:\.(\d{1,2}))?$/.exec(amount.trim());
	if (!match) throw new Error('Use a decimal amount with at most two fractional digits.');
	const [, sign, whole, fraction = ''] = match;
	const cents = BigInt(whole) * 100n + BigInt(fraction.padEnd(2, '0'));
	return sign ? -cents : cents;
}

// The runtime evaluates this as binary floating-point, so the displayed result
// exposes the representation detail rather than rounding it away.
export const familiarSum = 0.1 + 0.2;
GoTolerance-aware measurement comparison and exact minor-unit parsing
floating-point.go
package main

import (
	"errors"
	"math"
	"math/big"
	"regexp"
	"strings"
)

// ApproximatelyEqual uses tolerances chosen for the quantity's scale and units.
func ApproximatelyEqual(a, b, absTolerance, relTolerance float64) bool {
	if math.IsNaN(a) || math.IsNaN(b) || math.IsInf(a, 0) || math.IsInf(b, 0) ||
		math.IsNaN(absTolerance) || math.IsNaN(relTolerance) ||
		math.IsInf(absTolerance, 0) || math.IsInf(relTolerance, 0) ||
		absTolerance < 0 || relTolerance < 0 {
		return false
	}
	difference := math.Abs(a - b)
	if difference <= absTolerance {
		return true
	}
	scale := math.Max(math.Abs(a), math.Abs(b))
	return scale > 0 && math.Abs(a/scale-b/scale) <= relTolerance
}

var moneyPattern = regexp.MustCompile(`^(-?)(0|[1-9][0-9]*)(?:\.([0-9]{1,2}))?$`)

// ParseCents parses a decimal currency string exactly into arbitrary-precision cents.
func ParseCents(amount string) (*big.Int, error) {
	parts := moneyPattern.FindStringSubmatch(strings.TrimSpace(amount))
	if parts == nil {
		return nil, errors.New("use a decimal amount with at most two fractional digits")
	}
	whole := new(big.Int)
	whole.SetString(parts[2], 10)
	cents := new(big.Int).Mul(whole, big.NewInt(100))
	fraction := parts[3] + strings.Repeat("0", 2-len(parts[3]))
	if fraction != "" {
		minor, _ := new(big.Int).SetString(fraction, 10)
		cents.Add(cents, minor)
	}
	if parts[1] == "-" {
		cents.Neg(cents)
	}
	return cents, nil
}

func main() {
	_, _ = ParseCents("19.99")
}
07 / Diagnose before changing code

Find the first point where the stored value stops matching the intended quantity.

  1. State the quantity. Identify its unit, currency, scale, source, and whether it is a measurement, estimate, identifier, or amount owed.
  2. Reproduce raw inputs. Capture unformatted operands and the exact operation order; formatting can hide the relevant digits.
  3. Locate the first divergence. Inspect parsing, arithmetic, rounding, conversion, serialization, and database boundaries one at a time.
  4. Calculate an independent reference. Use rational or integer arithmetic for exact values, and record the domain's rounding point and mode.
  5. Test boundaries and transfer. Include values near zero, a comparison threshold, integer safe range, scale changes, and negative/remainder cases.
Transfer exercise

A dashboard reports 0.30000000000000004 seconds against a 300 ms target.

Before changing the comparison, convert both values to the same unit. Is the actual issue floating-point representation, a seconds-to-milliseconds conversion, rounding for display, or the threshold convention? Write down the evidence you would inspect and choose a rule whose tolerance reflects the timer's resolution and the decision's consequence.

For language details, see MDN's Number.EPSILON guidance and Number.MAX_SAFE_INTEGER. The Go specification describes floating-point operations and constants in its floating-point types section, and Go's math package documents machine epsilon and rounding functions. These references describe language formats and operations; the application's comparison and money rules still come from its domain.