Startup, readiness,
and liveness probes.

The incident report says the ticket database slowed down, API replicas restarted, and reconnect work rose during recovery. That sequence deserves attention, but it does not yet prove the probe caused the restarts. A process exit, memory kill, node disruption, or rollout can happen near the same dependency incident.

A process can be running while it initializes, running while a database is unavailable, or running while its work is stuck. A single green/red flag collapses those states. You need to decide which response would help before writing the check.

A probe is a repeated check whose result feeds a controller’s decision. In Kubernetes, the kubelet runs container probes. A failing result is therefore part of the control loop, not just a status message on a dashboard.

Once startup succeeds, its check stops for that container lifetime. Readiness can change repeatedly. Liveness runs independently of readiness after the startup gate opens; an unready container can still fail liveness and restart. These are the platform’s distinct probe semantics.

Give this service three explicit endpoint contracts. Return a small 200 response for success and 503 for failure. Keep the endpoints cheap, unauthenticated for the kubelet’s access path, and free of sensitive diagnostics. Avoid redirecting a probe to a login page.

In a real service, readiness might read a recent, bounded dependency observation from a background monitor. Define how old that observation may be and how recovery refreshes it. A cached “good” result that never expires is a false promise; synchronous probes that pile up slow database calls can become extra load during the outage.

For liveness, choose evidence tied to useful local progress. An HTTP response from the request event loop can reveal that loop being blocked; a separate health thread returning 200 says little about a deadlocked worker. If a heartbeat drives the check, define when progress is expected so an idle worker is not mistaken for a stuck one.

The fixture takes 50 seconds to start. Give startup about 120 seconds of nominal allowance: 24 failed samples at a five-second period. That leaves room for a slow cold start without forcing the running service’s liveness check to wait two minutes before noticing a fault.

This excerpt lives under the container in a Deployment. The named port http maps to port 8080. The complete manifest and disposable service appear in the drill below.

timeoutSeconds bounds one attempt. failureThreshold counts consecutive failed attempts before action. A success breaks that failure streak. Readiness also uses successThreshold to control recovery; it is one here. Startup and liveness require a success threshold of one. See the Kubernetes configuration reference.

The two readiness failures tolerate a brief blip before withdrawing traffic. The three liveness failures tolerate a little more before restart. Their nominal windows are about 10 and 15 seconds; they are not precise deadlines measured from the onset of a fault. Probe scheduling, timeouts, and the phase of the first failure affect detection. Termination and endpoint propagation add their own delays.

Choose these numbers from cold-start measurements, normal local pauses, probe latency, and the time users can tolerate a broken replica. CPU throttling or overload can delay a healthy process. Aggressive thresholds can then remove capacity at exactly the wrong time.

Reusing /readyz for liveness looks tidy. For this service, it turns database failure into a request to restart the application. Nothing in that action makes the database available again.

That last step is a risk to investigate, not an automatic numerical claim. This page’s experiment counts restarts in a small model; it does not measure database load or outage duration. It makes the unnecessary intervention visible.

In the database scenario, local liveness leaves the process running. Readiness withdraws traffic, and the same process can become ready when the dependency returns. The database-dependent policy restarts twice in the authored schedule. In the stuck-process scenario, a restart is useful: both policies recover local progress.

If all three replicas depend on the same failed database, all three may become unready. There is still a user-visible outage. Readiness does not manufacture a working replica; keeping liveness local prevents additional damage while you repair the dependency or activate a designed fallback.

A health handler test can prove which status code a condition produces. It cannot prove that your manifest points to that handler, the kubelet reaches the port, or traffic stops going to an unready endpoint. Verify the application contract and the deployed reaction separately.

Capture the initial restart count so an earlier crash does not look like a new liveness action. Inspect Pod events and the previous container’s termination reason. An out-of-memory kill, application exit, and liveness-triggered restart need different fixes.

Endpoint readiness is control-plane evidence. Also send representative requests through the Service from another Pod and watch which replica serves them. Direct Pod access and port forwarding bypass ordinary Service selection; they cannot establish that unready replicas have stopped receiving Service traffic.

If the expected withdrawal or restart never happens, check the configured path, named port, probe events, and actual response first. If the response is correct but traffic persists, inspect the routing path and existing connections. If restarts happen without liveness failures, follow the termination reason instead of tuning probe thresholds.

A liveness probe earns its place when it can detect a failure that a restart is likely to repair. If your process already exits reliably on unrecoverable local failure, the restart policy may be enough. Adding a second failure detector is a design decision with false positives and operating costs.

Keep monitoring alongside probes. Alert on user-visible errors and latency, sustained loss of ready capacity, and repeated restarts. A green health endpoint does not establish that real requests meet their promise.

Leave a short note with the configuration: what each endpoint means, where its signal comes from, why its tolerances fit, and how to exercise its failure and recovery. That note helps the next person resist “simplifying” three different questions into one.

The question to keep: if this check fails across every replica, does the action help the system recover?