Designing for Failure: A Backend Engineer's Mental Model

What you actually need to know

• Every external dependency can fail. Design for it.
• Timeouts on every call. Default ones are too long.
• Retries with exponential backoff and maximum attempts. Idempotency required.
• Circuit breakers stop calling failing dependencies.
• Bulkheads isolate failure. Graceful degradation preserves the user experience.

Pattern

What it prevents

Timeout

Hung callers

Retry with backoff

Cascading failures

Circuit breaker

Capacity exhaustion

Bulkhead

Failure spreading

Graceful degradation

Customer facing breakage

Idempotency

Duplicate side effects on retry

Dead letter

Lost work on terminal failure

Backpressure

Producer overrunning consumer

The core argument

Production systems fail, not occasionally but often: dependencies go slow, networks partition, disks fill, services restart. The teams that pretend failure is exceptional ship systems that fail badly the moment reality intrudes. The teams that design for failure ship systems that degrade gracefully and recover on their own.

The mental model is small. Every external call can fail. The application has to handle it. The handling is a small set of patterns applied consistently. Timeouts on every call. Retries with backoff. Circuit breakers for repeat failures. Bulkheads to isolate. Graceful degradation to preserve customer experience.

The cost of the patterns is small. The libraries exist. The application code adds a few lines per call site. The reasoning is the discipline. The team has to think about what happens when the call fails. The answer should be specific to the call. A retry might be right. A circuit breaker might be right. A graceful fallback might be right.

The teams that do this well have production systems that survive dependency failures the customer never sees. The teams that do not have systems that cascade. A slow third party API takes down the application. A database hiccup propagates to every endpoint. The team learns by causing incidents.

The patterns reference

Pattern

Implementation

Timeout

Per call, set deliberately. Usually a few seconds for hot path.

Retry with backoff

Exponential. Max attempts. Idempotent required.

Circuit breaker

After N failures in a window, stop calling. Half open test to recover.

Bulkhead

Separate resource pools per dependency.

Graceful degradation

Fallback that is worse but not broken.

Idempotency

Server side support so retries are safe.

Dead letter

Failed work captured for later inspection.

Backpressure

Producer slows when consumer is overwhelmed.

Compensating transaction

Roll back a side effect when later step fails.

Hedged requests

Send to multiple replicas, take the first.

How much does this cost

The cost is small once the patterns are in the team's repertoire. A few hours per critical surface to apply them. The libraries do the heavy lifting. The reasoning is the discipline. The cost is mostly the time to think about failure, not the code to handle it.

Features the failure design must have

• Timeouts on every external call.
• Retry policy documented per call site.
• Circuit breakers on the highest stakes dependencies.
• Bulkheads where dependencies could starve each other.
• Graceful fallbacks for user facing features.
• Idempotency on write endpoints.
• Dead letter handling for background work.
• Monitoring on timeout rate, retry rate, breaker state.
• Chaos testing of failure modes.

Expert opinion

The teams that design for failure ship systems that survive the kinds of incidents that take down their peers. The patterns are small. The discipline is to apply them consistently. The teams that do not design for failure spend years learning the patterns by causing incidents. The cost of incidents is large. The cost of applying the patterns by default is small.

Yashveer Singh, founder of Yashveer Labs

How this played out on a real project

A client hit a major incident when their payment processor slowed down. The application called the processor synchronously, with no timeout. The slow calls held threads, the threads filled the application's pool, and the whole application stopped responding. The incident lasted four hours.

We applied the failure patterns. Timeout on every external call. Circuit breaker on the payment processor. Graceful degradation that put orders into a pending state when the processor was unavailable. Dead letter queue for the orders that needed manual reprocessing.

A similar event six months later was contained. The circuit breaker opened. The application kept serving other traffic. Orders went into the pending state. The processor came back online. The pending orders were retried. The customers saw a brief delay on payment confirmations instead of an outage.

For more on the related work, see resilience patterns circuit breakers retries bulkheads and timeouts the setting most engineers get wrong.

Common mistakes teams make

1. Default timeouts. Usually too long.
2. No retry policy. Or retries without backoff.
3. No circuit breakers on dependencies that fail.
4. No bulkheads. One dependency starves others.
5. No graceful degradation. Features break entirely on dependency failure.
6. No idempotency. Retries duplicate side effects.
7. No dead letter handling. Failed work disappears.
8. No chaos testing. Failure modes are theoretical.

A 30 day plan to put patterns in place

1. Week one. Audit external calls. Identify the ones without timeouts.
2. Week two. Add timeouts. Add retries where appropriate.
3. Week three. Add circuit breakers on high stakes dependencies.
4. Week four. Add graceful fallbacks for user facing features. Run chaos tests.

For more on the related work, read resilience patterns circuit breakers retries bulkheads and chaos engineering at startup scale. On the broader reliability side, the outbox pattern a SaaS reliability cheat code is the natural next read.

FAQ

Frequently asked

• What is the failure mental model?
• What is the most basic failure pattern?
• What about retries?
• What is a circuit breaker?
• What is a bulkhead?
• What does graceful degradation look like?
• What is the worst failure design mistake?

Author

About the author and why it matters

Yashveer Singh wrote this. I run Yashveer Labs out of New Delhi. The work I take on tends to come from founders who have been burned by an agency, a freelancer, or their own ambition. I do not promise miracles. I promise that the system will be online, the code will be readable, and the next engineer who touches it will not curse me. That is rarer than it should be.

Start the conversation See the work DM on Instagram