Sometimes the best way to make a system more resilient is to teach it when to stop.
Imagine an order service needs to call a payment service.
Normally, the payment service responds in 100 milliseconds.
Then something goes wrong.
The first request waits 10 seconds and fails.
The order service tries again.
Another 10 seconds.
Another retry.
Meanwhile, hundreds of customers are placing orders. Every request is waiting for the same struggling payment service.
Threads become occupied. Connections pile up. Queues grow. Memory consumption increases.
One unhealthy service has started making healthy services unhealthy.
This is exactly the problem the circuit breaker pattern tries to prevent.
A circuit breaker watches calls to another service and temporarily stops sending requests when failures cross a defined threshold.
Instead of repeatedly asking a broken service whether it is still broken, the system gives it some room to recover.
Distributed Systems Turn Small Failures Into Big Ones
In a monolithic application, a failing function may be relatively easy to contain.
In distributed systems, one request can travel through several network calls.
An order might involve:
Order Service → Inventory Service → Payment Service → Shipping Service → Notification Service
Microservices give teams useful independence, but they also introduce dependencies over networks.
Networks are unreliable.
Services become overloaded.
Containers restart.
Databases slow down.
Deployments go wrong.
Connections time out.
A service therefore has to assume that another service will eventually become unavailable.
The interesting question isn’t whether failure happens.
It is how the system behaves when it does.
Retries Can Make the Problem Worse
The obvious response to a failed request is:
“Try again.”
And sometimes that is exactly right.
Perhaps a network connection briefly disappeared. A retry 200 milliseconds later succeeds and the user never notices.
But retries have a dangerous side.
Imagine a service already struggling under 10,000 requests.
Every caller automatically retries each failure three times.
The unhealthy service may now receive several times the traffic precisely when it has the least capacity to handle it.
Retries intended to improve reliability can delay recovery.
This is why resilient systems need more than retries.
They also need to know when retrying is pointless.
The Circuit Breaker Has Three States
The circuit breaker pattern borrows its name from electrical circuits.
When something dangerous happens, the circuit is interrupted rather than allowing the problem to continue.
In software, a circuit breaker typically moves between three states:
closed → open → half-open
The names can initially feel backwards.
A closed circuit means requests are allowed through.
An open circuit means requests are blocked.
The half-open state carefully tests whether the dependency has recovered.
Once those three states make sense, the entire pattern becomes much easier to understand.
Closed: Everything Looks Healthy
The circuit normally begins closed.
Requests pass through to the downstream service as usual.
The circuit breaker watches what happens.
Imagine 100 calls reach the payment service.
Most succeed.
A few fail.
That may be completely normal.
Distributed systems occasionally experience errors, so one failed request shouldn’t necessarily shut down communication.
Instead, the circuit breaker tracks failures against a failure threshold.
Perhaps the rule says:
Open the circuit if more than 50% of the last 20 requests fail.
As long as failures remain below that threshold, the circuit stays closed.
Then the payment service starts having serious problems.
Failures climb.
The threshold is crossed.
The circuit opens.
Open: Stop Sending Traffic to Something That’s Already Failing
When the circuit is open, requests are no longer sent to the unhealthy dependency.
They fail immediately or use some fallback behavior.
This is called failing fast.
Instead of waiting ten seconds for a timeout, the application may know almost immediately that the payment service is temporarily unavailable.
That sounds harsh.
It is actually protective.
The caller doesn’t waste connections waiting for requests that are unlikely to succeed.
The struggling service receives less traffic.
Resources are preserved.
Other parts of the system get a better chance of remaining healthy.
This is how circuit breakers help prevent cascading failures.
Cascading Failures Are the Real Enemy
Suppose Service C is unhealthy.
Service B depends on Service C, so its requests start waiting.
Service A depends on Service B, so its requests also begin waiting.
Now Service A’s clients retry.
More traffic reaches Service A.
Its connection pools fill.
Memory rises.
Service A becomes unhealthy too.
A single failure at the bottom of the chain has travelled upward through the architecture.
This is one of the unpleasant realities of microservices.
Services can fail together without sharing any code.
They only need to share dependencies.
Circuit breakers create boundaries around those failures.
Service C can be unhealthy without automatically dragging every caller down with it.
Timeouts Decide How Long We’re Willing to Wait
Circuit breakers work particularly well alongside timeouts.
A timeout answers:
How long should I wait for this individual request?
A circuit breaker answers:
Should I keep making requests to this dependency at all?
Those are different decisions.
Without sensible timeouts, requests can remain stuck while resources accumulate.
Without a circuit breaker, the application can continue making requests that repeatedly hit those timeouts.
Together they provide stronger protection.
A request waits only so long.
Repeated failures eventually stop future calls.
Half-Open: Has the Service Recovered Yet?
An open circuit shouldn’t remain open forever.
Eventually, the downstream service may recover.
After a configured period, the circuit breaker enters the half-open state.
This is the cautious state.
Instead of immediately restoring all traffic, the circuit allows a limited number of requests through.
Think of someone opening a door slightly after hearing a suspicious noise.
If those requests succeed, the dependency appears healthy again.
The circuit returns to closed.
Normal traffic resumes.
If those test requests fail, the circuit opens again.
The dependency gets more recovery time.
This creates a controlled path back to normal operation.
Fallback Behavior Makes Failure Less Painful
Opening the circuit does not always mean showing the user an error.
Sometimes the system has a useful fallback.
If a recommendation service is unavailable, the website might display popular products instead.
If a live pricing service fails, perhaps cached prices can temporarily be shown where business rules allow it.
If a notification service is unavailable, the message might be queued for later processing.
Fallback behavior depends heavily on the business operation.
A payment system should not casually invent a successful transaction because its dependency is unavailable.
Sometimes the correct fallback is simply:
“Payment is temporarily unavailable. Please try again later.”
Resilience doesn’t mean pretending nothing failed.
It means failing in a controlled way.
Fault Tolerance Isn’t the Same as Preventing Failure
This distinction matters.
A fault-tolerant system isn’t a system where nothing goes wrong.
It is a system designed so that something going wrong doesn’t automatically destroy everything around it.
Circuit breakers contribute to that resilience by limiting the impact of dependency failures.
They work alongside other patterns and controls such as:
timeouts → retries → circuit breakers → fallbacks → monitoring
Each handles a different part of failure.
Timeouts prevent endless waiting.
Retries handle temporary problems.
Circuit breakers stop repeated calls to unhealthy dependencies.
Fallbacks provide alternative behavior.
Monitoring tells humans what is actually happening.
The strength comes from how those pieces work together.
Recovery Should Be Designed, Not Hoped For
One of the most useful things about the circuit breaker pattern is that it treats recovery as part of normal system behavior.
The system expects services to fail.
It expects them to recover.
And it has rules for both situations.
That mindset changes how distributed systems are designed.
Instead of asking:
“What happens if this service fails?”
The better question becomes:
“What should every dependent service do when this service eventually fails?”
That small change removes the assumption that availability is permanent.
Circuit Breakers Are Really About Protecting the Rest of the System
At first glance, the circuit breaker pattern looks like a mechanism for handling one unreliable service.
Its real value is larger.
It protects everything surrounding that service.
Without a circuit breaker, timeouts and aggressive retries can turn a local outage into a system-wide problem.
With one, the application can recognize a pattern of failure, stop unnecessary traffic, provide fallback behavior where appropriate, and cautiously test for recovery.
The cycle becomes:
closed → failures increase → threshold reached → open → recovery period → half-open → test → closed
That is a small state machine.
But it captures an important principle of resilient software.
Sometimes resilience comes from retrying.
Sometimes it comes from recovering.
And sometimes the smartest thing a distributed system can do is recognize that another service is struggling and simply leave it alone for a while.