# When Retrying Makes Things Worse

Whenever a request fails, one of the first things that comes to our mind is to retry it.

The request failed? Try again.

The API timed out? Send it again.

Maybe it will work this time.

And honestly, this makes sense. Sometimes failures are temporary. There might be a network issue, a brief service disruption, or something that went wrong for just a few milliseconds. Retrying the request might be enough to recover without anyone having to intervene.

I used to think of retries as a mechanism to fix a broken service. If something didn't work, why not try it again?

But while exploring timeouts, retries, and cascading failures, I started looking at them differently.

What if retrying doesn't fix the problem but actually makes it worse?

Think about it. If a service is already struggling to process requests, sending it even more requests might be the last thing it needs.

What surprised me most was that a timeout doesn't necessarily mean the operation failed. The server might have completed the payment, but the response never reached the client.

I initially thought retrying was simply a way to recover from failure. But as I explored the problem, I realised that retrying can sometimes repeat an operation or add more load to a service that's already struggling.

That changed how I think about retries. Instead of asking only whether we should retry, we also need to ask whether it's safe to do so and what happens if the next attempt fails too.

Let's start with something simpler.

## What does a timeout actually mean?

Imagine that you're making a payment of ₹500.

You click the Pay button, and your application sends a request to the server. The server processes the payment successfully, but for some reason, the response never reaches your application.

After waiting for some time, the client times out.

Now, what do we know?

The client didn't receive a response within the expected time. But does that mean the payment failed?

No.

The payment might have failed. The server might still be processing it. Or the payment might have succeeded, and only the response got lost.

The client doesn't know which of these happened.

This is where things become interesting.

If the client assumes that the payment failed and sends the same request again, the server might process it as a completely new payment.

And now the customer gets charged twice.

The server might have processed the first request correctly. The problem is that the client doesn't know what happened.

![](https://cdn.hashnode.com/uploads/covers/624ea8510667a860b2552e4f/527cb312-ba6c-4c82-a6d2-75e9dc20b78e.png align="center")

A timeout doesn't necessarily mean that the operation failed. It means that the client didn't receive a response in time.

This is an important distinction because retries are often introduced precisely when we don't know what happened to the original request.

So, how do we make sure that retrying doesn't repeat an operation that has already succeeded?

Lets try to understand this through a small coding experiment.

## Making the same request safe to retry

I started with a very basic Python function to process a payment.

```python
payment = 0

def process_payment(amount):
    global payment
    payment += amount
    print(payment)
```

Nothing complicated here. We pass an amount, and the function adds it to the total.

But what happens if the same request is processed twice?

```python
process_payment(500)
process_payment(500)
```

The total becomes ₹1,000.

The function is doing exactly what we asked it to do. We called it twice, and it added ₹500 twice.

But that's not what we wanted. We intended to make one payment of ₹500.

The problem is that our function doesn't know whether the request is new or whether it's a retry of an operation that has already happened.

### Introducing idempotency

To solve this, I introduced a payment ID.

The idea is simple. Every logical payment gets an identifier. If the same request comes again with the same ID, the system should recognise it instead of processing the payment again.

I first used an in-memory set to keep track of the payment IDs that had already been processed.

```python
payment = 0
ids = set()

def process_payment(amount, payment_id):
    global payment

    if payment_id in ids:
        print("Duplicate payment detected.")
        return

    ids.add(payment_id)
    payment += amount

    print("Payment processed:", amount)
```

Now, if we call the function twice with the same payment ID, the second call doesn't increase the total again.

This is the basic idea behind idempotency.

![](https://cdn.hashnode.com/uploads/covers/624ea8510667a860b2552e4f/20c01c29-a9e4-4c0e-b31c-8ed358b24818.png align="center")

In simple terms, an operation is idempotent when repeating the same logical operation doesn't repeat its effect.

For a payment API, we want the same payment request to be safe to retry. If the payment has already succeeded, the next request should return the original result instead of creating another charge.

Of course, this first implementation has limitations. The set exists only in memory, so it disappears when the application restarts. It also doesn't safely coordinate concurrent requests.

Before moving to a database, I wanted to see what this problem looks like when a payment succeeds but the response is lost.

### Simulating a double charge

Here's a small experiment that simulates a server processing a payment and then losing the response.

The client times out and retries the same payment.

```python
class PaymentServer:
    def __init__(self, idempotent=False):
        self.charges = []
        self.results = {}
        self.idempotent = idempotent
        self.drop_next_response = True

    def process_payment(self, payment_id, amount):
        if self.idempotent and payment_id in self.results:
            return self.results[payment_id]

        # Simulate processing the payment.
        self.charges.append(amount)
        result = f"Payment successful: ₹{amount}"

        if self.idempotent:
            self.results[payment_id] = result

        # The payment succeeds, but the response is lost.
        if self.drop_next_response:
            self.drop_next_response = False
            raise TimeoutError("Response timed out")

        return result


def make_payment(server):
    for _ in range(2):
        try:
            return server.process_payment("pay-123", 500)
        except TimeoutError:
            print("Timeout. Retrying...")

    return "Request failed"


for idempotent in (False, True):
    server = PaymentServer(idempotent=idempotent)
    make_payment(server)

    print(f"Idempotency enabled: {idempotent}")
    print("Charges:", server.charges)
    print("Total:", sum(server.charges))
    print()
```

The output looks like this:

```text
Timeout. Retrying...
Idempotency enabled: False
Charges: [500, 500]
Total: 1000

Timeout. Retrying...
Idempotency enabled: True
Charges: [500]
Total: 500
```

In the first case, the payment succeeds, but the client never receives the response. When it retries, the server processes the payment again.

In the second case, the server recognises the same payment ID and returns the original result.

This is a simplified demonstration using in-memory state. It helps explain the idea, but a real payment system needs durable idempotency records and appropriate handling of external payment-provider responses.

### Moving from memory to a database

I wanted to improve the implementation so that duplicate payment IDs could be prevented at the database level.

I moved the experiment to SQLite and used a primary key to enforce uniqueness.

```sql
CREATE TABLE payments (
    payment_id TEXT PRIMARY KEY,
    amount INTEGER NOT NULL,
    status TEXT NOT NULL DEFAULT 'processed'
);
```

Now, the database prevents two payment records from having the same payment ID.

But uniqueness alone isn't enough. We also need to decide what should happen when a request comes in with an ID that already exists.

If the same ID and amount arrive again, the application can return the existing result. If the same ID arrives with a different amount, that should be rejected as a conflict rather than silently changing the original payment.

Here's the implementation:

```python
import sqlite3

conn = sqlite3.connect("payments.db")

conn.execute("""
    CREATE TABLE IF NOT EXISTS payments (
        payment_id TEXT PRIMARY KEY,
        amount INTEGER NOT NULL,
        status TEXT NOT NULL DEFAULT 'processed'
    )
""")

def process_payment(amount, payment_id):
    try:
        with conn:
            conn.execute(
                """
                INSERT INTO payments (payment_id, amount, status)
                VALUES (?, ?, 'processed')
                """,
                (payment_id, amount),
            )

        return "processed"

    except sqlite3.IntegrityError:
        row = conn.execute(
            """
            SELECT amount, status
            FROM payments
            WHERE payment_id = ?
            """,
            (payment_id,),
        ).fetchone()

        # This was an integrity error unrelated
        # to an existing payment ID.
        if row is None:
            raise

        # Same payment ID and amount:
        # return the existing status.
        if row[0] == amount:
            return row[1]

        # Same ID with a different amount:
        # reject the request.
        raise ValueError(
            "Conflict: payment_id reused "
            "with a different amount"
        )
```

This experiment helped me understand how we can add guardrails to a seemingly simple function.

We start with a basic solution, discover an edge case, and improve the implementation to handle it.

But there's an important limitation here. Preventing duplicate records in SQLite doesn't, by itself, guarantee that an external payment provider won't charge a customer twice. A real payment system needs to handle external provider responses, durable operation states, and ambiguous outcomes. It also needs to coordinate concurrent requests safely.

Still, this was a useful starting point.

And it raised a bigger question.

What if the problem isn't just that a request might repeat an operation, but that retries themselves start overwhelming the system?

**Idempotency helps with the first problem. It doesn't solve the second.**

## When retries start creating more work

Let's move from one function to a system where multiple services depend on each other.

Imagine three services:

*   **Service A** receives a request from the client.
    
*   **Service B** handles the business logic.
    
*   **Service C** accesses a database or another downstream dependency.
    

The request flows from A to B and then from B to C.

Under normal conditions, everything works fine.

But now imagine that Service C becomes slow.

Service B sends a request to C and waits for the response. C takes too long, so B times out and retries.

Meanwhile, Service A is waiting for B. B isn't responding quickly enough, so A also retries its request.

Now we have retries happening at multiple layers.

Service C was already struggling to process requests, and now it's receiving even more traffic because the services depending on it keep retrying.

As the load increases, C might become slower. More requests time out, which trigger even more retries.

And the cycle continues.

This is the basic mechanism behind a cascading failure: an initial problem creates additional failures, and those failures make the original problem worse.

The retry mechanism that was supposed to help the system recover can actually prevent it from recovering.

### How quickly can retries multiply?

Let's take a simple example.

![](https://cdn.hashnode.com/uploads/covers/624ea8510667a860b2552e4f/554e4d60-7a38-4c1f-ad6a-f559535b61be.png align="center")

Service A calls Service B. Service B calls Service C.

Suppose A makes up to three attempts to call B, and each attempt can cause B to make up to three attempts to call C.

In the worst case, a single original request can generate:

3 X 3 = 9

attempts against Service C.

Add a third layer with the same retry policy, and the number can become:

3 X 3 X 3 = 27

Now imagine hundreds or thousands of clients doing this at the same time.

This is a simplified example. The actual number depends on which attempts fail, how the retry policies are configured, and whether requests are cancelled or rejected. But it illustrates how retries at multiple layers can multiply the work sent to a downstream service.

This overload can create a positive feedback loop that leads to cascading failures. The system doesn't necessarily need to be completely down for this to begin. It might simply be unable to keep up with the work arriving at it.

And that's why we can't think about retries only in the context of one service.

We need to understand how our services are connected and how their retry policies interact with each other.

### Coordinating retries across services

Each service can have a perfectly reasonable retry policy on its own. Service A retries a failed request to B. B retries a failed request to C. Both have a limit of three attempts.

But those limits apply independently.

A doesn't know how many attempts B has already made against C. And B doesn't know whether A has already sent the same logical request multiple times.

This is how individually reasonable retry policies can combine into a system-wide problem.

One way to address this is to designate a single layer to retry a particular downstream operation. In our example, B could be responsible for retrying requests to C, while A does not retry B's explicit overload response.

Another guardrail is a **retry budget**: a limit on how many additional attempts a service or client can make within a given period. For example, a service might allow at most 60 retries per minute. Once that budget is exhausted, it stops retrying and fails fast instead of adding more load.

These two controls solve related but different problems. Limiting retries to one layer prevents independent retry policies from multiplying attempts. A retry budget limits the total retry traffic.

But how much difference does this make in practice? Let's test it.

### Testing the difference with a small simulation

Let's compare two scenarios using a simplified model in which each retrying layer allows up to three attempts.

In the first scenario, three layers retry independently. In the second, only one layer retries the downstream operation.

```python
MAX_ATTEMPTS = 3


def simulate_retries(retry_at_every_layer=True):
    attempts_reaching_dependency = 0

    if retry_at_every_layer:
        # Three nested layers, each allowing 3 attempts.
        for _ in range(MAX_ATTEMPTS):
            for _ in range(MAX_ATTEMPTS):
                for _ in range(MAX_ATTEMPTS):
                    attempts_reaching_dependency += 1
    else:
        # Only one layer retries the downstream operation.
        for _ in range(MAX_ATTEMPTS):
            attempts_reaching_dependency += 1

    return attempts_reaching_dependency


print(
    "Retries at every layer:",
    simulate_retries(True),
)

print(
    "Retries at one layer only:",
    simulate_retries(False),
)
```

The output:

```text
Retries at every layer: 27
Retries at one layer only: 3
```

The difference is significant: in this simplified example, the downstream dependency receives 27 attempts instead of 3 for the same original operation.

This is an illustrative model, not a full distributed-system simulation. It assumes every attempt reaches the next layer and that every retry is triggered. Real systems behave differently depending on timeouts, cancellation, error types, and retry policies. It demonstrates the arithmetic, not the effects of actual network failures or an enforced retry budget. Don't present it as a production benchmark.

The takeaway is that retry limits alone aren't enough. We also need to coordinate retries across service boundaries.

## Retries are useful, but they need guardrails

The solution isn't to stop retrying everything.

Retries are useful. They can help recover from temporary network failures and other transient errors.

The problem is assuming that retries are always the right response to a failure.

Think about it this way: if a service is already overloaded, sending it more requests might make the situation worse.

We need to understand why the request failed before deciding what to do next.

Here are some techniques that can help.

![](https://cdn.hashnode.com/uploads/covers/624ea8510667a860b2552e4f/544d1f76-816e-43d5-98ee-a2dcd6f48a3b.png align="center")

### 1\. Timeouts and deadlines

We need to decide how long a client should wait for a response.

Without appropriate timeouts, requests might hold on to connections and resources for too long. But if timeouts are too aggressive, we might retry requests that would have succeeded if we had waited a little longer.

Timeouts should be chosen based on the expected latency of the service and the overall time budget of the operation.

We should also remember that a client timing out doesn't automatically cancel the work happening on the server. Where possible, cancellation should be propagated so unnecessary work can stop.

### 2\. Bounded retries

We shouldn't retry forever.

If a request keeps failing, repeatedly sending it may consume resources without improving the chance of success.

A retry policy should have a maximum number of attempts and should distinguish between failures that might be temporary and failures that won't be fixed by sending the same request again.

Retrying a temporary connection failure might make sense. Retrying an invalid request without changing it usually doesn't.

There is another important consideration: retries should be bounded across the system, not just within an individual request.

A retry budget can limit how many additional requests a service sends over a given period. When the budget is exhausted, the service stops retrying and propagates the failure.

We should also avoid retrying the same operation independently at multiple layers whenever possible. If B is responsible for retrying C, A should respect B's explicit overload response rather than starting another round of retries.

This helps prevent a struggling dependency from receiving increasingly more work precisely when it has the least capacity to handle it.

### 3\. Exponential backoff

Instead of retrying immediately, we can wait before trying again.

With exponential backoff, the delay increases after successive failures, usually with a maximum cap.

For example:

| Retry attempt | Approximate delay |
| --- | --- |
| First retry | 200 ms |
| Second retry | 400 ms |
| Third retry | 800 ms |

These numbers are just an illustration, not universal defaults.

The idea is to give the downstream service time to recover instead of immediately adding more work.

### 4\. Jitter

But even exponential backoff has a problem.

Imagine thousands of clients experience a failure at the same time. If they all wait for exactly 200 milliseconds, they might all retry together.

This creates another burst of traffic.

Jitter adds randomness to retry delays. Instead of every client retrying at the same time, their attempts are spread out.

Exponential backoff increases the delay between retries, while jitter helps prevent multiple clients from retrying together and overwhelming the service.

### 5\. Circuit breakers

Sometimes, we need to stop sending requests to a struggling dependency temporarily.

A circuit breaker can help with this.

When failures cross a configured threshold, the circuit breaker opens and stops sending normal requests to the dependency. After a cooldown period, it allows a limited number of test requests to check whether the dependency has recovered.

If the service is healthy again, normal traffic can resume.

The goal isn't to stop calling the service permanently. It's to avoid continuously sending requests while the dependency is struggling, then cautiously resume requests to check whether it has recovered.

### 6\. Load shedding

Sometimes, the best response is to reduce the amount of work the system is doing.

We might reject some low-priority requests, pause nonessential batch jobs, or return a simpler response that requires fewer resources.

Restarting a server might sometimes be necessary, but if the underlying problem is that too much traffic keeps arriving, restarting alone won't solve it. The same overload can happen again.

The goal is to give the system enough breathing room to recover.

These techniques solve different problems, and they may be used together. The important thing is to choose them based on how the system behaves under failure, rather than adding retries and assuming everything will work out.

## Conclusion

Retries are an important tool for building reliable systems. They help us recover from temporary failures and make applications more resilient.

But retries are not a universal solution.

They can repeat operations that have already succeeded, increase the load on an already struggling service, and contribute to cascading failures when multiple services retry independently.

The lesson I take away is simple.

Understand the requirement. Understand how your services depend on one another. Think about what happens when a request fails, when its response is lost, and when the same operation is attempted again.

And then put the right guardrails in place.

I think of retries like medicine. The right amount can help you recover, but the wrong amount can make things worse.

Because sometimes, the thing we introduce to fix a problem can become part of the problem itself.

Retries can help a system recover. But without understanding the system and the failure, retrying can make things worse.

## References

*   [Google SRE Book: Addressing Cascading Failures](https://sre.google/sre-book/addressing-cascading-failures/)
    
*   [AWS Builders' Library: Timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/)
    

*The payment code in this article is a learning experiment intended to illustrate idempotency and failure handling. It is not a production-ready payment implementation.*