Lessons learned from operating high-throughput services in production.

Tooling is necessary but it is rarely the bottleneck; the bottleneck is the team’s shared mental model of how the system actually behaves under stress. Production data, sampled and anonymised, is worth a thousand synthetic benchmarks when you are tuning anything to do with Designing Resilient Laravel APIs at Scale. Documentation written next to the code, in the same change, is the only documentation that survives the next refactor; everything else slowly turns into folklore.

Why resilience matters

Whenever I onboard a new engineer, I now hand them a short reading list focused on the failure modes of Why resilience matters, because nothing else accelerates senior judgement faster. The best engineers I know spend a strange amount of time deleting code; every line removed is a line that cannot break, cannot confuse and cannot drift out of date. If your tests cannot tell you within five seconds whether a regression has hit Why resilience matters, your tests are not the bottleneck — your test design is. Tooling is necessary but it is rarely the bottleneck; the bottleneck is the team’s shared mental model of how the system actually behaves under stress.

Performance, security, and developer ergonomics tend to pull in different directions; Why resilience matters is one of those topics where you can usually only pick two without compromise. When you sit down to ship something around Why resilience matters, the first instinct is to dive into the code, but the most useful thirty minutes are usually spent in front of a notebook. The best engineers I know spend a strange amount of time deleting code; every line removed is a line that cannot break, cannot confuse and cannot drift out of date.

The shape of a resilient request

There is no replacement for sitting next to a real user and watching them use the feature; everything else is an indirect signal at best. Performance, security, and developer ergonomics tend to pull in different directions; The shape of a resilient request is one of those topics where you can usually only pick two without compromise. Pay attention to the way your team talks about The shape of a resilient request in design reviews — vague verbs and passive voice almost always signal a hidden assumption that will bite later.

Pay attention to the way your team talks about The shape of a resilient request in design reviews — vague verbs and passive voice almost always signal a hidden assumption that will bite later. Whenever I onboard a new engineer, I now hand them a short reading list focused on the failure modes of The shape of a resilient request, because nothing else accelerates senior judgement faster. When you can sketch the failure mode of a feature on a napkin, you understand it; when you can only sketch the happy path, you do not, and The shape of a resilient request is no exception.

  • Start by asking what you actually want to measure about The shape of a resilient request

  • Make the simplest version work end-to-end before you tune it

  • Write one test that fails, then make it pass — no bigger steps

  • Document the surprises, not the obvious — readers reward you for it

  • Review the assumptions every quarter, especially the ones you didn’t question

Idempotency as a contract

I have learned to ask “what would have to be true for this to be wrong?” before I ship anything that touches Idempotency as a contract, and it has saved me embarrassment many times. There is a recurring pattern across every team I have worked with: the people who ship reliably are the ones who treat boring details — naming, timeouts, error paths — as a craft rather than a chore. When you sit down to ship something around Idempotency as a contract, the first instinct is to dive into the code, but the most useful thirty minutes are usually spent in front of a notebook.

We made every classic mistake with Idempotency as a contract — silent retries, retries with backoff but no jitter, retries that bypassed idempotency — and each one cost us a postmortem. When in doubt, write the smaller change first, ship it behind a flag, and let production teach you what the design should actually be. Reading other people’s production code, especially open source projects that handle Idempotency as a contract at scale, has done more for my judgement than any course I have ever taken.

Code is read far more often than it is written, and Idempotency as a contract is no exception — favour clarity over cleverness.

Timeouts, retries and backoff

When in doubt, write the smaller change first, ship it behind a flag, and let production teach you what the design should actually be. I am increasingly convinced that the right metric for engineering quality is mean time to recover, not bug count or velocity, and Timeouts, retries and backoff is a great place to apply that lens. Tooling is necessary but it is rarely the bottleneck; the bottleneck is the team’s shared mental model of how the system actually behaves under stress. Reading other people’s production code, especially open source projects that handle Timeouts, retries and backoff at scale, has done more for my judgement than any course I have ever taken.

Production data, sampled and anonymised, is worth a thousand synthetic benchmarks when you are tuning anything to do with Timeouts, retries and backoff. Most of the systems I admire are the ones that do less than they could and do it well — the systems that try to do everything tend to do everything poorly. If your tests cannot tell you within five seconds whether a regression has hit Timeouts, retries and backoff, your tests are not the bottleneck — your test design is. Reading other people’s production code, especially open source projects that handle Timeouts, retries and backoff at scale, has done more for my judgement than any course I have ever taken.

class CreateOrderHandler {
    public function __invoke(CreateOrder $command): Order {
        return DB::transaction(function () use ($command) {
            $idempotency = IdempotencyKey::for($command->key);
            if ($result = $idempotency->cachedResult()) return $result;

            $order = Order::create($command->toArray());
            $idempotency->capture($order);
            return $order;
        });
    }
}

Performance, security, and developer ergonomics tend to pull in different directions; Timeouts, retries and backoff is one of those topics where you can usually only pick two without compromise. Tooling is necessary but it is rarely the bottleneck; the bottleneck is the team’s shared mental model of how the system actually behaves under stress. Communicating tradeoffs honestly upward is part of the job — there is no version of Timeouts, retries and backoff where every constraint can be satisfied at the same time.

Circuit breakers in PHP

Tooling is necessary but it is rarely the bottleneck; the bottleneck is the team’s shared mental model of how the system actually behaves under stress. There is a recurring pattern across every team I have worked with: the people who ship reliably are the ones who treat boring details — naming, timeouts, error paths — as a craft rather than a chore. Whenever I onboard a new engineer, I now hand them a short reading list focused on the failure modes of Circuit breakers in PHP, because nothing else accelerates senior judgement faster. It is tempting to reach for a framework feature when a five-line plain function would do, and Circuit breakers in PHP is the area where that tradeoff bites hardest in year three of the codebase.

Tooling is necessary but it is rarely the bottleneck; the bottleneck is the team’s shared mental model of how the system actually behaves under stress. The best engineers I know spend a strange amount of time deleting code; every line removed is a line that cannot break, cannot confuse and cannot drift out of date. Communicating tradeoffs honestly upward is part of the job — there is no version of Circuit breakers in PHP where every constraint can be satisfied at the same time. There is a recurring pattern across every team I have worked with: the people who ship reliably are the ones who treat boring details — naming, timeouts, error paths — as a craft rather than a chore.

Where to take this next

We made every classic mistake with Designing Resilient Laravel APIs at Scale — silent retries, retries with backoff but no jitter, retries that bypassed idempotency — and each one cost us a postmortem. Operability is a feature, and the cheapest place to bake it in is at design time; it gets exponentially more expensive each release after that. There is no replacement for sitting next to a real user and watching them use the feature; everything else is an indirect signal at best.