Why Your Automation Broke at 2 AM (And How to Build One That Doesn’t)

Getting a workflow to run once, with clean test data, while you’re watching, is about 30% of the job. The other 70% is everything that happens when nobody’s watching. This is the part that separates an automation you trust from one you check every morning “just in case” — which is not automation at all.

Here’s what actually breaks in production, and what to do about each.

The five things that break

API rate limits. You process 500 records in a loop, the API allows 60 requests a minute, and the run dies at record 61. Fix: batch your items and add a small wait between batches. Slower is fine. Failed is not.

Timeouts. An external service is having a bad day and takes 40 seconds to respond instead of 400ms. Your workflow gives up. This is temporary and worth retrying.

Changed data shape. Someone renames a field in the CRM, or a customer leaves the phone number blank for the first time in a year, and a step that expected text gets nothing. This is the most common one by far, and retrying won’t help — the data is genuinely different now.

Expired credentials. OAuth tokens expire. API keys get rotated. Someone leaves the company and their connected account goes with them. The workflow doesn’t fail loudly; it just quietly stops working.

Partial runs. The workflow created the invoice, then failed before sending the email. You fix the bug, re-run it, and now there are two invoices.

Retry, but only for the right things

Retries fix temporary problems: timeouts, rate limits, a 503 from a service that’s rebooting. They don’t fix wrong data, bad credentials, or logic errors — retrying those just fails five times instead of once, and takes five times as long to tell you.

A useful rule of thumb, borrowed from how HTTP status codes are meant to be read:

Response	Meaning	Retry?
429	Rate limited	Yes — and honour the Retry-After header if there is one
500, 502, 503, 504	Their server is struggling	Yes
Timeout / connection reset	Network or load	Yes
400, 422	Your data is wrong	No — it will be wrong next time too
401, 403	Credential problem	No — alert a human
404	The record isn’t there	No (usually)

When you do retry, wait longer each time — 5s, 15s, 45s — rather than hammering immediately, and cap it at three attempts. Most automation platforms have this as a per-step setting; use it on the steps that call external services and leave it off everywhere else.

If you’re processing a batch, add a little randomness to those waits. Fifty items that all fail at once and all retry after exactly 5 seconds will hit the recovering API as a single wall of traffic and knock it over again. A random extra 0–3 seconds per item spreads the load and costs you nothing.

Idempotency — the concept worth learning properly

Idempotent means: running it twice produces the same result as running it once. This is the single most useful idea in reliable automation, and it’s simpler than it sounds.

The pattern is always the same. Every incoming item has something stable and unique about it — an order ID, an invoice number, a message ID. Before you act on it, check whether you’ve already handled that ID. Keep a small store of processed IDs (a sheet, a database table, a key-value store) and write to it after each successful run.

Now a re-run is safe. A duplicate webhook is safe. Fixing a bug and replaying yesterday’s failures is safe. Without this, every re-run is a gamble, and you’ll be reluctant to touch the workflow at all — which is how automations rot.

Two refinements worth knowing. First, where the destination app supports it, use its own version — most invoicing and payment APIs accept an idempotency key or external reference and will refuse to create a second record with the same one. That’s stronger than your own check, because it’s enforced at the place the damage would happen.

Second, if a run has several side effects, record progress between them rather than only at the end. A workflow that creates an invoice, charges a card and sends an email should know which of those three already happened, so a re-run resumes instead of restarting. Without that, “safe to re-run” is only true for single-step workflows.

Fail loudly

The dangerous failure isn’t the one that throws an error. It’s the one that silently does nothing for three weeks while everyone assumes the leads are being routed.

Set up a single error handler that catches failures from every workflow and sends one message somewhere a human will see it — Slack, email, whatever your team actually reads. That message needs four things:

• Which workflow failed
• Which item (the ID — not “an item failed”)
• What the error actually said
• A direct link to the failed run

Anything less and you’ll be digging through logs instead of fixing it.

Then add a heartbeat for anything critical. If the nightly sync hasn’t reported success by 7am, tell someone. Silence should be an alert, not a comfort. This is the check that catches the failure mode nothing else does: a workflow that stopped being triggered at all, and therefore never errors, because it never runs.

Don’t create an alert nobody reads

There’s a failure mode on the other side of this, and it arrives about a month in. If every transient timeout pings the channel, people mute the channel, and you’re back to silent failure with extra steps.

Alert only on failures that survived their retries. Group repeats — “37 items failed with the same error” is one message, not thirty-seven. And separate “someone must act now” from “worth a look this week”; the first goes to chat, the second to a daily digest.

Give failures somewhere to go

When an item fails permanently, it should land somewhere — not vanish into a log entry. Borrowing the term from message queues, this is a dead letter store: a table or sheet holding the failed item’s ID, its full original payload, the error, and a timestamp.

The value shows up on a bad day. When an API change breaks 200 records overnight, you fix the workflow, then replay the dead letter store — because you kept the original payloads, and because the workflow is idempotent. Without it, your recovery plan is asking the source system for a date range and hoping it can tell you exactly which records failed.

Review it weekly. A dead letter store nobody looks at is just a slower way of losing data.

Validate at the door

Add a check as the very first step: does this item have the fields I need, in roughly the shape I expect? If not, stop, log it, notify. Failing at step one with “customer email is missing” is a five-minute fix. Failing at step nine with a cryptic type error, after three side effects have already happened, is an afternoon.

Check the handful of fields you actually use, and check for the realistic problems: missing, empty string, wrong type, a number arriving as text, a date in an unexpected format. You are not writing a schema validator — you’re stopping the run before it does damage.

Credentials expire on a schedule; plan for it

This deserves its own habit because it’s the failure that always arrives at the worst moment and never announces itself.

Keep a short list of every credential a workflow uses: what it is, which account owns it, when it expires, and who to ask if it breaks. Put the renewal dates in a shared calendar with a reminder two weeks before. Use a service account rather than a personal one wherever the platform allows it — the most common version of this failure is an integration authenticated as an employee who has since left.

Before you turn it on

• Run it with an empty input. Does it exit cleanly or crash?
• Run it with a weird input — missing field, wrong type, an emoji in a name, a 4,000-character message, an apostrophe in a surname.
• Run it twice with the same input. Do you get duplicates?
• Kill it halfway. What state is the data in? Can you safely re-run?
• Check every credential’s expiry and put the renewal date in a calendar.
• Break something on purpose and confirm the alert actually arrives. An untested error handler is just a theory — and it is genuinely common for the alerting step itself to be misconfigured.
• Run it against ten times your normal volume once, to find the rate limit before your busiest day does.

Fifteen minutes of this saves the 2am message. And more importantly, it’s what makes the difference between a workflow your team relies on and one they’ve quietly gone back to doing by hand.

Automations need maintenance, and that’s normal

One honest expectation to set: a workflow is not a thing you build once. The tools it connects will change their APIs, your business will change its process, and volumes will grow. Budget a little time each month to look at the error log and the dead letter store, and treat a workflow that has needed no attention in a year with mild suspicion — it’s worth confirming it’s still running at all.

Before any of this matters, you need the right trigger — see webhook, poll or schedule, particularly the reconciliation-sweep pattern, which is the safety net for everything described here. And if you’re choosing your first project, the five-day audit is where to start.

Every automation I build includes this scaffolding as standard, plus 30 days of support after launch — see n8n workflow automation or how a project runs.