Reliability
Idempotency, retries, fallbacks and failure handling.
Articles in Reliability (page 1 of 2)
- A Postgres Job Queue for AI Workflows with SKIP LOCKED — Model calls are slow, costly and rate-limited, so they belong in a queue. If you already run Postgres you may not need a broker. This guide covers the table, SKIP LOCKED claims, leases, retries, idempotency, monitoring and when to outgrow it.
- Dates, Times and Time Zones in AI Automation: Bugs and Fixes — Reports that are off by a day, reminders at the wrong hour and duplicates at daylight-saving changes all come from the same handful of mistakes. This guide shows how to store, convert, parse and test dates and time zones correctly.
- API Pagination Patterns for Automation Builders: Offset, Cursor and Beyond — Most APIs return data in pages, and most automation bugs here are silent: missed rows, duplicates, or a loop that never ends. This guide covers each pagination style, safe loop design, rate limits, checkpoints and how to test.
- Idempotent AI Automation: How to Stop Duplicates, Retries and Replays from Doing Damage — A retry creates two CRM records and two welcome emails, and nothing was technically a bug. Idempotency is the design idea that prevents it. This guide covers where duplicates come from and the concrete patterns, code and tests that make workflows safe to repeat.
- When an AI Agent Has Too Many Tools: Fixing Tool-Selection Ambiguity — Give an agent three different ways to look up a customer -- by email, by ID, by name search -- and it will pick a different one each time for reasons that have nothing to do with which is actually best.
- When an AI Agent Calls a Tool That Doesn't Exist — A looping agent at least calls a real tool over and over. A more disorienting failure is an agent confidently invoking a function name that was never in its tool list at all.
- Designing an AI Automation as a Resumable State Machine Instead of a Linear Script — A linear script has no idea where it was when it crashed. An AI automation state machine persists its current state after every step, so a container restart resumes from the last completed state instead of re-running the whole thing or silently continuing from the wrong place.
- Rotating API Keys in AI Automation Pipelines Without Downtime — An API key that has been sitting unrotated for years is a bigger liability than most teams treat it as -- and the reason it stays unrotated is usually not oversight, it is fear of breaking something. A rotation process designed correctly removes that fear.
- Writing an Incident Response Runbook Specifically for AI Automation Outages — A maintenance runbook explains how a workflow works so the next person can modify it calmly. An incident response runbook is a different document entirely -- built to be read in the first five panicked minutes of an active outage, when nobody has time to understand the whole system first.
- Load Testing an AI Automation Before It Goes Live — An automation that handles ten requests smoothly in testing can behave completely differently at the volume real launch traffic actually brings -- provider rate limits appear, a queue starts backing up, and latency that was invisible at low volume suddenly is not. None of this shows up until someone actually tests at scale.
- Circuit Breakers for AI Automation: Stopping Calls to a Failing Service Before It Makes Things Worse — A single failing LLM call is a nuisance. An automation that keeps retrying that same failing call, at volume, while the provider is degraded, turns a nuisance into a pile-up -- slower failures, wasted spend, and a queue that never drains. A circuit breaker exists to stop that from happening.
- Canary Releases for AI Automation: Rolling Out a New Prompt or Model to a Fraction of Traffic First — A new prompt version or a model upgrade that looks better in testing can still fail in ways testing never surfaced -- an edge case in real traffic, a subtle regression on a specific input type. A canary release limits the damage to a small slice of traffic instead of everyone at once.
- Cost Anomaly Detection: Catching a Runaway AI Automation Bill Before the Invoice Arrives — A prompt change that accidentally triples token usage, a retry loop that never stops, an upstream data change that sends ten times the normal volume -- none of these show up until the monthly invoice, unless something is actively watching spend as it happens.
- Shadow Mode: Running a New AI Automation Alongside the Old Process Before Cutting Over — A dry run tests an automation against sample data in isolation. Shadow mode is a step further -- running the new automation against real, live traffic, silently, without ever acting on its output, so you can compare it against the current process before trusting it to actually make decisions.
- Graceful Degradation: What an AI Automation Should Do When the LLM Call Fails — The default behavior for a failed LLM call is usually "the whole automation stops." That is rarely the right answer -- most automations have a reasonable, lower-quality fallback available if someone actually designs for it instead of treating an LLM failure as an unrecoverable dead end.
- Disaster Recovery Planning for AI Automation Pipelines — Most disaster recovery planning assumes the disaster is infrastructure -- a server dies, a data center goes down. An AI automation pipeline has an additional failure mode traditional DR planning does not cover: the automation platform itself, or a critical LLM provider, becoming unavailable entirely.
- Setting SLAs for AI-Automated Business Processes: Latency, Accuracy, and Escalation — Most AI automations run without any defined service level at all -- no agreed latency, no agreed accuracy floor, no escalation path when either slips. A practical framework for setting SLAs on an automated process the way you would for any other production system.
- Automation Drift: Why a Workflow That Worked Perfectly Stops Working Months Later — No error, no crash, no code change — and the output is quietly wrong anyway. Here's what automation drift actually is and how to catch it before it costs you.
- Rolling Back an AI Automation Pipeline Version, Not Just a Prompt — A prompt rollback undoes the prompt. If the model, the logic, and the config all changed together, rolling back only the prompt can leave the pipeline in a state that never actually shipped.
- Avoiding Alert Fatigue When an AI Automation Fails Every Few Minutes — The second alert for the same failure isn't new information. By the fiftieth, the channel is muted — and the next genuinely different failure gets muted along with it.
- Upstream API Schema Drift: What It Silently Breaks in Your AI Automation — A version bump tells you an API changed. A silent field rename doesn't — and your automation keeps running, just on assumptions that stopped being true.
- Polling vs Webhooks: Getting Results Back From a Long-Running AI Automation Step — A step that takes ninety seconds needs a different result-retrieval design than one that takes ninety milliseconds. Polling and webhooks solve that differently, at different costs.
- Compensating Actions: What to Do When Step 3 of a 5-Step AI Automation Fails — A five-step automation that fails on step three doesn't leave you at step two — it leaves you with two completed side effects and no built-in way to undo them.
- Validating LLM Output Against a Schema Before It Reaches Production Data — A model returning valid JSON isn't the same as a model returning correct data. Schema validation catches the gap between the two before it becomes a production bug.