The problems
Durable-execution engines like Temporal already handle the failures — crashes, retries, lost messages, non-deterministic replay. They also do something most architectures only promise: they genuinely isolate your business logic from the outside world. Clean Architecture, hexagonal ports, and their kin lean on convention — a boundary nothing enforces, so I/O leaks across it the moment discipline slips — whereas Temporal makes the split structural: workflow code is kept deterministic, and every effect on the outside world has to go through an activity. The abstraction doesn’t leak.
But none of this is free. The promise is real, and it is expensive: workers to run and tune, the determinism constraints to respect, and a whole web of signals, timers, and handlers to get right — operating durable workflows is anything but trivial, and it can be genuinely painful. And increasingly this is the substrate for AI agentic systems, where the agent decides at runtime which handlers, branches, and signals fire — driving the very interleavings these bugs hide in.
You always own the fix — the platform never applies one for you — but the bugs split into two families by where the answer comes from: the ones the platform already has a tool or pattern for, and the ones no generic tool can even find. And even after you have paid the full price, its behavior in production can still cost you sleep.
Family one — the platform already has an answer
Section titled “Family one — the platform already has an answer”Generic, well-known Temporal hazards the platform and its tooling can solve — but only with your guidance, expertise, and close monitoring; left on defaults, they won’t solve themselves. Increasingly, though, that expertise can be automated: in 2026 a Temporal-savvy AI coding agent — with dedicated skills and MCP tooling — can likely handle most of these, since the answers are generic and well-known.
- Replay non-determinism. Recorded histories fail to replay after a code change — an unversioned refactor or module-global mutable state can leave most of a long-running workflow’s in-flight executions unable to resume. Among the most damaging failures in practice, and preventable with
patched()gates, worker versioning, and replay CI. But the safety net is only as good as the histories you feed it: it catches nothing you haven’t captured, so someone has to periodically export fresh histories into the repo, judge which ones give real coverage, decide how many and how often — and simply remember to keep doing it. The platform ships the mechanism; keeping it meaningful is an unassisted, forever chore that is easy to let slip. - Missing or misconfigured timeouts and retries. Activities with no explicit timeout or retry policy — or a plausible-but-wrong one: a timeout under the activity’s real p99, retries that endlessly re-run a poison message, a heartbeat that never fires. The platform gives you the knobs but no way to tell your values are right for this workflow. Is one retry enough for this activity? Two? Do you just reach for three everywhere by default and hope? There is no principled way to answer under a tight deadline — so the number gets picked by guesswork, and catching a bad config before production is developer discipline and experience, not a Temporal feature. And even a well-chosen timeout hides a trap: when an activity overruns, Temporal just starts a fresh attempt while the original may still be running — terminating that orphaned work is something only discipline makes you build, easy to forget, and genuinely hard to test.
- Idempotent activities. Because Temporal runs activities at least once — retries, timeouts, and orphaned attempts all mean one can fire more than once — every activity with a side effect has to be idempotent. Nothing enforces it: Temporal happily runs a non-idempotent activity, and the duplicate only bites in production. Do you remember to make each one idempotent, know the right way to do it (idempotency keys, upserts, dedup), and actually test the double-execution path? All on you.
- Heartbeats. Long-running activities need heartbeats so a dead worker gets noticed instead of silently hanging — but at what interval? Too rare and a stall goes undetected for ages; too frequent and you pile on load. And you have to stop heartbeating and react to cancellation correctly on every error and timeout path. The platform gives you the primitive; the right cadence and the cleanup logic are yours to invent, remember, and get right — with nothing telling you when you’ve got them wrong.
- Unbounded history growth. A long-lived workflow with no
continueAsNew, accumulating state until it hits history limits — and well before that, the oversized history crashes the Web UI, runs up billable actions, and becomes impossible to load and investigate. - Worker and queue capacity. Under saturation the symptoms look like bugs but aren’t: tasks pile up in the queue, schedule-to-start timeouts fire, activities and workflows stall, and end users see latency spikes and intermittent timeout errors that are maddening to trace back to their real cause. The platform exposes the signals — task-queue backlog, schedule-to-start latency — but noticing them and tuning worker capacity is on you.
- Error delivery and observability gaps. Failures that never surface to the client or a dashboard. Temporal emits raw metrics and a Web UI, but no ready-made dashboards for your workflow’s health — you build them yourself, and knowing what to watch takes experience you may not have yet. It also lies to you by omission: the visibility store that backs list, search, and the Web UI filters updates asynchronously, so it lags real state — a workflow you just started or completed may not show up for a while, which makes both live dashboards and integration tests that query it flaky and timing-dependent. And even done well, monitoring only detects the symptoms of the deeper bugs — a stuck run shows as “running, no progress” — it never fixes the logic. A dashboard is not a solution.
Family two — the bugs no generic tool can find
Section titled “Family two — the bugs no generic tool can find”These live in the workflow’s own logic — its interleavings and the coordination it hand-rolls — and they surface as violations of invariants only this workflow defines. Generic tooling can’t catch them, because catching them means reasoning about this workflow. And here even an AI coding agent with the right skills goes only so far: the interleavings explode combinatorially, and that inherent complexity can’t be reasoned away.
- Completion before in-flight work finishes. A completion race can hide for months before anyone notices: the run finishes before an in-flight update resolves, and the caller silently never receives its result.
allHandlersFinished()guards handlers but not the fire-and-forget promises they spawned. - Detached write outlives the run. A fire-and-forget write lands after the workflow has already exited, with no workflow left to observe it.
- Rejected floating promise. An unawaited fire-and-forget promise rejects mid-run and takes the whole workflow task down, because nothing local caught it.
- Terminal op before preconditions settle. A submit or export fires while a prerequisite job is still mid-flight, so it runs on half-built state.
- Initialization race. A signal lands before setup finishes and mutates state that isn’t there yet.
- Never-resolving await. The run blocks forever on a
condition, activity, or coordinator that never resolves — e.g. a slot grant that never returns after a cancel. Temporal sees a healthy workflow; it is simply waiting. - Lost wakeup. A signal resets the “has work” flag a beat before the check that would have consumed it, so the work is never picked up.
- Delete-when-empty against an external store. A guard reads the pending items in memory but not the in-flight ones, so it deletes a document while an upload is still committing to the store. One of the most likely invariants to break, and the kind testing never catches — no run happens to hit the exact ordering.
- Stale read of a detached writer’s value. Check-then-act on a ref that a background process writes with no coordination, so the read misses or supersedes a write it needed.
- Overlapping jobs, last-writer-wins. Two regenerations race; the one over the smaller input finishes first and its result wins.
- Config toggle races a captured value. A flag flip doesn’t reach a process that already captured the old value — a race that shows up under real production timing, not just in theory.
- One-shot budget burned by a cancel. A “run exactly once” counter is incremented before its callback and never refunded on cancel, so “once” quietly becomes “never.”
- Home-grown coordinator bugs. A scheduler that drops a request when the grantee already finished; a back-off whose minimum wait computes to zero and busy-loops; a mutex built as an unbounded promise chain.
- Config that’s only wrong given the workflow’s logic. A retry policy that is fine in isolation but re-runs a non-idempotent activity and duplicates its side effect; a timeout tuned on its own that violates an ordering the workflow silently assumed. Whether a config is safe depends on this workflow’s own semantics — which the platform can’t check for you.
- Knowing what to observe requires knowing the workflow. A generic dashboard shows latency and task backlog, never “this run is silently wedged” or “this invariant just broke” — those are defined only by the workflow’s own logic. Instrumenting the signals that actually matter means first knowing where this workflow can go wrong, which is exactly the reasoning no generic tool does for you.
- Logic that drives runaway growth. History rarely balloons on its own — a loop that never converges, a self-rescheduling timer, or a signal storm the workflow keeps answering pushes it there.
continueAsNewonly caps the symptom; the underlying loophole is a workflow-logic bug that surfaces at scale in production. Predicting it beforehand means proving this workflow’s loops actually terminate — which is on the developer, not a generic check.
The shape underneath
Section titled “The shape underneath”Step back and the whole catalog classifies on two axes. The first is where the answer comes from — the two families above: off-the-shelf platform tooling, or reasoning about this workflow. The second orders family two, which otherwise looks like a dozen unrelated bugs: it is really one root, four shapes, and one trap.
The root — the interleaving surface. Temporal runs every signal and update handler on one coroutine that interleaves at each await; the only atomic regions are the await-free runs between them. Logical concurrency is handlers × await points, and most of the bugs above are just two of them parked in an order nobody pictured. And you don’t control the timing: signals arrive asynchronously, with no synchronous confirmation the workflow acted on one, so the exact ordering that breaks is something you can neither force nor observe — which is what makes these races so hard to reproduce and nearly impossible to test. This is also why the fix can’t be generic — the blast radius is defined by this workflow’s handlers and state, which is the same reason picking a safe config or the right thing to monitor needs workflow-specific reasoning, not a platform default.
The failures then sort into four shapes:
- Completion and lifecycle — the run ends or transitions at the wrong moment: completion before in-flight work, detached write outlives the run, rejected floating promise, terminal op before preconditions, initialization race.
- Liveness — the run silently stops advancing: never-resolving await, lost wakeup, logic that drives runaway growth.
- Shared-state races — handlers read-then-write across an
await, and the truth may live in a store the workflow doesn’t mirror: delete-when-empty, stale detached read, overlapping jobs, config toggle races a captured value. - Home-grown coordination — bespoke rate limiters and schedulers carrying their own algorithm bugs: one-shot budget burned by a cancel, coordinator, back-off, and mutex bugs.
The trap — correctness is emergent, not structural. Often no single component owns the invariant. “Complete exactly once” holds only because every handler repeats the same guard-then-set discipline; liveness holds only if every flag reset is followed by a re-check. A behavior-preserving refactor — reorder two lines, add an await — silently breaks it, which is why the same class of bug returns with each new feature. It is precisely what a model checker can catch and a linter cannot.
The boundary. One class sits outside both families: the quality of what an activity actually produces — a bad AI output, an unhandled input edge case. That is business correctness, not concurrency; input validation and property tests own it, and it is out of scope here.