How recovery works
A worker claims one agent, runs one step (a model call or a batch of tool calls), saves it, and releases the agent. A model response is saved with its tool calls and timeline event in one transaction, a tool result with its event, and event webhooks with the event they describe. Streamed text deltas are sent live and never saved; a client that reconnects reads the finished response from the event log. Each claim is a lease that the worker renews about every 30 seconds. If a worker crashes, its lease expires, the maintenance service releases the agent, and another worker claims it, usually within 60 to 90 seconds. A worker that has lost its lease can’t save anything afterward, so a slow or partitioned worker can’t overwrite the work of the worker that replaced it. Self-hosted deployments run the maintenance service alongside the API and workers. Planned shutdowns and deploys don’t wait for a lease to expire. A stopping worker stops claiming new work, gives the work in progressOMNARA_WORKER_DRAIN_TIMEOUT (20 seconds by default) to finish, and then releases its agents to other workers.
Model calls
A model call cut off by a crash, a shutdown, or a lost lease is retried from saved state. Partial output from the interrupted attempt isn’t kept, and a steering message or config change that arrived in the meantime applies to the retry. Rate limits, timeouts, and provider outages are retried with exponential backoff, honoringRetry-After, up to eight times. Authentication, billing, and invalid-request errors end the turn right away, with the error in the agent’s history. When the context is too long for the model, Omnara compacts it automatically.
As with any system that retries an interrupted request, the provider can bill for an attempt that finished on its side before the worker died.
Tools
- Commands.
run_commandhands a command to a machine once, and it’s never run again automatically. The command keeps running if the worker that started it dies. If the machine loses the process itself, the result comes back as unknown. - Tools that call outside services, like web search, web fetch, MCP servers, and integrations, aren’t rerun after a crash. The agent gets “Tool call was interrupted; its external outcome is unknown.” and decides whether to check or call it again.
- Tools that change only Omnara’s own state, like starting subagents, save their effect and their result in one transaction.
- Custom tools wait in Postgres until your backend posts a result, without holding a worker, so neither Omnara’s restarts nor yours affect them. The first result wins, and later submissions return
409.
Approvals, questions, and machines
Approvals and questions wait in Postgres for as long as it takes and survive worker crashes and restarts. The first answer wins: the same answer sent again by the same person is safe, and a different answer is rejected. A command a machine hasn’t accepted yet never runs late. It fails if the machine doesn’t accept it within five minutes, or once the machine has been offline for 30 seconds with no wake attempt in progress. If the machine goes offline while the agent is waiting on a command it already accepted, the agent gets a retryablemachine_unreachable result after 30 seconds, with the command’s last known state, so it can move on, for example to another machine. The command may still be running on the offline machine, so redoing its work elsewhere can repeat it.
What your code should handle
- Deduplicate event webhooks by
Webhook-Id, and custom tool work by tool-call ID. Webhooks are retried for up to ten minutes and can arrive more than once. - Treat a
409on a custom tool result as “check the call’s state”: your earlier submission may have succeeded with its response lost. - Make custom tools and MCP servers safe to receive the same request twice.
- Send an
Idempotency-Keywhen you create inputs, so a retried request doesn’t deliver a message twice. - After a stream reconnects, re-read pending tool calls with
GET /tool-calls, since lifecycle updates aren’t replayed.