.env.example in the repo as a starting point.
Required refers to production. Setting
OMNARA_ALLOW_INSECURE_DEV_DEFAULTS=1
fills the fallbacks marked dev below (local Compose services, console email,
an insecure secret-encryption key). Never set it in production.Core
Monitoring agent execution
Alert on these counters:increase(omnara_maintenance_job_items_total{outcome="mismatch"}[1h]) > 0: an agent’s stored execution state disagrees with its history.increase(omnara_worker_claim_deferrals_total[15m]) > 0: a worker could not claim an agent and retried it a minute later. Worker error logs name the agent.
omnara_worker_capacity (configured OMNARA_WORKER_CAPACITY),
omnara_worker_slots_used (occupied slots, including ones reserved while claiming or
backing off after a failure), and omnara_worker_executions_active (claimed work currently
executing). Graph cluster utilization as
100 * sum(omnara_worker_slots_used) / sum(omnara_worker_capacity).
Maintenance samples the agent backlog about every 15 seconds:
omnara_agent_ready_backlog counts due wakeups for non-archived agents without a runtime
lock, and omnara_agent_oldest_ready_age_seconds is how long the oldest one has been
claimable (zero when empty). A wakeup becomes claimable at its ready time or when its
agent’s previous runtime lock was released, whichever is later. Both report NaN before
the first sample or after a failed one; check
omnara_agent_backlog_sample_last_success_timestamp_seconds for freshness. Use the maximum
across maintenance replicas rather than summing.
Alert on sustained omnara_agent_oldest_ready_age_seconds with a fresh sample. High
utilization alongside the backlog means workers are at capacity; low utilization points to
a claiming or scheduling problem instead.
Integration inbox operation
Inbox concurrency is independent of agent-turn concurrency. All worker activity shares the database pool; raising inbox capacity does not raise its connection limit. Measure queue lag and database contention before increasing concurrency or replica counts. Workers sampleomnara_integration_inbox_oldest_ready_lag_seconds about every 30 seconds,
including while every consumer is busy. This measures time since next_attempt_at
for the oldest due queued receipt, including inactive integrations awaiting recovery;
future retry delays and processing leases are excluded. Scrapes read cached
values and never query the inbox. The metric has no integration or project labels and
reports zero for an empty ready queue, or NaN before sampling or after a failed
sample. Check omnara_integration_inbox_lag_sample_last_success_timestamp_seconds for
freshness; it remains unchanged on failure. Each worker observes the same global
queue, so compare the maximum lag across workers rather than summing it.
When processing confirms that a provider receipt has a content-free plan with no
recipients, completion discards its raw message payload. The receipt and duplicate-detection
key remain until normal cleanup. Pending or failed receipts retain their payloads, as do
receipts with message content in their plan and scheduled receipts whose payload is
needed to validate retries. This does not filter events before durable intake.
Inbox retention runs concurrently with the other core maintenance tasks. Completed/failed receipts
become eligible for deletion seven days after finishing, measured by the database
clock. Locks or cleanup backlogs can delay deletion. Deleting the integration, project or
organization can purge receipts earlier; disconnecting an integration preserves normal
retention.
Failed receipts are terminal, release unfinished launch
reservations, and follow the same retention policy. Queued and processing
receipts are never removed by terminal retention. Deleting a receipt ends its
transport deduplication window; exceptionally late replays can be accepted again.
Accepted agent history is independent and remains intact. Newer messages may pass
an older receipt waiting to retry.
Size database capacity for retained receipts as well as peak processing. Retained
row counts grow with incoming event rate and retention time. Measure representative
routing, payloads and competing workloads before sizing the deployment.
Failed integration events
Accepted events retry automatically up to eight attempts with exponential delay. A terminal failure releases unfinished launch reservations. Already committed agents and inputs remain intact. A new mention or message can retry transient failures, but a removed subscription must be restored before that conversation can forward messages. The worker makes a best-effort failure notice for unfinished human requests when provider access still works. Empty or fully admitted plans do not produce an error message. Scheduled failures remain visible in the schedule’s latest run; a known opening thread can also receive a notice. Crash recovery can miss notices and unused-upload cleanup without blocking new work. Inspectintegration inbox admission failed logs using the receipt, integration and project IDs. See the
inbox operation and retention.
Shared-webhook setup failures
Independent saved integrations can share a physical bot and webhook endpoint. Known invalid signatures and revoked/missing setups are skipped, but unknown secret, decryption/KMS or database errors still return non-2xx. Healthy siblings commit their receipts before that response; redelivery deduplicates those receipts. Findprovider event integration failed logs and use integration_id, project_id, provider,
setup_revision, stage and retryable to identify the affected setup. Logs
include the error type, not raw error text, payloads or credentials. Restore a
shared database/KMS outage first. For a persistently corrupt setup, reconnect
that saved integration with valid credentials for the same provider identity, or
disconnect it until repaired so independent healthy integrations can continue. Do not
acknowledge unknown failures merely because another integration succeeded.
After repair, recover events that failed before intake persisted them. Use the
provider’s redelivery where available; otherwise, ask the sender to resend the message
or action. See the integration guides for provider-specific
setup. Disconnecting preserves retained inbox diagnostics. Reconnect before sending
fresh requests; previously failed work remains terminal.
Database and Redis
PostgreSQL 18 or newer is required. API and maintenance pools default to a maximum
of 10 connections per process. Worker pools default to
max(10, ceil(OMNARA_WORKER_CAPACITY / 8)) maximum connections (256 capacity gives
32 connections). All three default to a minimum of 1 connection, a 30-minute idle
timeout, a 30-minute connection lifetime, and five minutes of lifetime jitter.
Explicit pool_* settings in OMNARA_DATABASE_URL override these defaults. Budget
connections across all service replicas.
Direct PostgreSQL, RDS Proxy, and PgBouncer session pooling work with these
settings. PgBouncer transaction pooling requires max_prepared_statements > 0;
statement pooling is not supported.
The migrator defaults to a 30-second lock timeout and a 15-minute statement
timeout. Explicit values in OMNARA_DATABASE_URL take precedence.
Web serving and proxy
OMNARA_EMAIL_DRIVER. In production, use smtp or sendgrid — or disable both flows with OMNARA_AUTH_SIGNUP_ENABLED=0 and OMNARA_AUTH_PASSWORD_RESET_ENABLED=0.
Auth and SSO connectors
GitHub uses
kind:"github" (endpoints default to github.com; GitHub-compatible providers must supply issuer, authorization_url, token_url, and userinfo_url). Google and generic SSO providers — Keycloak, Okta, Auth0, Azure, ZITADEL — use kind:"oidc" with issuer discovery. An example is shown below:
Secret encryption
Secret values are envelope-encrypted with keys from this ring.
Generate each key with
openssl rand -base64 32. To rotate: add the new key to the ring, set it active, rewrap stored versions, then remove the old key once nothing references it.
Skill download capabilities
Skill archive offers use a short-lived capability bound to one machine, skill, and revision.Blob storage
Artifact content lives in S3-compatible object storage; a bucket is required. Grant the service credentialss3:GetObject, s3:PutObject, and s3:DeleteObject
on the bucket’s artifacts/* and skills/* objects.
Memory storage
The directory must be writable by the service user and support case-sensitive paths, atomic rename, file/directory sync, and shared advisory locks. File search and scripted edits require a Linux worker with Landlock ABI 2 or newer and seccomp; on macOS, use Compose with its named volume. See deployment for shared mounts and backups.
Native workers need
rg, omnara-sandbox, and omnara-chroot-sandbox on PATH. For Debian/Ubuntu, install sed and libcap2-bin, then run make install-chroot-sandbox to install the chroot helper and its fixed filesystem jail. Rerun it after upgrading Omnara or the host’s sed, libraries, or locale files. The worker image includes this setup.
Workers also need a writable temporary directory (TMPDIR, normally /tmp) for memory search. If the container root filesystem is read-only, mount a writable volume there.
Scripted edits require a non-root worker with SYS_CHROOT available in its capability bounding set and no_new_privs disabled. Only the chroot helper acquires the capability; it drops it before running sed.
Machine daemon and pools
API daemon configuration
Machine pool templates automatically create a base machine pool for any new organization. Templates use the same provider settings as custom pools; see Sandbox providers for each provider.
Operator-provisioned default model provider
An installation can give each new organization a model provider without storing the upstream credential in configuration. Leave these variables unset to let organization owners configure providers themselves.
A template names the provider and lists the models each organization receives:
POST /model-provider-credentialsreceives the organization ID, creator user ID, and provider template. A successful response is201with{"credential_value":"..."};202and5xxare retryable, while409represents a durable identity or provisioner conflict.- The organization ID is the stable issuance identity. The same request may be repeated until Omnara commits the credential, provider, models, grants, and job completion together. The credential service must recover ambiguous issuance without returning an unusable credential.
- Organization creation never depends on credential issuance. The API attempts provisioning after commit, and maintenance retries failures.
- Apply migrations before deploying both binaries. The API and maintenance must use the same template, credential-service configuration, and secret-encryption key ring.
Host daemon configuration
Web tools
Web tools work out of the box:web_fetch has no vendor dependency, and web_search defaults to Exa’s keyless endpoint.