Skip to main content

Horizontal scaling

The frontend holds no state of its own and keeps nothing between requests, so it scales freely — and when the reverse proxy routes /api/* straight to the backend, as Deployment describes, the chat streams do not pass through it at all. Everything below is about the backend.

What running more than one replica requires​

RequirementWhy
PostgreSQLSQLite is single-writer and single-process. All replicas share one database, which is also what coordinates them
A shared SKILLS_DIREvery replica must mount the same agent skill store (ReadWriteMany). A run pins the skill revision it started with, and any replica may serve the next request in that run
No transaction-pooling proxySee Connection pooling below
A shared MCP_PROXY_TLS_DIREvery replica writes and reads the material that identifies it to the MCP proxy there, and the proxy reads the same directory. A replica that had its own would issue material the proxy has never seen
Sticky sessionsNot required. The advisory lock below does the job routing affinity would be reached for

The MCP proxy itself scales on its own terms: it keeps nothing between calls but a download cache, so replicas of it need no coordination at all — they may share MCP_SANDBOX_DIR or each keep their own. Size it against how many tool calls run at once, not against how many backend replicas there are.

One driver per conversation​

Writes to one conversation are already safe across replicas: the session service takes SELECT ... FOR UPDATE on the row it appends to for the whole transaction, so appends are serialized and neither the conversation's state nor its event rows can be lost.

Reads are the part that needs help. The agent runtime holds one in-memory session for the length of an invocation, so events another replica appends during that window never reach it, and the rest of the run reasons over a conversation that is missing them. Serializing writes cannot repair that — only keeping a session to one driver at a time can.

So each agent run (POST /api/v1/workflow-executions/{id}/agent, POST /api/v1/workflows/{id}/agent) takes a PostgreSQL session-level advisory lock and holds it for the whole SSE stream:

A second concurrent run of the same session is refused before any SSE headers are sent, rather than being left to diverge quietly. Different sessions never contend, and the lock is briefly waited on before it gives up, so a client that aborts a stream and immediately retries is not rejected while the abandoned run is still tearing down.

Human-in-the-loop is unaffected: a frontend tool call ends the run and closes the stream, releasing the lock, and the approval resumes as a new turn that may run on any replica.

Every replica also runs a session runner that performs the turns no browser drives — a run's kickoff and its resumption after an approval. It takes the same lock without waiting, so a session already being run elsewhere is left to that replica, and it picks up a turn that a stopped replica left unfinished. How many such turns one replica runs at once is set by SESSION_RUNNER_CONCURRENCY (see the configuration reference).

Schema migrations during a rollout​

Every replica applies pending schema migrations when it starts, and on a fresh database it also creates the initial accounts and settings. Several replicas rolling out together would all try at once, so they take turns: the first one to start does the work, and the others wait for it to finish — up to ten minutes — before carrying on with nothing left to do. A replica that is still waiting after that stops, and its orchestrator restarts it.

Connection pooling​

Because the lock is session-level rather than transaction-level, the deployment must not place a transaction-pooling proxy — PgBouncer in transaction mode, and most serverless PostgreSQL poolers — between the app and PostgreSQL. The lock would not survive between statements. The same goes for live chat updates: each backend process keeps one connection open to hear about turns written on other replicas, so that a chat open on one replica shows a turn running on another as it happens. Session-level pooling, or a direct connection, is required.

Connection budget​

Each backend process opens database connections from these sources:

SourceConnections
The pool serving API requestsUp to DB_POOL_SIZE + DB_MAX_OVERFLOW
The pool for conversation historyUp to DB_POOL_SIZE + DB_MAX_OVERFLOW
Each agent run in progress — a chat open in a browser, or a turn the server runs on its own (up to SESSION_RUNNER_CONCURRENCY at once)1 for the length of the run
The notification email sender, and background jobs such as a skill pull or a knowledge-base ingest1 each while they run
Live chat updates from other replicas1, always open

With the defaults that is up to 30 pooled connections per process, plus one for live chat updates and one per run in progress. PostgreSQL's own default max_connections is 100, so four or five busy replicas can reach it. Before adding replicas, either raise max_connections on the server, or lower DB_POOL_SIZE and DB_MAX_OVERFLOW (see the configuration reference). If that is still not enough, put PgBouncer in session mode in front of PostgreSQL.

Notification email​

The outgoing-email worker needs no coordination from you: whether it runs in the API processes or as a dedicated one, the email-queue advisory lock elects exactly one sender across the whole deployment, so replicas never double-send and the configured rate limit stays exact. Adding worker replicas buys failover, not throughput — see Process layout.