Nexadata Production Deployment

Use this guide when Nexadata must run across multiple containers, survive a container failure, or process a large workflow volume. It covers a separate worker pool, DNS-based clustering, load balancing, and an AWS ECS reference implementation. The same pattern works on any container platform.

Requirements

In a production deployment, it is recommended to run multiple nodes. Application nodes and worker nodes both use the same Docker image, but operate in different modes depending on configuration. Application nodes host the web interface and should be operated behind a load balancer. Worker nodes run in the background, polling for and executing workflows from a persistent queue.

Each node needs:

• 1 vCPU
• 4 GB RAM
• 20 GB storage

For provisioning, we recommend:

• 2 or more application nodes, ideally behind a load balancer
• 2 or more worker nodes

When running multiple nodes, clustering is required. See Clustering for how to set up a cluster so the nodes are aware of each other.

You also need the following infrastructure:

Requirement	Details
Container platform	Any platform that can run the Nexadata image and supply secrets
Postgres	Managed Postgres recommended; connection limit must cover all application and worker containers
S3-compatible storage	AWS S3 or another S3-compatible service
SMTP endpoint	Required for platform email
Load balancer	HTTP health checks and TLS termination
DNS service discovery	A DNS name that resolves to every application and worker container
Secrets	SECRET_KEY_BASE, VAULT_KEY, RELEASE_COOKIE, database credentials, storage keys, SMTP credentials, LLM keys

For a single container, see Nexadata Self-Hosted Quickstart.

Topology

                    ┌────────────────────────────┐
  users ── TLS ───▶ │ load balancer              │
                    └─────────────┬──────────────┘
                                  │
                    ┌─────────────▼──────────────┐
                    │ Nexadata application pool  │
                    │ 2 or more containers       │
                    │ web endpoint, port 4000    │
                    └─────────────┬──────────────┘
                                  │
          ┌───────────────────────┼───────────────────────┐
          │                       │                       │
 ┌────────▼────────┐    ┌─────────▼────────┐    ┌─────────▼────────┐
 │ Postgres        │    │ S3-compatible    │    │ SMTP endpoint    │
 │ data and jobs   │    │ object storage   │    │ email delivery   │
 └─────────────────┘    └──────────────────┘    └──────────────────┘

                    ┌────────────────────────────┐
                    │ Nexadata worker pool       │
                    │ separate containers        │
                    │ background jobs only       │
                    └────────────────────────────┘

Application containers serve HTTP and do not process background jobs. Worker containers process jobs and do not serve HTTP. Both use the same image, database, storage, and SMTP settings.

Containers

Setting	Application container	Worker container
BACKGROUND_WORKER	false	true
INLINE_WORKER	false (default)	not used
Exposed port	4000	none
Health check	GET /health	platform/process check only
Load balancer	yes	no
Scaling signal	requests or CPU	queue depth

Both roles use the same NEXADATA_IMAGE, the same RELEASE_COOKIE, and the same database and storage configuration. Every container applies database migrations before starting. Postgres serializes concurrent migrations, so rolling deploys are safe.

Initial Organization and Admin

After the application service is running, create the initial organization and admin user once. The bootstrap script is included with the Nexadata container and it can be executed on the container:

./bootstrap \
  --email admin@example.com \
  --password 'ChangeMe123!' \
  --org 'Your Organization'

Run this inside one application container (or as a one-off task with the same image and environment). The command creates the database and loads the schema when needed, applies migrations, and is idempotent; it does not reset an existing admin password.

Configuration

The configuration in Nexadata Self-Hosted Quickstart applies, with these changes:

• Set INLINE_WORKER=false (or leave it unset).
• Run application containers with BACKGROUND_WORKER=false.
• Run worker containers with BACKGROUND_WORKER=true.
• Enable DNS clustering for every application and worker container.
• Use a managed Postgres and S3 bucket; do not run database or object storage inside the application containers.

Common Settings

Two variables of particular importance are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. When not running the application on AWS, these variables are still required, as they serve as keys for most S3-compatible storage options that you would run yourself (i.e. MinIO). These variables are required unless you are running on AWS and use instance role authentication.

Variable	Required	Description
PUBLIC_URL	yes	Public base URL for the application, including port (if not 80 or 443)
SECRET_KEY_BASE	yes	Random session signing key; openssl rand -base64 64
VAULT_KEY	yes	Encryption key for secrets stored in DB, exactly 32 bytes; openssl rand -hex 16
RELEASE_COOKIE	yes	Random value to verify clustered nodes can connect to one another
DATABASE_URL	yes	Postgres connection URL, preferably with sslmode=require
FILE_MANAGER_BUCKET	yes	Private bucket for uploads and workflow outputs
STATIC_FILES_BUCKET	for profile images	Publicly readable bucket
AWS_REGION	yes	Storage region, relevant for S3-compatible storage even if not on AWS
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY	yes, with caveat (see above)	Omit when using an instance or task role
S3_ENDPOINT	for non-AWS storage	S3-compatible endpoint
S3_FORCE_PATH_STYLE	for non-AWS storage	Usually true
MAILER_ADAPTER	yes	smtp
SMTP_HOST, SMTP_PORT, SMTP_USERNAME, SMTP_PASSWORD	yes	SMTP endpoint
SMTP_TLS, SMTP_AUTH, SMTP_SSL	no	Match your SMTP provider
MAILER_FROM_EMAIL, MAILER_FROM_NAME	no	Default sender
APP_NAME, BUSINESS_NAME	no	Set both to Nexadata
POOL_SIZE	no	Database connections per container; default 10
PORT	no	Internal HTTP port (port inside Docker container); default 4000

Clustering

Variable	Required	Description
CLUSTER_STRATEGY	yes	dns
DNS_POLL_QUERY	yes	DNS name that resolves to every application and worker container

The Nexadata application relies upon a multi-node strategy where each node is aware of the others, forming a cluster. Nodes will be automatically discovered if using the dns cluster strategy.

The DNS_POLL_QUERY represents a hostname that the application should query to return a list of node IPs. From there, the application will attempt to connect all of the nodes together. This also requires certain ports to be opened, and the RELEASE_COOKIE environment variable to be the same across all nodes. Allow container-to-container traffic on the following ports:

• port 4369 for node discovery;
• ports 9000-9010 for node communication;
• port 9090 for remote calls.

Note

Do not expose these ports through the public load balancer.

Queue Scaling

Variable	Description
QUEUE_METRICS=cloudwatch	Publishes queue depth to AWS CloudWatch for worker autoscaling
QUEUE_METRICS=sink	Logs queue depth instead; use when your platform scales workers another way

Queue metrics are produced by worker containers. Application containers should use sink. If you are deployed to AWS, this will report the depth of the queue to CloudWatch, which could become the basis for an autoscaling strategy for worker nodes.

Optional Capabilities

Variable	Description
LLM_PROVIDER	anthropic (requires LLM_API_KEY) or bedrock (AWS credentials)
LLM_API_KEY, LLM_MODEL, ADVANCED_LLM_MODEL, LLM_BASE_URL	AI provider settings
OCR_PROVIDER	none, or textract with AWS S3 and Textract permissions
OTEL_EXPORTER_OTLP_ENDPOINT	OTLP collector endpoint; leave unset to disable telemetry
OTEL_EXPORTER_OTLP_HEADERS	OTLP vendor headers
CSP_EXTRA_SOURCES	Comma-separated additional Content Security Policy sources, added to default-src and img-src
SAML variables	SAML_SP_ID, SAML_SP_ENTITY_ID, SAML_SP_CERT/SAML_SP_KEY (PEM contents), SAML_SP_CERTFILE/SAML_SP_KEYFILE (paths), SAML_ORG_*, SAML_CONTACT_*
OAuth variables	HUBSPOT_CLIENT_ID/_SECRET

Note

Textract requires a real AWS S3 bucket. MinIO and other S3-compatible services are not supported for PDF extraction.

OpenTelemetry

Nexadata exports traces and metrics over OTLP when an endpoint is configured. Configure both the application and worker containers:

Variable	Required	Description
OTEL_EXPORTER_OTLP_ENDPOINT	yes	Collector base URL, without /v1/traces or /v1/metrics
OTEL_EXPORTER_OTLP_PROTOCOL	yes	http/protobuf, or grpc if your collector supports it
OTEL_SERVICE_NAME	recommended	Service name shown by your telemetry backend
OTEL_EXPORTER_OTLP_HEADERS	if required	Comma-separated authentication headers, for example x-api-key=...
OTEL_RESOURCE_ATTRIBUTES	optional	Additional resource attributes, for example deployment.environment=production

The exporter appends the signal paths to OTEL_EXPORTER_OTLP_ENDPOINT. Do not set OTEL_SDK_DISABLED=true, because that disables the SDK. The metrics exporter is enabled by the application; no separate metrics flag is required.

SAML Service Provider Files

The Nexadata image does not include a SAML service-provider certificate or private key. Provide them at runtime and point the application at the files:

SAML_SP_CERTFILE=/run/secrets/saml_sp_cert
SAML_SP_KEYFILE=/run/secrets/saml_sp_key
SAML_SP_ENTITY_ID=https://app.example.com

With Docker Compose, mount them as read-only secrets:

secrets:
  saml_sp_cert:
    file: ./secrets/saml-sp.crt
  saml_sp_key:
    file: ./secrets/saml-sp.key

services:
  nexadata:
    secrets:
      - saml_sp_cert
      - saml_sp_key

With docker run, mount the files directly:

-v "$PWD/secrets/saml-sp.crt:/run/secrets/saml_sp_cert:ro" \
-v "$PWD/secrets/saml-sp.key:/run/secrets/saml_sp_key:ro"

The files must be readable by the container user (UID 61000 in the supplied image). If SAML is not used, leave both paths unset.

You may also inject them as environment variables: SAML_SP_KEY and SAML_SP_CERT.

If you inject them this way, at startup the container writes them to /tmp/nexadata-secrets with owner-only permissions and sets SAML_SP_CERTFILE / SAML_SP_KEYFILE automatically. Do not also set the file-path variables.

Load Balancing

• Health check: GET /health, expecting HTTP 200.
• Terminate TLS at the load balancer and forward to container port 4000 (or the exposed port provided by the PORT environment variable).
• Redirect HTTP to HTTPS, and set PUBLIC_URL to the public HTTPS origin so generated links use the correct scheme and port.
• Enable sticky sessions so a browser stays on one application container.
• Set the idle timeout to at least 120 seconds for browser WebSocket connections.
• Use a deregistration delay of at least 30 seconds for nodes behind a load balancer.
• Do not route web traffic to worker containers (if using a load balancer, do not register them with the load balancer).

Temporary Files

Workflow files are written to container-local disk before upload to S3. With clustering enabled, downloads can read the file from the container that wrote it. If you run multiple application containers without clustering, mount shared storage (for example EFS or NFS) at /tmp on every application and worker container.

AWS ECS Reference

AWS ECS/Fargate is one implementation of this topology. The same environment and port rules apply to Kubernetes or another orchestrator.

Task Definitions

Run one task definition for the application and one for workers, both with the same image and common environment. Use awsvpc networking and give both task roles S3, SMTP/SES, CloudWatch, and any LLM/Textract permissions you enable.

Application container:

{
  "name": "nexadata",
  "image": "<nexadata-image>",
  "portMappings": [{ "containerPort": 4000, "protocol": "tcp" }],
  "environment": [
    { "name": "BACKGROUND_WORKER", "value": "false" },
    { "name": "INLINE_WORKER", "value": "false" },
    { "name": "CLUSTER_STRATEGY", "value": "dns" },
    { "name": "DNS_POLL_QUERY", "value": "nexadata.local" },
    { "name": "PUBLIC_URL", "value": "https://app.example.com" },
    { "name": "FILE_MANAGER_BUCKET", "value": "nexadata-files" },
    { "name": "STATIC_FILES_BUCKET", "value": "nexadata-static" },
    { "name": "QUEUE_METRICS", "value": "sink" }
  ],
  "secrets": [
    { "name": "SECRET_KEY_BASE", "valueFrom": "<secret-arn>" },
    { "name": "VAULT_KEY", "valueFrom": "<secret-arn>" },
    { "name": "RELEASE_COOKIE", "valueFrom": "<secret-arn>" },
    { "name": "DATABASE_URL", "valueFrom": "<secret-arn>" },
    { "name": "SMTP_PASSWORD", "valueFrom": "<secret-arn>" }
  ]
}

Worker container: identical, but:

{
  "environment": [
    { "name": "BACKGROUND_WORKER", "value": "true" },
    { "name": "CLUSTER_STRATEGY", "value": "dns" },
    { "name": "DNS_POLL_QUERY", "value": "nexadata.local" },
    { "name": "QUEUE_METRICS", "value": "cloudwatch" }
  ]
}

The worker has no portMappings and is not attached to the load balancer.

ECS Services

• Put both services in the same Cloud Map namespace and make DNS_POLL_QUERY resolve to all task IPs.
• Attach only the application service to the ALB target group.
• Allow the task security group to reach itself on 4369, 9000-9010, and 9090.
• Use RDS for Postgres and S3 for storage.
• Use secrets for SECRET_KEY_BASE, VAULT_KEY, RELEASE_COOKIE, DATABASE_URL, and SMTP/LLM credentials.
• Keep the same RELEASE_COOKIE in both task definitions.

Worker Autoscaling

With QUEUE_METRICS=cloudwatch, workers publish queue depth to CloudWatch:

• Namespace: Workflows/QueueDepth
• Metric: Depth
• Dimension: QueueName=default

Register the worker ECS service as a scalable target and add a target-tracking policy:

aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/nexadata-prod/nexadata-prod-worker \
  --min-capacity 1 \
  --max-capacity 20

aws application-autoscaling put-scaling-policy \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/nexadata-prod/nexadata-prod-worker \
  --policy-name nexadata-worker-queue-depth \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 10,
    "ScaleInCooldown": 300,
    "ScaleOutCooldown": 60,
    "CustomizedMetricSpecification": {
      "MetricName": "Depth",
      "Namespace": "Workflows/QueueDepth",
      "Statistic": "Maximum",
      "Dimensions": [{"Name": "QueueName", "Value": "default"}]
    }
  }'

The task role needs cloudwatch:PutMetricData. Scale application containers from request count or CPU, not queue depth.

Operations

Upgrades

1. Back up Postgres and the S3 buckets.
2. Deploy the new image to workers, then application containers.
3. Keep SECRET_KEY_BASE, VAULT_KEY, and RELEASE_COOKIE unchanged.
4. Verify /health and run a test workflow.

Every container applies migrations before starting. No separate migration step is required; Postgres serializes concurrent migrations during a rolling deploy.

Backups

• Back up Postgres with your provider’s automated backups or pg_dump.
• Enable versioning or replicate the file-manager and static buckets.
• Back up the secrets store, especially VAULT_KEY.

Danger

Losing VAULT_KEY makes stored credentials unrecoverable. Back it up before you go live.

Troubleshooting

Symptom	Check
Container exits at startup	Logs contain a configuration error naming every invalid variable
RELEASE_COOKIE is not set	Set the same cookie on every container
Containers do not cluster	CLUSTER_STRATEGY, DNS_POLL_QUERY, cookie, DNS resolution, ports 4369/9000-9010/9090
Workflows stay queued	Worker containers are running with BACKGROUND_WORKER=true and can reach Postgres
No worker autoscaling metrics	QUEUE_METRICS=cloudwatch, cloudwatch:PutMetricData, worker logs
Downloads fail after scaling	Clustering is enabled, or shared /tmp is mounted on every container
OCR fails	Textract requires real AWS S3 and AWS permissions; use OCR_PROVIDER=none otherwise
Email does not arrive	SMTP host, port, credentials, TLS, and sender address