> ## Documentation Index
> Fetch the complete documentation index at: https://getoverlay.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS reference architecture

> Build and operate Overlay on ECS with a self-hosted Convex backend, RDS PostgreSQL, ElastiCache Redis, S3, and customer-controlled CI/CD.

This guide describes the customer-controlled AWS deployment used for an Overlay
pilot. Complete [Docker On EC2](https://getoverlay.io/docs/self-hosting/install/docker.md) first when the goal
is a quick connectivity and product sanity check.

> [!WARNING]
>
> Do not turn the EC2 sanity host or the disposable AWS rehearsal stack into
> production by changing its name. Production infrastructure must follow the
> customer's networking, security, backup, observability, and change-control
> standards.

## Reference architecture

```text theme={null}
Users
  -> customer DNS and HTTPS
  -> WAF or approved edge
  -> Application Load Balancer
  -> ECS Fargate web service (2+ tasks)
       -> self-hosted convex-backend service (realtime + functions)
       -> RDS/Aurora PostgreSQL: Better Auth data
       -> ElastiCache Redis over TLS
       -> private encrypted S3 bucket
       -> approved OIDC provider and model gateway

ECS convex-backend service (1 active task + standby)
  -> RDS/Aurora PostgreSQL in the same AZ (Convex persistence engine)
  -> S3 buckets for files, modules, exports, snapshot imports, and search
  -> internal ALB on ports 3210 (client/realtime) and 3211 (HTTP actions)

Customer CI -> ECR immutable digest -> one-off migration -> all ECS roles
Secrets Manager -> ECS task secret references and BYOK provider-key records
CloudWatch, Container Insights, and SNS -> logs, metrics, alarms, and operators
```

Convex is the application backend. The self-hosted `convex-backend` service
executes functions, realtime sync, scheduled jobs, and durable work; there are
no separate worker or scheduler services. Postgres remains only as the Convex
backend's persistence engine and as Better Auth's separate auth database — the
app never runs SQL against application data. Deploy and operate the backend per
[Self-hosted Convex backend](https://getoverlay.io/docs/self-hosting/install/self-hosted-convex.md).

## Required ownership

Before provisioning, name the owners for:

* AWS account, Region, VPC, quotas, tags, and budget.
* Source repository, protected branches, CI/CD, and release approval.
* DNS, ACM certificates, ingress, WAF, and outbound egress.
* RDS, Redis, S3, KMS, backups, restore tests, and retention.
* Google Workspace or other OIDC application and pilot users.
* Secrets Manager values and rotation.
* CloudWatch alarms and the incident notification destination.
* Model gateway and every optional external processor.
* Pilot acceptance, rollback, and go-live.

Use a dedicated nonproduction account when possible. Human access should be
federated, named, MFA-protected, time-limited, least-privilege, and audited by
CloudTrail. CI should use GitHub Actions OIDC or CodeBuild roles, not static AWS
access keys.

## AWS resources

| Layer | Minimum pilot requirement |
| - | - |
| Network | At least two Availability Zones; public ALB subnets; private ECS subnets; isolated RDS/Redis subnets; approved NAT, proxy, or VPC endpoints. |
| Compute | ECS cluster; web service with at least two tasks; convex-backend service (active + standby); one-off migration task. |
| Database | Encrypted Multi-AZ RDS PostgreSQL or Aurora PostgreSQL writer serving the convex-backend persistence engine, plus a separate auth-scoped database and role for Better Auth; automated backups and restore test. Backend and RDS must be in the same region, ideally the same AZ. |
| Redis | TLS ElastiCache with fail-closed application behavior; Multi-AZ automatic failover for the target pilot. |
| Objects | Private encrypted, versioned S3 bucket; block public access; lifecycle, logging, backup/recovery, and deletion policy. |
| Ingress | HTTPS ALB listener, healthy target group, customer DNS, ACM certificate, and WAF/edge controls required by policy. |
| Delivery | Customer repository; CI OIDC or CodeBuild; immutable ECR tags; image scanning; protected staging and production environments. |
| Operations | CloudWatch log groups, Container Insights, dashboard, alarms, SNS destination, RDS metrics, ALB access logs, and defined retention. |

## Secrets Manager

Create separate paths for staging and production. ECS injects application
secrets as environment variables. For BYOK, the application makes narrowly
scoped runtime calls to Secrets Manager using its ECS task role; each user API
key is a separate opaque secret ARN stored in Convex, never a plaintext
database value. Keep non-secret selectors and hostnames in task-definition
environment variables.

Enable AWS-backed BYOK in the customer runtime configuration:

```text theme={null}
OVERLAY_PROVIDER_SECRETS=aws-secrets-manager
AWS_REGION=<customer-region>
OVERLAY_AWS_SECRETS_PREFIX=jpgs/overlay/byok
OVERLAY_AWS_SECRETS_KMS_KEY_ID=<customer-kms-key-arn>
```

Grant the web ECS task role only `secretsmanager:CreateSecret`,
`GetSecretValue`, `PutSecretValue`, and `DeleteSecret` for that prefix, plus
`kms:Encrypt`, `kms:Decrypt`, and `kms:GenerateDataKey` on the customer KMS
key. Do not grant list, broad read, cross-environment, or wildcard customer
secret permissions.

At minimum, create entries for:

```text theme={null}
SESSION_SECRET
SESSION_TRANSFER_KEY
SESSION_COOKIE_ENCRYPTION_KEY
INTERNAL_API_SECRET
INTERNAL_SERVICE_AUTH_SECRET
MCP_CREDENTIAL_ENCRYPTION_KEY
API_KEY_HASH_SECRET
PROVIDER_KEYS_SECRET
HOOKS_TOKEN_SALT
BETTER_AUTH_SECRET
BETTER_AUTH_OIDC_CLIENT_SECRET
BETTER_AUTH_DATABASE_URL
OVERLAY_REDIS_URL
AI_GATEWAY_API_KEY
```

Add provider credentials only for approved, enabled features. On ECS or EC2,
prefer a bucket-scoped task or instance role and omit `S3_ACCESS_KEY_ID` and
`S3_SECRET_ACCESS_KEY`. For non-AWS S3-compatible storage, configure both keys,
store them in Secrets Manager, and rotate them. Do not place any secret in a
`NEXT_PUBLIC_*` variable, source file, task-definition plaintext environment
value, build log, document, ticket, email, or chat.

## Deployment procedure

**1. Prepare the customer repository**

Fork or mirror the public
[LayerNorm/overlay-web](https://github.com/LayerNorm/overlay-web) repository into
the customer's GitHub organization. Clone that customer-controlled repository,
retain the public repository as `upstream`, and branch from an approved
published release. The example defaults to the tested `v0.1.0-rc.1` release.

```bash theme={null}
export CUSTOMER_REPOSITORY_URL="https://github.com/<CUSTOMER_ORGANIZATION>/<CUSTOMER_REPOSITORY>.git"
export OVERLAY_RELEASE="${OVERLAY_RELEASE:-v0.1.0-rc.1}"

git clone "$CUSTOMER_REPOSITORY_URL" overlay-web
cd overlay-web
git remote add upstream https://github.com/layernorm/overlay-web.git
git fetch upstream --tags --force
git checkout -b "release/${OVERLAY_RELEASE#v}-customer.1" "$OVERLAY_RELEASE"
git push -u origin "release/${OVERLAY_RELEASE#v}-customer.1"
```

Merge only through a reviewed pull request with required checks.

**2. Establish customer-controlled CI access**

Configure GitHub Actions OIDC or CodeBuild. Restrict the deployment role to
the exact repository, branch/environment, ECR repository, ECS services,
task roles, log groups, and approved infrastructure stacks. Restrict
`iam:PassRole` to the named ECS execution and task roles.

**3. Provision infrastructure as code**

Use the customer's approved Terraform, CloudFormation, CDK, or platform
catalog. Keep environment values outside application source. Review the
generated plan or change set before applying it.

Required network flows are:

```text theme={null}
Internet/edge -> ALB: HTTPS only
ALB -> web tasks: application port only
web tasks -> convex-backend internal ALB: 3210 and 3211
convex-backend -> RDS: 5432 by security-group reference (same AZ)
migration -> RDS: 5432 by security-group reference (Better Auth database)
web tasks -> Redis: TLS port by security-group reference
convex-backend -> S3: VPC endpoints or approved egress
runtime -> Secrets/ECR/CloudWatch: VPC endpoints or approved egress
runtime -> OIDC/model/customer APIs: explicit approved HTTPS destinations
```

**4. Build and push one immutable image**

Build from the reviewed commit in CI, not on a production host:

```bash theme={null}
export AWS_REGION=<REGION>
export AWS_ACCOUNT_ID=<ACCOUNT_ID>
export ECR_REPOSITORY=overlay-web
export GIT_SHA="$(git rev-parse HEAD)"
export IMAGE="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$GIT_SHA"

aws ecr get-login-password --region "$AWS_REGION" | \
  docker login --username AWS --password-stdin \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"

docker buildx build \
  --platform linux/amd64 \
  --file docker/Dockerfile.enterprise-runtime \
  --build-arg NEXT_PUBLIC_CONVEX_URL="https://<CONVEX_BACKEND_CLOUD_ORIGIN>" \
  --build-arg NEXT_PUBLIC_APP_URL="https://<OVERLAY_HOST>" \
  --tag "$IMAGE" \
  --push .

export IMAGE_DIGEST="$(aws ecr describe-images \
  --repository-name "$ECR_REPOSITORY" \
  --image-ids imageTag="$GIT_SHA" \
  --query 'imageDetails[0].imageDigest' \
  --output text)"

test -n "$IMAGE_DIGEST" && test "$IMAGE_DIGEST" != None
printf 'Release %s -> %s@%s\n' "$GIT_SHA" "$ECR_REPOSITORY" "$IMAGE_DIGEST"
```

Record the digest and use `repository@sha256:...` in every task definition.
Do not rebuild the image between staging and production. `NEXT_PUBLIC_*`
values are inlined at build time, so the baked `NEXT_PUBLIC_CONVEX_URL` must
be the origin web tasks and browsers can both reach — the backend's internal
ALB hostname works when web tasks, operator browsers, and the backend share
routable DNS; otherwise use the public backend origin.

**5. Define the ECS roles**

All task definitions use the same image digest and configuration family:

| Role | Command | Scaling |
| - | - | - |
| Web | `node server.js` | At least two tasks behind the ALB. |
| Convex backend | Official `convex-backend` image | One active task plus a standby; see the self-hosted runbook. |
| Migration | See the command below | One-off task only; never an ECS service. |

Migration command:

```bash theme={null}
NODE_OPTIONS=--conditions=react-server node_modules/.bin/tsx scripts/db/better-auth-migrate.ts
```

Convex schema and function deployment is a separate `npx convex deploy` step
against the self-hosted backend; see
[Self-hosted Convex backend](https://getoverlay.io/docs/self-hosting/install/self-hosted-convex.md).

Configure health checks, log groups, CPU/memory, graceful stop time, read-only
filesystem exceptions, database pool budgets, and secret ARN references per role.

**6. Snapshot and run migrations once**

Record the current image digest, task-definition revisions, schema version,
a pre-deploy RDS snapshot, and a `npx convex export` of the Convex deployment.

Run the Better Auth migration task in the private ECS subnets with the direct
RDS writer endpoint. Wait for the task to stop, inspect its exit code, and
retain its redacted logs. Do not deploy the services when the migration exit
code is nonzero. Deploy the Convex revision per the self-hosted runbook and
watch the backend logs for `Executing Migration n/m`.

**7. Deploy the same digest to every service**

Register new task-definition revisions containing the recorded digest, then
update the convex-backend and web services. For web rolling deployments use:

```text theme={null}
desiredCount: 2 or more
minimumHealthyPercent: 100
maximumPercent: 200
ALB stickiness: disabled
ALB idle timeout: at least 60 seconds
```

Wait for every ECS service to become stable and every ALB target to become
healthy. The convex-backend service must hold its persistence lease; a promoted
standby acquiring the lease is expected HA behavior, not a split brain.

**8. Configure DNS, TLS, and OIDC**

Point the approved hostname to the ALB or approved edge. Register the exact
production values with the OIDC owner:

```text theme={null}
Callback: https://<OVERLAY_HOST>/api/better-auth/sso/callback/<PROVIDER_ID>
Logout:   https://<OVERLAY_HOST>
Origin:   https://<OVERLAY_HOST>
```

Set `BETTER_AUTH_URL`, trusted origins, JWT issuer/audience, and JWKS URL to
that same HTTPS origin. Store only the public client ID in plaintext config;
the client secret remains in Secrets Manager.

**9. Qualify staging**

At minimum verify:

1. `/api/v1/capabilities`, `/api/auth/session`, and the ALB health path.
2. OIDC sign-in, refresh, new tab, sign-out, and account deletion.
3. Chat create, stream, stop, refresh recovery, rename, and delete across two web tasks.
4. Files, notes, projects, memory, knowledge, and citations when enabled.
5. Convex cron execution and realtime subscriptions observed in backend logs.
6. Redis fail-closed behavior and recovery.
7. RDS failover and convex-backend lease recovery; automatic web recovery.
8. S3 signed URL expiry, ownership checks, deletion, and reconciliation.
9. Backup restore to a new database and application connection to the restore.
10. CloudWatch alarms and delivery to the named operator.
11. Convex connectivity: web tasks reach the backend on 3210/3211 and no other deployment URL.

Keep the test accounts and data disposable. Run destructive contracts only
against a separate contract-test database.

## Rollback

Before every release, record the previous image digest and task definitions.
When the new schema remains backward-compatible, roll all services back to the
previous digest and verify auth, chat, files, and queue state.

If the previous runtime is not schema-compatible, do not improvise a down
migration. Restore the pre-deploy snapshot into a new database, point the
previous release at the restored writer, and preserve the failed database and
logs for investigation.

## Product updates and customer customizations

Overlay updates never deploy automatically into the customer account:

```bash theme={null}
git fetch upstream --tags
git checkout -b upgrade/overlay-vX.Y.Z main
git merge --no-ff <APPROVED_UPSTREAM_TAG>
# Resolve conflicts, update customer extensions, and run all gates.
git push -u origin upgrade/overlay-vX.Y.Z
```

Reusable fixes should go upstream. Customer policy, consent, curriculum, and
proprietary data-lake logic should remain behind provider interfaces, service
contracts, APIs, MCP servers, or extension modules where practical. Every
customer customization needs tests, an owner, a data-flow description, an
upgrade note, and rollback behavior.

## Paste into a coding agent

Use this prompt only after the agent is authenticated through the customer's
approved AWS and source-control mechanisms:

```text theme={null}
Deploy Overlay to the customer-controlled AWS staging environment by following
docs/self-hosting/install/aws.mdx.

Operating constraints:
- Work only in the approved account, Region, repository, and staging environment.
- Start by reporting `aws sts get-caller-identity`, Git remote URLs, branch, and SHA.
- Use existing customer IaC and security standards. Do not create a parallel VPC
  or bypass approval because it is faster.
- Use federated access and CI OIDC/CodeBuild roles. Never create static AWS keys.
- Never print, echo, retrieve into logs, commit, or return secret values.
- Reference Secrets Manager ARNs from ECS. Return only names and ARNs.
- Keep RDS, Redis, the convex-backend internal ALB, and ECS private. Expose only
  the approved HTTPS ingress.
- Build one immutable image and use its exact digest for migration and web. The
  convex-backend uses the pinned official image; run migrations and the Convex
  deploy once before service rollout.
- Stop and request approval before applying IaC, running migrations, changing
  DNS/OIDC, triggering failover, deleting data, or promoting to production.

Execution checkpoints:
1. Inventory customer inputs, existing infrastructure, quotas, and missing owners.
2. Produce an architecture delta and IaC plan/change set; wait for approval.
3. Validate config and tests, build the reviewed commit in CI, push to ECR, scan,
   and record the immutable digest.
4. Record backup and rollback state; run the one-off migration and verify exit code.
5. Deploy the digest to the convex-backend and web services; wait for stable services.
6. Pause for interactive OIDC setup/login where required.
7. Execute staging product, isolation, recovery, backup, alarm, deletion, and
   Convex-connectivity QA from the guide.
8. Return a redacted deployment report: account/Region, resource IDs, Git SHA,
   image digest, task revisions, schema version, QA results, alarm/restore proof,
   costs, known issues, rollback instructions, and PASS/FAIL. Never return secrets.
```

## Handover gate

The deployment is not handed over until the customer PoC can:

1. Identify the deployed Git SHA, image digest, schema version, and task revisions.
2. Redeploy the current digest through CI without changing source.
3. Find web, convex-backend, migration, RDS, Redis, ALB, and S3 telemetry.
4. Trigger and receive a test alarm.
5. Roll back to the previous digest and roll forward again.
6. Locate backups, restore instructions, secret rotation ownership, and incident contacts.
7. Revoke or reduce the temporary Overlay operator role while CI continues to work.

For configuration details and provider matrices, continue with
[Self-Hosting](https://getoverlay.io/docs/self-hosting/overview.md) and
[Security](https://getoverlay.io/docs/self-hosting/operate/security.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.