API Staging — One-Time Droplet Setup

This page documents the one-time provisioning steps to prepare a fresh DigitalOcean droplet for running the Sema Link API. These steps are manual and run once. All subsequent deployments are automated via GitHub Actions — see API CI/CD Pipeline.

Current staging droplet: 209.38.197.79 — Ubuntu 24.04 LTS, 2 vCPU / 4 GB RAM, Frankfurt (FRA1)

---

1. Create the Droplet

In the DigitalOcean control panel:

1. Create Droplet → Ubuntu 24.04 LTS x64
2. Size: Basic, 2 vCPU / 4 GB RAM / 80 GB SSD ($24/mo)
3. Region: Frankfurt (FRA1) — matches Neon and Upstash data residency
4. Add your SSH public key at creation time so you get root access immediately
5. No additional volumes or VPCs needed for staging

---

2. Prepare the OS

SSH in as root:

ssh root@<droplet-ip>

Update packages:

apt update && apt upgrade -y

---

3. Install Docker

The entire API stack runs in Docker. Install Docker Engine using the official convenience script:

curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker

Verify:

docker --version          # Docker 26.x or higher
docker compose version    # Docker Compose v2.x (bundled with Docker Engine)

The API uses docker compose (Compose V2, plugin), not the legacy docker-compose command. Make sure you get Docker Engine ≥ 23 which includes Compose V2 as a plugin.

---

4. Create the deploy User

GitHub Actions SSHes into the droplet as a non-root deploy user. This limits blast radius if the SSH key is ever compromised.

useradd -m -s /bin/bash deploy
usermod -aG docker deploy     # allow docker without sudo

---

5. Add the GitHub Actions SSH Key

Generate an ED25519 key pair locally (not on the droplet):

ssh-keygen -t ed25519 -C "github-actions-staging" -f ~/.ssh/semalink_staging_deploy

This produces:

• ~/.ssh/semalink_staging_deploy — private key (goes into GitHub secret)
• ~/.ssh/semalink_staging_deploy.pub — public key (goes on the droplet)

On the droplet, add the public key to the deploy user's authorized keys:

mkdir -p /home/deploy/.ssh
cat >> /home/deploy/.ssh/authorized_keys <<'EOF'
<paste contents of semalink_staging_deploy.pub here>
EOF
chown -R deploy:deploy /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

Test from your local machine:

ssh -i ~/.ssh/semalink_staging_deploy deploy@<droplet-ip>

You should land in a shell as deploy with no password prompt.

---

6. Configure GitHub Secrets

In the semalink-api GitHub repository → Settings → Environments → staging, add these secrets:

Secret	Description
STAGING_DROPLET_IP	The droplet's public IPv4 address
STAGING_SSH_KEY	Contents of ~/.ssh/semalink_staging_deploy (private key, including BEGIN/END lines)
DATABASE_URL	Full Neon connection string (see below)
REDIS_URL	Full Upstash Redis URL (see below)
RABBITMQ_PASS	Password for the RabbitMQ semalink user (32+ random chars)
JWT_SECRET	HS256 signing secret for access tokens (64+ random chars)
JWT_REFRESH_SECRET	Signing secret for refresh tokens (64+ random chars, different from JWT_SECRET)
MAILGUN_API_KEY	Mailgun private API key
MAILGUN_DOMAIN	mailer.semalink.africa
MAILGUN_FROM	noreply@mailer.semalink.africa
CELCOM_API_URL	https://api.celcomafrica.com
CELCOM_API_KEY	Celcom Africa API key
SLACK_WEBHOOK_URL	Slack incoming webhook for deployment notifications

GitHub expands these secrets into the .env file at deploy time — see API CI/CD Pipeline for how this works.

---

7. Clone the Repository

On the droplet as root, create the deployment directory and clone the repo:

mkdir -p /opt/semalink-api
git clone https://github.com/Sema-Link/semalink-api.git /opt/semalink-api
chown -R deploy:deploy /opt/semalink-api

Switch to the staging branch:

cd /opt/semalink-api
git checkout staging

---

8. Create the .env File

Copy the example template and fill in real values:

cp /opt/semalink-api/.env.staging.example /opt/semalink-api/.env
nano /opt/semalink-api/.env

From the second deploy onwards, GitHub Actions overwrites .env from secrets on every run — you only need to populate it manually for the very first deploy.

Database — Neon (serverless Postgres)

The staging API connects to a Neon serverless Postgres instance in Frankfurt. Neon uses a branching model: production uses the main database branch; staging uses a separate staging branch, which is a live fork of production schema with isolated data.

DATABASE_URL=postgresql://<user>:<password>@<host>-pooler.c-3.eu-central-1.aws.neon.tech/neondb?sslmode=require&channel_binding=require

Key notes:

• The URL includes -pooler in the hostname — this is Neon's PgBouncer connection pooler, which is required for serverless/edge workloads where connection count spikes matter
• sslmode=require enforces TLS
• channel_binding=require enables SCRAM-SHA-256-PLUS for stronger auth
• Find the URL in the Neon dashboard under Connection Details for the staging branch

Cache — Upstash Redis

Redis is used for caching and session/rate-limit state. The staging instance runs on Upstash in Frankfurt (free tier, sufficient for staging).

REDIS_URL=rediss://default:<password>@<hostname>.upstash.io:6379

Key notes:

• rediss:// (double-s) = TLS-encrypted connection — always required for Upstash
• Port 6379 is Upstash's TLS Redis port
• Find the URL in the Upstash console under Details → REST URL / Redis URL

Message Queue — RabbitMQ (local Docker container)

RabbitMQ runs as a local Docker container on the droplet (no managed service). Its password is set via the RABBITMQ_PASS environment variable:

RABBITMQ_PASS=<32+ random characters>

The API connects using the internal Docker network hostname:

RABBITMQ_URL=amqp://semalink:${RABBITMQ_PASS}@rabbitmq:5672

This URL is constructed in docker-compose.staging.yml — you only need to supply RABBITMQ_PASS in .env.

---

9. Run the First Deploy

From /opt/semalink-api as the deploy user:

docker compose -f docker-compose.staging.yml build
docker compose -f docker-compose.staging.yml up -d

The first build takes 2–5 minutes (downloading base images, installing npm dependencies, compiling TypeScript).

---

10. Verify Everything Is Running

Check container status

docker compose -f docker-compose.staging.yml ps

Expected output — all three services should show running (healthy) or running:

NAME                    IMAGE               STATUS
semalink-api-caddy-1    caddy:2-alpine      running
semalink-api-rabbitmq-1 rabbitmq:3.13-...   running (healthy)
semalink-api-api-1      semalink-api-api    running

Check API startup logs

docker compose -f docker-compose.staging.yml logs api

You should see:

[entrypoint] Running database migrations...
[entrypoint] Starting API server...
Server listening at http://0.0.0.0:3000

If migrations fail, the container will exit (due to set -e in entrypoint.sh). Check DATABASE_URL in .env.

Health check via Caddy

curl -k https://localhost/health

Expected: {"status":"ok"}

The -k flag skips TLS verification (Caddy uses a self-signed cert on port 443 with tls internal). Cloudflare sits in front and terminates the public HTTPS, so the self-signed cert is only ever seen by Cloudflare's edge servers.

Health check via public domain

curl https://staging-arc.semalink.africa/health

Expected: {"status":"ok"}

This hits Cloudflare → Caddy → API.

---

Service Architecture Detail

Docker Compose services

Three services run in a single Compose project (docker-compose.staging.yml):

caddy (image: caddy:2-alpine)

• The only container with ports bound to the host: 80:80 and 443:443
• Reads ./Caddyfile (mounted read-only)
• Persistent volume caddy_data stores ACME certificates and Caddy's internal state
• Forwards all HTTP and HTTPS traffic to api:3000 via Docker's internal DNS

rabbitmq (image: rabbitmq:3.13-management-alpine)

• Non-default user semalink — the default guest:guest credentials are blocked for remote connections
• Ports 5672 (AMQP) and 15672 (management UI) bound to the host for local debugging
• Healthcheck: rabbitmq-diagnostics ping — the api container waits for this before starting
• No external broker or managed service — runs entirely on the droplet

api (built from Dockerfile)

• Port 3000 is internal only — not bound to the host
• Receives traffic only from the Caddy container on Docker's bridge network
• depends_on: rabbitmq: condition: service_healthy — Compose will not start this container until RabbitMQ passes its healthcheck

Caddy reverse proxy

:443 (tls internal) → reverse_proxy api:3000
:80               → reverse_proxy api:3000

tls internal tells Caddy to generate a self-signed certificate using its built-in CA. This is required because Cloudflare's Full SSL mode connects to the origin server on port 443 and validates that a TLS handshake succeeds (though it does not verify the cert's CA). Without this, Cloudflare would get a connection refused on 443 and return a 521 error.

If you ever switch Cloudflare SSL mode to Full (strict), you'll need a real certificate. Use tls /path/to/cert /path/to/key in the Caddyfile or configure an ACME provider.

Docker image — multi-stage build

Stage 1 — builder (node:22-alpine)
  ├── npm ci (installs all deps including devDeps)
  ├── tsc (compiles TypeScript → dist/)
  └── output: dist/

Stage 2 — runtime (node:22-alpine)
  ├── npm ci (installs deps again — devDeps included, needed for drizzle-kit)
  ├── ENV NODE_ENV=production (set AFTER npm ci so devDeps are not skipped)
  ├── Copies dist/ from builder
  ├── Copies src/db/migrations/ from builder
  └── ENTRYPOINT entrypoint.sh

The runtime stage uses npm ci (not npm ci --omit=dev) because drizzle-kit — a dev dependency — is required at container startup to run database migrations. NODE_ENV=production is set after npm ci deliberately so npm does not skip dev dependencies.

Entrypoint

scripts/entrypoint.sh:

#!/bin/sh
set -e
echo "[entrypoint] Running database migrations..."
npx drizzle-kit migrate
echo "[entrypoint] Starting API server..."
exec node dist/server.js

• set -e: any non-zero exit code aborts the script — if migrations fail, the container exits and Docker restarts it (per restart: unless-stopped), rather than starting the server against an inconsistent schema
• exec replaces the shell process with Node so signals (SIGTERM from docker stop) are delivered directly to the Node process

---

Troubleshooting

Container exits immediately

docker compose -f docker-compose.staging.yml logs api

Most likely causes:

• Migration failure: check DATABASE_URL in .env — wrong password, wrong host, or network issue
• Port conflict: another process is using port 3000 on the host (though port 3000 is internal only, so this shouldn't happen)
• RabbitMQ not ready: if you start the API standalone without the healthcheck, it fails because AMQP connection is refused

Caddy 521 (Connection Refused)

Cloudflare could not reach port 80 or 443 on the droplet. Check:

• docker compose ... ps — is Caddy running?
• ufw status — is port 443 open? Run ufw allow 443 if not
• ss -tlnp | grep ':443' — is something bound to 443?

cannot open '.git/FETCH_HEAD': Permission denied

The repo directory is owned by root but the deploy user is doing git pull. Fix:

chown -R deploy:deploy /opt/semalink-api

RabbitMQ management UI

Access at http://<droplet-ip>:15672 using credentials semalink / <RABBITMQ_PASS>. Useful for inspecting queues, exchanges, and message rates during debugging.

Port 15672 is directly bound to the host. If you want to restrict access, add a UFW rule: ufw allow from <your-ip> to any port 15672.