How Helen Gets Made A making-of, not the live show Watch her live

Workshop · Hosting · what runs where

What runs where, and the two Caddy blocks that serve it

The first deploy of the show site returned 403 on every URL: an archive built on another machine, extracted as root, every file mode 0700, and Caddy running as caddy. That outage is now one line in the deploy script, a second outage is another, and four more became the rules at the bottom. The box they run on is a one-vCPU VPS with 3.8 GB of RAM that does everything that touches the public; a desktop PC at home does everything that touches video (Frigate, the detector, the 24/7 streams, the Shorts pipeline).

Everything below is the VPS. The home side is on the Windows page.

Helen, a dilute tortoiseshell calico, looking into the camera
The public never touches the box that watches her. It touches a small server that serves HTML, proxies JPEGs, and takes payments.

There are two sites. helencam.com, the one you are reading, is hand-written HTML in a directory. helenthecatlive.com is a Vite build (the three.js court) plus a Node backend for payments, the treat path and the live-camera proxy. Both are served by one Caddy on one small VPS. Splitting it that way means the machine that can see the cameras is never the machine that faces the internet, and the one that faces the internet is small enough that there is nothing on it worth taking.

Topology

Who runs what
HOME (Windows 11 desktop, i7-8700, WSL2)              VPS (1 vCPU, 3.8 GB, 48 GB disk, Ubuntu)
  docker: frigate 0.17          :5000 :8554             caddy ── helencam.com          static  /var/www/helencam-howto
  docker: home-assistant        :8123                         ── helenthecatlive.com   static  /var/www/helenthecatlive/current/dist
  systemd: ov-gpu-detector      :5555 (zmq)                                            /api/*, /live-cam/* → 127.0.0.1:5178
  scheduled tasks: shorts pipeline, 24/7 streams,             ── stats.helencam.com    → 127.0.0.1:3117 (Umami)
                   boot + keepalive                    pm2:     helenthecatlive        node server/api.mjs   127.0.0.1:5178
                                                       pm2:     helen-cam-proxy       express               127.0.0.1:3113 ──┐
          ▲ Tailscale only; no ports forwarded         systemd: ha-chain              socat 127.0.0.1:8472 ─────────────────┼─▶ home :8123
          └────────────────────────────────────────────systemd: helencam-suggest      python 127.0.0.1:3127                │
                                                       systemd: caddy, postgresql (Umami), tailscaled, fail2ban            └─▶ home :5000

Everything that listens on the VPS binds to loopback and Caddy is the only thing on 80 and 443. The two arrows to the home box go over Tailscale; the home router forwards nothing. The VPS never decodes a frame of video: the live cameras reach the site as JPEGs through the cam proxy, and the 24/7 video goes straight from the home box to YouTube.

helencam.com — a static site with three tiny APIs

/etc/caddy/Caddyfile — the helencam.com block
helencam.com, www.helencam.com {
	encode gzip
	handle /<GSC_VERIFICATION_FILE>.html {
		respond "google-site-verification: <GSC_VERIFICATION_FILE>.html" 200
	}
	handle /<INDEXNOW_KEY>.txt {
		respond "<INDEXNOW_KEY>" 200
	}
	handle_path /helencam/* {
		reverse_proxy 127.0.0.1:3113
	}
	@suggestapi path /suggest/list /suggest/send
	handle @suggestapi {
		reverse_proxy 127.0.0.1:3127
	}
	@shop path /shop /shop/*
	redir @shop https://helenthecatlive.com/shop/ permanent
	@about path /about /about/*
	redir @about https://helenthecatlive.com/about/ permanent
	@livepaths path /live /live/* /watch /watch/*
	redir @livepaths https://helenthecatlive.com/live-cat-cams/ permanent
	@gone path /operator /operator/* /cart /cart/* /checkout /checkout/*
	handle @gone {
		header X-Robots-Tag "noindex, nofollow"
		respond "Gone" 410
	}
	handle {
		header Cache-Control "no-store, must-revalidate"
		header Clear-Site-Data "\"cache\""
		root * /var/www/helencam-howto
		try_files {path} {path}/index.html /index.html
		file_server
	}
}

Line by line, the parts that are not obvious:

  • Verification files as respond. Search Console and IndexNow want a file at the root with a token in it. Answering from the Caddyfile means a redeploy of the site directory can never delete them.
  • handle_path /helencam/* strips the prefix and hands the rest to the cam proxy on 3113, which is how this site's refreshing stills work. handle_path, not handle, or the proxy sees /helencam/api/... and 404s.
  • The suggestion inbox is a 100-line Python ThreadingHTTPServer under systemd, only reachable at two paths. A static site with one form does not need a framework.
  • The redirect map exists because this domain used to be the show as well as the making-of. When the show moved to its own domain every old storefront path got a 301 to its new home, and the paths that had no new home (/operator, /cart, /checkout) return a real 410 with noindex, so search engines drop them instead of retrying forever.
  • try_files {path} {path}/index.html /index.html means nested folders work with no build step (/frigate/windows/setup/ is just a directory with an index.html), and also that an unknown URL returns the homepage with a 200. That is a trade we accepted for simplicity; if you want real 404s, drop the last argument and add a handle_errors block like the second site has.
  • no-store plus Clear-Site-Data: "cache" on every HTML response. The site is edited by hand and deployed often; a visitor should never see yesterday's page. The cost is one extra fetch per page view on a site that weighs almost nothing.

helenthecatlive.com — static build plus a Node API

The second site has more going on: a 3D scene with multi-megabyte models, hashed asset filenames, live JPEGs, an API, and a real 404. Its block lives in its own file and is pulled in with import sites/*.caddy from the main Caddyfile.

/etc/caddy/sites/helenthecatlive.caddy
helenthecatlive.com, www.helenthecatlive.com {
	@www host www.helenthecatlive.com
	redir @www https://helenthecatlive.com{uri} 308

	root * /var/www/helenthecatlive/current/dist
	encode gzip zstd {
		match {
			header Content-Type text/*
			header Content-Type application/javascript*
			header Content-Type application/json*
			header Content-Type application/manifest+json*
			header Content-Type application/wasm*
			header Content-Type image/svg+xml*
			header Content-Type model/gltf-binary*
		}
	}
	header {
		X-Content-Type-Options nosniff
		Referrer-Policy strict-origin-when-cross-origin
		X-Frame-Options SAMEORIGIN
		Permissions-Policy "camera=(), microphone=(), geolocation=()"
	}
	@hashed path /assets/*
	header @hashed Cache-Control "public, max-age=31536000, immutable"
	@cams path /cams/*
	header @cams Cache-Control "public, max-age=60"
	@day path /models/* /site/* /merch/* /vendor/* /icons/*
	header @day Cache-Control "public, max-age=604800, stale-while-revalidate=86400"
	@html path *.html /
	header @html Cache-Control "no-cache"
	@glb path *.glb
	header @glb Content-Type model/gltf-binary
	@wasm path *.wasm
	header @wasm Content-Type application/wasm
	@mp4 path *.mp4
	header @mp4 Content-Type video/mp4
	handle /api/* {
		reverse_proxy 127.0.0.1:5178
	}
	handle /live-cam/* {
		reverse_proxy 127.0.0.1:5178
	}
	handle {
		try_files {path} {path}/
		file_server
	}
	handle_errors {
		@notFound `{err.status_code} == 404`
		handle @notFound {
			rewrite * /404.html
			file_server
		}
	}
}
The cache tiers, and why each one
PathHeader and reason
/assets/*immutable, 1 year. Vite puts a content hash in every filename, so a changed file is a new URL. The 648 kB three.js chunk is fetched once per browser, ever.
/cams/*60 s. Static copies of the latest camera frames, written by the home box. Sixty seconds is the honest freshness, and it lets this site's pages show stills without hitting the rate-limited proxy.
/models/* /site/* /merch/*7 days + stale-while-revalidate. Models and textures change rarely and carry a ?v= query when they do. The stale-while-revalidate day means a returning visitor never waits on a 738 kB GLB.
HTMLno-cache: revalidate every time, but conditional requests still get a 304.
.glb .wasm .mp4Explicit MIME types. Caddy's defaults are fine for most, but model/gltf-binary is what lets the encode matcher above compress models; gzip takes another ten percent or so off a meshopt-compressed file.

The root points at current/dist. current is a symlink, and that symlink is the entire deploy mechanism.

Releases: build into a fresh directory, flip one symlink

/var/www/helenthecatlive/ on the VPS
/var/www/helenthecatlive/
├── <YYYYMMDDHHMMSS>/        three older releases, kept for rollback
├── <YYYYMMDDHHMMSS>/
├── <YYYYMMDDHHMMSS>/
├── 20260902150437/          ← newest (four on disk at the time of writing)
│   ├── dist/                what Caddy serves
│   ├── server/api.mjs       what pm2 runs
│   ├── package.json
│   ├── .env     -> /var/lib/helenthecatlive/.env            (outside the release: survives deploys)
│   └── .runtime -> /var/lib/helenthecatlive/runtime         (treat stamps + ledger, same reason)
└── current -> /var/www/helenthecatlive/20260902150437

The rules this layout enforces, in the order they were learned:

  1. Git should be the source of truth, and here it is not yet. The rule is that every release directory is a build of a commit and nothing is edited in place on the server. We learned it on a different site where a hot-patched server directory was the only copy of a month of work. On this box the show site's repository has no initial commit yet, and the making-of site you are reading is a plain directory whose rollback is a tarball taken before each deploy. The release directories and the symlink swap below work without git; the commit history is the part still owed.
  2. Build into a fresh directory, never over the live one. A half-written dist/ under a live root serves half-written pages. A new timestamped directory is invisible to Caddy until the symlink moves.
  3. Promote with an atomic symlink swap. ln -sfn new current.tmp && mv -Tf current.tmp current. The mv -T is a single rename(2), so there is no instant where current does not exist. Rollback is the same command pointed at the previous directory.
  4. Promotion is a deliberate flip, not a loop. A push may trigger a build; it must not trigger an automatic rebuild-and-restart on the box. We ran that loop once on another server and it is one of the four incidents below.
  5. Verify with curl after every deploy. The script's last act is to fetch five URLs and require 200 from each. "Success" from a script is not evidence; a 200 from the public hostname is.
scripts/deploy.sh (in the site repo; runs from the dev machine)
#!/usr/bin/env bash
#   ./scripts/deploy.sh            build, upload, swap, restart, verify
#   ./scripts/deploy.sh --rollback swap back to the previous release
set -euo pipefail

HOST="${HOST:-root@<PUBLIC_IP>}"
ROOT="${REMOTE_ROOT:-/var/www/helenthecatlive}"
APP="${PM2_APP:-helenthecatlive}"
KEEP="${KEEP:-5}"
ENV_FILE="${REMOTE_ENV:-/var/lib/helenthecatlive/.env}"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
SSH_OPTS=(-o StrictHostKeyChecking=accept-new -i "${SSH_KEY:-$HOME/.ssh/<DEPLOY_KEY>}")
rsh() { ssh "${SSH_OPTS[@]}" "$HOST" "$@"; }

verify() {
  local bad=0
  for path in / /shop/ /about/ /checkout/ /data/catalog.json; do
    local code
    code=$(curl -s -o /dev/null -w '%{http_code}' "https://helenthecatlive.com${path}" || echo 000)
    printf '  %-22s %s\n' "$path" "$code"
    [[ "$code" == "200" ]] || bad=1
  done
  return $bad
}

if [[ "${1:-}" == "--rollback" ]]; then
  prev=$(rsh "ls -1d $ROOT/*/ | sed 's#/\$##' | sort | grep -B1 \"\$(readlink -f $ROOT/current)\" | head -1")
  [[ -n "$prev" ]] || { echo "no earlier release found" >&2; exit 1; }
  echo "==> Rolling back to $prev"
  rsh "ln -sfn '$prev' $ROOT/current.tmp && mv -Tf $ROOT/current.tmp $ROOT/current"
  rsh "cd '$prev' && set -a && . '$ENV_FILE' && set +a && pm2 delete $APP >/dev/null 2>&1; pm2 start server/api.mjs --name $APP --cwd '$prev' --update-env >/dev/null && sleep 3 && pm2 save >/dev/null"
  sleep 2; verify && echo "Rolled back." || { echo "STILL FAILING after rollback." >&2; exit 1; }
  exit 0
fi

echo "==> Building"
cd "$HERE"
npm run build
npm run check-shop

[[ -f dist/index.html ]] || { echo "dist/index.html missing - refusing to deploy" >&2; exit 1; }
[[ ! -d dist/ref && ! -d dist/clips ]] || { echo "dist still has ref/ or clips/ - prune did not run" >&2; exit 1; }

REL="$(date -u +%Y%m%d%H%M%S)"
TGT="$ROOT/$REL"
PREV="$(rsh "readlink -f $ROOT/current" || true)"
echo "==> Release $REL   (previous: ${PREV:-none})"

# --owner/--group/--mode so the archive never carries this machine's identity.
tar -czf /tmp/helen-release.tgz --owner=0 --group=0 --mode='u+rw,go+r' dist server package.json deploy
rsh "mkdir -p '$TGT'"
scp "${SSH_OPTS[@]}" /tmp/helen-release.tgz "$HOST:$TGT/release.tgz"
rm -f /tmp/helen-release.tgz

echo "==> Unpacking and preflighting"
rsh bash -s <<REMOTE
set -euo pipefail
cd '$TGT'
tar -xzf release.tgz && rm -f release.tgz
chown -R root:root '$TGT'
find '$TGT' -type d -exec chmod 755 {} +
find '$TGT' -type f -exec chmod 644 {} +
# .env and .runtime live outside the release so keys and fulfilment records
# survive a deploy instead of being wiped with the old directory.
mkdir -p /var/lib/helenthecatlive/runtime
ln -sfn '$ENV_FILE' '$TGT/.env'
rm -rf '$TGT/.runtime'
ln -sfn /var/lib/helenthecatlive/runtime '$TGT/.runtime'
test -f '$TGT/dist/index.html'
node --check '$TGT/server/api.mjs'
sudo -u caddy test -r '$TGT/dist/index.html' || { echo 'caddy cannot read the new release - NOT swapping'; exit 1; }
echo PREFLIGHT_OK
REMOTE

echo "==> Swapping and restarting"
rsh bash -s <<REMOTE
set -euo pipefail
ln -sfn '$TGT' $ROOT/current.tmp
mv -Tf $ROOT/current.tmp $ROOT/current
cd '$TGT'
set -a; . '$ENV_FILE'; set +a
pm2 delete $APP >/dev/null 2>&1 || true
pm2 start server/api.mjs --name $APP --cwd '$TGT' --update-env >/dev/null
sleep 3
pm2 save >/dev/null
ls -1d $ROOT/*/ | sed 's#/\$##' | sort -r | tail -n +$((KEEP + 1)) | xargs -r rm -rf
REMOTE

echo "==> Verifying"
sleep 2
if verify; then
  echo "Deployed $REL."
else
  echo "Verification FAILED - rolling back." >&2
  "$0" --rollback
  exit 1
fi

Two lines in that script each exist because of an outage:

  • sudo -u caddy test -r … || exit 1. The first deploy extracted an archive built on another machine, as root, and every file came out mode 0700 owned by root. Caddy runs as caddy. Every URL on the site returned 403. Now ownership and modes are normalised, the archive is built with --owner=0 --mode=go+r so it cannot carry the dev machine's identity, and the swap refuses to happen unless the caddy user can read index.html.
  • set -a; . "$ENV_FILE"; set +a before pm2 start … --update-env. The API does not read .env itself, and pm2 has no --env-file. A plain pm2 restart started the server with no Stripe keys and it came up "healthy": /api/health said ok, the shop rendered, and checkout was silently disabled. The env has to be sourced into the shell that starts pm2, and pm2 delete + pm2 start is used instead of restart so the process picks up the new --cwd and environment.

The tail -n +$((KEEP + 1)) | xargs rm -rf line is the retention. Five releases stay; older ones go. That number matters more than it looks, for the reason in the first incident below.

systemd or pm2: who owns what

Process management on the VPS
ServiceManager, and why
Caddy, PostgreSQL, tailscaled, fail2bansystemd, from the distro packages. Never touched by a deploy.
ha-chain (socat tunnel to Home Assistant)systemd. One ExecStart, Restart=always, ordered After=tailscaled.service. Shown on the treat page.
helencam-suggest (the suggestion inbox)systemd, as www-data. A Python stdlib server with no dependencies has no business in a Node process manager.
helenthecatlive (the site API)pm2. It is replaced on every deploy with a new --cwd, and pm2's delete/start/save cycle is the least friction for that.
helen-cam-proxypm2. Node, restarted rarely, reads a .env next to itself.
Umamipm2, behind stats.helencam.com. Its Postgres is systemd.

The rule: anything with a fixed path and a fixed environment goes in systemd; anything that changes directory on every release goes in pm2, and it is written down which is which. The worst hour we have spent on this box was restarting the wrong manager, watching "restarted" scroll by, and wondering why nothing changed. On a machine with both, systemctl is-active and pm2 ls are the first two commands of every incident.

/etc/systemd/system/helencam-suggest.service — the shape of a one-file service
[Unit]
Description=helencam public suggestion inbox
After=network.target

[Service]
ExecStart=/usr/bin/python3 /opt/helencam-suggest/suggest.py
WorkingDirectory=/opt/helencam-suggest
Restart=always
User=www-data
Group=www-data

[Install]
WantedBy=multi-user.target

Four incidents, four rules

These did not all happen on this VPS. They happened on servers we run the same way, and each one changed something on this page.

  1. A per-minute snapshotter with no retention filled a production disk to 100 %. A backup-before-patch helper took a copy of a site directory every minute and never deleted one. Within days the disk was full, the database stopped writing, and every site on the box went down at once. The rule: any automation that creates files must also prune them. Keep the newest N, throttle the rate, and alert at 85 % disk. That is why KEEP=5 is in the deploy script and not left to memory.
  2. Caddy with PrivateTmp=true could not reload. A systemctl reload caddy failed with an error about /tmp, while caddy validate said the config was fine. The unit's private /tmp namespace was the problem. The fix is a drop-in with PrivateTmp=no, or reloading with caddy reload --config /etc/caddy/Caddyfile directly. Related, and easier to hit: a file in sites/ that is not world-readable makes validate pass and reload fail. Files there are 0644, always.
  3. A 502 is usually DNS or Caddy, not the app. Three times now a site has shown 502 or no response, the app was blamed, and the app was fine. The first check is curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/ on the box. If loopback answers, the problem is in front of the app: the Caddy block, the certificate, or the DNS record. If loopback does not answer, then it is the app.
  4. An orphaned next start squatted the port. pm2 stop reported success; the site kept serving the old build; the new one crashed on EADDRINUSE. A child process had outlived its parent and was still bound to the port, and a pm2 resurrect in cron was fighting the manual stop. The rule: when a port will not free, find the PID with ss -ltnp and kill that, and make sure only one thing is allowed to bring services back after a reboot.

Is one vCPU enough?

Yes, because the VPS never touches video. At the time of writing it runs Caddy, Node, a Python inbox, Umami and its PostgreSQL, Tailscale and fail2ban on 3.8 GB of RAM and 48 GB of disk (38 % used), and the busiest thing it does is hand out 648 kB of JavaScript and a 738 kB cat to each new visitor, compressed. The moment you put ffmpeg or an object detector on a box like this you need a different box; that is the whole reason the detector lives at home on an Intel iGPU and the streams go to YouTube from there.