Both files run on the same desktop as Frigate, against http://127.0.0.1:5000. helen_pipeline.py (2,059 lines) is the library: the Frigate client, the vision and caption calls, the caption renderer, the YouTube uploader, the state files. helen_multiangle.py (2,603 lines) imports it and owns the editorial logic: stations, visits, the shot list, the crop, the publishing clock. If you only read one thing, read the shot list; that is where the rules on the cut page became code.
What it asks Frigate for
Four endpoints. No MQTT, no webhooks, no Frigate plugin; the pipeline is a client of the same REST API the Frigate web UI uses, so nothing in Frigate had to change.
| Call | Used for |
|---|---|
GET /api/config | Camera discovery. Every camera name in the live config is matched against station aliases; unknown names are ignored or refused (below). Adding a camera to Frigate is enough to add it here. |
GET /api/events?camera=<cam>&label=cat&after=<epoch>&limit=120&has_clip=1 | Per camera, not system-wide: the system-wide list is newest-first across every detecting camera (the three house cameras have detect off and never produce events), and a busy hour at one bowl pushes a quiet camera's events off the end. |
GET /api/<cam>/start/<s>/end/<e>/clip.mp4 | Each shot, as a time-range export from continuous recording (see retention: this is why the cat cameras keep three days of everything, not just events). Frigate honours the start and overshoots the end by 1.1–2.1 s per clip, measured. |
GET /api/events/<id>, GET /api/events/<id>/snapshot.jpg | The event's bounding box and path for the crop, and the hero frame for the caption model and the thumbnail. |
FRIGATE = "http://127.0.0.1:5000"
def frigate_events_camera(camera, after_epoch, limit=120):
"""Recent cat events for one camera since after_epoch (epoch seconds)."""
url = "%s/api/events?camera=%s&label=cat&after=%s&limit=%d&has_clip=1" % (
FRIGATE, urllib.parse.quote(str(camera)), repr(float(after_epoch)), limit)
try:
return http_json(url, timeout=20)
except Exception as e:
log("frigate camera poll error (%s): %s" % (camera, e))
return []def fetch_range(cam, start, end, dest, timeout=180):
"""One shot, straight out of Frigate's continuous recording."""
url = "%s/api/%s/start/%s/end/%s/clip.mp4" % (
P.FRIGATE, cam, repr(float(start)), repr(float(end)))
dst, n = P.http_download(url, dest, timeout=timeout)
if n <= 2000 or not Path(dst).exists() or Path(dst).stat().st_size <= 2000:
raise RuntimeError("empty range clip for %s" % cam)
return dstVisits: events become one thing
Frigate emits an event per camera per object. A cat that drinks, walks to the food bowl and eats is three or four events on two cameras. The pipeline's unit is the visit: events from any camera, sorted by start time, joined while the next one starts within 45 s of the latest end seen so far, capped at seven minutes. Events still in progress (no end_time) are left for the next run.
SESSION_GAP_SEC = _envf("HELEN_MA_GAP", 45.0) # events this close = one visit
SESSION_MAX_SEC = _envf("HELEN_MA_SPAN", 420.0) # hard cap on one visit's span
def build_visits(events, cam2st, gap=None, max_span=None):
"""Cluster finished cat events across all Helen cams into visits.
A visit is a run of events where each next event starts no later than
`gap` seconds after the latest end seen so far. Because the clustering is
done across cameras, two angles of the same moment land in one visit, and
so does a cat that walks water -> food inside the gap.
"""
gap = SESSION_GAP_SEC if gap is None else gap
max_span = SESSION_MAX_SEC if max_span is None else max_span
usable = []
for ev in events or []:
try:
cam = ev.get("camera")
if cam not in cam2st:
continue
if not ev.get("has_clip"):
continue
if ev.get("start_time") is None or ev.get("end_time") is None:
continue # still in progress -- wait for a later run
usable.append(ev)
except Exception:
continue
usable.sort(key=lambda e: float(e["start_time"]))
visits = []
cur = []
cur_end = None
cur_start = None
for ev in usable:
s = float(ev["start_time"])
e = float(ev["end_time"])
if cur and (s <= cur_end + gap) and ((e - cur_start) <= max_span):
cur.append(ev)
cur_end = max(cur_end, e)
else:
if cur:
visits.append(cur)
cur = [ev]
cur_start = s
cur_end = e
if cur:
visits.append(cur)
return visitsA visit is eligible if the cat was in frame for at least six seconds (the union of the event spans, not the padded clip length: "a 2-second blip padded to 12 s is still a 2-second video of a cat"), and then either it touches the Water station, which is always eligible, or its whole span plus padding clears a 20-second floor. When a visit publishes, every event in it is marked processed, so the same drink cannot become a second Short from another angle.
The shot list, and the rule that a cut never goes backwards
Each event becomes a beat: camera, start, end, score. Beats on the same camera that touch within two seconds merge (Frigate regularly emits a short event nested inside a long one on the same camera). Then:
SHOT_MIN_SEC = _envf("HELEN_MA_SHOT_MIN", 2.5)
SHOT_MAX_SEC = _envf("HELEN_MA_SHOT_MAX", 5.0)
MAX_SHOTS = int(_envf("HELEN_MA_MAX_SHOTS", 4))
SHOT_PAD_SEC = _envf("HELEN_MA_SHOT_PAD", 1.5) # lead-in/out around each beat
# episode length comes from helen_pipeline: MIN_EPISODE 14 s, MAX_EPISODE 20 s
def build_shotlist(visit, cam2st, min_total=None, max_total=None, lead=None,
open_cam=None, max_shots=None):
"""Chronological (camera, start, end) shots covering the visit.
Rules, in order:
1. One beat per camera-run, in the order the cat triggered them.
2. Consecutive beats from the SAME camera are merged -- a cut back to the
angle you are already on is not a cut.
3. Too many beats to fit the Shorts window: keep the highest-scoring ones,
then restore chronological order. Never more than MAX_SHOTS.
4. Shot lengths are clamped to [SHOT_MIN, SHOT_MAX], then scaled so the
total lands inside MIN_EPISODE..MAX_EPISODE (14-20s as of
2026-08-02; was 8-15s, which ended just below the best
performing length bucket across the first 81 uploads).
""" # 2026-08-25 HARD RULE: a cut never goes backwards. The centring branch
# above places a short beat around its own timestamp with no regard for
# where the edit already is, which is what put a 14.9s-earlier shot last
# in an aired episode. Clamp to the cursor; if the beat's own footage is
# then behind the edit, drop the beat rather than time-travel.
if cursor is not None and start + 0.001 < cursor:
start = cursor
end = start + dur
if shots:
_ov = min(end, b["end"]) - max(start, b["start"])
if _ov < min(1.5, dur * 0.4):
log("shot dropped: %s beat is behind the edit (%.1fs of usable "
"footage) -- keeping the cut chronological" % (b["cam"], _ov))
continue
cursor = endThat is the whole fix for the backwards Short (the cut page tells the story from the viewer's side). Every shot now starts at or after the previous shot's end on the one clock all cameras share, and a beat whose footage would fall behind that point is dropped instead of reordered. The self-test suite has a case for it (13b. cuts never travel backwards).
The vertical crop
Each shot is cropped to 1080×1920 before the concat, at full source resolution, so every angle keeps its native detail and each camera can carry its own horizontal bias (the Face camera, for instance, is steered away from the bowl edge). The crop centre comes from the event: GET /api/events/<id> gives the box and, if Frigate tracked one, the path, and a piecewise-linear pan expression follows it.
def _pan_expr(knots):
"""ffmpeg per-frame expression for cx(t): piecewise linear through knots."""
expr = "%.4f" % knots[-1][1]
for (ta, ca), (tb, cb) in reversed(list(zip(knots, knots[1:]))):
seg = ("(%.4f+(%.4f)*((t-%.3f)/%.3f))"
% (ca, cb - ca, ta, max(0.001, tb - ta)))
expr = "if(lt(t,%.3f),%s,%s)" % (tb, seg, expr)
return "if(lt(t,%.3f),%.4f,%s)" % (knots[0][0], knots[0][1], expr) # 2026-08-24: -t BEFORE -i, so each shot contributes exactly the length
# build_shotlist assigned it. fetch_range asks Frigate for start->end and
# Frigate returns more (measured +1.1s to +2.1s per shot, +5.84s over four),
# and this loop used to ingest every file whole - which is how a 20.0s
# shotlist became a 25.1s Short. Frigate honours the START, so trimming from
# the front of each file keeps exactly the chosen window.
for _i, p in enumerate(shots_paths):
_d = None
try:
_d = float(durs[_i]) if durs else None
except Exception:
_d = None
if _d and _d > 0.2:
cmd += ["-t", "%.3f" % _d]
cmd += ["-i", str(p)]The caption
Two model calls per video, both resolved by role through a small resolver with an ordered fallback chain and an allowlist, so no file in the pipeline names a vendor. The vision role samples three frames across the assembled clip (not Frigate's single best-score snapshot, which captions a walk-past as a meal) and returns a sentence about what she did. The caption role turns that into the decree. The resolver, the chains and the day one vendor went down are on the AI page.
DEFAULT_ROLES = {
# Vision: must be able to see the frame.
"vision": [
"openrouter:google/gemini-2.5-flash",
"openrouter:google/gemini-2.5-flash-lite",
"together:google/gemma-3n-E4B-it",
"gemini:gemini-2.5-flash",
"anthropic:claude-haiku-4-5-20251001",
],
# Caption: short creative text, cheap tier first.
"caption": [
"openrouter:google/gemini-2.5-flash-lite",
"openrouter:google/gemini-2.5-flash",
"together:meta-llama/Llama-3.3-70B-Instruct-Turbo",
"gemini:gemini-2.5-flash-lite",
"anthropic:claude-haiku-4-5-20251001",
],
}
MAX_CALLS_PER_RUN = int(os.environ.get("HELEN_MAX_MODEL_CALLS", "60"))def make_decree(behavior, camera, frame_hint=None):
touch = cam_touch(camera)
user = (
"WHAT HELEN JUST DID:\n %s\n\n"
"Write ONE very short thought or observation about what Helen just did -- a brief "
"line, ideally 5-9 words and NEVER more than 10, warm and a little wry. You MAY fold in this single local "
"touch or omit it: '%s'. No history. Return only the line, no period needed." % (behavior, touch)
)A lane that returns a billing or quota error is marked dead for the run and the next one is tried. If the vision call fails everywhere, the description falls back to a plain sentence built from the station; if the caption call fails everywhere, the run stops rather than uploading a video with no words on it, and a separate backfill task fills the slot from a library whose captions come from a local file, not an API. The gold text is burned on with Pillow and ffmpeg at 1080×1920; the rules for where it sits are on the cut page, and the prompt, the guardrails around it and the person who still reads every line are on the AI page.
Guardrails, every one of which was earned
| Gate | What it does |
|---|---|
NEVER_PUBLISH deny list | Camera names that may record and be watched but may never reach YouTube. Checked first, before the station map, so no future edit can rescue one. Written after an audit found that an unrecognised camera used to be adopted as a new station and green-lit. |
may_publish() fails closed | Returns False for None, for an empty string, and for any station not in the known table. It used to return True for anything whenever the restriction env var was unset, which was always, in production. |
HOUSE_DENY prefixes | front_door, back_yard, driveway, porch, remote_ and friends are ignored at discovery. The house cameras exist in Frigate for the house; see naming. |
| Person guard | At the single choke point every publishing path goes through, a visit that overlaps a Frigate person event on the same camera is refused. This one fails closed even if Frigate cannot be reached: 10 of 75 eligible visits over 72 hours had a person in them, which is why person is tracked on the cat cameras at all. A separate task every 20 minutes re-checks what has already been posted. |
| Minimum cat time | MIN_CAT_SEC = 6.0: the union of event spans, not clip length. |
| Never the same station twice running | A hard rule, not a score penalty. Eleven consecutive Water Cam uploads on 2026-07-31 earned it. If only that station is available, the run holds. |
| Rate | MAX_PER_RUN = 1, COOLDOWN_MIN = 25 between uploads, and a 45-minute window per publishing slot that can be consumed once per day. |
| Age | Events older than 30 hours are ignored in the multi-angle cutter (the library default is 12). She is nocturnal: a week of Frigate events showed peaks at 21:00, 01:00, 22:00 and 00:00 and nothing at all 09:00–14:00, so a 01:00 visit has to survive until the 19:00 slot. |
| A model's opinion | Never asked. None of the gates in this table calls a model, and the two model calls that exist sit downstream of every one of them. Two of these gates were found missing by a coding assistant reading this file on 2026-08-23: may_publish() then returned true for anything, and two cameras were being discarded every run as unrecognised. The same session wrote the resolver, after the caption vendor's billing limit had taken the channel dark that morning. |
| YouTube says no | uploadLimitExceeded (a 400, the channel's daily cap) pauses uploads for eight hours; quotaExceeded, invalid_grant and a few others are treated as permanent for the run instead of retried. More cameras made the pipeline more selective, not more prolific. |
The upload
svc = build("youtube", "v3", credentials=creds, cache_discovery=False)
snip = {"title": title[:95], "description": description[:4900],
"categoryId": "15"} # Pets & Animals
if tags:
# YouTube caps the tag list at 500 characters total, so trim rather than
# let the whole insert fail on a 400.
picked, used = [], 0
for t in tags:
if used + len(t) + 1 > 480:
break
picked.append(t)
used += len(t) + 1
snip["tags"] = picked
body = {
"snippet": snip,
"status": _hero_status(privacy, publish_at), # public, not made for kids
}
# chunksize=-1 sends the whole file in ONE request, which silently defeats
# resumable=True. Fine for a 3 MB clip, fatal for a 200 MB compilation.
media = MediaFileUpload(str(mp4), chunksize=8 * 1024 * 1024,
resumable=True, mimetype="video/*")
req = svc.videos().insert(part="snippet,status", body=body, media_body=media)The OAuth token is asked for the scopes it already holds (youtube.upload and youtube.force-ssl); for weeks the code requested only upload, which is why every thumbnail call failed with insufficient permissions and every video shipped with a YouTube-picked frame. A Short is just a vertical video under sixty seconds; there is no Shorts flag in the API.
What schedules it
One Windows scheduled task, HelenCamUploader2, runs python.exe helen_multiangle.py --once every fifteen minutes as SYSTEM. Most runs exit in a second with outside a publishing slot -- nothing to do. Inside a slot the run polls, builds visits, picks the best one that prefers the slot's station, and publishes at most one.
# (HH, MM, preferred station or None for "best available")
# 2026-08-02 RETIMED from 81 published videos. Median views by publish hour,
# SOLO uploads only -- batched uploads are excluded because they poison every
# bucket they touch:
#
# 05:00 150 | 06:00 94 | 07:00 150 | 08:00 44 | 09:00 72 | 10:00 32
# 11:00 4 | 12:00 2 | 13:00 2 | 15:00 0 | 19:00 159
#
# Per-hour samples are small (n=2-6), so treat this as "stop publishing into
# proven dead hours", not as a precise optimum.
# 2026-08-23 CUT FROM 6 TO 4, operator: "3 or 4" a day.
PUBLISH_SLOTS = [
(5, 0, "water"), # 05:00 measured 150; 05:30 straddled 06:00's 94
(7, 0, "food"), # 150, joint-best morning hour
(17, 0, "court"), # the overhead; unmeasured hour, ramps into the peak
(19, 0, "face"), # 159, the best hour -- given to the most productive
# camera on the system (helen_food_face, 38 events
# on 2026-08-22)
]
SLOT_WINDOW_MIN = _envf("HELEN_MA_SLOT_WINDOW", 45.0)The station is a preference, not a requirement: if there is no Court visit at 17:00 the run takes the best available and says so in the log. Local time, because the schedule is about when people watch, and the machine is in Eastern.
| Task | When · what |
|---|---|
HelenCamUploader2 | every 15 min · helen_multiangle.py --once, the page you are reading |
HelenCamDrinkingComp | daily 20:00 · helen_pipeline.py --drinking-comp: every drink of the day as one reel, "The Royal Drinking Hours" |
HelenCamBackfill | every 30 min · fills an unconsumed slot from the library when the live path had nothing |
HelenPersonPurge | every 20 min · a second pass of the person check on its own clock, independent of the uploader |
HelenCamHealth, HelenTaskGuard, HelenPublishProof | daily · every 5 min · 4×/day: health, task-liveness and publish-proof checks that page the operator instead of failing quietly |
HelenLongArcWeekly, HelenSaga, HelenNewsletterWeekly, HelenVoteTally | weekly and daily: the long-arc recap, the season serial, the newsletter, the vote count. Separate scripts, same library. |
What it does not do
- It does not review. A person looks at the posted frames on a phone after the fact and fixes what is wrong. The rules above are what keep that list short; they are not a substitute.
- It does not know what is funny. The caption model gets a sentence about what she did and a station touch. Everything it writes is short enough to be harmless when it is wrong.
- It is one machine. Frigate, the recordings, the pipeline and both live streams share a desktop. When Windows Update reboots it, the 15-minute task resumes; a slot missed during the reboot is gone.