Puppet Stage: a browser puppet that lip-syncs to local text-to-speech and takes stage directions, live or from a screenplay.
Puppet Stage — notes for AI sessions
Browser puppet with Rhubarb-based lip-sync and a cue-driven direction system. Read README.md first for user-facing behavior. This file is the map of the code and the invariants that are easy to break.
Layout
server.js— the whole backend, zero npm deps. TTS rendering + Rhubarb
viseme extraction (prepareSpeech, prepareExternalAudio), screenplay parser/runner (parseScript, runScript), SSE cue broadcast, static files.
public/stage.js— the whole frontend runtime.Puppetis now per-frame
(mouth shapes, actions, walking, camera views, idle breathing/gaze/blink, Web Audio speech playback); a Stage manager owns the map of frames and routes cues by cue.frame. Stage boots one implicit main frame (slot full, default character) so existing behavior is unchanged when no frame cue is ever sent. See docs/design/frames-and-scenes.md.
public/index.html— layout and CSS only, no logic beyond the?shothook.characters/<name>/{puppet.svg, manifest.json}— data-only characters.
pip is the minimal example, ava exercises every manifest feature, bo is a second (vintage rubber-hose-style) character for two-frame scenes.
assets/{backgrounds,props,overlays}/<id>.<ext>— non-character assets,
served like characters/. An id resolves format-agnostically (.svg first, then raster) so an AI-generated PNG drops into the same slot as a hand-authored SVG with no code change.
examples/<name>.txt— runnable screenplays (paste into the Screenplay box
or POST /api/script); troll-fable.txt exercises frames, lower-thirds, captions, and transitions together. See examples/README.md.
cache/— generated audio + viseme JSON, keyed by content hash. Disposable.tools/— Rhubarb binary, installed bysetup.sh. Both gitignored.integrations/dbbasic/— the app packaged for dbbasic-object-server
(three Python objects; see its README). stage.py is GENERATED by build.py from public/ + characters/ + shim.js — never edit it by hand, and re-run build.py after frontend/character changes. NOTE: test the package against a throwaway clone of the object server — never a live or shared checkout you don't own.
- The screenplay grammar (including frame/scene/overlay directions, plus
captions on|off and iris|fade in|out [ms] transition cues) lives in BOTH server.js (parseScript / directionCue) and integrations/dbbasic/shim.js. Keep them in sync, then re-run build.py.
Integration points (in order of preference)
- External TTS audio →
POST /api/speak-audio?text=<transcript>with raw
audio bytes as the body (any ffmpeg-readable format). Runs Rhubarb, plays on every connected stage. The text hint noticeably improves accuracy.
- Text via local TTS →
POST /api/say {text, engine, voice, rate}. - Gestures/movement/camera →
POST /api/cuewith `{type: action|walk|look|
view|character, ...} — see handleCue` in stage.js for the full set.
- Scripted sequences →
POST /api/script {script}(screenplay format, see
README) — the server paces cues; /api/script/stop cancels.
- Building a different frontend → consume
GET /api/events(SSE); every
cue the stage reacts to flows through it.
Invariants — do not break casually
- Audio plays in the browser, never on the server. Sync depends on
scheduling mouth shapes against the Web Audio clock (ctx.currentTime).
- Speech playback must use Web Audio buffer sources, not
<audio>.
Speak cues arrive over SSE (outside user gestures); Safari blocks HTMLMediaElement.play() there. The pointerdown/keydown listener in stage.js resumes the context AND starts a silent buffer — Safari needs both. Symptom of regression: lips move, no sound, Firefox works.
- Characters are data-only. The runtime animates whatever the manifest
declares; never special-case a character name in JS. New capabilities go into the manifest contract (document them in README) with safe defaults so old characters keep working.
- Viseme set is Rhubarb's A–H + X (rhubarb runs with
--extendedShapes GHX). Every character SVG needs all nine #mouth-* groups; exactly one is shown at a time via display toggling.
- Speech results are cached in
cache/by content hash — changing TTS
flags/pipeline won't take effect for previously-spoken lines unless the hash inputs change or the cache is cleared.
- Server pacing of scripts uses duration estimates (
ACTION_SECONDS); the
browser is the source of truth for how animations actually look.
Verifying changes without a visible browser
Headless Firefox works even while regular Firefox is open, via a separate profile (Chrome is not installed on this machine):
/Applications/Firefox.app/Contents/MacOS/firefox --headless --new-instance \
--profile /tmp/ffprof --screenshot out.png --window-size=1280,900 \
'http://localhost:3123/?shot'
?shot makes the page hold its load event ~3 s (via /slowpx) so the screenshot captures the booted stage. To capture a mid-scene state, POST cues on a delay in a background subshell before launching the screenshot. Headless Firefox cannot play audio, so audio: not started yet in a screenshot is expected, and mouth movement can't be captured this way — verify timelines via the JSON returned by /api/say instead.