Puppet Stage: a browser puppet that lip-syncs to local text-to-speech and takes stage directions, live or from a screenplay.

0 downloads

dbbasic-face/CLAUDE.md
Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-face.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-face.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-face.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

Dan Quellhorst Converse: characters know they have bodies 52ca998 4 months, 1 week ago History
CLAUDE.md · 5.3 KB · 93 lines

Puppet Stage — notes for AI sessions

Browser puppet with Rhubarb-based lip-sync and a cue-driven direction system. Read README.md first for user-facing behavior. This file is the map of the code and the invariants that are easy to break.

Layout

  • server.js — the whole backend, zero npm deps. TTS rendering + Rhubarb

viseme extraction (prepareSpeech, prepareExternalAudio), screenplay parser/runner (parseScript, runScript), SSE cue broadcast, static files.

  • public/stage.js — the whole frontend runtime. Puppet is now per-frame

(mouth shapes, actions, walking, camera views, idle breathing/gaze/blink, Web Audio speech playback); a Stage manager owns the map of frames and routes cues by cue.frame. Stage boots one implicit main frame (slot full, default character) so existing behavior is unchanged when no frame cue is ever sent. See docs/design/frames-and-scenes.md.

  • public/index.html — layout and CSS only, no logic beyond the ?shot hook.
  • characters/<name>/{puppet.svg, manifest.json} — data-only characters.

pip is the minimal example, ava exercises every manifest feature, bo is a second (vintage rubber-hose-style) character for two-frame scenes.

  • assets/{backgrounds,props,overlays}/<id>.<ext> — non-character assets,

served like characters/. An id resolves format-agnostically (.svg first, then raster) so an AI-generated PNG drops into the same slot as a hand-authored SVG with no code change.

  • examples/<name>.txt — runnable screenplays (paste into the Screenplay box

or POST /api/script); troll-fable.txt exercises frames, lower-thirds, captions, and transitions together. See examples/README.md.

  • cache/ — generated audio + viseme JSON, keyed by content hash. Disposable.
  • tools/ — Rhubarb binary, installed by setup.sh. Both gitignored.
  • integrations/dbbasic/ — the app packaged for dbbasic-object-server

(three Python objects; see its README). stage.py is GENERATED by build.py from public/ + characters/ + shim.js — never edit it by hand, and re-run build.py after frontend/character changes. NOTE: test the package against a throwaway clone of the object server — never a live or shared checkout you don't own.

  • The screenplay grammar (including frame/scene/overlay directions, plus

captions on|off and iris|fade in|out [ms] transition cues) lives in BOTH server.js (parseScript / directionCue) and integrations/dbbasic/shim.js. Keep them in sync, then re-run build.py.

Integration points (in order of preference)

  1. External TTS audio → POST /api/speak-audio?text=<transcript> with raw

audio bytes as the body (any ffmpeg-readable format). Runs Rhubarb, plays on every connected stage. The text hint noticeably improves accuracy.

  1. Text via local TTS → POST /api/say {text, engine, voice, rate}.
  2. Gestures/movement/camera → POST /api/cue with `{type: action|walk|look|

view|character, ...} — see handleCue` in stage.js for the full set.

  1. Scripted sequences → POST /api/script {script} (screenplay format, see

README) — the server paces cues; /api/script/stop cancels.

  1. Building a different frontend → consume GET /api/events (SSE); every

cue the stage reacts to flows through it.

Invariants — do not break casually

  • Audio plays in the browser, never on the server. Sync depends on

scheduling mouth shapes against the Web Audio clock (ctx.currentTime).

  • Speech playback must use Web Audio buffer sources, not <audio>.

Speak cues arrive over SSE (outside user gestures); Safari blocks HTMLMediaElement.play() there. The pointerdown/keydown listener in stage.js resumes the context AND starts a silent buffer — Safari needs both. Symptom of regression: lips move, no sound, Firefox works.

  • Characters are data-only. The runtime animates whatever the manifest

declares; never special-case a character name in JS. New capabilities go into the manifest contract (document them in README) with safe defaults so old characters keep working.

  • Viseme set is Rhubarb's A–H + X (rhubarb runs with

--extendedShapes GHX). Every character SVG needs all nine #mouth-* groups; exactly one is shown at a time via display toggling.

  • Speech results are cached in cache/ by content hash — changing TTS

flags/pipeline won't take effect for previously-spoken lines unless the hash inputs change or the cache is cleared.

  • Server pacing of scripts uses duration estimates (ACTION_SECONDS); the

browser is the source of truth for how animations actually look.

Verifying changes without a visible browser

Headless Firefox works even while regular Firefox is open, via a separate profile (Chrome is not installed on this machine):

/Applications/Firefox.app/Contents/MacOS/firefox --headless --new-instance \
  --profile /tmp/ffprof --screenshot out.png --window-size=1280,900 \
  'http://localhost:3123/?shot'

?shot makes the page hold its load event ~3 s (via /slowpx) so the screenshot captures the booted stage. To capture a mid-scene state, POST cues on a delay in a background subshell before launching the screenshot. Headless Firefox cannot play audio, so audio: not started yet in a screenshot is expected, and mouth movement can't be captured this way — verify timelines via the JSON returned by /api/say instead.