Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
talk.py · 78.5 KB · 1831 lines
1"""Talk: the shell as a stage instead of a transcript.
2
3Same brain as /shell -- one POST /api/ai/chat, the same prefs/model/tools,
4the same shell_commands table -- projected differently. The shell renders
5a scrolling keyboard log; Talk fills the screen with whatever the
6conversation just produced (a spoken answer or a materialized /views page)
7and reduces the transcript to a single caption strip. No server route is
8new here: this object is a second window onto the same conversation.
9
10Every turn is still recorded to shell_commands (Talk and the shell share
11that record), but Talk does NOT replay old shell_commands rows into the
12model's context on load. A model given a transcript of *any* age of
13in-context examples will imitate the habits those examples demonstrate
14even when the current instructions say otherwise -- concretely, replaying
15turns recorded before the [[view:<id>]] marker convention existed taught
16the model to keep ignoring the marker instruction, no matter how the
17system prompt was reworded. So each Talk page load starts aiHistory empty
18and it only ever accumulates turns from *this* session.
19
20_BASE_CAPABILITIES below is copied verbatim from shell.py's system prompt
21(including the views MATERIALIZE PAGES block) so the two stay in sync --
22edit both together. Talk adds one addendum on top: short spoken replies,
23and never speaking ids/urls/paths aloud.
24"""
25
26import json
27
28# Page-unique layout only; palette, chrome, and inputs come from /style, so
29# Talk reskins with the active theme like every other page.
30_STYLE = """
31/* The page must FIT the viewport: the mic, the captions and the text box
32 are one control surface, and a layout where you scroll between the nav
33 and the input is one you cannot operate. The stage takes what is left
34 after the fixed-height rows rather than claiming 85vh of its own --
35 which is what pushed the input off-screen on a tall tablet.
36 dvh where supported (iOS Safari's toolbars shrink the visible area and
37 100vh lies about it); -webkit-fill-available for older iPadOS that has
38 neither dvh nor svh; plain vh last for everything else. */
39.talkwrap { display: flex; flex-direction: column;
40 height: calc(100vh - 3.5rem);
41 height: -webkit-fill-available;
42 height: calc(100dvh - 3.5rem);
43 max-height: calc(100dvh - 3.5rem); overflow: hidden; }
44.stage { flex: 1 1 auto; min-height: 0; display: flex; align-items: center;
45 justify-content: center; padding: 1rem; overflow: hidden; }
46.stage .card { background: var(--panel); border: 1px solid var(--line); border-radius: var(--radius-md);
47 padding: 2rem; max-width: 640px; max-height: 100%; overflow-y: auto;
48 font-size: 1.4rem; line-height: 1.5; text-align: center; }
49.stage .card.placeholder { color: var(--muted); font-size: 1.1rem; }
50.stage .card a { color: var(--accent-strong); }
51.stage .card code { background: var(--panel-2); color: var(--accent-strong);
52 padding: 1px 5px; border-radius: 3px; font-size: 0.9em; }
53.stageframe { width: 100%; height: 100%; border: 1px solid var(--line); border-radius: var(--radius-md); }
54.bar { border-top: 1px solid var(--line); padding: 0.75rem 1rem; display: flex; flex-direction: column; gap: 0.5rem; }
55.captions { min-height: 2.6rem; font-size: 0.95rem; }
56.endpointhint { font-size: 0.78rem; opacity: 0.75; padding: 0 0.2rem 0.2rem; }
57#sessionpick { max-width: 11rem; font-size: 0.78rem; opacity: 0.85; }
58.cap.user { color: var(--muted); }
59.cap.assistant { color: var(--text); font-weight: 600; }
60.controls { display: flex; align-items: center; gap: 0.75rem; }
61.micwrap { position: relative; flex: 0 0 auto; }
62#mic { width: 4.5rem; height: 4.5rem; border-radius: 50%; font-size: 0.75rem;
63 background: var(--panel-2); border: 1px solid var(--line); color: var(--muted); cursor: pointer; }
64#mic.on { color: var(--danger); border-color: var(--danger); background: var(--panel); }
65/* Level meter ring: a privacy indicator as much as a VU meter -- it must
66 read as visibly dead (opacity 0, no shadow) whenever the mic is off. */
67.miclevel { position: absolute; inset: -6px; border-radius: 50%; pointer-events: none;
68 opacity: 0; box-shadow: 0 0 0 0 var(--danger); transition: opacity 80ms linear; }
69form#prompt { flex: 1; display: flex; gap: 0.5rem; }
70form#prompt input { flex: 1; }
71.backlink { color: var(--muted); font-size: 0.82rem; white-space: nowrap; }
72/* #cloudvoice is a <button> wearing .backlink's look, not an <a> --
73 strip the browser's default button chrome so it reads as the same
74 plain inline text control as the "back to shell" link beside it. */
75#cloudvoice, #headphonestoggle { background: none; border: none; padding: 0; font: inherit; cursor: pointer; }
76/* Caption states: armed (waiting for the wake word), active (live capture
77 or the assistant's reply), sent (what was just submitted, muted). */
78.cap.user.armed { font-style: italic; opacity: 0.7; }
79.cap.user.active { color: var(--text); opacity: 1; font-style: normal; }
80.cap.user.sent { color: var(--muted); opacity: 1; font-style: normal; }
81"""
82
83# Copied verbatim from shell.py's /api/ai/chat system prompt so the two
84# stay in sync. Edit both together.
85_BASE_CAPABILITIES = (
86 "You are the shell of this user's object server. Answer in plain terminal text "
87 "with no markdown formatting. Be concise. Use your tools when the question is "
88 "about the user's records. "
89 "You can also MATERIALIZE PAGES: the views collection turns records into live "
90 "pages. When the user asks for a page/dashboard/view (or an answer clearly worth "
91 "keeping as one), create a views record: fields title, layout 'single', "
92 "owner_id (the user), pinned 'false', is_public 'false', and blocks = a JSON "
93 "string of a list of block objects. Block kinds: "
94 "{kind:'count', collection, filters:{field:value}, label, warn_over?} | "
95 "{kind:'list', collection, filters?, sort?:'newest'|'oldest', title?} | "
96 "{kind:'form', collection, record_id?} | "
97 "{kind:'detail', collection, record_id} | "
98 "{kind:'markdown', text}. "
99 "After creating it, tell the user the page is at /views/{id} (the record id). "
100 "Prefer a count block above a list block for status-style pages. "
101 "You do NOT know from memory which collections exist. Call list_collections "
102 "and read its result before answering; if the collection the user named "
103 "appears in that result, USE it -- never say it is missing when it is in "
104 "the list. The user's tasks live in the collection named tasks. "
105 "To show one specific record on screen, create a view whose blocks contain a "
106 "detail block for it. Never claim something is on screen unless you created "
107 "or updated a views record in this same turn. "
108 "Whenever the screen should show a view -- newly created OR one that already "
109 "exists -- end your reply with the marker [[view:<record id>]] alone on the "
110 "last line. The marker is machine-read; it is never displayed or spoken, so "
111 "it does not violate the no-ids-aloud rule. "
112 'Example reply: "Here are your open tasks. '
113 '[[view:26b247ed-3b1a-4206-b060-1d92847194de]]"'
114 " You can also READ WEB PAGES with the read_page tool when the user gives a "
115 "URL or asks you to read/summarize a page: it returns the page text and its "
116 'links numbered in order. Offer the links as "link 1, link 2, ..." so the '
117 'user can say "open link N" and you read_page that link\'s url next.'
118)
119
120_TALK_ADDENDUM = (
121 " You are in voice mode. Reply in one or two short spoken sentences. When the "
122 "user asks to see, list, or track anything, create (or update) a views record "
123 "and say you have put it on screen. NEVER read ids, urls, uuids, or paths aloud."
124)
125
126TALK_SYSTEM = _BASE_CAPABILITIES + _TALK_ADDENDUM
127
128_SCRIPT = """
129// One conversation per page load. The server stamps this on every turn it
130// records, which is what makes a session listable and resumable later.
131let SESSION_ID = freshSessionId();
132function freshSessionId() {
133 return crypto.randomUUID ? crypto.randomUUID()
134 : "s-" + Math.random().toString(36).slice(2);
135}
136
137const esc = (s) => String(s ?? "").replace(/[&<>"']/g,
138 (c) => ({"&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;"}[c]));
139const stage = document.getElementById("stage");
140const capUser = document.getElementById("capUser");
141const capAssistant = document.getElementById("capAssistant");
142let prefs = {id: OWNER_ID, ai_model: "anthropic:claude-sonnet-5",
143 tools: "global_search,list_collections,list_records,get_record,create_record,update_record,read_page",
144 talk_wake_word: "computer", talk_end_word: "over",
145 talk_endpoint: "silence", talk_silence_ms: "1400", talk_tts: "auto",
146 talk_tts_engine: "local", talk_stt_engine: "browser"};
147let aiHistory = [];
148// The real link data from the most recent read_page call -- {final_url,
149// title, links: [{n, label, href}]} or null. Conversation history
150// (aiHistory) only ever carries the model's own prose, never a tool's
151// actual result, so a page's real hrefs used to vanish the instant the
152// model finished replying -- a later "go to that story" had nothing but
153// the model's own memory of what it said to resolve against, which broke
154// completely on any page where voice mode's brevity instruction won over
155// citing link numbers (a busy news homepage, for instance). Replaced
156// wholesale by the next read_page call; otherwise persists for the rest
157// of the session, same as aiHistory.
158let lastReaderPage = null;
159
160// Grounding text appended to the system prompt every turn while
161// lastReaderPage is set -- real hrefs and labels the model can resolve a
162// vague reference against ("go to that story"), not just its own memory
163// of a summary it gave several turns ago. Empty string (a no-op append)
164// once no page has been read yet, or after a session reset.
165function readerContextNote() {
166 if (!lastReaderPage || !Array.isArray(lastReaderPage.links) || !lastReaderPage.links.length) return "";
167 const list = lastReaderPage.links.map((l) => l.n + ". " + l.label + " (" + l.href + ")").join(" ");
168 return " The last web page you read was '" + (lastReaderPage.title || lastReaderPage.final_url)
169 + "' (" + lastReaderPage.final_url + "). Its links, in case the user asks to open one or refers "
170 + "to something on that page: " + list;
171}
172
173const TTS_MAX_CHARS = 800;
174const VIEWS_PATH_RE = /\\/views\\/[A-Za-z0-9_-]+/;
175
176// Preference reads all go through pref() rather than raw prefs.x access:
177// loadPrefs() replaces `prefs` wholesale with whatever record comes back
178// from the server, and a record written before a field existed in the
179// schema simply won't have that key. pref() falls back to the shipped
180// default only when the key is *missing* -- an explicit empty string (the
181// wake/end word's "off" setting) is a real value, not a gap, and must be
182// left alone.
183const DEFAULT_WAKE_WORD = "computer";
184const DEFAULT_END_WORD = "over";
185const DEFAULT_ENDPOINT = "silence";
186const DEFAULT_SILENCE_MS = 1400;
187const DEFAULT_TOOLS = "global_search,list_collections,list_records,get_record,create_record,update_record,read_page";
188const DEFAULT_MODEL = "anthropic:claude-sonnet-5";
189const DEFAULT_TALK_TTS = "auto";
190const DEFAULT_TALK_TTS_ENGINE = "local";
191const DEFAULT_TALK_STT_ENGINE = "browser";
192
193function pref(name, fallback) {
194 const v = prefs[name];
195 return v === undefined || v === null ? fallback : v;
196}
197function wakeWord() { return String(pref("talk_wake_word", DEFAULT_WAKE_WORD)).trim(); }
198function endWord() { return String(pref("talk_end_word", DEFAULT_END_WORD)).trim(); }
199function endpointMode() {
200 const m = String(pref("talk_endpoint", DEFAULT_ENDPOINT)).trim();
201 return (m === "word" || m === "manual" || m === "silence") ? m : DEFAULT_ENDPOINT;
202}
203function talkTtsMode() {
204 const m = String(pref("talk_tts", DEFAULT_TALK_TTS)).trim();
205 const mode = (m === "server" || m === "browser") ? m : DEFAULT_TALK_TTS;
206 // A resolved engine of "openai" -- from settings OR a ?tts= URL
207 // override -- is an explicit request to hear the cloud voice. "auto"
208 // otherwise prefers ANY browser-native speechSynthesis voice over the
209 // server, and on many setups (Chrome on Linux, notably) that native
210 // voice is ITSELF backed by espeak -- indistinguishable by ear from
211 // this server's own local engine. Without this check, choosing openai
212 // (or passing ?tts=openai) would silently do nothing whenever the
213 // browser had any local voice at all: exactly what one live test hit.
214 if (mode === "auto" && talkTtsEngine() === "openai") return "server";
215 return mode;
216}
217// ?tts=openai|local and ?stt=openai|browser override the stored preference
218// for THIS page load only -- nothing is saved back to shell_preferences.
219// Exists so a link (e.g. "/talk?stt=openai") can be tried or shared
220// without visiting settings first; the stored preference underneath is
221// exactly what a plain "/talk" load still honors.
222function urlOverride(param, allowed) {
223 const v = new URLSearchParams(location.search).get(param);
224 return allowed.includes(v) ? v : null;
225}
226function talkTtsEngine() {
227 const override = urlOverride("tts", ["local", "openai"]);
228 if (override) return override;
229 const m = String(pref("talk_tts_engine", DEFAULT_TALK_TTS_ENGINE)).trim();
230 return m === "openai" ? "openai" : DEFAULT_TALK_TTS_ENGINE;
231}
232// "openai" here is deliberately opt-in only, never an automatic fallback
233// for browsers lacking SpeechRecognition -- a Firefox user who never
234// touched this setting (or the URL override) and has no stored OpenAI key
235// must keep seeing today's exact behavior (mic hidden), not a confusing
236// "no key stored" error from a feature they never asked for. See
237// initMic()/initCloudMic().
238function talkSttEngine() {
239 const override = urlOverride("stt", ["browser", "openai"]);
240 if (override) return override;
241 const m = String(pref("talk_stt_engine", DEFAULT_TALK_STT_ENGINE)).trim();
242 return m === "openai" ? "openai" : DEFAULT_TALK_STT_ENGINE;
243}
244function silenceMs() {
245 const n = Number(pref("talk_silence_ms", DEFAULT_SILENCE_MS));
246 return Number.isFinite(n) && n > 0 ? n : DEFAULT_SILENCE_MS;
247}
248
249// Strip markdown -- and, unlike the shell's stripForSpeech, urls and
250// /views paths too -- down to sentences worth speaking. The talk-mode
251// system prompt tells the model never to read an id/url/path aloud; this
252// is the belt to that suspenders in case it slips one in anyway.
253function stripForSpeech(text) {
254 return String(text ?? "")
255 .replace(/```[\\s\\S]*?```/g, " ")
256 .replace(/`[^`]*`/g, " ")
257 .replace(/!\\[[^\\]]*\\]\\([^)]*\\)/g, " ")
258 .replace(/\\[([^\\]]*)\\]\\([^)]*\\)/g, "$1")
259 .replace(/https?:\\/\\/\\S+/g, " ")
260 .replace(/\\/views\\/\\S+/g, " ")
261 .replace(/\\[\\[view:[^\\]]*\\]\\]/g, " ")
262 .replace(/[*_#>~]/g, " ")
263 .replace(/\\s+/g, " ")
264 .trim()
265 .slice(0, TTS_MAX_CHARS);
266}
267
268function renderCard(text) {
269 const card = document.createElement("div");
270 card.className = "card";
271 card.innerHTML = window.dbbasicMarkdown ? window.dbbasicMarkdown(text) : esc(text);
272 stage.innerHTML = "";
273 stage.appendChild(card);
274}
275
276function renderStageView(path) {
277 stage.innerHTML = `<iframe src="${esc(path)}?embed=1" class="stageframe"></iframe>`;
278}
279
280// The machine channel: the model ends a reply with [[view:<id>]] when the
281// stage should show that view (new OR already-existing) -- explicit
282// signaling instead of path-sniffing, since spoken replies never carry
283// paths. The marker is stripped from everything displayed and spoken.
284const VIEW_MARKER_RE = /\\[\\[view:([A-Za-z0-9-]+)\\]\\]/;
285
286function stripViewMarker(text) {
287 return String(text ?? "").replace(/\\[\\[view:[^\\]]*\\]\\]/g, " ").trim();
288}
289
290function viewPathFromReply(text) {
291 const marker = String(text || "").match(VIEW_MARKER_RE);
292 if (marker) return "/views/" + marker[1];
293 const match = String(text || "").match(VIEWS_PATH_RE);
294 return match ? match[0].replace(/[.,;:)]+$/, "") : null;
295}
296
297// Fallback for when the model materialized a page but its spoken reply
298// never mentions the path -- inspect the tool calls the chat turn made
299// (that's what /api/ai/chat actually returns: name/arguments/http_status
300// per call) for a create_record or update_record on the views collection.
301function viewPathFromToolCalls(toolCalls) {
302 if (!Array.isArray(toolCalls)) return null;
303 for (let i = toolCalls.length - 1; i >= 0; i--) {
304 const call = toolCalls[i];
305 const args = call && call.arguments;
306 if (!args || args.collection !== "views") continue;
307 const id = (call.name === "create_record" && args.record && args.record.id)
308 || (call.name === "update_record" && args.record_id);
309 if (id) return "/views/" + id;
310 }
311 return null;
312}
313
314let speaking = false;
315
316// Voice picking for the browser speechSynthesis engine: prefer a
317// natural-sounding named voice, then the first on-device (localService)
318// English voice, else leave utter.voice unset and let the browser use its
319// own default.
320const PREFERRED_VOICE_RE = /Samantha|Ava|Karen|Daniel/;
321function getVoices() {
322 return (window.speechSynthesis && window.speechSynthesis.getVoices()) || [];
323}
324function pickVoice() {
325 const voices = getVoices();
326 if (!voices.length) return null;
327 const byName = voices.find((v) => PREFERRED_VOICE_RE.test(v.name));
328 if (byName) return byName;
329 const localEn = voices.find((v) => v.localService && /^en/i.test(v.lang));
330 return localEn || null;
331}
332// The "auto" engine gate: is there at least one on-device voice at all
333// (any language) -- if so, speaking never has to leave the device.
334function hasLocalVoice() {
335 return getVoices().some((v) => v.localService);
336}
337
338// speechSynthesis-only speaking path, used both when talk_tts is "browser"
339// and as the fallback when "server"/"auto" server TTS fails. Carries the
340// exact same speaking-flag/listen-pause contract as the server path below:
341// set before speaking, cleared and resumed only once the utterance ends.
342async function speakBrowser(text) {
343 if (!window.speechSynthesis) { speaking = false; resumeListeningIfNeeded(); return; }
344 speaking = true;
345 await stopListeningAndWaitForRelease();
346 window.speechSynthesis.cancel();
347 const utter = new SpeechSynthesisUtterance(text);
348 const voice = pickVoice();
349 if (voice) utter.voice = voice;
350 utter.addEventListener("end", stopSpeaking);
351 utter.addEventListener("error", stopSpeaking);
352 window.speechSynthesis.speak(utter);
353}
354
355// Speak one assistant reply, routed by the talk_tts preference:
356// "server" -- always server TTS (POST /api/tts), speechSynthesis on failure.
357// "browser" -- always speechSynthesis, no server round-trip.
358// "auto" -- speechSynthesis if an on-device voice is available, else
359// server-with-fallback, same as "server".
360// Whichever path runs, the mic is kept stopped for the whole span and only
361// resumed once playback actually ends, so the recognizer never hears the
362// machine talking to itself.
363async function speak(text) {
364 const spoken = stripForSpeech(text);
365 if (!spoken) { resumeListeningIfNeeded(); return; }
366
367 const mode = talkTtsMode();
368 if (mode === "browser") { speakBrowser(spoken); return; }
369 if (mode === "auto" && window.speechSynthesis && hasLocalVoice()) {
370 speakBrowser(spoken);
371 return;
372 }
373
374 speaking = true;
375 await stopListeningAndWaitForRelease();
376 try {
377 const res = await fetch("/api/tts", {
378 method: "POST", credentials: "same-origin",
379 headers: {"content-type": "application/json", accept: "audio/*"},
380 body: JSON.stringify({text: spoken, engine: talkTtsEngine()}),
381 });
382 if (!res.ok) throw new Error("tts endpoint failed");
383 const url = URL.createObjectURL(await res.blob());
384 // The SAME element every time, never a fresh one. iOS allows play()
385 // outside a user gesture only on an element that has already played
386 // inside one -- primeAudioForIOS() plays this element silent during
387 // the mic tap, and changing src afterwards keeps the unlock. `new
388 // Audio(url)` here was born locked, its play() rejected, and the
389 // catch fell back to speechSynthesis... which the mute switch
390 // silences. Server voice configured, server WAV fetched, and still
391 // no sound: this line is why.
392 const player = sharedTtsAudio();
393 player.pause();
394 player.src = url;
395 currentTtsObjectUrl = url;
396 // EVERY exit clears `speaking`, not just the happy one. It gates the
397 // mic restart AND the transcript-stability submit, so a playback that
398 // ends without an `ended` event -- a bad WAV, a decode error, an
399 // interrupted element -- used to leave the page deaf and mute
400 // forever: mic never resumed, submits silently blocked, and nothing
401 // on screen said why. Found by the harness, whose spied play() never
402 // completed, which is exactly the shape of a real playback failure.
403 player.onended = stopSpeaking;
404 player.onerror = stopSpeaking;
405 player.onstalled = stopSpeaking;
406 await player.play();
407 } catch (e) {
408 speakBrowser(spoken);
409 }
410}
411
412// The object URL a cloud-TTS playback is currently using, or null --
413// tracked at module scope (not just inside speak()'s closure) so
414// stopSpeaking() can revoke it even when called from OUTSIDE a normal
415// playback-ended event, i.e. an interrupt.
416let currentTtsObjectUrl = null;
417
418// Shared end-of-speech cleanup, whether playback finished naturally (an
419// "ended"/"error"/"stalled" event) or was interrupted deliberately
420// (headphones mode's VAD hearing the user start talking mid-reply -- see
421// startHeadphonesMode below). Every exit clears `speaking`, not just the
422// happy one: a playback that never fires its own end event used to leave
423// the page deaf and mute forever (mic never resumed, submits silently
424// blocked). Safe to call when nothing is actually playing (window.speechSynthesis.cancel()
425// and player.pause() are no-ops in that case).
426function stopSpeaking() {
427 const player = sharedTtsAudio();
428 player.onended = player.onerror = player.onstalled = null;
429 player.pause();
430 if (currentTtsObjectUrl) { URL.revokeObjectURL(currentTtsObjectUrl); currentTtsObjectUrl = null; }
431 if (window.speechSynthesis) window.speechSynthesis.cancel();
432 speaking = false;
433 resumeListeningIfNeeded();
434}
435
436// Conversation-mode mic: the button toggles a mode, not a single listen.
437// While the mode is on, onend auto-restarts recognition -- unless we are
438// currently speaking (playing TTS), in which case resumeListeningIfNeeded()
439// (called from speak()'s completion handlers above) is what restarts it.
440const SpeechRecognitionCtor = window.SpeechRecognition || window.webkitSpeechRecognition;
441let recognizer = null;
442let conversationMode = false;
443let listening = false;
444
445// --- Radio protocol: wake word / end word / VAD silence endpointing ------
446//
447// `buffer` holds finalized transcript from earlier recognizer sessions in
448// the current utterance; `sessionLive` holds the current (possibly still
449// interim) recognizer session's transcript. Both are already wake-word-
450// stripped once armed flips false, so most readers just concatenate them.
451//
452// Word-boundary matching is done by tokenizing on whitespace rather than
453// regex word-boundary escapes -- this file has already had one bug from a
454// backslash escape that didn't survive its Python-string layer, so the
455// word-match helpers below avoid backslash metacharacters entirely.
456let buffer = "";
457let sessionLive = "";
458let armed = true;
459let finalDebounceTimer = null;
460let stabilityTimer = null;
461
462function tokenize(text) {
463 return String(text ?? "").trim().split(/\\s+/).filter(Boolean);
464}
465
466// Strip leading/trailing punctuation from one token for comparison --
467// "computer," or "over." still match "computer" / "over".
468function wordKey(token) {
469 return token.replace(/[.,!?;:]+$/, "").replace(/^[.,!?;:]+/, "").toLowerCase();
470}
471
472// If `word` appears as a whole token in `text`, return everything after
473// its first occurrence (joined back with spaces); otherwise null.
474// Bounded edit distance for wake-word tolerance. Recognition mishears:
475// an iPad delivered "compture" for "computer", and an exact match then
476// silently discarded everything the user said after it -- armed forever,
477// capturing nothing, with the transcript ON SCREEN making it look heard.
478// The budget scales with length (short words stay exact: "hey" must not
479// match "they"), and it is capped at 2 because a wake word is a gate, not
480// a search. Worst case of a false arm: the mic captures a sentence the
481// user then sees -- recoverable. Worst case of a strict miss: everything
482// said is thrown away invisibly -- not recoverable, and that asymmetry is
483// the whole argument for leniency here.
484function editDistanceAtMost(a, b, budget) {
485 if (Math.abs(a.length - b.length) > budget) return false;
486 // Damerau (OSA), not plain Levenshtein, and the difference is the
487 // whole point: a transposition costs 1. The observed mishear was
488 // "compture" for "computer" -- two adjacent swaps, Damerau distance 2,
489 // plain distance 3 -- so plain Levenshtein would have rejected the
490 // exact case this function exists for. Swapped letters are what
491 // recognition mishears LOOK like.
492 let prevPrev = null;
493 let prev = [];
494 for (let j = 0; j <= b.length; j++) prev.push(j);
495 for (let i = 1; i <= a.length; i++) {
496 const row = [i];
497 let best = i;
498 for (let j = 1; j <= b.length; j++) {
499 const cost = a[i - 1] === b[j - 1] ? 0 : 1;
500 let v = Math.min(prev[j] + 1, row[j - 1] + 1, prev[j - 1] + cost);
501 if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) {
502 v = Math.min(v, prevPrev[j - 2] + 1);
503 }
504 row.push(v);
505 if (v < best) best = v;
506 }
507 if (best > budget) return false; // the whole row is over budget
508 prevPrev = prev;
509 prev = row;
510 }
511 return prev[b.length] <= budget;
512}
513
514function wakeBudget(target) {
515 if (target.length >= 7) return 2;
516 if (target.length >= 4) return 1;
517 return 0;
518}
519
520function matchesWakeWord(tokenKey, target) {
521 if (tokenKey === target) return true;
522 return editDistanceAtMost(tokenKey, target, wakeBudget(target));
523}
524
525function findWakeSplit(text, word) {
526 const target = word.trim().toLowerCase();
527 if (!target) return null;
528 const tokens = tokenize(text);
529 for (let i = 0; i < tokens.length; i++) {
530 const key = wordKey(tokens[i]);
531 if (matchesWakeWord(key, target)) return tokens.slice(i + 1).join(" ");
532 // A no-space transcript fuses the wake word onto what follows
533 // ("comptureshowmemynotes"). If a long token's PREFIX matches the
534 // wake word, the remainder of the token is the start of the command
535 // -- unspaced, but the model reads unspaced text far better than
536 // this page reads a discarded utterance. Prefix lengths of the
537 // target and one either side, so a mishear inside the fused prefix
538 // still arms.
539 if (key.length > target.length + wakeBudget(target)) {
540 for (const len of [target.length, target.length + 1, target.length - 1]) {
541 if (len < 3 || len >= key.length) continue;
542 if (matchesWakeWord(key.slice(0, len), target)) {
543 const rest = [key.slice(len)].concat(tokens.slice(i + 1));
544 return rest.join(" ").trim();
545 }
546 }
547 }
548 }
549 return null;
550}
551
552// If `text`'s last token is `word`, return everything before it (joined
553// back with spaces); otherwise null. Tolerant of trailing punctuation on
554// the last token ("...turn it on, over." still matches "over").
555function tailMatchesEndWord(text, word) {
556 const target = word.trim().toLowerCase();
557 if (!target) return null;
558 const tokens = tokenize(text);
559 if (!tokens.length) return null;
560 if (wordKey(tokens[tokens.length - 1]) !== target) return null;
561 return tokens.slice(0, -1).join(" ");
562}
563
564function resetTalkState() {
565 buffer = "";
566 sessionLive = "";
567 armed = true;
568 if (finalDebounceTimer) { clearTimeout(finalDebounceTimer); finalDebounceTimer = null; }
569 utteranceStarted = false;
570 silenceStartTs = null;
571 speechHoldStart = null;
572}
573
574function updateCaptionArmed() {
575 const w = wakeWord();
576 capUser.textContent = w ? `say "${w}" to address me` : "";
577 capUser.classList.remove("active", "sent");
578 capUser.classList.add("armed");
579}
580
581function updateCaptionActive(text) {
582 capUser.textContent = text || "…";
583 capUser.classList.remove("armed", "sent");
584 capUser.classList.add("active");
585}
586
587// What would be submitted right now, with wake word and end word both
588// stripped if present -- used by VAD endpointing and by manual send/Enter,
589// which fold in whatever voice buffer exists regardless of protocol.
590function pendingText() {
591 let text = ((buffer ? buffer + " " : "") + sessionLive).trim();
592 const w = wakeWord();
593 if (w) {
594 const split = findWakeSplit(text, w);
595 if (split !== null) text = split;
596 }
597 const ew = endWord();
598 if (ew) {
599 const stripped = tailMatchesEndWord(text, ew);
600 if (stripped !== null) text = stripped;
601 }
602 return text;
603}
604
605function bufferHasContent() {
606 return ((buffer ? buffer + " " : "") + sessionLive).trim().length > 0;
607}
608
609// The single funnel for every voice-triggered submission (end word, VAD
610// silence, isFinal-debounce fallback): clear/re-arm first so new speech
611// during the in-flight chat call starts a fresh utterance, then hand off
612// to the same submitTurn() the text box and Enter use.
613function finalizeVoiceSubmit(text) {
614 if (finalDebounceTimer) { clearTimeout(finalDebounceTimer); finalDebounceTimer = null; }
615 if (stabilityTimer) { clearTimeout(stabilityTimer); stabilityTimer = null; }
616 const clean = String(text ?? "").trim();
617 resetTalkState();
618 if (!clean) { if (conversationMode) updateCaptionArmed(); return; }
619 // The wake word ALONE is a knock, not a question. One leaked through a
620 // gate seam on an iPad and bought a 2.4-second provider round-trip
621 // that answered "" -- money and silence for a turn nobody asked. The
622 // check is fuzzy like the gate itself, so a misheard bare "compture"
623 // is also swallowed.
624 const w = wakeWord();
625 if (w) {
626 const only = tokenize(clean);
627 if (only.length === 1 && matchesWakeWord(wordKey(only[0]), w.toLowerCase())) {
628 if (conversationMode) updateCaptionArmed();
629 return;
630 }
631 }
632 submitTurn(clean);
633}
634
635// Called on every recognizer result (interim and final). Handles the wake
636// gate, live caption, end-word override, and -- on a final result -- folds
637// the session transcript into `buffer` and (endpoint "word" with no end
638// word configured) arms the isFinal-debounce fallback.
639function processTranscript(isFinal) {
640 const combinedRaw = ((buffer ? buffer + " " : "") + sessionLive).trim();
641 const w = wakeWord();
642
643 // The split runs on EVERY event, not once at arming. Recognition
644 // transcripts are cumulative -- each interim restates the utterance
645 // from the beginning -- so a split applied only at arm time is undone
646 // by the very next event, and the wake word rides into the submitted
647 // command. Not hypothetical: the server's recorded turns include
648 // "Show me my notes computer show me my notes", the wake word twice,
649 // from exactly this. Splitting per event is idempotent: once buffer
650 // holds only command text, findWakeSplit finds nothing and passes it
651 // through untouched.
652 const split = w ? findWakeSplit(combinedRaw, w) : null;
653 if (armed) {
654 if (w && split === null) {
655 updateCaptionArmed();
656 if (isFinal) sessionLive = "";
657 return;
658 }
659 armed = false; // empty wake word: capture everything while the mic is on
660 }
661
662 const active = (split !== null ? split : combinedRaw).trim();
663 updateCaptionActive(active);
664
665 const endpoint = endpointMode();
666 const ew = endWord();
667 if (ew && (endpoint === "word" || endpoint === "silence")) {
668 const stripped = tailMatchesEndWord(active, ew);
669 if (stripped !== null) { finalizeVoiceSubmit(stripped); return; }
670 }
671
672 // TRANSCRIPT-STABILITY endpointing: on a device where the level meter
673 // cannot run (the iPad's exclusive microphone), end-of-utterance used
674 // to wait for iOS to volunteer an isFinal -- which it does 2-6 seconds
675 // after you stop, sometimes never. But the recognizer's own interims
676 // ARE a silence detector: while you speak the text keeps changing, and
677 // when it stops changing for the silence window, you have finished.
678 // Same signal the RMS meter derived from amplitude, taken from the
679 // transcript instead -- no second microphone stream, no contention.
680 // Backstop even when the meter runs: a mis-calibrated meter (started
681 // mid-speech, threshold learned from the voice itself) fails SILENT --
682 // no error, no submit, ever. The transcript stalling is evidence the
683 // meter cannot fake, so this fires a beat after the VAD would have; a
684 // healthy VAD always wins first, and finalizeVoiceSubmit clears both
685 // timers so the loser never double-fires.
686 if (!armed && endpointMode() === "silence" && active) {
687 if (stabilityTimer) clearTimeout(stabilityTimer);
688 // Close over THIS event's already-split text. Rebuilding from the
689 // raw sessionLive here would resurrect the wake word the split just
690 // removed -- the timer is reset by every event, so the latest
691 // event's `active` is by definition the whole utterance so far.
692 stabilityTimer = setTimeout(() => {
693 stabilityTimer = null;
694 if (active && conversationMode && !speaking) finalizeVoiceSubmit(active);
695 }, silenceMs() + (meterRunning ? 500 : 0));
696 }
697
698 if (isFinal) {
699 buffer = active;
700 sessionLive = "";
701 // `silence` mode with no working meter has no other way to end an
702 // utterance, so isFinal becomes the endpoint. Without this the mode
703 // is a dead end on any device where the mic cannot be opened twice.
704 if ((endpoint === "word" && !ew) || (endpoint === "silence" && !meterRunning)) {
705 // No end word configured for word mode: fall back to the original
706 // isFinal-triggered submit, debounced so a recognizer that fires
707 // several quick finals in a row (common near silence) only submits
708 // once.
709 if (finalDebounceTimer) clearTimeout(finalDebounceTimer);
710 finalDebounceTimer = setTimeout(() => {
711 finalDebounceTimer = null;
712 finalizeVoiceSubmit(buffer);
713 }, 300);
714 }
715 }
716}
717
718// --- Mic level meter + VAD silence endpointing ----------------------------
719//
720// A separate getUserMedia stream (SpeechRecognition exposes no raw audio)
721// feeds an AnalyserNode; a rAF loop computes RMS to both draw the level
722// ring around the mic button and, in "silence" endpoint mode, decide when
723// an utterance has ended. The stream is only open while conversation mode
724// is on -- it doubles as the privacy indicator, so it must go fully dead
725// (tracks stopped, context closed) the instant the mode turns off.
726const MIN_SPEECH_THRESHOLD = 0.015;
727const CALIBRATION_MS = 800;
728const SPEECH_HOLD_MS = 150;
729
730// Did the VAD meter actually come up? On iOS the microphone is EXCLUSIVE:
731// SpeechRecognition and getUserMedia contend for it, so the second one to
732// ask simply fails. Silence endpointing depends entirely on that second
733// stream, and when it is absent nothing in `silence` mode ever submits --
734// the user speaks, sees their words, and has to tap the mic again to make
735// anything happen. Reported from an older iPad, and the fallback below is
736// the fix: fall back to the recognizer's own isFinal, which every browser
737// that supports recognition at all still gives us.
738let meterRunning = false;
739// Set the first time recognition returns a result. Until then the meter
740// must not open a competing microphone stream.
741let recognitionProven = false;
742// The deferred meter start above is NOT sufficient on its own, and an
743// iPad proved it: recognition delivered ONE result ("computer"), that
744// proof started the meter, and the meter's stream then took the
745// microphone back -- first word on screen, then nothing, which is the
746// original bug moved one result later. So the meter must be able to
747// LOSE, and it convicts itself with its own evidence: if it can hear
748// voice (RMS above the speech threshold) while recognition has produced
749// nothing for CONTENTION_MS, the two are not sharing the microphone --
750// on a working platform, results flow continuously while you speak.
751// Convicted once, it stays off for the session and isFinal endpointing
752// takes over.
753let meterContended = false;
754let lastResultTs = 0;
755const CONTENTION_MS = 2000;
756let micStream = null;
757let audioCtx = null;
758let analyser = null;
759let meterRafId = null;
760let noiseFloor = 0;
761let speechThreshold = MIN_SPEECH_THRESHOLD;
762let calibrating = false;
763let calibrationStart = 0;
764let calibrationSamples = [];
765let utteranceStarted = false;
766let silenceStartTs = null;
767let speechHoldStart = null;
768
769// Absent capability, stated -- never a stub that pretends to work. The
770// hint is the only way a user can tell "it is waiting for me to stop
771// talking" from "it is waiting for me to tap".
772function noteEndpointFallback() {
773 meterRunning = false;
774 const hint = document.getElementById("endpointhint");
775 if (hint && endpointMode() === "silence") {
776 hint.textContent = "No level meter on this device, so it sends when it "
777 + "hears you finish rather than after a silence.";
778 hint.hidden = false;
779 }
780}
781
782
783function updateMeterVisual(rms) {
784 const ring = document.getElementById("miclevel");
785 if (!ring) return;
786 // Force the ring to idle during TTS playback -- the stream is still
787 // open (so the mic doesn't visibly "go dead" mid-conversation), but a
788 // level driven by the machine's own voice bleeding into the mic would
789 // be a misleading, distracting reading.
790 const level = speaking ? 0 : Math.min(1, rms * 6);
791 ring.style.opacity = String(Math.min(0.65, level * 1.3));
792 ring.style.boxShadow = "0 0 0 " + (level * 14).toFixed(1) + "px var(--danger)";
793}
794
795function evaluateVAD(rms, now) {
796 if (speaking || !conversationMode) { silenceStartTs = null; speechHoldStart = null; return; }
797 if (endpointMode() !== "silence") return;
798 const above = rms >= speechThreshold;
799 if (above) {
800 silenceStartTs = null;
801 if (speechHoldStart === null) speechHoldStart = now;
802 if (!utteranceStarted && now - speechHoldStart >= SPEECH_HOLD_MS) utteranceStarted = true;
803 return;
804 }
805 speechHoldStart = null;
806 if (!utteranceStarted || !bufferHasContent()) return;
807 if (silenceStartTs === null) { silenceStartTs = now; return; }
808 if (now - silenceStartTs < silenceMs()) return;
809 silenceStartTs = null;
810 utteranceStarted = false;
811 if (armed) {
812 // Never woken -- silence ended the attempt, discard rather than submit.
813 resetTalkState();
814 updateCaptionArmed();
815 } else {
816 finalizeVoiceSubmit(pendingText());
817 }
818}
819
820async function startMeter() {
821 meterRunning = false;
822 if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
823 noteEndpointFallback(); return;
824 }
825 let stream;
826 try {
827 stream = await navigator.mediaDevices.getUserMedia({audio: true});
828 } catch (e) { noteEndpointFallback(); return; }
829 if (!conversationMode) { stream.getTracks().forEach((t) => t.stop()); return; }
830 const Ctor = window.AudioContext || window.webkitAudioContext;
831 if (!Ctor) {
832 stream.getTracks().forEach((t) => t.stop()); noteEndpointFallback(); return;
833 }
834 meterRunning = true;
835 lastResultTs = performance.now();
836 micStream = stream;
837 audioCtx = new Ctor();
838 const source = audioCtx.createMediaStreamSource(micStream);
839 analyser = audioCtx.createAnalyser();
840 analyser.fftSize = 512;
841 source.connect(analyser);
842 const data = new Uint8Array(analyser.fftSize);
843
844 calibrating = true;
845 calibrationStart = performance.now();
846 calibrationSamples = [];
847 utteranceStarted = false;
848 silenceStartTs = null;
849 speechHoldStart = null;
850 const ring = document.getElementById("miclevel");
851 if (ring) ring.hidden = false;
852
853 function frame() {
854 if (!analyser) return; // meter was stopped
855 analyser.getByteTimeDomainData(data);
856 let sumSquares = 0;
857 for (let i = 0; i < data.length; i++) {
858 const v = (data[i] - 128) / 128;
859 sumSquares += v * v;
860 }
861 const rms = Math.sqrt(sumSquares / data.length);
862 const now = performance.now();
863
864 if (calibrating) {
865 calibrationSamples.push(rms);
866 if (now - calibrationStart >= CALIBRATION_MS) {
867 const avg = calibrationSamples.reduce((a, b) => a + b, 0) / calibrationSamples.length;
868 noiseFloor = avg;
869 speechThreshold = Math.max(avg * 3, MIN_SPEECH_THRESHOLD);
870 calibrating = false;
871 }
872 } else {
873 // The contention check runs BEFORE the VAD: the meter hearing
874 // voice while recognition says nothing is proof the microphone is
875 // exclusive and the meter took it. Stopping the meter frees the
876 // mic; stopListening() kicks the starved recognizer, whose onend
877 // handler restarts it fresh -- and this time nothing competes.
878 if (rms >= speechThreshold && now - lastResultTs > CONTENTION_MS) {
879 meterContended = true;
880 stopMeter();
881 noteEndpointFallback();
882 stopListening();
883 return;
884 }
885 evaluateVAD(rms, now);
886 }
887
888 updateMeterVisual(rms);
889 meterRafId = requestAnimationFrame(frame);
890 }
891 meterRafId = requestAnimationFrame(frame);
892}
893
894function stopMeter() {
895 if (meterRafId) cancelAnimationFrame(meterRafId);
896 meterRafId = null;
897 analyser = null;
898 if (audioCtx) { try { audioCtx.close(); } catch (e) { /* already closed */ } audioCtx = null; }
899 if (micStream) { micStream.getTracks().forEach((t) => t.stop()); micStream = null; }
900 calibrating = false;
901 utteranceStarted = false;
902 silenceStartTs = null;
903 speechHoldStart = null;
904 const ring = document.getElementById("miclevel");
905 if (ring) { ring.style.opacity = "0"; ring.style.boxShadow = "none"; ring.hidden = true; }
906}
907
908function stopListening() {
909 listening = false;
910 if (recognizer) { try { recognizer.stop(); } catch (e) { /* already stopped */ } }
911}
912
913// The mic and the speaker share ONE audio session on iOS, and
914// recognizer.stop() is a REQUEST, not an instant release -- teardown
915// completes asynchronously and onend fires later. Starting playback
916// before that teardown finishes was Dan's own diagnosis: "listening
917// while talking, maybe not allowed" -- iOS can leave the session
918// configured for recording through the transition, and audio played
919// into a still-recording session can come out silent or ducked to
920// nothing, with no error anywhere to say so. speak() now WAITS for the
921// release rather than racing it.
922let _listeningStoppedWaiters = [];
923function _drainListeningStopped() {
924 const waiters = _listeningStoppedWaiters;
925 _listeningStoppedWaiters = [];
926 waiters.forEach((resolve) => resolve());
927}
928function stopListeningAndWaitForRelease() {
929 stopListening();
930 if (!recognizer) return Promise.resolve();
931 return new Promise((resolve) => {
932 // A safety net, not the expected path: some browsers/recognizer
933 // states never fire onend for a stop() that was already a no-op
934 // (already stopped, never started). Real iOS teardown is much
935 // faster than this; it only pays the timeout when there was
936 // nothing to wait for in the first place.
937 const timer = setTimeout(resolve, 400);
938 _listeningStoppedWaiters.push(() => { clearTimeout(timer); resolve(); });
939 });
940}
941
942function startListening() {
943 if (!recognizer || listening || speaking) return;
944 listening = true;
945 try { recognizer.start(); } catch (e) { listening = false; }
946}
947
948function resumeListeningIfNeeded() {
949 if (conversationMode && !speaking && !listening) startListening();
950}
951
952function initMic() {
953 const mic = document.getElementById("mic");
954 const ring = document.getElementById("miclevel");
955 if (!mic) return;
956
957 // talk_stt_engine "openai" is an explicit opt-in (see talkSttEngine's
958 // comment) -- it never silently activates for a browser that merely
959 // lacks SpeechRecognition, so someone who never touched this setting
960 // keeps seeing exactly today's behavior either way.
961 if (talkSttEngine() === "openai") {
962 if (!navigator.mediaDevices || !window.MediaRecorder) {
963 mic.hidden = true;
964 if (ring) ring.hidden = true;
965 return;
966 }
967 mic.hidden = false;
968 // No live level meter here: it is driven by SpeechRecognition's own
969 // partial-result timing (see startMeter's callers), which push-to-talk
970 // cloud capture has no equivalent signal for.
971 if (ring) ring.hidden = true;
972 initCloudMic(mic);
973 return;
974 }
975
976 if (!SpeechRecognitionCtor) {
977 mic.hidden = true;
978 if (ring) ring.hidden = true;
979 return;
980 }
981 mic.hidden = false;
982 recognizer = new SpeechRecognitionCtor();
983 recognizer.continuous = false;
984 recognizer.interimResults = true;
985 recognizer.lang = "en-US";
986
987 // Interim and final results both feed the same processTranscript(), which
988 // is the single funnel for the wake gate, live caption, and end-word/
989 // isFinal submission -- there is no separate "submit" path for voice.
990 recognizer.onresult = (event) => {
991 lastResultTs = performance.now();
992 // Feeds the contention verdict and the no-recognition hint; the
993 // meter no longer waits on this (it must calibrate BEFORE speech).
994 recognitionProven = true;
995 let text = "";
996 // Segments are joined with an explicit space. Chrome includes leading
997 // spaces on continuation segments; iOS does NOT, and concatenating
998 // raw produced "comptureshowmemynotes" from an iPad -- one giant
999 // token the whitespace tokenizer cannot split, so the wake gate could
1000 // never match and every utterance was invisibly discarded. Trimming
1001 // then joining is correct on both: the tokenizer collapses runs of
1002 // whitespace anyway, so a doubled space costs nothing.
1003 for (let i = 0; i < event.results.length; i++) {
1004 const seg = event.results[i][0].transcript.trim();
1005 if (seg) text += (text ? " " : "") + seg;
1006 }
1007 sessionLive = text;
1008 const isFinal = event.results[event.results.length - 1].isFinal;
1009 processTranscript(isFinal);
1010 if (isFinal) stopListening();
1011 };
1012 recognizer.onerror = () => { listening = false; };
1013 recognizer.onend = () => {
1014 _drainListeningStopped();
1015 // CRITICAL: stop recognition before playing TTS (done in speak()) and
1016 // only resume after playback ends -- if mode is still on AND we are
1017 // not currently speaking, restart here too, so a recognizer that ends
1018 // for any other reason (silence timeout, browser quirk) still keeps
1019 // conversation mode going instead of going quiet.
1020 listening = false;
1021 if (conversationMode && !speaking) startListening();
1022 };
1023
1024 // If recognition has produced nothing at all a few seconds after the
1025 // mic went on, the device has not given us speech recognition -- say so
1026 // instead of leaving a listening button that will never do anything.
1027 let provenTimer = null;
1028 function watchForRecognition() {
1029 if (provenTimer) clearTimeout(provenTimer);
1030 provenTimer = setTimeout(() => {
1031 provenTimer = null;
1032 if (!conversationMode || recognitionProven) return;
1033 const hint = document.getElementById("endpointhint");
1034 if (hint) {
1035 hint.textContent = "This browser is not returning speech recognition."
1036 + " Type below instead \u2014 the mic cannot hear you here.";
1037 hint.hidden = false;
1038 }
1039 }, 6000);
1040 }
1041
1042 mic.addEventListener("click", () => {
1043 // iOS will not speak or play audio that was not started by a user
1044 // gesture, and a reply arrives asynchronously long after this click.
1045 // Priming both engines HERE -- inside the gesture -- is what makes the
1046 // later programmatic speak() audible. Silent by construction: an empty
1047 // utterance and a zero-length resume.
1048 primeAudioForIOS();
1049 conversationMode = !conversationMode;
1050 mic.classList.toggle("on", conversationMode);
1051 mic.textContent = conversationMode ? "listening…" : "mic";
1052 if (conversationMode) {
1053 resetTalkState();
1054 updateCaptionArmed();
1055 // Recognition and the meter BOTH start here, and the ordering war
1056 // between them is settled by evidence, not scheduling. History of
1057 // this line: starting both stole the mic from recognition on iOS
1058 // (exclusive microphone -- ring animated, no transcript), so the
1059 // meter was deferred until recognition's first result. That fix
1060 // broke every DESKTOP: a meter started mid-speech CALIBRATES ON
1061 // SPEECH, sets its threshold at 3x the user's own voice, and the
1062 // VAD never fires again -- transcript on screen, listening
1063 // forever, nothing sent. Calibration must happen in the pre-speech
1064 // silence, which means the meter must start at the click.
1065 //
1066 // The iOS case is owned by the CONVICTION instead: a meter that
1067 // can hear voice while recognition produces nothing for two
1068 // seconds took the microphone, and is stopped for the session --
1069 // recognition restarts and the stability endpointer takes over.
1070 // Each platform loses the component it cannot support, on proof.
1071 startListening();
1072 if (endpointMode() === "silence" && !meterContended) startMeter();
1073 watchForRecognition();
1074 } else {
1075 recognitionProven = false;
1076 stopListening();
1077 stopMeter();
1078 resetTalkState();
1079 capUser.textContent = "";
1080 capUser.classList.remove("armed", "active", "sent");
1081 }
1082 });
1083}
1084
1085// Cloud STT is push-to-talk by default: one tap starts a real
1086// MediaRecorder capture, a second tap stops it and uploads the clip to
1087// /api/stt for transcription. Raw audio has no equivalent to
1088// SpeechRecognition's interimResults, so there is no live partial
1089// transcript to drive a wake word or continuous conversation the way the
1090// browser path has for free. This is deliberately a smaller, honest
1091// feature on speakers: it promises only "record what you say between two
1092// taps, transcribe it, send it" -- and unlike SpeechRecognition, it works
1093// on every browser with a microphone, including ones the Web Speech API
1094// never supported (Firefox).
1095//
1096// talk_headphones_mode (see startHeadphonesMode below) is the exception:
1097// with headphones on, the mic physically cannot hear the assistant's own
1098// voice, so the "one tap per utterance" safety rail push-to-talk needs on
1099// speakers becomes pure friction instead of a real problem to guard
1100// against. That mode builds real audio-level VAD (the same
1101// amplitude-analysis technique the browser path's silence endpoint
1102// already uses, pointed at a MediaRecorder loop instead of
1103// SpeechRecognition's text buffer) for genuinely hands-free,
1104// no-tap-needed listening -- including staying live WHILE the assistant
1105// is speaking, so starting to talk mid-reply interrupts it.
1106let cloudRecorder = null;
1107let cloudChunks = [];
1108let cloudStream = null;
1109let cloudRecording = false;
1110
1111function talkHeadphonesMode() {
1112 return String(pref("talk_headphones_mode", "false")).trim() === "true";
1113}
1114
1115function initCloudMic(mic) {
1116 mic.addEventListener("click", async () => {
1117 primeAudioForIOS();
1118 if (talkHeadphonesMode()) {
1119 if (headphonesArmed) stopHeadphonesMode(mic); else await startHeadphonesMode(mic);
1120 return;
1121 }
1122 if (cloudRecording) { stopCloudRecording(); return; }
1123 await startCloudRecording(mic);
1124 });
1125}
1126
1127function showEndpointHint(message) {
1128 const hint = document.getElementById("endpointhint");
1129 if (hint) { hint.textContent = message; hint.hidden = false; }
1130}
1131
1132async function startCloudRecording(mic) {
1133 let stream;
1134 try {
1135 stream = await navigator.mediaDevices.getUserMedia({audio: true});
1136 } catch (e) {
1137 showEndpointHint("Microphone access was denied or unavailable.");
1138 return;
1139 }
1140 const candidates = ["audio/webm", "audio/mp4", "audio/ogg"];
1141 const mimeType = window.MediaRecorder && MediaRecorder.isTypeSupported
1142 ? candidates.find((t) => MediaRecorder.isTypeSupported(t))
1143 : undefined;
1144 try {
1145 cloudRecorder = mimeType ? new MediaRecorder(stream, {mimeType}) : new MediaRecorder(stream);
1146 } catch (e) {
1147 stream.getTracks().forEach((t) => t.stop());
1148 showEndpointHint("This browser cannot record audio for cloud speech recognition.");
1149 return;
1150 }
1151 cloudStream = stream;
1152 cloudChunks = [];
1153 cloudRecorder.ondataavailable = (e) => { if (e.data && e.data.size) cloudChunks.push(e.data); };
1154 cloudRecorder.onstop = () => onCloudRecordingStopped(mic);
1155 cloudRecorder.start();
1156 cloudRecording = true;
1157 mic.classList.add("on");
1158 mic.textContent = "recording… tap to stop";
1159 capUser.textContent = "";
1160 capUser.classList.add("armed");
1161}
1162
1163function stopCloudRecording() {
1164 if (cloudRecorder && cloudRecorder.state !== "inactive") cloudRecorder.stop();
1165 cloudRecording = false;
1166}
1167
1168async function onCloudRecordingStopped(mic) {
1169 if (cloudStream) { cloudStream.getTracks().forEach((t) => t.stop()); cloudStream = null; }
1170 mic.classList.remove("on");
1171 mic.textContent = "transcribing…";
1172 const blobType = (cloudRecorder && cloudRecorder.mimeType) || "audio/webm";
1173 const blob = new Blob(cloudChunks, {type: blobType});
1174 cloudChunks = [];
1175 if (!blob.size) {
1176 mic.textContent = "mic";
1177 return;
1178 }
1179 try {
1180 const res = await fetch("/api/stt", {
1181 method: "POST", credentials: "same-origin",
1182 headers: {"content-type": blobType, accept: "application/json"},
1183 body: blob,
1184 });
1185 const data = await res.json().catch(() => ({}));
1186 mic.textContent = "mic";
1187 if (!res.ok || data.status !== "ok" || typeof data.text !== "string") {
1188 showEndpointHint(data.error || "Transcription failed.");
1189 return;
1190 }
1191 const text = data.text.trim();
1192 if (text) submitTurn(text);
1193 } catch (e) {
1194 mic.textContent = "mic";
1195 showEndpointHint("Transcription request failed.");
1196 }
1197}
1198
1199// ---- Headphones mode: continuous, hands-free cloud STT --------------------
1200//
1201// One tap arms it; the mic stays open and listening until a second tap
1202// disarms it -- no per-utterance tap. Silence-segmented the same way the
1203// browser path's VAD is (calibrate ambient noise briefly, then watch for
1204// a speech-then-silence cycle), just pointed at a MediaRecorder loop
1205// (stop the current segment, upload+transcribe it, immediately start the
1206// next one) instead of SpeechRecognition's own text buffer. Deliberately
1207// a SEPARATE, independent state machine from the browser path's meter
1208// code above (evaluateVAD/startMeter/etc.) rather than a shared one: that
1209// code is tightly coupled to SpeechRecognition-specific state
1210// (conversationMode, armed, lastResultTs, wake-word buffering) that
1211// doesn't apply here, and forcing a shared implementation would have
1212// meant threading cloud-vs-browser branches through code that's already
1213// dense with iPad-specific hard-won fixes -- not worth the risk of
1214// breaking that path to save writing a second, smaller VAD loop.
1215//
1216// The one genuinely new thing this mode does versus push-to-talk: it
1217// keeps recording WHILE the assistant is speaking, and interrupts
1218// playback (stopSpeaking()) the moment it hears real speech during that
1219// window -- safe specifically because headphones mean the mic can't be
1220// hearing the assistant's own voice, so any detected speech is real.
1221let headphonesArmed = false;
1222let headphonesStream = null;
1223let headphonesRecorder = null;
1224let headphonesChunks = [];
1225let headphonesAudioCtx = null;
1226let headphonesAnalyser = null;
1227let headphonesCalibrating = false;
1228let headphonesCalibrationStart = 0;
1229let headphonesCalibrationSamples = [];
1230let headphonesSpeechThreshold = MIN_SPEECH_THRESHOLD;
1231let headphonesUtteranceStarted = false;
1232let headphonesSilenceStartTs = null;
1233let headphonesSpeechHoldStart = null;
1234let headphonesRafId = null;
1235// Guards the ENTIRE finish-a-segment sequence (stop -> read chunks ->
1236// clear -> start next -> upload -> transcribe -> submit), not just the
1237// network part: headphonesChunks is one shared array reused across
1238// segments, and two overlapping finishes racing to read/clear it would
1239// corrupt each other's audio. The cost is real but bounded: back-to-back
1240// utterances faster than one full transcribe round-trip apart get
1241// recorded into the same segment and submitted together, rather than
1242// split -- simple and correct beats a fully pipelined multi-segment
1243// system for what this is.
1244let headphonesSegmentPending = false;
1245
1246function headphonesRms(analyser, data) {
1247 analyser.getByteTimeDomainData(data);
1248 let sumSquares = 0;
1249 for (let i = 0; i < data.length; i++) {
1250 const v = (data[i] - 128) / 128;
1251 sumSquares += v * v;
1252 }
1253 return Math.sqrt(sumSquares / data.length);
1254}
1255
1256async function startHeadphonesMode(mic) {
1257 let stream;
1258 try {
1259 stream = await navigator.mediaDevices.getUserMedia({audio: true});
1260 } catch (e) {
1261 showEndpointHint("Microphone access was denied or unavailable.");
1262 return;
1263 }
1264 const Ctor = window.AudioContext || window.webkitAudioContext;
1265 if (!Ctor || !window.MediaRecorder) {
1266 stream.getTracks().forEach((t) => t.stop());
1267 showEndpointHint("This browser cannot run headphones mode.");
1268 return;
1269 }
1270
1271 headphonesArmed = true;
1272 headphonesStream = stream;
1273 headphonesAudioCtx = new Ctor();
1274 const source = headphonesAudioCtx.createMediaStreamSource(stream);
1275 headphonesAnalyser = headphonesAudioCtx.createAnalyser();
1276 headphonesAnalyser.fftSize = 512;
1277 source.connect(headphonesAnalyser);
1278 const data = new Uint8Array(headphonesAnalyser.fftSize);
1279
1280 headphonesCalibrating = true;
1281 headphonesCalibrationStart = performance.now();
1282 headphonesCalibrationSamples = [];
1283 headphonesUtteranceStarted = false;
1284 headphonesSilenceStartTs = null;
1285 headphonesSpeechHoldStart = null;
1286
1287 mic.classList.add("on");
1288 mic.textContent = "listening (headphones)… tap to stop";
1289 capUser.textContent = "";
1290 capUser.classList.add("armed");
1291
1292 startHeadphonesSegment();
1293
1294 function frame() {
1295 if (!headphonesAnalyser) return; // disarmed mid-flight
1296 const rms = headphonesRms(headphonesAnalyser, data);
1297 const now = performance.now();
1298
1299 if (headphonesCalibrating) {
1300 headphonesCalibrationSamples.push(rms);
1301 if (now - headphonesCalibrationStart >= CALIBRATION_MS) {
1302 const avg = headphonesCalibrationSamples.reduce((a, b) => a + b, 0) / headphonesCalibrationSamples.length;
1303 headphonesSpeechThreshold = Math.max(avg * 3, MIN_SPEECH_THRESHOLD);
1304 headphonesCalibrating = false;
1305 }
1306 } else {
1307 const above = rms >= headphonesSpeechThreshold;
1308 if (above) {
1309 // The headline behavior: real speech heard while the assistant is
1310 // still talking is the interrupt itself, not a signal to ignore.
1311 if (speaking) stopSpeaking();
1312 headphonesSilenceStartTs = null;
1313 if (headphonesSpeechHoldStart === null) headphonesSpeechHoldStart = now;
1314 if (!headphonesUtteranceStarted && now - headphonesSpeechHoldStart >= SPEECH_HOLD_MS) {
1315 headphonesUtteranceStarted = true;
1316 capUser.classList.remove("armed");
1317 capUser.classList.add("active");
1318 capUser.textContent = "listening…";
1319 }
1320 } else {
1321 headphonesSpeechHoldStart = null;
1322 if (headphonesUtteranceStarted && !headphonesSegmentPending) {
1323 if (headphonesSilenceStartTs === null) {
1324 headphonesSilenceStartTs = now;
1325 } else if (now - headphonesSilenceStartTs >= silenceMs()) {
1326 headphonesSilenceStartTs = null;
1327 headphonesUtteranceStarted = false;
1328 finishHeadphonesSegment();
1329 }
1330 }
1331 }
1332 }
1333 headphonesRafId = requestAnimationFrame(frame);
1334 }
1335 headphonesRafId = requestAnimationFrame(frame);
1336}
1337
1338function startHeadphonesSegment() {
1339 if (!headphonesStream) return;
1340 const candidates = ["audio/webm", "audio/mp4", "audio/ogg"];
1341 const mimeType = window.MediaRecorder && MediaRecorder.isTypeSupported
1342 ? candidates.find((t) => MediaRecorder.isTypeSupported(t))
1343 : undefined;
1344 try {
1345 headphonesRecorder = mimeType
1346 ? new MediaRecorder(headphonesStream, {mimeType})
1347 : new MediaRecorder(headphonesStream);
1348 } catch (e) {
1349 stopHeadphonesMode(document.getElementById("mic"));
1350 showEndpointHint("This browser cannot record audio for headphones mode.");
1351 return;
1352 }
1353 headphonesChunks = [];
1354 headphonesRecorder.ondataavailable = (e) => { if (e.data && e.data.size) headphonesChunks.push(e.data); };
1355 headphonesRecorder.start();
1356}
1357
1358async function finishHeadphonesSegment() {
1359 if (!headphonesRecorder || headphonesRecorder.state === "inactive") return;
1360 headphonesSegmentPending = true;
1361 try {
1362 const recorder = headphonesRecorder;
1363 const blobType = recorder.mimeType || "audio/webm";
1364 const stopped = new Promise((resolve) => { recorder.onstop = resolve; });
1365 recorder.stop();
1366 await stopped;
1367 const blob = new Blob(headphonesChunks, {type: blobType});
1368 headphonesChunks = [];
1369
1370 // Arm the NEXT segment immediately -- "always listening" must not
1371 // have a gap while this segment's transcription is still in flight.
1372 if (headphonesArmed) startHeadphonesSegment();
1373
1374 if (!blob.size) return;
1375 const res = await fetch("/api/stt", {
1376 method: "POST", credentials: "same-origin",
1377 headers: {"content-type": blobType, accept: "application/json"},
1378 body: blob,
1379 });
1380 const data = await res.json().catch(() => ({}));
1381 if (!res.ok || data.status !== "ok" || typeof data.text !== "string") {
1382 showEndpointHint(data.error || "Transcription failed.");
1383 return;
1384 }
1385 const text = data.text.trim();
1386 if (text) submitTurn(text);
1387 } catch (e) {
1388 showEndpointHint("Transcription request failed.");
1389 } finally {
1390 headphonesSegmentPending = false;
1391 }
1392}
1393
1394function stopHeadphonesMode(mic) {
1395 headphonesArmed = false;
1396 if (headphonesRafId) cancelAnimationFrame(headphonesRafId);
1397 headphonesRafId = null;
1398 headphonesAnalyser = null;
1399 if (headphonesAudioCtx) { headphonesAudioCtx.close(); headphonesAudioCtx = null; }
1400 if (headphonesRecorder && headphonesRecorder.state !== "inactive") headphonesRecorder.stop();
1401 headphonesRecorder = null;
1402 if (headphonesStream) { headphonesStream.getTracks().forEach((t) => t.stop()); headphonesStream = null; }
1403 headphonesUtteranceStarted = false;
1404 headphonesSilenceStartTs = null;
1405 headphonesSpeechHoldStart = null;
1406 if (mic) { mic.classList.remove("on"); mic.textContent = "mic"; }
1407 capUser.textContent = "";
1408 capUser.classList.remove("armed", "active", "sent");
1409}
1410
1411// One-time unlock, inside a user gesture. Safari on iOS refuses both
1412// speechSynthesis and HTMLAudioElement.play() outside one, and gives no
1413// error worth reading when it refuses -- the reply simply never becomes
1414// sound, which is exactly what an older iPad reported.
1415let ttsAudioEl = null;
1416function sharedTtsAudio() {
1417 if (!ttsAudioEl) ttsAudioEl = new Audio();
1418 return ttsAudioEl;
1419}
1420
1421let audioPrimed = false;
1422function primeAudioForIOS() {
1423 if (audioPrimed) return;
1424 audioPrimed = true;
1425 try {
1426 if (window.speechSynthesis) {
1427 const warm = new SpeechSynthesisUtterance("");
1428 warm.volume = 0;
1429 window.speechSynthesis.speak(warm);
1430 // getVoices() is empty until the engine loads on iOS; asking inside
1431 // the gesture is what populates it in time for hasLocalVoice().
1432 window.speechSynthesis.getVoices();
1433 }
1434 } catch (e) { /* nothing to unlock */ }
1435 try {
1436 // Unlock the ONE audio element every server-TTS reply will reuse: a
1437 // zero-length silent play inside the gesture is what licenses every
1438 // later programmatic play() on the same element.
1439 const player = sharedTtsAudio();
1440 player.muted = true;
1441 player.src = "data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAIlYAAESsAAACABAAZGF0YQAAAAA=";
1442 const attempt = player.play();
1443 if (attempt && attempt.catch) attempt.catch(() => {});
1444 setTimeout(() => { player.muted = false; }, 150);
1445 } catch (e) { /* nothing to unlock */ }
1446 try {
1447 const Ctor = window.AudioContext || window.webkitAudioContext;
1448 if (Ctor) {
1449 const ctx = new Ctor();
1450 if (ctx.state === "suspended" && ctx.resume) ctx.resume();
1451 // Kept open deliberately: closing it re-locks playback on iOS.
1452 window.__dbbasicTalkUnlockCtx = ctx;
1453 }
1454 } catch (e) { /* nothing to unlock */ }
1455}
1456
1457
1458async function api(method, path, payload) {
1459 const res = await fetch(path, {
1460 method, credentials: "same-origin",
1461 headers: {"content-type": "application/json", accept: "application/json"},
1462 body: payload === undefined ? undefined : JSON.stringify(payload),
1463 });
1464 return [res.ok, await res.json()];
1465}
1466
1467async function loadPrefs() {
1468 const res = await fetch(`/collections/shell_preferences/records/${OWNER_ID}`,
1469 {credentials: "same-origin", headers: {accept: "application/json"}});
1470 if (res.ok) { const body = await res.json(); prefs = body.record || prefs; }
1471}
1472
1473async function savePrefs(changes) {
1474 Object.assign(prefs, changes);
1475 const [ok] = await api("PUT", `/collections/shell_preferences/records/${OWNER_ID}`, changes);
1476 if (!ok) await api("POST", "/collections/shell_preferences/records", prefs);
1477}
1478
1479// One combined switch for both engines together -- talk_tts_engine and
1480// talk_stt_engine are separate prefs (fine-grained control still lives in
1481// the settings form), but "get back to this mode" means the SAME
1482// push-to-talk-plus-cloud-voice combination just tried, not two things to
1483// remember to flip separately.
1484function resolvedCloudVoiceOn() {
1485 return talkTtsEngine() === "openai" && talkSttEngine() === "openai";
1486}
1487
1488function initCloudVoiceToggle() {
1489 const btn = document.getElementById("cloudvoice");
1490 if (!btn) return;
1491 btn.hidden = false;
1492 btn.textContent = resolvedCloudVoiceOn() ? "cloud voice: on" : "cloud voice: off";
1493 btn.addEventListener("click", async () => {
1494 const turningOn = !resolvedCloudVoiceOn();
1495 btn.disabled = true;
1496 btn.textContent = "saving…";
1497 await savePrefs({
1498 talk_tts_engine: turningOn ? "openai" : "local",
1499 talk_stt_engine: turningOn ? "openai" : "browser",
1500 });
1501 // A clean reload, no ?tts=/?stt= -- the freshly-saved preference is
1502 // the only source of truth for the next load. Without this, a page
1503 // reached via a "?stt=openai" link could look like the toggle did
1504 // nothing (still "on" from the URL) or, worse, silently keep forcing
1505 // cloud back on after the user just explicitly turned it off.
1506 location.href = location.pathname;
1507 });
1508}
1509
1510// Only meaningful under cloud speech recognition (talk_stt_engine ===
1511// "openai") -- hidden otherwise rather than shown inert, same as the mic
1512// button itself is conditionally shown. Unlike the cloud-voice toggle
1513// above, this one does NOT reload the page: it only changes behavior
1514// INSIDE initCloudMic's already-wired click handler (talkHeadphonesMode()
1515// is read fresh on every tap), never which top-level mic implementation
1516// initMic() chose at load, so there's nothing that needs re-initializing.
1517function initHeadphonesToggle() {
1518 const btn = document.getElementById("headphonestoggle");
1519 if (!btn || talkSttEngine() !== "openai") return;
1520 btn.hidden = false;
1521 btn.textContent = talkHeadphonesMode() ? "headphones: on" : "headphones: off";
1522 btn.addEventListener("click", async () => {
1523 const turningOn = !talkHeadphonesMode();
1524 btn.disabled = true;
1525 btn.textContent = "saving…";
1526 await savePrefs({talk_headphones_mode: turningOn ? "true" : "false"});
1527 btn.textContent = turningOn ? "headphones: on" : "headphones: off";
1528 btn.disabled = false;
1529 });
1530}
1531
1532// The 10-second silence problem: a tool-using model legitimately takes
1533// that long, and a page that shows a frozen ellipsis for it reads as
1534// dead -- "we don't know because after 10 seconds..." is a verbatim
1535// user report. Two signals, chosen for a voice-first surface: a short
1536// BLIP the moment the utterance is accepted (the ear knows it fired
1537// without looking at the screen), and a visible elapsed counter while
1538// the model works (a counter that is moving is a page that is alive,
1539// and "thinking... 9s" and "stuck" stop being the same picture).
1540let thinkingTimer = null;
1541function startThinking() {
1542 const started = performance.now();
1543 capAssistant.textContent = "thinking\\u2026";
1544 if (thinkingTimer) clearInterval(thinkingTimer);
1545 thinkingTimer = setInterval(() => {
1546 const s = Math.round((performance.now() - started) / 1000);
1547 if (s >= 2) capAssistant.textContent = "thinking\\u2026 " + s + "s";
1548 }, 1000);
1549}
1550function stopThinking() {
1551 if (thinkingTimer) { clearInterval(thinkingTimer); thinkingTimer = null; }
1552}
1553function blip() {
1554 try {
1555 const ctx = window.__dbbasicTalkUnlockCtx;
1556 if (!ctx || ctx.state !== "running") return;
1557 const osc = ctx.createOscillator();
1558 const gain = ctx.createGain();
1559 osc.frequency.value = 880;
1560 gain.gain.setValueAtTime(0.08, ctx.currentTime);
1561 gain.gain.exponentialRampToValueAtTime(0.0001, ctx.currentTime + 0.12);
1562 osc.connect(gain); gain.connect(ctx.destination);
1563 osc.start(); osc.stop(ctx.currentTime + 0.13);
1564 } catch (e) { /* a missing blip is not an error */ }
1565}
1566
1567// Every path that can produce an utterance -- the voice endpointer
1568// (VAD, stability backstop, isFinal debounce) and the manual send/Enter
1569// handler -- funnels here. Nothing serialized them, and a real session
1570// proved it: "showed me my notes" and "show me my notes" reached the
1571// server ONE SECOND apart, two provider calls billed for one utterance,
1572// the reply spoken twice. Cause not fully isolated (a stale recognizer
1573// instance surviving its own onend restart is the leading suspect), but
1574// the fix does not need the cause: a turn already in flight must simply
1575// refuse a second one, for ANY reason it might arrive.
1576let turnInFlight = false;
1577
1578async function submitTurn(input) {
1579 if (turnInFlight) return; // a second utterance chasing the first is dropped, not queued
1580 turnInFlight = true;
1581 capUser.textContent = input;
1582 capUser.classList.remove("armed", "active");
1583 capUser.classList.add("sent");
1584 blip();
1585 startThinking();
1586 stopListening();
1587
1588 let ok, body;
1589 try {
1590 // pref(), not raw prefs.tools -- a shell_preferences record written
1591 // before this field existed in the schema won't have it, and
1592 // undefined.split() would throw here and silently kill the turn.
1593 const tools = String(pref("tools", DEFAULT_TOOLS)).split(",").map((t) => t.trim()).filter(Boolean);
1594 [ok, body] = await api("POST", "/api/ai/chat",
1595 {message: input,
1596 model: String(pref("talk_model", "")).trim() || pref("ai_model", DEFAULT_MODEL),
1597 tools, history: aiHistory.slice(-20),
1598 session_id: SESSION_ID, source: "talk",
1599 system: TALK_SYSTEM + " Current local date/time: " + new Date().toString() + "." + readerContextNote()});
1600 } finally {
1601 // Cleared before the reply renders, not after: a fetch that throws
1602 // must release the gate too, or one failed turn wedges the page shut
1603 // for the rest of the session.
1604 turnInFlight = false;
1605 }
1606
1607 stopThinking();
1608 // An empty SUCCESS is the worst reply a voice surface can relay: the
1609 // round-trip happened, nothing is shown, nothing is spoken, and the
1610 // user is left staring at a page that looks exactly like a hang. Turn
1611 // it into words -- one really did come back empty, recorded as out:""
1612 // on the server.
1613 const rawReply = ok ? (body.reply || "I came back empty \u2014 ask me again?")
1614 : (body.error || "Something went wrong.");
1615 const replyText = stripViewMarker(rawReply);
1616 capAssistant.textContent = stripForSpeech(replyText) || replyText;
1617
1618 if (ok) {
1619 aiHistory.push({role: "user", content: input});
1620 aiHistory.push({role: "assistant", content: body.reply});
1621 // Same "tool calls are ground truth" reasoning as the view marker
1622 // below: a read_page call's real result (not the model's prose about
1623 // it) is what a LATER turn needs to resolve "open that link"/"go to
1624 // that story" against. See lastReaderPage's own comment.
1625 const readerCall = (body.tool_calls || []).find((c) => c.name === "read_page" && c.result);
1626 if (readerCall) lastReaderPage = readerCall.result;
1627 // Tool calls are ground truth for what was actually created/updated this
1628 // turn; the [[view:id]] marker is the model retyping an id and can typo or
1629 // hallucinate it (pointing the stage at a phantom view). So trust the tool
1630 // call first, fall back to the marker only for "show an existing view again"
1631 // turns that touched no records.
1632 const viewPath = viewPathFromToolCalls(body.tool_calls) || viewPathFromReply(rawReply);
1633 if (viewPath) renderStageView(viewPath); else renderCard(replyText);
1634 } else {
1635 renderCard(replyText);
1636 }
1637 speak(replyText);
1638 // No client-side history write: the SERVER records every AI turn as
1639 // part of /api/ai/chat itself -- stamped, session-grouped, and immune
1640 // to this page dying before a fire-and-forget write lands (a real turn
1641 // was lost exactly that way).
1642}
1643
1644// Manual send/Enter always submits whatever is in the voice buffer plus
1645// whatever is typed, regardless of endpoint protocol or wake-word arming
1646// -- an explicit click/Enter is its own address signal. pendingText()
1647// still strips a wake/end word if one happens to be present so a
1648// half-spoken command doesn't come along for the ride.
1649document.getElementById("prompt").addEventListener("submit", (event) => {
1650 event.preventDefault();
1651 const box = event.target.elements["line"];
1652 const typed = box.value.trim();
1653 const voice = conversationMode ? pendingText() : "";
1654 box.value = "";
1655 if (finalDebounceTimer) { clearTimeout(finalDebounceTimer); finalDebounceTimer = null; }
1656 resetTalkState();
1657 const input = [voice, typed].filter(Boolean).join(" ").trim();
1658 if (!input) { if (conversationMode) updateCaptionArmed(); return; }
1659 submitTurn(input);
1660});
1661// initMic()'s job is deciding WHICH mic implementation to wire up at all
1662// (browser SpeechRecognition vs. cloud push-to-talk) -- a one-time,
1663// load-bearing decision, unlike wakeWord()/endpointMode()/etc., which are
1664// read live and repeatedly deep into an already-running conversation
1665// (loadPrefs() has always finished long before a user starts talking).
1666// Calling it before loadPrefs() resolves used to mean it ALWAYS saw the
1667// hardcoded defaults, never the real stored talk_stt_engine -- a real bug
1668// (settings-based cloud STT could never activate, only a ?stt= URL
1669// override could, since that alone doesn't depend on prefs). initCloudVoiceToggle()
1670// has the same real dependency, for the same reason: its initial label
1671// must reflect what's actually stored, not the shipped default.
1672loadPrefs().then(() => { initMic(); initCloudVoiceToggle(); initHeadphonesToggle(); });
1673
1674
1675// === sessions =================================================================
1676//
1677// The same fold the shell does over the server-recorded turns, with one
1678// deliberate difference: Talk NEVER auto-resumes. A voice page that
1679// silently reloads twenty old turns into model context changes what the
1680// next answer means without the user hearing anything change -- resume
1681// here is a hand action on the picker, always. (The shell auto-resumes
1682// inside a 30-minute window; eyes can see a restored transcript, ears
1683// cannot.)
1684let sessions = [];
1685
1686function foldSessions(rows) {
1687 const by = {};
1688 for (const row of rows) {
1689 const sid = row.session_id || "";
1690 if (!sid) continue;
1691 (by[sid] = by[sid] || []).push(row);
1692 }
1693 return Object.entries(by).map(([id, rs]) => {
1694 rs.sort((a, b) => String(a.created_at || "").localeCompare(String(b.created_at || "")));
1695 const first = rs.find((r) => r.input) || rs[0];
1696 return {id, rows: rs, count: rs.length,
1697 title: String((first && first.input) || "(untitled)").slice(0, 40),
1698 last: String(rs[rs.length - 1].created_at || "")};
1699 }).sort((a, b) => b.last.localeCompare(a.last)).slice(0, 10);
1700}
1701
1702function agoShort(iso) {
1703 const ms = Date.now() - Date.parse(iso || 0);
1704 if (!isFinite(ms) || ms < 0) return "";
1705 const m = Math.round(ms / 60000);
1706 if (m < 1) return "now";
1707 if (m < 60) return m + "m";
1708 const h = Math.round(m / 60);
1709 if (h < 24) return h + "h";
1710 return Math.round(h / 24) + "d";
1711}
1712
1713function renderSessionPicker() {
1714 const sel = document.getElementById("sessionpick");
1715 if (!sel) return;
1716 const opts = ['<option value="__new">\u2795 new session</option>'];
1717 for (const s of sessions) {
1718 const label = s.title + (s.last ? " \u2014 " + agoShort(s.last) : "");
1719 opts.push('<option value="' + esc(s.id) + '"'
1720 + (s.id === SESSION_ID ? " selected" : "") + ">"
1721 + esc(label) + "</option>");
1722 }
1723 sel.innerHTML = opts.join("");
1724 if (!sessions.some((s) => s.id === SESSION_ID)) sel.value = "__new";
1725}
1726
1727async function loadSessions() {
1728 const res = await fetch("/collections/shell_commands/records?limit=1000",
1729 {credentials: "same-origin", headers: {accept: "application/json"}});
1730 if (!res.ok) return;
1731 sessions = foldSessions((await res.json()).records || []);
1732 renderSessionPicker();
1733
1734 const sel = document.getElementById("sessionpick");
1735 if (sel) sel.addEventListener("change", () => {
1736 if (sel.value === "__new") {
1737 SESSION_ID = freshSessionId();
1738 aiHistory = [];
1739 lastReaderPage = null;
1740 if (headphonesArmed) stopHeadphonesMode(document.getElementById("mic"));
1741 capAssistant.textContent = "new session";
1742 renderSessionPicker();
1743 return;
1744 }
1745 const s = sessions.find((x) => x.id === sel.value);
1746 if (!s) return;
1747 SESSION_ID = s.id;
1748 aiHistory = [];
1749 lastReaderPage = null;
1750 if (headphonesArmed) stopHeadphonesMode(document.getElementById("mic"));
1751 for (const row of s.rows.slice(-10)) {
1752 if (row.kind === "ai" && row.output) {
1753 aiHistory.push({role: "user", content: row.input});
1754 aiHistory.push({role: "assistant", content: row.output});
1755 }
1756 }
1757 // Say what just happened, on the surface the user is actually using:
1758 // the context changed, and a voice page must not do that silently.
1759 capAssistant.textContent = "resumed \u201c" + s.title + "\u201d ("
1760 + s.count + " turns)";
1761 renderSessionPicker();
1762 });
1763}
1764loadSessions();
1765
1766"""
1767
1768
1769def GET(request):
1770 identity = request.get("_identity", {})
1771 user_id = identity.get("user_id")
1772 _logger.info("site_talk served", user_id=user_id or "anonymous")
1773
1774 if not user_id:
1775 body = '<p class="hint"><a href="/login?next=/talk">Sign in</a> to use Talk.</p>'
1776 script = ""
1777 else:
1778 body = """
1779<div class="talkwrap">
1780<div id="stage" class="stage"><div class="card placeholder">Tap the mic and talk, or type below.</div></div>
1781<div class="bar">
1782<div class="captions">
1783<div id="capUser" class="cap user"></div>
1784<div id="capAssistant" class="cap assistant"></div>
1785</div>
1786<div id="endpointhint" class="endpointhint" hidden></div>
1787<div class="controls">
1788<div class="micwrap">
1789<button type="button" id="mic" hidden aria-label="toggle conversation mode">mic</button>
1790<span id="miclevel" class="miclevel" hidden></span>
1791</div>
1792<form id="prompt" autocomplete="off">
1793<input name="line" placeholder="or type..." autofocus>
1794<button type="submit" class="btn primary" aria-label="send">send</button>
1795</form>
1796<select id="sessionpick" aria-label="conversation">
1797<option value="__new">&#10133; new session</option></select>
1798<button type="button" id="cloudvoice" class="backlink" hidden>cloud voice: off</button>
1799<button type="button" id="headphonestoggle" class="backlink" hidden>headphones: off</button>
1800<a class="backlink" href="/shell">back to shell</a>
1801</div>
1802</div>
1803</div>
1804"""
1805 script = f"<script>const OWNER_ID = {user_id!r}; const TALK_SYSTEM = {json.dumps(TALK_SYSTEM)};{_SCRIPT}</script>"
1806
1807 who = (
1808 f"signed in as <strong>{user_id}</strong>"
1809 if user_id
1810 else '<a href="/login?next=/talk">sign in</a>'
1811 )
1812 html = f"""<!doctype html>
1813<html lang="en">
1814<head>
1815<meta charset="utf-8">
1816<meta name="viewport" content="width=device-width, initial-scale=1">
1817<title>Talk</title>
1818<link rel="stylesheet" href="/style">
1819<style>{_STYLE}</style>
1820</head>
1821<body>
1822<div class="wrap">
1823<header class="app"><h1>Talk</h1><div class="who">{who}</div></header>
1824{body}
1825</div>
1826<script src="/markdown"></script>
1827{script}
1828<script src="/nav"></script>
1829</body>
1830</html>"""
1831 return {"content_type": "text/html; charset=utf-8", "body": html}