askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
talk.py · 78.5 KB · 1831 lines
| 1 | """Talk: the shell as a stage instead of a transcript. |
| 2 | |
| 3 | Same brain as /shell -- one POST /api/ai/chat, the same prefs/model/tools, |
| 4 | the same shell_commands table -- projected differently. The shell renders |
| 5 | a scrolling keyboard log; Talk fills the screen with whatever the |
| 6 | conversation just produced (a spoken answer or a materialized /views page) |
| 7 | and reduces the transcript to a single caption strip. No server route is |
| 8 | new here: this object is a second window onto the same conversation. |
| 9 | |
| 10 | Every turn is still recorded to shell_commands (Talk and the shell share |
| 11 | that record), but Talk does NOT replay old shell_commands rows into the |
| 12 | model's context on load. A model given a transcript of *any* age of |
| 13 | in-context examples will imitate the habits those examples demonstrate |
| 14 | even when the current instructions say otherwise -- concretely, replaying |
| 15 | turns recorded before the [[view:<id>]] marker convention existed taught |
| 16 | the model to keep ignoring the marker instruction, no matter how the |
| 17 | system prompt was reworded. So each Talk page load starts aiHistory empty |
| 18 | and it only ever accumulates turns from *this* session. |
| 19 | |
| 20 | _BASE_CAPABILITIES below is copied verbatim from shell.py's system prompt |
| 21 | (including the views MATERIALIZE PAGES block) so the two stay in sync -- |
| 22 | edit both together. Talk adds one addendum on top: short spoken replies, |
| 23 | and never speaking ids/urls/paths aloud. |
| 24 | """ |
| 25 | |
| 26 | import json |
| 27 | |
| 28 | # Page-unique layout only; palette, chrome, and inputs come from /style, so |
| 29 | # Talk reskins with the active theme like every other page. |
| 30 | _STYLE = """ |
| 31 | /* The page must FIT the viewport: the mic, the captions and the text box |
| 32 | are one control surface, and a layout where you scroll between the nav |
| 33 | and the input is one you cannot operate. The stage takes what is left |
| 34 | after the fixed-height rows rather than claiming 85vh of its own -- |
| 35 | which is what pushed the input off-screen on a tall tablet. |
| 36 | dvh where supported (iOS Safari's toolbars shrink the visible area and |
| 37 | 100vh lies about it); -webkit-fill-available for older iPadOS that has |
| 38 | neither dvh nor svh; plain vh last for everything else. */ |
| 39 | .talkwrap { display: flex; flex-direction: column; |
| 40 | height: calc(100vh - 3.5rem); |
| 41 | height: -webkit-fill-available; |
| 42 | height: calc(100dvh - 3.5rem); |
| 43 | max-height: calc(100dvh - 3.5rem); overflow: hidden; } |
| 44 | .stage { flex: 1 1 auto; min-height: 0; display: flex; align-items: center; |
| 45 | justify-content: center; padding: 1rem; overflow: hidden; } |
| 46 | .stage .card { background: var(--panel); border: 1px solid var(--line); border-radius: var(--radius-md); |
| 47 | padding: 2rem; max-width: 640px; max-height: 100%; overflow-y: auto; |
| 48 | font-size: 1.4rem; line-height: 1.5; text-align: center; } |
| 49 | .stage .card.placeholder { color: var(--muted); font-size: 1.1rem; } |
| 50 | .stage .card a { color: var(--accent-strong); } |
| 51 | .stage .card code { background: var(--panel-2); color: var(--accent-strong); |
| 52 | padding: 1px 5px; border-radius: 3px; font-size: 0.9em; } |
| 53 | .stageframe { width: 100%; height: 100%; border: 1px solid var(--line); border-radius: var(--radius-md); } |
| 54 | .bar { border-top: 1px solid var(--line); padding: 0.75rem 1rem; display: flex; flex-direction: column; gap: 0.5rem; } |
| 55 | .captions { min-height: 2.6rem; font-size: 0.95rem; } |
| 56 | .endpointhint { font-size: 0.78rem; opacity: 0.75; padding: 0 0.2rem 0.2rem; } |
| 57 | #sessionpick { max-width: 11rem; font-size: 0.78rem; opacity: 0.85; } |
| 58 | .cap.user { color: var(--muted); } |
| 59 | .cap.assistant { color: var(--text); font-weight: 600; } |
| 60 | .controls { display: flex; align-items: center; gap: 0.75rem; } |
| 61 | .micwrap { position: relative; flex: 0 0 auto; } |
| 62 | #mic { width: 4.5rem; height: 4.5rem; border-radius: 50%; font-size: 0.75rem; |
| 63 | background: var(--panel-2); border: 1px solid var(--line); color: var(--muted); cursor: pointer; } |
| 64 | #mic.on { color: var(--danger); border-color: var(--danger); background: var(--panel); } |
| 65 | /* Level meter ring: a privacy indicator as much as a VU meter -- it must |
| 66 | read as visibly dead (opacity 0, no shadow) whenever the mic is off. */ |
| 67 | .miclevel { position: absolute; inset: -6px; border-radius: 50%; pointer-events: none; |
| 68 | opacity: 0; box-shadow: 0 0 0 0 var(--danger); transition: opacity 80ms linear; } |
| 69 | form#prompt { flex: 1; display: flex; gap: 0.5rem; } |
| 70 | form#prompt input { flex: 1; } |
| 71 | .backlink { color: var(--muted); font-size: 0.82rem; white-space: nowrap; } |
| 72 | /* #cloudvoice is a <button> wearing .backlink's look, not an <a> -- |
| 73 | strip the browser's default button chrome so it reads as the same |
| 74 | plain inline text control as the "back to shell" link beside it. */ |
| 75 | #cloudvoice, #headphonestoggle { background: none; border: none; padding: 0; font: inherit; cursor: pointer; } |
| 76 | /* Caption states: armed (waiting for the wake word), active (live capture |
| 77 | or the assistant's reply), sent (what was just submitted, muted). */ |
| 78 | .cap.user.armed { font-style: italic; opacity: 0.7; } |
| 79 | .cap.user.active { color: var(--text); opacity: 1; font-style: normal; } |
| 80 | .cap.user.sent { color: var(--muted); opacity: 1; font-style: normal; } |
| 81 | """ |
| 82 | |
| 83 | # Copied verbatim from shell.py's /api/ai/chat system prompt so the two |
| 84 | # stay in sync. Edit both together. |
| 85 | _BASE_CAPABILITIES = ( |
| 86 | "You are the shell of this user's object server. Answer in plain terminal text " |
| 87 | "with no markdown formatting. Be concise. Use your tools when the question is " |
| 88 | "about the user's records. " |
| 89 | "You can also MATERIALIZE PAGES: the views collection turns records into live " |
| 90 | "pages. When the user asks for a page/dashboard/view (or an answer clearly worth " |
| 91 | "keeping as one), create a views record: fields title, layout 'single', " |
| 92 | "owner_id (the user), pinned 'false', is_public 'false', and blocks = a JSON " |
| 93 | "string of a list of block objects. Block kinds: " |
| 94 | "{kind:'count', collection, filters:{field:value}, label, warn_over?} | " |
| 95 | "{kind:'list', collection, filters?, sort?:'newest'|'oldest', title?} | " |
| 96 | "{kind:'form', collection, record_id?} | " |
| 97 | "{kind:'detail', collection, record_id} | " |
| 98 | "{kind:'markdown', text}. " |
| 99 | "After creating it, tell the user the page is at /views/{id} (the record id). " |
| 100 | "Prefer a count block above a list block for status-style pages. " |
| 101 | "You do NOT know from memory which collections exist. Call list_collections " |
| 102 | "and read its result before answering; if the collection the user named " |
| 103 | "appears in that result, USE it -- never say it is missing when it is in " |
| 104 | "the list. The user's tasks live in the collection named tasks. " |
| 105 | "To show one specific record on screen, create a view whose blocks contain a " |
| 106 | "detail block for it. Never claim something is on screen unless you created " |
| 107 | "or updated a views record in this same turn. " |
| 108 | "Whenever the screen should show a view -- newly created OR one that already " |
| 109 | "exists -- end your reply with the marker [[view:<record id>]] alone on the " |
| 110 | "last line. The marker is machine-read; it is never displayed or spoken, so " |
| 111 | "it does not violate the no-ids-aloud rule. " |
| 112 | 'Example reply: "Here are your open tasks. ' |
| 113 | '[[view:26b247ed-3b1a-4206-b060-1d92847194de]]"' |
| 114 | " You can also READ WEB PAGES with the read_page tool when the user gives a " |
| 115 | "URL or asks you to read/summarize a page: it returns the page text and its " |
| 116 | 'links numbered in order. Offer the links as "link 1, link 2, ..." so the ' |
| 117 | 'user can say "open link N" and you read_page that link\'s url next.' |
| 118 | ) |
| 119 | |
| 120 | _TALK_ADDENDUM = ( |
| 121 | " You are in voice mode. Reply in one or two short spoken sentences. When the " |
| 122 | "user asks to see, list, or track anything, create (or update) a views record " |
| 123 | "and say you have put it on screen. NEVER read ids, urls, uuids, or paths aloud." |
| 124 | ) |
| 125 | |
| 126 | TALK_SYSTEM = _BASE_CAPABILITIES + _TALK_ADDENDUM |
| 127 | |
| 128 | _SCRIPT = """ |
| 129 | // One conversation per page load. The server stamps this on every turn it |
| 130 | // records, which is what makes a session listable and resumable later. |
| 131 | let SESSION_ID = freshSessionId(); |
| 132 | function freshSessionId() { |
| 133 | return crypto.randomUUID ? crypto.randomUUID() |
| 134 | : "s-" + Math.random().toString(36).slice(2); |
| 135 | } |
| 136 | |
| 137 | const esc = (s) => String(s ?? "").replace(/[&<>"']/g, |
| 138 | (c) => ({"&": "&", "<": "<", ">": ">", '"': """, "'": "'"}[c])); |
| 139 | const stage = document.getElementById("stage"); |
| 140 | const capUser = document.getElementById("capUser"); |
| 141 | const capAssistant = document.getElementById("capAssistant"); |
| 142 | let prefs = {id: OWNER_ID, ai_model: "anthropic:claude-sonnet-5", |
| 143 | tools: "global_search,list_collections,list_records,get_record,create_record,update_record,read_page", |
| 144 | talk_wake_word: "computer", talk_end_word: "over", |
| 145 | talk_endpoint: "silence", talk_silence_ms: "1400", talk_tts: "auto", |
| 146 | talk_tts_engine: "local", talk_stt_engine: "browser"}; |
| 147 | let aiHistory = []; |
| 148 | // The real link data from the most recent read_page call -- {final_url, |
| 149 | // title, links: [{n, label, href}]} or null. Conversation history |
| 150 | // (aiHistory) only ever carries the model's own prose, never a tool's |
| 151 | // actual result, so a page's real hrefs used to vanish the instant the |
| 152 | // model finished replying -- a later "go to that story" had nothing but |
| 153 | // the model's own memory of what it said to resolve against, which broke |
| 154 | // completely on any page where voice mode's brevity instruction won over |
| 155 | // citing link numbers (a busy news homepage, for instance). Replaced |
| 156 | // wholesale by the next read_page call; otherwise persists for the rest |
| 157 | // of the session, same as aiHistory. |
| 158 | let lastReaderPage = null; |
| 159 | |
| 160 | // Grounding text appended to the system prompt every turn while |
| 161 | // lastReaderPage is set -- real hrefs and labels the model can resolve a |
| 162 | // vague reference against ("go to that story"), not just its own memory |
| 163 | // of a summary it gave several turns ago. Empty string (a no-op append) |
| 164 | // once no page has been read yet, or after a session reset. |
| 165 | function readerContextNote() { |
| 166 | if (!lastReaderPage || !Array.isArray(lastReaderPage.links) || !lastReaderPage.links.length) return ""; |
| 167 | const list = lastReaderPage.links.map((l) => l.n + ". " + l.label + " (" + l.href + ")").join(" "); |
| 168 | return " The last web page you read was '" + (lastReaderPage.title || lastReaderPage.final_url) |
| 169 | + "' (" + lastReaderPage.final_url + "). Its links, in case the user asks to open one or refers " |
| 170 | + "to something on that page: " + list; |
| 171 | } |
| 172 | |
| 173 | const TTS_MAX_CHARS = 800; |
| 174 | const VIEWS_PATH_RE = /\\/views\\/[A-Za-z0-9_-]+/; |
| 175 | |
| 176 | // Preference reads all go through pref() rather than raw prefs.x access: |
| 177 | // loadPrefs() replaces `prefs` wholesale with whatever record comes back |
| 178 | // from the server, and a record written before a field existed in the |
| 179 | // schema simply won't have that key. pref() falls back to the shipped |
| 180 | // default only when the key is *missing* -- an explicit empty string (the |
| 181 | // wake/end word's "off" setting) is a real value, not a gap, and must be |
| 182 | // left alone. |
| 183 | const DEFAULT_WAKE_WORD = "computer"; |
| 184 | const DEFAULT_END_WORD = "over"; |
| 185 | const DEFAULT_ENDPOINT = "silence"; |
| 186 | const DEFAULT_SILENCE_MS = 1400; |
| 187 | const DEFAULT_TOOLS = "global_search,list_collections,list_records,get_record,create_record,update_record,read_page"; |
| 188 | const DEFAULT_MODEL = "anthropic:claude-sonnet-5"; |
| 189 | const DEFAULT_TALK_TTS = "auto"; |
| 190 | const DEFAULT_TALK_TTS_ENGINE = "local"; |
| 191 | const DEFAULT_TALK_STT_ENGINE = "browser"; |
| 192 | |
| 193 | function pref(name, fallback) { |
| 194 | const v = prefs[name]; |
| 195 | return v === undefined || v === null ? fallback : v; |
| 196 | } |
| 197 | function wakeWord() { return String(pref("talk_wake_word", DEFAULT_WAKE_WORD)).trim(); } |
| 198 | function endWord() { return String(pref("talk_end_word", DEFAULT_END_WORD)).trim(); } |
| 199 | function endpointMode() { |
| 200 | const m = String(pref("talk_endpoint", DEFAULT_ENDPOINT)).trim(); |
| 201 | return (m === "word" || m === "manual" || m === "silence") ? m : DEFAULT_ENDPOINT; |
| 202 | } |
| 203 | function talkTtsMode() { |
| 204 | const m = String(pref("talk_tts", DEFAULT_TALK_TTS)).trim(); |
| 205 | const mode = (m === "server" || m === "browser") ? m : DEFAULT_TALK_TTS; |
| 206 | // A resolved engine of "openai" -- from settings OR a ?tts= URL |
| 207 | // override -- is an explicit request to hear the cloud voice. "auto" |
| 208 | // otherwise prefers ANY browser-native speechSynthesis voice over the |
| 209 | // server, and on many setups (Chrome on Linux, notably) that native |
| 210 | // voice is ITSELF backed by espeak -- indistinguishable by ear from |
| 211 | // this server's own local engine. Without this check, choosing openai |
| 212 | // (or passing ?tts=openai) would silently do nothing whenever the |
| 213 | // browser had any local voice at all: exactly what one live test hit. |
| 214 | if (mode === "auto" && talkTtsEngine() === "openai") return "server"; |
| 215 | return mode; |
| 216 | } |
| 217 | // ?tts=openai|local and ?stt=openai|browser override the stored preference |
| 218 | // for THIS page load only -- nothing is saved back to shell_preferences. |
| 219 | // Exists so a link (e.g. "/talk?stt=openai") can be tried or shared |
| 220 | // without visiting settings first; the stored preference underneath is |
| 221 | // exactly what a plain "/talk" load still honors. |
| 222 | function urlOverride(param, allowed) { |
| 223 | const v = new URLSearchParams(location.search).get(param); |
| 224 | return allowed.includes(v) ? v : null; |
| 225 | } |
| 226 | function talkTtsEngine() { |
| 227 | const override = urlOverride("tts", ["local", "openai"]); |
| 228 | if (override) return override; |
| 229 | const m = String(pref("talk_tts_engine", DEFAULT_TALK_TTS_ENGINE)).trim(); |
| 230 | return m === "openai" ? "openai" : DEFAULT_TALK_TTS_ENGINE; |
| 231 | } |
| 232 | // "openai" here is deliberately opt-in only, never an automatic fallback |
| 233 | // for browsers lacking SpeechRecognition -- a Firefox user who never |
| 234 | // touched this setting (or the URL override) and has no stored OpenAI key |
| 235 | // must keep seeing today's exact behavior (mic hidden), not a confusing |
| 236 | // "no key stored" error from a feature they never asked for. See |
| 237 | // initMic()/initCloudMic(). |
| 238 | function talkSttEngine() { |
| 239 | const override = urlOverride("stt", ["browser", "openai"]); |
| 240 | if (override) return override; |
| 241 | const m = String(pref("talk_stt_engine", DEFAULT_TALK_STT_ENGINE)).trim(); |
| 242 | return m === "openai" ? "openai" : DEFAULT_TALK_STT_ENGINE; |
| 243 | } |
| 244 | function silenceMs() { |
| 245 | const n = Number(pref("talk_silence_ms", DEFAULT_SILENCE_MS)); |
| 246 | return Number.isFinite(n) && n > 0 ? n : DEFAULT_SILENCE_MS; |
| 247 | } |
| 248 | |
| 249 | // Strip markdown -- and, unlike the shell's stripForSpeech, urls and |
| 250 | // /views paths too -- down to sentences worth speaking. The talk-mode |
| 251 | // system prompt tells the model never to read an id/url/path aloud; this |
| 252 | // is the belt to that suspenders in case it slips one in anyway. |
| 253 | function stripForSpeech(text) { |
| 254 | return String(text ?? "") |
| 255 | .replace(/```[\\s\\S]*?```/g, " ") |
| 256 | .replace(/`[^`]*`/g, " ") |
| 257 | .replace(/!\\[[^\\]]*\\]\\([^)]*\\)/g, " ") |
| 258 | .replace(/\\[([^\\]]*)\\]\\([^)]*\\)/g, "$1") |
| 259 | .replace(/https?:\\/\\/\\S+/g, " ") |
| 260 | .replace(/\\/views\\/\\S+/g, " ") |
| 261 | .replace(/\\[\\[view:[^\\]]*\\]\\]/g, " ") |
| 262 | .replace(/[*_#>~]/g, " ") |
| 263 | .replace(/\\s+/g, " ") |
| 264 | .trim() |
| 265 | .slice(0, TTS_MAX_CHARS); |
| 266 | } |
| 267 | |
| 268 | function renderCard(text) { |
| 269 | const card = document.createElement("div"); |
| 270 | card.className = "card"; |
| 271 | card.innerHTML = window.dbbasicMarkdown ? window.dbbasicMarkdown(text) : esc(text); |
| 272 | stage.innerHTML = ""; |
| 273 | stage.appendChild(card); |
| 274 | } |
| 275 | |
| 276 | function renderStageView(path) { |
| 277 | stage.innerHTML = `<iframe src="${esc(path)}?embed=1" class="stageframe"></iframe>`; |
| 278 | } |
| 279 | |
| 280 | // The machine channel: the model ends a reply with [[view:<id>]] when the |
| 281 | // stage should show that view (new OR already-existing) -- explicit |
| 282 | // signaling instead of path-sniffing, since spoken replies never carry |
| 283 | // paths. The marker is stripped from everything displayed and spoken. |
| 284 | const VIEW_MARKER_RE = /\\[\\[view:([A-Za-z0-9-]+)\\]\\]/; |
| 285 | |
| 286 | function stripViewMarker(text) { |
| 287 | return String(text ?? "").replace(/\\[\\[view:[^\\]]*\\]\\]/g, " ").trim(); |
| 288 | } |
| 289 | |
| 290 | function viewPathFromReply(text) { |
| 291 | const marker = String(text || "").match(VIEW_MARKER_RE); |
| 292 | if (marker) return "/views/" + marker[1]; |
| 293 | const match = String(text || "").match(VIEWS_PATH_RE); |
| 294 | return match ? match[0].replace(/[.,;:)]+$/, "") : null; |
| 295 | } |
| 296 | |
| 297 | // Fallback for when the model materialized a page but its spoken reply |
| 298 | // never mentions the path -- inspect the tool calls the chat turn made |
| 299 | // (that's what /api/ai/chat actually returns: name/arguments/http_status |
| 300 | // per call) for a create_record or update_record on the views collection. |
| 301 | function viewPathFromToolCalls(toolCalls) { |
| 302 | if (!Array.isArray(toolCalls)) return null; |
| 303 | for (let i = toolCalls.length - 1; i >= 0; i--) { |
| 304 | const call = toolCalls[i]; |
| 305 | const args = call && call.arguments; |
| 306 | if (!args || args.collection !== "views") continue; |
| 307 | const id = (call.name === "create_record" && args.record && args.record.id) |
| 308 | || (call.name === "update_record" && args.record_id); |
| 309 | if (id) return "/views/" + id; |
| 310 | } |
| 311 | return null; |
| 312 | } |
| 313 | |
| 314 | let speaking = false; |
| 315 | |
| 316 | // Voice picking for the browser speechSynthesis engine: prefer a |
| 317 | // natural-sounding named voice, then the first on-device (localService) |
| 318 | // English voice, else leave utter.voice unset and let the browser use its |
| 319 | // own default. |
| 320 | const PREFERRED_VOICE_RE = /Samantha|Ava|Karen|Daniel/; |
| 321 | function getVoices() { |
| 322 | return (window.speechSynthesis && window.speechSynthesis.getVoices()) || []; |
| 323 | } |
| 324 | function pickVoice() { |
| 325 | const voices = getVoices(); |
| 326 | if (!voices.length) return null; |
| 327 | const byName = voices.find((v) => PREFERRED_VOICE_RE.test(v.name)); |
| 328 | if (byName) return byName; |
| 329 | const localEn = voices.find((v) => v.localService && /^en/i.test(v.lang)); |
| 330 | return localEn || null; |
| 331 | } |
| 332 | // The "auto" engine gate: is there at least one on-device voice at all |
| 333 | // (any language) -- if so, speaking never has to leave the device. |
| 334 | function hasLocalVoice() { |
| 335 | return getVoices().some((v) => v.localService); |
| 336 | } |
| 337 | |
| 338 | // speechSynthesis-only speaking path, used both when talk_tts is "browser" |
| 339 | // and as the fallback when "server"/"auto" server TTS fails. Carries the |
| 340 | // exact same speaking-flag/listen-pause contract as the server path below: |
| 341 | // set before speaking, cleared and resumed only once the utterance ends. |
| 342 | async function speakBrowser(text) { |
| 343 | if (!window.speechSynthesis) { speaking = false; resumeListeningIfNeeded(); return; } |
| 344 | speaking = true; |
| 345 | await stopListeningAndWaitForRelease(); |
| 346 | window.speechSynthesis.cancel(); |
| 347 | const utter = new SpeechSynthesisUtterance(text); |
| 348 | const voice = pickVoice(); |
| 349 | if (voice) utter.voice = voice; |
| 350 | utter.addEventListener("end", stopSpeaking); |
| 351 | utter.addEventListener("error", stopSpeaking); |
| 352 | window.speechSynthesis.speak(utter); |
| 353 | } |
| 354 | |
| 355 | // Speak one assistant reply, routed by the talk_tts preference: |
| 356 | // "server" -- always server TTS (POST /api/tts), speechSynthesis on failure. |
| 357 | // "browser" -- always speechSynthesis, no server round-trip. |
| 358 | // "auto" -- speechSynthesis if an on-device voice is available, else |
| 359 | // server-with-fallback, same as "server". |
| 360 | // Whichever path runs, the mic is kept stopped for the whole span and only |
| 361 | // resumed once playback actually ends, so the recognizer never hears the |
| 362 | // machine talking to itself. |
| 363 | async function speak(text) { |
| 364 | const spoken = stripForSpeech(text); |
| 365 | if (!spoken) { resumeListeningIfNeeded(); return; } |
| 366 | |
| 367 | const mode = talkTtsMode(); |
| 368 | if (mode === "browser") { speakBrowser(spoken); return; } |
| 369 | if (mode === "auto" && window.speechSynthesis && hasLocalVoice()) { |
| 370 | speakBrowser(spoken); |
| 371 | return; |
| 372 | } |
| 373 | |
| 374 | speaking = true; |
| 375 | await stopListeningAndWaitForRelease(); |
| 376 | try { |
| 377 | const res = await fetch("/api/tts", { |
| 378 | method: "POST", credentials: "same-origin", |
| 379 | headers: {"content-type": "application/json", accept: "audio/*"}, |
| 380 | body: JSON.stringify({text: spoken, engine: talkTtsEngine()}), |
| 381 | }); |
| 382 | if (!res.ok) throw new Error("tts endpoint failed"); |
| 383 | const url = URL.createObjectURL(await res.blob()); |
| 384 | // The SAME element every time, never a fresh one. iOS allows play() |
| 385 | // outside a user gesture only on an element that has already played |
| 386 | // inside one -- primeAudioForIOS() plays this element silent during |
| 387 | // the mic tap, and changing src afterwards keeps the unlock. `new |
| 388 | // Audio(url)` here was born locked, its play() rejected, and the |
| 389 | // catch fell back to speechSynthesis... which the mute switch |
| 390 | // silences. Server voice configured, server WAV fetched, and still |
| 391 | // no sound: this line is why. |
| 392 | const player = sharedTtsAudio(); |
| 393 | player.pause(); |
| 394 | player.src = url; |
| 395 | currentTtsObjectUrl = url; |
| 396 | // EVERY exit clears `speaking`, not just the happy one. It gates the |
| 397 | // mic restart AND the transcript-stability submit, so a playback that |
| 398 | // ends without an `ended` event -- a bad WAV, a decode error, an |
| 399 | // interrupted element -- used to leave the page deaf and mute |
| 400 | // forever: mic never resumed, submits silently blocked, and nothing |
| 401 | // on screen said why. Found by the harness, whose spied play() never |
| 402 | // completed, which is exactly the shape of a real playback failure. |
| 403 | player.onended = stopSpeaking; |
| 404 | player.onerror = stopSpeaking; |
| 405 | player.onstalled = stopSpeaking; |
| 406 | await player.play(); |
| 407 | } catch (e) { |
| 408 | speakBrowser(spoken); |
| 409 | } |
| 410 | } |
| 411 | |
| 412 | // The object URL a cloud-TTS playback is currently using, or null -- |
| 413 | // tracked at module scope (not just inside speak()'s closure) so |
| 414 | // stopSpeaking() can revoke it even when called from OUTSIDE a normal |
| 415 | // playback-ended event, i.e. an interrupt. |
| 416 | let currentTtsObjectUrl = null; |
| 417 | |
| 418 | // Shared end-of-speech cleanup, whether playback finished naturally (an |
| 419 | // "ended"/"error"/"stalled" event) or was interrupted deliberately |
| 420 | // (headphones mode's VAD hearing the user start talking mid-reply -- see |
| 421 | // startHeadphonesMode below). Every exit clears `speaking`, not just the |
| 422 | // happy one: a playback that never fires its own end event used to leave |
| 423 | // the page deaf and mute forever (mic never resumed, submits silently |
| 424 | // blocked). Safe to call when nothing is actually playing (window.speechSynthesis.cancel() |
| 425 | // and player.pause() are no-ops in that case). |
| 426 | function stopSpeaking() { |
| 427 | const player = sharedTtsAudio(); |
| 428 | player.onended = player.onerror = player.onstalled = null; |
| 429 | player.pause(); |
| 430 | if (currentTtsObjectUrl) { URL.revokeObjectURL(currentTtsObjectUrl); currentTtsObjectUrl = null; } |
| 431 | if (window.speechSynthesis) window.speechSynthesis.cancel(); |
| 432 | speaking = false; |
| 433 | resumeListeningIfNeeded(); |
| 434 | } |
| 435 | |
| 436 | // Conversation-mode mic: the button toggles a mode, not a single listen. |
| 437 | // While the mode is on, onend auto-restarts recognition -- unless we are |
| 438 | // currently speaking (playing TTS), in which case resumeListeningIfNeeded() |
| 439 | // (called from speak()'s completion handlers above) is what restarts it. |
| 440 | const SpeechRecognitionCtor = window.SpeechRecognition || window.webkitSpeechRecognition; |
| 441 | let recognizer = null; |
| 442 | let conversationMode = false; |
| 443 | let listening = false; |
| 444 | |
| 445 | // --- Radio protocol: wake word / end word / VAD silence endpointing ------ |
| 446 | // |
| 447 | // `buffer` holds finalized transcript from earlier recognizer sessions in |
| 448 | // the current utterance; `sessionLive` holds the current (possibly still |
| 449 | // interim) recognizer session's transcript. Both are already wake-word- |
| 450 | // stripped once armed flips false, so most readers just concatenate them. |
| 451 | // |
| 452 | // Word-boundary matching is done by tokenizing on whitespace rather than |
| 453 | // regex word-boundary escapes -- this file has already had one bug from a |
| 454 | // backslash escape that didn't survive its Python-string layer, so the |
| 455 | // word-match helpers below avoid backslash metacharacters entirely. |
| 456 | let buffer = ""; |
| 457 | let sessionLive = ""; |
| 458 | let armed = true; |
| 459 | let finalDebounceTimer = null; |
| 460 | let stabilityTimer = null; |
| 461 | |
| 462 | function tokenize(text) { |
| 463 | return String(text ?? "").trim().split(/\\s+/).filter(Boolean); |
| 464 | } |
| 465 | |
| 466 | // Strip leading/trailing punctuation from one token for comparison -- |
| 467 | // "computer," or "over." still match "computer" / "over". |
| 468 | function wordKey(token) { |
| 469 | return token.replace(/[.,!?;:]+$/, "").replace(/^[.,!?;:]+/, "").toLowerCase(); |
| 470 | } |
| 471 | |
| 472 | // If `word` appears as a whole token in `text`, return everything after |
| 473 | // its first occurrence (joined back with spaces); otherwise null. |
| 474 | // Bounded edit distance for wake-word tolerance. Recognition mishears: |
| 475 | // an iPad delivered "compture" for "computer", and an exact match then |
| 476 | // silently discarded everything the user said after it -- armed forever, |
| 477 | // capturing nothing, with the transcript ON SCREEN making it look heard. |
| 478 | // The budget scales with length (short words stay exact: "hey" must not |
| 479 | // match "they"), and it is capped at 2 because a wake word is a gate, not |
| 480 | // a search. Worst case of a false arm: the mic captures a sentence the |
| 481 | // user then sees -- recoverable. Worst case of a strict miss: everything |
| 482 | // said is thrown away invisibly -- not recoverable, and that asymmetry is |
| 483 | // the whole argument for leniency here. |
| 484 | function editDistanceAtMost(a, b, budget) { |
| 485 | if (Math.abs(a.length - b.length) > budget) return false; |
| 486 | // Damerau (OSA), not plain Levenshtein, and the difference is the |
| 487 | // whole point: a transposition costs 1. The observed mishear was |
| 488 | // "compture" for "computer" -- two adjacent swaps, Damerau distance 2, |
| 489 | // plain distance 3 -- so plain Levenshtein would have rejected the |
| 490 | // exact case this function exists for. Swapped letters are what |
| 491 | // recognition mishears LOOK like. |
| 492 | let prevPrev = null; |
| 493 | let prev = []; |
| 494 | for (let j = 0; j <= b.length; j++) prev.push(j); |
| 495 | for (let i = 1; i <= a.length; i++) { |
| 496 | const row = [i]; |
| 497 | let best = i; |
| 498 | for (let j = 1; j <= b.length; j++) { |
| 499 | const cost = a[i - 1] === b[j - 1] ? 0 : 1; |
| 500 | let v = Math.min(prev[j] + 1, row[j - 1] + 1, prev[j - 1] + cost); |
| 501 | if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1]) { |
| 502 | v = Math.min(v, prevPrev[j - 2] + 1); |
| 503 | } |
| 504 | row.push(v); |
| 505 | if (v < best) best = v; |
| 506 | } |
| 507 | if (best > budget) return false; // the whole row is over budget |
| 508 | prevPrev = prev; |
| 509 | prev = row; |
| 510 | } |
| 511 | return prev[b.length] <= budget; |
| 512 | } |
| 513 | |
| 514 | function wakeBudget(target) { |
| 515 | if (target.length >= 7) return 2; |
| 516 | if (target.length >= 4) return 1; |
| 517 | return 0; |
| 518 | } |
| 519 | |
| 520 | function matchesWakeWord(tokenKey, target) { |
| 521 | if (tokenKey === target) return true; |
| 522 | return editDistanceAtMost(tokenKey, target, wakeBudget(target)); |
| 523 | } |
| 524 | |
| 525 | function findWakeSplit(text, word) { |
| 526 | const target = word.trim().toLowerCase(); |
| 527 | if (!target) return null; |
| 528 | const tokens = tokenize(text); |
| 529 | for (let i = 0; i < tokens.length; i++) { |
| 530 | const key = wordKey(tokens[i]); |
| 531 | if (matchesWakeWord(key, target)) return tokens.slice(i + 1).join(" "); |
| 532 | // A no-space transcript fuses the wake word onto what follows |
| 533 | // ("comptureshowmemynotes"). If a long token's PREFIX matches the |
| 534 | // wake word, the remainder of the token is the start of the command |
| 535 | // -- unspaced, but the model reads unspaced text far better than |
| 536 | // this page reads a discarded utterance. Prefix lengths of the |
| 537 | // target and one either side, so a mishear inside the fused prefix |
| 538 | // still arms. |
| 539 | if (key.length > target.length + wakeBudget(target)) { |
| 540 | for (const len of [target.length, target.length + 1, target.length - 1]) { |
| 541 | if (len < 3 || len >= key.length) continue; |
| 542 | if (matchesWakeWord(key.slice(0, len), target)) { |
| 543 | const rest = [key.slice(len)].concat(tokens.slice(i + 1)); |
| 544 | return rest.join(" ").trim(); |
| 545 | } |
| 546 | } |
| 547 | } |
| 548 | } |
| 549 | return null; |
| 550 | } |
| 551 | |
| 552 | // If `text`'s last token is `word`, return everything before it (joined |
| 553 | // back with spaces); otherwise null. Tolerant of trailing punctuation on |
| 554 | // the last token ("...turn it on, over." still matches "over"). |
| 555 | function tailMatchesEndWord(text, word) { |
| 556 | const target = word.trim().toLowerCase(); |
| 557 | if (!target) return null; |
| 558 | const tokens = tokenize(text); |
| 559 | if (!tokens.length) return null; |
| 560 | if (wordKey(tokens[tokens.length - 1]) !== target) return null; |
| 561 | return tokens.slice(0, -1).join(" "); |
| 562 | } |
| 563 | |
| 564 | function resetTalkState() { |
| 565 | buffer = ""; |
| 566 | sessionLive = ""; |
| 567 | armed = true; |
| 568 | if (finalDebounceTimer) { clearTimeout(finalDebounceTimer); finalDebounceTimer = null; } |
| 569 | utteranceStarted = false; |
| 570 | silenceStartTs = null; |
| 571 | speechHoldStart = null; |
| 572 | } |
| 573 | |
| 574 | function updateCaptionArmed() { |
| 575 | const w = wakeWord(); |
| 576 | capUser.textContent = w ? `say "${w}" to address me` : ""; |
| 577 | capUser.classList.remove("active", "sent"); |
| 578 | capUser.classList.add("armed"); |
| 579 | } |
| 580 | |
| 581 | function updateCaptionActive(text) { |
| 582 | capUser.textContent = text || "…"; |
| 583 | capUser.classList.remove("armed", "sent"); |
| 584 | capUser.classList.add("active"); |
| 585 | } |
| 586 | |
| 587 | // What would be submitted right now, with wake word and end word both |
| 588 | // stripped if present -- used by VAD endpointing and by manual send/Enter, |
| 589 | // which fold in whatever voice buffer exists regardless of protocol. |
| 590 | function pendingText() { |
| 591 | let text = ((buffer ? buffer + " " : "") + sessionLive).trim(); |
| 592 | const w = wakeWord(); |
| 593 | if (w) { |
| 594 | const split = findWakeSplit(text, w); |
| 595 | if (split !== null) text = split; |
| 596 | } |
| 597 | const ew = endWord(); |
| 598 | if (ew) { |
| 599 | const stripped = tailMatchesEndWord(text, ew); |
| 600 | if (stripped !== null) text = stripped; |
| 601 | } |
| 602 | return text; |
| 603 | } |
| 604 | |
| 605 | function bufferHasContent() { |
| 606 | return ((buffer ? buffer + " " : "") + sessionLive).trim().length > 0; |
| 607 | } |
| 608 | |
| 609 | // The single funnel for every voice-triggered submission (end word, VAD |
| 610 | // silence, isFinal-debounce fallback): clear/re-arm first so new speech |
| 611 | // during the in-flight chat call starts a fresh utterance, then hand off |
| 612 | // to the same submitTurn() the text box and Enter use. |
| 613 | function finalizeVoiceSubmit(text) { |
| 614 | if (finalDebounceTimer) { clearTimeout(finalDebounceTimer); finalDebounceTimer = null; } |
| 615 | if (stabilityTimer) { clearTimeout(stabilityTimer); stabilityTimer = null; } |
| 616 | const clean = String(text ?? "").trim(); |
| 617 | resetTalkState(); |
| 618 | if (!clean) { if (conversationMode) updateCaptionArmed(); return; } |
| 619 | // The wake word ALONE is a knock, not a question. One leaked through a |
| 620 | // gate seam on an iPad and bought a 2.4-second provider round-trip |
| 621 | // that answered "" -- money and silence for a turn nobody asked. The |
| 622 | // check is fuzzy like the gate itself, so a misheard bare "compture" |
| 623 | // is also swallowed. |
| 624 | const w = wakeWord(); |
| 625 | if (w) { |
| 626 | const only = tokenize(clean); |
| 627 | if (only.length === 1 && matchesWakeWord(wordKey(only[0]), w.toLowerCase())) { |
| 628 | if (conversationMode) updateCaptionArmed(); |
| 629 | return; |
| 630 | } |
| 631 | } |
| 632 | submitTurn(clean); |
| 633 | } |
| 634 | |
| 635 | // Called on every recognizer result (interim and final). Handles the wake |
| 636 | // gate, live caption, end-word override, and -- on a final result -- folds |
| 637 | // the session transcript into `buffer` and (endpoint "word" with no end |
| 638 | // word configured) arms the isFinal-debounce fallback. |
| 639 | function processTranscript(isFinal) { |
| 640 | const combinedRaw = ((buffer ? buffer + " " : "") + sessionLive).trim(); |
| 641 | const w = wakeWord(); |
| 642 | |
| 643 | // The split runs on EVERY event, not once at arming. Recognition |
| 644 | // transcripts are cumulative -- each interim restates the utterance |
| 645 | // from the beginning -- so a split applied only at arm time is undone |
| 646 | // by the very next event, and the wake word rides into the submitted |
| 647 | // command. Not hypothetical: the server's recorded turns include |
| 648 | // "Show me my notes computer show me my notes", the wake word twice, |
| 649 | // from exactly this. Splitting per event is idempotent: once buffer |
| 650 | // holds only command text, findWakeSplit finds nothing and passes it |
| 651 | // through untouched. |
| 652 | const split = w ? findWakeSplit(combinedRaw, w) : null; |
| 653 | if (armed) { |
| 654 | if (w && split === null) { |
| 655 | updateCaptionArmed(); |
| 656 | if (isFinal) sessionLive = ""; |
| 657 | return; |
| 658 | } |
| 659 | armed = false; // empty wake word: capture everything while the mic is on |
| 660 | } |
| 661 | |
| 662 | const active = (split !== null ? split : combinedRaw).trim(); |
| 663 | updateCaptionActive(active); |
| 664 | |
| 665 | const endpoint = endpointMode(); |
| 666 | const ew = endWord(); |
| 667 | if (ew && (endpoint === "word" || endpoint === "silence")) { |
| 668 | const stripped = tailMatchesEndWord(active, ew); |
| 669 | if (stripped !== null) { finalizeVoiceSubmit(stripped); return; } |
| 670 | } |
| 671 | |
| 672 | // TRANSCRIPT-STABILITY endpointing: on a device where the level meter |
| 673 | // cannot run (the iPad's exclusive microphone), end-of-utterance used |
| 674 | // to wait for iOS to volunteer an isFinal -- which it does 2-6 seconds |
| 675 | // after you stop, sometimes never. But the recognizer's own interims |
| 676 | // ARE a silence detector: while you speak the text keeps changing, and |
| 677 | // when it stops changing for the silence window, you have finished. |
| 678 | // Same signal the RMS meter derived from amplitude, taken from the |
| 679 | // transcript instead -- no second microphone stream, no contention. |
| 680 | // Backstop even when the meter runs: a mis-calibrated meter (started |
| 681 | // mid-speech, threshold learned from the voice itself) fails SILENT -- |
| 682 | // no error, no submit, ever. The transcript stalling is evidence the |
| 683 | // meter cannot fake, so this fires a beat after the VAD would have; a |
| 684 | // healthy VAD always wins first, and finalizeVoiceSubmit clears both |
| 685 | // timers so the loser never double-fires. |
| 686 | if (!armed && endpointMode() === "silence" && active) { |
| 687 | if (stabilityTimer) clearTimeout(stabilityTimer); |
| 688 | // Close over THIS event's already-split text. Rebuilding from the |
| 689 | // raw sessionLive here would resurrect the wake word the split just |
| 690 | // removed -- the timer is reset by every event, so the latest |
| 691 | // event's `active` is by definition the whole utterance so far. |
| 692 | stabilityTimer = setTimeout(() => { |
| 693 | stabilityTimer = null; |
| 694 | if (active && conversationMode && !speaking) finalizeVoiceSubmit(active); |
| 695 | }, silenceMs() + (meterRunning ? 500 : 0)); |
| 696 | } |
| 697 | |
| 698 | if (isFinal) { |
| 699 | buffer = active; |
| 700 | sessionLive = ""; |
| 701 | // `silence` mode with no working meter has no other way to end an |
| 702 | // utterance, so isFinal becomes the endpoint. Without this the mode |
| 703 | // is a dead end on any device where the mic cannot be opened twice. |
| 704 | if ((endpoint === "word" && !ew) || (endpoint === "silence" && !meterRunning)) { |
| 705 | // No end word configured for word mode: fall back to the original |
| 706 | // isFinal-triggered submit, debounced so a recognizer that fires |
| 707 | // several quick finals in a row (common near silence) only submits |
| 708 | // once. |
| 709 | if (finalDebounceTimer) clearTimeout(finalDebounceTimer); |
| 710 | finalDebounceTimer = setTimeout(() => { |
| 711 | finalDebounceTimer = null; |
| 712 | finalizeVoiceSubmit(buffer); |
| 713 | }, 300); |
| 714 | } |
| 715 | } |
| 716 | } |
| 717 | |
| 718 | // --- Mic level meter + VAD silence endpointing ---------------------------- |
| 719 | // |
| 720 | // A separate getUserMedia stream (SpeechRecognition exposes no raw audio) |
| 721 | // feeds an AnalyserNode; a rAF loop computes RMS to both draw the level |
| 722 | // ring around the mic button and, in "silence" endpoint mode, decide when |
| 723 | // an utterance has ended. The stream is only open while conversation mode |
| 724 | // is on -- it doubles as the privacy indicator, so it must go fully dead |
| 725 | // (tracks stopped, context closed) the instant the mode turns off. |
| 726 | const MIN_SPEECH_THRESHOLD = 0.015; |
| 727 | const CALIBRATION_MS = 800; |
| 728 | const SPEECH_HOLD_MS = 150; |
| 729 | |
| 730 | // Did the VAD meter actually come up? On iOS the microphone is EXCLUSIVE: |
| 731 | // SpeechRecognition and getUserMedia contend for it, so the second one to |
| 732 | // ask simply fails. Silence endpointing depends entirely on that second |
| 733 | // stream, and when it is absent nothing in `silence` mode ever submits -- |
| 734 | // the user speaks, sees their words, and has to tap the mic again to make |
| 735 | // anything happen. Reported from an older iPad, and the fallback below is |
| 736 | // the fix: fall back to the recognizer's own isFinal, which every browser |
| 737 | // that supports recognition at all still gives us. |
| 738 | let meterRunning = false; |
| 739 | // Set the first time recognition returns a result. Until then the meter |
| 740 | // must not open a competing microphone stream. |
| 741 | let recognitionProven = false; |
| 742 | // The deferred meter start above is NOT sufficient on its own, and an |
| 743 | // iPad proved it: recognition delivered ONE result ("computer"), that |
| 744 | // proof started the meter, and the meter's stream then took the |
| 745 | // microphone back -- first word on screen, then nothing, which is the |
| 746 | // original bug moved one result later. So the meter must be able to |
| 747 | // LOSE, and it convicts itself with its own evidence: if it can hear |
| 748 | // voice (RMS above the speech threshold) while recognition has produced |
| 749 | // nothing for CONTENTION_MS, the two are not sharing the microphone -- |
| 750 | // on a working platform, results flow continuously while you speak. |
| 751 | // Convicted once, it stays off for the session and isFinal endpointing |
| 752 | // takes over. |
| 753 | let meterContended = false; |
| 754 | let lastResultTs = 0; |
| 755 | const CONTENTION_MS = 2000; |
| 756 | let micStream = null; |
| 757 | let audioCtx = null; |
| 758 | let analyser = null; |
| 759 | let meterRafId = null; |
| 760 | let noiseFloor = 0; |
| 761 | let speechThreshold = MIN_SPEECH_THRESHOLD; |
| 762 | let calibrating = false; |
| 763 | let calibrationStart = 0; |
| 764 | let calibrationSamples = []; |
| 765 | let utteranceStarted = false; |
| 766 | let silenceStartTs = null; |
| 767 | let speechHoldStart = null; |
| 768 | |
| 769 | // Absent capability, stated -- never a stub that pretends to work. The |
| 770 | // hint is the only way a user can tell "it is waiting for me to stop |
| 771 | // talking" from "it is waiting for me to tap". |
| 772 | function noteEndpointFallback() { |
| 773 | meterRunning = false; |
| 774 | const hint = document.getElementById("endpointhint"); |
| 775 | if (hint && endpointMode() === "silence") { |
| 776 | hint.textContent = "No level meter on this device, so it sends when it " |
| 777 | + "hears you finish rather than after a silence."; |
| 778 | hint.hidden = false; |
| 779 | } |
| 780 | } |
| 781 | |
| 782 | |
| 783 | function updateMeterVisual(rms) { |
| 784 | const ring = document.getElementById("miclevel"); |
| 785 | if (!ring) return; |
| 786 | // Force the ring to idle during TTS playback -- the stream is still |
| 787 | // open (so the mic doesn't visibly "go dead" mid-conversation), but a |
| 788 | // level driven by the machine's own voice bleeding into the mic would |
| 789 | // be a misleading, distracting reading. |
| 790 | const level = speaking ? 0 : Math.min(1, rms * 6); |
| 791 | ring.style.opacity = String(Math.min(0.65, level * 1.3)); |
| 792 | ring.style.boxShadow = "0 0 0 " + (level * 14).toFixed(1) + "px var(--danger)"; |
| 793 | } |
| 794 | |
| 795 | function evaluateVAD(rms, now) { |
| 796 | if (speaking || !conversationMode) { silenceStartTs = null; speechHoldStart = null; return; } |
| 797 | if (endpointMode() !== "silence") return; |
| 798 | const above = rms >= speechThreshold; |
| 799 | if (above) { |
| 800 | silenceStartTs = null; |
| 801 | if (speechHoldStart === null) speechHoldStart = now; |
| 802 | if (!utteranceStarted && now - speechHoldStart >= SPEECH_HOLD_MS) utteranceStarted = true; |
| 803 | return; |
| 804 | } |
| 805 | speechHoldStart = null; |
| 806 | if (!utteranceStarted || !bufferHasContent()) return; |
| 807 | if (silenceStartTs === null) { silenceStartTs = now; return; } |
| 808 | if (now - silenceStartTs < silenceMs()) return; |
| 809 | silenceStartTs = null; |
| 810 | utteranceStarted = false; |
| 811 | if (armed) { |
| 812 | // Never woken -- silence ended the attempt, discard rather than submit. |
| 813 | resetTalkState(); |
| 814 | updateCaptionArmed(); |
| 815 | } else { |
| 816 | finalizeVoiceSubmit(pendingText()); |
| 817 | } |
| 818 | } |
| 819 | |
| 820 | async function startMeter() { |
| 821 | meterRunning = false; |
| 822 | if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) { |
| 823 | noteEndpointFallback(); return; |
| 824 | } |
| 825 | let stream; |
| 826 | try { |
| 827 | stream = await navigator.mediaDevices.getUserMedia({audio: true}); |
| 828 | } catch (e) { noteEndpointFallback(); return; } |
| 829 | if (!conversationMode) { stream.getTracks().forEach((t) => t.stop()); return; } |
| 830 | const Ctor = window.AudioContext || window.webkitAudioContext; |
| 831 | if (!Ctor) { |
| 832 | stream.getTracks().forEach((t) => t.stop()); noteEndpointFallback(); return; |
| 833 | } |
| 834 | meterRunning = true; |
| 835 | lastResultTs = performance.now(); |
| 836 | micStream = stream; |
| 837 | audioCtx = new Ctor(); |
| 838 | const source = audioCtx.createMediaStreamSource(micStream); |
| 839 | analyser = audioCtx.createAnalyser(); |
| 840 | analyser.fftSize = 512; |
| 841 | source.connect(analyser); |
| 842 | const data = new Uint8Array(analyser.fftSize); |
| 843 | |
| 844 | calibrating = true; |
| 845 | calibrationStart = performance.now(); |
| 846 | calibrationSamples = []; |
| 847 | utteranceStarted = false; |
| 848 | silenceStartTs = null; |
| 849 | speechHoldStart = null; |
| 850 | const ring = document.getElementById("miclevel"); |
| 851 | if (ring) ring.hidden = false; |
| 852 | |
| 853 | function frame() { |
| 854 | if (!analyser) return; // meter was stopped |
| 855 | analyser.getByteTimeDomainData(data); |
| 856 | let sumSquares = 0; |
| 857 | for (let i = 0; i < data.length; i++) { |
| 858 | const v = (data[i] - 128) / 128; |
| 859 | sumSquares += v * v; |
| 860 | } |
| 861 | const rms = Math.sqrt(sumSquares / data.length); |
| 862 | const now = performance.now(); |
| 863 | |
| 864 | if (calibrating) { |
| 865 | calibrationSamples.push(rms); |
| 866 | if (now - calibrationStart >= CALIBRATION_MS) { |
| 867 | const avg = calibrationSamples.reduce((a, b) => a + b, 0) / calibrationSamples.length; |
| 868 | noiseFloor = avg; |
| 869 | speechThreshold = Math.max(avg * 3, MIN_SPEECH_THRESHOLD); |
| 870 | calibrating = false; |
| 871 | } |
| 872 | } else { |
| 873 | // The contention check runs BEFORE the VAD: the meter hearing |
| 874 | // voice while recognition says nothing is proof the microphone is |
| 875 | // exclusive and the meter took it. Stopping the meter frees the |
| 876 | // mic; stopListening() kicks the starved recognizer, whose onend |
| 877 | // handler restarts it fresh -- and this time nothing competes. |
| 878 | if (rms >= speechThreshold && now - lastResultTs > CONTENTION_MS) { |
| 879 | meterContended = true; |
| 880 | stopMeter(); |
| 881 | noteEndpointFallback(); |
| 882 | stopListening(); |
| 883 | return; |
| 884 | } |
| 885 | evaluateVAD(rms, now); |
| 886 | } |
| 887 | |
| 888 | updateMeterVisual(rms); |
| 889 | meterRafId = requestAnimationFrame(frame); |
| 890 | } |
| 891 | meterRafId = requestAnimationFrame(frame); |
| 892 | } |
| 893 | |
| 894 | function stopMeter() { |
| 895 | if (meterRafId) cancelAnimationFrame(meterRafId); |
| 896 | meterRafId = null; |
| 897 | analyser = null; |
| 898 | if (audioCtx) { try { audioCtx.close(); } catch (e) { /* already closed */ } audioCtx = null; } |
| 899 | if (micStream) { micStream.getTracks().forEach((t) => t.stop()); micStream = null; } |
| 900 | calibrating = false; |
| 901 | utteranceStarted = false; |
| 902 | silenceStartTs = null; |
| 903 | speechHoldStart = null; |
| 904 | const ring = document.getElementById("miclevel"); |
| 905 | if (ring) { ring.style.opacity = "0"; ring.style.boxShadow = "none"; ring.hidden = true; } |
| 906 | } |
| 907 | |
| 908 | function stopListening() { |
| 909 | listening = false; |
| 910 | if (recognizer) { try { recognizer.stop(); } catch (e) { /* already stopped */ } } |
| 911 | } |
| 912 | |
| 913 | // The mic and the speaker share ONE audio session on iOS, and |
| 914 | // recognizer.stop() is a REQUEST, not an instant release -- teardown |
| 915 | // completes asynchronously and onend fires later. Starting playback |
| 916 | // before that teardown finishes was Dan's own diagnosis: "listening |
| 917 | // while talking, maybe not allowed" -- iOS can leave the session |
| 918 | // configured for recording through the transition, and audio played |
| 919 | // into a still-recording session can come out silent or ducked to |
| 920 | // nothing, with no error anywhere to say so. speak() now WAITS for the |
| 921 | // release rather than racing it. |
| 922 | let _listeningStoppedWaiters = []; |
| 923 | function _drainListeningStopped() { |
| 924 | const waiters = _listeningStoppedWaiters; |
| 925 | _listeningStoppedWaiters = []; |
| 926 | waiters.forEach((resolve) => resolve()); |
| 927 | } |
| 928 | function stopListeningAndWaitForRelease() { |
| 929 | stopListening(); |
| 930 | if (!recognizer) return Promise.resolve(); |
| 931 | return new Promise((resolve) => { |
| 932 | // A safety net, not the expected path: some browsers/recognizer |
| 933 | // states never fire onend for a stop() that was already a no-op |
| 934 | // (already stopped, never started). Real iOS teardown is much |
| 935 | // faster than this; it only pays the timeout when there was |
| 936 | // nothing to wait for in the first place. |
| 937 | const timer = setTimeout(resolve, 400); |
| 938 | _listeningStoppedWaiters.push(() => { clearTimeout(timer); resolve(); }); |
| 939 | }); |
| 940 | } |
| 941 | |
| 942 | function startListening() { |
| 943 | if (!recognizer || listening || speaking) return; |
| 944 | listening = true; |
| 945 | try { recognizer.start(); } catch (e) { listening = false; } |
| 946 | } |
| 947 | |
| 948 | function resumeListeningIfNeeded() { |
| 949 | if (conversationMode && !speaking && !listening) startListening(); |
| 950 | } |
| 951 | |
| 952 | function initMic() { |
| 953 | const mic = document.getElementById("mic"); |
| 954 | const ring = document.getElementById("miclevel"); |
| 955 | if (!mic) return; |
| 956 | |
| 957 | // talk_stt_engine "openai" is an explicit opt-in (see talkSttEngine's |
| 958 | // comment) -- it never silently activates for a browser that merely |
| 959 | // lacks SpeechRecognition, so someone who never touched this setting |
| 960 | // keeps seeing exactly today's behavior either way. |
| 961 | if (talkSttEngine() === "openai") { |
| 962 | if (!navigator.mediaDevices || !window.MediaRecorder) { |
| 963 | mic.hidden = true; |
| 964 | if (ring) ring.hidden = true; |
| 965 | return; |
| 966 | } |
| 967 | mic.hidden = false; |
| 968 | // No live level meter here: it is driven by SpeechRecognition's own |
| 969 | // partial-result timing (see startMeter's callers), which push-to-talk |
| 970 | // cloud capture has no equivalent signal for. |
| 971 | if (ring) ring.hidden = true; |
| 972 | initCloudMic(mic); |
| 973 | return; |
| 974 | } |
| 975 | |
| 976 | if (!SpeechRecognitionCtor) { |
| 977 | mic.hidden = true; |
| 978 | if (ring) ring.hidden = true; |
| 979 | return; |
| 980 | } |
| 981 | mic.hidden = false; |
| 982 | recognizer = new SpeechRecognitionCtor(); |
| 983 | recognizer.continuous = false; |
| 984 | recognizer.interimResults = true; |
| 985 | recognizer.lang = "en-US"; |
| 986 | |
| 987 | // Interim and final results both feed the same processTranscript(), which |
| 988 | // is the single funnel for the wake gate, live caption, and end-word/ |
| 989 | // isFinal submission -- there is no separate "submit" path for voice. |
| 990 | recognizer.onresult = (event) => { |
| 991 | lastResultTs = performance.now(); |
| 992 | // Feeds the contention verdict and the no-recognition hint; the |
| 993 | // meter no longer waits on this (it must calibrate BEFORE speech). |
| 994 | recognitionProven = true; |
| 995 | let text = ""; |
| 996 | // Segments are joined with an explicit space. Chrome includes leading |
| 997 | // spaces on continuation segments; iOS does NOT, and concatenating |
| 998 | // raw produced "comptureshowmemynotes" from an iPad -- one giant |
| 999 | // token the whitespace tokenizer cannot split, so the wake gate could |
| 1000 | // never match and every utterance was invisibly discarded. Trimming |
| 1001 | // then joining is correct on both: the tokenizer collapses runs of |
| 1002 | // whitespace anyway, so a doubled space costs nothing. |
| 1003 | for (let i = 0; i < event.results.length; i++) { |
| 1004 | const seg = event.results[i][0].transcript.trim(); |
| 1005 | if (seg) text += (text ? " " : "") + seg; |
| 1006 | } |
| 1007 | sessionLive = text; |
| 1008 | const isFinal = event.results[event.results.length - 1].isFinal; |
| 1009 | processTranscript(isFinal); |
| 1010 | if (isFinal) stopListening(); |
| 1011 | }; |
| 1012 | recognizer.onerror = () => { listening = false; }; |
| 1013 | recognizer.onend = () => { |
| 1014 | _drainListeningStopped(); |
| 1015 | // CRITICAL: stop recognition before playing TTS (done in speak()) and |
| 1016 | // only resume after playback ends -- if mode is still on AND we are |
| 1017 | // not currently speaking, restart here too, so a recognizer that ends |
| 1018 | // for any other reason (silence timeout, browser quirk) still keeps |
| 1019 | // conversation mode going instead of going quiet. |
| 1020 | listening = false; |
| 1021 | if (conversationMode && !speaking) startListening(); |
| 1022 | }; |
| 1023 | |
| 1024 | // If recognition has produced nothing at all a few seconds after the |
| 1025 | // mic went on, the device has not given us speech recognition -- say so |
| 1026 | // instead of leaving a listening button that will never do anything. |
| 1027 | let provenTimer = null; |
| 1028 | function watchForRecognition() { |
| 1029 | if (provenTimer) clearTimeout(provenTimer); |
| 1030 | provenTimer = setTimeout(() => { |
| 1031 | provenTimer = null; |
| 1032 | if (!conversationMode || recognitionProven) return; |
| 1033 | const hint = document.getElementById("endpointhint"); |
| 1034 | if (hint) { |
| 1035 | hint.textContent = "This browser is not returning speech recognition." |
| 1036 | + " Type below instead \u2014 the mic cannot hear you here."; |
| 1037 | hint.hidden = false; |
| 1038 | } |
| 1039 | }, 6000); |
| 1040 | } |
| 1041 | |
| 1042 | mic.addEventListener("click", () => { |
| 1043 | // iOS will not speak or play audio that was not started by a user |
| 1044 | // gesture, and a reply arrives asynchronously long after this click. |
| 1045 | // Priming both engines HERE -- inside the gesture -- is what makes the |
| 1046 | // later programmatic speak() audible. Silent by construction: an empty |
| 1047 | // utterance and a zero-length resume. |
| 1048 | primeAudioForIOS(); |
| 1049 | conversationMode = !conversationMode; |
| 1050 | mic.classList.toggle("on", conversationMode); |
| 1051 | mic.textContent = conversationMode ? "listening…" : "mic"; |
| 1052 | if (conversationMode) { |
| 1053 | resetTalkState(); |
| 1054 | updateCaptionArmed(); |
| 1055 | // Recognition and the meter BOTH start here, and the ordering war |
| 1056 | // between them is settled by evidence, not scheduling. History of |
| 1057 | // this line: starting both stole the mic from recognition on iOS |
| 1058 | // (exclusive microphone -- ring animated, no transcript), so the |
| 1059 | // meter was deferred until recognition's first result. That fix |
| 1060 | // broke every DESKTOP: a meter started mid-speech CALIBRATES ON |
| 1061 | // SPEECH, sets its threshold at 3x the user's own voice, and the |
| 1062 | // VAD never fires again -- transcript on screen, listening |
| 1063 | // forever, nothing sent. Calibration must happen in the pre-speech |
| 1064 | // silence, which means the meter must start at the click. |
| 1065 | // |
| 1066 | // The iOS case is owned by the CONVICTION instead: a meter that |
| 1067 | // can hear voice while recognition produces nothing for two |
| 1068 | // seconds took the microphone, and is stopped for the session -- |
| 1069 | // recognition restarts and the stability endpointer takes over. |
| 1070 | // Each platform loses the component it cannot support, on proof. |
| 1071 | startListening(); |
| 1072 | if (endpointMode() === "silence" && !meterContended) startMeter(); |
| 1073 | watchForRecognition(); |
| 1074 | } else { |
| 1075 | recognitionProven = false; |
| 1076 | stopListening(); |
| 1077 | stopMeter(); |
| 1078 | resetTalkState(); |
| 1079 | capUser.textContent = ""; |
| 1080 | capUser.classList.remove("armed", "active", "sent"); |
| 1081 | } |
| 1082 | }); |
| 1083 | } |
| 1084 | |
| 1085 | // Cloud STT is push-to-talk by default: one tap starts a real |
| 1086 | // MediaRecorder capture, a second tap stops it and uploads the clip to |
| 1087 | // /api/stt for transcription. Raw audio has no equivalent to |
| 1088 | // SpeechRecognition's interimResults, so there is no live partial |
| 1089 | // transcript to drive a wake word or continuous conversation the way the |
| 1090 | // browser path has for free. This is deliberately a smaller, honest |
| 1091 | // feature on speakers: it promises only "record what you say between two |
| 1092 | // taps, transcribe it, send it" -- and unlike SpeechRecognition, it works |
| 1093 | // on every browser with a microphone, including ones the Web Speech API |
| 1094 | // never supported (Firefox). |
| 1095 | // |
| 1096 | // talk_headphones_mode (see startHeadphonesMode below) is the exception: |
| 1097 | // with headphones on, the mic physically cannot hear the assistant's own |
| 1098 | // voice, so the "one tap per utterance" safety rail push-to-talk needs on |
| 1099 | // speakers becomes pure friction instead of a real problem to guard |
| 1100 | // against. That mode builds real audio-level VAD (the same |
| 1101 | // amplitude-analysis technique the browser path's silence endpoint |
| 1102 | // already uses, pointed at a MediaRecorder loop instead of |
| 1103 | // SpeechRecognition's text buffer) for genuinely hands-free, |
| 1104 | // no-tap-needed listening -- including staying live WHILE the assistant |
| 1105 | // is speaking, so starting to talk mid-reply interrupts it. |
| 1106 | let cloudRecorder = null; |
| 1107 | let cloudChunks = []; |
| 1108 | let cloudStream = null; |
| 1109 | let cloudRecording = false; |
| 1110 | |
| 1111 | function talkHeadphonesMode() { |
| 1112 | return String(pref("talk_headphones_mode", "false")).trim() === "true"; |
| 1113 | } |
| 1114 | |
| 1115 | function initCloudMic(mic) { |
| 1116 | mic.addEventListener("click", async () => { |
| 1117 | primeAudioForIOS(); |
| 1118 | if (talkHeadphonesMode()) { |
| 1119 | if (headphonesArmed) stopHeadphonesMode(mic); else await startHeadphonesMode(mic); |
| 1120 | return; |
| 1121 | } |
| 1122 | if (cloudRecording) { stopCloudRecording(); return; } |
| 1123 | await startCloudRecording(mic); |
| 1124 | }); |
| 1125 | } |
| 1126 | |
| 1127 | function showEndpointHint(message) { |
| 1128 | const hint = document.getElementById("endpointhint"); |
| 1129 | if (hint) { hint.textContent = message; hint.hidden = false; } |
| 1130 | } |
| 1131 | |
| 1132 | async function startCloudRecording(mic) { |
| 1133 | let stream; |
| 1134 | try { |
| 1135 | stream = await navigator.mediaDevices.getUserMedia({audio: true}); |
| 1136 | } catch (e) { |
| 1137 | showEndpointHint("Microphone access was denied or unavailable."); |
| 1138 | return; |
| 1139 | } |
| 1140 | const candidates = ["audio/webm", "audio/mp4", "audio/ogg"]; |
| 1141 | const mimeType = window.MediaRecorder && MediaRecorder.isTypeSupported |
| 1142 | ? candidates.find((t) => MediaRecorder.isTypeSupported(t)) |
| 1143 | : undefined; |
| 1144 | try { |
| 1145 | cloudRecorder = mimeType ? new MediaRecorder(stream, {mimeType}) : new MediaRecorder(stream); |
| 1146 | } catch (e) { |
| 1147 | stream.getTracks().forEach((t) => t.stop()); |
| 1148 | showEndpointHint("This browser cannot record audio for cloud speech recognition."); |
| 1149 | return; |
| 1150 | } |
| 1151 | cloudStream = stream; |
| 1152 | cloudChunks = []; |
| 1153 | cloudRecorder.ondataavailable = (e) => { if (e.data && e.data.size) cloudChunks.push(e.data); }; |
| 1154 | cloudRecorder.onstop = () => onCloudRecordingStopped(mic); |
| 1155 | cloudRecorder.start(); |
| 1156 | cloudRecording = true; |
| 1157 | mic.classList.add("on"); |
| 1158 | mic.textContent = "recording… tap to stop"; |
| 1159 | capUser.textContent = ""; |
| 1160 | capUser.classList.add("armed"); |
| 1161 | } |
| 1162 | |
| 1163 | function stopCloudRecording() { |
| 1164 | if (cloudRecorder && cloudRecorder.state !== "inactive") cloudRecorder.stop(); |
| 1165 | cloudRecording = false; |
| 1166 | } |
| 1167 | |
| 1168 | async function onCloudRecordingStopped(mic) { |
| 1169 | if (cloudStream) { cloudStream.getTracks().forEach((t) => t.stop()); cloudStream = null; } |
| 1170 | mic.classList.remove("on"); |
| 1171 | mic.textContent = "transcribing…"; |
| 1172 | const blobType = (cloudRecorder && cloudRecorder.mimeType) || "audio/webm"; |
| 1173 | const blob = new Blob(cloudChunks, {type: blobType}); |
| 1174 | cloudChunks = []; |
| 1175 | if (!blob.size) { |
| 1176 | mic.textContent = "mic"; |
| 1177 | return; |
| 1178 | } |
| 1179 | try { |
| 1180 | const res = await fetch("/api/stt", { |
| 1181 | method: "POST", credentials: "same-origin", |
| 1182 | headers: {"content-type": blobType, accept: "application/json"}, |
| 1183 | body: blob, |
| 1184 | }); |
| 1185 | const data = await res.json().catch(() => ({})); |
| 1186 | mic.textContent = "mic"; |
| 1187 | if (!res.ok || data.status !== "ok" || typeof data.text !== "string") { |
| 1188 | showEndpointHint(data.error || "Transcription failed."); |
| 1189 | return; |
| 1190 | } |
| 1191 | const text = data.text.trim(); |
| 1192 | if (text) submitTurn(text); |
| 1193 | } catch (e) { |
| 1194 | mic.textContent = "mic"; |
| 1195 | showEndpointHint("Transcription request failed."); |
| 1196 | } |
| 1197 | } |
| 1198 | |
| 1199 | // ---- Headphones mode: continuous, hands-free cloud STT -------------------- |
| 1200 | // |
| 1201 | // One tap arms it; the mic stays open and listening until a second tap |
| 1202 | // disarms it -- no per-utterance tap. Silence-segmented the same way the |
| 1203 | // browser path's VAD is (calibrate ambient noise briefly, then watch for |
| 1204 | // a speech-then-silence cycle), just pointed at a MediaRecorder loop |
| 1205 | // (stop the current segment, upload+transcribe it, immediately start the |
| 1206 | // next one) instead of SpeechRecognition's own text buffer. Deliberately |
| 1207 | // a SEPARATE, independent state machine from the browser path's meter |
| 1208 | // code above (evaluateVAD/startMeter/etc.) rather than a shared one: that |
| 1209 | // code is tightly coupled to SpeechRecognition-specific state |
| 1210 | // (conversationMode, armed, lastResultTs, wake-word buffering) that |
| 1211 | // doesn't apply here, and forcing a shared implementation would have |
| 1212 | // meant threading cloud-vs-browser branches through code that's already |
| 1213 | // dense with iPad-specific hard-won fixes -- not worth the risk of |
| 1214 | // breaking that path to save writing a second, smaller VAD loop. |
| 1215 | // |
| 1216 | // The one genuinely new thing this mode does versus push-to-talk: it |
| 1217 | // keeps recording WHILE the assistant is speaking, and interrupts |
| 1218 | // playback (stopSpeaking()) the moment it hears real speech during that |
| 1219 | // window -- safe specifically because headphones mean the mic can't be |
| 1220 | // hearing the assistant's own voice, so any detected speech is real. |
| 1221 | let headphonesArmed = false; |
| 1222 | let headphonesStream = null; |
| 1223 | let headphonesRecorder = null; |
| 1224 | let headphonesChunks = []; |
| 1225 | let headphonesAudioCtx = null; |
| 1226 | let headphonesAnalyser = null; |
| 1227 | let headphonesCalibrating = false; |
| 1228 | let headphonesCalibrationStart = 0; |
| 1229 | let headphonesCalibrationSamples = []; |
| 1230 | let headphonesSpeechThreshold = MIN_SPEECH_THRESHOLD; |
| 1231 | let headphonesUtteranceStarted = false; |
| 1232 | let headphonesSilenceStartTs = null; |
| 1233 | let headphonesSpeechHoldStart = null; |
| 1234 | let headphonesRafId = null; |
| 1235 | // Guards the ENTIRE finish-a-segment sequence (stop -> read chunks -> |
| 1236 | // clear -> start next -> upload -> transcribe -> submit), not just the |
| 1237 | // network part: headphonesChunks is one shared array reused across |
| 1238 | // segments, and two overlapping finishes racing to read/clear it would |
| 1239 | // corrupt each other's audio. The cost is real but bounded: back-to-back |
| 1240 | // utterances faster than one full transcribe round-trip apart get |
| 1241 | // recorded into the same segment and submitted together, rather than |
| 1242 | // split -- simple and correct beats a fully pipelined multi-segment |
| 1243 | // system for what this is. |
| 1244 | let headphonesSegmentPending = false; |
| 1245 | |
| 1246 | function headphonesRms(analyser, data) { |
| 1247 | analyser.getByteTimeDomainData(data); |
| 1248 | let sumSquares = 0; |
| 1249 | for (let i = 0; i < data.length; i++) { |
| 1250 | const v = (data[i] - 128) / 128; |
| 1251 | sumSquares += v * v; |
| 1252 | } |
| 1253 | return Math.sqrt(sumSquares / data.length); |
| 1254 | } |
| 1255 | |
| 1256 | async function startHeadphonesMode(mic) { |
| 1257 | let stream; |
| 1258 | try { |
| 1259 | stream = await navigator.mediaDevices.getUserMedia({audio: true}); |
| 1260 | } catch (e) { |
| 1261 | showEndpointHint("Microphone access was denied or unavailable."); |
| 1262 | return; |
| 1263 | } |
| 1264 | const Ctor = window.AudioContext || window.webkitAudioContext; |
| 1265 | if (!Ctor || !window.MediaRecorder) { |
| 1266 | stream.getTracks().forEach((t) => t.stop()); |
| 1267 | showEndpointHint("This browser cannot run headphones mode."); |
| 1268 | return; |
| 1269 | } |
| 1270 | |
| 1271 | headphonesArmed = true; |
| 1272 | headphonesStream = stream; |
| 1273 | headphonesAudioCtx = new Ctor(); |
| 1274 | const source = headphonesAudioCtx.createMediaStreamSource(stream); |
| 1275 | headphonesAnalyser = headphonesAudioCtx.createAnalyser(); |
| 1276 | headphonesAnalyser.fftSize = 512; |
| 1277 | source.connect(headphonesAnalyser); |
| 1278 | const data = new Uint8Array(headphonesAnalyser.fftSize); |
| 1279 | |
| 1280 | headphonesCalibrating = true; |
| 1281 | headphonesCalibrationStart = performance.now(); |
| 1282 | headphonesCalibrationSamples = []; |
| 1283 | headphonesUtteranceStarted = false; |
| 1284 | headphonesSilenceStartTs = null; |
| 1285 | headphonesSpeechHoldStart = null; |
| 1286 | |
| 1287 | mic.classList.add("on"); |
| 1288 | mic.textContent = "listening (headphones)… tap to stop"; |
| 1289 | capUser.textContent = ""; |
| 1290 | capUser.classList.add("armed"); |
| 1291 | |
| 1292 | startHeadphonesSegment(); |
| 1293 | |
| 1294 | function frame() { |
| 1295 | if (!headphonesAnalyser) return; // disarmed mid-flight |
| 1296 | const rms = headphonesRms(headphonesAnalyser, data); |
| 1297 | const now = performance.now(); |
| 1298 | |
| 1299 | if (headphonesCalibrating) { |
| 1300 | headphonesCalibrationSamples.push(rms); |
| 1301 | if (now - headphonesCalibrationStart >= CALIBRATION_MS) { |
| 1302 | const avg = headphonesCalibrationSamples.reduce((a, b) => a + b, 0) / headphonesCalibrationSamples.length; |
| 1303 | headphonesSpeechThreshold = Math.max(avg * 3, MIN_SPEECH_THRESHOLD); |
| 1304 | headphonesCalibrating = false; |
| 1305 | } |
| 1306 | } else { |
| 1307 | const above = rms >= headphonesSpeechThreshold; |
| 1308 | if (above) { |
| 1309 | // The headline behavior: real speech heard while the assistant is |
| 1310 | // still talking is the interrupt itself, not a signal to ignore. |
| 1311 | if (speaking) stopSpeaking(); |
| 1312 | headphonesSilenceStartTs = null; |
| 1313 | if (headphonesSpeechHoldStart === null) headphonesSpeechHoldStart = now; |
| 1314 | if (!headphonesUtteranceStarted && now - headphonesSpeechHoldStart >= SPEECH_HOLD_MS) { |
| 1315 | headphonesUtteranceStarted = true; |
| 1316 | capUser.classList.remove("armed"); |
| 1317 | capUser.classList.add("active"); |
| 1318 | capUser.textContent = "listening…"; |
| 1319 | } |
| 1320 | } else { |
| 1321 | headphonesSpeechHoldStart = null; |
| 1322 | if (headphonesUtteranceStarted && !headphonesSegmentPending) { |
| 1323 | if (headphonesSilenceStartTs === null) { |
| 1324 | headphonesSilenceStartTs = now; |
| 1325 | } else if (now - headphonesSilenceStartTs >= silenceMs()) { |
| 1326 | headphonesSilenceStartTs = null; |
| 1327 | headphonesUtteranceStarted = false; |
| 1328 | finishHeadphonesSegment(); |
| 1329 | } |
| 1330 | } |
| 1331 | } |
| 1332 | } |
| 1333 | headphonesRafId = requestAnimationFrame(frame); |
| 1334 | } |
| 1335 | headphonesRafId = requestAnimationFrame(frame); |
| 1336 | } |
| 1337 | |
| 1338 | function startHeadphonesSegment() { |
| 1339 | if (!headphonesStream) return; |
| 1340 | const candidates = ["audio/webm", "audio/mp4", "audio/ogg"]; |
| 1341 | const mimeType = window.MediaRecorder && MediaRecorder.isTypeSupported |
| 1342 | ? candidates.find((t) => MediaRecorder.isTypeSupported(t)) |
| 1343 | : undefined; |
| 1344 | try { |
| 1345 | headphonesRecorder = mimeType |
| 1346 | ? new MediaRecorder(headphonesStream, {mimeType}) |
| 1347 | : new MediaRecorder(headphonesStream); |
| 1348 | } catch (e) { |
| 1349 | stopHeadphonesMode(document.getElementById("mic")); |
| 1350 | showEndpointHint("This browser cannot record audio for headphones mode."); |
| 1351 | return; |
| 1352 | } |
| 1353 | headphonesChunks = []; |
| 1354 | headphonesRecorder.ondataavailable = (e) => { if (e.data && e.data.size) headphonesChunks.push(e.data); }; |
| 1355 | headphonesRecorder.start(); |
| 1356 | } |
| 1357 | |
| 1358 | async function finishHeadphonesSegment() { |
| 1359 | if (!headphonesRecorder || headphonesRecorder.state === "inactive") return; |
| 1360 | headphonesSegmentPending = true; |
| 1361 | try { |
| 1362 | const recorder = headphonesRecorder; |
| 1363 | const blobType = recorder.mimeType || "audio/webm"; |
| 1364 | const stopped = new Promise((resolve) => { recorder.onstop = resolve; }); |
| 1365 | recorder.stop(); |
| 1366 | await stopped; |
| 1367 | const blob = new Blob(headphonesChunks, {type: blobType}); |
| 1368 | headphonesChunks = []; |
| 1369 | |
| 1370 | // Arm the NEXT segment immediately -- "always listening" must not |
| 1371 | // have a gap while this segment's transcription is still in flight. |
| 1372 | if (headphonesArmed) startHeadphonesSegment(); |
| 1373 | |
| 1374 | if (!blob.size) return; |
| 1375 | const res = await fetch("/api/stt", { |
| 1376 | method: "POST", credentials: "same-origin", |
| 1377 | headers: {"content-type": blobType, accept: "application/json"}, |
| 1378 | body: blob, |
| 1379 | }); |
| 1380 | const data = await res.json().catch(() => ({})); |
| 1381 | if (!res.ok || data.status !== "ok" || typeof data.text !== "string") { |
| 1382 | showEndpointHint(data.error || "Transcription failed."); |
| 1383 | return; |
| 1384 | } |
| 1385 | const text = data.text.trim(); |
| 1386 | if (text) submitTurn(text); |
| 1387 | } catch (e) { |
| 1388 | showEndpointHint("Transcription request failed."); |
| 1389 | } finally { |
| 1390 | headphonesSegmentPending = false; |
| 1391 | } |
| 1392 | } |
| 1393 | |
| 1394 | function stopHeadphonesMode(mic) { |
| 1395 | headphonesArmed = false; |
| 1396 | if (headphonesRafId) cancelAnimationFrame(headphonesRafId); |
| 1397 | headphonesRafId = null; |
| 1398 | headphonesAnalyser = null; |
| 1399 | if (headphonesAudioCtx) { headphonesAudioCtx.close(); headphonesAudioCtx = null; } |
| 1400 | if (headphonesRecorder && headphonesRecorder.state !== "inactive") headphonesRecorder.stop(); |
| 1401 | headphonesRecorder = null; |
| 1402 | if (headphonesStream) { headphonesStream.getTracks().forEach((t) => t.stop()); headphonesStream = null; } |
| 1403 | headphonesUtteranceStarted = false; |
| 1404 | headphonesSilenceStartTs = null; |
| 1405 | headphonesSpeechHoldStart = null; |
| 1406 | if (mic) { mic.classList.remove("on"); mic.textContent = "mic"; } |
| 1407 | capUser.textContent = ""; |
| 1408 | capUser.classList.remove("armed", "active", "sent"); |
| 1409 | } |
| 1410 | |
| 1411 | // One-time unlock, inside a user gesture. Safari on iOS refuses both |
| 1412 | // speechSynthesis and HTMLAudioElement.play() outside one, and gives no |
| 1413 | // error worth reading when it refuses -- the reply simply never becomes |
| 1414 | // sound, which is exactly what an older iPad reported. |
| 1415 | let ttsAudioEl = null; |
| 1416 | function sharedTtsAudio() { |
| 1417 | if (!ttsAudioEl) ttsAudioEl = new Audio(); |
| 1418 | return ttsAudioEl; |
| 1419 | } |
| 1420 | |
| 1421 | let audioPrimed = false; |
| 1422 | function primeAudioForIOS() { |
| 1423 | if (audioPrimed) return; |
| 1424 | audioPrimed = true; |
| 1425 | try { |
| 1426 | if (window.speechSynthesis) { |
| 1427 | const warm = new SpeechSynthesisUtterance(""); |
| 1428 | warm.volume = 0; |
| 1429 | window.speechSynthesis.speak(warm); |
| 1430 | // getVoices() is empty until the engine loads on iOS; asking inside |
| 1431 | // the gesture is what populates it in time for hasLocalVoice(). |
| 1432 | window.speechSynthesis.getVoices(); |
| 1433 | } |
| 1434 | } catch (e) { /* nothing to unlock */ } |
| 1435 | try { |
| 1436 | // Unlock the ONE audio element every server-TTS reply will reuse: a |
| 1437 | // zero-length silent play inside the gesture is what licenses every |
| 1438 | // later programmatic play() on the same element. |
| 1439 | const player = sharedTtsAudio(); |
| 1440 | player.muted = true; |
| 1441 | player.src = "data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAIlYAAESsAAACABAAZGF0YQAAAAA="; |
| 1442 | const attempt = player.play(); |
| 1443 | if (attempt && attempt.catch) attempt.catch(() => {}); |
| 1444 | setTimeout(() => { player.muted = false; }, 150); |
| 1445 | } catch (e) { /* nothing to unlock */ } |
| 1446 | try { |
| 1447 | const Ctor = window.AudioContext || window.webkitAudioContext; |
| 1448 | if (Ctor) { |
| 1449 | const ctx = new Ctor(); |
| 1450 | if (ctx.state === "suspended" && ctx.resume) ctx.resume(); |
| 1451 | // Kept open deliberately: closing it re-locks playback on iOS. |
| 1452 | window.__dbbasicTalkUnlockCtx = ctx; |
| 1453 | } |
| 1454 | } catch (e) { /* nothing to unlock */ } |
| 1455 | } |
| 1456 | |
| 1457 | |
| 1458 | async function api(method, path, payload) { |
| 1459 | const res = await fetch(path, { |
| 1460 | method, credentials: "same-origin", |
| 1461 | headers: {"content-type": "application/json", accept: "application/json"}, |
| 1462 | body: payload === undefined ? undefined : JSON.stringify(payload), |
| 1463 | }); |
| 1464 | return [res.ok, await res.json()]; |
| 1465 | } |
| 1466 | |
| 1467 | async function loadPrefs() { |
| 1468 | const res = await fetch(`/collections/shell_preferences/records/${OWNER_ID}`, |
| 1469 | {credentials: "same-origin", headers: {accept: "application/json"}}); |
| 1470 | if (res.ok) { const body = await res.json(); prefs = body.record || prefs; } |
| 1471 | } |
| 1472 | |
| 1473 | async function savePrefs(changes) { |
| 1474 | Object.assign(prefs, changes); |
| 1475 | const [ok] = await api("PUT", `/collections/shell_preferences/records/${OWNER_ID}`, changes); |
| 1476 | if (!ok) await api("POST", "/collections/shell_preferences/records", prefs); |
| 1477 | } |
| 1478 | |
| 1479 | // One combined switch for both engines together -- talk_tts_engine and |
| 1480 | // talk_stt_engine are separate prefs (fine-grained control still lives in |
| 1481 | // the settings form), but "get back to this mode" means the SAME |
| 1482 | // push-to-talk-plus-cloud-voice combination just tried, not two things to |
| 1483 | // remember to flip separately. |
| 1484 | function resolvedCloudVoiceOn() { |
| 1485 | return talkTtsEngine() === "openai" && talkSttEngine() === "openai"; |
| 1486 | } |
| 1487 | |
| 1488 | function initCloudVoiceToggle() { |
| 1489 | const btn = document.getElementById("cloudvoice"); |
| 1490 | if (!btn) return; |
| 1491 | btn.hidden = false; |
| 1492 | btn.textContent = resolvedCloudVoiceOn() ? "cloud voice: on" : "cloud voice: off"; |
| 1493 | btn.addEventListener("click", async () => { |
| 1494 | const turningOn = !resolvedCloudVoiceOn(); |
| 1495 | btn.disabled = true; |
| 1496 | btn.textContent = "saving…"; |
| 1497 | await savePrefs({ |
| 1498 | talk_tts_engine: turningOn ? "openai" : "local", |
| 1499 | talk_stt_engine: turningOn ? "openai" : "browser", |
| 1500 | }); |
| 1501 | // A clean reload, no ?tts=/?stt= -- the freshly-saved preference is |
| 1502 | // the only source of truth for the next load. Without this, a page |
| 1503 | // reached via a "?stt=openai" link could look like the toggle did |
| 1504 | // nothing (still "on" from the URL) or, worse, silently keep forcing |
| 1505 | // cloud back on after the user just explicitly turned it off. |
| 1506 | location.href = location.pathname; |
| 1507 | }); |
| 1508 | } |
| 1509 | |
| 1510 | // Only meaningful under cloud speech recognition (talk_stt_engine === |
| 1511 | // "openai") -- hidden otherwise rather than shown inert, same as the mic |
| 1512 | // button itself is conditionally shown. Unlike the cloud-voice toggle |
| 1513 | // above, this one does NOT reload the page: it only changes behavior |
| 1514 | // INSIDE initCloudMic's already-wired click handler (talkHeadphonesMode() |
| 1515 | // is read fresh on every tap), never which top-level mic implementation |
| 1516 | // initMic() chose at load, so there's nothing that needs re-initializing. |
| 1517 | function initHeadphonesToggle() { |
| 1518 | const btn = document.getElementById("headphonestoggle"); |
| 1519 | if (!btn || talkSttEngine() !== "openai") return; |
| 1520 | btn.hidden = false; |
| 1521 | btn.textContent = talkHeadphonesMode() ? "headphones: on" : "headphones: off"; |
| 1522 | btn.addEventListener("click", async () => { |
| 1523 | const turningOn = !talkHeadphonesMode(); |
| 1524 | btn.disabled = true; |
| 1525 | btn.textContent = "saving…"; |
| 1526 | await savePrefs({talk_headphones_mode: turningOn ? "true" : "false"}); |
| 1527 | btn.textContent = turningOn ? "headphones: on" : "headphones: off"; |
| 1528 | btn.disabled = false; |
| 1529 | }); |
| 1530 | } |
| 1531 | |
| 1532 | // The 10-second silence problem: a tool-using model legitimately takes |
| 1533 | // that long, and a page that shows a frozen ellipsis for it reads as |
| 1534 | // dead -- "we don't know because after 10 seconds..." is a verbatim |
| 1535 | // user report. Two signals, chosen for a voice-first surface: a short |
| 1536 | // BLIP the moment the utterance is accepted (the ear knows it fired |
| 1537 | // without looking at the screen), and a visible elapsed counter while |
| 1538 | // the model works (a counter that is moving is a page that is alive, |
| 1539 | // and "thinking... 9s" and "stuck" stop being the same picture). |
| 1540 | let thinkingTimer = null; |
| 1541 | function startThinking() { |
| 1542 | const started = performance.now(); |
| 1543 | capAssistant.textContent = "thinking\\u2026"; |
| 1544 | if (thinkingTimer) clearInterval(thinkingTimer); |
| 1545 | thinkingTimer = setInterval(() => { |
| 1546 | const s = Math.round((performance.now() - started) / 1000); |
| 1547 | if (s >= 2) capAssistant.textContent = "thinking\\u2026 " + s + "s"; |
| 1548 | }, 1000); |
| 1549 | } |
| 1550 | function stopThinking() { |
| 1551 | if (thinkingTimer) { clearInterval(thinkingTimer); thinkingTimer = null; } |
| 1552 | } |
| 1553 | function blip() { |
| 1554 | try { |
| 1555 | const ctx = window.__dbbasicTalkUnlockCtx; |
| 1556 | if (!ctx || ctx.state !== "running") return; |
| 1557 | const osc = ctx.createOscillator(); |
| 1558 | const gain = ctx.createGain(); |
| 1559 | osc.frequency.value = 880; |
| 1560 | gain.gain.setValueAtTime(0.08, ctx.currentTime); |
| 1561 | gain.gain.exponentialRampToValueAtTime(0.0001, ctx.currentTime + 0.12); |
| 1562 | osc.connect(gain); gain.connect(ctx.destination); |
| 1563 | osc.start(); osc.stop(ctx.currentTime + 0.13); |
| 1564 | } catch (e) { /* a missing blip is not an error */ } |
| 1565 | } |
| 1566 | |
| 1567 | // Every path that can produce an utterance -- the voice endpointer |
| 1568 | // (VAD, stability backstop, isFinal debounce) and the manual send/Enter |
| 1569 | // handler -- funnels here. Nothing serialized them, and a real session |
| 1570 | // proved it: "showed me my notes" and "show me my notes" reached the |
| 1571 | // server ONE SECOND apart, two provider calls billed for one utterance, |
| 1572 | // the reply spoken twice. Cause not fully isolated (a stale recognizer |
| 1573 | // instance surviving its own onend restart is the leading suspect), but |
| 1574 | // the fix does not need the cause: a turn already in flight must simply |
| 1575 | // refuse a second one, for ANY reason it might arrive. |
| 1576 | let turnInFlight = false; |
| 1577 | |
| 1578 | async function submitTurn(input) { |
| 1579 | if (turnInFlight) return; // a second utterance chasing the first is dropped, not queued |
| 1580 | turnInFlight = true; |
| 1581 | capUser.textContent = input; |
| 1582 | capUser.classList.remove("armed", "active"); |
| 1583 | capUser.classList.add("sent"); |
| 1584 | blip(); |
| 1585 | startThinking(); |
| 1586 | stopListening(); |
| 1587 | |
| 1588 | let ok, body; |
| 1589 | try { |
| 1590 | // pref(), not raw prefs.tools -- a shell_preferences record written |
| 1591 | // before this field existed in the schema won't have it, and |
| 1592 | // undefined.split() would throw here and silently kill the turn. |
| 1593 | const tools = String(pref("tools", DEFAULT_TOOLS)).split(",").map((t) => t.trim()).filter(Boolean); |
| 1594 | [ok, body] = await api("POST", "/api/ai/chat", |
| 1595 | {message: input, |
| 1596 | model: String(pref("talk_model", "")).trim() || pref("ai_model", DEFAULT_MODEL), |
| 1597 | tools, history: aiHistory.slice(-20), |
| 1598 | session_id: SESSION_ID, source: "talk", |
| 1599 | system: TALK_SYSTEM + " Current local date/time: " + new Date().toString() + "." + readerContextNote()}); |
| 1600 | } finally { |
| 1601 | // Cleared before the reply renders, not after: a fetch that throws |
| 1602 | // must release the gate too, or one failed turn wedges the page shut |
| 1603 | // for the rest of the session. |
| 1604 | turnInFlight = false; |
| 1605 | } |
| 1606 | |
| 1607 | stopThinking(); |
| 1608 | // An empty SUCCESS is the worst reply a voice surface can relay: the |
| 1609 | // round-trip happened, nothing is shown, nothing is spoken, and the |
| 1610 | // user is left staring at a page that looks exactly like a hang. Turn |
| 1611 | // it into words -- one really did come back empty, recorded as out:"" |
| 1612 | // on the server. |
| 1613 | const rawReply = ok ? (body.reply || "I came back empty \u2014 ask me again?") |
| 1614 | : (body.error || "Something went wrong."); |
| 1615 | const replyText = stripViewMarker(rawReply); |
| 1616 | capAssistant.textContent = stripForSpeech(replyText) || replyText; |
| 1617 | |
| 1618 | if (ok) { |
| 1619 | aiHistory.push({role: "user", content: input}); |
| 1620 | aiHistory.push({role: "assistant", content: body.reply}); |
| 1621 | // Same "tool calls are ground truth" reasoning as the view marker |
| 1622 | // below: a read_page call's real result (not the model's prose about |
| 1623 | // it) is what a LATER turn needs to resolve "open that link"/"go to |
| 1624 | // that story" against. See lastReaderPage's own comment. |
| 1625 | const readerCall = (body.tool_calls || []).find((c) => c.name === "read_page" && c.result); |
| 1626 | if (readerCall) lastReaderPage = readerCall.result; |
| 1627 | // Tool calls are ground truth for what was actually created/updated this |
| 1628 | // turn; the [[view:id]] marker is the model retyping an id and can typo or |
| 1629 | // hallucinate it (pointing the stage at a phantom view). So trust the tool |
| 1630 | // call first, fall back to the marker only for "show an existing view again" |
| 1631 | // turns that touched no records. |
| 1632 | const viewPath = viewPathFromToolCalls(body.tool_calls) || viewPathFromReply(rawReply); |
| 1633 | if (viewPath) renderStageView(viewPath); else renderCard(replyText); |
| 1634 | } else { |
| 1635 | renderCard(replyText); |
| 1636 | } |
| 1637 | speak(replyText); |
| 1638 | // No client-side history write: the SERVER records every AI turn as |
| 1639 | // part of /api/ai/chat itself -- stamped, session-grouped, and immune |
| 1640 | // to this page dying before a fire-and-forget write lands (a real turn |
| 1641 | // was lost exactly that way). |
| 1642 | } |
| 1643 | |
| 1644 | // Manual send/Enter always submits whatever is in the voice buffer plus |
| 1645 | // whatever is typed, regardless of endpoint protocol or wake-word arming |
| 1646 | // -- an explicit click/Enter is its own address signal. pendingText() |
| 1647 | // still strips a wake/end word if one happens to be present so a |
| 1648 | // half-spoken command doesn't come along for the ride. |
| 1649 | document.getElementById("prompt").addEventListener("submit", (event) => { |
| 1650 | event.preventDefault(); |
| 1651 | const box = event.target.elements["line"]; |
| 1652 | const typed = box.value.trim(); |
| 1653 | const voice = conversationMode ? pendingText() : ""; |
| 1654 | box.value = ""; |
| 1655 | if (finalDebounceTimer) { clearTimeout(finalDebounceTimer); finalDebounceTimer = null; } |
| 1656 | resetTalkState(); |
| 1657 | const input = [voice, typed].filter(Boolean).join(" ").trim(); |
| 1658 | if (!input) { if (conversationMode) updateCaptionArmed(); return; } |
| 1659 | submitTurn(input); |
| 1660 | }); |
| 1661 | // initMic()'s job is deciding WHICH mic implementation to wire up at all |
| 1662 | // (browser SpeechRecognition vs. cloud push-to-talk) -- a one-time, |
| 1663 | // load-bearing decision, unlike wakeWord()/endpointMode()/etc., which are |
| 1664 | // read live and repeatedly deep into an already-running conversation |
| 1665 | // (loadPrefs() has always finished long before a user starts talking). |
| 1666 | // Calling it before loadPrefs() resolves used to mean it ALWAYS saw the |
| 1667 | // hardcoded defaults, never the real stored talk_stt_engine -- a real bug |
| 1668 | // (settings-based cloud STT could never activate, only a ?stt= URL |
| 1669 | // override could, since that alone doesn't depend on prefs). initCloudVoiceToggle() |
| 1670 | // has the same real dependency, for the same reason: its initial label |
| 1671 | // must reflect what's actually stored, not the shipped default. |
| 1672 | loadPrefs().then(() => { initMic(); initCloudVoiceToggle(); initHeadphonesToggle(); }); |
| 1673 | |
| 1674 | |
| 1675 | // === sessions ================================================================= |
| 1676 | // |
| 1677 | // The same fold the shell does over the server-recorded turns, with one |
| 1678 | // deliberate difference: Talk NEVER auto-resumes. A voice page that |
| 1679 | // silently reloads twenty old turns into model context changes what the |
| 1680 | // next answer means without the user hearing anything change -- resume |
| 1681 | // here is a hand action on the picker, always. (The shell auto-resumes |
| 1682 | // inside a 30-minute window; eyes can see a restored transcript, ears |
| 1683 | // cannot.) |
| 1684 | let sessions = []; |
| 1685 | |
| 1686 | function foldSessions(rows) { |
| 1687 | const by = {}; |
| 1688 | for (const row of rows) { |
| 1689 | const sid = row.session_id || ""; |
| 1690 | if (!sid) continue; |
| 1691 | (by[sid] = by[sid] || []).push(row); |
| 1692 | } |
| 1693 | return Object.entries(by).map(([id, rs]) => { |
| 1694 | rs.sort((a, b) => String(a.created_at || "").localeCompare(String(b.created_at || ""))); |
| 1695 | const first = rs.find((r) => r.input) || rs[0]; |
| 1696 | return {id, rows: rs, count: rs.length, |
| 1697 | title: String((first && first.input) || "(untitled)").slice(0, 40), |
| 1698 | last: String(rs[rs.length - 1].created_at || "")}; |
| 1699 | }).sort((a, b) => b.last.localeCompare(a.last)).slice(0, 10); |
| 1700 | } |
| 1701 | |
| 1702 | function agoShort(iso) { |
| 1703 | const ms = Date.now() - Date.parse(iso || 0); |
| 1704 | if (!isFinite(ms) || ms < 0) return ""; |
| 1705 | const m = Math.round(ms / 60000); |
| 1706 | if (m < 1) return "now"; |
| 1707 | if (m < 60) return m + "m"; |
| 1708 | const h = Math.round(m / 60); |
| 1709 | if (h < 24) return h + "h"; |
| 1710 | return Math.round(h / 24) + "d"; |
| 1711 | } |
| 1712 | |
| 1713 | function renderSessionPicker() { |
| 1714 | const sel = document.getElementById("sessionpick"); |
| 1715 | if (!sel) return; |
| 1716 | const opts = ['<option value="__new">\u2795 new session</option>']; |
| 1717 | for (const s of sessions) { |
| 1718 | const label = s.title + (s.last ? " \u2014 " + agoShort(s.last) : ""); |
| 1719 | opts.push('<option value="' + esc(s.id) + '"' |
| 1720 | + (s.id === SESSION_ID ? " selected" : "") + ">" |
| 1721 | + esc(label) + "</option>"); |
| 1722 | } |
| 1723 | sel.innerHTML = opts.join(""); |
| 1724 | if (!sessions.some((s) => s.id === SESSION_ID)) sel.value = "__new"; |
| 1725 | } |
| 1726 | |
| 1727 | async function loadSessions() { |
| 1728 | const res = await fetch("/collections/shell_commands/records?limit=1000", |
| 1729 | {credentials: "same-origin", headers: {accept: "application/json"}}); |
| 1730 | if (!res.ok) return; |
| 1731 | sessions = foldSessions((await res.json()).records || []); |
| 1732 | renderSessionPicker(); |
| 1733 | |
| 1734 | const sel = document.getElementById("sessionpick"); |
| 1735 | if (sel) sel.addEventListener("change", () => { |
| 1736 | if (sel.value === "__new") { |
| 1737 | SESSION_ID = freshSessionId(); |
| 1738 | aiHistory = []; |
| 1739 | lastReaderPage = null; |
| 1740 | if (headphonesArmed) stopHeadphonesMode(document.getElementById("mic")); |
| 1741 | capAssistant.textContent = "new session"; |
| 1742 | renderSessionPicker(); |
| 1743 | return; |
| 1744 | } |
| 1745 | const s = sessions.find((x) => x.id === sel.value); |
| 1746 | if (!s) return; |
| 1747 | SESSION_ID = s.id; |
| 1748 | aiHistory = []; |
| 1749 | lastReaderPage = null; |
| 1750 | if (headphonesArmed) stopHeadphonesMode(document.getElementById("mic")); |
| 1751 | for (const row of s.rows.slice(-10)) { |
| 1752 | if (row.kind === "ai" && row.output) { |
| 1753 | aiHistory.push({role: "user", content: row.input}); |
| 1754 | aiHistory.push({role: "assistant", content: row.output}); |
| 1755 | } |
| 1756 | } |
| 1757 | // Say what just happened, on the surface the user is actually using: |
| 1758 | // the context changed, and a voice page must not do that silently. |
| 1759 | capAssistant.textContent = "resumed \u201c" + s.title + "\u201d (" |
| 1760 | + s.count + " turns)"; |
| 1761 | renderSessionPicker(); |
| 1762 | }); |
| 1763 | } |
| 1764 | loadSessions(); |
| 1765 | |
| 1766 | """ |
| 1767 | |
| 1768 | |
| 1769 | def GET(request): |
| 1770 | identity = request.get("_identity", {}) |
| 1771 | user_id = identity.get("user_id") |
| 1772 | _logger.info("site_talk served", user_id=user_id or "anonymous") |
| 1773 | |
| 1774 | if not user_id: |
| 1775 | body = '<p class="hint"><a href="/login?next=/talk">Sign in</a> to use Talk.</p>' |
| 1776 | script = "" |
| 1777 | else: |
| 1778 | body = """ |
| 1779 | <div class="talkwrap"> |
| 1780 | <div id="stage" class="stage"><div class="card placeholder">Tap the mic and talk, or type below.</div></div> |
| 1781 | <div class="bar"> |
| 1782 | <div class="captions"> |
| 1783 | <div id="capUser" class="cap user"></div> |
| 1784 | <div id="capAssistant" class="cap assistant"></div> |
| 1785 | </div> |
| 1786 | <div id="endpointhint" class="endpointhint" hidden></div> |
| 1787 | <div class="controls"> |
| 1788 | <div class="micwrap"> |
| 1789 | <button type="button" id="mic" hidden aria-label="toggle conversation mode">mic</button> |
| 1790 | <span id="miclevel" class="miclevel" hidden></span> |
| 1791 | </div> |
| 1792 | <form id="prompt" autocomplete="off"> |
| 1793 | <input name="line" placeholder="or type..." autofocus> |
| 1794 | <button type="submit" class="btn primary" aria-label="send">send</button> |
| 1795 | </form> |
| 1796 | <select id="sessionpick" aria-label="conversation"> |
| 1797 | <option value="__new">➕ new session</option></select> |
| 1798 | <button type="button" id="cloudvoice" class="backlink" hidden>cloud voice: off</button> |
| 1799 | <button type="button" id="headphonestoggle" class="backlink" hidden>headphones: off</button> |
| 1800 | <a class="backlink" href="/shell">back to shell</a> |
| 1801 | </div> |
| 1802 | </div> |
| 1803 | </div> |
| 1804 | """ |
| 1805 | script = f"<script>const OWNER_ID = {user_id!r}; const TALK_SYSTEM = {json.dumps(TALK_SYSTEM)};{_SCRIPT}</script>" |
| 1806 | |
| 1807 | who = ( |
| 1808 | f"signed in as <strong>{user_id}</strong>" |
| 1809 | if user_id |
| 1810 | else '<a href="/login?next=/talk">sign in</a>' |
| 1811 | ) |
| 1812 | html = f"""<!doctype html> |
| 1813 | <html lang="en"> |
| 1814 | <head> |
| 1815 | <meta charset="utf-8"> |
| 1816 | <meta name="viewport" content="width=device-width, initial-scale=1"> |
| 1817 | <title>Talk</title> |
| 1818 | <link rel="stylesheet" href="/style"> |
| 1819 | <style>{_STYLE}</style> |
| 1820 | </head> |
| 1821 | <body> |
| 1822 | <div class="wrap"> |
| 1823 | <header class="app"><h1>Talk</h1><div class="who">{who}</div></header> |
| 1824 | {body} |
| 1825 | </div> |
| 1826 | <script src="/markdown"></script> |
| 1827 | {script} |
| 1828 | <script src="/nav"></script> |
| 1829 | </body> |
| 1830 | </html>""" |
| 1831 | return {"content_type": "text/html; charset=utf-8", "body": html} |