askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
doc_pages.tsv · 450.1 KB · 10612 lines
| 1 | id title category nav_order summary content |
| 2 | why Why DBBASIC — The Advantages, Honestly Stated Start Here 100 A plain-spoken case for DBBASIC's design — live changes, apps as data, one permission engine — with its current boundaries stated just as plainly. "DBBASIC is a small server with strong opinions. This page states the |
| 3 | advantages plainly, and the boundaries next to them, because a claim |
| 4 | without its boundary is marketing. |
| 5 | |
| 6 | ## Change is live |
| 7 | |
| 8 | Objects load per execution: edit an object and the next request serves |
| 9 | the new code. Schemas, permission rules, routes, and site pages are |
| 10 | data, changed over HTTP. Only changes to the server file itself need a |
| 11 | restart (seconds). There is no build, no deploy pipeline, no migration |
| 12 | ceremony. |
| 13 | |
| 14 | Safety comes from **reversibility instead of ceremony**: object source |
| 15 | is versioned with rollback, schemas keep history, records keep |
| 16 | changelogs, permission changes and admin actions are audited. Ship in |
| 17 | steps of one; undo in steps of one. |
| 18 | |
| 19 | ## Apps are data |
| 20 | |
| 21 | An application is a package: schema + permission rules + optionally one |
| 22 | page object ([the app suite](/docs/app-packages)). The server grows only |
| 23 | when a genuinely new *kind* of capability appears — search, sharing, |
| 24 | file storage, AI — and each lands once, generically, then every app |
| 25 | uses it by declaring so in its schema. |
| 26 | |
| 27 | The consequence: **capability grows with data, not code.** Add a |
| 28 | schema for any obscure domain and it immediately has validated writes, |
| 29 | generated forms, search, sharing, MCP tools, and AI conversation about |
| 30 | it. Nobody ships function-calling for your beekeeping records; here it |
| 31 | is free. |
| 32 | |
| 33 | ## One permission engine, every surface |
| 34 | |
| 35 | Browser pages, the records API, MCP agents, global search, file |
| 36 | downloads, and AI tool calls all pass the same policy: role rules, row |
| 37 | filters (`$user_id`, `$accessible_projects`, `$owned_projects`), field |
| 38 | redaction, audit. A rule written once binds everywhere — no surface can |
| 39 | leak what another surface hides. Sharing is records (`project_access` |
| 40 | rows), so granting and revoking are audited record writes. |
| 41 | |
| 42 | ## Fast because it does less |
| 43 | |
| 44 | A request here is: a dict-router match, a TSV read the OS page cache |
| 45 | already holds, permission string-checks, JSON out. No ORM, no |
| 46 | connection pool, no query planner, no middleware stack, no signal |
| 47 | cascade. At personal-and-team data sizes (thousands of rows per |
| 48 | collection), scanning a hot file in-process beats an indexed query |
| 49 | across a socket. |
| 50 | |
| 51 | **Boundary:** this inverts somewhere past hundreds of thousands of rows |
| 52 | in one collection. Collections are per-file, so a hot one can gain an |
| 53 | index later without changing the platform; the search contract promises |
| 54 | no ranking, so an index can slot in behind it. |
| 55 | |
| 56 | ## AI-native without AI lock-in |
| 57 | |
| 58 | The whole admin surface is MCP; agents operate the server with their |
| 59 | own identities and audited sessions. Users store their own provider |
| 60 | keys (write-only), pick their own models, and hand the AI a chosen |
| 61 | subset of tools ([the shell](/docs/shell-and-ai)). The server never meters |
| 62 | your tokens or requires a vendor. An AI acting for a user is exactly as |
| 63 | powerful as the user — no more. |
| 64 | |
| 65 | ## Yours |
| 66 | |
| 67 | Stdlib-only Python plus uvicorn; one VM; data in human-readable TSV and |
| 68 | JSON files you can grep, back up, and take elsewhere. Small enough for |
| 69 | one person to read. MIT licensed, with the operator console (Scroll) |
| 70 | open source too. Multi-domain hosting means one instance serves many |
| 71 | sites with one identity store and one audit trail. |
| 72 | |
| 73 | ## Current boundaries, plainly |
| 74 | |
| 75 | - One VM; no cluster correctness claims. |
| 76 | - Realtime push is live over one permission-filtered websocket (`/ws`); |
| 77 | the durable event log remains as the poll fallback. |
| 78 | - A background-job runtime exists (`app-runner`, plus the daemon's |
| 79 | scheduler and queue); what it lacks is handlers for long media work |
| 80 | such as PDF extraction and thumbnails. |
| 81 | - Untrusted arbitrary code execution is not offered; object writes are |
| 82 | operator/admin actions, and the worker pool isolates processes without |
| 83 | CPU or memory limits. |
| 84 | - Open signup and password reset flows are not built; identities are |
| 85 | operator-created today. |
| 86 | - Permission enforcement is opt-in per deployment rather than the shipped |
| 87 | default (staging runs with it on). |
| 88 | |
| 89 | The direction on each is recorded in status. The rule for |
| 90 | new capability is unchanged: it must pass the same permission engine, |
| 91 | audit trail, and reversibility bar as everything before it. |
| 92 | " |
| 93 | quickstart Quickstart — From a Fresh VM to Your First App Start Here 110 A linear, thirty-minute path from a bare Debian/Ubuntu VM to a running DBBASIC Object Server with login, HTTPS, and your first app installed. "This is the linear path: a bare Debian/Ubuntu VM to a running object |
| 94 | server, a login, a public domain, and a first app installed — in one |
| 95 | sitting, about thirty minutes, most of it waiting on `apt`. Every step |
| 96 | is a real command you can paste. |
| 97 | |
| 98 | For the full reference (systemd hardening, journald caps, backups, |
| 99 | permission rollout, per-route proxy allowlists), see |
| 100 | [single-vm deployment](/docs/single-vm-deployment). This page is the happy |
| 101 | path; that one is the operator's manual. |
| 102 | |
| 103 | ```mermaid |
| 104 | flowchart LR |
| 105 | A[""Fresh VM<br/>+ a domain""] --> B[""Run the installer""] |
| 106 | B --> C[""Create your admin user<br/>+ enable login""] |
| 107 | C --> D[""Caddy: domain + TLS""] |
| 108 | D --> E[""Enable package installs""] |
| 109 | E --> F[""Install an app<br/>→ live at /notes""] |
| 110 | ``` |
| 111 | |
| 112 | ## 0. What you need |
| 113 | |
| 114 | - A small VM (a $5–$7 droplet is plenty) running Debian 12 or Ubuntu |
| 115 | 22.04+, with root or `sudo`. |
| 116 | - Optional but recommended: a domain name with an `A` record pointing at |
| 117 | the VM's IP, so Caddy can get you automatic HTTPS. |
| 118 | |
| 119 | ## 1. Run the installer |
| 120 | |
| 121 | The installer is a readable script — download it, read it, run it. It |
| 122 | installs dependencies, creates the service user and layout, builds a |
| 123 | virtualenv, generates an admin token, and starts a systemd service on |
| 124 | `127.0.0.1:8001`. It is idempotent, so re-running it is safe. |
| 125 | |
| 126 | ```bash |
| 127 | git clone <repository-url> dbbasic-object-server |
| 128 | cd dbbasic-object-server |
| 129 | less scripts/install.sh # read it first |
| 130 | sudo bash scripts/install.sh |
| 131 | ``` |
| 132 | |
| 133 | When it finishes you have a running server. Confirm: |
| 134 | |
| 135 | ```bash |
| 136 | curl http://127.0.0.1:8001/health |
| 137 | ``` |
| 138 | |
| 139 | Your admin token lives in `/etc/dbbasic-object-server.env` as |
| 140 | `DBBASIC_ADMIN_TOKEN` (readable by root and the service group only). |
| 141 | Everything the installer did on purpose *not* to do — login, a public |
| 142 | domain, app installs — is below, as short deliberate steps. |
| 143 | |
| 144 | ## 2. Create your admin user and turn on login |
| 145 | |
| 146 | ```bash |
| 147 | cd /opt/dbbasic-object-server |
| 148 | sudo -u dbbasic DBBASIC_DATA_DIR=/var/lib/dbbasic-object-server/data \ |
| 149 | .venv/bin/python object_identity_cli.py create-superuser \ |
| 150 | --user-id you --email you@example.com |
| 151 | ``` |
| 152 | |
| 153 | It prompts for a password twice (no echo) and stores only a scrypt hash |
| 154 | under the runtime data directory. Now enable password login and restart: |
| 155 | |
| 156 | ```bash |
| 157 | sudo sed -i 's/^DBBASIC_ENABLE_PASSWORD_LOGIN=false/DBBASIC_ENABLE_PASSWORD_LOGIN=true/' \ |
| 158 | /etc/dbbasic-object-server.env |
| 159 | sudo systemctl restart dbbasic-object-server |
| 160 | ``` |
| 161 | |
| 162 | ## 3. Put a domain and HTTPS in front |
| 163 | |
| 164 | Keep uvicorn bound to localhost and let Caddy terminate TLS. If a domain |
| 165 | points at the VM, this is all it takes: |
| 166 | |
| 167 | ```bash |
| 168 | sudo apt install -y caddy |
| 169 | sudo tee /etc/caddy/Caddyfile >/dev/null <<'EOF' |
| 170 | your-domain.com { |
| 171 | handle / { |
| 172 | rewrite * /objects/site_home |
| 173 | reverse_proxy 127.0.0.1:8001 |
| 174 | } |
| 175 | handle { |
| 176 | reverse_proxy 127.0.0.1:8001 |
| 177 | } |
| 178 | } |
| 179 | EOF |
| 180 | sudo systemctl reload caddy |
| 181 | ``` |
| 182 | |
| 183 | Caddy fetches a certificate automatically. Visit `https://your-domain.com` |
| 184 | — you should see the home object, and `https://your-domain.com/login` |
| 185 | should accept the user you just created. |
| 186 | |
| 187 | > The catch-all `handle` above proxies every path to the server, which |
| 188 | > relies on the permission policy as the gate. For a locked-down public |
| 189 | > staging box that instead exposes only named routes, use the per-route |
| 190 | > allowlist in [single-vm deployment](/docs/single-vm-deployment#https-proxy). |
| 191 | |
| 192 | ## 4. Enable app installs and install your first app |
| 193 | |
| 194 | Apps are packages that ship in the repo under `packages/`. Turn on |
| 195 | installs, then ask the server to install one. (Installs are gated by the |
| 196 | admin token, so this stays an operator action.) |
| 197 | |
| 198 | ```bash |
| 199 | sudo sed -i 's/^DBBASIC_ENABLE_PACKAGE_INSTALLS=false/DBBASIC_ENABLE_PACKAGE_INSTALLS=true/' \ |
| 200 | /etc/dbbasic-object-server.env |
| 201 | sudo systemctl restart dbbasic-object-server |
| 202 | |
| 203 | TOKEN=$(sudo grep -oP 'DBBASIC_ADMIN_TOKEN=\K.*' /etc/dbbasic-object-server.env) |
| 204 | |
| 205 | # Projects is the hub other apps relate to; install it first, then notes. |
| 206 | for app in app-projects app-notes; do |
| 207 | curl -s -X POST http://127.0.0.1:8001/packages/$app/install \ |
| 208 | -H ""Authorization: Token $TOKEN"" -H 'Content-Type: application/json' -d '{}' |
| 209 | echo |
| 210 | done |
| 211 | ``` |
| 212 | |
| 213 | `app-notes` includes a page object. To turn permission enforcement on so |
| 214 | each user sees only their own notes, follow the audit-then-enforce |
| 215 | rollout in [permissions](/docs/permissions-model#route-enforcement); on a |
| 216 | single-operator box you can leave it open to start. |
| 217 | |
| 218 | Visit `https://your-domain.com/notes`, sign in, and you have a working |
| 219 | notes app — quick capture, projects, search, public sharing — that you |
| 220 | never wrote a line of server code for. Browse |
| 221 | [the app suite](/docs/app-packages) for the rest (tasks, contacts, |
| 222 | articles, calendar, files, the shell) and install the ones you want. |
| 223 | |
| 224 | ## 5. Optional: the talk-to-everything shell |
| 225 | |
| 226 | Install `app-shell`, then store an AI provider key and chat with your |
| 227 | data — see [the shell and AI guide](/docs/shell-and-ai). Storing a key is |
| 228 | one command in the shell (`/key anthropic sk-...`), and the AI operates |
| 229 | the server with your permissions. |
| 230 | |
| 231 | ## What to do next |
| 232 | |
| 233 | - **Back it up.** All state is files under |
| 234 | `/var/lib/dbbasic-object-server/data`; see [backup and restore](/docs/backup-restore). |
| 235 | - **Harden it.** systemd sandboxing, journald caps, and the readiness |
| 236 | gates are in [single-vm deployment](/docs/single-vm-deployment). |
| 237 | - **Understand the model.** [Why DBBASIC](/docs/why) and |
| 238 | [comparisons](/docs/comparisons) explain what you are and are not |
| 239 | signing up for. |
| 240 | - **Build.** Copy `app-notes` to make your own app |
| 241 | ([package authoring](/docs/package-authoring)), or add a system-tool |
| 242 | object ([capability objects](/docs/capability-objects)). |
| 243 | |
| 244 | ## Upgrading later |
| 245 | |
| 246 | ```bash |
| 247 | cd /opt/dbbasic-object-server |
| 248 | sudo -u dbbasic git pull --ff-only |
| 249 | sudo -u dbbasic .venv/bin/python -m pip install -e '.[server]' |
| 250 | sudo systemctl restart dbbasic-object-server |
| 251 | ``` |
| 252 | |
| 253 | Changes to live objects and schemas need no deploy at all — that is the |
| 254 | point. Only server-code upgrades need the pull-and-restart above. |
| 255 | " |
| 256 | architecture Architecture — The Layers, and Where Each Doc Fits Start Here 120 A map of the DBBASIC Object Server stack, from storage through access surfaces, with pointers to the focused doc for each layer. "DBBASIC is a small stack. Everything is a record or an object in a |
| 257 | human-readable store; each layer above adds capability without hiding the one |
| 258 | below. This page is the map — read it first, then branch into the focused doc |
| 259 | for whichever layer you care about. |
| 260 | |
| 261 | ```mermaid |
| 262 | flowchart TB |
| 263 | A[""Access surfaces — Web · HTTP API · Shell / AI / MCP · Files""] |
| 264 | P[""Permission engine — one policy enforced on every surface""] |
| 265 | G[""Generative UI — list / table / board / tree / calendar · forms · detail · filters · search""] |
| 266 | C[""Capabilities — comments · attachments · sharing""] |
| 267 | D[""Data — schemas (types · validation · relations · transitions) + collections""] |
| 268 | R[""Runtime — ASGI object execution · realtime change log · daemon (triggers · schedules)""] |
| 269 | S[""Storage — human-readable TSV + object source / state / logs / files / versions""] |
| 270 | |
| 271 | A --> P --> G |
| 272 | G --> C |
| 273 | G --> D |
| 274 | C --> D |
| 275 | D --> R --> S |
| 276 | R -. live updates .-> G |
| 277 | ``` |
| 278 | |
| 279 | Every app is a **package** installed onto this stack — a schema, permission |
| 280 | rules, and at most one page object, with no app-specific server code. The same |
| 281 | stack serves the web page, the API, search, files, and the AI. |
| 282 | |
| 283 | ## The layers, bottom to top |
| 284 | |
| 285 | **Storage.** Collections are plain TSV you can `grep`, in one of two modes |
| 286 | (classic overwrite, or append-only with tombstones + compaction). Each object |
| 287 | keeps its source, state, logs, files, and version trail next to it. |
| 288 | See `storage-modes.md`, `append-only-storage-design.md`, |
| 289 | `durability-and-recovery.md`, `backup-restore.md`. |
| 290 | |
| 291 | **Runtime.** A long-running ASGI server executes objects in-process, pooled, or |
| 292 | in a subprocess. A daemon runs scheduled passes, compaction, and |
| 293 | trigger→target automation. A websocket change log lets any surface re-render |
| 294 | live. |
| 295 | See `runtime-contract.md`, `realtime.md`, `asgi-realtime-direction.md`, |
| 296 | `object-authoring.md`, `capability-objects.md`. |
| 297 | |
| 298 | **Data & schema.** A collection's schema declares its fields — types, enums, |
| 299 | relations (validated pointers), bounds, defaults, and status transitions — and |
| 300 | those are enforced on every write, not just in the form. |
| 301 | See `schema-forms.md`, `validation-and-logic.md`, |
| 302 | `business-logic-patterns.md`, `logic-decisions.md`. |
| 303 | |
| 304 | **Permissions.** One engine decides access for every surface: access modes, |
| 305 | role/object/action rules, ownership, row and field filters, temporary paid |
| 306 | access, and record sharing. Routing maps URLs; the policy decides what a |
| 307 | caller may see or write. |
| 308 | See `permissions-model.md`, `site-routing.md`, `secrets-and-credentials.md`. |
| 309 | |
| 310 | **Generative UI.** One renderer turns a schema into a live |
| 311 | list/table/board/tree/calendar, a form, and a composed detail page — with |
| 312 | filters, sort, search, relation labels, and realtime updates — no per-app UI |
| 313 | code. The design system is a semantic contract, not prescribed widgets. |
| 314 | See `generative-ui.md`, `design-system.md`, `ui-decisions.md`. |
| 315 | |
| 316 | **Capabilities.** Generic per-collection *behaviors* on top of the display |
| 317 | layer: a schema flag grows a comment thread, attachment list, or owner-checked |
| 318 | sharing, with no per-app table. |
| 319 | See `capabilities.md`. |
| 320 | |
| 321 | **Access surfaces.** The same records reach a web page, the HTTP API, |
| 322 | cross-collection search, file up/download, MCP agents, and a talk-to-everything |
| 323 | shell — all behind the one permission engine. |
| 324 | See `http-api-contract.md`, `rest-and-object-messages.md`, `shell-and-ai.md`. |
| 325 | |
| 326 | **Packages.** Apps are installable packages with a data-preserving upgrade |
| 327 | system (provenance baselines, three-way reconcile, override objects, feature |
| 328 | flags). |
| 329 | See `app-packages.md`, `package-authoring.md`, `upgrade-and-customization.md`. |
| 330 | |
| 331 | ## Operating it |
| 332 | |
| 333 | See `quickstart.md` (fresh VM to first app), `single-vm-deployment.md`, |
| 334 | `docker-deployment.md`, `traffic-limits.md`, `status.md`. |
| 335 | |
| 336 | ## Why it's shaped this way |
| 337 | |
| 338 | See `why-dbbasic.md` (the advantages, honestly stated) and `comparisons.md` |
| 339 | (what DBBASIC deletes compared to Django/Rails/no-code tools, and the cost of |
| 340 | each deletion). |
| 341 | " |
| 342 | zero-to-code From Zero Code to Some Code Start Here 130 The staged path every app on the DBBASIC Object Server follows, from a schema with no code at all through hooks, event handlers, verbs, scheduled runners, and hand-written pages. "The question every generative system eventually gets asked: *fine, the |
| 343 | metadata builds the app — but when I need real code, where does it go?* |
| 344 | This page is the known path. Every app on this server grows through the |
| 345 | same stages, in the same order, and the order is the point: **code enters |
| 346 | through the write path first and the page last** — the exact reverse of |
| 347 | the classic web framework, where code entered through the controller |
| 348 | first and that is where the logic rotted. |
| 349 | |
| 350 | ## Stage 0 — an app with no code at all |
| 351 | |
| 352 | A schema (`schemas/notes.json`), permission rules, and two seeded view |
| 353 | records (an index at `/notes`, a detail at `/notes/{id}`). From those |
| 354 | alone: list, search, sort, create/edit forms, a detail page, realtime |
| 355 | updates, the JSON API, MCP tools for agents, and global search. All of it |
| 356 | reads the schema; none of it is emitted as code |
| 357 | ([`generative-ui.md`](/docs/generative-ui), including why this is *not* |
| 358 | scaffolding). |
| 359 | |
| 360 | `app-notes` ships this way today: a package registering zero objects. |
| 361 | |
| 362 | ## Stage 1 — logic as declarations (still no code) |
| 363 | |
| 364 | Most business rules are not code and never should be: |
| 365 | |
| 366 | - shape, bounds, required, enums, FK-exists → **schema validation** |
| 367 | - who may do what, whose rows are whose → **permission rules** |
| 368 | - legal state moves and who may make them → **`transitions` + `when`** |
| 369 | - same-record derivations → **`formula` fields** |
| 370 | - sums over children → **`rollup` fields** |
| 371 | |
| 372 | Every surface enforces these because they live on the shared write path, |
| 373 | not in a page. A large fraction of real apps end here. |
| 374 | |
| 375 | ## Stage 2 — the first line of Python is a gate |
| 376 | |
| 377 | The first rule that *cannot* be declared is almost always a cross-record |
| 378 | check: ""a payment may not exceed the invoice balance"", ""a journal must |
| 379 | balance before posting"", ""a loss move cannot have a destination"". That is |
| 380 | a **pre-write hook** — one file, one function: |
| 381 | |
| 382 | ```python |
| 383 | # objects/hook/payments.py schema: ""hooks"": {""before_write"": ""hook_payments""} |
| 384 | def BEFORE_WRITE(request): |
| 385 | if too_much(request[""record""]): |
| 386 | return {""error"": ""..."", ""status"": 409} # reject |
| 387 | record = dict(request[""record""]); record[""rate""] = looked_up # or stamp |
| 388 | return {""record"": record} |
| 389 | ``` |
| 390 | |
| 391 | Hooks run on the shared write path, so the form, the API, and an AI agent |
| 392 | all hit the same gate. They fail closed. This is where code enters the |
| 393 | system, and it enters *enforcing*, not rendering. |
| 394 | |
| 395 | ## Stage 3 — reactions |
| 396 | |
| 397 | Something should *happen* after a write: a status flips, a journal is |
| 398 | composed, someone is notified. Notifications are data (`notify_rules`); |
| 399 | follow-on writes are an **event handler** — a `system/` object with |
| 400 | `HANDLES = [""payments.record.created""]` and an `EVENT(request)` function. |
| 401 | Reactions are post-commit and best-effort: they may never block the write |
| 402 | that already happened ([`logic-decisions.md`](/docs/logic-decisions) #6), and |
| 403 | anything they generate is idempotent by provenance (#7). |
| 404 | |
| 405 | ## Stage 4 — verbs |
| 406 | |
| 407 | Users (and agents) do things that are not CRUD: *reverse this journal*, |
| 408 | *apply this count*, *import this statement*, *resolve this bank line*. |
| 409 | Each is an **action object** — `objects/action/<verb>.py` with a `POST` |
| 410 | that checks identity, validates, and writes through the same records |
| 411 | layer. A verb is the honest home for anything a button should do that a |
| 412 | form cannot. |
| 413 | |
| 414 | ## Stage 5 — time |
| 415 | |
| 416 | Aging, dunning, recurring entries, escalation: time-driven state belongs |
| 417 | to the **daemon** (#2), as a runner object registered on the scheduler |
| 418 | (the `/scheduler` board shows every run's outcome) or a message on the |
| 419 | queue (delay, retry, backoff — already built). A runner is just an object |
| 420 | with a `POST`; it is callable by hand for testing and by the clock in |
| 421 | production. |
| 422 | |
| 423 | ## Stage 6 — presentation code, last of all |
| 424 | |
| 425 | Only when a page must *show* something no generic block can express: |
| 426 | |
| 427 | - first an **`object` block** — one hand-written panel inside an otherwise |
| 428 | generated page (`{""kind"": ""object"", ""object_id"": ""site_my_panel""}`); |
| 429 | - then, rarely, a **full page object** — `site/notes.py` overrides the |
| 430 | generated `/notes` by convention the moment the file exists, and |
| 431 | deleting it reverts. Reports are the legitimate case: the trial balance, |
| 432 | the statements, the reconciliation tie-out are folds across collections |
| 433 | with audit-grade warnings, and they stay hand-written on purpose. |
| 434 | |
| 435 | The rule that keeps stage 6 safe: **a page may never be the only place a |
| 436 | rule exists.** A gate written into a page does not gate the API or an |
| 437 | agent — it provably does not hold — so enforcement stays in stages 1–2 |
| 438 | where every surface shares it. Pages show; they never decide. |
| 439 | |
| 440 | ## The whole path at a glance |
| 441 | |
| 442 | | stage | you add | it lives in | enforced for | |
| 443 | |---|---|---|---| |
| 444 | | 0 | schema + rules + 2 view rows | data | every surface | |
| 445 | | 1 | declarations (transitions, formulas, rollups) | schema | every surface | |
| 446 | | 2 | a gate | `hook/` on the write path | every surface | |
| 447 | | 3 | a reaction | `system/` handler, `notify_rules` | post-commit | |
| 448 | | 4 | a verb | `action/` | its callers | |
| 449 | | 5 | a schedule | runner + scheduler/queue | the clock | |
| 450 | | 6 | a panel, then a page | `object` block → `site/` | presentation only | |
| 451 | |
| 452 | ## Where this page came from |
| 453 | |
| 454 | This path was not designed and then implemented — it was **extracted**. A |
| 455 | week of building real modules concretely (payments, the books spine, bank |
| 456 | reconciliation, inventory losses, cash counts) kept producing the same |
| 457 | placement decisions, and once every kind of business rule had exactly one |
| 458 | home, the homes sorted themselves into the order an app acquires them. |
| 459 | The taxonomy is the load-bearing thing; the path is its corollary. |
| 460 | |
| 461 | That is also why earlier frameworks never had a page like this. They |
| 462 | documented their *mechanisms* (models, controllers, middleware, signals) |
| 463 | but never fixed where *logic* lives — any rule could go anywhere, so there |
| 464 | was no gradient from zero code to some code, only a cliff. ""Fat models, |
| 465 | skinny controllers"" was a slogan, not a placement table with enforcement |
| 466 | behind it. A path only exists where placement is decided. |
| 467 | |
| 468 | The runtime twin of this page is [`write-pipeline.md`](/docs/write-pipeline): |
| 469 | once logic is placed, that page says exactly when it runs, in what order, |
| 470 | and what can stop a write. |
| 471 | |
| 472 | Placement questions beyond this page: the flowchart in |
| 473 | [`business-logic-patterns.md`](/docs/business-logic-patterns) decides where |
| 474 | any individual rule goes, and [`logic-decisions.md`](/docs/logic-decisions) |
| 475 | records why. |
| 476 | " |
| 477 | object-authoring Object Authoring Building on DBBASIC 200 How to write DBBASIC objects: source layout, HTTP-style methods, request identity, runtime helpers, response shapes, and public deployment safety. "This guide describes the object shape supported by the current public server release. It's intentionally small, and is designed to stay compatible with the hardened runtime as more of that runtime's functionality moves into this repository over time. |
| 478 | |
| 479 | ## Source Location |
| 480 | |
| 481 | Object source lives under `objects/` by default: |
| 482 | |
| 483 | ```text |
| 484 | objects/site/home.py |
| 485 | objects/basics/counter.py |
| 486 | objects/users/42/deals.py |
| 487 | ``` |
| 488 | |
| 489 | The source root can be changed for deployments: |
| 490 | |
| 491 | ```text |
| 492 | DBBASIC_OBJECTS_DIR=/var/lib/dbbasic-object-server/objects |
| 493 | ``` |
| 494 | |
| 495 | Object IDs are derived from paths: |
| 496 | |
| 497 | - `objects/site/home.py` becomes `site_home` |
| 498 | - `objects/basics/counter.py` becomes `basics_counter` |
| 499 | - `objects/users/42/deals.py` becomes `u_42_deals` |
| 500 | |
| 501 | ## Methods |
| 502 | |
| 503 | An object is a Python file that can define HTTP-style methods: |
| 504 | |
| 505 | ```python |
| 506 | def GET(request): |
| 507 | return {""status"": ""ok""} |
| 508 | |
| 509 | def POST(request): |
| 510 | return {""status"": ""created"", ""payload"": request} |
| 511 | ``` |
| 512 | |
| 513 | `GET` receives query parameters. `POST`, `PUT`, and `DELETE` receive the parsed |
| 514 | request body plus query parameters where the HTTP contract allows it. |
| 515 | |
| 516 | ## Request Identity |
| 517 | |
| 518 | Every request payload includes a server-controlled `_identity` key describing |
| 519 | the resolved caller, so objects can render per-user pages and make per-user |
| 520 | decisions without hand-rolling auth: |
| 521 | |
| 522 | ```python |
| 523 | def GET(request): |
| 524 | identity = request[""_identity""] |
| 525 | if identity[""user_id""] is None: |
| 526 | return {""status"": 401, ""error"": ""Sign in first""} |
| 527 | return {""hello"": identity[""user_id""], ""roles"": identity[""roles""]} |
| 528 | ``` |
| 529 | |
| 530 | `_identity` contains `user_id`, `account_id`, `roles`, `subscriptions`, and |
| 531 | `auth_method` (`admin_token`, `session_token`, `session_cookie`, |
| 532 | `trusted_headers`, or `anonymous`). The server overwrites any client-supplied |
| 533 | `_identity` value before the object runs, so it cannot be spoofed through the |
| 534 | request body or query string. Anonymous requests still receive the key with |
| 535 | `user_id` set to `None`. |
| 536 | |
| 537 | ## Runtime Helpers |
| 538 | |
| 539 | Loaded object modules receive a small set of runtime helpers. The current public |
| 540 | runtime injects: |
| 541 | |
| 542 | ```python |
| 543 | _state_manager.get(key, default=None) |
| 544 | _state_manager.set(key, value) |
| 545 | _state_manager.get_all() |
| 546 | _state_manager.reload() |
| 547 | |
| 548 | _logger.debug(message, **fields) |
| 549 | _logger.info(message, **fields) |
| 550 | _logger.warning(message, **fields) |
| 551 | _logger.error(message, **fields) |
| 552 | ``` |
| 553 | |
| 554 | State is stored under `data/state/{object_id}/state.tsv`. Logs are stored under |
| 555 | `data/logs/{object_id}/log.tsv`. The active log is plain TSV. Rotated logs are |
| 556 | gzip-compressed by default and can still be read by the server. |
| 557 | |
| 558 | ## Default Storage And Integrations |
| 559 | |
| 560 | The default DBBASIC object should not need a separate database. Source, state, |
| 561 | logs, files, and versions are file-backed so a useful app can start on one VM |
| 562 | with normal Unix paths and backups. |
| 563 | |
| 564 | Objects can still connect to SQL databases, SQLite files, HTTP APIs, AI APIs, or |
| 565 | other services when a workload needs them. Those integrations should be explicit |
| 566 | object behavior or package behavior, not hidden platform requirements. |
| 567 | |
| 568 | This keeps the base stack small while leaving room for: |
| 569 | |
| 570 | - SQL/reporting objects |
| 571 | - schema and relationship inspectors |
| 572 | - API connector objects |
| 573 | - migration objects |
| 574 | - AI-assisted query and data tools |
| 575 | |
| 576 | DBBASIC Scroll can expose SQL, schema, and diagram screens, but those should be |
| 577 | capabilities of the object system rather than proof that every app needs a |
| 578 | database tier by default. |
| 579 | |
| 580 | Schema still matters. The difference is that DBBASIC should let schemas, |
| 581 | validation rules, relationships, and generated admin screens live close to the |
| 582 | objects and packages that use them. That avoids forcing every small app through |
| 583 | an external database, migration, ORM, admin, and deployment loop before the app |
| 584 | has proved it needs that weight. |
| 585 | |
| 586 | ```mermaid |
| 587 | flowchart LR |
| 588 | subgraph DB[""Database-first loop""] |
| 589 | D1[""Design schema""] --> D2[""Run migration""] |
| 590 | D2 --> D3[""Update model/ORM""] |
| 591 | D3 --> D4[""Update route/view/controller""] |
| 592 | D4 --> D5[""Deploy""] |
| 593 | D5 --> D6[""Inspect errors""] |
| 594 | D6 --> D1 |
| 595 | end |
| 596 | |
| 597 | subgraph OBJ[""DBBASIC object loop""] |
| 598 | O1[""Edit object or package metadata""] --> O2[""Run it""] |
| 599 | O2 --> O3[""Read output, logs, state, errors""] |
| 600 | O3 --> O4[""Fix source or schema metadata""] |
| 601 | O4 --> O5[""Keep versions""] |
| 602 | O5 --> O1 |
| 603 | end |
| 604 | |
| 605 | O3 -.-> I[""Optional integrations: SQL, SQLite, HTTP APIs, AI APIs""] |
| 606 | ``` |
| 607 | |
| 608 | ## JSON Object |
| 609 | |
| 610 | ```python |
| 611 | def GET(request): |
| 612 | count = int(_state_manager.get(""count"", 0)) + 1 |
| 613 | _state_manager.set(""count"", count) |
| 614 | _logger.info(""counter served"", count=count) |
| 615 | return { |
| 616 | ""status"": ""ok"", |
| 617 | ""count"": count, |
| 618 | } |
| 619 | ``` |
| 620 | |
| 621 | Normal dictionaries return JSON. |
| 622 | |
| 623 | ## HTML Object |
| 624 | |
| 625 | ```python |
| 626 | def GET(request): |
| 627 | count = int(_state_manager.get(""count"", 0)) + 1 |
| 628 | _state_manager.set(""count"", count) |
| 629 | _logger.info(""home served"", count=count, response_type=""html"") |
| 630 | |
| 631 | html = f""""""<!doctype html> |
| 632 | <html lang=""en""> |
| 633 | <head> |
| 634 | <meta charset=""utf-8""> |
| 635 | <meta name=""viewport"" content=""width=device-width, initial-scale=1""> |
| 636 | <title>DBBASIC Object</title> |
| 637 | </head> |
| 638 | <body> |
| 639 | <h1>DBBASIC Object</h1> |
| 640 | <p>This object has been served {count} times.</p> |
| 641 | </body> |
| 642 | </html> |
| 643 | """""" |
| 644 | |
| 645 | return { |
| 646 | ""content_type"": ""text/html; charset=utf-8"", |
| 647 | ""body"": html, |
| 648 | } |
| 649 | ``` |
| 650 | |
| 651 | Dicts with `content_type` and `body` become raw HTTP responses. This is the |
| 652 | shape used for pages, generated views, images, and other non-JSON object |
| 653 | responses. |
| 654 | |
| 655 | Plain strings also return `text/html; charset=utf-8`. Plain bytes return |
| 656 | `application/octet-stream`. |
| 657 | |
| 658 | ## HTML Form Object |
| 659 | |
| 660 | Forms are how design changes ship without a deploy: the page, the handler, and |
| 661 | the state live in one object. `GET` renders the form; `POST` receives the |
| 662 | submitted fields (form posts from a signed-in browser carry the session cookie, |
| 663 | so `request[""_identity""]` identifies the user in both methods): |
| 664 | |
| 665 | ```python |
| 666 | def GET(request): |
| 667 | identity = request[""_identity""] |
| 668 | if identity[""user_id""] is None: |
| 669 | return (303, [(""Location"", ""/login?next=/objects/notes_quick"")], b"""") |
| 670 | |
| 671 | saved = _state_manager.get(f""note:{identity['user_id']}"", """") |
| 672 | html = f""""""<!doctype html> |
| 673 | <html lang=""en""> |
| 674 | <body> |
| 675 | <h1>Quick note for {identity['user_id']}</h1> |
| 676 | <form method=""post""> |
| 677 | <textarea name=""note"">{saved}</textarea> |
| 678 | <button type=""submit"">Save</button> |
| 679 | </form> |
| 680 | </body> |
| 681 | </html>"""""" |
| 682 | return {""content_type"": ""text/html; charset=utf-8"", ""body"": html} |
| 683 | |
| 684 | |
| 685 | def POST(request): |
| 686 | identity = request[""_identity""] |
| 687 | if identity[""user_id""] is None: |
| 688 | return {""status"": 401, ""error"": ""Sign in first""} |
| 689 | |
| 690 | _state_manager.set(f""note:{identity['user_id']}"", str(request.get(""note"", """"))) |
| 691 | _logger.info(""note saved"", user_id=identity[""user_id""]) |
| 692 | return (303, [(""Location"", ""/objects/notes_quick"")], b"""") |
| 693 | ``` |
| 694 | |
| 695 | Cookie-authenticated form posts are origin-checked by the server (same-site |
| 696 | only), so objects do not need their own CSRF plumbing for basic forms. Records |
| 697 | and schemas are the step up from object state when the data needs listing, |
| 698 | validation, and change history — see the collections sections of |
| 699 | [`http-api-contract.md`](/docs/http-api-contract). |
| 700 | |
| 701 | ## Upload Forms |
| 702 | |
| 703 | `multipart/form-data` posts (forms with `enctype=""multipart/form-data""`) are |
| 704 | parsed for you: text fields arrive as plain payload keys, and each uploaded |
| 705 | file arrives under `request[""_files""]` with its content base64-encoded so the |
| 706 | payload survives the subprocess execution boundary: |
| 707 | |
| 708 | ```python |
| 709 | import base64 |
| 710 | |
| 711 | def POST(request): |
| 712 | upload = request[""_files""][""document""] |
| 713 | content = base64.b64decode(upload[""content_base64""]) |
| 714 | _logger.info(""upload received"", filename=upload[""filename""], size=upload[""size""]) |
| 715 | return {""saved"": upload[""filename""], ""bytes"": len(content)} |
| 716 | ``` |
| 717 | |
| 718 | Each `_files` entry has `filename`, `content_type`, `size`, and |
| 719 | `content_base64`. Upload size is bounded by `DBBASIC_MAX_REQUEST_BYTES`. |
| 720 | |
| 721 | An object that receives an upload can process it with system tools — |
| 722 | OCR, transcode, thumbnail, extract — by shelling out to binaries like |
| 723 | `tesseract` or `ffmpeg`. See [capability objects](/docs/capability-objects) |
| 724 | for the subprocess model, worked examples, and the trust boundary. |
| 725 | Store durable uploads through the object file APIs so they inherit |
| 726 | permissions and change history. |
| 727 | |
| 728 | ## Low-Level Response |
| 729 | |
| 730 | When an object needs exact status and headers, return a tuple: |
| 731 | |
| 732 | ```python |
| 733 | def POST(request): |
| 734 | return (201, [(""Content-Type"", ""text/plain"")], [b""created""]) |
| 735 | ``` |
| 736 | |
| 737 | The tuple shape is: |
| 738 | |
| 739 | ```text |
| 740 | (status, headers, body) |
| 741 | ``` |
| 742 | |
| 743 | `body` may be bytes, text, or a list of bytes/text parts. |
| 744 | |
| 745 | ## Authoring Rule |
| 746 | |
| 747 | Keep each object small enough to inspect, run, log, fix, and version quickly. |
| 748 | The point of DBBASIC is the short loop: |
| 749 | |
| 750 | ```mermaid |
| 751 | flowchart LR |
| 752 | A[""edit object""] --> B[""run""] |
| 753 | B --> C[""read output, logs, state, errors""] |
| 754 | C --> D[""fix""] |
| 755 | D --> E[""keep version""] |
| 756 | E --> A |
| 757 | ``` |
| 758 | |
| 759 | Objects can call other objects, emit events, handle jobs, serve pages, or expose |
| 760 | APIs, but the first rule is the same: make the running object easy to inspect. |
| 761 | |
| 762 | ## Public Deployment Safety |
| 763 | |
| 764 | Until role, object, and row-level permissions are enforced, do not expose the |
| 765 | full `/objects` API on a public hostname. |
| 766 | |
| 767 | For public staging: |
| 768 | |
| 769 | - keep `DBBASIC_ENABLE_SOURCE_WRITES=false` |
| 770 | - generate a deployment-specific `DBBASIC_ADMIN_TOKEN` outside the source tree |
| 771 | - bind uvicorn to `127.0.0.1` |
| 772 | - allowlist explicit public object routes in a reverse proxy |
| 773 | - keep object source, state, logs, versions, and object listing private from |
| 774 | normal public users |
| 775 | " |
| 776 | package-authoring Package Authoring Building on DBBASIC 210 Practical guide to building and installing a DBBASIC package: manifest structure, install semantics, navigation and attention registries, and the authoring workflow. "This is the practical guide for building a DBBASIC package: one installable |
| 777 | directory that ships an app piece — objects, schemas, and seed data — without |
| 778 | a deploy pipeline. It is written for humans and AI tools that generate |
| 779 | packages. The runtime contract details live in `runtime-contract.md#packages`; |
| 780 | this page is the authoring workflow. |
| 781 | |
| 782 | ## What A Package Is |
| 783 | |
| 784 | A package is a directory under `packages/` (or `DBBASIC_PACKAGES_DIR`) with a |
| 785 | manifest and package-relative content: |
| 786 | |
| 787 | ```text |
| 788 | packages/{package_id}/ |
| 789 | dbbasic-package.json |
| 790 | objects/ # object source files |
| 791 | schemas/ # collection schema JSON files |
| 792 | seed/ # initial records TSV files |
| 793 | permissions/ # declared, not yet installed (merge semantics pending) |
| 794 | migrations/ # declared, not yet run (run semantics pending) |
| 795 | ``` |
| 796 | |
| 797 | A good package is one app primitive: contacts, notes, tasks, articles, |
| 798 | invoices. Small enough to review, complete enough to install and use. |
| 799 | |
| 800 | ## The Manifest |
| 801 | |
| 802 | `dbbasic-package.json` is plain JSON: |
| 803 | |
| 804 | ```json |
| 805 | { |
| 806 | ""id"": ""contacts"", |
| 807 | ""name"": ""Contacts"", |
| 808 | ""version"": ""0.1.0"", |
| 809 | ""description"": ""Contact records with a browse/edit page"", |
| 810 | ""compatibility"": {""dbbasic_object_server"": "">=0.1.0""}, |
| 811 | ""dependencies"": [], |
| 812 | ""objects"": [ |
| 813 | {""id"": ""contacts_directory"", ""path"": ""objects/contacts/directory.py""} |
| 814 | ], |
| 815 | ""schemas"": [ |
| 816 | {""collection"": ""contacts"", ""path"": ""schemas/contacts.json""} |
| 817 | ], |
| 818 | ""seed"": [ |
| 819 | {""collection"": ""contacts"", ""path"": ""seed/contacts.tsv""} |
| 820 | ], |
| 821 | ""permissions"": [], |
| 822 | ""migrations"": [], |
| 823 | ""schedules"": [ |
| 824 | {""id"": ""contacts_dedupe_nightly"", |
| 825 | ""object_id"": ""contacts_directory"", |
| 826 | ""schedule"": ""10 6 * * *"", |
| 827 | ""description"": ""Flag likely duplicate contacts overnight.""} |
| 828 | ], |
| 829 | ""nav"": [ |
| 830 | {""id"": ""contacts"", ""label"": ""Contacts"", ""path"": ""/contacts"", |
| 831 | ""blurb"": ""People, the organizations they belong to, and interactions"", |
| 832 | ""surface"": ""member"", ""group"": ""Work"", ""order"": 60} |
| 833 | ] |
| 834 | } |
| 835 | ``` |
| 836 | |
| 837 | Rules the server enforces: |
| 838 | |
| 839 | - `id` must be a safe package identifier; object ids and collection names are |
| 840 | validated with the same rules as the rest of the server. |
| 841 | - All paths are package-relative. Absolute paths, null bytes, and `..` |
| 842 | traversal are rejected at parse time. |
| 843 | - Unknown manifest sections are rejected rather than ignored. |
| 844 | |
| 845 | ## What Install Actually Does Today |
| 846 | |
| 847 | Installs are deliberately conservative: |
| 848 | |
| 849 | - `objects` entries are written under the configured objects root. Existing |
| 850 | objects require `allow_replace`. |
| 851 | - `schemas` entries are validated and written under `data/schemas/`. Existing |
| 852 | schemas require `allow_replace`. |
| 853 | - `seed` TSV is **install-once**: it is written only when the target |
| 854 | collection has no records file yet. On a reinstall or upgrade of a package |
| 855 | whose collection already holds data, seeding is skipped (reported as |
| 856 | `status: ""skipped""`, `action: ""skip""`) — it never overwrites live data, and |
| 857 | it never blocks the install. This is what makes an in-place upgrade safe: |
| 858 | reinstall with `allow_replace` to ship new object code, a migrated schema, |
| 859 | and re-merged permissions while every existing record is preserved. Live |
| 860 | records live outside the package, in `data/collections/<name>/records.tsv` |
| 861 | keyed by collection, so an upgrade has no reason to touch them. (Evolving a |
| 862 | schema across an upgrade — backfilling a newly added field on old rows — is |
| 863 | the separate `migrations` story below, not yet implemented; adding a |
| 864 | read-only/server-set field like `created_at` is safe because old rows simply |
| 865 | read as empty for it.) |
| 866 | - `permissions` entries MERGE: each fragment file is |
| 867 | `{""rules"": [ ... ]}` using the same rule shape as the policy document |
| 868 | (`effect`, `principal`, `actions`, optional `collection`/`object_id`, |
| 869 | row filters, field lists). On install the rules are validated, stamped |
| 870 | with `""package"": ""<package_id>""` provenance, and appended to the |
| 871 | deployment policy — skipping any rule that already exists, so reinstalls |
| 872 | are idempotent. This is how a site package makes its own pages public: |
| 873 | ship the grant with the code instead of hand-editing policy after install. |
| 874 | Invalid fragments block the whole install. |
| 875 | - `migrations` are accepted in the manifest and reported in dry-runs, but |
| 876 | install rejects them until explicit run semantics land. |
| 877 | - `schedules` become `task_<id>` rows in the scheduler trigger's state, |
| 878 | which is where `object_daemon.process_scheduler` reads the board from. |
| 879 | **If your app needs a recurring pass, declare it here.** A schedule that |
| 880 | exists only in a running server's state is invisible to review, absent |
| 881 | from a fresh install, and gone the next time the box is rebuilt — that |
| 882 | is precisely how a demo server ended up with four nightly passes that |
| 883 | appeared nowhere in the package sources. Fields: `id` (becomes the state |
| 884 | key, `[a-z][a-z0-9_]*`), `object_id`, `schedule`, and optional `type` |
| 885 | (`cron` — the default — or `onetime`), `method` (default `POST`), |
| 886 | `payload`, `description`. |
| 887 | |
| 888 | Three properties make declaring a schedule safe to do repeatedly: |
| 889 | |
| 890 | - **Run history survives.** `last_run`, `run_count` and `next_run` |
| 891 | belong to the daemon and are never reset by an install. An upgrade |
| 892 | that forgot when a pass last worked would make ""is this actually |
| 893 | running?"" unanswerable. |
| 894 | - **A pause is honoured.** The package declares what *should* run; an |
| 895 | operator decides what *does* right now. A reinstall never restarts a |
| 896 | task somebody deliberately paused. |
| 897 | - **A changed expression reschedules.** Editing the cron clears |
| 898 | `next_run` so the daemon recomputes it; leaving it alone keeps |
| 899 | tonight's firing exactly where it was, so shipping a patch does not |
| 900 | skip a run. |
| 901 | |
| 902 | Two things are refused outright, at install time rather than at 3am: |
| 903 | an unparseable cron (the daemon reads that as ""no next run"" and moves |
| 904 | on in silence — a pass that never runs and never complains), and a |
| 905 | schedule aimed at an object neither this package nor the server |
| 906 | provides. Removing a schedule from a manifest does **not** delete the |
| 907 | task, matching the rest of install: deregistering never destroys. |
| 908 | - `nav` entries become rows in the `nav_entries` collection, which every |
| 909 | navigation surface folds over. **If your app serves a page, declare its |
| 910 | door here.** See the section below. |
| 911 | - `attention` entries become rows in the `attention_sources` collection, |
| 912 | which the daemon's rollup pass reads its list of providers from. |
| 913 | **If your app has a queue that waits on a person, declare it here.** |
| 914 | See the section below. |
| 915 | - The HTTP install route creates a restore point first and appends changelog |
| 916 | rows under `data/package_changes/{package_id}/changes.jsonl`. |
| 917 | |
| 918 | ## Navigation (`nav`) |
| 919 | |
| 920 | A menu maintained by hand rots the moment somebody ships without editing |
| 921 | it, and the rot is invisible — nothing fails, the front door just |
| 922 | advertises a server that no longer exists. One deployment of this system |
| 923 | accumulated three separate lists of the same apps: an app switcher's JS |
| 924 | array (25 entries), a home page's tile grid (21 entries), and a |
| 925 | collection-to-URL map for search hits. They disagreed with each other and |
| 926 | between them named none of the eight newest apps. Worse, the home page's |
| 927 | tile grid existed in no package at all — it lived only on a running |
| 928 | server, so a rebuilt box came back with no front door and said nothing |
| 929 | about it, which is precisely the failure the `schedules` section was |
| 930 | built to end for cron. |
| 931 | |
| 932 | So a door is declared by the app that owns it, and every navigation |
| 933 | surface becomes a fold over one registry: |
| 934 | |
| 935 | ```json |
| 936 | ""nav"": [ |
| 937 | {""id"": ""shop"", ""label"": ""Shop"", ""path"": ""/shop"", |
| 938 | ""blurb"": ""Browse what is for sale and buy it"", |
| 939 | ""surface"": ""public"", ""group"": ""Commerce"", ""order"": 400} |
| 940 | ] |
| 941 | ``` |
| 942 | |
| 943 | - `id` becomes the `nav_entries` record id, so `[a-z][a-z0-9_]*`. It must |
| 944 | be unique across every installed package: two packages writing one row |
| 945 | is two packages fighting over one door. |
| 946 | - `label` and `path` are required. `path` starts with `/` and holds no |
| 947 | whitespace. |
| 948 | - `blurb` is one honest line in the present tense saying what the app |
| 949 | does — it is rendered under the label on the home grid. No marketing. |
| 950 | - `surface` is `public` | `member` | `operator` | `hidden`, default |
| 951 | `member`. This governs the MENU and is not access control: permissions |
| 952 | decide who may open a page, and leaving a door off the menu hides |
| 953 | nothing that was not already protected. `/shop` and the public |
| 954 | product/pay/portal pages are `public`; ordinary app pages are `member`; |
| 955 | `/scheduler`, `/flow`, `/urls`, `/dashboard` and other admin or debug |
| 956 | pages are `operator`. |
| 957 | - `group` is free text, default `Apps`. The house groups are **Work**, |
| 958 | **Publishing**, **Money**, **Commerce**, **Warehouse** and **System**. |
| 959 | - `order` is an integer, default `100`. Packages band it by group (Work |
| 960 | in the 0s, Publishing the 200s, Money the 300s, Commerce the 400s, |
| 961 | Warehouse the 500s, System the 600s) because a group sorts where its |
| 962 | earliest entry sorts — that way the reading order of the groups falls |
| 963 | out of the same number, instead of a seventh list holding it. |
| 964 | |
| 965 | Three properties make declaring a door safe to do repeatedly: |
| 966 | |
| 967 | - **The operator's `hidden` survives.** `operator_hidden` is a separate |
| 968 | boolean column and an install never writes it on an existing row. The |
| 969 | package says what the app IS; the operator says what they want to SEE. |
| 970 | An upgrade that silently put back a page somebody deliberately removed |
| 971 | would be the same class of incident as one that restarts a paused |
| 972 | nightly pass — so an operator flips `operator_hidden` (a `role:manager` |
| 973 | update on `nav_entries`) instead of editing somebody else's manifest. |
| 974 | - **Unchanged means unwritten.** Every package-owned field — label, path, |
| 975 | blurb, group, surface, order — is compared before anything is written, |
| 976 | so a plan that reports `unchanged` is one where nothing at all is about |
| 977 | to happen. A dry run that under-reports is worse than no dry run. |
| 978 | - **Removing an entry does not delete the row.** Same as the rest of |
| 979 | install: deregistering never destroys. |
| 980 | |
| 981 | Two things are refused, and one deliberately is not: |
| 982 | |
| 983 | - A **duplicate id inside one manifest**, and an **invalid surface, path |
| 984 | or order**, fail at parse time. |
| 985 | - A **collision with another package** blocks the install: an id already |
| 986 | registered by a different package, or a path a different package |
| 987 | already claims. Whichever installed last would otherwise silently win, |
| 988 | and the loser's menu entry would point somewhere its own package never |
| 989 | chose. |
| 990 | - Whether a path is actually **served** is NOT checked. Routing has three |
| 991 | sources (convention, `site_routes` records, `views` records) and two of |
| 992 | them are data the same install may be about to seed, so resolving a |
| 993 | route here would produce false blockers. The registry records the door |
| 994 | the package intends; `/urls` is where you go to see what the server |
| 995 | actually answers. |
| 996 | |
| 997 | If the `nav_entries` collection does not exist yet (app-nav is not |
| 998 | installed), nav entries are reported as `skipped` with a reason and the |
| 999 | install proceeds. A package must never fail because the navigation app is |
| 1000 | absent. The corollary is an ordering note for a fresh box: **install |
| 1001 | `app-nav` first**, or reinstall afterwards, since entries declared before |
| 1002 | the registry exists are skipped rather than queued. |
| 1003 | |
| 1004 | ### The opt-out |
| 1005 | |
| 1006 | `tests/test_package_nav.py` asserts that every package shipping a `site_*` |
| 1007 | object or a `site_routes` seed declares at least one nav entry. That test |
| 1008 | is the thing that stops the drift coming back. Some packages legitimately |
| 1009 | serve a URL that is not a door — a mounted widget (`site_thread`, |
| 1010 | `site_share`), a fragment inside another page |
| 1011 | (`site_materialize_run_button`), a POST endpoint (`site_timer_actions`), |
| 1012 | or a per-record document reached from the record itself |
| 1013 | (`site_packing_slip`, `site_receiving_sheet`, `site_return_form`). Those |
| 1014 | declare: |
| 1015 | |
| 1016 | ```json |
| 1017 | ""nav_optional"": true |
| 1018 | ``` |
| 1019 | |
| 1020 | which makes ""this app has no front page"" a reviewable claim in the |
| 1021 | manifest rather than an omission nobody notices. |
| 1022 | |
| 1023 | ## Attention (`attention`) |
| 1024 | |
| 1025 | A menu answers ""where can I go"". The only question a home screen is |
| 1026 | actually asked is **does anything need me?**, and this server is unusually |
| 1027 | well placed to answer it, because it is built almost entirely out of gates |
| 1028 | and derived states. Every one of them produces a queue of things a machine |
| 1029 | deliberately refused to decide: scans sitting at `extracted` waiting to be |
| 1030 | confirmed into expenses, time entries `submitted` waiting for an approver |
| 1031 | who is by design not the person who logged them, invoices past their due |
| 1032 | date, bank lines nothing matched, POs a supplier shorted, orders committed |
| 1033 | and not yet in a box, returns that arrived and were never judged. |
| 1034 | |
| 1035 | The system already computes all of it, on a schedule, and throws it away |
| 1036 | at the end of each pass. `system_scan_processor` has been returning a |
| 1037 | field literally named `needing_a_human` since the day it was written and |
| 1038 | no surface has ever read it. The arithmetic was never missing; a place to |
| 1039 | declare that the arithmetic *means* something was — and that declaration |
| 1040 | can only honestly live in the package that owns the domain, because |
| 1041 | nobody else knows that `extracted` is a waiting room and `confirmed` is |
| 1042 | not. |
| 1043 | |
| 1044 | So it is declared, third use of the pattern that already fixed cron |
| 1045 | (`schedules`) and navigation (`nav`): |
| 1046 | |
| 1047 | ```json |
| 1048 | ""attention"": [ |
| 1049 | {""id"": ""scans_to_confirm"", ""object_id"": ""system_scan_attention"", |
| 1050 | ""label"": ""Receipts to confirm"", ""path"": ""/scans?status=extracted"", |
| 1051 | ""nav_id"": ""scans"", ""group"": ""Money"", ""severity"": ""normal""} |
| 1052 | ] |
| 1053 | ``` |
| 1054 | |
| 1055 | - `id` becomes the `attention_sources` record id, so `[a-z][a-z0-9_]*`, |
| 1056 | and it must be unique across every installed package. |
| 1057 | - `object_id` names the provider (below). Required, and checked: a |
| 1058 | counter aimed at an object neither this package nor the server provides |
| 1059 | **blocks the install**, exactly as a schedule does. A counter pointing |
| 1060 | at nothing reads zero forever, and a queue that reads zero forever is |
| 1061 | indistinguishable from a queue that is empty — the silent failure this |
| 1062 | whole section exists to end. |
| 1063 | - `label` is what a person calls the queue, in their words: ""Receipts to |
| 1064 | confirm"", not ""scans.status=extracted"". |
| 1065 | - `path` is where the count clicks through to. Required, starts with `/`, |
| 1066 | no whitespace. **A number that cannot be acted on in one click does not |
| 1067 | belong on a home page.** Whether anything actually serves the path is |
| 1068 | not checked, for the same reason `nav` does not check it. |
| 1069 | - `nav_id` is optional: the `nav_entries` row this count decorates, so the |
| 1070 | app list itself reads ""Invoices — 5 overdue"" with no second surface to |
| 1071 | maintain. Omit it when the package ships no door at all (app-intake). |
| 1072 | - `group` is free text, default `Apps`, and uses the same house groups as |
| 1073 | `nav`. |
| 1074 | - `severity` is `normal` | `warning` | `urgent`, default `normal`, and is |
| 1075 | the first sort key on the home band. `normal` is work waiting; |
| 1076 | `warning` is work that is already late (an overdue invoice); `urgent` |
| 1077 | is something broken (a scheduler pass that failed). Three steps, |
| 1078 | because a scale with more is a scale nobody calibrates. |
| 1079 | |
| 1080 | ### The COUNT contract |
| 1081 | |
| 1082 | A provider is an ordinary object exposing: |
| 1083 | |
| 1084 | ```python |
| 1085 | def COUNT(request): |
| 1086 | return {""count"": 3, ""detail"": ""1 over a week old""} |
| 1087 | ``` |
| 1088 | |
| 1089 | `COUNT` and not `GET` or `POST`, deliberately: a distinct verb means the |
| 1090 | same object stays free to answer a human on the other methods without |
| 1091 | either use accidentally serving the other. `count` must be a real |
| 1092 | non-negative integer — the rollup refuses to coerce a string or a float, |
| 1093 | because every coercion here is a way for a broken provider to look like |
| 1094 | an empty queue. `detail` is optional and is the sentence a bare number |
| 1095 | cannot say (""62 hours awaiting approval"", ""18 units still owed""). |
| 1096 | |
| 1097 | A provider reads its own collections and **degrades to `{""count"": 0}` |
| 1098 | when they are absent** — a deployment that never installed your app |
| 1099 | should not produce an error every five minutes. A provider that genuinely |
| 1100 | fails should raise; see below for what happens then. |
| 1101 | |
| 1102 | ### Counts are a rollup, not a live fold |
| 1103 | |
| 1104 | `object_daemon.process_attention` executes every declared provider on an |
| 1105 | interval (`DBBASIC_ATTENTION_INTERVAL_SECONDS`, default 300) and writes |
| 1106 | the answers to the `attention_counts` collection. The home page and the |
| 1107 | app list read that one small table. |
| 1108 | |
| 1109 | This is not an optimisation, it is a rule. Folding a dozen collections on |
| 1110 | every home render is how a home page becomes the slowest page, and worse: |
| 1111 | one box went to 675MB resident and started swapping because something |
| 1112 | folded a big collection on a timer. A home screen that folds a dozen |
| 1113 | collections per render is that same mistake with a nicer name, and it |
| 1114 | makes the front page's cost grow with the number of installed packages — |
| 1115 | the one place cost must not grow. Staleness is bounded by the interval |
| 1116 | and **stated on the page** (""as of 6 minutes ago"") rather than pretended |
| 1117 | away. |
| 1118 | |
| 1119 | Two consequences worth knowing when you write a provider: |
| 1120 | |
| 1121 | - **A broken provider must look broken, never look calm.** One that |
| 1122 | raises, or answers with a non-integer count, has its `error` written |
| 1123 | and its *previous* count kept, with `computed_at` left where it was. |
| 1124 | Zeroing it would report an empty queue when what happened is that |
| 1125 | nobody looked. |
| 1126 | - **Do not do expensive work per row.** The pass runs every five minutes |
| 1127 | on the same box that serves requests. A full scan of your own |
| 1128 | collections is the intended cost; calling out to a network is not. |
| 1129 | |
| 1130 | ### Zeros disappear |
| 1131 | |
| 1132 | A count of zero is stored (the absence of a row means the provider has |
| 1133 | never run, which is a different fact) and **rendered nowhere**. When |
| 1134 | every queue is empty the home page's ""Needs you"" band is absent entirely |
| 1135 | — not ""0 pending"", not an empty panel with a heading: gone. |
| 1136 | |
| 1137 | This is the rule that separates an inbox from a dashboard. A board that |
| 1138 | says ""0 pending approvals"" every day trains people to stop reading it, |
| 1139 | and once they have stopped they do not start again on the day it says |
| 1140 | three. A page that is EMPTY when nothing needs them, and short when |
| 1141 | something does, gets read every time. ""Nothing needs you"" is a complete |
| 1142 | and welcome answer, and blank space gives it better than any sentence |
| 1143 | would. |
| 1144 | |
| 1145 | ### Idempotency and the operator |
| 1146 | |
| 1147 | The same three properties `nav` has: |
| 1148 | |
| 1149 | - **The operator's `operator_muted` survives.** A deployment that does |
| 1150 | not work a queue mutes it (a `role:manager` update on |
| 1151 | `attention_sources`) instead of editing somebody else's manifest, and |
| 1152 | an install never writes that column. A muted source is not executed at |
| 1153 | all and its stored count is removed, so it costs nothing and shows |
| 1154 | nothing. |
| 1155 | - **Unchanged means unwritten.** Every package-owned field — |
| 1156 | `object_id`, `label`, `path`, `nav_id`, `group`, `severity` — is |
| 1157 | compared before anything is written. |
| 1158 | - **Removing an entry does not delete the row.** Deregistering never |
| 1159 | destroys. |
| 1160 | |
| 1161 | A **collision with another package** on an `id` blocks the install: two |
| 1162 | packages writing one row is two definitions of what needs a human, and |
| 1163 | whichever installed last would silently win. Unlike `nav` there is no |
| 1164 | path-collision rule — two queues legitimately point at one list, and |
| 1165 | blocking that would be a false blocker. |
| 1166 | |
| 1167 | If the `attention_sources` collection does not exist (app-nav is not |
| 1168 | installed) the entries are reported as `skipped` with a reason and the |
| 1169 | install proceeds, exactly as `nav` entries are. |
| 1170 | |
| 1171 | ## Authoring Workflow |
| 1172 | |
| 1173 | 1. Create the package directory and manifest. |
| 1174 | 2. Write the objects (see [`object-authoring.md`](/docs/object-authoring) for method shape, state, logs, |
| 1175 | `request[""_identity""]`, and HTML form patterns). |
| 1176 | 3. Write each schema JSON (`{""fields"": [{""name"": ""id""}, ...]}` — see the |
| 1177 | schemas section of [`http-api-contract.md`](/docs/http-api-contract) for validation and field |
| 1178 | permission options). |
| 1179 | 4. Seed TSV: first row is the header, tab-separated, and must include `id`. |
| 1180 | 5. Dry-run before any install: |
| 1181 | |
| 1182 | ```http |
| 1183 | GET /packages/{package_id}?dry_run=true |
| 1184 | Authorization: Token <token> |
| 1185 | ``` |
| 1186 | |
| 1187 | The plan reports what would be created, replaced, merged, applied, or |
| 1188 | skipped, with warnings. Nothing is written. |
| 1189 | 6. Install (requires `DBBASIC_ENABLE_PACKAGE_INSTALLS=true` plus the admin |
| 1190 | gate): |
| 1191 | |
| 1192 | ```http |
| 1193 | POST /packages/{package_id}/install |
| 1194 | Authorization: Token <token> |
| 1195 | ``` |
| 1196 | |
| 1197 | 7. Verify: run the objects, read the records, check |
| 1198 | `GET /packages/{package_id}/changes` for the install changelog, and use |
| 1199 | `POST /packages/{package_id}/restore` if the install needs to be rolled |
| 1200 | back (requires `DBBASIC_ENABLE_PACKAGE_RESTORE=true`). |
| 1201 | |
| 1202 | ## A Minimal Working Example |
| 1203 | |
| 1204 | `packages/hello-world/` in this repository is the smallest complete package. |
| 1205 | `packages/admin-write-probe/` is the smallest complete *app-shaped* package: |
| 1206 | one HTML page object, one schema, one seed file, exercising records end to |
| 1207 | end. `packages/app-projects/` and `packages/app-notes/` are the reference |
| 1208 | *user apps*: an owner-scoped schema with `search` metadata, permission |
| 1209 | rules granting signed-in users their own rows (`row_filter` on |
| 1210 | `$user_id`), and a signed-in page that reads and writes records with the |
| 1211 | visitor's session cookie. Start a new app package from `app-notes`. |
| 1212 | |
| 1213 | `app-notes` also shows the public/private pattern: a boolean `is_public` |
| 1214 | field plus a second permission rule (`principal: public`, `row_filter: |
| 1215 | {""is_public"": ""true""}`) makes shared records readable by anyone across |
| 1216 | every surface — pages, records API, MCP, and search — with no |
| 1217 | visibility code in the app itself. Its permalink page (`site_note_view`) |
| 1218 | expects a `site_routes` record mapping `/notes/{note_id:uuid}` to it; |
| 1219 | route records are site data, so each deployment adds them next to its |
| 1220 | other routes. |
| 1221 | |
| 1222 | Object files inside packages are normal DBBASIC objects. A package page that |
| 1223 | renders per-user content reads `request[""_identity""]`; a package form posts to |
| 1224 | its own object id and writes records through the object runtime. Signed-in |
| 1225 | browser users carry their session cookie automatically. |
| 1226 | |
| 1227 | ## Rules For Generated Packages |
| 1228 | |
| 1229 | - Never put secrets, tokens, real hostnames, or private paths in package |
| 1230 | content — packages are meant to be shareable source. |
| 1231 | - Keep one package per app primitive; use `dependencies` to declare (not yet |
| 1232 | resolve) cross-package needs. |
| 1233 | - Bump `version` on any content change so install changelogs and restore |
| 1234 | points stay meaningful. |
| 1235 | - Prefer seed data that demonstrates the schema without pretending to be real |
| 1236 | data. |
| 1237 | - Dry-run output is the review artifact: a package that cannot explain itself |
| 1238 | in a dry-run plan is not ready to install. |
| 1239 | " |
| 1240 | schema-forms Schema Field Contract: Generated Forms and Views Building on DBBASIC 220 How one schema document drives desktop forms, web forms, MCP agents, and record-write validation from the same declared field semantics. "This is the contract that lets user interfaces generate themselves from schemas. One schema drives every surface at once: desktop forms, a server-rendered web form, an agent reading constraints over MCP, and the record-write APIs enforcing the same rules for all of them. |
| 1241 | |
| 1242 | The governing principle: **schemas declare semantics, never widgets.** A field says ""one of these values"" or ""points at a projects record"" — each interface maps that meaning onto its own idiom (a dropdown, a picker, a JSON constraint, a spoken prompt). New kinds of interface plug in later without touching the data. |
| 1243 | |
| 1244 | Everything here is enforced by the server on record writes today. None of it is a DSL — it is keys in the schema JSON document, checked by explicit code, versioned and rolled back through the normal schema history. |
| 1245 | |
| 1246 | ## Field Keys |
| 1247 | |
| 1248 | ```json |
| 1249 | { |
| 1250 | ""name"": ""urgency"", |
| 1251 | ""type"": ""enum"", |
| 1252 | ""required"": true, |
| 1253 | ""label"": ""Urgency"", |
| 1254 | ""help"": ""How soon this needs attention"", |
| 1255 | ""placeholder"": ""Choose…"", |
| 1256 | ""default"": ""normal"", |
| 1257 | ""enum"": [""low"", ""normal"", ""high"", ""critical""] |
| 1258 | } |
| 1259 | ``` |
| 1260 | |
| 1261 | - `name` — required; letters/digits/underscores. |
| 1262 | - `type` — `text` (default), `textarea`, `integer`, `number`, `boolean`, `date`, `datetime`, `enum`, `computed`. Types are validated on write (integers must parse, dates must be ISO, enums must match, and so on). Boolean fields accept several spellings on input (`True`, `1`, `yes`) but are stored canonically as `true`/`false`, so permission row filters like `{""is_public"": ""true""}` match reliably. |
| 1263 | - `required` — empty values rejected on write. |
| 1264 | - `label`, `help`, `placeholder` — presentation text for generated forms. |
| 1265 | - `default` — filled in when the field is not submitted. |
| 1266 | - `enum` — allowed values; writes outside the list are rejected. Renders as a dropdown/segment control in form UIs. |
| 1267 | - `computed` / `read_only` — the server rejects client writes to these fields; forms render them display-only. |
| 1268 | - `transitions` — for lifecycle fields, which values each current value may move to. Enforced on record update; values missing from the map are terminal. This is data plus one check, not a state machine framework — no hooks, no callbacks, no side effects: |
| 1269 | |
| 1270 | ```json |
| 1271 | {""name"": ""status"", ""type"": ""enum"", |
| 1272 | ""enum"": [""open"", ""assigned"", ""done""], |
| 1273 | ""transitions"": {""open"": [""assigned""], ""assigned"": [""done"", ""open""]}} |
| 1274 | ``` |
| 1275 | |
| 1276 | Form UIs can offer only the moves allowed from the record's current value; the server rejects everything else either way. |
| 1277 | |
| 1278 | A list entry can also be a guarded object instead of a plain string, adding *who* on top of *whether*: |
| 1279 | |
| 1280 | ```json |
| 1281 | {""transitions"": { |
| 1282 | ""assigned"": [{""to"": ""open"", ""when"": {""assigned_to"": ""$user_id""}}] |
| 1283 | }} |
| 1284 | ``` |
| 1285 | |
| 1286 | The move is allowed only once every `when` clause matches the record's current stored values against the resolved subject variable (`$user_id`, `$account_id`, `$accessible_projects`, `$owned_projects`, `$writable_projects`) or a literal string — the same closed set and matching rules row filters use (see `permissions-model.md`). Guards are only evaluated where a request subject is available (the HTTP update path); direct library callers still get the plain validity check (the move's `to` must be in the list) but not the guard. |
| 1287 | - `relation` — a validated pointer to another collection: |
| 1288 | |
| 1289 | ```json |
| 1290 | {""name"": ""project_id"", ""relation"": {""collection"": ""projects"", ""display_field"": ""name""}} |
| 1291 | ``` |
| 1292 | |
| 1293 | On write, the value must be an existing record id in that collection (`projects/p999` missing → 400). `display_field` tells form UIs which field of the target record to show in pickers. A bare string (`""relation"": ""projects""`) is shorthand. This is deliberately a pointer plus a display hint, not an association framework — no joins, no lazy loading, no cascades. |
| 1294 | - `validation` — explicit bounds, all enforced on write: |
| 1295 | |
| 1296 | ```json |
| 1297 | {""validation"": {""min_length"": 3, ""max_length"": 80, ""pattern"": ""^[a-z-]+$"", ""min"": 0, ""max"": 100}} |
| 1298 | ``` |
| 1299 | |
| 1300 | `pattern` is **full-matched** against the whole value (`re.fullmatch`), not searched — so it must describe the entire field, not a prefix. A URL check is `^https?://\S+$`, not `^https?://` (that would only match the literal scheme and reject every real URL). The `^`/`$` anchors are therefore redundant but fine to keep for readability. |
| 1301 | |
| 1302 | ## Schema Root Keys |
| 1303 | |
| 1304 | Presentation metadata lives beside the fields and versions with them — a ""Save Layout"" action in a desktop client persists here, so form design has history and rollback like source code: |
| 1305 | |
| 1306 | ```json |
| 1307 | { |
| 1308 | ""fields"": [...], |
| 1309 | ""forms"": { |
| 1310 | ""default"": {""fields"": [""title"", ""description"", ""project_id"", ""urgency"", ""due_date"", ""assigned_to""]} |
| 1311 | }, |
| 1312 | ""views"": { |
| 1313 | ""list_mode"": ""table"", |
| 1314 | ""list_fields"": [""title"", ""urgency"", ""due_date"", ""assigned_to""] |
| 1315 | } |
| 1316 | } |
| 1317 | ``` |
| 1318 | |
| 1319 | - `forms.default.fields` — field order for the generated record form. Additional named forms can define alternate layouts. |
| 1320 | - `views.list_mode` — how the generated list renders. The shared list renderer (`window.dbbasicList`) resolves it from the schema: |
| 1321 | - `table` — a dense, sortable HTML table over `list_fields`; cells format by type (money `_cents`, boolean, relative dates, enum badges), and relation columns show the target's label, not the raw id. |
| 1322 | - `board` — a kanban grouped by an enum field, drag-to-transition (the drag issues the ordinary status write, so `transitions` rules still gate it). A board collection that also has `list_fields` gets a **Board ⇄ Table** toggle (the choice persists per collection). |
| 1323 | - `tree` — nests a self-relation (`parent_id`) into a hierarchy. |
| 1324 | - `calendar` — buckets by a date field. |
| 1325 | - absent / `cards` / `feed` — the plain rich-row list. |
| 1326 | |
| 1327 | Every mode shares one fetch/sort/cap/search/realtime pipeline, so all of them inherit filtering, the 50-row cap, and live updates. A mode whose required field can't be derived falls back to the row list with a visible notice — never a blank page. |
| 1328 | - `views.list_fields` — columns (table), card fields (board), or summary fields (rows). Relation fields resolve to the referenced record's label. |
| 1329 | - `views.filter_fields` — fields to expose as a **filter bar** above the list (enum → a select of its options, boolean → Yes/No). A pick narrows the fetch server-side (`field=value`, after the permission row filter) and composes with the search box; works in every mode. |
| 1330 | - `search` — opts the collection into global search (`GET /api/search` and the `global_search` MCP tool): |
| 1331 | |
| 1332 | ```json |
| 1333 | {""search"": {""fields"": [""title"", ""content""], ""result_fields"": [""id"", ""title""]}} |
| 1334 | ``` |
| 1335 | |
| 1336 | `fields` are the searchable fields; `result_fields` (optional) trims what each search hit returns, defaulting to `id` plus `views.list_fields`. Collections without a `search` section never appear in search results. Search runs inside the permission engine, so row filters and field permissions bound what any caller can find. |
| 1337 | - `formula` / `rollup` (on a `type: computed` field) — derived values, materialized on write: a formula computes from sibling fields (`""first_name + \"" \"" + last_name""`); a rollup aggregates a child collection (`{""collection"": ""fin_journal_lines"", ""fk_field"": ""journal_id"", ""op"": ""sum"", ""field"": ""debit_cents""}`). Stored like any field, so every view/filter/sort sees them; kept current by the write path. See `validation-and-logic.md`. |
| 1338 | - `capabilities` — generic per-collection *behaviors* on top of this display layer: `{""comments"": true}`, `{""attachments"": true}`, `{""shareable"": true}` grow a comment thread, attachment list, and owner-checked sharing on the detail page from one flag each. See `capabilities.md`. |
| 1339 | |
| 1340 | ## Worked Example: tasks |
| 1341 | |
| 1342 | The acceptance test for this contract is a ""New Task"" screen, expressed entirely as data: |
| 1343 | |
| 1344 | ```json |
| 1345 | { |
| 1346 | ""fields"": [ |
| 1347 | {""name"": ""id""}, |
| 1348 | {""name"": ""title"", ""type"": ""text"", ""required"": true, ""label"": ""Title"", |
| 1349 | ""validation"": {""min_length"": 1, ""max_length"": 200}}, |
| 1350 | {""name"": ""description"", ""type"": ""textarea"", ""label"": ""Description""}, |
| 1351 | {""name"": ""project_id"", ""label"": ""Project"", |
| 1352 | ""relation"": {""collection"": ""projects"", ""display_field"": ""name""}}, |
| 1353 | {""name"": ""urgency"", ""type"": ""enum"", ""label"": ""Urgency"", ""default"": ""normal"", |
| 1354 | ""enum"": [""low"", ""normal"", ""high"", ""critical""]}, |
| 1355 | {""name"": ""due_date"", ""type"": ""date"", ""label"": ""Due Date""}, |
| 1356 | {""name"": ""assigned_to"", ""label"": ""Assigned To"", |
| 1357 | ""relation"": {""collection"": ""users"", ""display_field"": ""display_name""}} |
| 1358 | ], |
| 1359 | ""forms"": {""default"": {""fields"": [""title"", ""description"", ""project_id"", |
| 1360 | ""urgency"", ""due_date"", ""assigned_to""]}}, |
| 1361 | ""views"": {""list_mode"": ""table"", |
| 1362 | ""list_fields"": [""title"", ""urgency"", ""due_date"", ""assigned_to""]} |
| 1363 | } |
| 1364 | ``` |
| 1365 | |
| 1366 | A form renderer walks `forms.default.fields`, looks up each field's semantics, and emits the right control. The record POST enforces every rule the form promised — and the same rules bind a record created over MCP or raw HTTP, so no surface can bypass what another surface displays." |
| 1367 | business-logic-patterns Business Logic Patterns: Where Every Rule Goes Building on DBBASIC 230 A placement guide and decision procedure for where each kind of business rule belongs in the primitive stack, illustrated with worked examples from tax, aging, dunning, refunds, inventory, and bank reconciliation. "Every business rule has one correct home in the primitive stack. This page is the placement guide: the decision procedure, the map, and worked examples (sales tax, payment aging, dunning, refunds, inventory shrinkage, bank reconciliation) that most systems get wrong. |
| 1368 | |
| 1369 | The method behind all of it: build each module's rules **concretely** in the cheapest correct primitive — a hook is three lines of real code — and only promote a pattern to a declarative schema key after it has repeated in two or three real modules. Concrete → pattern → primitive, never primitive-first. Systems that build the workflow/rules engine first die when reality doesn't fit it; this one extracts engines from reality. |
| 1370 | |
| 1371 | ## Where does this rule go? |
| 1372 | |
| 1373 | ```mermaid |
| 1374 | flowchart TD |
| 1375 | Q[""a business rule""] --> D1{""Derivable from<br/>stored data?""} |
| 1376 | D1 -->|same record| F[""formula field""] |
| 1377 | D1 -->|child records| R[""rollup field""] |
| 1378 | D1 -->|no| D2{""Does it GATE<br/>a write?""} |
| 1379 | D2 -->|shape / bounds / FK| V[""schema validation""] |
| 1380 | D2 -->|legal state moves + who| T[""transitions + when guards""] |
| 1381 | D2 -->|cross-record or computed gate| H[""pre-write hook<br/>(reject or stamp)""] |
| 1382 | D2 -->|no| D3{""Does TIME<br/>cause it?""} |
| 1383 | D3 -->|yes| DM[""daemon: scheduled pass /<br/>auto-transition""] |
| 1384 | D3 -->|no| D4{""Is it a REACTION<br/>to a change?""} |
| 1385 | D4 -->|notify / email| N[""notify_rules (after write)""] |
| 1386 | D4 -->|run code| TR[""trigger → target object""] |
| 1387 | D4 -->|external system| C[""connector<br/>(desired state + reconcile)""] |
| 1388 | D4 -->|no| O[""plain object<br/>(page / API / worker)""] |
| 1389 | ``` |
| 1390 | |
| 1391 | ## The placement map |
| 1392 | |
| 1393 | | Kind of rule | Primitive | Lives in | Example | |
| 1394 | |---|---|---|---| |
| 1395 | | Shape, type, bounds, enum, FK-exists | schema validation | `object_records` write path | amount is integer cents; project must exist | |
| 1396 | | Same-record derivation | `formula` (materialized) | `object_computed` via `object_records` | `tax_cents = amount_cents * tax_rate` | |
| 1397 | | Aggregation over children | `rollup` (materialized, live) | same | `invoice.paid_cents = sum(payments)` | |
| 1398 | | Legal state moves, and who may move | `transitions` + `when` | `object_records` | `draft → posted`, owner only | |
| 1399 | | Cross-record gate, computed gate, point-in-time stamp | **pre-write hook** | `hooks.before_write` → `object_server` write path | journal must balance; stamp `tax_rate` | |
| 1400 | | Time-driven state | daemon scheduled pass / `after_hours` auto-transition | `object_daemon` | `sent → overdue` when past due | |
| 1401 | | Reaction: notify, email | `notify_rules` | daemon, **after** the write | dunning email on `overdue` | |
| 1402 | | Reaction: run code | trigger → target object | daemon | recompute, follow-on records | |
| 1403 | | External side effect | connector | desired-state records + reconcile | payment-provider refund call | |
| 1404 | | Correction to money / inventory | append-only compensating record | `storage: append` collections | refunds, stock moves, journal reversals | |
| 1405 | | Everything else | a plain object | anywhere | bespoke report, integration, one-off page | |
| 1406 | |
| 1407 | Two hard boundaries hold the map together: |
| 1408 | |
| 1409 | - **Gates before the write; reactions after.** A hook may reject or stamp but never send email or call out — a gate with side effects leaks that a write was attempted and couples saves to SMTP. A reaction may notify or spawn work but never block the write that already happened. |
| 1410 | - **Derived values are non-authoritative; gates are authoritative.** A broken formula stores `""""` and the write succeeds; a broken hook fails closed and the write is rejected. Never let a gate read only a derived value it could recompute — the journal-balance hook sums the lines itself even though rollup totals exist. |
| 1411 | |
| 1412 | ## Worked example 1: sales tax (the one everyone leaves out) |
| 1413 | |
| 1414 | Why it's left out: it hides a doctrine most systems never articulate — **stamp point-in-time facts; derive live facts**. The tax *rate* is a fact about the transaction moment: rates change, and the invoice must forever carry the rate it was issued under. |
| 1415 | |
| 1416 | 1. **Rates are data**: a `tax_rates` collection — `region`, `category`, `rate`, `valid_from`. Editable like any records; history visible. |
| 1417 | 2. **Stamping is a hook**: `hooks.before_write` on invoice lines looks up the applicable rate and *stamps* `tax_rate` onto the line — same posture as the server stamping `owner_id`. The client never supplies it; later rate changes never rewrite history. |
| 1418 | 3. **Amounts are formulas/rollups**: `line.tax_cents = amount_cents * tax_rate` (formula); `invoice.tax_total_cents` (rollup). Materialized, so they appear on every list/detail/report and update live. |
| 1419 | |
| 1420 | ## Worked example 2: payment aging (time makes the state) |
| 1421 | |
| 1422 | The trap: an invoice becomes overdue with **no write happening** — time did it. Any system that computes ""is it late?"" ad hoc in the UI grows a dozen disagreeing copies of that check. |
| 1423 | |
| 1424 | 1. `balance_cents` is a **rollup** (`total_cents` minus `sum(payments)`). |
| 1425 | 2. `sent → overdue` is a **daemon scheduled pass** through the ordinary transition machinery, when `due_date < today` and `balance_cents > 0` — so the state is *stored*, permission-checked, realtime-pushed, and every surface agrees (time-driven state belongs to the daemon, never to read-time checks). |
| 1426 | 3. Aging buckets (30/60/90) are a rollup/report over stored state. |
| 1427 | |
| 1428 | ## Worked example 3: dunning (reactions compose on state) |
| 1429 | |
| 1430 | Dunning is nothing new — it is the composition of the two layers above: |
| 1431 | |
| 1432 | 1. The daemon flips the state (`overdue`, later `overdue_30`, …). |
| 1433 | 2. `notify_rules` reacts: state change → email template to the invoice's contact. The rule is data; the daemon delivers after the write. |
| 1434 | 3. Escalation (""again at 30 days unless paid"") is *more time-driven state*, not a special notification feature — each escalation step is a transition the daemon makes, each with its own notify rule. |
| 1435 | |
| 1436 | ## Worked example 4: refunds (money moves, never mutates) |
| 1437 | |
| 1438 | **Money never mutates; it moves.** A refund is not an edit to a payment — it is a new, append-only compensating record, exactly like a stock move or a journal reversal. |
| 1439 | |
| 1440 | 1. `refunds` is a `storage: append` collection pointing at the payment. |
| 1441 | 2. The invariant is a **hook**: `refund ≤ payment − prior refunds` — a cross-record gate, summed from the records themselves. |
| 1442 | 3. `payment.refunded_cents` / `net_cents` are **rollup/formula** — display truth, derived. |
| 1443 | 4. ""Correctly credited"" is the balanced journal entry the refund composes — enforced by the journal-balance hook that already ships. The chain closes: append-only record → hook invariant → rollup display → balanced books. |
| 1444 | |
| 1445 | ## Worked example 5: inventory shrinkage (the events nobody codes for) |
| 1446 | |
| 1447 | Waste, breakage, theft, expiry, damage, disaster: systems are easy to build when quantities just flow buy → stock → sell; real usage is losses. Naming them is the point — ""adjustment"" is the euphemism that hides exactly what operators and auditors need to see. |
| 1448 | |
| 1449 | 1. **Losses are movements**: a loss is an append-only stock move with a loss `reason`, out of a location, to nowhere — never an edit to an on-hand number (levels stay derived). |
| 1450 | 2. **The hook stamps and gates**: `unit_cost_cents` is stamped from the product at the moment of movement (tomorrow's cost edit must not reprice yesterday's loss); a loss must leave a real location and can have no destination. |
| 1451 | 3. **The composer books the shadow**: a loss move composes a posted journal — DR shrinkage-expense / CR inventory-asset — with per-reason account overrides as settings (theft to its own account for the insurance claim). |
| 1452 | 4. **Count reconciliation is the same machinery**: a physical count writes one compensating adjustment move for the variance (out for shortage, in for overage) plus the mirror journal. The count is an *event*, not an edit. |
| 1453 | |
| 1454 | This was also the composer pattern's third repeat, which triggered the extraction of a shared composer: every generated journal now goes through `object_finance.compose_posted_journal` (idempotent by `generated_from`, verify-then-post), and the payment/refund/recurring/reversal composers shrank onto it. |
| 1455 | |
| 1456 | ## Worked example 6: bank reconciliation (evidence, not data entry) |
| 1457 | |
| 1458 | Comparing the books to a bank statement is the strongest practical control a small business has, and the reason is not arithmetic — it is that the bank's copy was written by someone with no stake in the numbers. Every design choice follows from protecting that independence. |
| 1459 | |
| 1460 | 1. **Imported lines are append-only and keep their original text** in `raw`; parsed fields are a view of it. A pre-write hook makes the factual fields immutable on update — corrections come from re-importing a corrected statement, never from editing a line to agree with the books. |
| 1461 | 2. **The statement checks itself.** Opening + lines = closing (tie-out), and this opening = the last statement's closing (continuity). Both run at import and *set* the import's status; a truncated or edited file fails arithmetic. The flags record which checks **ran**, so a balance-less CSV never reads as trustworthy as a verified one. |
| 1462 | 3. **Reading a bank's format is data**, not code: a saved column map, authored once with AI help and executed deterministically thereafter. AI at configuration time, code at run time — a control cannot be nondeterministic. |
| 1463 | 4. **The matcher proposes; a person disposes.** Suggestions are derived and non-authoritative; confirming is an ordinary attributed update, so the change log names who reconciled what. Only a two-signal match (reference *and* amount) is auto-confirmable, and even that is configurable to off. |
| 1464 | 5. **The unmatched tail is where the business logic lives**: fees, interest, own-account transfers, NSF bounces — each a resolution verb composing a posted journal through the shared composer, stamped with the line's id so resolving twice books once. These are exactly the entries small books never record, which is why their cash balance drifts until they stop reconciling at all. |
| 1465 | |
| 1466 | ## State-dependent behavior generally |
| 1467 | |
| 1468 | `transitions` + `when` guards define the machine; the board's drag is its UI; hooks gate specific moves with computed conditions (the balance check gates `draft → posted`); the daemon makes time-based moves. The known missing declarative is **""state freezes fields""** (a posted invoice's amounts are immutable). Per the method: express it in each module's hook first; promote it to a schema key (`locked_when`) only after the shape repeats. |
| 1469 | |
| 1470 | ## What this implies we build next |
| 1471 | |
| 1472 | The examples above want one substrate piece that doesn't exist yet: a **payments collection** (plain records — provider integration is a connector later). It unblocks aging, dunning, and refunds at once. Then: `tax_rates` + the stamping hook; then a delay/repeat notion for `notify_rules` if escalation schedules outgrow ""each step is its own transition."" |
| 1473 | " |
| 1474 | validation-and-logic Validation and Business Logic Building on DBBASIC 240 How DBBASIC Object Server enforces field validation and runs business rules from the schema alone, with no per-app code required. "Two things a schema-driven system has to get right without per-app code: |
| 1475 | rejecting bad writes, and running the business rules a record's lifecycle |
| 1476 | implies. DBBASIC does both from the schema, enforced on the server — the form |
| 1477 | is a convenience, not the gate. |
| 1478 | |
| 1479 | ## Validation — declared once, enforced on every write |
| 1480 | |
| 1481 | The write path (create and update) validates each field against the schema, |
| 1482 | server-side, regardless of which surface issued the write (web form, API, MCP |
| 1483 | agent, AI). A form's client-side checks are a nicety on top; they are not what |
| 1484 | protects the data. |
| 1485 | |
| 1486 | What is enforced today: |
| 1487 | |
| 1488 | | Check | Declared as | Enforced | |
| 1489 | |-------|-------------|----------| |
| 1490 | | Type parses | `type` (integer, number, date, datetime, boolean, enum) | Yes | |
| 1491 | | Required present | `required: true` | Yes | |
| 1492 | | Enum membership | `enum: [...]` | Yes (value must be in the list) | |
| 1493 | | Length bounds | `validation.min_length` / `max_length` | Yes | |
| 1494 | | Pattern | `validation.pattern` (regex) | Yes | |
| 1495 | | Referential integrity | `relation: {collection}` | Yes (id must exist in the target) | |
| 1496 | | State transition | field `transitions` map / `flow` | Yes (status can only move where allowed) | |
| 1497 | | Read-only / computed | `read_only` / `type: computed` | Yes (client value rejected) | |
| 1498 | | Storable in TSV | (substrate) | Yes (size, no NUL/control chars) | |
| 1499 | |
| 1500 | Referential integrity and enforced state transitions are the notable ones — |
| 1501 | most no-code tools skip both. A relation is a *validated pointer*: you cannot |
| 1502 | point a `project_id` at a project that doesn't exist. A status field with a |
| 1503 | `transitions` map is a state machine: an order can go `open → cancelled` only |
| 1504 | if the schema says so. |
| 1505 | |
| 1506 | ### Known gaps |
| 1507 | |
| 1508 | - **Numeric range** — we bound string length, not numeric value (a price can go |
| 1509 | negative). A `min`/`max` value rule is the fix. |
| 1510 | - **Uniqueness** — no unique constraint (e.g. unique email). |
| 1511 | - **Declarative cross-field rules** — no formula language yet for |
| 1512 | ""end_date > start_date"" (`visible_when` handles conditional *visibility*, |
| 1513 | not validation). The general mechanism exists, though: a **pre-write hook** |
| 1514 | (below) expresses any cross-field or cross-collection rule in real code. A |
| 1515 | Salesforce-style declarative rule (formula + message) remains the ergonomic |
| 1516 | follow-on for the common cases. |
| 1517 | - **Inline client feedback** — the form pre-checks `required`; length/pattern |
| 1518 | errors currently surface after a server round-trip, and messages are |
| 1519 | technical (""longer than max_length""). Friendlier, inline feedback is a |
| 1520 | UX polish item, not a safety gap. |
| 1521 | |
| 1522 | ## Pre-write hooks — code inside the write path |
| 1523 | |
| 1524 | For rules the schema can't express, a collection declares a **hook object**: |
| 1525 | |
| 1526 | ```json |
| 1527 | { ""name"": ""fin_journals"", ""hooks"": { ""before_write"": ""hook_fin_journals"" } } |
| 1528 | ``` |
| 1529 | |
| 1530 | The object's `BEFORE_WRITE(request)` runs **synchronously inside the generic |
| 1531 | HTTP write path** — after permission checks, before persist — for every public |
| 1532 | create/update on that collection. `request` carries `{collection, action, |
| 1533 | record, existing, changes, subject}`. It returns `None` to allow, |
| 1534 | `{""error"": ""..."", ""status"": 4xx}` to reject with its own message, or |
| 1535 | `{""record"": {...}}` to transform (the transformed record still passes full |
| 1536 | schema validation, and can never touch `id`, `owner_id`, or read-only/computed |
| 1537 | fields). |
| 1538 | |
| 1539 | The contract is deliberately strict: |
| 1540 | |
| 1541 | - **Fail closed** — a declared hook that is missing, raises, or returns a |
| 1542 | non-contract shape rejects the write. A gate that silently passes on error |
| 1543 | is not a gate. |
| 1544 | - **After permissions** — a denied subject never reaches the hook. |
| 1545 | - **Opt-in per collection** — collections without `hooks` pay one cached dict |
| 1546 | lookup; the original latency concern that deferred this feature stays |
| 1547 | bounded. |
| 1548 | - **Public surface only** — trusted server-side writers (the daemon, |
| 1549 | migrations, seed-merge) call the storage layer directly and bypass hooks. |
| 1550 | |
| 1551 | This is what keeps the generative form working when custom logic arrives: the |
| 1552 | form still POSTs to `/collections/{c}/records`; the rule lives server-side. |
| 1553 | First adopter: `fin_journals` refuses to post a journal whose lines don't |
| 1554 | balance — a cross-collection rule (it sums `fin_journal_lines`) no schema key |
| 1555 | could express. |
| 1556 | |
| 1557 | ## Business logic — the automation substrate |
| 1558 | |
| 1559 | Business rules run without per-app code through a set of declarative |
| 1560 | primitives, driven by a background daemon and the change log: |
| 1561 | |
| 1562 | ```mermaid |
| 1563 | flowchart LR |
| 1564 | W[""record write""] --> CL[""change log""] |
| 1565 | CL --> DAEMON[""daemon: poll triggers""] |
| 1566 | DAEMON --> NOTIFY[""notify_rules → notification / email""] |
| 1567 | DAEMON --> TARGET[""trigger object → target object""] |
| 1568 | W --> TRANS[""transition guards (flow: when clauses)""] |
| 1569 | SCHED[""schedule""] --> DAEMON |
| 1570 | DAEMON --> COMPUTE[""computed / materialize fields""] |
| 1571 | DAEMON --> CONN[""connectors → external systems + reconcile""] |
| 1572 | ``` |
| 1573 | |
| 1574 | - **Transitions with guards** — a `flow`/`transitions` map is a conditional |
| 1575 | state machine; a `when` clause gates whether a move is allowed. The board's |
| 1576 | drag and every status write run through it. |
| 1577 | - **notify_rules** — declarative ""a record changed → send a notification / |
| 1578 | email."" Data, not code. |
| 1579 | - **Trigger → target objects** — the daemon polls trigger objects and executes |
| 1580 | target objects: an event-handler path built on top of the daemon and its |
| 1581 | scheduled passes. |
| 1582 | - **Computed / materialized fields** — derived values (`type: computed`, |
| 1583 | `materialize_definitions`). |
| 1584 | - **Connectors** — external side effects expressed as desired-state records |
| 1585 | with a reconcile pass. |
| 1586 | - **Scheduled passes** — the daemon runs periodic work (compaction, |
| 1587 | auto-transitions). |
| 1588 | |
| 1589 | This is the same family as Salesforce flows/validation-rules, Hasura event |
| 1590 | triggers, and Directus Flows — declarative automation over a data store, here |
| 1591 | on plain-text records. |
| 1592 | |
| 1593 | ### Formula & rollup fields — shipped |
| 1594 | |
| 1595 | The spreadsheet feature, on collections. A `type: computed` field declares |
| 1596 | either a **formula** over sibling fields or a **rollup** over a child |
| 1597 | collection: |
| 1598 | |
| 1599 | ```json |
| 1600 | {""name"": ""full_name"", ""type"": ""computed"", |
| 1601 | ""formula"": ""first_name + \"" \"" + last_name""} |
| 1602 | |
| 1603 | {""name"": ""debit_total_cents"", ""type"": ""computed"", |
| 1604 | ""rollup"": {""collection"": ""fin_journal_lines"", ""fk_field"": ""journal_id"", |
| 1605 | ""op"": ""sum"", ""field"": ""debit_cents""}} |
| 1606 | ``` |
| 1607 | |
| 1608 | Values are **materialized on write at the storage layer** — stored in the TSV, |
| 1609 | recomputed whenever inputs change (formulas on writes of the record itself; |
| 1610 | rollups when source-collection records are created, updated, or deleted — |
| 1611 | single-hop, loop-guarded). Because the value is stored, every surface works |
| 1612 | with zero changes: lists, tables, detail, search, `field=value` filters, sort, |
| 1613 | CSV, backups — and **realtime falls out free**: edit a journal line and the |
| 1614 | journal's totals update live in every open browser. `rollup.op` is |
| 1615 | `sum | count | min | max | avg`, with an optional equality `where`. The |
| 1616 | formula language is a tiny safe expression grammar (fields, strings, numbers, |
| 1617 | `+ - * /`, parens — never a general-purpose `eval`). A broken formula stores an |
| 1618 | empty value and never fails the write — derived values are non-authoritative, |
| 1619 | the opposite posture of hooks. Definition changes are backfilled explicitly |
| 1620 | with `object_records.recompute_computed_fields(collection)` (install step, |
| 1621 | admin action, or cron — never a runtime loop). |
| 1622 | |
| 1623 | **Declarative validation rules** (cross-field formula + message) remain the |
| 1624 | natural next sugar over the hook. |
| 1625 | |
| 1626 | ## Related |
| 1627 | |
| 1628 | - **Placement of rules** — a companion guide walks through *where* each kind |
| 1629 | of rule goes (the placement flowchart) with worked examples, and the |
| 1630 | doctrine behind those placements. |
| 1631 | - **Field contract** — the schema field reference these checks read against. |
| 1632 | - **Runtime contract** — the daemon, triggers, schedules, and event contracts |
| 1633 | in full. |
| 1634 | - **Storage modes** — append-only collections, the immutable substrate for |
| 1635 | audit-shaped data (journals, stock moves). |
| 1636 | " |
| 1637 | capabilities Capabilities: Per-Collection Behaviors, Declared Not Coded Building on DBBASIC 250 How the platform's capability layer lets any collection opt into comments, attachments, and sharing by declaring a flag instead of hand-building each behavior per app. "The schema-driven UI has two layers. The **display** layer turns a schema into |
| 1638 | a list, table, board, form, and detail page (see |
| 1639 | [`schema-forms.md`](/docs/schema-forms)). The **capability** layer sits on top of |
| 1640 | it: generic *behaviors* a collection opts into by declaring a flag, wired by the |
| 1641 | platform instead of hand-built per app. |
| 1642 | |
| 1643 | A collection declares them under a root-level `capabilities` key: |
| 1644 | |
| 1645 | ```json |
| 1646 | { |
| 1647 | ""name"": ""tasks"", |
| 1648 | ""capabilities"": { ""comments"": true, ""attachments"": true, ""shareable"": true }, |
| 1649 | ""fields"": [ ... ] |
| 1650 | } |
| 1651 | ``` |
| 1652 | |
| 1653 | and its detail page grows a comment thread, an attachment list, and a share |
| 1654 | control — with no per-view block and no per-app table. The same three flags on |
| 1655 | `contacts`, `invoices`, or any other collection give the same behaviors. |
| 1656 | |
| 1657 | This is the ""define once, project everywhere"" rule applied to what a record |
| 1658 | *does* and *connects to*, not just how it looks. It replaces the pattern where |
| 1659 | every app hand-rolled its own version of the same idea: comments existed four |
| 1660 | separate times (`task_comments`, `thread_comments`, `profile_comments`, |
| 1661 | `interactions`) before one `capabilities.comments` flag collapsed them. |
| 1662 | |
| 1663 | ## The model |
| 1664 | |
| 1665 | Each capability is one **polymorphic collection** (a row attaches to a record in |
| 1666 | *any* collection via a `parent_collection` + `parent_id` pair), one **widget** |
| 1667 | served as a static script like `/list` and `/form`, and one line in the detail |
| 1668 | renderer that mounts the widget when the flag is present. |
| 1669 | |
| 1670 | ```mermaid |
| 1671 | flowchart TD |
| 1672 | S[""schema.capabilities flag""] --> API[""/api/schema exposes it""] |
| 1673 | API --> VR[""view_render: maybeMountCapabilities()""] |
| 1674 | VR -->|flag present| W[""window.dbbasicThread / Attachments / Share .mount()""] |
| 1675 | W --> C[""polymorphic collection<br/>parent_collection + parent_id""] |
| 1676 | C -->|permission-checked read and write| STORE[""object store""] |
| 1677 | D[""detail block""] --> VR |
| 1678 | ``` |
| 1679 | |
| 1680 | The wiring is one function, `maybeMountCapabilities` in |
| 1681 | `app-views/objects/site/view_render.py`: it fetches the record's schema, reads |
| 1682 | `capabilities`, and mounts each declared widget into its own sub-element under |
| 1683 | the detail. Absent flag or unloaded widget → nothing renders, never an error. |
| 1684 | |
| 1685 | `capabilities` is whitelisted in schema normalization |
| 1686 | (`object_schemas.py`) and surfaced by `/api/schema` so the client can wire it — |
| 1687 | the same ""carried, not interpreted"" posture as `flow` and `blocks`. |
| 1688 | |
| 1689 | ## The three capabilities |
| 1690 | |
| 1691 | | Flag | Widget (route) | Backing collection | Package | |
| 1692 | |------|----------------|--------------------|---------| |
| 1693 | | `comments` | `window.dbbasicThread` (`/thread`) | `thread_comments` | `app-thread` | |
| 1694 | | `attachments` | `window.dbbasicAttachments` (`/attachments`) | `files` (+`parent_*`) | `app-files` | |
| 1695 | | `shareable` | `window.dbbasicShare` (`/share`) | `record_shares` | `app-share` | |
| 1696 | |
| 1697 | ### comments |
| 1698 | |
| 1699 | A comment thread for one record. Reads/writes `thread_comments` filtered to the |
| 1700 | `(parent_collection, parent_id)` pair, posts with a minimal payload (the server |
| 1701 | owns `created_at`; `owner_id` is set from the session and redacted from other |
| 1702 | readers, so each comment also stamps a display `author_name`), owner-only |
| 1703 | delete, and re-renders over the change log like every other surface. |
| 1704 | |
| 1705 | ### attachments |
| 1706 | |
| 1707 | An upload/list/download/delete surface for files attached to one record. |
| 1708 | Uploads go multipart to `/api/files` — which owns `owner_id`/`size`/ |
| 1709 | `content_type` and saves the blob — carrying `parent_collection`/`parent_id` so |
| 1710 | the file lands on this record. Delete is blob-aware (`DELETE /api/files/{id}` |
| 1711 | removes the bytes *and* the metadata row; a plain record delete would orphan the |
| 1712 | blob). No per-collection foreign key on `files` is needed. |
| 1713 | |
| 1714 | ### shareable |
| 1715 | |
| 1716 | The foundational one — it changes the **permission check**, not just a widget. |
| 1717 | A record's owner grants other users read (or write) access to that individual |
| 1718 | record. See the security model below. |
| 1719 | |
| 1720 | ## Sharing: the security model |
| 1721 | |
| 1722 | Sharing is the one capability that grants access, so it is designed so that a |
| 1723 | grant can only ever *narrow* what a viewer could already reach, never widen it |
| 1724 | for the granter's own benefit. |
| 1725 | |
| 1726 | ```mermaid |
| 1727 | sequenceDiagram |
| 1728 | participant Owner |
| 1729 | participant API as /api/share |
| 1730 | participant Store as record_shares |
| 1731 | participant Grantee |
| 1732 | participant Engine as permission check |
| 1733 | |
| 1734 | Owner->>API: POST {collection, record_id, user_id} |
| 1735 | API->>API: does Owner own the TARGET record? |
| 1736 | Note right of API: cross-collection ownership check<br/>a row_filter cannot express |
| 1737 | API-->>Owner: 403 if not owner |
| 1738 | API->>Store: create grant (owner-only path) |
| 1739 | Grantee->>Engine: read record |
| 1740 | Engine->>Store: subject.shared_records[collection] |
| 1741 | Engine->>Engine: {id: $shared_records} row filter matches |
| 1742 | Engine-->>Grantee: allowed (only this record, only this collection) |
| 1743 | ``` |
| 1744 | |
| 1745 | Three properties make it safe: |
| 1746 | |
| 1747 | 1. **Grants are minted only by `/api/share`.** `record_shares` has *no* |
| 1748 | create/update/delete permission rule, so a raw collection write can't forge |
| 1749 | one. The endpoint authorizes each grant against the **target record's** |
| 1750 | `owner_id` — a cross-collection fact a row filter can't check. A non-owner |
| 1751 | gets `403 ""Only the record's owner can share it.""` |
| 1752 | 2. **Reads resolve through the subject.** Before a check runs, the server builds |
| 1753 | `subject.shared_records` (per collection) from `record_shares`, the same way |
| 1754 | it builds `project_ids` from `project_access`. A collection opts in with a |
| 1755 | `{""id"": ""$shared_records""}` read rule. |
| 1756 | 3. **`$shared_records` is collection-aware.** The collection being checked is |
| 1757 | threaded through `_record_matches_filter → _resolve_filter_value`, so a share |
| 1758 | on a *task* never matches a same-id row in another collection. (Covered by |
| 1759 | `test_shared_records_filter_is_collection_scoped`.) |
| 1760 | |
| 1761 | Verified end to end: a grantee reads a shared record (`200`), a non-grantee is |
| 1762 | denied (`403`), and a non-owner cannot create a grant (`403`). |
| 1763 | |
| 1764 | ## Adding a new capability |
| 1765 | |
| 1766 | The recipe, in order: |
| 1767 | |
| 1768 | 1. **A polymorphic collection** with `parent_collection` + `parent_id` (both |
| 1769 | *settable on create* — a field that is `required` **and** `read_only` can |
| 1770 | never be created through the HTTP write path, only a server-side |
| 1771 | `preserve_read_only` bypass). |
| 1772 | 2. **A widget** object at `site_{name}` serving `window.dbbasic{Name}` (public |
| 1773 | `execute` rule — it's a static script; the data it reads stays row-gated). |
| 1774 | 3. **Wire it** in `view_render`'s `maybeMountCapabilities` and add |
| 1775 | `<script src=""/{name}"">` to the view page. |
| 1776 | 4. **Adopt it** on a collection with `capabilities.{name}: true`. If the |
| 1777 | behavior grants access (like sharing), also add the permission rule. |
| 1778 | |
| 1779 | Leave `owner_id` to the server on create — a field with a `public: hidden` |
| 1780 | policy is server-set from the session and a client value is rejected. |
| 1781 | |
| 1782 | ## Related |
| 1783 | |
| 1784 | - [`schema-forms.md`](/docs/schema-forms) — the display layer (forms, views, list |
| 1785 | modes) capabilities sit on top of. |
| 1786 | - [`permissions-model.md`](/docs/permissions-model) — row/field filters, the |
| 1787 | `$`-variable row-filter language `$shared_records` extends, and sharing. |
| 1788 | - [`ui-decisions.md`](/docs/ui-decisions) — decisions #7–#14, the living log of the |
| 1789 | generative-UI and capability behaviors (with the gotchas found building them). |
| 1790 | " |
| 1791 | capability-objects Capability Objects: ffmpeg, OCR, and Other System Tools Building on DBBASIC 260 How to write DBBASIC objects that shell out to system binaries like tesseract and ffmpeg safely, with worked examples and a security checklist. "A DBBASIC object is plain Python running in a subprocess. There is no import sandbox and no call whitelist, so an object can do anything Python on that machine can do — including shell out to system binaries. That makes real capability work a normal object: OCR a scanned receipt with `tesseract`, transcode a clip with `ffmpeg`, render a thumbnail with ImageMagick, extract text from a PDF, call a local model. |
| 1792 | |
| 1793 | An earlier prototype of this platform shipped a pack of capability objects like these. The current runtime supports the same capability; this page documents how to write them safely. |
| 1794 | |
| 1795 | ## The execution model |
| 1796 | |
| 1797 | Each object call runs in a **spawned subprocess** with a wall-clock timeout (`DBBASIC_OBJECT_TIMEOUT_SECONDS`). Practical consequences: |
| 1798 | |
| 1799 | - The object has the full standard library, the filesystem, and the ability to launch its own subprocesses. `subprocess.run([...])` works. |
| 1800 | - A runaway object is killed at the timeout, so a hung `ffmpeg` cannot wedge the server. Set the timeout to fit the workload. |
| 1801 | - It is isolated from the parent's memory, not from the machine. There is no CPU or memory cap yet, so capability objects are **operator/admin code** — your own tools on your own box, the same trust level as a cron job. Writing object source is gated; visitors run these objects, they do not author them. |
| 1802 | |
| 1803 | ## Prerequisite: install the tool |
| 1804 | |
| 1805 | Capability objects call binaries that must exist on the VM. This is a normal operator step, explicit rather than hidden: |
| 1806 | |
| 1807 | ```bash |
| 1808 | sudo apt install tesseract-ocr ffmpeg imagemagick poppler-utils |
| 1809 | ``` |
| 1810 | |
| 1811 | If a binary is missing, the object's `subprocess.run` raises `FileNotFoundError`, which surfaces as a normal object execution error in the logs and change feed — no silent failure. |
| 1812 | |
| 1813 | ## Worked example: OCR an uploaded image |
| 1814 | |
| 1815 | A browser form posting `multipart/form-data` delivers files under `request[""_files""]` with base64 content. The object writes the bytes to a temp file, runs `tesseract`, and returns the text. |
| 1816 | |
| 1817 | ```python |
| 1818 | import base64 |
| 1819 | import subprocess |
| 1820 | import tempfile |
| 1821 | from pathlib import Path |
| 1822 | |
| 1823 | |
| 1824 | def POST(request): |
| 1825 | upload = request.get(""_files"", {}).get(""image"") |
| 1826 | if upload is None: |
| 1827 | return {""status"": ""error"", ""error"": ""post an image field""} |
| 1828 | |
| 1829 | content = base64.b64decode(upload[""content_base64""]) |
| 1830 | with tempfile.TemporaryDirectory() as work: |
| 1831 | source = Path(work) / ""in.png"" |
| 1832 | source.write_bytes(content) |
| 1833 | result = subprocess.run( |
| 1834 | [""tesseract"", str(source), ""stdout""], |
| 1835 | capture_output=True, text=True, timeout=30, |
| 1836 | ) |
| 1837 | |
| 1838 | if result.returncode != 0: |
| 1839 | _logger.error(""ocr failed"", stderr=result.stderr[:500]) |
| 1840 | return {""status"": ""error"", ""error"": ""OCR failed""} |
| 1841 | |
| 1842 | text = result.stdout.strip() |
| 1843 | _logger.info(""ocr done"", chars=len(text)) |
| 1844 | return {""status"": ""ok"", ""text"": text} |
| 1845 | ``` |
| 1846 | |
| 1847 | Set that object's timeout generously (`DBBASIC_OBJECT_TIMEOUT_SECONDS`) so a large scan has room, and grant it a public `execute` rule if the form is public. |
| 1848 | |
| 1849 | ## Worked example: transcode with ffmpeg |
| 1850 | |
| 1851 | ```python |
| 1852 | import base64 |
| 1853 | import subprocess |
| 1854 | import tempfile |
| 1855 | from pathlib import Path |
| 1856 | |
| 1857 | |
| 1858 | def POST(request): |
| 1859 | upload = request.get(""_files"", {}).get(""clip"") |
| 1860 | content = base64.b64decode(upload[""content_base64""]) |
| 1861 | with tempfile.TemporaryDirectory() as work: |
| 1862 | src = Path(work) / ""in"" |
| 1863 | out = Path(work) / ""out.mp3"" |
| 1864 | src.write_bytes(content) |
| 1865 | proc = subprocess.run( |
| 1866 | [""ffmpeg"", ""-y"", ""-i"", str(src), ""-vn"", ""-b:a"", ""128k"", str(out)], |
| 1867 | capture_output=True, timeout=120, |
| 1868 | ) |
| 1869 | if proc.returncode != 0 or not out.exists(): |
| 1870 | return {""status"": ""error"", ""error"": ""transcode failed""} |
| 1871 | audio = out.read_bytes() |
| 1872 | |
| 1873 | return { |
| 1874 | ""content_type"": ""audio/mpeg"", |
| 1875 | ""body"": base64.b64encode(audio).decode(""ascii""), |
| 1876 | ""encoding"": ""base64"", |
| 1877 | } |
| 1878 | ``` |
| 1879 | |
| 1880 | Small media (receipts, short clips, single images) fits inside a request. **Long media does not** — a two-second timeout is fine for a receipt but a 2 GB video transcode belongs in a background job, not a web request. Background jobs are a planned platform slice; until then, keep capability objects to work that finishes in seconds. |
| 1881 | |
| 1882 | ## Composing with files, records, and AI |
| 1883 | |
| 1884 | Capability objects get more useful combined with the rest of the platform: |
| 1885 | |
| 1886 | - **Files** — instead of accepting an upload each time, an object can read a stored file's bytes and OCR or transcode it, then write the result back as a new file or a record. (The clean helper for an object to read from the `files` collection by id is a small planned addition; today an object works from the request payload.) |
| 1887 | - **Records** — an OCR object can `create` a record with the extracted text so it becomes searchable through global search. |
| 1888 | - **AI + shell** — because the object is exposed as an `execute_object` MCP tool, the shell AI can call it. ""OCR my latest upload and note the total"" becomes: the AI reads the file, calls the OCR object, and creates a note — each step permission-checked and audited, no glue code. |
| 1889 | |
| 1890 | ## Safety checklist for a capability object |
| 1891 | |
| 1892 | - Set a timeout that fits the job; never rely on the default. |
| 1893 | - Validate the upload before spending CPU on it (size, content type). |
| 1894 | - Write to a `tempfile.TemporaryDirectory()` so nothing leaks between calls; never build a shell command by string-concatenating user input — pass argument lists to `subprocess.run`, never `shell=True`. |
| 1895 | - Log failures with `_logger` so they land in the change feed. |
| 1896 | - Treat these as operator-authored: source writes are gated, and a per-family ""builder role"" for scoped, non-admin object creation is a deliberate open design item, not a shipped capability. |
| 1897 | " |
| 1898 | storage-modes Collection Storage Modes Data & Storage 300 How collections choose between classic (rewrite-on-write) and append (log-structured) storage, with real numbers on write cost, compaction, and current limits. "A collection stores its records one of two ways. Both live in the same |
| 1899 | single `records.tsv`, both are served by the same API, permissions, cache, |
| 1900 | and generators, and both satisfy the same core invariant: the file alone |
| 1901 | is the complete truth. |
| 1902 | |
| 1903 | ## Classic (the default) |
| 1904 | |
| 1905 | Every write atomically rewrites the whole file (temp file + rename). Every |
| 1906 | row appears exactly once. This is the right mode for what collections |
| 1907 | usually are — hundreds to tens of thousands of records — and it is what |
| 1908 | every collection uses unless its schema says otherwise. Nothing existing |
| 1909 | changes modes on its own; there is no migration. |
| 1910 | |
| 1911 | - Write cost grows with collection size (measured: ~4ms at 1k rows, |
| 1912 | ~284ms at 100k, ~2.8s at 1M — per write). |
| 1913 | - The file never contains history; `cat` shows current state only. |
| 1914 | |
| 1915 | ## Append (opt-in per collection) |
| 1916 | |
| 1917 | Declare it in the schema: |
| 1918 | |
| 1919 | ```json |
| 1920 | { ""name"": ""events_log"", ""storage"": ""append"", ""fields"": [ ... ] } |
| 1921 | ``` |
| 1922 | |
| 1923 | The collection becomes a last-wins log inside the same file: create, |
| 1924 | update, and delete each append one row instead of rewriting. An internal |
| 1925 | first column `_op` marks ordinary rows (empty) and deletions (`del`) — it |
| 1926 | is never visible in schemas, API responses, or returned records, and no |
| 1927 | schema may declare a field named `_op`. Reads fold the log newest-wins; |
| 1928 | list order matches classic mode exactly. |
| 1929 | |
| 1930 | - Write cost is near-constant (measured: create ~4ms at 100k rows, ~34ms |
| 1931 | at 1M; update ~1–9ms — versus seconds in classic mode at that scale). |
| 1932 | - Until compaction, `cat` shows the row's full history and deletions — an |
| 1933 | inspectable audit trail, by design. |
| 1934 | - Mode transitions are just writes: adding `""storage"": ""append""` takes |
| 1935 | effect on the next write (one final rewrite adds `_op`); removing it |
| 1936 | compacts back to a classic file on the next write. |
| 1937 | |
| 1938 | ### Compaction (the VACUUM analog) |
| 1939 | |
| 1940 | Superseded rows and tombstones accumulate until compaction folds the file |
| 1941 | back to live rows only (atomic rewrite, under the writer lock): |
| 1942 | |
| 1943 | - Manually: `object_records.compact_collection(""name"", base_dir=...)` |
| 1944 | (returns rows/bytes before and after). |
| 1945 | - Automatically: when a read observes more dead rows than live ones and |
| 1946 | the file exceeds `DBBASIC_APPEND_COMPACT_MIN_ROWS` (default 10000) |
| 1947 | physical rows, the **next write** compacts instead of appending. Reads |
| 1948 | never rewrite anything. |
| 1949 | - Via HTTP (admin-gated, same posture as other mutating admin record |
| 1950 | routes): `POST /admin/collections/{collection}/compact` runs |
| 1951 | `compact_collection` and returns its rows/bytes-before/after summary |
| 1952 | plus `duration_ms`. 404 for an unknown collection, 400 for a collection |
| 1953 | not currently in append storage mode. `GET /admin/storage` reports |
| 1954 | per-collection compaction observability — `{""physical_rows"", |
| 1955 | ""live_rows"", ""dead_rows"", ""bloat_ratio"", ""file_bytes"", |
| 1956 | ""sidecar_present"", ""compaction_flagged""}` for every append-mode |
| 1957 | collection (`object_records.append_collection_stats` / |
| 1958 | `list_append_collection_stats`). This endpoint is cheap by |
| 1959 | construction: it never folds records.tsv to answer, preferring a warm |
| 1960 | in-process cache entry or the id→offset sidecar's own row count; a |
| 1961 | collection neither can answer comes back with `""estimated"": true` and |
| 1962 | only `file_bytes` populated (`physical_rows`/`live_rows` null) rather |
| 1963 | than paying an O(file) parse on a status call. |
| 1964 | - Via the daemon (`object_daemon.py`, optional — nothing else depends on |
| 1965 | it running): `process_compactions` polls every |
| 1966 | `DBBASIC_COMPACTION_INTERVAL_SECONDS` (default 3600, tracked by a |
| 1967 | `.compaction_last_run` marker file under the data dir rather than an |
| 1968 | in-process timer, so the interval survives a daemon restart) and |
| 1969 | compacts any append-mode collection whose `bloat_ratio` is at or above |
| 1970 | `DBBASIC_COMPACTION_BLOAT_RATIO` (default 1.0 — dead rows at least |
| 1971 | matching live rows) AND whose physical row count is at or above |
| 1972 | `DBBASIC_APPEND_COMPACT_MIN_ROWS`. A failure compacting one collection |
| 1973 | is logged and skipped; it never stops the rest of the pass. |
| 1974 | |
| 1975 | ### Crash behavior |
| 1976 | |
| 1977 | A write interrupted mid-row leaves a torn final line. Readers ignore it; |
| 1978 | the next writer truncates it away before appending. Compaction and classic |
| 1979 | mode keep full temp+rename atomicity. |
| 1980 | |
| 1981 | ### Current limits (honest) |
| 1982 | |
| 1983 | - Collections larger than `DBBASIC_RECORDS_CACHE_MAX_ROWS` (default |
| 1984 | 500k) fall out of the in-process cache. Point operations (create's |
| 1985 | duplicate check, get, update, delete) no longer pay a full fold-parse |
| 1986 | above that size: an id→offset sidecar (`.records.oidx` next to |
| 1987 | `records.tsv`, derived and disposable) answers them by seek instead |
| 1988 | (measured: create/update ~1ms, get ~0.3ms at 1M rows, versus ~2.8s for |
| 1989 | a full fold-parse before the sidecar existed). The sidecar builds |
| 1990 | lazily on first use past the threshold (one sequential scan, ~2s at 1M |
| 1991 | rows) and self-heals from any inconsistency by rebuilding. **List and |
| 1992 | read-all-records stay unchanged** above the threshold: they still fold |
| 1993 | the whole file, since ordering and windowing need the full picture — a |
| 1994 | secondary sidecar for that is future work. At or below the threshold, |
| 1995 | nothing changed: the in-process cache still serves everything, sidecar |
| 1996 | or not. |
| 1997 | - At or below the threshold, a warm cache entry for an append-mode |
| 1998 | collection that grows in place (another process, the CLI, or a sibling |
| 1999 | worker appending — same inode, size only) is caught up with a |
| 2000 | tail-delta fold of just the new rows instead of a full re-parse of the |
| 2001 | whole file: when a stat signature changes by growth alone, only the |
| 2002 | tail delta gets parsed. Measured: reading a 400,000-row collection |
| 2003 | right after an external +100-row append dropped from ~742–815ms (full |
| 2004 | re-fold every time, regardless of how small the growth was) to |
| 2005 | ~74–100ms (fold just the 100 new rows) — roughly 8x at this size, and |
| 2006 | the gap widens with collection size since the old cost was O(file) and |
| 2007 | the new one is O(delta). A delta containing any deletion falls back to |
| 2008 | a full fold (deletes shift every later row's position, which the cheap |
| 2009 | merge doesn't model); growth past the cache threshold is still evicted |
| 2010 | per the existing row-count rule above. |
| 2011 | - Cold reads of an append file cost ~1.6x a classic parse (fold |
| 2012 | overhead). Compaction restores parity. |
| 2013 | - Backup **preview/diff** interprets `_op`: for append-mode collections |
| 2014 | (backup side, live side, or both — a backup taken before a mode switch |
| 2015 | compares fine against a now-classic or now-append live file) it folds |
| 2016 | the log last-wins-by-id before diffing, so `_op` never appears as a |
| 2017 | field and a tombstoned id is treated as absent, not live. Backup/restore |
| 2018 | of the file itself is unaffected either way (the file is opaque bytes to |
| 2019 | backup). |
| 2020 | |
| 2021 | ## Choosing |
| 2022 | |
| 2023 | Classic for ordinary app data (the default is correct). Append for |
| 2024 | write-hot or log-shaped collections: event logs, impressions, telemetry, |
| 2025 | game/world state, anything updated far more often than it is read cold. |
| 2026 | When in doubt, stay classic — switching later is one schema edit." |
| 2027 | append-only-storage Append-Only Storage: Design Decisions Data & Storage 310 Design decisions for making collection writes O(1) via append-only TSV storage, while preserving the single-file-of-truth invariant. "This document lays out the design decisions for the next phase of the storage engine: making collection writes O(1) while preserving the system's defining constraint. As with other design work in this project, it's written before implementation, so the direction and its reasoning are on record before any code exists. |
| 2028 | |
| 2029 | ## The Invariant (Decided First, Everything Else Follows) |
| 2030 | |
| 2031 | **A collection either IS one `records.tsv` file, or can deterministically emit one.** The single TSV is the source of truth: complete on its own, greppable, diffable, backed up with `cp`. Anything clever — indexes, caches, offsets — is a *sidecar*: derived, disposable, and rebuildable from the TSV alone. No storage feature may ever be added that cannot be flattened back to the single file. (This also binds any future alternate backend: its contract includes materializing the canonical TSV on demand.) |
| 2032 | |
| 2033 | ## Why Change Anything |
| 2034 | |
| 2035 | Measured (2026-07-12, on an M1 machine, local SSD, after the read-cache work landed): `create` and `update` rewrite the whole file per record — 10.5s at 1M rows, 2.5min at 5M. Reads scale fine (O(1) indexed `get` holds at millions of rows); writes are the wall. Motivating workloads: game-server state (write-hot large collections), ad-exchange impression logs, price history — all append-shaped. |
| 2036 | |
| 2037 | ## The Design |
| 2038 | |
| 2039 | Append-only *within the single file*: |
| 2040 | |
| 2041 | - **Create** appends one row. O(1). |
| 2042 | - **Update** appends a superseding row with the same id. Reads resolve **last-wins** per id. This is a deliberate change from today's duplicate-id behavior, which is first-occurrence-wins. |
| 2043 | - **Delete** appends a tombstone row. The exact representation is decided at implementation time, but it must be a reserved marker that cannot collide with user data, must survive the CSV layer, and must be visible and obvious in a raw `cat`. |
| 2044 | - **Compaction** is the only remaining whole-file rewrite: collapse to live rows via the existing atomic temp+rename, run on a schedule or when the superseded-row ratio passes a threshold — never inline in a request. Until compaction, the file contains its own row history: grep shows every version a record went through. That is a feature (an inspectable audit trail), not merely tolerated. |
| 2045 | - **New-column handling**: today a new field triggers a full-file header rewrite. Append-only tolerates rows shorter than the current header (they read as empty — the semantics that already exist), so a header change can ride the next compaction rather than forcing an immediate rewrite. |
| 2046 | |
| 2047 | ## Sidecars |
| 2048 | |
| 2049 | - **id → byte-offset index** file per collection: maps each id to the offset of its *latest* row. Enables point reads by seek without parsing the file. Derived and disposable — rebuild is one sequential scan. Never required for correctness; readers without the sidecar fall back to scan. |
| 2050 | - **The in-process read cache** (already shipped) composes into incremental reads: cache entries remember the byte offset they consumed; when a stat signature changes by growth alone, parse only the tail delta — O(new rows), not O(file). This dissolves the ""big hot collection vs. cache eviction"" tension for write-hot workloads. |
| 2051 | - The read cache's staleness check currently relies on every write producing a fresh inode via atomic rename. Appends mutate in place instead (same inode; mtime/size change). The stat-signature check still detects the change correctly, but the reasoning behind it needs updating for the append case at implementation time, including the torn-tail rule described below. |
| 2052 | |
| 2053 | ## Crash Safety |
| 2054 | |
| 2055 | Rename-atomicity is replaced (for appends) by the classic log rule: a torn final line — no trailing newline, or a row that fails to parse at EOF — is ignored by readers and truncated/overwritten by the next writer. Appends are flushed per write; fsync policy is a deliberate knob (default: flush, no fsync per row — same durability class as today's writes; an env var enables fsync-per-append for workloads that want it). Compaction keeps full temp+rename atomicity. |
| 2056 | |
| 2057 | ## Concurrency |
| 2058 | |
| 2059 | Writers still serialize under the existing per-collection file lock — but the critical section shrinks from ""rewrite everything"" to ""append one line,"" so effective write throughput rises by orders of magnitude without new machinery. Readers remain lock-free; the stat-signature cache covers cross-process coherence, as today. |
| 2060 | |
| 2061 | ## What This Buys, Honestly |
| 2062 | |
| 2063 | - The measured interactive ceiling moves from tens of thousands of rows to **tens of millions** for ordinary use, and far beyond for append-heavy, read-by-id workloads (a 15GB file appends in microseconds and point-reads by seek). |
| 2064 | - What stays hard: rich secondary queries (by customer, by date range, by status) still scan unless a sidecar index for that field exists. Secondary sidecar indexes are future work, following the same rules: derived, disposable. |
| 2065 | - What is out of scope on purpose: cross-collection transactions, replication, multi-writer scaling. Single VM, one honest file, is still the product. |
| 2066 | |
| 2067 | ## Migration and Compatibility |
| 2068 | |
| 2069 | Opt-in per collection (schema flag or storage marker), default unchanged — existing collections keep their exact current behavior until switched. A compacted append-only file is byte-compatible with a classic one (header + unique rows), so switching a collection back is: compact, remove the marker. Package installs, backups, and restore already treat `records.tsv` as opaque content and keep working; the backup process gains nothing new to learn (the sidecar index is rebuildable, so backups may ignore it entirely). |
| 2070 | |
| 2071 | ## The Escape Hatch (Documented Now, Built Never — Until Real Need) |
| 2072 | |
| 2073 | If a collection someday truly outgrows one file (billions of rows, heavy secondary queries), the answer is a **pluggable per-collection backend** under the identical schema/permission/API surface — SQLite being the obvious candidate (single file, stdlib, public domain). The invariant above still binds it: such a backend must emit the canonical `records.tsv` on demand. This paragraph exists so nobody ever ""improves"" the TSV engine into a half-database; past the envelope, graft a real one underneath instead. |
| 2074 | |
| 2075 | ## Prior Art |
| 2076 | |
| 2077 | The delta-plus-merge shape has flat-file ancestry: Strozzi NoSQL (1998), the system that coined ""NoSQL,"" documented exactly this pattern for big tables — extract a small working table, edit it, background-merge it into the sorted master, keeping the old table and the delta ""for backup and/or journaling purposes."" This design automates that workflow inside one file. Its lock strategy (a `.lock` sidecar next to the table, cleaned on interrupt) is also the mechanism this codebase independently uses. The offset sidecar planned above is the same trick genomics' `tabix` proved on TSV at petabyte scale. |
| 2078 | |
| 2079 | ## Build Order |
| 2080 | |
| 2081 | 1. Tail-delta read cache (offset-tracking) — useful even before append-only lands, zero format change. |
| 2082 | 2. Append-only create/update/delete + torn-tail recovery + last-wins reads, behind the per-collection opt-in. |
| 2083 | 3. Compaction (manual/admin trigger first, then scheduled via the daemon). |
| 2084 | 4. id → offset sidecar for seek-based point reads on large collections. |
| 2085 | 5. Secondary-field sidecars — only when a named workload demands one. |
| 2086 | " |
| 2087 | write-pipeline The Write Pipeline: What Happens, In What Order, And What Can Stop It Data & Storage 320 A stage-by-stage reference for every synchronous, fan-out, and asynchronous step a record write goes through, with each stage's failure semantics. "[`zero-to-code.md`](/docs/zero-to-code) says where each kind of logic lives. |
| 2088 | This page is its runtime twin: when a record is written, **every stage that |
| 2089 | runs, in order, with its failure semantics** — because ""code working with |
| 2090 | code, signals going around"" is exactly where frameworks went dark. Django |
| 2091 | signals and Rails callbacks earned their reputation not because reactions |
| 2092 | are wrong but because nobody could answer *why did my record change?* |
| 2093 | without a debugger. Here the pipeline is fixed, short, and this page is its |
| 2094 | single description. Verified against `object_server.py` and |
| 2095 | `object_records.py`; if this page and the code disagree, fix one of them — |
| 2096 | that disagreement is a bug. |
| 2097 | |
| 2098 | ## The synchronous path (inside the request) |
| 2099 | |
| 2100 | Everything here happens before the client gets its response, in this order: |
| 2101 | |
| 2102 | | # | stage | on failure | lives in | |
| 2103 | |---|---|---|---| |
| 2104 | | 1 | **permission check** (collection + row rules) | 403, nothing ran | policy | |
| 2105 | | 2 | **field-write denial** (schema field permissions) | 403, names the fields | schema | |
| 2106 | | 3 | **owner stamp** — sessions get `owner_id` written server-side; clients cannot spoof it (admin *tokens* skip this: seeding and migration set ownership deliberately) | — | server | |
| 2107 | | 4 | **pre-write hook** (`hooks.before_write`) — may reject (its own 4xx) or transform (stamp fields; id/owner/read-only/computed protected) | **fails CLOSED**: a broken hook is a 500 and no write | `hook/` object | |
| 2108 | | 5 | **auto `created_at`**, then **computed fields materialize** — rollups first, then formulas, so a formula may read a rollup | a broken formula stores `""""` — **fails SOFT**, the write proceeds | `object_computed` | |
| 2109 | | 6 | **schema validation** — types, required, enums, relations-exist, integer-cents; runs on the *final* record, so a hook's transform is validated too | 400, no write | schema | |
| 2110 | | 7 | **transition check** — is this status move declared, and may *this* subject make it (`when` guards) | 403/409, no write | schema | |
| 2111 | | 8 | **persist under the file lock**, then the **change-log entry** (actor, before/after, changed fields) | write errors surface as 4xx/5xx | `object_records` | |
| 2112 | |
| 2113 | Two asymmetries in there are doctrine, not accident: **gates fail closed, |
| 2114 | derivations fail soft** (a validation gate that silently passes on error is |
| 2115 | not a gate; a caption that blanks on error is just a blank caption), and |
| 2116 | **hooks run before validation** so that what a hook stamps is still checked |
| 2117 | like any client input. |
| 2118 | |
| 2119 | ## The fan-out (after the write, same request, best-effort) |
| 2120 | |
| 2121 | Once the record is durably written, the server fans out — in this order, |
| 2122 | each stage isolated so one failing never blocks the next, and none of them |
| 2123 | can un-write the record: |
| 2124 | |
| 2125 | 1. **rollup re-triggers** — a payment write recomputes its invoice's rollup |
| 2126 | fields (single-hop by design: a thread-local guard stops A→B→A cascades; |
| 2127 | a chain that needs more hops is a design smell, not a setting) |
| 2128 | 2. **realtime push** — permission-filtered signal (collection, id, action); |
| 2129 | subscribers re-fetch through the enforced API, so the socket never |
| 2130 | leaks record bodies |
| 2131 | 3. **event handlers** (`HANDLES` objects) — the reaction layer: composers, |
| 2132 | status flips. Payload carries the raw verb (`create`) while the event |
| 2133 | name uses the participle (`…created`) — accept both |
| 2134 | 4. **durable event log** (when enabled) — for pollers and webhooks |
| 2135 | |
| 2136 | ## The asynchronous tail (the daemon, seconds later) |
| 2137 | |
| 2138 | Independent passes over the change log and collections, each with its own |
| 2139 | cursor, each at-least-once: **notify** (rules → in-app/email intents), |
| 2140 | **email outbox** drain, **change dispatch** (off by default — replays |
| 2141 | handlers for storage-level writes so reactions stop depending on which |
| 2142 | path wrote), **auto-transitions** (the 48h approve), **scheduled runners** |
| 2143 | (aging, dunning, recurring journals, matcher, escalation — every run's |
| 2144 | outcome a `scheduler_runs` record), **queue messages** (delay, retry, |
| 2145 | backoff). |
| 2146 | |
| 2147 | ## How chains work without becoming Django signals |
| 2148 | |
| 2149 | The reaction graph is real: a payment triggers the books composer, which |
| 2150 | writes a journal, whose lines re-trigger the journal's rollups. What keeps |
| 2151 | it legible instead of haunted: |
| 2152 | |
| 2153 | - **One hop is synchronous, everything else is a record.** A handler's |
| 2154 | write does not re-enter the dispatcher (storage-level writes don't |
| 2155 | dispatch), so there is no unbounded cascade — a deeper chain must go |
| 2156 | through the change-dispatch pass, which has a cursor, markers, and a log. |
| 2157 | - **Everything generated says who generated it**: `generated_from` |
| 2158 | provenance on composed records, `actor` on every change-log entry. |
| 2159 | ""Why did my record change?"" is a query, not a debugging session: |
| 2160 | `record_changes` names the actor; a composed journal names its source. |
| 2161 | - **Idempotency by provenance** means redelivery is safe by construction — |
| 2162 | a replayed event composes nothing, so at-least-once is a guarantee, not |
| 2163 | a hazard. |
| 2164 | - **Reactions may not gate and gates may not react** |
| 2165 | ([`logic-decisions.md`](/docs/logic-decisions) #6). The moment a reaction |
| 2166 | can block a write, or a gate can send an email, the pipeline above stops |
| 2167 | being a truthful description — that boundary is what makes this page |
| 2168 | possible. |
| 2169 | |
| 2170 | ## Multi-collection workflows |
| 2171 | |
| 2172 | An order becomes an invoice becomes a payment becomes a journal; a task is |
| 2173 | claimed, submitted, approved, settled. There is deliberately **no workflow |
| 2174 | engine** — a business process here is a chain of *(transition on one |
| 2175 | record) → (reaction that writes the next record)*, and each link is |
| 2176 | individually visible in the table above. The process's state is not hidden |
| 2177 | in an engine: it IS the records, their statuses, and their provenance |
| 2178 | stamps, which is why an AI operator can answer ""where is this order stuck?"" |
| 2179 | by reading collections. If a real orchestration need repeats (sagas, |
| 2180 | compensation across many collections), that is a doctrine-#4 moment — |
| 2181 | extract it *then*, from the concrete chains that exist. |
| 2182 | |
| 2183 | ## Reading a live system |
| 2184 | |
| 2185 | - `record_changes` — who did what, before/after, per record |
| 2186 | - `generated_from` — what machine wrote this, from what source |
| 2187 | - `/scheduler` — what the clock did and what each run returned |
| 2188 | - execution logs — every hook and handler run, observable like any object |
| 2189 | - boot lines — the daemon prints every pass's posture (`Books: NOT READY…`, |
| 2190 | `Queue: enabled`, `Change dispatch: disabled`) so a disabled layer is |
| 2191 | announced, not discovered |
| 2192 | </content> |
| 2193 | </StructuredOutput> |
| 2194 | </invoke> |
| 2195 | " |
| 2196 | durability-and-recovery Durability & Recovery Data & Storage 330 How the object server keeps your data safe and readable in plain text, with crash-safe writes, replayable change logs, and a dedicated conformance suite. "How the object server keeps your data safe, and why you can still read it in fifty years. |
| 2197 | |
| 2198 | ## The design principle: plain text is the source of truth |
| 2199 | |
| 2200 | Every record lives in a tab-separated-values file on disk — |
| 2201 | `data/collections/<name>/records.tsv` — that you can open with `cat`, `grep`, |
| 2202 | a spreadsheet, or any programming language's CSV reader, on any operating |
| 2203 | system, with no special software. There is no proprietary binary format your |
| 2204 | data can be trapped inside. |
| 2205 | |
| 2206 | Everything that makes the store *fast* — the id→offset index, the in-memory |
| 2207 | read cache — is **derived from the TSV and rebuildable from it**. Delete every |
| 2208 | index and cache and the database is byte-for-byte identical; you lose speed, |
| 2209 | never a byte. The canonical data is exactly the set of `.tsv` files, and |
| 2210 | nothing else is load-bearing. |
| 2211 | |
| 2212 | > **Why it matters:** a binary database entombs your bytes in a format that |
| 2213 | > needs *its own software* to read. Plain text means your data outlives the |
| 2214 | > software that wrote it. |
| 2215 | |
| 2216 | ## Graceful degradation |
| 2217 | |
| 2218 | Because the store is plain text, corruption **degrades gracefully**. A damaged |
| 2219 | `.tsv` — or a truncated backup of one — can be opened in any editor and |
| 2220 | salvaged by hand: at worst a single line is garbled; the file never becomes |
| 2221 | ""won't open."" A corrupted binary database page, by contrast, can render the |
| 2222 | whole database unreadable without vendor-specific recovery tools, and a torn |
| 2223 | write can damage internal tree structure catastrophically. Plain text has no |
| 2224 | structure to catastrophically damage. |
| 2225 | |
| 2226 | ## Crash safety |
| 2227 | |
| 2228 | - **Classic collections** are rewritten with a temp-file-plus-atomic-rename. |
| 2229 | A reader always sees either the entire previous version or the entire new |
| 2230 | one — never a half-written file. |
| 2231 | - **Append-only collections** (used for high-write logs) treat an interrupted |
| 2232 | write as a torn tail: only the last, incomplete entry is ignored on read; |
| 2233 | every fully-written entry before it survives — including rows whose quoted |
| 2234 | fields span multiple physical lines (the write-side repair is quote-aware; |
| 2235 | see [History: the torn-tail write gap](#history-the-torn-tail-write-gap-fixed)). |
| 2236 | - **Concurrency** is serialized with a kernel file lock (`flock`) that applies |
| 2237 | across both threads *and* separate processes, so concurrent writers on a |
| 2238 | multi-worker deployment cannot corrupt or lose each other's writes. |
| 2239 | |
| 2240 | ## Recovery by replay: backup + log |
| 2241 | |
| 2242 | Alongside each collection, every mutation is also appended to an independent |
| 2243 | **change log** — `record_changes/<collection>/changes.jsonl` — recording, for |
| 2244 | each create/update/delete: a timestamp, the actor, and full *before* and |
| 2245 | *after* snapshots of the record. |
| 2246 | |
| 2247 | This gives you the classic ""restore a backup and replay the log"" recovery |
| 2248 | model, in plain text: |
| 2249 | |
| 2250 | - **Reconstruct a collection from its change log alone.** Replaying the log in |
| 2251 | chronological order — apply each `after` snapshot on create/update, drop the |
| 2252 | record on delete — reproduces the collection's exact current state. If a |
| 2253 | collection file is ever lost or damaged, the change log rebuilds it. |
| 2254 | - **Point-in-time recovery.** Restore a plain-text backup, then replay the |
| 2255 | change log *forward* to a chosen timestamp. |
| 2256 | - **The change log is itself crash-safe by construction.** It is line-based |
| 2257 | (JSON-per-line); a read skips any incomplete final line, so a crash |
| 2258 | mid-append costs only that one in-flight entry. |
| 2259 | |
| 2260 | The result is **two independent plain-text records of every change** — the |
| 2261 | collection file and the change log — either of which can reconstruct the |
| 2262 | other. Your recovery, like your data, needs no special software: a short |
| 2263 | script replays the log. |
| 2264 | |
| 2265 | ## How we test durability |
| 2266 | |
| 2267 | Durability is verified by a dedicated conformance suite, run deliberately |
| 2268 | (like a benchmark) rather than on every commit, covering: |
| 2269 | |
| 2270 | - **Character-set fidelity** — ASCII and control bytes, multibyte UTF-8, |
| 2271 | emoji and grapheme clusters, adversarial Unicode, and arbitrary JSON, each |
| 2272 | round-tripped byte-exact through every storage and read path, with no silent |
| 2273 | normalization. |
| 2274 | - **Crash recovery** — a write interrupted at *every byte offset* is simulated |
| 2275 | and the store is checked to recover to a valid state with no loss of |
| 2276 | committed data — including multi-line quoted rows, whose every-offset sweep |
| 2277 | was held as strict xfails until the write-side repair was fixed and is now |
| 2278 | an ordinary regression battery. |
| 2279 | - **Concurrency** — real concurrent processes writing the same collection, |
| 2280 | checked for lost updates and corruption. |
| 2281 | - **Atomicity** — single-record writes are all-or-nothing, and a reader never |
| 2282 | observes a torn intermediate. |
| 2283 | - **Backup, restore, and compaction** — verified byte-exact, including exotic |
| 2284 | content, and crash-safe (an interrupted compaction never damages the |
| 2285 | original). |
| 2286 | - **Plain-text rebuild and replay** — deleting every index leaves the data |
| 2287 | fully intact and queryable from the TSV alone; a collection is reconstructed |
| 2288 | exactly from its change log. |
| 2289 | |
| 2290 | The philosophy follows the databases that made testing their reputation: a |
| 2291 | visible, exhaustive test suite is itself the assurance that the data is safe. |
| 2292 | |
| 2293 | ## History: the torn-tail write gap (fixed) |
| 2294 | |
| 2295 | **Status: fixed (2026-07-30).** This section replaces a ""Known gap"" that |
| 2296 | stood here for one day — kept as history because the shape of the bug and of |
| 2297 | its first, reverted fix are both instructive. |
| 2298 | |
| 2299 | **The bug.** Before an append lands, `_repair_torn_tail` trims any partial |
| 2300 | row left by an interrupted write. It found ""the last row boundary"" by |
| 2301 | scanning backwards for a newline — quote-blind, so a newline *inside* a |
| 2302 | quoted multi-line field looked like a boundary. Worse, its fast path |
| 2303 | returned untouched any file whose final byte was `\n`, including a newline |
| 2304 | inside an unclosed quote — skipping repair in exactly the case that needed |
| 2305 | it. A crash mid-multi-line-row could then swallow the next appended row into |
| 2306 | the open quote. Exposure: ~97% of such a row's write window. |
| 2307 | |
| 2308 | **The first fix, and why it was reverted.** It gated the repair on the |
| 2309 | records cache's `covered_bytes`. Two flaws, found the second time around: |
| 2310 | that value lives only in process memory (gone precisely when a crash makes |
| 2311 | repair matter), and — the actual killer — a full parse of a *torn* file |
| 2312 | stores `covered_bytes` equal to the whole file size, because the read fold |
| 2313 | ""accounts for"" the torn bytes by knowing they are torn. So a warm cache |
| 2314 | reported clean on exactly the files needing repair. Under pytest the cache |
| 2315 | was warm from other tests; standalone it was cold. ""Fails in pytest, passes |
| 2316 | standalone"" was that, nothing more. |
| 2317 | |
| 2318 | **The fix that stands.** The repair scans forward with the same quote-aware |
| 2319 | `committed_prefix_len` the read path and the backup index use — one |
| 2320 | implementation, three users, no copies to drift. A cache may only ever |
| 2321 | *skip* the scan, never choose a truncation point, and the only cache |
| 2322 | consulted is the id-sidecar's, whose `covered_bytes` values are genuine row |
| 2323 | boundaries by construction. Cold (post-crash) the repair pays one full scan, |
| 2324 | which is the honest price of the one moment a torn tail can exist. Two |
| 2325 | neighbouring holes were closed in the same pass: an intact-but-unterminated |
| 2326 | *header* no longer truncates to a headerless file that reports success and |
| 2327 | then fails every read (the header is restored from the schema), and a |
| 2328 | sidecar corrupted with non-UTF-8 bytes now rebuilds instead of raising on |
| 2329 | every read. |
| 2330 | |
| 2331 | **Verification.** The strict-xfail acceptance tests that mapped the trigger |
| 2332 | surface — every byte offset from a row's first embedded newline to its last |
| 2333 | — were rewritten into the regression guards they were designed to become, |
| 2334 | and the every-offset sweeps now pass as ordinary tests. |
| 2335 | " |
| 2336 | backup-restore Backup and Restore Data & Storage 340 How to create, verify, and restore runtime backups of a DBBASIC Object Server instance, including scheduled backups and the deliberately CLI-only restore path. "DBBASIC runtime backups protect the object loop by keeping source, state, logs, files, versions, schemas, and collection records recoverable together. |
| 2337 | |
| 2338 | The public backup helper is: |
| 2339 | |
| 2340 | ```bash |
| 2341 | python -m object_backup |
| 2342 | ``` |
| 2343 | |
| 2344 | It creates a tar/gzip archive with a JSON manifest and a narrow runtime backup set. |
| 2345 | |
| 2346 | ## Included |
| 2347 | |
| 2348 | Runtime backups include: |
| 2349 | |
| 2350 | ```text |
| 2351 | objects/ |
| 2352 | data/state/ |
| 2353 | data/logs/ |
| 2354 | data/versions/ |
| 2355 | data/source_changes/ |
| 2356 | data/schema_versions/ |
| 2357 | data/record_changes/ |
| 2358 | data/package_changes/ |
| 2359 | data/file_changes/ |
| 2360 | data/files/ |
| 2361 | data/schemas/ |
| 2362 | data/collections/ |
| 2363 | ``` |
| 2364 | |
| 2365 | That covers object source, object state, current logs, rotated compressed logs, source versions, source change history, schema change history, record change history, package change history, object file change history, object-owned files, schema metadata, and TSV-backed collection records. |
| 2366 | |
| 2367 | ## Excluded |
| 2368 | |
| 2369 | Runtime backups deliberately exclude: |
| 2370 | |
| 2371 | - deployment secrets |
| 2372 | - environment files |
| 2373 | - systemd service files |
| 2374 | - git history |
| 2375 | - virtualenvs |
| 2376 | - caches |
| 2377 | - lock files |
| 2378 | - temp files |
| 2379 | - ephemeral rate-limit files |
| 2380 | |
| 2381 | The environment file and service unit still need operator-managed backups, but they should be stored through a secret-aware server backup process, not inside the portable runtime archive. |
| 2382 | |
| 2383 | ## Create |
| 2384 | |
| 2385 | ```bash |
| 2386 | mkdir -p /var/backups/dbbasic-object-server |
| 2387 | |
| 2388 | python -m object_backup create \ |
| 2389 | /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \ |
| 2390 | --objects-dir /var/lib/dbbasic-object-server/objects \ |
| 2391 | --data-dir /var/lib/dbbasic-object-server/data |
| 2392 | ``` |
| 2393 | |
| 2394 | The command writes `dbbasic-backup-manifest.json` inside the archive. The manifest records the format version, timestamp, included roots, file count, byte count, and warnings. |
| 2395 | |
| 2396 | ## Restore points |
| 2397 | |
| 2398 | Server-side mutations that can rewrite multiple runtime files should create a restore point before writing. Package installs use: |
| 2399 | |
| 2400 | ```python |
| 2401 | object_backup.create_runtime_restore_point(""package-hello-world"") |
| 2402 | ``` |
| 2403 | |
| 2404 | By default, restore points are written under: |
| 2405 | |
| 2406 | ```text |
| 2407 | data/backups/ |
| 2408 | ``` |
| 2409 | |
| 2410 | Set `DBBASIC_BACKUPS_DIR` to move those archives to a mounted backup volume or another operator-managed path. Runtime backup archives do not include `data/backups/`, so restore points do not recursively back up other archives. |
| 2411 | |
| 2412 | Package restore uses the recorded restore point from a package install change, not an arbitrary package path. That package-scoped HTTP route (`POST /packages/{id}/restore`, gated by `DBBASIC_ENABLE_PACKAGE_RESTORE`) is separate from full-runtime restore, which stays CLI-only. It calls `restore_runtime_backup(..., overwrite=True, prune_extra=True)` so files created by the package are removed if they were not present in the snapshot. Pruning is limited to known runtime roots such as `objects/`, `data/state/`, `data/logs/`, `data/collections/`, `data/schemas/`, and `data/source_changes/`, `data/package_changes/`, `data/file_changes/`; `data/backups/` is left alone. |
| 2413 | |
| 2414 | ## Verify |
| 2415 | |
| 2416 | ```bash |
| 2417 | python -m object_backup verify \ |
| 2418 | /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \ |
| 2419 | --json |
| 2420 | ``` |
| 2421 | |
| 2422 | Verification checks that: |
| 2423 | |
| 2424 | - the archive can be read |
| 2425 | - the manifest is present |
| 2426 | - the manifest count matches the archive contents |
| 2427 | - archive paths stay under `objects/` or supported `data/` roots |
| 2428 | - symlinks, hard links, device files, and other unsafe member types are rejected |
| 2429 | |
| 2430 | ## Restore |
| 2431 | |
| 2432 | Restore is a **deliberate CLI operation**, not an HTTP verb. It is destructive and whole-instance, so it is intentionally not exposed on the admin surface (`/admin/status` reports `capabilities.backups.can_restore: false`); a ""restore"" button one misclick from wiping live data is a footgun. The pairing is: **download a backup from Scroll or `GET /admin/backups/{id}/download` to get it, and this CLI to apply it.** |
| 2433 | |
| 2434 | Runtime backups live under `data/backups/` (or `DBBASIC_BACKUPS_DIR`) and are named `YYYYMMDDTHHMMSSffffffZ-<label>.tar.gz` (label `manual` for on-demand, `package-<id>` for install restore points). Always verify first (see above). |
| 2435 | |
| 2436 | ### Dry-run into clean directories first |
| 2437 | |
| 2438 | ```bash |
| 2439 | mkdir -p /tmp/dbbasic-restore/objects /tmp/dbbasic-restore/data |
| 2440 | |
| 2441 | python -m object_backup restore \ |
| 2442 | /var/lib/dbbasic-object-server/data/backups/20260708T145041382734Z-manual.tar.gz \ |
| 2443 | --objects-dir /tmp/dbbasic-restore/objects \ |
| 2444 | --data-dir /tmp/dbbasic-restore/data \ |
| 2445 | --json |
| 2446 | ``` |
| 2447 | |
| 2448 | By default restore refuses to overwrite existing files, so this dry run never touches a live runtime. Start the server against the restored directories and check it before trusting the archive: |
| 2449 | |
| 2450 | ```bash |
| 2451 | DBBASIC_OBJECTS_DIR=/tmp/dbbasic-restore/objects \ |
| 2452 | DBBASIC_DATA_DIR=/tmp/dbbasic-restore/data \ |
| 2453 | uvicorn object_server:app --host 127.0.0.1 --port 8001 |
| 2454 | ``` |
| 2455 | |
| 2456 | Then verify `/health` and representative object routes. A real recovery drill should do this on a second clean VM. |
| 2457 | |
| 2458 | ### Recovering the live runtime |
| 2459 | |
| 2460 | Stop the service first so nothing writes during the restore. Use `--overwrite` to replace existing files, and `--prune-extra` (which requires `--overwrite`) only when you want an exact match to the archive, removing files added since — a powerful, opt-in flag: |
| 2461 | |
| 2462 | ```bash |
| 2463 | sudo systemctl stop dbbasic-object-server |
| 2464 | |
| 2465 | sudo -u dbbasic /opt/dbbasic-object-server/.venv/bin/python \ |
| 2466 | -m object_backup restore \ |
| 2467 | /var/lib/dbbasic-object-server/data/backups/<id>.tar.gz \ |
| 2468 | --objects-dir /var/lib/dbbasic-object-server/objects \ |
| 2469 | --data-dir /var/lib/dbbasic-object-server/data \ |
| 2470 | --overwrite --prune-extra --json |
| 2471 | |
| 2472 | sudo systemctl start dbbasic-object-server |
| 2473 | curl http://127.0.0.1:8001/health |
| 2474 | ``` |
| 2475 | |
| 2476 | Pruning is limited to known runtime roots (`objects/`, `data/state/`, `data/logs/`, `data/collections/`, `data/schemas/`, `data/source_changes/`, `data/package_changes/`, `data/file_changes/`); `data/backups/` is left alone, so restoring never destroys your other archives. |
| 2477 | |
| 2478 | ## HTTP surface |
| 2479 | |
| 2480 | On-demand backup and download are exposed on the admin API (and used by Scroll's Backup screen) — see [the HTTP contract](/docs/http-api-contract#backups): |
| 2481 | |
| 2482 | - `GET /admin/backups` — inventory (`id`, `created_at`, `size`, `kind`, `scope`). |
| 2483 | - `POST /admin/backups` — create a full-runtime backup now. |
| 2484 | - `GET /admin/backups/{id}/download` — stream the archive. |
| 2485 | |
| 2486 | All admin-gated (a backup contains credentials and service keys, so never public). Restore is deliberately absent from this surface — use the CLI above. |
| 2487 | |
| 2488 | ## Scheduling |
| 2489 | |
| 2490 | Automatic backups are a **config option, off by default**. The run itself is an external timer (systemd or cron) invoking `object_backup.py create` on a cadence with retention; `DBBASIC_BACKUP_SCHEDULE` records the operator's intent so `/admin/status` and Scroll can display it. A sample daily systemd timer: |
| 2491 | |
| 2492 | ```ini |
| 2493 | # /etc/systemd/system/dbbasic-backup.service |
| 2494 | [Service] |
| 2495 | Type=oneshot |
| 2496 | User=dbbasic |
| 2497 | EnvironmentFile=/etc/dbbasic-object-server.env |
| 2498 | WorkingDirectory=/opt/dbbasic-object-server |
| 2499 | ExecStart=/opt/dbbasic-object-server/.venv/bin/python -m object_backup create \ |
| 2500 | /var/lib/dbbasic-object-server/data/backups/scheduled-%%i.tar.gz |
| 2501 | ``` |
| 2502 | |
| 2503 | ```ini |
| 2504 | # /etc/systemd/system/dbbasic-backup.timer |
| 2505 | [Timer] |
| 2506 | OnCalendar=daily |
| 2507 | Persistent=true |
| 2508 | [Install] |
| 2509 | WantedBy=timers.target |
| 2510 | ``` |
| 2511 | |
| 2512 | Note that on-box backups do not survive losing the VM — pair them with provider-level disk backups (or copy archives to a bucket) for disaster recovery. On-demand create and download always work regardless of any schedule. |
| 2513 | " |
| 2514 | per-tenant-storage Per-Tenant Storage: The SaaS Decomposition Data & Storage 350 A direction note on decomposing one shared collection into per-tenant storage roots — what it buys, what it costs, and why the industry is already headed there. "This is a direction note, not a build plan. It's recorded alongside the [append-only storage design guide](append-only-storage-design) because the two together answer ""how far can one TSV go, and what happens at SaaS scale"" — and the answer to the second question is not ""a bigger table."" |
| 2515 | |
| 2516 | ## The observation |
| 2517 | |
| 2518 | The measured storage envelope (see the append-only storage design guide: tens of thousands of rows interactive today, tens of millions after append-only) looks like a limit under the one-big-table model. Per-tenant storage inverts it: a SaaS with 100,000 customers and ""a billion orders"" does not have a billion-row problem — it has 100,000 ten-thousand-row problems, each of which sits in the comfortable center of the measured envelope. The giant commingled table is an artifact of shared-database design, not a law of SaaS. |
| 2519 | |
| 2520 | ## What decomposition buys |
| 2521 | |
| 2522 | - **Physical isolation.** Row-level security (`WHERE tenant_id = ?`, one missing predicate from a breach) becomes filesystem separation. The permission engine's row filters harden into file boundaries. |
| 2523 | - **GDPR and export for free.** Right-to-be-forgotten is removing the tenant's directory. ""Export my data"" is handing over the tenant's own TSV files — human-readable, Excel-openable, no export pipeline. |
| 2524 | - **Per-customer backup/restore already exists.** Scoped restore plus a read-only preview already ship as platform features, and they apply per tenant with zero new code: ""restore customer X's notes to yesterday"" is the same operation as any other scoped restore. |
| 2525 | - **Per-tenant migrations.** Package upgrades ride the provenance/reconcile system tenant-by-tenant: incremental rollout, conflicts parked per tenant, no all-customers-at-once schema locks. |
| 2526 | - **Noisy neighbors decompose.** One tenant's huge collection cannot slow another's small one; the records-cache LRU bound doubles as per-tenant cache fairness. |
| 2527 | - **Tenant mobility is file movement.** Rebalancing tenants across VMs, or graduating one outlier to its own instance or to the per-collection backend escape hatch, is `mv` of a directory. Shared-nothing per tenant; horizontal scale without a distributed database. |
| 2528 | - **Metering is `du`.** Storage billing per tenant is a directory size. |
| 2529 | |
| 2530 | ## Honest costs |
| 2531 | |
| 2532 | - **Cross-tenant queries become scatter-gather.** Admin analytics (""orders today, all customers"") need rollup collections maintained by the daemon (aggregates are just another collection), or a scan across tenant directories. This is the standard sharding trade-off and should be designed as rollups, not ad-hoc scans. |
| 2533 | - **File-count hygiene.** 100k tenants times N collections is roughly 1M files — fine for modern filesystems, but the directory layout should shard (e.g. `tenants/ab/abcd.../collections/...`) from day one. |
| 2534 | - **The single-TSV invariant holds per collection per tenant** — and the whole-instance view (""all tenants' notes as one TSV"") is itself just a deterministic concatenation with a tenant column, satisfying the emit-one-file rule at the aggregate level whenever it's needed. |
| 2535 | |
| 2536 | ## Prior art |
| 2537 | |
| 2538 | The industry is arriving at the same place from several directions: Cloudflare Durable Objects (SQLite per entity, storage co-located with logic), Turso (database-per-tenant as a product), and the SQLite-per-user pattern generally. Per-tenant single-file storage next to per-tenant code is the current direction of travel; this system's version is the flavor you can `cat`. |
| 2539 | |
| 2540 | There's also older advice in the flat-file lineage worth citing directly. Strozzi NoSQL's ""Big tables"" documentation opens with *""big tables should be avoided in the first place""* — use the Unix filesystem to pre-organize relations and keep each one small. Per-tenant decomposition is that doctrine, systematized. |
| 2541 | |
| 2542 | ## Current state and gap |
| 2543 | |
| 2544 | Today the platform is multi-tenant by *rows* (`owner_id` row filters, one shared collection per name) and multi-site by *hosts* (`site_hosts`). The gap this note names: collections themselves are instance-global. The eventual shape is tenant-scoped collection roots under one instance — same schemas, same permission engine, same generators, same API surface, with the tenant resolved from identity the way hosts are resolved from requests today. |
| 2545 | |
| 2546 | This isn't scheduled work. It's recorded here so the append-only and cache work keep it reachable — nothing in either conflicts with tenant-scoped roots, since both operate per file. |
| 2547 | " |
| 2548 | site-routing Site Routing Sites, UI & Realtime 400 How to turn the object server into a webserver with clean public URLs, a route table, multi-domain hosting, and static asset handling. "Site routing turns the object server into a webserver: clean public URLs |
| 2549 | (`/about`, `/articles/{uuid}`) resolve to objects, so a whole website can be |
| 2550 | built, edited, and rolled back through the normal object loop with no deploy |
| 2551 | step and no per-page reverse-proxy rewrites. |
| 2552 | |
| 2553 | It is off by default and enabled per deployment: |
| 2554 | |
| 2555 | ```text |
| 2556 | DBBASIC_ENABLE_SITE_ROUTES=true |
| 2557 | ``` |
| 2558 | |
| 2559 | ## Resolution Order |
| 2560 | |
| 2561 | Site routing runs only after every built-in route family has declined the |
| 2562 | path, so reserved surfaces (`/objects`, `/admin`, `/api`, `/identity`, |
| 2563 | `/login`, `/health`, `/collections`, `/schemas`, `/events`, `/packages`, |
| 2564 | `/daemon`, `/permissions`, `/logout`) can never be shadowed by a site page. |
| 2565 | |
| 2566 | For everything else: |
| 2567 | |
| 2568 | 1. **Convention.** `/` maps to `site_home`, `/about` to `site_about`, |
| 2569 | `/docs/install` to `site_docs_install`. Hyphens become underscores |
| 2570 | (`/getting-started` → `site_getting_started`). Creating a page object |
| 2571 | creates its URL — the filesystem-routing feel of the original prototype, |
| 2572 | kept because it makes the common case zero-config. |
| 2573 | 2. **The `site_routes` collection.** Records with `pattern`, `object_id`, and |
| 2574 | optional `priority` handle what conventions cannot express: |
| 2575 | |
| 2576 | | id | pattern | object_id | priority | |
| 2577 | |----|---------|-----------|----------| |
| 2578 | | r1 | `/articles/{article_id:uuid}` | `articles_view` | 10 | |
| 2579 | | r2 | `/blog/{slug}` | `blog_post` | 20 | |
| 2580 | |
| 2581 | Pattern segments are literal text, `{name}` (any single segment), or |
| 2582 | `{name:uuid}` (segment must be UUID-shaped — the common case for record |
| 2583 | ids, which are UUIDv4 by default). Captured params are merged into the |
| 2584 | object's request payload, so `articles_view` reads |
| 2585 | `request[""article_id""]`. Lower `priority` wins; ties go to the pattern |
| 2586 | with more literal segments. |
| 2587 | 3. **`site_404`.** If nothing resolves and a `site_404` object exists, it |
| 2588 | runs with the missed `path` in its payload and should return |
| 2589 | `status_code: 404`. Without it, the plain JSON 404 stays. |
| 2590 | |
| 2591 | ## Design Decisions |
| 2592 | |
| 2593 | These choices came out of comparing the original prototype's pure filesystem |
| 2594 | router, the route-table-in-object-state design from the early dbbasic.com |
| 2595 | plan, and what the public server has grown since: |
| 2596 | |
| 2597 | - **Routing only maps URLs; authorization stays in the permission policy.** |
| 2598 | The resolved object executes through the normal execution path, so |
| 2599 | enforcement, audit entries, execution logs, timeouts, concurrency limits, |
| 2600 | and correlation ids all apply. Publishing a page is two audited steps: |
| 2601 | create the object, grant `execute` to `public` (or whichever principal). |
| 2602 | A file appearing on disk never silently becomes public under enforcement. |
| 2603 | - **The route table lives in records, not object state.** Records give the |
| 2604 | table schema validation, change history, rollback, Scroll editing, and MCP |
| 2605 | tools for free. Editing the sitemap is a data change with an audit trail. |
| 2606 | - **Convention for the 90%, data for the 10%.** Plain pages need no table |
| 2607 | entry. The table exists for parameterized routes, renames, and pointing |
| 2608 | multiple URLs at one object. |
| 2609 | - **TLS and static assets stay in the reverse proxy.** Caddy terminates TLS |
| 2610 | and should serve `/static/*` from disk with cache headers; site routing |
| 2611 | replaces per-page rewrites with one catch-all proxy block. |
| 2612 | |
| 2613 | ## Deployment |
| 2614 | |
| 2615 | With site routing enabled, the Caddy shape becomes: explicit handles for the |
| 2616 | reserved admin/API surfaces you want public, a `/static/*` file server, and a |
| 2617 | catch-all `reverse_proxy` instead of a catch-all 404. Under permission |
| 2618 | enforcement, unrouted or unauthorized paths still return controlled 403/404 |
| 2619 | responses from the server. |
| 2620 | |
| 2621 | ## Multiple Domains |
| 2622 | |
| 2623 | One object server can serve many websites. The `site_hosts` collection maps |
| 2624 | request hosts to object prefixes: |
| 2625 | |
| 2626 | | id | host | prefix | home_object | not_found_object | |
| 2627 | |----|------|--------|-------------|-------------------| |
| 2628 | | h1 | dbbasic.com | db | | | |
| 2629 | | h2 | example.org | eg | | | |
| 2630 | |
| 2631 | With those records, `/` on dbbasic.com serves `db_home` and `/pricing` |
| 2632 | serves `db_pricing`, while `/` on example.org serves `eg_home` — same |
| 2633 | server, same data directory, fully separate page namespaces. `home_object` |
| 2634 | and `not_found_object` default to `{prefix}_home` and `{prefix}_404` |
| 2635 | (falling back to the generic `site_404`). Hosts without a record keep the |
| 2636 | default `site_*` behavior, so single-site deployments need no |
| 2637 | configuration. `site_routes` pattern records take an optional `host` |
| 2638 | column; host-specific rows only match their domain and beat host-agnostic |
| 2639 | rows. |
| 2640 | |
| 2641 | Pointing a new domain at the server is then: |
| 2642 | |
| 2643 | 1. DNS: point the domain's A record at the VM. |
| 2644 | 2. Caddy: add a site block for it and reload — Caddy only answers (and only |
| 2645 | fetches certificates) for hostnames it has a block for, and DNS must |
| 2646 | already point at the VM for the certificate challenge to succeed. With |
| 2647 | the `public_site` snippet defined once at the top of the Caddyfile, each |
| 2648 | new domain is three lines: |
| 2649 | |
| 2650 | ```caddyfile |
| 2651 | (public_site) { |
| 2652 | handle /login { reverse_proxy 127.0.0.1:8001 } |
| 2653 | handle /logout { reverse_proxy 127.0.0.1:8001 } |
| 2654 | handle_path /static/* { |
| 2655 | root * /var/lib/dbbasic-static/{args[0]} |
| 2656 | file_server |
| 2657 | header Cache-Control ""public, max-age=3600"" |
| 2658 | } |
| 2659 | handle { reverse_proxy 127.0.0.1:8001 } |
| 2660 | } |
| 2661 | |
| 2662 | newsite.example { |
| 2663 | import public_site newsite |
| 2664 | } |
| 2665 | ``` |
| 2666 | |
| 2667 | Public domains expose only the site surface; keep the admin/API/MCP |
| 2668 | handles only on the operations domain, so public sites never route to |
| 2669 | the operator surface at all. |
| 2670 | 3. Add the `site_hosts` record and create `{prefix}_home` — the site is live. |
| 2671 | |
| 2672 | Identity, sessions, records, policy, and audit are shared across domains: |
| 2673 | one operator account works everywhere, and browsers scope session cookies |
| 2674 | per domain automatically. Domains that must not share those belong on their |
| 2675 | own server instance instead (one process per trust boundary). |
| 2676 | |
| 2677 | ## Static Assets |
| 2678 | |
| 2679 | Site CSS, JavaScript, images, and fonts are served by the reverse proxy from |
| 2680 | a plain directory — never through object execution — so they get file-server |
| 2681 | performance, gzip, and cache headers: |
| 2682 | |
| 2683 | ```caddyfile |
| 2684 | handle_path /static/* { |
| 2685 | root * /var/lib/dbbasic-static |
| 2686 | file_server |
| 2687 | header Cache-Control ""public, max-age=3600"" |
| 2688 | } |
| 2689 | ``` |
| 2690 | |
| 2691 | Pages reference them normally (`<link href=""/static/site.css"">` in the |
| 2692 | rendered HTML body). The directory is deploy-time content owned by the |
| 2693 | service user; treat it like object source (versioned in the site package), |
| 2694 | not like runtime data. User-uploaded files are different: keep those in the |
| 2695 | object file APIs so they inherit permissions, audit, and backups. |
| 2696 | |
| 2697 | ## Writing Site Pages |
| 2698 | |
| 2699 | Site pages are normal objects (see the object authoring guide): return a |
| 2700 | dict for JSON, a `content_type`/`body` dict for HTML, use |
| 2701 | `request[""_identity""]` for per-user rendering, and handle `POST` for forms — |
| 2702 | browser form posts arrive as parsed fields with the session cookie applied. |
| 2703 | Ship a site as a package (see the package authoring guide) so it installs |
| 2704 | with its schema, seed records, and route table rows in one reviewed unit. |
| 2705 | " |
| 2706 | generative-ui Generative UI — One Renderer, Every App Sites, UI & Realtime 410 How a collection's schema alone drives list, form, and detail rendering across every app, and why customizing a generated page never means forking generated code. "There is no per-app UI code. A collection's schema drives a single set of |
| 2707 | generative renderers, served as static scripts (`/list`, `/form`, `/detail`), |
| 2708 | that turn records into live, permission-checked, realtime surfaces. This page |
| 2709 | is the overview; the field-level contract is in |
| 2710 | [`schema-forms.md`](/docs/schema-forms), the behavior decisions in |
| 2711 | [`ui-decisions.md`](/docs/ui-decisions), and the visual system in |
| 2712 | [`design-system.md`](/docs/design-system). |
| 2713 | |
| 2714 | ```mermaid |
| 2715 | flowchart LR |
| 2716 | SCH[""schema<br/>fields · views · capabilities""] --> LIST[""window.dbbasicList""] |
| 2717 | SCH --> FORM[""window.dbbasicForm""] |
| 2718 | SCH --> DET[""window.dbbasicDetail""] |
| 2719 | LIST --> MODES[""list · table · board · tree · calendar""] |
| 2720 | LIST --> FSR[""filters · search · sort · row cap""] |
| 2721 | FORM --> CTRL[""type-aware controls · validation""] |
| 2722 | DET --> COMP[""detail + related + capabilities""] |
| 2723 | CL[""change log (websocket)""] -. re-render .-> LIST |
| 2724 | CL -. re-render .-> DET |
| 2725 | ``` |
| 2726 | |
| 2727 | ## The list, in five modes |
| 2728 | |
| 2729 | `window.dbbasicList(collection, cfg)` resolves `views.list_mode` from the |
| 2730 | schema and renders one of: |
| 2731 | |
| 2732 | - **list** (default) — rich rows: avatar, title, subtitle, meta, tag pills, |
| 2733 | owner edit/delete. |
| 2734 | - **table** — a dense, sortable HTML table over `list_fields`; cells format by |
| 2735 | type (money `_cents` → `$`, boolean → Yes/No, datetime → relative, enum → |
| 2736 | colored badge), and **relation columns show the target's label, not the raw |
| 2737 | id**. Owned rows get the same edit/delete the list has. |
| 2738 | - **board** — a kanban grouped by an enum field; dragging a card issues the |
| 2739 | ordinary status write, so `flow` transitions still gate the move. A board |
| 2740 | collection that also has `list_fields` shows a **Board ⇄ Table** toggle whose |
| 2741 | choice persists per collection. |
| 2742 | - **tree** — nests a self-relation (`parent_id`) into a hierarchy. |
| 2743 | - **calendar** — buckets records by a date field. |
| 2744 | |
| 2745 | Every mode shares **one** fetch/sort/cap/search/subscribe pipeline, so all of |
| 2746 | them inherit: |
| 2747 | |
| 2748 | - **filters** — `views.filter_fields` renders a filter bar (enum → select, |
| 2749 | boolean → Yes/No); a pick narrows the fetch server-side (`field=value`, after |
| 2750 | the permission row filter) and composes with the search box. |
| 2751 | - **search** — the toolbar box, server-side over the collection's `search` |
| 2752 | fields. |
| 2753 | - **the 50-row cap** with a Show-all toggle (no surface ever renders a |
| 2754 | 50,000px page). |
| 2755 | - **realtime** — each surface subscribes to the change log and re-renders when |
| 2756 | a record changes, from any tab, user, or agent. |
| 2757 | |
| 2758 | A mode whose required field can't be derived (no enum for board, no |
| 2759 | self-relation for tree, no date for calendar) falls back to the row list with a |
| 2760 | visible notice — never a blank page. |
| 2761 | |
| 2762 | ## The form |
| 2763 | |
| 2764 | `window.dbbasicForm(collection)` builds a record form from the schema: field |
| 2765 | order from `forms.default.fields`, controls from field semantics (enum → |
| 2766 | select, relation → picker, boolean → checkbox, date → date input, textarea, |
| 2767 | number, text), and labels/help/required/max-length from the field. It handles |
| 2768 | create (POST) and edit (PUT), sets `id`/`owner_id`/`created_at` automatically, |
| 2769 | never writes computed/read-only fields, and supports conditional field |
| 2770 | visibility (`visible_when`). See [`schema-forms.md`](/docs/schema-forms). |
| 2771 | |
| 2772 | ## The detail page |
| 2773 | |
| 2774 | `window.dbbasicDetail` renders one record read-only by reusing the form's own |
| 2775 | field renderer in read-only mode — no second field renderer to keep in sync. It |
| 2776 | adds owner-only Edit/Delete (reusing the form's edit pipeline), always shows |
| 2777 | record metadata (`created_at`/`updated_at`) even when `detail_fields` curates |
| 2778 | the main fields, and auto-mounts any declared **capabilities** (comments, |
| 2779 | attachments, sharing — see [`capabilities.md`](/docs/capabilities)) below the |
| 2780 | detail. Composed detail pages are `views` records with a `detail` block plus |
| 2781 | `related` child blocks; the view renderer |
| 2782 | (`app-views/objects/site/view_render.py`) assembles them. |
| 2783 | |
| 2784 | A `textarea`-type field shown read-only gets a safe, minimal markdown |
| 2785 | subset (bold, italic, links, paragraph breaks) — the same transform the |
| 2786 | standalone `markdown` block below applies — instead of plain escaped text, |
| 2787 | which used to collapse multi-line content into one run-together line. |
| 2788 | |
| 2789 | ## Other view block kinds |
| 2790 | |
| 2791 | Beyond `list`/`form`/`detail`/`related`, a view's `blocks` array can also |
| 2792 | declare: |
| 2793 | |
| 2794 | - **`markdown`** — a block of static prose baked directly into the view |
| 2795 | definition (`{kind: ""markdown"", text: ""...""}`), the same escape-first, |
| 2796 | safe-subset renderer as the textarea treatment above. |
| 2797 | - **`reader`** — `{kind: ""reader"", url: ""...""}` fetches a URL through the |
| 2798 | flag-gated, signed-in-only `POST /api/read` and renders title + body text |
| 2799 | + a numbered link list. See [`shell-and-ai.md`](/docs/shell-and-ai)'s |
| 2800 | ""Reading the Web"" section for the SSRF posture and the voice-navigation |
| 2801 | loop this backs. |
| 2802 | - **`count`** / **`aggregate`** — live stat cards over a collection (row |
| 2803 | count, or a sum of numeric/`_cents` fields, with an optional |
| 2804 | balanced/not-balanced badge for double-entry-style checks). |
| 2805 | - **`thread`** — sugar for a comment/reply thread capability. |
| 2806 | - **`object`** — mounts an arbitrary object's own rendered output inline. |
| 2807 | |
| 2808 | Every kind subscribes to its source collection and re-renders live, the |
| 2809 | same as `list`/`related`. An unrecognized kind (or one missing a required |
| 2810 | field) renders a small ""unsupported"" placeholder card rather than breaking |
| 2811 | the page. |
| 2812 | |
| 2813 | ## Why this is the point |
| 2814 | |
| 2815 | Adding an app is a schema file. The list, the table, the board, the toggle, |
| 2816 | the filters, the form, the detail page, badges, the row cap, relation labels, |
| 2817 | realtime, and owner actions all fall out of it — the same code for every app. |
| 2818 | The measure isn't the feature list (anyone can claim those); it's the amount of |
| 2819 | per-app code, which is a schema and at most one page object. |
| 2820 | |
| 2821 | ## This is not scaffolding |
| 2822 | |
| 2823 | Rails scaffolds and Django's startapp share one failure that defined a |
| 2824 | generation of frameworks: the generator emits **code**, you edit it, and from |
| 2825 | that moment you own all of it. The generator can never help that screen again |
| 2826 | — template improvements don't flow, upgrades don't apply, and nothing records |
| 2827 | which parts you changed. The fork happens at generation time and diverges |
| 2828 | silently forever. ""Eject"" is the same failure with a button on it. |
| 2829 | |
| 2830 | The rule that keeps this system on the other side of that line: |
| 2831 | |
| 2832 | > **Generated output is never materialized, so it can never be edited.** |
| 2833 | > A page is a view record *interpreted at request time* by a renderer that is |
| 2834 | > itself an ordinary object. Customizing never means editing emitted code, |
| 2835 | > because there is no emitted code. You either change **data** (and keep |
| 2836 | > receiving renderer improvements) or you write **your own object** (which was |
| 2837 | > never generated, so nothing was forked). |
| 2838 | |
| 2839 | ### The customization ladder |
| 2840 | |
| 2841 | Each rung says exactly what you take ownership of and what keeps flowing: |
| 2842 | |
| 2843 | 1. **Edit the block config** (filters, `title_field`, sort, `row_limit`) — you |
| 2844 | own the config; every renderer improvement still flows. |
| 2845 | 2. **Add blocks** — a view is a list of blocks; put an `aggregate` summary or |
| 2846 | `markdown` intro above the generated list. Same ownership as (1). |
| 2847 | 3. **Customize a shipped view record** — the upgrade system detects it against |
| 2848 | the package baseline and parks a reconcile instead of clobbering: the system |
| 2849 | *records* what you own and shows you when upstream moved. Scaffolding's |
| 2850 | fork is silent; this one has a ledger. |
| 2851 | 4. **Drop an `object` block into a generated page** — one hand-written panel |
| 2852 | among generated ones (`{""kind"": ""object"", ""object_id"": ""site_my_panel""}`). |
| 2853 | You own that panel; the rest of the page keeps flowing. It runs through the |
| 2854 | ordinary execution path, so permissions and audit apply — the block widens |
| 2855 | what a page can *show*, never what a viewer may *see*. |
| 2856 | 5. **Write the whole page object.** Convention routing means creating |
| 2857 | `site/notes.py` instantly overrides the generated `/notes` — and deleting |
| 2858 | the file reverts to it. The override is total, explicit, and reversible, |
| 2859 | and it is visible in the objects listing rather than buried in a diff |
| 2860 | against generator output. |
| 2861 | |
| 2862 | ### Why replacing a page is safe here and wasn't in Rails |
| 2863 | |
| 2864 | The deep failure of scaffolds wasn't the HTML — it was that the scaffold |
| 2865 | contained **rules** (validations in the controller, logic in the view), so |
| 2866 | editing it forked the rules. Here the doctrine is that *a page may never be |
| 2867 | the only place a rule exists*: enforcement lives in the schema, hooks, |
| 2868 | transitions, and permissions, on the shared write path that the JSON API, MCP |
| 2869 | agents, and the websocket all go through. |
| 2870 | |
| 2871 | That doctrine is not just policy — the architecture makes the wrong place |
| 2872 | **fail loudly**. A rule written into a page object simply does not gate API or |
| 2873 | MCP writes, so the mistake shows up the first time anything that isn't the |
| 2874 | page touches the collection. In Rails, logic in the controller *worked* for |
| 2875 | web users, which is exactly why the fork stayed invisible for years. The |
| 2876 | residual risk here is someone treating rung 4–5 as a place to put gates; the |
| 2877 | system's answer is that gates there provably don't hold, and the hook is one |
| 2878 | file away. |
| 2879 | |
| 2880 | ## Related |
| 2881 | |
| 2882 | - [`schema-forms.md`](/docs/schema-forms) — the field-level contract these |
| 2883 | renderers read. |
| 2884 | - [`ui-decisions.md`](/docs/ui-decisions) — the living log of interaction |
| 2885 | decisions (detail-vs-edit, the row cap, board flex, filters, relation labels, |
| 2886 | capabilities) and the reasons behind them. |
| 2887 | - [`design-system.md`](/docs/design-system) — semantic tokens, themes as data, and |
| 2888 | the stylesheet served as an object. |
| 2889 | - [`capabilities.md`](/docs/capabilities) — the behavior layer mounted on detail. |
| 2890 | " |
| 2891 | design-system The Design System Sites, UI & Realtime 420 How DBBASIC's design system works as a semantic contract with a thin, themeable renderer, delivered as a live object rather than a static component library. "DBBASIC's design system is not a component library you depend on — it is |
| 2892 | a **semantic contract with a thin, themeable renderer, delivered as an |
| 2893 | object.** It has three layers, and only the top one is pixels. |
| 2894 | |
| 2895 | ```mermaid |
| 2896 | flowchart TD |
| 2897 | SEM[""Layer 1 — Semantics<br/>schema: types, enum, relation,<br/>list_mode, transitions, permissions""] --> REND |
| 2898 | TOK[""Layer 2 — Tokens<br/>semantic roles: surface, text,<br/>accent, positive/warning/danger""] --> REND |
| 2899 | REND[""Layer 3 — Renderers<br/>web pages · Scroll · shell · AI""] |
| 2900 | TOK -.theme = a set of role values.-> THEMES[""Themes as data:<br/>base · paper · terminal · yours""] |
| 2901 | ``` |
| 2902 | |
| 2903 | ## The governing idea: define once, project everywhere, own no copies |
| 2904 | |
| 2905 | Everything below is one principle seen from different angles. State the |
| 2906 | design once — as tokens, as a shared stylesheet object, as a nav object, |
| 2907 | as a schema — and let every surface *project* it. No page owns a copy of |
| 2908 | the theme; it references `/style`. No page owns a copy of the nav; it |
| 2909 | includes `/nav`. No app hardcodes a widget; it declares a field's |
| 2910 | meaning. Because nothing downstream owns a copy, changing the source |
| 2911 | changes the whole product at once, with nothing to chase. |
| 2912 | |
| 2913 | This is why rebranding the entire twelve-app suite and giving it a shared |
| 2914 | navigation bar was **two objects edited live** — no deploy, no migration, |
| 2915 | no per-page rework. If a future change to the look or the chrome ever |
| 2916 | requires editing many pages, that is the signal that a copy leaked in and |
| 2917 | should be pulled back into a shared object. Guard this property; it is the |
| 2918 | whole value. |
| 2919 | |
| 2920 | ## Layer 1 — Semantics (the source of truth) |
| 2921 | |
| 2922 | Design decisions live in the schema, not in markup. A field that declares |
| 2923 | `enum` *means* ""one of these values""; each surface picks its own control |
| 2924 | (dropdown on the web, segmented control in Scroll, a spoken list to the |
| 2925 | AI). `views.list_mode` picks table / cards / feed. `transitions` says |
| 2926 | which state moves are legal. Permissions decide which controls exist at |
| 2927 | all. See [schema forms](/docs/schema-forms) and [permissions](/docs/permissions-model). |
| 2928 | |
| 2929 | **We deliberately do not use a widget-prescriptive `ui_schema`.** A |
| 2930 | predecessor system paired each schema with a React-JSON-Schema-Form |
| 2931 | `ui_schema` that said things like `""ui:widget"": ""radio""`. That only ever |
| 2932 | renders on one surface — a `radio` hint is meaningless to an AI, the |
| 2933 | shell, a voice interface, or a native app. Semantics render everywhere; |
| 2934 | widget instructions render once. This is the single most important choice |
| 2935 | in the system, and the reason a design here can span surfaces that do not |
| 2936 | exist yet. |
| 2937 | |
| 2938 | **Permissions are affordances.** A control exists because the caller can |
| 2939 | perform the action, not because markup drew it and then hid it. A delete |
| 2940 | button appears when the policy allows delete; a project filter appears |
| 2941 | because there is a `project_id` relation; a column appears because |
| 2942 | `list_fields` lists it. The design system renders *capability* — which is |
| 2943 | why an AI acting for a user, or a shared component, can never show a |
| 2944 | control the user could not actually use. |
| 2945 | |
| 2946 | ## Layer 2 — Tokens (roles, not colors) |
| 2947 | |
| 2948 | The visual vocabulary is a small set of **semantic roles** — surface |
| 2949 | (`bg`, `panel`, `line`), text (`text`, `muted`), `accent`, and status |
| 2950 | (`positive`, `warning`, `danger`) — plus spacing, radii, shadow, and two |
| 2951 | type families. Components are written against the roles, never against |
| 2952 | raw colors, so a retheme changes values in one place and nothing in any |
| 2953 | page moves. |
| 2954 | |
| 2955 | ## Layer 3 — Renderers, and the design system *as an object* |
| 2956 | |
| 2957 | The stylesheet is itself a DBBASIC object: **`site_style`, served at |
| 2958 | `/style`.** Every page links to it (`<link rel=""stylesheet"" href=""/style"">`) |
| 2959 | instead of carrying its own CSS. Because it is an object, the whole visual |
| 2960 | system is versioned, rollback-able, and live-editable like everything |
| 2961 | else — retheming is one reversible object change, not a deploy. |
| 2962 | |
| 2963 | Other renderers consume the same two layers: Scroll's generated Flutter |
| 2964 | forms and lists, the shell's terminal, and the AI (whose ""UI"" is the |
| 2965 | schema's `label`/`help` text — good microcopy is design work that serves |
| 2966 | the form, the accessibility label, and the AI tool description at once). |
| 2967 | |
| 2968 | ## The app shell — chrome contributed, not copied |
| 2969 | |
| 2970 | Navigation is part of the design system, and it follows the same rule. |
| 2971 | The **`site_nav` object, served at `/nav`,** injects one shared top bar |
| 2972 | into every page that includes `<script src=""/nav""></script>`: brand, an |
| 2973 | app switcher across installed apps, global search (`⌘K` / `Ctrl-K`) over |
| 2974 | `/api/search`, an Ask-AI link to the shell, a notification bell reading |
| 2975 | the `notifications` collection, and a user menu (appearance, sign out). |
| 2976 | Its styling lives in `/style`, so it themes with everything else. |
| 2977 | |
| 2978 | The point is that a *package* (`app-theme`) contributes this bar to every |
| 2979 | other app without those apps knowing — a thing a Django app could never do |
| 2980 | to its siblings. Adding a nav item is one edit to one object; every app |
| 2981 | gets it. If you find yourself pasting nav markup into a page, stop: add it |
| 2982 | to `site_nav`. |
| 2983 | |
| 2984 | The bell is **live**: it subscribes to the `notifications` collection over |
| 2985 | the `/ws` websocket and updates the instant something happens, with a 20s |
| 2986 | poll as backup. Every app page is live the same way — each subscribes to |
| 2987 | its collection and auto-refreshes (150ms debounced) when a record it can |
| 2988 | see changes, over the one shared socket, via |
| 2989 | `window.dbbasicSubscribe(collection, handler)`. Add a live list to a new |
| 2990 | page in one line. Auto-update on events is the thing older |
| 2991 | dashboard/Channels-based stacks never did cleanly — see the |
| 2992 | [realtime contract](/docs/http-api-contract#realtime-push-websocket). |
| 2993 | |
| 2994 | ## Themes are data — and packages |
| 2995 | |
| 2996 | A **theme is just a set of values for the token roles.** `site_style` |
| 2997 | ships three built in: |
| 2998 | |
| 2999 | - **base** — the DBBASIC identity: a warm dark (per the earth-theme rule |
| 3000 | ""warm, not cold blue-black"") with a terracotta/ember accent, carried |
| 3001 | forward from an earlier internal palette. |
| 3002 | - **paper** — a warm light theme. |
| 3003 | - **terminal** — high-contrast green-on-black. |
| 3004 | |
| 3005 | The active theme lives in the object's state. There is a visual chooser |
| 3006 | at **`/appearance`** (the `site_appearance` object): it reads the |
| 3007 | available themes and their preview swatches from `/style?info=true` and |
| 3008 | lets an admin click one to reskin the whole instance; non-admins see the |
| 3009 | active theme read-only. Or switch over HTTP: |
| 3010 | |
| 3011 | ```http |
| 3012 | POST /style {""theme"": ""terminal""} # admin session only |
| 3013 | POST /style {""tokens"": {""accent"": ""#5aa7ff""}} # custom role overrides |
| 3014 | GET /style?info=true # {active, available, previews, tokens} |
| 3015 | ``` |
| 3016 | |
| 3017 | `site_style` self-gates writes to admin sessions (using the unspoofable |
| 3018 | injected identity), so a public `execute` grant can serve the CSS to |
| 3019 | everyone while only operators change the look. |
| 3020 | |
| 3021 | **A theme is a package.** Installing a package that ships its own |
| 3022 | `site_style` (or that sets custom tokens) reskins the whole instance — |
| 3023 | design distributed the same way apps are. A community can publish themes; |
| 3024 | you install the one you like; you can fork it because it is one small, |
| 3025 | readable object. Per-user theming (each account choosing a theme) is a |
| 3026 | planned extension built on the same roles plus the `shell_preferences` |
| 3027 | pattern. |
| 3028 | |
| 3029 | ## Forms, tables, search, and filters |
| 3030 | |
| 3031 | The base stylesheet carries a genuinely complete component vocabulary so |
| 3032 | generated UIs and hand-written pages share one look: form fields with |
| 3033 | labels/help/required markers and validation states; tables with sortable |
| 3034 | headers, hover, numeric alignment, and pagination; a toolbar with a search |
| 3035 | input and filter chips; buttons as **intents** (primary / neutral / |
| 3036 | danger) rather than colors; badges; and first-class empty / loading / |
| 3037 | denied / error states. |
| 3038 | |
| 3039 | That is the visual layer. The **generative renderer** that reads a schema |
| 3040 | and emits these — the successor to the declarative table/filter libraries |
| 3041 | (`django_tables2` + `django_filters`) used in an earlier internal system, |
| 3042 | which worked because they were declarative from a single source — unified |
| 3043 | across table, cards, and feed with search and filters wired to |
| 3044 | `search.fields` — is the next build. The honest lesson from that earlier |
| 3045 | system is that this substrate must be *one* renderer, not the four |
| 3046 | disconnected implementations it grew (`django_tables2` for tables, two |
| 3047 | incompatible form engines, hand-written cards and feeds). The semantics |
| 3048 | and the token vocabulary are now in place for it. |
| 3049 | |
| 3050 | ## Building a page against the design system |
| 3051 | |
| 3052 | ```html |
| 3053 | <link rel=""stylesheet"" href=""/style""> |
| 3054 | ... |
| 3055 | <div class=""wrap""> |
| 3056 | <header class=""app""><h1>Notes</h1><div class=""who"">…</div></header> |
| 3057 | <div class=""toolbar""> |
| 3058 | <input class=""search grow"" placeholder=""Search…""> |
| 3059 | <div class=""filters""><button class=""chip"" aria-pressed=""true"">All</button></div> |
| 3060 | </div> |
| 3061 | <div class=""cards"">…</div> |
| 3062 | </div> |
| 3063 | ``` |
| 3064 | |
| 3065 | No inline CSS, no framework, no build step — just the shared classes. When |
| 3066 | the instance's theme changes, this page changes with it. Page-unique |
| 3067 | layout (a form's column grid, an edit-mode toggle) may stay in a small |
| 3068 | `<style>` block, but it must use the token variables (`var(--panel)`, |
| 3069 | `var(--line)`, `var(--accent)`) — never hardcoded colors — so it reskins |
| 3070 | too. |
| 3071 | |
| 3072 | ## Why this is a design system, not a style guide |
| 3073 | |
| 3074 | The test that separates the two: **change the source, and does the whole |
| 3075 | product change with no copies to chase?** Here it does — edit `/style`, |
| 3076 | every page reskins; edit `/nav`, every page's chrome updates; edit a |
| 3077 | schema, every surface re-renders that field. A folder of CSS classes and a |
| 3078 | component catalog is a style guide; you still hand-apply it and it drifts. |
| 3079 | A design system propagates from one place. Protect that difference: the |
| 3080 | day a change stops propagating is the day it quietly became a style guide. |
| 3081 | |
| 3082 | ## Why not a dashboard template |
| 3083 | |
| 3084 | The web-template industry (Bootstrap admin themes, and the marketplaces |
| 3085 | that sell them — an earlier internal project shipped on the ""Volt"" |
| 3086 | dashboard) assumed **everything is a dashboard**: a fixed chrome of |
| 3087 | sidebar, topbar, cards, and a data table, into which you pour your app. |
| 3088 | That is widgets-before-meaning — the shape is the product, and the only |
| 3089 | app you can express is the one the widgets already draw. It is the same |
| 3090 | mistake as a widget-prescriptive `ui_schema`, at the scale of a whole |
| 3091 | product. It is also why that earlier project accumulated 110 KB of |
| 3092 | override CSS *fighting* its template: when the chrome is copied into |
| 3093 | every page, changing it means fighting it everywhere. |
| 3094 | |
| 3095 | Reading modes (`table` / `cards` / `feed`) are the direct rejection of the |
| 3096 | one-shape assumption: a note is read individually (cards), a task is |
| 3097 | compared to others (table), a message is sequential (feed). The dashboard |
| 3098 | world only had the table. |
| 3099 | |
| 3100 | The reason no one built it this way earlier is not taste, it is |
| 3101 | arithmetic. Projecting one schema into a web form, a table, a Flutter |
| 3102 | widget, an AI tool description, and an accessibility label meant |
| 3103 | hand-writing five renderers per app — insane in 2015, so buying the frozen |
| 3104 | template was rational. What changed is that the schema is now legible to a |
| 3105 | model, a model can render a surface, and the runtime hot-loads it with no |
| 3106 | deploy — so the marginal cost of a surface collapsed, and the correct |
| 3107 | architecture (declare meaning once, project everywhere) finally pays. Build |
| 3108 | for that, not for the dashboard era. |
| 3109 | |
| 3110 | ## Current edges (honest) |
| 3111 | |
| 3112 | - **Theme-complete, component-incomplete.** The token/theme/app-shell |
| 3113 | layers are done and propagate. The component *vocabulary* exists in |
| 3114 | `/style`, but pages still assemble it by hand — the **generative |
| 3115 | renderer** (schema → dense list with row actions, filters, and relative |
| 3116 | timestamps) is not built yet. It is the next big design build, and it |
| 3117 | needs a `created_at` field added to schemas first (nothing captures |
| 3118 | creation time today, so there are no relative timestamps to show). |
| 3119 | - **Web + tokens today.** Scroll shares the token *roles* but the web and |
| 3120 | Flutter renderers are not unified against one contract yet. |
| 3121 | - **Instance-level theming.** Per-user theme choice is a planned extension |
| 3122 | on the same roles plus `shell_preferences`. |
| 3123 | - **The shell page** (`/shell`) deliberately keeps its own terminal palette |
| 3124 | and does not link `/style` or `/nav` — a focused full-screen mode. The |
| 3125 | admin dashboard is not yet on the shared shell either. |
| 3126 | |
| 3127 | ## Extending it correctly — rules for future builds |
| 3128 | |
| 3129 | When adding to the design system, keep the propagation property intact: |
| 3130 | |
| 3131 | 1. **Contribute, don't copy.** New chrome goes in a shared object |
| 3132 | (`site_style`, `site_nav`), never pasted into pages. If it must live in |
| 3133 | a page, it is page-unique and must use tokens. |
| 3134 | 2. **Tokens are roles, never hex.** Add a role (e.g. `--info`) if you need |
| 3135 | a new meaning; never a raw color in a component or a page. |
| 3136 | 3. **Semantics, not widgets.** Express new capability as schema meaning |
| 3137 | (a new field key, a new `views` mode) that every surface can interpret |
| 3138 | — resist any per-surface ""render as X"" instruction; that is `ui_schema` |
| 3139 | returning, and it collapses you back to one surface. |
| 3140 | 4. **Let permissions draw the controls.** Show an action because the |
| 3141 | policy allows it, not because you wrote the button. |
| 3142 | 5. **One renderer per concern.** When the generative renderer lands, route |
| 3143 | all lists/forms through it — do not grow a second form engine. A |
| 3144 | previous system grew four; that fragmentation is the failure mode to |
| 3145 | avoid. |
| 3146 | 6. **No framework to fight.** Do not adopt a CSS framework you will then |
| 3147 | override. The system is small and authoritative on purpose. |
| 3148 | 7. **If a change stops propagating, fix the leak** — pull the copy back |
| 3149 | into the shared object — rather than editing many pages. |
| 3150 | |
| 3151 | Follow these and the next person (or the next model, or a stranger who |
| 3152 | forks this) can retheme, re-chrome, or extend the whole product from a |
| 3153 | handful of objects — which is the point. |
| 3154 | " |
| 3155 | rest-and-object-messages REST and Object Messages Sites, UI & Realtime 430 How the DBBASIC Object Server splits its HTTP API into RESTful resources, object messages, and realtime streams — and why that split matters. "DBBASIC should be RESTful where it exposes resources and direct where it executes behavior. |
| 3156 | |
| 3157 | The goal is not to pretend every operation is pure REST. The goal is to keep a clean split. The HTTP API breaks down into three kinds of surface: |
| 3158 | |
| 3159 | - **Resources** — `source`, `state`, `logs`, `files`, `versions`, `metadata` |
| 3160 | - **Object messages** — execute method, run job, handle webhook, perform admin action |
| 3161 | - **Streams** — WebSocket events, Server-Sent Events |
| 3162 | |
| 3163 | ## Why This Matters |
| 3164 | |
| 3165 | Many APIs called REST are really RPC over HTTP: |
| 3166 | |
| 3167 | ```text |
| 3168 | POST /doThing |
| 3169 | POST /executeAction |
| 3170 | POST /api/run |
| 3171 | ``` |
| 3172 | |
| 3173 | That shape is easy to add, but it hides what the system actually owns. DBBASIC objects own real resources: source, state, logs, files, versions, and metadata. Those should be exposed as resources with stable paths, predictable HTTP methods, status codes, and JSON representations. |
| 3174 | |
| 3175 | Object behavior is different. Executing an object is message passing. Jobs, events, webhooks, and realtime commands are also messages. They should be documented as messages instead of disguised as fake CRUD resources. |
| 3176 | |
| 3177 | ## RESTful Resource Surface |
| 3178 | |
| 3179 | DBBASIC should keep these resource-style endpoints boring and predictable: |
| 3180 | |
| 3181 | ```text |
| 3182 | GET /objects |
| 3183 | POST /objects |
| 3184 | GET /objects/{object_id}?source=true |
| 3185 | PUT /objects/{object_id}?source=true |
| 3186 | GET /objects/{object_id}?state=true |
| 3187 | GET /objects/{object_id}?logs=true |
| 3188 | GET /objects/{object_id}?versions=true |
| 3189 | GET /objects/{object_id}?version=N |
| 3190 | DELETE /objects/{object_id}?destroy=true |
| 3191 | ``` |
| 3192 | |
| 3193 | These endpoints expose things the object server owns. |
| 3194 | |
| 3195 | They should preserve: |
| 3196 | |
| 3197 | - stable object identity |
| 3198 | - standard HTTP status codes |
| 3199 | - JSON response shapes existing clients can keep using |
| 3200 | - explicit authorization checks |
| 3201 | - useful cache behavior where safe |
| 3202 | - compatibility with the HTTP API contract |
| 3203 | |
| 3204 | ## Object Message Surface |
| 3205 | |
| 3206 | Object execution is not only CRUD. It is behavior: |
| 3207 | |
| 3208 | ```text |
| 3209 | GET /objects/{object_id} |
| 3210 | POST /objects/{object_id} |
| 3211 | PUT /objects/{object_id} |
| 3212 | DELETE /objects/{object_id} |
| 3213 | ``` |
| 3214 | |
| 3215 | Those calls deliver a method and payload to the object. |
| 3216 | |
| 3217 | That model is closer to: |
| 3218 | |
| 3219 | ```text |
| 3220 | object_id + method + payload -> result |
| 3221 | ``` |
| 3222 | |
| 3223 | than: |
| 3224 | |
| 3225 | ```text |
| 3226 | table + row + CRUD operation |
| 3227 | ``` |
| 3228 | |
| 3229 | This is intentional. A DBBASIC object can represent a report, worker, webhook, view, business process, queue handler, generated UI, or admin tool. |
| 3230 | |
| 3231 | ## Rollback |
| 3232 | |
| 3233 | Rollback is the awkward case: |
| 3234 | |
| 3235 | ```text |
| 3236 | POST /objects/{object_id} |
| 3237 | {""action"": ""rollback"", ""version_id"": 1} |
| 3238 | ``` |
| 3239 | |
| 3240 | That is a message, not a pure REST resource update. It is kept for compatibility with existing clients. |
| 3241 | |
| 3242 | Future server code may add a cleaner resource-style alias, but it should not break the existing message shape without a migration: |
| 3243 | |
| 3244 | ```text |
| 3245 | POST /objects/{object_id}/versions/{version_id}/rollback |
| 3246 | ``` |
| 3247 | |
| 3248 | ## Design Rule |
| 3249 | |
| 3250 | Do not turn every operation into `/api/doThing`. |
| 3251 | |
| 3252 | Use resources for object-owned data: |
| 3253 | |
| 3254 | ```text |
| 3255 | source, state, logs, files, versions, metadata |
| 3256 | ``` |
| 3257 | |
| 3258 | Use messages for object-owned behavior: |
| 3259 | |
| 3260 | ```text |
| 3261 | execute, run, handle, repair, migrate, broadcast |
| 3262 | ``` |
| 3263 | |
| 3264 | Use streams for object-owned realtime feedback: |
| 3265 | |
| 3266 | ```text |
| 3267 | source changed, state changed, log appended, version saved, execution failed |
| 3268 | ``` |
| 3269 | |
| 3270 | That is the honest shape of DBBASIC: REST where resources are real, object messages where behavior is real, and streams where live feedback is real. |
| 3271 | " |
| 3272 | realtime Realtime: The Change Log and Live Surfaces Sites, UI & Realtime 440 How DBBASIC Object Server pushes change notifications over a websocket without ever putting record data on the wire, and why that design choice matters. "Every generative surface — every list, table, board, detail page, comment thread, and attachment list — subscribes to the change log and re-renders when a record it can see changes, from any tab, user, or agent. This falls out of one small websocket contract with a deliberate rule: **the wire carries a signal, never the record body.** |
| 3273 | |
| 3274 | ```mermaid |
| 3275 | sequenceDiagram |
| 3276 | participant Writer as any write (web/API/agent) |
| 3277 | participant Server |
| 3278 | participant Hub as realtime hub |
| 3279 | participant Client as a live surface |
| 3280 | |
| 3281 | Client->>Server: connect /ws (session), subscribe [collections] |
| 3282 | Server->>Server: may this subject READ each collection? |
| 3283 | Server-->>Client: {type: subscribed, collections} |
| 3284 | Writer->>Server: create/update/delete a record |
| 3285 | Server->>Hub: publish {collection, record_id, action} |
| 3286 | Hub->>Hub: per subscriber, re-run the READ decision |
| 3287 | Hub-->>Client: {type: record, collection, record_id, action} |
| 3288 | Client->>Server: refetch through the normal permission-enforced API |
| 3289 | Client->>Client: re-render |
| 3290 | ``` |
| 3291 | |
| 3292 | ## Transport |
| 3293 | |
| 3294 | `GET /ws` (upgraded to a websocket). The connection is **authenticated** — a session cookie or bearer token — and anonymous connections are refused (`1008`). Realtime is behind a flag; when off, surfaces fall back to polling. |
| 3295 | |
| 3296 | ## Client protocol |
| 3297 | |
| 3298 | Messages are JSON. On accept, the server sends `{""type"": ""welcome"", ""user"": ...}`. Then: |
| 3299 | |
| 3300 | | Client sends | Server replies | |
| 3301 | |--------------|----------------| |
| 3302 | | `{""action"": ""subscribe"", ""collections"": [""tasks"", ""notes""]}` | `{""type"": ""subscribed"", ""collections"": [...]}` — only the collections the subject may read are kept | |
| 3303 | | `{""action"": ""unsubscribe"", ""collections"": [...]}` | `{""type"": ""subscribed"", ""collections"": [...]}` | |
| 3304 | | `{""action"": ""ping""}` | `{""type"": ""pong""}` | |
| 3305 | |
| 3306 | Subscriptions are capped per connection, and each collection name is validated. |
| 3307 | |
| 3308 | In the browser this is wrapped as one helper, `window.dbbasicSubscribe(collection, handler)` (served by the `nav` object): it opens the socket once, subscribes on demand, reconnects with backoff, and runs a ~20-second poll as a fallback for missed events and multi-worker deployments. A page never talks to `/ws` directly — it calls `dbbasicSubscribe` and re-renders in the handler. |
| 3309 | |
| 3310 | ## The event |
| 3311 | |
| 3312 | A change pushes exactly: |
| 3313 | |
| 3314 | ```json |
| 3315 | { ""type"": ""record"", ""collection"": ""tasks"", ""record_id"": ""…"", ""action"": ""update"" } |
| 3316 | ``` |
| 3317 | |
| 3318 | `action` is `create` / `update` / `delete`. **No field values travel** — only which record changed. The client reacts by *refetching* through the ordinary API, which re-applies the full permission policy and field redaction. So a live surface can never show a field the API wouldn't have served. |
| 3319 | |
| 3320 | ## Permission filtering — twice |
| 3321 | |
| 3322 | Realtime is inside the permission engine, not beside it: |
| 3323 | |
| 3324 | 1. **On subscribe** — a subject may follow a collection only if it may `READ` it (`_ws_can_subscribe` runs the same check a `GET` would). |
| 3325 | 2. **On publish** — for each subscriber, the server re-runs the READ decision *against the changed record*, so a row-filtered subscriber only hears about rows its filter would let it see. If the policy can't be loaded to filter safely, nothing is pushed. |
| 3326 | |
| 3327 | Combined with the signal-only payload, this means the change log leaks neither the existence of records you can't see (filtered per-subscriber) nor their contents (never sent). |
| 3328 | |
| 3329 | ## Why signal, not body |
| 3330 | |
| 3331 | - **Security** — the wire can't leak a field, because it carries none. All reads go back through the one enforced path. |
| 3332 | - **Simplicity** — there is no second serialization/redaction path to keep in sync with the API; the client just refetches. |
| 3333 | - **Correctness under races** — the refetch always returns the current row, so a burst of edits collapses to one correct re-render rather than replaying stale bodies. |
| 3334 | |
| 3335 | The durable, poll-based event log remains the fallback for missed pushes and for deployments with more than one worker process. |
| 3336 | |
| 3337 | ## Related reading |
| 3338 | |
| 3339 | - **ASGI and realtime direction** — why the server uses plain ASGI and how websocket/SSE object events fit that direction. |
| 3340 | - **Generative UI** — the surfaces that subscribe to the change log. |
| 3341 | - **Permissions model** — the READ decision reused on subscribe and on every push. |
| 3342 | - **Runtime contract** — the daemon and event contracts behind the durable log. |
| 3343 | " |
| 3344 | asgi-realtime-direction ASGI and Realtime Direction Sites, UI & Realtime 450 Why DBBASIC's ASGI layer stays close to persistent CGI, what it borrows from Phoenix/Erlang, and how the websocket realtime overlay sits on top of the durable event log. "The working prototype uses plain ASGI with Uvicorn, not Django ASGI, Channels, FastAPI, or Starlette. |
| 3345 | |
| 3346 | The shape is simple: |
| 3347 | |
| 3348 | ```mermaid |
| 3349 | flowchart LR |
| 3350 | A[""Uvicorn""] --> B[""ASGI app""] |
| 3351 | B --> C[""Filesystem router""] |
| 3352 | C --> D[""api/*.py handlers""] |
| 3353 | ``` |
| 3354 | |
| 3355 | That keeps the useful part of classic CGI: |
| 3356 | |
| 3357 | ```mermaid |
| 3358 | flowchart LR |
| 3359 | A[""URL""] --> B[""File""] |
| 3360 | B --> C[""Function""] |
| 3361 | ``` |
| 3362 | |
| 3363 | but without classic CGI's process-per-request cost. |
| 3364 | |
| 3365 | That is a better comparison than saying DBBASIC is a Rails-like app structure. The goal is closer to persistent CGI with a reusable object runtime: |
| 3366 | |
| 3367 | - filesystem paths can map to runnable code |
| 3368 | - the server stays warm between requests |
| 3369 | - routing is derived from object source placement |
| 3370 | - common helpers provide state, logs, files, versions, and scheduling |
| 3371 | - object boundaries stay small enough for humans and AI tools to inspect |
| 3372 | |
| 3373 | Classic CGI made deployment easy but left too much work to each script. DBBASIC keeps the direct path-to-code idea and adds the shared runtime pieces that make objects reusable as application building blocks. |
| 3374 | |
| 3375 | ## Why ASGI Matters |
| 3376 | |
| 3377 | ASGI gives DBBASIC one server surface for: |
| 3378 | |
| 3379 | - normal HTTP requests |
| 3380 | - long-lived WebSocket connections |
| 3381 | - Server-Sent Events |
| 3382 | - startup and shutdown lifecycle hooks |
| 3383 | |
| 3384 | That matters because DBBASIC objects are not only pages or REST handlers. They can also be jobs, event handlers, admin actions, realtime screens, data feeds, and AI-edited live application pieces. |
| 3385 | |
| 3386 | The important loop is: |
| 3387 | |
| 3388 | ```mermaid |
| 3389 | flowchart LR |
| 3390 | A[""Object changes""] --> B[""Version saved""] |
| 3391 | B --> C[""Object event""] |
| 3392 | C --> D[""UI/client updates""] |
| 3393 | ``` |
| 3394 | |
| 3395 | HTTP can handle the command. WebSockets or SSE can carry the feedback. |
| 3396 | |
| 3397 | ## Phoenix and Erlang |
| 3398 | |
| 3399 | Phoenix and Erlang/BEAM show why realtime application systems can feel so much better than request/response-only web frameworks: |
| 3400 | |
| 3401 | - long-lived connections are normal |
| 3402 | - PubSub is a core idea |
| 3403 | - server-side events can update clients quickly |
| 3404 | - stateful processes can model active parts of the application |
| 3405 | - failures are isolated and supervised |
| 3406 | |
| 3407 | DBBASIC should learn from that direction. |
| 3408 | |
| 3409 | The DBBASIC version is not Erlang and does not get BEAM supervision for free. Python ASGI does not automatically give per-object process isolation, hot code replacement, distributed process registries, or Phoenix PubSub semantics. |
| 3410 | |
| 3411 | What DBBASIC can still use well is the product shape: |
| 3412 | |
| 3413 | ```mermaid |
| 3414 | flowchart TD |
| 3415 | A[""Object source""] |
| 3416 | B[""Object state""] |
| 3417 | C[""Object logs""] |
| 3418 | D[""Object versions""] |
| 3419 | E[""Realtime object events""] |
| 3420 | A --> E |
| 3421 | B --> E |
| 3422 | C --> E |
| 3423 | D --> E |
| 3424 | ``` |
| 3425 | |
| 3426 | That can support live admin screens, Scroll object inspectors, AI repair loops, dashboards, collaborative editing, queues, and business app interfaces without forcing everything through a Rails/Django-style request/deploy cycle. |
| 3427 | |
| 3428 | ## Public v1 Direction |
| 3429 | |
| 3430 | The public v1 server stays single-node first and keeps clustering separate until replication and conflict behavior are proven. |
| 3431 | |
| 3432 | The first public ASGI layer is small and runnable. It includes: |
| 3433 | |
| 3434 | - `GET /health` |
| 3435 | - `GET /objects?format=json` |
| 3436 | - `GET /objects/{object_id}?source=true&format=json` |
| 3437 | - object execution |
| 3438 | - source update and rollback |
| 3439 | - state/log/version endpoints |
| 3440 | - metadata endpoints |
| 3441 | - temporary admin-token gate for object listing and introspection reads |
| 3442 | |
| 3443 | Role, object, and row-level permissions, resource limits, and backup/restore checks have all shipped. The websocket event stream is now implemented too (see below). |
| 3444 | |
| 3445 | ## Realtime: Implemented |
| 3446 | |
| 3447 | Realtime arrived as an event system, not a decorative feature, and it reuses the durable event log rather than inventing a parallel path: |
| 3448 | |
| 3449 | - **Durable log (source of truth).** Every record write already publishes `collection.record.created/updated/deleted` to `object_events`, which is persisted, capped, and pollable with a cursor. This is the reliable, cross-restart, cross-worker path. |
| 3450 | - **Live overlay (`/ws`).** An in-process pub/sub hub (`object_realtime`) pushes those same events to connected websockets the instant they happen. See the [HTTP API contract](/docs/http-api-contract#realtime-push-websocket) for the connect + subscribe protocol. |
| 3451 | |
| 3452 | Events are boring JSON signals — collection, record id, action — never the record body, so a push can never leak a field. Delivery obeys the permission engine: you can only subscribe to a collection you may read, and with enforcement on you only receive events for records your row filters would show you. The client refetches through the normal enforced API. |
| 3453 | |
| 3454 | **Single-process assumption.** The live hub lives in one process; with one Uvicorn worker every write and every socket share the event loop, so push is immediate and consistent. Across multiple workers each has its own hub, and clients fall back to polling the durable log — which is why the durable log, not the socket, is the source of truth. Keep the public v1 single-worker (or add a shared broker before scaling workers) so live push stays whole; either way correctness never depends on the socket. |
| 3455 | |
| 3456 | An object-scoped room model (`/ws/objects/{object_id}` for source/state/log changes) is a natural next addition on the same hub, using the existing source/state/version change events. |
| 3457 | |
| 3458 | ## Design Rule |
| 3459 | |
| 3460 | Keep the ASGI layer thin. |
| 3461 | |
| 3462 | The server should route requests and streams to object operations. It should not become a second framework with its own competing object model. |
| 3463 | </content> |
| 3464 | </StructuredOutput> |
| 3465 | |
| 3466 | " |
| 3467 | shell-and-ai The Shell and AI — Talk to the Whole System AI 500 How the DBBASIC shell routes instant record commands, search, and AI chat through the same MCP tools and permission engine, plus voice and web-reading features. "The shell (`/shell`) is one input that reaches everything: instant |
| 3468 | record commands, global search, and an AI that operates the server |
| 3469 | through the same MCP tools any agent uses — with **your key, your |
| 3470 | model choice, and your permissions**. |
| 3471 | |
| 3472 | ```mermaid |
| 3473 | flowchart TD |
| 3474 | U[""You type""] --> D{""prefix?""} |
| 3475 | D -->|""$ . ^""| REC[""Create note / task / link<br/>(records API, your session)""] |
| 3476 | D -->|""~""| SRCH[""GET /api/search""] |
| 3477 | D -->|""/""| CMD[""Built-ins: /help /key /model /tools /voice""] |
| 3478 | D -->|""anything else""| CHAT[""POST /api/ai/chat""] |
| 3479 | CHAT --> PROV[""AI provider<br/>(your stored key)""] |
| 3480 | PROV -->|""tool calls""| MCP[""MCP tool subset<br/>dispatched with YOUR credentials""] |
| 3481 | MCP --> PERM[""Permission engine + audit""] |
| 3482 | PERM --> PROV |
| 3483 | PROV --> REPLY[""Reply + tool log + token usage + cost""] |
| 3484 | ``` |
| 3485 | |
| 3486 | ## Setup: Your Key, Your Model |
| 3487 | |
| 3488 | AI features use each user's own provider key — the server stores it |
| 3489 | write-only and calls providers on your behalf. Key material never |
| 3490 | appears in any response, record, or backup. Two providers are supported |
| 3491 | today (`object_ai.SUPPORTED_SERVICES`): `anthropic` and `openai` — no |
| 3492 | Gemini, no local/on-device inference. An `openai` key, once stored, also |
| 3493 | covers the cloud voice paths below (`POST /api/tts` with `engine: ""openai""`, |
| 3494 | `POST /api/stt`) — one key, no separate signup for voice. |
| 3495 | |
| 3496 | In the shell: |
| 3497 | |
| 3498 | ``` |
| 3499 | /key anthropic sk-ant-... store a key (masked, never logged) |
| 3500 | /keys which services have keys |
| 3501 | /model anthropic:claude-haiku-4-5 pick a model (service:model) |
| 3502 | /model openai:gpt-5-mini or the other provider |
| 3503 | ``` |
| 3504 | |
| 3505 | Or over HTTP: `PUT /identity/users/{you}/service-keys` with your |
| 3506 | session (see the [HTTP API contract](/docs/http-api-contract)). |
| 3507 | |
| 3508 | ## Tool Subsets: Small Context, Configurable Power |
| 3509 | |
| 3510 | The AI is offered a **named subset** of the MCP tool catalog, not the |
| 3511 | whole thing. Small subsets keep the context small enough for fast, |
| 3512 | inexpensive models — and the subset is your safety dial: |
| 3513 | |
| 3514 | ``` |
| 3515 | /tools global_search,list_records,get_record,create_record conversation mode |
| 3516 | /tools global_search,list_objects,get_object_source,create_object,update_object_source,execute_object builder mode |
| 3517 | ``` |
| 3518 | |
| 3519 | Every tool call the model makes is dispatched through the server's own |
| 3520 | routing **with your credentials**: the AI can do exactly what you could |
| 3521 | do yourself — row filters, field redaction, and the audit trail apply. |
| 3522 | An AI acting for a user is never more powerful than the user. |
| 3523 | |
| 3524 | ## Conversation Memory |
| 3525 | |
| 3526 | The shell logs every exchange to the `shell_commands` collection (your |
| 3527 | history is just records — searchable and owner-scoped). A new browser |
| 3528 | session replays recent history and sends prior AI turns back with each |
| 3529 | message, so conversations resume. The server stays stateless about |
| 3530 | chats: `POST /api/ai/chat` accepts a `history` list, and what to |
| 3531 | remember is the client's choice. |
| 3532 | |
| 3533 | ## Cost Recording |
| 3534 | |
| 3535 | Every chat turn is priced server-side, not by the caller. `POST |
| 3536 | /api/ai/chat` reads token counts straight from the provider's own response, |
| 3537 | looks up the model's price in the `ai_prices` collection (editable records, |
| 3538 | never a hardcoded table -- so a price change is a data edit, not a |
| 3539 | deploy), and computes the cost in integer cents: `tokens * |
| 3540 | per_million_cents // 1_000_000`, input and output added separately, no |
| 3541 | float division on money. The result -- `tokens_in`, `tokens_out`, |
| 3542 | `cost_cents`, `model`, `provider` -- lands in the `ai_usage` collection, |
| 3543 | written by the chat handler itself with the caller's user id as actor, so |
| 3544 | the record can't be skipped or forged the way a client-written log could |
| 3545 | be. A model with no matching price row still gets its tokens recorded; |
| 3546 | only the cost is null. The same numbers come back in the chat response's |
| 3547 | `usage` object so a surface can show them immediately. |
| 3548 | |
| 3549 | ## Coding Without Coding |
| 3550 | |
| 3551 | With builder-mode tools, the AI can create and edit live objects: |
| 3552 | |
| 3553 | > make an object called site_dice that renders a page rolling two dice |
| 3554 | |
| 3555 | The model writes the source, calls `create_object`, and the page is |
| 3556 | live at `/dice` immediately — objects load per execution, so there is |
| 3557 | no deploy step between the AI writing code and the code serving |
| 3558 | traffic. Edits use `update_object_source`; every version lands in |
| 3559 | source history with rollback; and create/update responses report the |
| 3560 | methods the code actually exposes, so the model can self-correct. |
| 3561 | |
| 3562 | Object writes ride the admin gate (an admin-role session today), and |
| 3563 | source writes require `DBBASIC_ENABLE_SOURCE_WRITES=true` — the same |
| 3564 | boundaries that govern humans. |
| 3565 | |
| 3566 | ## Voice |
| 3567 | |
| 3568 | The shell doubles as a push-to-talk terminal. Speech in and speech out each |
| 3569 | have two paths: browser-native (free, on-device, always available if the |
| 3570 | browser supports it) and cloud (a real per-user OpenAI key, real cost, but |
| 3571 | ""not on device"" — works on browsers/hardware the native APIs don't cover, |
| 3572 | and sounds noticeably better than the local synthesis engines). |
| 3573 | |
| 3574 | - **Speech in, browser path**: the mic button uses `window.SpeechRecognition` |
| 3575 | (or the `webkit` prefix); interim words show live in the input, and the |
| 3576 | final transcript submits through the same form-submit path a typed line |
| 3577 | does — no separate send code. Neither Chrome nor Safari expose the API |
| 3578 | outside a secure context (`https://` or `localhost`), and without it the |
| 3579 | button simply stays hidden. This is the default (`talk_stt_engine: ""browser""`). |
| 3580 | - **Speech in, cloud path**: when `talk_stt_engine` is `""openai""`, the mic |
| 3581 | becomes push-to-talk instead of continuous listening — tap to start a |
| 3582 | real `MediaRecorder` capture, tap again to stop and upload the clip to |
| 3583 | `POST /api/stt`, which transcribes it via the caller's own stored OpenAI |
| 3584 | key and returns text. No wake word, no VAD silence endpoint: raw audio has |
| 3585 | no equivalent to `SpeechRecognition`'s live partial-result signal, so |
| 3586 | there is no live signal to drive one. See `POST /api/stt` below. |
| 3587 | - **Speech out, browser path**: `speechSynthesis.speak` — free, on-device, |
| 3588 | no server round-trip. |
| 3589 | - **Speech out, cloud path**: when `talk_tts_engine` is `""openai""`, assistant |
| 3590 | replies (markdown and code fences stripped first — only prose is spoken, |
| 3591 | never tool-call noise) POST to `POST /api/tts` with `""engine"": ""openai""` |
| 3592 | and play the returned `audio/mpeg`, via the caller's own stored OpenAI key. |
| 3593 | If the request fails for any reason, the browser's own `speechSynthesis` |
| 3594 | says the line instead, so voice mode never goes silent because of a |
| 3595 | server-side gap. |
| 3596 | |
| 3597 | `/voice`, `/voice on`, `/voice off` toggle voice mode inline (mirrors |
| 3598 | `/model`); the setting persists on `shell_preferences.voice_enabled` and is |
| 3599 | honored on page load like `ai_model` and `tools`. `talk_tts_engine` and |
| 3600 | `talk_stt_engine` (both default to the free/local path) live on the same |
| 3601 | `shell_preferences` record — the Talk page (`/talk`) additionally has a |
| 3602 | single in-page toggle that flips both together, since ""cloud voice"" is one |
| 3603 | combined mode to remember, not two settings. |
| 3604 | |
| 3605 | ### `POST /api/tts` |
| 3606 | |
| 3607 | ```http |
| 3608 | POST /api/tts |
| 3609 | Authorization: Bearer <session-token> (or the session cookie) |
| 3610 | |
| 3611 | {""text"": ""one note matches"", ""voice"": ""en-us"", ""engine"": ""local""} |
| 3612 | ``` |
| 3613 | |
| 3614 | Response: audio bytes (`audio/wav` for `engine: ""local""`, `audio/mpeg` for |
| 3615 | `engine: ""openai""`), or the usual `{""status"": ""error"", ""error""}` JSON shape |
| 3616 | on failure. Requires `DBBASIC_ENABLE_TTS=true` (off by default) and a |
| 3617 | signed-in session — same posture as `/api/ai/chat`. `text` is capped at 800 |
| 3618 | characters (413 beyond that). `engine` is optional, `""local""` or `""openai""`, |
| 3619 | defaulting to `""local""` — the two paths never silently fall back to each |
| 3620 | other; a caller who asked for cloud quality and hits a provider error sees |
| 3621 | that error. |
| 3622 | |
| 3623 | **`engine: ""local""`** (default): discovered at call time, first match wins: |
| 3624 | `espeak-ng`, `espeak`, then macOS `say` (development convenience — `say` |
| 3625 | writes AIFF and the also-stock `afconvert` turns it into WAV; if `afconvert` |
| 3626 | isn't present the endpoint returns 501 rather than growing a bespoke audio |
| 3627 | pipeline). No engine on `PATH` at all is a 503 with a clear message. |
| 3628 | |
| 3629 | **`engine: ""openai""`**: calls OpenAI's TTS API (`gpt-4o-mini-tts` by |
| 3630 | default) using the caller's own key, stored via `PUT |
| 3631 | /identity/users/{user_id}/service-keys` (`service: ""openai""`) — no key |
| 3632 | stored is a 400 with the exact PUT to fix it. A provider-side failure (bad |
| 3633 | key, rate limit) is a 502. `DBBASIC_TTS_CLOUD_TIMEOUT_SECONDS` bounds the |
| 3634 | outbound call (default 30s). |
| 3635 | |
| 3636 | Successful audio is cached at |
| 3637 | `data/tts-cache/{sha256(engine|voice|text[|model])}.wav` regardless of |
| 3638 | engine — a repeat of the same line is a cache read, not a re-synthesis. |
| 3639 | There's no eviction in v1; operators who want a bound should prune that |
| 3640 | directory on their own schedule. |
| 3641 | |
| 3642 | ### `POST /api/stt` |
| 3643 | |
| 3644 | ```http |
| 3645 | POST /api/stt |
| 3646 | Authorization: Bearer <session-token> (or the session cookie) |
| 3647 | Content-Type: audio/webm |
| 3648 | |
| 3649 | <raw audio bytes> |
| 3650 | ``` |
| 3651 | |
| 3652 | Response: `{""status"": ""ok"", ""text"": ""...""}`, or the usual |
| 3653 | `{""status"": ""error"", ""error""}` shape. The request body is the raw audio |
| 3654 | bytes themselves (whatever `MediaRecorder` produced client-side — typically |
| 3655 | `audio/webm` on Chrome/Firefox, `audio/mp4`-ish on Safari), not JSON; the |
| 3656 | `Content-Type` header says what format it is. Requires |
| 3657 | `DBBASIC_ENABLE_STT=true` (off by default, unlike TTS which many operators |
| 3658 | turn on immediately — STT costs money on every utterance, not just the |
| 3659 | occasional reply) and a signed-in session. Capped at 10MB per clip (413 |
| 3660 | beyond that). |
| 3661 | |
| 3662 | Unlike TTS, there is **no local-engine path here at all** — every call goes |
| 3663 | to OpenAI's transcription API (`gpt-4o-mini-transcribe` by default), using |
| 3664 | the caller's own stored key exactly like the TTS cloud path. No key stored |
| 3665 | is a 400; a provider-side failure is a 502. |
| 3666 | `DBBASIC_STT_TIMEOUT_SECONDS` bounds the outbound call (default 30s). |
| 3667 | Nothing is persisted server-side beyond the ordinary `ai_usage`/cost |
| 3668 | accounting every provider call gets — the audio itself is uploaded, |
| 3669 | transcribed, and discarded within the request. |
| 3670 | |
| 3671 | ## Reading the Web |
| 3672 | |
| 3673 | The AI can fetch and read a real web page — the lynx/w3m move for an AI + |
| 3674 | voice consumer: strip a URL down to `{title, text, links}`, links numbered |
| 3675 | in document order so they double as speakable navigation targets (""open |
| 3676 | link three""). It is a real, tested loop: read a page, get a short spoken |
| 3677 | summary with numbered links, say ""open link one,"" and it resolves against |
| 3678 | the previous turn's link list, fetches that page, and summarizes it too — |
| 3679 | follow a chain of pages by voice alone. |
| 3680 | |
| 3681 | The `read_page` tool is offered like any other MCP tool (add it to a |
| 3682 | `/tools` subset to use it) and calls `object_reader.py` server-side, gated |
| 3683 | behind `DBBASIC_ENABLE_READER` (off by default). Fetching is SSRF-hardened: |
| 3684 | every hostname — the original and every redirect hop — is resolved and |
| 3685 | checked before the socket opens, refusing loopback/link-local/private |
| 3686 | ranges and failing closed on any resolution ambiguity; `http`/`https` only, |
| 3687 | a small redirect cap, and hard timeout/size limits. HTML is stripped with |
| 3688 | `html.parser` (no regex scraping), never a headless browser — JS-rendered |
| 3689 | pages, images, and login-walled pages are out of scope by design. |
| 3690 | |
| 3691 | A fetched page can also be shown on screen as a `{""kind"": ""reader"", ""url"": |
| 3692 | ...}` view block (title + body text + the numbered link list), fetched |
| 3693 | client-side through `POST /api/read` — the same flag and SSRF posture |
| 3694 | apply. |
| 3695 | |
| 3696 | ## The Instant Commands |
| 3697 | |
| 3698 | | Input | Effect | |
| 3699 | |---|---| |
| 3700 | | `$ pay the hosting bill` | quick note | |
| 3701 | | `. fix the header` | quick task | |
| 3702 | | `^https://example.com docs` | save a link | |
| 3703 | | `~flywheel` | global search across collections | |
| 3704 | | `/help` | list commands | |
| 3705 | |
| 3706 | These never touch the AI — they are one permission-checked record |
| 3707 | write each, which is why they feel instant. |
| 3708 | |
| 3709 | ## For Agents |
| 3710 | |
| 3711 | Everything above is equally available to AI agents connecting over MCP |
| 3712 | (`POST /api/mcp`) with their own identities and labeled sessions, and |
| 3713 | to headless callers hitting `POST /api/ai/chat` directly. One surface, |
| 3714 | many kinds of operator. |
| 3715 | " |
| 3716 | permissions-model Permissions Model Security & Limits 600 A reference guide to DBBASIC's server-side permissions primitive: access modes, rules, row/field filters, sharing, subscriptions, and route enforcement. "DBBASIC permissions are a server-side primitive. A visual editor like Scroll can edit, preview, and explain rules, but the object server must make the final allow/deny decision. |
| 3717 | |
| 3718 | Many web frameworks treat permissions as something each app has to rediscover: blog visibility, admin-only pages, intranet roles, or social-app follows. Those cases matter, but they are not enough for a general business platform. |
| 3719 | |
| 3720 | DBBASIC needs object-level permissions that can model SaaS, customer portals, employee roles, contractors, paid content, shared workspaces, internal admin tools, and AI-generated app surfaces with the same small set of primitives. Getting this right means most apps need less custom permission code. |
| 3721 | |
| 3722 | The model needs to cover the common app cases without forcing every object into one framework shape: |
| 3723 | |
| 3724 | - public visitors |
| 3725 | - signed-in users |
| 3726 | - paid subscribers |
| 3727 | - temporary paid access |
| 3728 | - admins and employees |
| 3729 | - customer accounts with their own employees |
| 3730 | - object owners |
| 3731 | - users or accounts an owner shared with |
| 3732 | - per-row and per-field restrictions |
| 3733 | |
| 3734 | ## Decision shape |
| 3735 | |
| 3736 | The core API is: |
| 3737 | |
| 3738 | ```python |
| 3739 | check_permission(subject, action, policy, collection=None, object_id=None, record=None) |
| 3740 | ``` |
| 3741 | |
| 3742 | It returns: |
| 3743 | |
| 3744 | ```python |
| 3745 | PermissionDecision( |
| 3746 | allowed=True/False, |
| 3747 | reason=""..."", |
| 3748 | code=""allowed|authentication_required|payment_required|forbidden"", |
| 3749 | http_status=200|401|402|403, |
| 3750 | row_filter={...}, |
| 3751 | ) |
| 3752 | ``` |
| 3753 | |
| 3754 | When a list/query request has a row filter but no specific record, the decision can return `allowed=True` with the row filter attached. The caller then applies that filter before returning rows. |
| 3755 | |
| 3756 | ```mermaid |
| 3757 | flowchart TD |
| 3758 | A[""request""] --> B[""identify subject""] |
| 3759 | B --> C[""load policy""] |
| 3760 | C --> D{""admin role?""} |
| 3761 | D -- yes --> Z[""allow""] |
| 3762 | D -- no --> E{""explicit deny matches?""} |
| 3763 | E -- yes --> Y[""deny with reason""] |
| 3764 | E -- no --> F{""access mode allows?""} |
| 3765 | F -- yes --> Z |
| 3766 | F -- no --> G{""allow rule matches?""} |
| 3767 | G -- yes --> H[""allow with optional row/field filters""] |
| 3768 | G -- no --> I{""owner fallback?""} |
| 3769 | I -- yes --> Z |
| 3770 | I -- no --> Y |
| 3771 | ``` |
| 3772 | |
| 3773 | ## Persistence and API |
| 3774 | |
| 3775 | The current public server stores the active policy as: |
| 3776 | |
| 3777 | ```text |
| 3778 | data/permissions/policy.json |
| 3779 | ``` |
| 3780 | |
| 3781 | If the file does not exist, the server uses a conservative default: |
| 3782 | |
| 3783 | ```json |
| 3784 | { |
| 3785 | ""access_mode"": ""role_based"", |
| 3786 | ""roles"": {}, |
| 3787 | ""user_roles"": {}, |
| 3788 | ""rules"": [], |
| 3789 | ""admin_roles"": [""admin"", ""superuser""] |
| 3790 | } |
| 3791 | ``` |
| 3792 | |
| 3793 | That default has no allow grants, so enforcement readiness will block until a real policy is written. `object_permission_store.starter_policy_payload()` returns the documented starter policy for a fresh deployment: anonymous visitors can execute the public pages (`site_home`, `system_dashboard`, `system_write_probe`) and read the `dbbasic_probe` demo records; signed-in users can read and execute objects and write probe records; admin-role subjects bypass rules. Install it with the policy endpoint: |
| 3794 | |
| 3795 | ```bash |
| 3796 | python3 -c ""import json, object_permission_store as s; print(json.dumps({'policy': s.starter_policy_payload()}))"" \ |
| 3797 | | curl -s -X PUT https://your-host/permissions/policy \ |
| 3798 | -H ""Authorization: Token <admin-token>"" \ |
| 3799 | -H ""Content-Type: application/json"" --data-binary @- |
| 3800 | ``` |
| 3801 | |
| 3802 | Then edit grants per app instead of widening the starter rules in place. Run audit mode before enforcement to see what live traffic would be denied. |
| 3803 | |
| 3804 | The policy endpoints are still admin-gated while broader login, account management, and permission editing mature: |
| 3805 | |
| 3806 | ```http |
| 3807 | GET /permissions/policy |
| 3808 | Authorization: Token <token> |
| 3809 | ``` |
| 3810 | |
| 3811 | ```http |
| 3812 | PUT /permissions/policy |
| 3813 | Authorization: Token <token> |
| 3814 | Content-Type: application/json |
| 3815 | ``` |
| 3816 | |
| 3817 | ```json |
| 3818 | { |
| 3819 | ""policy"": { |
| 3820 | ""access_mode"": ""role_based"", |
| 3821 | ""roles"": {""sales"": {""label"": ""Sales""}}, |
| 3822 | ""user_roles"": {""7"": [""sales""]}, |
| 3823 | ""rules"": [] |
| 3824 | } |
| 3825 | } |
| 3826 | ``` |
| 3827 | |
| 3828 | Identity subjects can come from trusted headers, the admin token, or DBBASIC session tokens. The session tokens are backed by a small local registry: |
| 3829 | |
| 3830 | ```text |
| 3831 | data/identity/accounts.tsv |
| 3832 | data/identity/users.tsv |
| 3833 | data/identity/sessions.tsv |
| 3834 | ``` |
| 3835 | |
| 3836 | Accounts carry tenant/subscription defaults. Users carry account membership, roles, and user-level subscriptions. A session minted for a registered user can inherit that account, role, and subscription data, giving permission checks the same subject shape whether the request came from a visual editor, a gateway, or a future DBBASIC login object. |
| 3837 | |
| 3838 | New DBBASIC-created accounts, users, sessions, shared links, record IDs, and change/audit IDs should use UUIDv4 values. Imported systems can keep their old integer or slug IDs as compatibility columns, but new routes and generated UI should not depend on sequential IDs. This keeps SaaS-style tenant data, temporary links, subscriptions, shared records, and future package exports from being tied to one database sequence. The identity registry still accepts string IDs so relational-database data and older prototypes can migrate without a flag day. |
| 3839 | |
| 3840 | Editors and tests can preview decisions without persisting draft rules: |
| 3841 | |
| 3842 | ```http |
| 3843 | POST /permissions/check |
| 3844 | Authorization: Token <token> |
| 3845 | Content-Type: application/json |
| 3846 | ``` |
| 3847 | |
| 3848 | ```json |
| 3849 | { |
| 3850 | ""policy"": { |
| 3851 | ""access_mode"": ""role_based"", |
| 3852 | ""rules"": [ |
| 3853 | { |
| 3854 | ""effect"": ""allow"", |
| 3855 | ""principal"": ""role:sales"", |
| 3856 | ""actions"": [""read""], |
| 3857 | ""collection"": ""contacts"", |
| 3858 | ""row_filter"": {""owner_id"": ""$user_id""} |
| 3859 | } |
| 3860 | ] |
| 3861 | }, |
| 3862 | ""subject"": {""user_id"": ""7"", ""roles"": [""sales""]}, |
| 3863 | ""action"": ""read"", |
| 3864 | ""collection"": ""contacts"" |
| 3865 | } |
| 3866 | ``` |
| 3867 | |
| 3868 | If `policy` is omitted, `/permissions/check` uses the persisted policy file. The optional `now` field accepts an ISO timestamp for testing temporary access windows. |
| 3869 | |
| 3870 | ## Route enforcement |
| 3871 | |
| 3872 | The public server can enforce the persisted policy against object routes and collection-record routes, but it is intentionally opt-in while auth and the editing tools mature. |
| 3873 | |
| 3874 | ```text |
| 3875 | DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true |
| 3876 | ``` |
| 3877 | |
| 3878 | That flag requests enforcement. The server only makes it effective when the readiness checks from `/permissions/status` pass. If recovery, identity, or policy readiness is incomplete, the request stays in audit/shadow mode and responses continue to follow the pre-enforcement route behavior. Current blockers include: |
| 3879 | |
| 3880 | - no admin recovery token |
| 3881 | - unreadable identity store |
| 3882 | - invalid permission policy |
| 3883 | - `password` mode before a password verifier exists |
| 3884 | - `role_based` mode with no allow grants |
| 3885 | - private, registered, subscription, or role-based modes with no non-admin identity path |
| 3886 | |
| 3887 | Operators can see this through: |
| 3888 | |
| 3889 | ```json |
| 3890 | { |
| 3891 | ""permissions"": { |
| 3892 | ""enforcement_requested"": true, |
| 3893 | ""enforcement_enabled"": false, |
| 3894 | ""enforcement_blocked"": true |
| 3895 | } |
| 3896 | } |
| 3897 | ``` |
| 3898 | |
| 3899 | The explicit recovery/test override is: |
| 3900 | |
| 3901 | ```text |
| 3902 | DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true |
| 3903 | ``` |
| 3904 | |
| 3905 | Do not use that override as a normal production setting. |
| 3906 | |
| 3907 | When enforcement is effective, protected routes call the same decision engine before serving or executing data: |
| 3908 | |
| 3909 | ```mermaid |
| 3910 | flowchart LR |
| 3911 | A[""HTTP request""] --> B[""map subject""] |
| 3912 | B --> C[""map route to action""] |
| 3913 | C --> D[""load data/permissions/policy.json""] |
| 3914 | D --> E[""check_permission""] |
| 3915 | E --> F{""allowed?""} |
| 3916 | F -- yes --> G[""serve route""] |
| 3917 | F -- no --> H[""return 401 / 402 / 403""] |
| 3918 | E --> I[""append data/permissions/audit.jsonl""] |
| 3919 | ``` |
| 3920 | |
| 3921 | Route actions currently map like this: |
| 3922 | |
| 3923 | - `GET /objects/{id}` -> `execute` |
| 3924 | - `POST /objects/{id}` -> `execute` |
| 3925 | - `PUT /objects/{id}` -> `update` |
| 3926 | - `DELETE /objects/{id}` -> `delete` |
| 3927 | - `?source=true` -> `source` |
| 3928 | - `?state=true` -> `state` |
| 3929 | - `?logs=true` -> `logs` |
| 3930 | - `?versions=true` or `?version=N` -> `versions` |
| 3931 | - `?metadata=true` -> `read` |
| 3932 | - `GET /collections/{collection}/records` -> `read` |
| 3933 | - `POST /collections/{collection}/records` -> `create` |
| 3934 | - `GET /collections/{collection}/records/{record_id}` -> `read` |
| 3935 | - `PUT /collections/{collection}/records/{record_id}` -> `update` |
| 3936 | - `DELETE /collections/{collection}/records/{record_id}` -> `delete` |
| 3937 | |
| 3938 | Collection record lists apply row filters before pagination. Record detail checks include the selected record, so owner, account, subscription, temporary access, row-filter, and field-redaction rules can all be enforced by the server. Record writes check the affected record. Updates also check the candidate row after applying the submitted changes, so a user cannot pass a row filter on the old row and then move the row into another owner/account by changing fields. |
| 3939 | |
| 3940 | By default the collection record routes are admin-token gated. Read routes can use audit or enforcement mode with the active policy. Write routes require the admin token unless permission enforcement is enabled and the active policy allows the matching `create`, `update`, or `delete` action. Audit-only mode logs write decisions, but it does not authorize mutations by itself. |
| 3941 | |
| 3942 | Audit-only mode logs decisions without blocking the request: |
| 3943 | |
| 3944 | ```text |
| 3945 | DBBASIC_ENABLE_PERMISSION_AUDIT=true |
| 3946 | ``` |
| 3947 | |
| 3948 | That is useful before turning enforcement on for a live app. Enforcement also writes audit entries. |
| 3949 | |
| 3950 | Operators and editing tools can read recent audit entries through the admin-gated audit endpoint: |
| 3951 | |
| 3952 | ```http |
| 3953 | GET /permissions/audit?limit=100&object_id=site_home |
| 3954 | Authorization: Token <token> |
| 3955 | ``` |
| 3956 | |
| 3957 | Supported filters are `action`, `object_id`, `collection`, `allowed`, and `enforced`. |
| 3958 | |
| 3959 | By default, route checks only trust the admin token and anonymous public traffic. If a reverse proxy or auth gateway has already authenticated the request, trusted identity headers can be enabled explicitly: |
| 3960 | |
| 3961 | ```text |
| 3962 | DBBASIC_PERMISSION_TRUST_HEADERS=true |
| 3963 | ``` |
| 3964 | |
| 3965 | Supported headers: |
| 3966 | |
| 3967 | - `X-DBBASIC-User-Id` |
| 3968 | - `X-DBBASIC-Account-Id` |
| 3969 | - `X-DBBASIC-Roles` |
| 3970 | - `X-DBBASIC-Subscriptions` |
| 3971 | |
| 3972 | Only enable trusted headers behind infrastructure that strips or overwrites client-supplied copies. |
| 3973 | |
| 3974 | The server can also mint DBBASIC identity sessions: |
| 3975 | |
| 3976 | ```http |
| 3977 | POST /identity/sessions |
| 3978 | Authorization: Token <admin-token> |
| 3979 | ``` |
| 3980 | |
| 3981 | The admin route can mint arbitrary operator-controlled sessions. For a login gateway or local trusted client, `POST /identity/session` can mint a session for an existing active user when `DBBASIC_ENABLE_SESSION_LOGIN=true` and the caller presents `DBBASIC_SESSION_LOGIN_TOKEN`. That token is separate from the admin token. The route accepts only `user_id`, `label`, and `ttl_seconds`; role, account, and subscription overrides are rejected so the subject comes from the registered user and account. |
| 3982 | |
| 3983 | Sessions store only a token hash under `data/identity/sessions.tsv`. Requests can then send `Authorization: Token <session-token>` or `Authorization: Bearer <session-token>`, and the server resolves the subject directly from its local identity store. This is the first built-in replacement for each app hand-rolling the same ""current user, current account, current roles, current subscription"" lookup. |
| 3984 | |
| 3985 | Clients can inspect or revoke their own active session without the admin token: |
| 3986 | |
| 3987 | ```http |
| 3988 | GET /identity/session |
| 3989 | DELETE /identity/session |
| 3990 | Authorization: Token <session-token> |
| 3991 | ``` |
| 3992 | |
| 3993 | Those routes require an active session token and do not treat the admin token as a user session. |
| 3994 | |
| 3995 | The active subject can be inspected through: |
| 3996 | |
| 3997 | ```http |
| 3998 | GET /identity |
| 3999 | Authorization: Token <token> |
| 4000 | ``` |
| 4001 | |
| 4002 | Admin-token-gated operator routes still require `DBBASIC_ADMIN_TOKEN` by default. A deployment can opt in to `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`; when enabled, an active DBBASIC session whose subject has one of the policy admin roles can pass the same gate. This is intended for operator clients that should stop carrying the raw deployment token after login exists. |
| 4003 | |
| 4004 | That endpoint returns the normalized `user_id`, `account_id`, roles, subscriptions, auth method, and permission-mode flags the server will use for route checks. |
| 4005 | |
| 4006 | Operators and editing tools can inspect permission rollout readiness through: |
| 4007 | |
| 4008 | ```http |
| 4009 | GET /permissions/status |
| 4010 | Authorization: Token <admin-token> |
| 4011 | ``` |
| 4012 | |
| 4013 | That endpoint summarizes the active mode flags, identity counts, persisted policy shape, covered route groups, blockers, and warnings. It is meant to be used before enabling enforcement on a live app. For identity-gated modes, one of these non-admin identity paths must exist before enforcement becomes effective: trusted proxy headers, guarded session login for existing users, or an active DBBASIC session. |
| 4014 | |
| 4015 | ## Access modes |
| 4016 | |
| 4017 | Access modes answer who gets through the front door. |
| 4018 | |
| 4019 | - `public` - anyone can read or execute public objects. |
| 4020 | - `password` - a shared gate authenticates before policy checks. |
| 4021 | - `registered` - any signed-in user can read or execute. |
| 4022 | - `subscription` - signed-in users with an active subscription can read or execute. |
| 4023 | - `role_based` - rules decide by role, user, account, owner, row, field, and action. |
| 4024 | - `private` - owner-only fallback unless explicit rules grant access. |
| 4025 | |
| 4026 | ## Rules |
| 4027 | |
| 4028 | Rules answer what the subject can do once they are known. |
| 4029 | |
| 4030 | Actions should stay small and explicit: |
| 4031 | |
| 4032 | - `create` |
| 4033 | - `read` |
| 4034 | - `update` |
| 4035 | - `delete` |
| 4036 | - `execute` |
| 4037 | - `source` |
| 4038 | - `state` |
| 4039 | - `logs` |
| 4040 | - `files` |
| 4041 | - `versions` |
| 4042 | - `share` |
| 4043 | - `admin` |
| 4044 | |
| 4045 | Principal strings are portable on purpose: |
| 4046 | |
| 4047 | - `public` |
| 4048 | - `registered` |
| 4049 | - `owner` |
| 4050 | - `role:admin` |
| 4051 | - `role:sales` |
| 4052 | - `user:42` |
| 4053 | - `account:customer-acme` |
| 4054 | - `subscription:pro` |
| 4055 | |
| 4056 | Explicit denies are checked before allow rules. |
| 4057 | |
| 4058 | ## Ownership |
| 4059 | |
| 4060 | The old object convention still matters: |
| 4061 | |
| 4062 | ```text |
| 4063 | u_42_report |
| 4064 | ``` |
| 4065 | |
| 4066 | That object belongs to user `42`. Owners can read, execute, update, inspect, share, and delete their own user objects unless a future stricter policy blocks that behavior. |
| 4067 | |
| 4068 | Records can also carry ownership fields: |
| 4069 | |
| 4070 | ```json |
| 4071 | { |
| 4072 | ""owner_id"": ""42"", |
| 4073 | ""customer_account_id"": ""customer-acme"" |
| 4074 | } |
| 4075 | ``` |
| 4076 | |
| 4077 | The server can use `owner_id`, `user_id`, or `created_by` as owner fields for the basic fallback. Application objects can add richer policy around account, department, project, team, or customer ownership. |
| 4078 | |
| 4079 | ## Sharing |
| 4080 | |
| 4081 | Sharing is just a rule: |
| 4082 | |
| 4083 | ```python |
| 4084 | PermissionRule.allow( |
| 4085 | ""user:99"", |
| 4086 | [""read"", ""update""], |
| 4087 | object_id=""u_42_shared_report"", |
| 4088 | reason=""shared by owner"", |
| 4089 | ) |
| 4090 | ``` |
| 4091 | |
| 4092 | Account sharing uses the same shape: |
| 4093 | |
| 4094 | ```python |
| 4095 | PermissionRule.allow( |
| 4096 | ""account:customer-acme"", |
| 4097 | [""read""], |
| 4098 | collection=""invoices"", |
| 4099 | row_filter={""customer_account_id"": ""$account_id""}, |
| 4100 | ) |
| 4101 | ``` |
| 4102 | |
| 4103 | That covers the common case where a customer has multiple employees and they all need access to the customer portal, invoices, files, or support tickets. |
| 4104 | |
| 4105 | Project sharing — ""records I own OR records in a project shared with me"" — uses grant records plus one filter value. Grants are rows in the plain `project_access` collection (`project_id`, `user_id`), so sharing is data: browseable, audited, versioned. Before checks run, the server resolves the subject's grants; rules opt in with `$accessible_projects`, which matches when the record's field value is any granted project id: |
| 4106 | |
| 4107 | ```python |
| 4108 | PermissionRule.allow( |
| 4109 | ""registered"", |
| 4110 | [""read""], |
| 4111 | collection=""notes"", |
| 4112 | row_filter={""project_id"": ""$accessible_projects""}, |
| 4113 | reason=""notes in shared projects are readable"", |
| 4114 | ) |
| 4115 | ``` |
| 4116 | |
| 4117 | Pair it with an owner rule on the same collection: matching stays first-rule-wins at the collection level, but list, single-record, and search reads evaluate rules per record, so ""own"" and ""shared"" rules combine naturally. The engine itself stays pure — grant resolution is IO the server does once per request, and the subject carries the result. |
| 4118 | |
| 4119 | Granting is self-serve through the companion value `$owned_projects`, which resolves to the projects whose `owner_id` is the subject. ""Only a project's owner may share it"" — a cross-record condition — becomes a plain row filter on the grant collection itself: |
| 4120 | |
| 4121 | ```python |
| 4122 | PermissionRule.allow( |
| 4123 | ""registered"", |
| 4124 | [""create"", ""read"", ""delete""], |
| 4125 | collection=""project_access"", |
| 4126 | row_filter={""project_id"": ""$owned_projects""}, |
| 4127 | reason=""project owners share and revoke access to their own projects"", |
| 4128 | ) |
| 4129 | ``` |
| 4130 | |
| 4131 | Creating a grant is creating a record; revoking is deleting it; both are audited and appear in the change history like every other write. |
| 4132 | |
| 4133 | Grants carry a `permission` of `read` or `write`. Most rules only need membership (`$accessible_projects`), but some need to tell read grants apart from write grants — ""only project writers may publish"" — and that narrower set is `$writable_projects`, resolved from the same `project_access` rows filtered to `permission == ""write""`: |
| 4134 | |
| 4135 | ```python |
| 4136 | PermissionRule.allow( |
| 4137 | ""registered"", |
| 4138 | [""update""], |
| 4139 | collection=""docs"", |
| 4140 | row_filter={""project_id"": ""$writable_projects""}, |
| 4141 | reason=""only writers may edit docs in a shared project"", |
| 4142 | ) |
| 4143 | ``` |
| 4144 | |
| 4145 | It resolves the same way and is available everywhere `$accessible_projects` and `$owned_projects` are: row filters and, for schema-declared lifecycle fields, transition guards (see the `transitions` field key in the schema-forms guide). |
| 4146 | |
| 4147 | ## Row and field rules |
| 4148 | |
| 4149 | Row filters model rules like: |
| 4150 | |
| 4151 | ```text |
| 4152 | sales reps only see their own leads |
| 4153 | ``` |
| 4154 | |
| 4155 | As data: |
| 4156 | |
| 4157 | ```python |
| 4158 | PermissionRule.allow( |
| 4159 | ""role:sales"", |
| 4160 | [""read""], |
| 4161 | collection=""contacts"", |
| 4162 | row_filter={""owner_id"": ""$user_id""}, |
| 4163 | ) |
| 4164 | ``` |
| 4165 | |
| 4166 | Field rules model cases like hiding salary, internal notes, cost, or private metadata: |
| 4167 | |
| 4168 | ```python |
| 4169 | PermissionRule.allow( |
| 4170 | ""account:customer-acme"", |
| 4171 | [""read""], |
| 4172 | collection=""invoices"", |
| 4173 | fields=[""invoice_id"", ""status"", ""total""], |
| 4174 | denied_fields=[""internal_notes""], |
| 4175 | ) |
| 4176 | ``` |
| 4177 | |
| 4178 | Schema field permissions add the generated-form version of the same idea. They live next to field metadata so a visual editor can render the right controls and the server can enforce the same decision: |
| 4179 | |
| 4180 | ```json |
| 4181 | { |
| 4182 | ""name"": ""margin"", |
| 4183 | ""type"": ""currency"", |
| 4184 | ""permissions"": { |
| 4185 | ""admin"": ""edit"", |
| 4186 | ""sales"": ""hidden"", |
| 4187 | ""viewer"": ""hidden"" |
| 4188 | } |
| 4189 | } |
| 4190 | ``` |
| 4191 | |
| 4192 | The access levels are: |
| 4193 | |
| 4194 | - `edit` - visible and writable |
| 4195 | - `read` - visible but not writable |
| 4196 | - `hidden` - removed from reads and not writable |
| 4197 | |
| 4198 | Principal keys can be role names such as `sales`, or full principals such as `role:sales`, `user:42`, `account:customer-acme`, `subscription:pro`, `registered`, `public`, or `owner`. Grouped access lists are also accepted: |
| 4199 | |
| 4200 | ```json |
| 4201 | { |
| 4202 | ""name"": ""notes"", |
| 4203 | ""permissions"": { |
| 4204 | ""edit"": [""owner"", ""role:admin""], |
| 4205 | ""read"": [""role:support""], |
| 4206 | ""default"": ""hidden"" |
| 4207 | } |
| 4208 | } |
| 4209 | ``` |
| 4210 | |
| 4211 | When multiple principals match one subject, the most restrictive field access wins. Schema field permissions only refine fields after the broader policy has already allowed access to the row. |
| 4212 | |
| 4213 | ## Subscription and temporary access |
| 4214 | |
| 4215 | Subscriptions can be broad: |
| 4216 | |
| 4217 | ```python |
| 4218 | PermissionPolicy(access_mode=""subscription"") |
| 4219 | ``` |
| 4220 | |
| 4221 | Or plan-specific: |
| 4222 | |
| 4223 | ```python |
| 4224 | PermissionRule.allow( |
| 4225 | ""subscription:pro"", |
| 4226 | [""read""], |
| 4227 | collection=""premium_reports"", |
| 4228 | ) |
| 4229 | ``` |
| 4230 | |
| 4231 | Temporary pay-per-view style access is a time-boxed grant: |
| 4232 | |
| 4233 | ```python |
| 4234 | PermissionRule.allow( |
| 4235 | ""user:42"", |
| 4236 | [""read""], |
| 4237 | object_id=""reports_market_snapshot"", |
| 4238 | valid_from=""2026-06-01T00:00:00Z"", |
| 4239 | expires_at=""2026-07-01T00:00:00Z"", |
| 4240 | ) |
| 4241 | ``` |
| 4242 | |
| 4243 | If access should be consumed after one or more views, that should be tracked by an entitlement or ledger object. The permission rule says access is possible; the ledger object records purchases, consumption, refunds, and audit history. |
| 4244 | |
| 4245 | For HTTP APIs, missing or expired paid entitlement should map to: |
| 4246 | |
| 4247 | ```text |
| 4248 | 402 Payment Required |
| 4249 | ``` |
| 4250 | |
| 4251 | That is different from: |
| 4252 | |
| 4253 | - `401 Unauthorized` - the server does not know who the user is yet. |
| 4254 | - `403 Forbidden` - the server knows the user, but the user is not allowed. |
| 4255 | - `402 Payment Required` - the user may be allowed after subscription, purchase, credit, or another payment entitlement. |
| 4256 | |
| 4257 | ## Editor contract |
| 4258 | |
| 4259 | A visual editing tool should eventually call a server endpoint that exposes the same decision shape: |
| 4260 | |
| 4261 | ```json |
| 4262 | { |
| 4263 | ""allowed"": true, |
| 4264 | ""reason"": ""sales reps only see own contacts"", |
| 4265 | ""code"": ""allowed"", |
| 4266 | ""http_status"": 200, |
| 4267 | ""row_filter"": {""owner_id"": ""$user_id""}, |
| 4268 | ""fields"": null, |
| 4269 | ""denied_fields"": [] |
| 4270 | } |
| 4271 | ``` |
| 4272 | |
| 4273 | That lets an editor show permission previews, generated matrices, ""test as role"", and AI-generated rules without becoming the authority. |
| 4274 | " |
| 4275 | secrets-and-credentials Secrets, Credentials, and the Trust Boundary Security & Limits 610 Where secrets live in DBBASIC, why a secret is never a record field, and the honest threat model behind the write-only identity vault. "This is the doctrine for **where secrets live** in DBBASIC and why. It matters |
| 4276 | to anyone writing a package that touches a credential — a user's third-party |
| 4277 | API key, an outbound-mail relay password, an email-hosting mailbox password, a |
| 4278 | signing token. Get this wrong and a secret ends up in a record collection, |
| 4279 | which means it ends up in an export, a backup, or an API read. Get it right and |
| 4280 | secrets stay in one place, with one honest boundary, that every other part of |
| 4281 | the system already trusts. |
| 4282 | |
| 4283 | The one-line rule: **a secret is never a record field.** Records are readable, |
| 4284 | listable, exportable, and included in portable backups by design. A secret must |
| 4285 | be none of those things. So secrets go somewhere else, through a different door. |
| 4286 | |
| 4287 | ## Where secrets live |
| 4288 | |
| 4289 | There are exactly two homes, by lifetime and ownership: |
| 4290 | |
| 4291 | ### 1. Deploy-time, server-wide secrets → the environment file |
| 4292 | |
| 4293 | A credential the *whole deployment* shares — the SMTP relay password |
| 4294 | (`DBBASIC_SMTP_PASSWORD`), a connector's API key (`DBBASIC_MAILCOW_API_KEY`), |
| 4295 | the admin token (`DBBASIC_ADMIN_TOKEN`) — lives in |
| 4296 | `/etc/dbbasic-object-server.env`, `640 root:dbbasic` (see |
| 4297 | [`single-vm-deployment.md`](/docs/single-vm-deployment)). It is set by the operator, read by the server |
| 4298 | process, and never appears in a schema field, a record, an API response, or a |
| 4299 | backup archive. A package that needs one **declares the env var name in its |
| 4300 | docs and reads it at runtime** — it never ships the value and never stores it. |
| 4301 | |
| 4302 | ### 2. Per-user recoverable secrets → the identity vault |
| 4303 | |
| 4304 | A secret that belongs to *one user* and must be *recoverable to be used* (a BYO |
| 4305 | API key the server calls a provider with; a mailbox password handed to the mail |
| 4306 | server) lives in the **write-only identity vault**, `object_service_keys` |
| 4307 | (`identity/service_keys.tsv`). Its contract: |
| 4308 | |
| 4309 | - **Owner-only file**, `0600`, next to `credentials.tsv` — same filesystem |
| 4310 | trust boundary as the admin token. |
| 4311 | - **Write-only**: a caller can *set* a key, *list which services have one*, and |
| 4312 | *delete* one. **No surface ever reads key material back** — not the HTTP API, |
| 4313 | not the shell, not an agent, not a package. The server uses a stored key |
| 4314 | internally (to call the provider) so it never travels to a browser. |
| 4315 | - **Stored as-provided, not encrypted, not hashed.** Passwords are hashed |
| 4316 | because they only need to be *verified*; these must be *used*, so they must be |
| 4317 | recoverable. On-record encryption with a key derived from a server secret |
| 4318 | (the pattern many frameworks reach for) is *theater* here — the ciphertext and |
| 4319 | the key it's decrypted with live in the same blast radius, so they leak |
| 4320 | together (see the Trust Boundary section). The real protection is the `0600` |
| 4321 | file and the write-only contract, not a cipher. |
| 4322 | - **Excluded from portable backups and source control** — so a leaked backup |
| 4323 | archive, the most common accidental exposure, contains no secret material. |
| 4324 | (See [`backup-restore.md`](/docs/backup-restore); a package must never undo this by copying a secret |
| 4325 | into a record it *does* back up.) |
| 4326 | |
| 4327 | **Concrete example**: the AI chat, cloud TTS, and cloud STT features |
| 4328 | (`docs/shell-and-ai.md`) all read from this vault, not the env file — a |
| 4329 | user's own `anthropic` and/or `openai` key, set via `PUT |
| 4330 | /identity/users/{you}/service-keys`. One stored `openai` key covers chat, |
| 4331 | `POST /api/tts` with `engine: ""openai""`, and `POST /api/stt` — no separate |
| 4332 | signup per feature. This is also why those features are per-user opt-in |
| 4333 | rather than deployment-wide: `DBBASIC_ENABLE_AI_CHAT`/`_TTS`/`_STT` in the |
| 4334 | env file only unlock the *capability*; each user still needs their own |
| 4335 | stored key before any of it does anything for them. |
| 4336 | |
| 4337 | ## How a secret gets written (it is not a record write) |
| 4338 | |
| 4339 | The vault has its own door, deliberately separate from the generic record API: |
| 4340 | |
| 4341 | ``` |
| 4342 | GET /identity/users/{user_id}/service-keys → which services are set (status only) |
| 4343 | PUT /identity/users/{user_id}/service-keys → { ""service"": ""..."", ""key"": ""..."" } |
| 4344 | DELETE /identity/users/{user_id}/service-keys/{service} → clear one |
| 4345 | ``` |
| 4346 | |
| 4347 | - **Owner-scoped**: only the account owner's own session may write their vault |
| 4348 | (or the operator admin gate). Identity is resolved server-side from an opaque |
| 4349 | session token — it cannot be spoofed by a client-supplied user id. |
| 4350 | - **CSRF-guarded**: cross-origin cookie writes are refused. |
| 4351 | - **Never echoes material**: `PUT` returns metadata (`service`, timestamps) |
| 4352 | only; `GET` returns presence only. |
| 4353 | |
| 4354 | `POST /collections/{c}/records` — the generic record write — **cannot reach the |
| 4355 | vault at all.** That is the point: there is no record surface to misconfigure |
| 4356 | into leaking a secret. |
| 4357 | |
| 4358 | **In a generated UI**, a secret is a *write-only field*: it renders as ""set / |
| 4359 | not set"" with an input that only ever POSTs to the vault endpoint above and |
| 4360 | never displays a stored value. The form still looks uniform to the user — one |
| 4361 | field just writes to a different door than the record. |
| 4362 | |
| 4363 | ## Reserved namespaces: user-writable vs platform-owned |
| 4364 | |
| 4365 | A vault entry is keyed by a **service name**. Two kinds: |
| 4366 | |
| 4367 | - **User-writable** (a person's own BYO credential): a plain name — `openai`, |
| 4368 | `anthropic`, `stripe`. The self-service endpoint above writes these. |
| 4369 | - **Platform-owned** (a secret the *platform* provisions on the user's behalf — |
| 4370 | a generated mailbox password): a **reserved name**, prefixed `sys-` |
| 4371 | (e.g. `sys-mailbox-<mailbox_id>`). **The self-service HTTP endpoint refuses to |
| 4372 | write or delete a reserved name.** Only in-process server code (a connector, a |
| 4373 | provisioning verb) may set one. |
| 4374 | |
| 4375 | This is the control that stops a user from **overwriting a secret the platform |
| 4376 | owns**. Without it, a user could `PUT` their own `sys-mailbox-<id>` and clobber |
| 4377 | the password the mail server actually uses. `object_service_keys.is_reserved_service()` |
| 4378 | is the single source of truth for the rule; the HTTP handler enforces it. |
| 4379 | |
| 4380 | Two more rules for platform-owned secrets a package must follow: |
| 4381 | |
| 4382 | - **Scope every read by the record's owner**, never by caller input. Resolve a |
| 4383 | mailbox's secret from *the mailbox record's `owner_id`* — never from a |
| 4384 | `mailbox_id` a request supplied — or a user could plant an entry in their own |
| 4385 | vault and hope a global lookup reads it. |
| 4386 | - **Generate, don't collect, when you can.** A provisioned password should be |
| 4387 | generated server-side (`secrets.token_urlsafe`), stored in the vault, and |
| 4388 | revealed to the user *exactly once* in the provisioning response — never read |
| 4389 | back afterward. |
| 4390 | |
| 4391 | ## The trust boundary (threat model) |
| 4392 | |
| 4393 | Be honest about what actually protects the vault, because it changes what's |
| 4394 | worth building. A secret is reachable only by one of these, and **every one of |
| 4395 | them also compromises the entire system** — so the vault is not a weaker link, |
| 4396 | it inherits the system's security: |
| 4397 | |
| 4398 | | Vector | What it takes | Does encryption-at-rest help? | |
| 4399 | |---|---|---| |
| 4400 | | **A logged-in user** reaching another user's secret | Impossible via the API: owner-scoped, server-derived identity, reserved-namespace guard | N/A — blocked at the app layer | |
| 4401 | | **Code execution as the service user** | An RCE in the server, or an installed package/connector (both run as the service user and legitimately read the vault) | **No** — the running process holds the key too | |
| 4402 | | **Root on the host** | Kernel/sudo/other-service exploit; `0600` is nothing to root | **No** — same blast radius | |
| 4403 | | **The raw disk** | A VM snapshot, an attached volume, a stolen *unencrypted* backup | **Only here** — and only if the key lives off-box (a KMS) | |
| 4404 | | **The deploy path** | A dev-box SSH key or a push-to-prod pipeline → arbitrary code as the service user | **No** — it's code execution with a login | |
| 4405 | |
| 4406 | Conclusions that follow: |
| 4407 | |
| 4408 | - **A plain user cannot reach it.** They are boxed into their own namespace by a |
| 4409 | server-resolved identity and the reserved-namespace guard. |
| 4410 | - **On-box encryption of the vault is theater** against every vector except a |
| 4411 | raw-disk/snapshot theft — because for all the others the decryption key rides |
| 4412 | in the same boundary. The genuinely stronger option is **envelope encryption |
| 4413 | with a KMS-held root key**, and only when you have a KMS (not on a single VM). |
| 4414 | Until then, the higher-leverage move is to **encrypt backups/snapshots with an |
| 4415 | off-box key and lock the hosting account with 2FA** — the deploy credential |
| 4416 | and the hosting account are, in effect, vault keys. |
| 4417 | - **""Installed package = code execution as the service user""** is a stated trust |
| 4418 | fact, which is why package installs (and any dynamically loaded connector |
| 4419 | code) are admin-gated. Fine while packages are first-party; it needs a |
| 4420 | stronger answer before any third-party marketplace. |
| 4421 | |
| 4422 | ## Checklist for package authors |
| 4423 | |
| 4424 | - **Never** put a secret in a schema field or a record. |
| 4425 | - Server-wide credential → read from a documented `DBBASIC_*` env var; never ship or store the value. |
| 4426 | - Per-user recoverable secret → the identity vault (`object_service_keys`), write-only. |
| 4427 | - Platform-provisioned secret → a `sys-`-prefixed reserved service name; write it only from server-side code. |
| 4428 | - Resolve a secret by the owning record's `owner_id`, never by caller-supplied input. |
| 4429 | - Prefer generating a secret and revealing it once over collecting and storing it. |
| 4430 | - In a UI, a secret field is write-only (set/not-set), posting to the vault endpoint, never rendering the value. |
| 4431 | - Never copy a secret into anything that lands in a backup or an export. |
| 4432 | |
| 4433 | ## See also |
| 4434 | |
| 4435 | - [`single-vm-deployment.md`](/docs/single-vm-deployment) — the env-file trust boundary and admin-route posture. |
| 4436 | - [`permissions-model.md`](/docs/permissions-model) — owner-scoping and the admin bypass. |
| 4437 | - [`backup-restore.md`](/docs/backup-restore) — what is and isn't in a portable backup. |
| 4438 | - [`package-authoring.md`](/docs/package-authoring) — how a package ships schemas, permissions, and (soon) connectors. |
| 4439 | " |
| 4440 | traffic-limits Traffic Limits Security & Limits 620 How the DBBASIC Object Server rejects oversized, overloaded, or runaway traffic before object code runs, and the env vars that control each limit. "DBBASIC should reject bad or oversized traffic before object code runs. The |
| 4441 | runtime keeps the object loop fast by failing early when a request is too large |
| 4442 | or when the server is under pressure. |
| 4443 | |
| 4444 | ## Request Body Size |
| 4445 | |
| 4446 | The ASGI server enforces a request body cap: |
| 4447 | |
| 4448 | ```text |
| 4449 | DBBASIC_MAX_REQUEST_BYTES=1048576 |
| 4450 | ``` |
| 4451 | |
| 4452 | The default is `1048576` bytes, or 1 MiB. If a request body is larger than the |
| 4453 | configured value, the server returns: |
| 4454 | |
| 4455 | ```http |
| 4456 | 413 Payload Too Large |
| 4457 | ``` |
| 4458 | |
| 4459 | ```json |
| 4460 | { |
| 4461 | ""status"": ""error"", |
| 4462 | ""error"": ""Request body too large"", |
| 4463 | ""max_bytes"": 1048576 |
| 4464 | } |
| 4465 | ``` |
| 4466 | |
| 4467 | This applies before JSON parsing and before object execution. It checks |
| 4468 | `Content-Length` when present and also counts streamed ASGI body chunks. |
| 4469 | |
| 4470 | Large file upload paths should get their own file API and their own explicit |
| 4471 | limits. Normal object `POST`, `PUT`, and `DELETE` requests should stay small. |
| 4472 | |
| 4473 | ## Concurrency |
| 4474 | |
| 4475 | The ASGI server also enforces per-process concurrency caps: |
| 4476 | |
| 4477 | ```text |
| 4478 | DBBASIC_MAX_CONCURRENT_REQUESTS=64 |
| 4479 | DBBASIC_MAX_CONCURRENT_EXECUTIONS=8 |
| 4480 | ``` |
| 4481 | |
| 4482 | `DBBASIC_MAX_CONCURRENT_REQUESTS` limits non-health HTTP requests that are in |
| 4483 | flight inside one server process. `/health` bypasses this limit so a monitor can |
| 4484 | still check whether the process is alive under load. |
| 4485 | |
| 4486 | `DBBASIC_MAX_CONCURRENT_EXECUTIONS` limits object method executions inside one |
| 4487 | server process. Introspection routes, health checks, and other non-execution |
| 4488 | requests do not consume object execution slots. |
| 4489 | |
| 4490 | When either limit is full, the server returns: |
| 4491 | |
| 4492 | ```http |
| 4493 | 503 Service Unavailable |
| 4494 | ``` |
| 4495 | |
| 4496 | ```json |
| 4497 | { |
| 4498 | ""status"": ""error"", |
| 4499 | ""error"": ""Server is busy"", |
| 4500 | ""limit"": ""object_executions"", |
| 4501 | ""max_concurrent"": 8 |
| 4502 | } |
| 4503 | ``` |
| 4504 | |
| 4505 | Set either concurrency value to `0` to disable that app-level limit for local |
| 4506 | experiments. Production and staging deployments should keep explicit limits. |
| 4507 | |
| 4508 | These caps are per process. If uvicorn runs multiple workers, each worker has |
| 4509 | its own counters. A reverse proxy or process manager still controls the total |
| 4510 | machine-level shape. |
| 4511 | |
| 4512 | ## Rate Limits |
| 4513 | |
| 4514 | Rate limiting is opt-in: |
| 4515 | |
| 4516 | ```text |
| 4517 | DBBASIC_RATE_LIMIT_REQUESTS=1000 |
| 4518 | DBBASIC_RATE_LIMIT_WINDOW_SECONDS=60 |
| 4519 | ``` |
| 4520 | |
| 4521 | The default `DBBASIC_RATE_LIMIT_REQUESTS=0` leaves the app-level limiter |
| 4522 | disabled for local development. Public staging and production deployments should |
| 4523 | set an explicit value. |
| 4524 | |
| 4525 | When enabled, the server checks the rate limit before reading the request body |
| 4526 | or running object code. Plain `GET /health` is excluded so external monitoring |
| 4527 | can still tell whether the process is alive. Detailed health via |
| 4528 | `capacity=true` or `metrics=true` is rate-limited and still requires the admin |
| 4529 | token. |
| 4530 | |
| 4531 | The limiter stores one timestamp bucket per identity under: |
| 4532 | |
| 4533 | ```text |
| 4534 | data/ratelimit/ |
| 4535 | ``` |
| 4536 | |
| 4537 | Those files are runtime guardrails, not application data. Backups exclude them. |
| 4538 | |
| 4539 | Valid admin-token requests use a token bucket. Other requests use the client IP. |
| 4540 | By default the server ignores `X-Forwarded-For` and `X-Real-IP`; enable proxy |
| 4541 | header trust only when uvicorn is bound privately behind a trusted reverse |
| 4542 | proxy: |
| 4543 | |
| 4544 | ```text |
| 4545 | DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS=true |
| 4546 | ``` |
| 4547 | |
| 4548 | When the limit is full, the server returns: |
| 4549 | |
| 4550 | ```http |
| 4551 | 429 Too Many Requests |
| 4552 | Retry-After: 30 |
| 4553 | ``` |
| 4554 | |
| 4555 | ```json |
| 4556 | { |
| 4557 | ""status"": ""error"", |
| 4558 | ""error"": ""Rate limit exceeded"", |
| 4559 | ""retry_after"": 30, |
| 4560 | ""limit"": 1000, |
| 4561 | ""window_seconds"": 60 |
| 4562 | } |
| 4563 | ``` |
| 4564 | |
| 4565 | ## Object Execution Timeout |
| 4566 | |
| 4567 | Object execution timeout is opt-in: |
| 4568 | |
| 4569 | ```text |
| 4570 | DBBASIC_OBJECT_TIMEOUT_SECONDS=5 |
| 4571 | ``` |
| 4572 | |
| 4573 | The default `0` keeps local development on the direct in-process execution path. |
| 4574 | When the value is above zero, object methods run in a short-lived worker process. |
| 4575 | If the object does not finish before the wall-clock timeout, the server |
| 4576 | terminates the worker and returns: |
| 4577 | |
| 4578 | ```http |
| 4579 | 504 Gateway Timeout |
| 4580 | ``` |
| 4581 | |
| 4582 | ```json |
| 4583 | { |
| 4584 | ""status"": ""error"", |
| 4585 | ""error"": ""Execution failed: GET timed out for object basics_slow after 5 seconds"" |
| 4586 | } |
| 4587 | ``` |
| 4588 | |
| 4589 | Timeout failures are written to the object log with |
| 4590 | `ObjectExecutionTimeoutError`, so Scroll and operators can see which object |
| 4591 | timed out. |
| 4592 | |
| 4593 | Trusted server-owned objects can opt back into the fast in-process path even |
| 4594 | when timeouts are enabled: |
| 4595 | |
| 4596 | ```text |
| 4597 | DBBASIC_TRUSTED_IN_PROCESS_OBJECTS=site_home,system_health |
| 4598 | ``` |
| 4599 | |
| 4600 | Use that only for reviewed objects shipped or controlled by the server operator, |
| 4601 | such as a public homepage that receives frequent hits. Untrusted user objects |
| 4602 | should stay on the timeout-enabled worker path. |
| 4603 | |
| 4604 | This boundary is intentionally narrow. It stops hung object work and releases |
| 4605 | the execution slot, but it is not the final sandbox. CPU quotas, memory limits, |
| 4606 | and a longer-lived worker pool are still future hardening steps. |
| 4607 | |
| 4608 | ## Capacity Health |
| 4609 | |
| 4610 | `GET /health` stays public and cheap: |
| 4611 | |
| 4612 | ```json |
| 4613 | { |
| 4614 | ""status"": ""ok"" |
| 4615 | } |
| 4616 | ``` |
| 4617 | |
| 4618 | Detailed capacity is an operator surface and requires the admin token: |
| 4619 | |
| 4620 | ```http |
| 4621 | GET /health?capacity=true |
| 4622 | Authorization: Token <token> |
| 4623 | ``` |
| 4624 | |
| 4625 | It reports the configured request and object execution limits, current |
| 4626 | in-flight slot counts, object count, storage status, version, uptime, request |
| 4627 | counts, recent response timing, and basic process/system information. Scroll and |
| 4628 | future station routers should use this detailed shape instead of guessing from |
| 4629 | logs. |
| 4630 | |
| 4631 | `GET /health?metrics=true` includes the same capacity payload plus detailed |
| 4632 | request metrics such as status counts, top paths, and recent HTTP errors. |
| 4633 | |
| 4634 | ## High-Traffic Shape |
| 4635 | |
| 4636 | Use layered limits: |
| 4637 | |
| 4638 | - reverse proxy body limit before Python receives the request |
| 4639 | - app body limit through `DBBASIC_MAX_REQUEST_BYTES` |
| 4640 | - app request and object execution limits through the concurrency env vars |
| 4641 | - per-IP and per-token rate limits before write or execute paths |
| 4642 | - object execution wall-clock timeouts |
| 4643 | - later CPU and memory isolation around worker processes |
| 4644 | |
| 4645 | When the server is overloaded, it should return `429 Too Many Requests` for rate |
| 4646 | limits or `503 Service Unavailable` for temporary capacity pressure. That is |
| 4647 | better than letting a small VM run out of memory or accumulate unbounded object |
| 4648 | work. |
| 4649 | |
| 4650 | ## Staging Defaults |
| 4651 | |
| 4652 | For the first public staging VM: |
| 4653 | |
| 4654 | ```text |
| 4655 | DBBASIC_MAX_REQUEST_BYTES=1048576 |
| 4656 | DBBASIC_MAX_CONCURRENT_REQUESTS=64 |
| 4657 | DBBASIC_MAX_CONCURRENT_EXECUTIONS=8 |
| 4658 | DBBASIC_OBJECT_TIMEOUT_SECONDS=5 |
| 4659 | DBBASIC_RATE_LIMIT_REQUESTS=1000 |
| 4660 | DBBASIC_RATE_LIMIT_WINDOW_SECONDS=60 |
| 4661 | DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS=true |
| 4662 | DBBASIC_ENABLE_SOURCE_WRITES=false |
| 4663 | ``` |
| 4664 | |
| 4665 | Keep source writes closed until auth, permissions, backups, and rollback checks |
| 4666 | are working. Keep public routes narrow. Let the object server prove the loop |
| 4667 | without allowing arbitrary public code changes. |
| 4668 | |
| 4669 | ## Cluster Direction |
| 4670 | |
| 4671 | Local capacity should become a station signal. A single station can already |
| 4672 | return `503` when it is full. Later, station heartbeat or health data should |
| 4673 | include capacity fields so a cluster router can avoid saturated stations before |
| 4674 | forwarding object work. |
| 4675 | |
| 4676 | That signal should include request slots, execution slots, recent error rate, |
| 4677 | and basic machine load such as CPU and memory pressure. That keeps the cluster |
| 4678 | model simple: each station knows its own limits, and the router uses those |
| 4679 | limits to choose where work should go. |
| 4680 | |
| 4681 | ## Logs And Garbage Collection |
| 4682 | |
| 4683 | Traffic controls also need disk controls: |
| 4684 | |
| 4685 | - rotate object logs before they grow forever |
| 4686 | - gzip rotated logs at rest |
| 4687 | - keep a bounded number of rotated logs per object |
| 4688 | - disable noisy uvicorn access logs in systemd and rely on object logs, |
| 4689 | request metrics, and deliberate analytics objects |
| 4690 | - cap journald retention for process logs |
| 4691 | - keep backup retention explicit |
| 4692 | - delete temp files and stale lock files only through known runtime paths |
| 4693 | |
| 4694 | The active object log stays plain TSV so tools can tail it. Rotated logs can be |
| 4695 | read in place with normal Unix tools such as `gzip -cd`. |
| 4696 | |
| 4697 | ## Next Limits |
| 4698 | |
| 4699 | The request, concurrency, rate-limit, and timeout caps are only the first |
| 4700 | boundaries. The next production-hardening steps are: |
| 4701 | |
| 4702 | - CPU and memory isolation for untrusted object code |
| 4703 | - a longer-lived worker pool so timeout-enabled execution does not need a new |
| 4704 | process per request |
| 4705 | " |
| 4706 | single-vm-deployment Single VM Deployment Deployment & Operations 700 How to deploy DBBASIC Object Server on a single VM: layout, install, environment, systemd, HTTPS proxy, backups, and upgrade procedure. "This is the first deployment target for DBBASIC Object Server: one small VM, |
| 4707 | one ASGI process, one object source directory, and one data directory. |
| 4708 | |
| 4709 | The goal is not to claim the public runtime is production-ready. The goal is to |
| 4710 | keep a clean staging server running so install, restart, logs, object storage, |
| 4711 | and the edit/run/inspect loop are tested on a real machine. |
| 4712 | |
| 4713 | This shape should stay cheap and understandable: one small VM can run the |
| 4714 | server, objects, TSV-backed state, logs, versions, HTTPS proxy, and monitoring |
| 4715 | without requiring a separate database tier for the first useful app. |
| 4716 | |
| 4717 | ## Deployment Goal |
| 4718 | |
| 4719 | The first deployment path should stay closer to a simple Unix/PHP-style upload |
| 4720 | than a large multi-service app stack: |
| 4721 | |
| 4722 | - one VM is enough for the first useful business app |
| 4723 | - no container runtime is required for the base deployment |
| 4724 | - no separate database server is required before the app proves itself |
| 4725 | - server code, live object source, runtime data, and secrets live in separate |
| 4726 | Unix paths |
| 4727 | - normal upgrades are `git pull`, package install, service restart, checks |
| 4728 | - object edits do not require a full server redeploy |
| 4729 | - every manual step should become scriptable after it stays boring |
| 4730 | |
| 4731 | This does not mean skipping safety. The simple path still uses a service user, |
| 4732 | systemd, a reverse proxy, HTTPS, private config, filesystem checks, backups, and |
| 4733 | explicit public route allowlists. |
| 4734 | |
| 4735 | ## Verified Baseline |
| 4736 | |
| 4737 | The first staging install was verified on: |
| 4738 | |
| 4739 | - Ubuntu 24.04.4 LTS |
| 4740 | - Python 3.12.3 |
| 4741 | - Git 2.43.0 |
| 4742 | - Caddy 2.11.4 |
| 4743 | - systemd with the DBBASIC service and Caddy running as separate units |
| 4744 | |
| 4745 | Record the baseline on each new VM. Run the Caddy commands after Caddy is |
| 4746 | installed, if the VM does not already have it. |
| 4747 | |
| 4748 | ```bash |
| 4749 | lsb_release -a |
| 4750 | python3 --version |
| 4751 | git --version |
| 4752 | caddy version |
| 4753 | systemctl is-active dbbasic-object-server caddy |
| 4754 | cd /opt/dbbasic-object-server |
| 4755 | sudo -u dbbasic git rev-parse --short HEAD |
| 4756 | ``` |
| 4757 | |
| 4758 | Do not commit the VM's real IP address, hostname, or provider-specific details |
| 4759 | to this repository. |
| 4760 | |
| 4761 | ## System Updates |
| 4762 | |
| 4763 | Before using a new VM for staging, apply normal OS updates and reboot if Ubuntu |
| 4764 | requests it: |
| 4765 | |
| 4766 | ```bash |
| 4767 | sudo apt update |
| 4768 | apt list --upgradable |
| 4769 | sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade -y |
| 4770 | sudo DEBIAN_FRONTEND=noninteractive apt-get autoremove -y |
| 4771 | test -f /var/run/reboot-required && cat /var/run/reboot-required || true |
| 4772 | ``` |
| 4773 | |
| 4774 | If packages are kept back because they need new dependencies, install them |
| 4775 | explicitly when they are normal platform packages: |
| 4776 | |
| 4777 | ```bash |
| 4778 | sudo DEBIAN_FRONTEND=noninteractive apt-get install -y fwupd |
| 4779 | ``` |
| 4780 | |
| 4781 | After a reboot, verify the services and endpoint again: |
| 4782 | |
| 4783 | ```bash |
| 4784 | sudo systemctl is-active dbbasic-object-server caddy |
| 4785 | curl https://dbbasic.example.com/ |
| 4786 | ``` |
| 4787 | |
| 4788 | ## Provider Monitoring |
| 4789 | |
| 4790 | For DigitalOcean Droplets, the metrics agent is useful on staging VMs because it |
| 4791 | adds CPU, memory, disk, and network graphs plus alerting while the server is |
| 4792 | under active development. DigitalOcean describes this as a free opt-in service. |
| 4793 | It sends system telemetry for monitoring, not customer content. |
| 4794 | |
| 4795 | Install it after normal OS updates: |
| 4796 | |
| 4797 | ```bash |
| 4798 | curl -fsSL https://repos.insights.digitalocean.com/install.sh -o /tmp/do-agent-install.sh |
| 4799 | sudo bash /tmp/do-agent-install.sh |
| 4800 | ``` |
| 4801 | |
| 4802 | Verify it: |
| 4803 | |
| 4804 | ```bash |
| 4805 | systemctl is-active do-agent |
| 4806 | systemctl is-enabled do-agent |
| 4807 | dpkg-query -W do-agent |
| 4808 | systemctl --no-pager --full status do-agent |
| 4809 | ``` |
| 4810 | |
| 4811 | Metrics usually appear in the Droplet's Insights tab after a few minutes. This |
| 4812 | is provider-specific VM monitoring; it does not replace DBBASIC's own object |
| 4813 | logs, state, versions, or future application/runtime metrics. |
| 4814 | |
| 4815 | ## Layout |
| 4816 | |
| 4817 | Use separate paths for server code, object source, and runtime data: |
| 4818 | |
| 4819 | ```text |
| 4820 | /opt/dbbasic-object-server |
| 4821 | /var/lib/dbbasic-object-server/objects |
| 4822 | /var/lib/dbbasic-object-server/data |
| 4823 | ``` |
| 4824 | |
| 4825 | - `/opt/dbbasic-object-server` is the git checkout. |
| 4826 | - `/var/lib/dbbasic-object-server/objects` contains live object source files. |
| 4827 | - `/var/lib/dbbasic-object-server/data` contains state, logs, and versions. |
| 4828 | |
| 4829 | Keeping objects and data outside the checkout lets the server be upgraded |
| 4830 | without mixing application data into git history. |
| 4831 | |
| 4832 | ## Install |
| 4833 | |
| 4834 | On a fresh Debian or Ubuntu VM: |
| 4835 | |
| 4836 | ```bash |
| 4837 | sudo apt update |
| 4838 | sudo apt install -y python3 python3-venv python3-pip git |
| 4839 | sudo useradd --system --home /var/lib/dbbasic-object-server --shell /usr/sbin/nologin dbbasic |
| 4840 | sudo mkdir -p /opt/dbbasic-object-server |
| 4841 | sudo mkdir -p /var/lib/dbbasic-object-server/objects |
| 4842 | sudo mkdir -p /var/lib/dbbasic-object-server/data |
| 4843 | sudo chown -R dbbasic:dbbasic /opt/dbbasic-object-server /var/lib/dbbasic-object-server |
| 4844 | sudo chmod 755 /opt/dbbasic-object-server |
| 4845 | sudo chmod 750 /var/lib/dbbasic-object-server |
| 4846 | sudo chmod 750 /var/lib/dbbasic-object-server/objects |
| 4847 | sudo chmod 750 /var/lib/dbbasic-object-server/data |
| 4848 | ``` |
| 4849 | |
| 4850 | Clone and install the server: |
| 4851 | |
| 4852 | ```bash |
| 4853 | sudo -u dbbasic git clone <repository-url> /opt/dbbasic-object-server |
| 4854 | cd /opt/dbbasic-object-server |
| 4855 | sudo -u dbbasic python3 -m venv .venv |
| 4856 | sudo -u dbbasic .venv/bin/python -m pip install -e '.[server]' |
| 4857 | ``` |
| 4858 | |
| 4859 | ## Environment |
| 4860 | |
| 4861 | Create `/etc/dbbasic-object-server.env`: |
| 4862 | |
| 4863 | ```text |
| 4864 | DBBASIC_OBJECTS_DIR=/var/lib/dbbasic-object-server/objects |
| 4865 | DBBASIC_DATA_DIR=/var/lib/dbbasic-object-server/data |
| 4866 | DBBASIC_BACKUPS_DIR=/var/lib/dbbasic-object-server/data/backups |
| 4867 | DBBASIC_PACKAGES_DIR=/opt/dbbasic-object-server/packages |
| 4868 | DBBASIC_ENABLE_SOURCE_WRITES=false |
| 4869 | DBBASIC_ENABLE_FILE_WRITES=false |
| 4870 | DBBASIC_ENABLE_PACKAGE_INSTALLS=false |
| 4871 | DBBASIC_ENABLE_PACKAGE_RESTORE=false |
| 4872 | # AI chat, voice, and reader -- each is its own flag; enabling AI chat does |
| 4873 | # NOT enable TTS/STT/reader. Each also needs a per-user provider key stored |
| 4874 | # via PUT /identity/users/{id}/service-keys before it does anything -- see |
| 4875 | # docs/shell-and-ai.md and docs/secrets-and-credentials.md. |
| 4876 | DBBASIC_ENABLE_AI_CHAT=false |
| 4877 | DBBASIC_AI_TIMEOUT_SECONDS=60 |
| 4878 | DBBASIC_AI_DEFAULT_MODEL=anthropic:claude-haiku-4-5 |
| 4879 | DBBASIC_ENABLE_TTS=false |
| 4880 | DBBASIC_TTS_CLOUD_TIMEOUT_SECONDS=30 |
| 4881 | # STT costs real money on every utterance (not just occasional replies |
| 4882 | # like TTS) -- leave this off even where TTS is on unless you mean it. |
| 4883 | DBBASIC_ENABLE_STT=false |
| 4884 | DBBASIC_STT_TIMEOUT_SECONDS=30 |
| 4885 | DBBASIC_ENABLE_READER=false |
| 4886 | DBBASIC_ADMIN_TOKEN=replace-with-a-generated-token |
| 4887 | DBBASIC_MAX_REQUEST_BYTES=1048576 |
| 4888 | DBBASIC_MAX_OBJECT_FILE_BYTES=1048576 |
| 4889 | DBBASIC_MAX_CONCURRENT_REQUESTS=64 |
| 4890 | DBBASIC_MAX_CONCURRENT_EXECUTIONS=8 |
| 4891 | DBBASIC_OBJECT_TIMEOUT_SECONDS=5 |
| 4892 | DBBASIC_TRUSTED_IN_PROCESS_OBJECTS=site_home |
| 4893 | DBBASIC_RATE_LIMIT_REQUESTS=1000 |
| 4894 | DBBASIC_RATE_LIMIT_WINDOW_SECONDS=60 |
| 4895 | DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS=true |
| 4896 | DBBASIC_ENABLE_PERMISSION_AUDIT=false |
| 4897 | DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=false |
| 4898 | DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=false |
| 4899 | DBBASIC_PERMISSION_TRUST_HEADERS=false |
| 4900 | DBBASIC_ENABLE_SESSION_LOGIN=false |
| 4901 | DBBASIC_SESSION_LOGIN_TOKEN= |
| 4902 | DBBASIC_ENABLE_PASSWORD_LOGIN=false |
| 4903 | DBBASIC_COOKIE_SECURE=true |
| 4904 | DBBASIC_LOGIN_LOCKOUT_ATTEMPTS=5 |
| 4905 | DBBASIC_LOGIN_LOCKOUT_WINDOW_SECONDS=900 |
| 4906 | DBBASIC_ENABLE_RECORD_EVENTS=true |
| 4907 | DBBASIC_EVENT_KEEP_COUNT=1000 |
| 4908 | DBBASIC_EVENT_KEEP_SECONDS=604800 |
| 4909 | DBBASIC_LOG_MAX_BYTES=10485760 |
| 4910 | DBBASIC_LOG_COMPRESS_ROTATED=true |
| 4911 | DBBASIC_LOG_KEEP_ROTATED=32 |
| 4912 | # Other feature gates worth knowing exist -- off by default, turn on what |
| 4913 | # you actually use rather than everything at once. |
| 4914 | DBBASIC_ENABLE_USER_FILES=false |
| 4915 | DBBASIC_ENABLE_REALTIME=false |
| 4916 | DBBASIC_ENABLE_SITE_ROUTES=false |
| 4917 | DBBASIC_ENABLE_FEED=false |
| 4918 | DBBASIC_ENABLE_FILTERING=false |
| 4919 | DBBASIC_ENABLE_CONCURRENCY=false |
| 4920 | DBBASIC_ENABLE_SESSION_ADMIN_GATES=false |
| 4921 | ``` |
| 4922 | |
| 4923 | Generate a local token on the VM: |
| 4924 | |
| 4925 | ```bash |
| 4926 | python3 -c ""import secrets; print(secrets.token_urlsafe(32))"" |
| 4927 | ``` |
| 4928 | |
| 4929 | Use that generated value for `DBBASIC_ADMIN_TOKEN`. Do not reuse test tokens, |
| 4930 | README placeholders, or tokens from another server. It is required for local or |
| 4931 | admin object listing and introspection requests such as source, state, logs, |
| 4932 | metadata, and versions. |
| 4933 | |
| 4934 | ## Bootstrap Browser Login |
| 4935 | |
| 4936 | To enable browser login on a fresh VM, create the first admin user from the |
| 4937 | shell and turn on password login: |
| 4938 | |
| 4939 | ```bash |
| 4940 | cd /opt/dbbasic-object-server |
| 4941 | sudo -u dbbasic DBBASIC_DATA_DIR=/var/lib/dbbasic-object-server/data \ |
| 4942 | .venv/bin/python object_identity_cli.py create-superuser \ |
| 4943 | --user-id operator --email you@example.com |
| 4944 | ``` |
| 4945 | |
| 4946 | The password is prompted twice without echo and stored only as a scrypt hash |
| 4947 | under `data/identity/credentials.tsv` (owner-only permissions, inside the |
| 4948 | runtime data directory that never enters source control). Then set |
| 4949 | `DBBASIC_ENABLE_PASSWORD_LOGIN=true` in the environment file, restart the |
| 4950 | service, and allow `/login` and `/logout` through the reverse proxy. Keep |
| 4951 | `DBBASIC_COOKIE_SECURE=true` behind HTTPS; set it to `false` only for |
| 4952 | plain-HTTP local development. Additional users can be managed with the same |
| 4953 | CLI (`create-user`, `set-password`, `list-users`) or through the admin HTTP |
| 4954 | surface from Scroll. |
| 4955 | |
| 4956 | For the first VM boot, leave source writes disabled. After health checks, |
| 4957 | backups, and proxy access are working, a staging server can enable source |
| 4958 | writes: |
| 4959 | |
| 4960 | ```text |
| 4961 | DBBASIC_ENABLE_SOURCE_WRITES=true |
| 4962 | ``` |
| 4963 | |
| 4964 | Leave file writes disabled on public staging until quotas, permissions, and |
| 4965 | content policy are ready for the use case. A controlled operator VM can enable |
| 4966 | them separately: |
| 4967 | |
| 4968 | ```text |
| 4969 | DBBASIC_ENABLE_FILE_WRITES=true |
| 4970 | ``` |
| 4971 | |
| 4972 | Permission audit and enforcement should also stay closed on first boot. Turn on |
| 4973 | audit mode first to watch route decisions without blocking users: |
| 4974 | |
| 4975 | ```text |
| 4976 | DBBASIC_ENABLE_PERMISSION_AUDIT=true |
| 4977 | ``` |
| 4978 | |
| 4979 | Request blocking only after the persisted policy and auth gateway are confirmed: |
| 4980 | |
| 4981 | ```text |
| 4982 | DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true |
| 4983 | ``` |
| 4984 | |
| 4985 | That flag is a request, not a blind switch. The server checks the same |
| 4986 | readiness blockers exposed by `/permissions/status`; if the policy is invalid or |
| 4987 | the default role-based policy has no grants, the route stays in audit/shadow |
| 4988 | mode and `permissions.enforcement_blocked` reports `true`. |
| 4989 | |
| 4990 | For manual recovery or a narrow test, the readiness guard can be bypassed: |
| 4991 | |
| 4992 | ```text |
| 4993 | DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true |
| 4994 | ``` |
| 4995 | |
| 4996 | Do not leave that override enabled on public staging. |
| 4997 | |
| 4998 | `DBBASIC_PERMISSION_TRUST_HEADERS=true` is only appropriate when a trusted proxy |
| 4999 | or auth gateway strips client-supplied identity headers and writes fresh |
| 5000 | `X-DBBASIC-*` headers itself. |
| 5001 | |
| 5002 | Do not commit real tokens, VM hostnames, private URLs, or deployment-specific |
| 5003 | paths to this repository. |
| 5004 | |
| 5005 | After creating the file, keep it readable by root and the service group only: |
| 5006 | |
| 5007 | ```bash |
| 5008 | sudo chown root:dbbasic /etc/dbbasic-object-server.env |
| 5009 | sudo chmod 640 /etc/dbbasic-object-server.env |
| 5010 | ``` |
| 5011 | |
| 5012 | ## Filesystem Check |
| 5013 | |
| 5014 | The public package includes a small single-VM layout checker. Run it after |
| 5015 | installing the server and after changing ownership or deployment paths: |
| 5016 | |
| 5017 | ```bash |
| 5018 | cd /opt/dbbasic-object-server |
| 5019 | set -a |
| 5020 | . /etc/dbbasic-object-server.env |
| 5021 | set +a |
| 5022 | .venv/bin/python -m deployment_checks |
| 5023 | ``` |
| 5024 | |
| 5025 | The checker validates the normal Unix boundary: |
| 5026 | |
| 5027 | - server code under `/opt/dbbasic-object-server` |
| 5028 | - live object source under `/var/lib/dbbasic-object-server/objects` |
| 5029 | - runtime data under `/var/lib/dbbasic-object-server/data` |
| 5030 | - deployment secrets under `/etc/dbbasic-object-server.env` |
| 5031 | - systemd unit under `/etc/systemd/system/dbbasic-object-server.service` |
| 5032 | |
| 5033 | Errors should be fixed before exposing routes publicly. Warnings usually mean a |
| 5034 | runtime path is visible to other local users. On a single-purpose staging VM |
| 5035 | that may not break anything, but the safer default is `750` for object and data |
| 5036 | directories. |
| 5037 | |
| 5038 | ## Traffic Limits |
| 5039 | |
| 5040 | Keep the public server narrow while it is staging. |
| 5041 | |
| 5042 | `DBBASIC_MAX_REQUEST_BYTES` limits inbound HTTP request bodies before JSON |
| 5043 | parsing or object execution. The default is 1 MiB: |
| 5044 | |
| 5045 | ```text |
| 5046 | DBBASIC_MAX_REQUEST_BYTES=1048576 |
| 5047 | DBBASIC_MAX_CONCURRENT_REQUESTS=64 |
| 5048 | DBBASIC_MAX_CONCURRENT_EXECUTIONS=8 |
| 5049 | ``` |
| 5050 | |
| 5051 | Configure the same or smaller request body limit in the reverse proxy so Caddy |
| 5052 | or nginx can reject oversized traffic before Python handles it. The app-level |
| 5053 | limit stays in place because proxy configuration can drift. |
| 5054 | |
| 5055 | The concurrency limits are per process. Under overload, the server returns |
| 5056 | `503` rather than queueing unlimited work on a small VM. Future production |
| 5057 | hardening still needs CPU/memory isolation and a longer-lived worker pool. |
| 5058 | `DBBASIC_OBJECT_TIMEOUT_SECONDS` runs object methods in a worker process and |
| 5059 | returns `504` if the wall-clock timeout is exceeded. |
| 5060 | `DBBASIC_TRUSTED_IN_PROCESS_OBJECTS` is a comma-separated allowlist for |
| 5061 | reviewed server-owned objects that should keep the fast in-process path, such as |
| 5062 | the public homepage. Do not add user-created objects to this list. |
| 5063 | The rate limit values above return `429` with `Retry-After` when one IP or |
| 5064 | valid admin token exceeds the configured window. |
| 5065 | `DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS` is appropriate only because uvicorn is |
| 5066 | bound to `127.0.0.1` behind Caddy. |
| 5067 | |
| 5068 | See [`traffic-limits.md`](/docs/traffic-limits) for the operating model. |
| 5069 | |
| 5070 | ## Log Maintenance |
| 5071 | |
| 5072 | Object logs are part of the runtime feedback loop, so staging servers should not |
| 5073 | let them grow forever. |
| 5074 | |
| 5075 | The active log for an object stays plain TSV: |
| 5076 | |
| 5077 | ```text |
| 5078 | /var/lib/dbbasic-object-server/data/logs/{object_id}/log.tsv |
| 5079 | ``` |
| 5080 | |
| 5081 | When it reaches `DBBASIC_LOG_MAX_BYTES`, the server rotates it. New rotated logs |
| 5082 | are gzip-compressed by default: |
| 5083 | |
| 5084 | ```text |
| 5085 | /var/lib/dbbasic-object-server/data/logs/{object_id}/log-YYYYMMDD-HHMMSS-ffffff.tsv.gz |
| 5086 | ``` |
| 5087 | |
| 5088 | Inspect compressed logs without expanding a second copy: |
| 5089 | |
| 5090 | ```bash |
| 5091 | gzip -cd /var/lib/dbbasic-object-server/data/logs/site_home/log-*.tsv.gz |
| 5092 | ``` |
| 5093 | |
| 5094 | Garbage collection is controlled by `DBBASIC_LOG_KEEP_ROTATED`. The default |
| 5095 | keeps the newest 32 rotated files per object and deletes older rotated logs |
| 5096 | after a successful rotation. Set it to `0` to keep all rotated logs. |
| 5097 | |
| 5098 | This is at-rest compression. For in-motion compression, start with reverse-proxy |
| 5099 | HTTP compression for large log responses. Later station replication should |
| 5100 | compress batches, not individual tiny events. |
| 5101 | |
| 5102 | Event rows are a delivery queue, not the durable audit log. The server and |
| 5103 | daemon prune old `data/state/events/state.tsv` event rows with: |
| 5104 | |
| 5105 | ```bash |
| 5106 | DBBASIC_EVENT_KEEP_COUNT=1000 |
| 5107 | DBBASIC_EVENT_KEEP_SECONDS=604800 |
| 5108 | ``` |
| 5109 | |
| 5110 | Set either value to `0` to disable that retention rule. Subscriptions stay in |
| 5111 | place, and any event referenced by a subscription `last_event_id` is protected so |
| 5112 | the daemon keeps its delivery cursor. Failed delivery attempts are protected too, |
| 5113 | so callback failures stay visible and retryable until delivery succeeds or the |
| 5114 | subscription is repaired. |
| 5115 | |
| 5116 | ## Packages |
| 5117 | |
| 5118 | Packages are read from `DBBASIC_PACKAGES_DIR`. The public server currently |
| 5119 | supports package listing, dry-run planning, and package installs only when |
| 5120 | `DBBASIC_ENABLE_PACKAGE_INSTALLS=true`. Dry-run and install requests append |
| 5121 | compact audit rows under |
| 5122 | `/var/lib/dbbasic-object-server/data/package_changes/{package_id}/changes.jsonl`. |
| 5123 | Package installs create a runtime restore point before writing source, schema, |
| 5124 | or seed files. Restore points default to |
| 5125 | `/var/lib/dbbasic-object-server/data/backups/`; set `DBBASIC_BACKUPS_DIR` if |
| 5126 | they should live on another mounted volume. |
| 5127 | |
| 5128 | Package restore is separate and requires `DBBASIC_ENABLE_PACKAGE_RESTORE=true`. |
| 5129 | It restores a recorded install restore point with overwrite and snapshot |
| 5130 | pruning, so package-created runtime files are removed if they were not in the |
| 5131 | archive. |
| 5132 | |
| 5133 | Keep `DBBASIC_ENABLE_PACKAGE_INSTALLS=false` and |
| 5134 | `DBBASIC_ENABLE_PACKAGE_RESTORE=false` on public staging unless `/packages` is |
| 5135 | reachable only from a private admin surface. The first install path writes |
| 5136 | objects and schemas, creates seed TSV files only when no data exists, and |
| 5137 | rejects permissions/migrations until merge/run semantics are explicit. |
| 5138 | |
| 5139 | ## Minimal Object |
| 5140 | |
| 5141 | Create one object so the VM can prove execution, state, and logs: |
| 5142 | |
| 5143 | ```bash |
| 5144 | sudo -u dbbasic mkdir -p /var/lib/dbbasic-object-server/objects/site |
| 5145 | sudo -u dbbasic tee /var/lib/dbbasic-object-server/objects/site/home.py >/dev/null <<'PY' |
| 5146 | def GET(request): |
| 5147 | count = _state_manager.get(""count"", 0) + 1 |
| 5148 | _state_manager.set(""count"", count) |
| 5149 | return { |
| 5150 | ""status"": ""ok"", |
| 5151 | ""message"": ""DBBASIC Object Server is running"", |
| 5152 | ""count"": count, |
| 5153 | } |
| 5154 | PY |
| 5155 | ``` |
| 5156 | |
| 5157 | This object resolves as `site_home`. |
| 5158 | |
| 5159 | For a public staging page, the same object can return HTML instead of JSON: |
| 5160 | |
| 5161 | ```python |
| 5162 | def GET(request): |
| 5163 | count = int(_state_manager.get(""count"", 0)) + 1 |
| 5164 | _state_manager.set(""count"", count) |
| 5165 | _logger.info(""site_home served"", count=count, response_type=""html"") |
| 5166 | |
| 5167 | return { |
| 5168 | ""content_type"": ""text/html; charset=utf-8"", |
| 5169 | ""body"": f""<!doctype html><h1>DBBASIC Object Server</h1><p>Served {count} times.</p>"", |
| 5170 | } |
| 5171 | ``` |
| 5172 | |
| 5173 | That keeps the staging page honest: it is a live object with state and logs, not |
| 5174 | a static file. |
| 5175 | |
| 5176 | ## Run Manually |
| 5177 | |
| 5178 | Before creating a service, run the server by hand: |
| 5179 | |
| 5180 | ```bash |
| 5181 | cd /opt/dbbasic-object-server |
| 5182 | set -a |
| 5183 | . /etc/dbbasic-object-server.env |
| 5184 | set +a |
| 5185 | .venv/bin/uvicorn object_server:app --host 127.0.0.1 --port 8001 --no-access-log |
| 5186 | ``` |
| 5187 | |
| 5188 | In another shell: |
| 5189 | |
| 5190 | ```bash |
| 5191 | curl http://127.0.0.1:8001/health |
| 5192 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/health?capacity=true' |
| 5193 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" http://127.0.0.1:8001/objects |
| 5194 | curl http://127.0.0.1:8001/objects/site_home |
| 5195 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/objects/site_home?state=true' |
| 5196 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/objects/site_home?logs=true' |
| 5197 | ``` |
| 5198 | |
| 5199 | ## systemd |
| 5200 | |
| 5201 | Create `/etc/systemd/system/dbbasic-object-server.service`: |
| 5202 | |
| 5203 | ```ini |
| 5204 | [Unit] |
| 5205 | Description=DBBASIC Object Server |
| 5206 | After=network-online.target |
| 5207 | Wants=network-online.target |
| 5208 | |
| 5209 | [Service] |
| 5210 | User=dbbasic |
| 5211 | Group=dbbasic |
| 5212 | WorkingDirectory=/opt/dbbasic-object-server |
| 5213 | EnvironmentFile=/etc/dbbasic-object-server.env |
| 5214 | ExecStart=/opt/dbbasic-object-server/.venv/bin/uvicorn object_server:app --host 127.0.0.1 --port 8001 --no-access-log |
| 5215 | Restart=on-failure |
| 5216 | RestartSec=2 |
| 5217 | NoNewPrivileges=true |
| 5218 | PrivateTmp=true |
| 5219 | ProtectSystem=full |
| 5220 | ReadWritePaths=/var/lib/dbbasic-object-server |
| 5221 | |
| 5222 | [Install] |
| 5223 | WantedBy=multi-user.target |
| 5224 | ``` |
| 5225 | |
| 5226 | Start it: |
| 5227 | |
| 5228 | ```bash |
| 5229 | sudo systemctl daemon-reload |
| 5230 | sudo systemctl enable --now dbbasic-object-server |
| 5231 | sudo systemctl status dbbasic-object-server |
| 5232 | curl http://127.0.0.1:8001/health |
| 5233 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/health?capacity=true' |
| 5234 | ``` |
| 5235 | |
| 5236 | Watch logs: |
| 5237 | |
| 5238 | ```bash |
| 5239 | sudo journalctl -u dbbasic-object-server -f |
| 5240 | ``` |
| 5241 | |
| 5242 | `--no-access-log` keeps successful request lines out of journald. DBBASIC keeps |
| 5243 | the useful application trail in object logs and `/health?metrics=true`, which |
| 5244 | are the surfaces Scroll should read. If access analytics are needed later, |
| 5245 | build them as an explicit app/object instead of relying on raw process logs. |
| 5246 | |
| 5247 | Cap journald so process logs cannot consume the VM: |
| 5248 | |
| 5249 | ```bash |
| 5250 | sudo mkdir -p /etc/systemd/journald.conf.d |
| 5251 | sudo tee /etc/systemd/journald.conf.d/99-dbbasic.conf >/dev/null <<'EOF' |
| 5252 | [Journal] |
| 5253 | SystemMaxUse=128M |
| 5254 | SystemKeepFree=1G |
| 5255 | MaxRetentionSec=7day |
| 5256 | Compress=yes |
| 5257 | EOF |
| 5258 | sudo systemctl restart systemd-journald |
| 5259 | sudo journalctl --vacuum-size=128M |
| 5260 | sudo journalctl --disk-usage |
| 5261 | ``` |
| 5262 | |
| 5263 | ## Upgrade |
| 5264 | |
| 5265 | **Two different things live at `/opt` and two different things get served |
| 5266 | at `/var/lib` — know which one you changed before declaring a deploy done.** |
| 5267 | |
| 5268 | Core server modules (`object_server.py`, `object_ai.py`, and every other |
| 5269 | `.py` at the repo root) are imported directly from |
| 5270 | `/opt/dbbasic-object-server` — the steps below are complete for these: |
| 5271 | |
| 5272 | ```bash |
| 5273 | cd /opt/dbbasic-object-server |
| 5274 | sudo -u dbbasic git pull --ff-only |
| 5275 | sudo -u dbbasic .venv/bin/python -m pip install -e '.[server]' |
| 5276 | sudo systemctl restart dbbasic-object-server |
| 5277 | ``` |
| 5278 | |
| 5279 | **Anything under `packages/<name>/` (objects, schemas, seed data) is |
| 5280 | different.** The running server reads these from a separately *installed* |
| 5281 | copy at `/var/lib/dbbasic-object-server/objects/` and |
| 5282 | `/var/lib/dbbasic-object-server/data/schemas/` — `git pull` alone never |
| 5283 | syncs into it, no matter how many times you restart. A `git pull`-only |
| 5284 | ""deploy"" of a `packages/` change is a silent no-op: the commit lands, the |
| 5285 | tests pass, the service restarts clean, and the live server keeps running |
| 5286 | the old code with no error anywhere. After the `git pull` above, also run: |
| 5287 | |
| 5288 | ```bash |
| 5289 | curl -s -X POST https://dbbasic.example.com/packages/app-shell/install \ |
| 5290 | -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" -H 'Content-Type: application/json' \ |
| 5291 | -d '{""allow_replace"": true}' |
| 5292 | ``` |
| 5293 | |
| 5294 | (swap `app-shell` for whichever package changed; `DBBASIC_ENABLE_PACKAGE_INSTALLS` |
| 5295 | must be `true`). Check the response's `install.objects[].status` — `""updated""` |
| 5296 | means a real change landed, `""unchanged""` means nothing to sync. A schema-only |
| 5297 | change can also go straight to `PUT /admin/schemas/{collection}` with the |
| 5298 | updated schema JSON, same admin-token gate. Either way, **restart the |
| 5299 | service afterward** — object modules are cached in the running process, so |
| 5300 | even a correctly-installed file has no effect until the next restart clears |
| 5301 | that cache: |
| 5302 | |
| 5303 | ```bash |
| 5304 | sudo systemctl restart dbbasic-object-server |
| 5305 | ``` |
| 5306 | |
| 5307 | To confirm a `packages/` deploy actually landed rather than trusting the |
| 5308 | install response alone, diff the live file against the source directly: |
| 5309 | |
| 5310 | ```bash |
| 5311 | diff /var/lib/dbbasic-object-server/objects/site/<name>.py \ |
| 5312 | /opt/dbbasic-object-server/packages/<pkg>/objects/site/<name>.py |
| 5313 | ``` |
| 5314 | |
| 5315 | A non-empty diff means the install step (or the restart) didn't happen yet. |
| 5316 | |
| 5317 | Then verify the local service, filesystem layout, public proxy, and deployed |
| 5318 | commit: |
| 5319 | |
| 5320 | ```bash |
| 5321 | cd /opt/dbbasic-object-server |
| 5322 | set -a |
| 5323 | . /etc/dbbasic-object-server.env |
| 5324 | set +a |
| 5325 | .venv/bin/python -m deployment_checks |
| 5326 | curl http://127.0.0.1:8001/health |
| 5327 | curl https://dbbasic.example.com/ |
| 5328 | sudo systemctl is-active dbbasic-object-server caddy |
| 5329 | sudo -u dbbasic git rev-parse --short HEAD |
| 5330 | ``` |
| 5331 | |
| 5332 | If the upgrade only changes live objects that were edited **directly** |
| 5333 | under `/var/lib/dbbasic-object-server/objects` (not via a `packages/` git |
| 5334 | change), the server should not need a git deploy at all. That is the |
| 5335 | object loop: edit one object, run it, inspect state/logs, and keep the |
| 5336 | version trail. This is a third, separate path from both cases above — it |
| 5337 | never touches `/opt` or git at all. |
| 5338 | |
| 5339 | ## HTTPS Proxy |
| 5340 | |
| 5341 | For a public or staging hostname, put Caddy or nginx in front of uvicorn and keep |
| 5342 | uvicorn bound to `127.0.0.1`. |
| 5343 | |
| 5344 | Install Caddy if the VM does not already have a reverse proxy: |
| 5345 | |
| 5346 | ```bash |
| 5347 | sudo apt install -y caddy |
| 5348 | ``` |
| 5349 | |
| 5350 | Example Caddyfile using documentation domains: |
| 5351 | |
| 5352 | ```caddyfile |
| 5353 | dbbasic.example.com { |
| 5354 | reverse_proxy 127.0.0.1:8001 |
| 5355 | } |
| 5356 | ``` |
| 5357 | |
| 5358 | Replace `dbbasic.example.com` only on the VM. Do not commit real deployment |
| 5359 | hostnames into this repository. |
| 5360 | |
| 5361 | If Caddy was already installed, inspect what it is serving before taking over |
| 5362 | the hostname: |
| 5363 | |
| 5364 | ```bash |
| 5365 | sudo systemctl status caddy --no-pager |
| 5366 | sudo sed -n '1,220p' /etc/caddy/Caddyfile |
| 5367 | sudo find /usr/share/caddy -maxdepth 3 -type f -printf '%p\t%s bytes\n' |
| 5368 | ``` |
| 5369 | |
| 5370 | Back up the old config before changing it: |
| 5371 | |
| 5372 | ```bash |
| 5373 | sudo cp -a /etc/caddy/Caddyfile /etc/caddy/Caddyfile.before-dbbasic |
| 5374 | ``` |
| 5375 | |
| 5376 | For the earliest public staging endpoint, expose only the hello object, system |
| 5377 | dashboard object, admin status and admin inspection routes, health check, and |
| 5378 | current-session self-service route until auth, permissions, and source |
| 5379 | visibility are ready. Keep the broad `/objects` listing blocked: |
| 5380 | |
| 5381 | ```caddyfile |
| 5382 | dbbasic.example.com { |
| 5383 | handle / { |
| 5384 | rewrite * /objects/site_home |
| 5385 | reverse_proxy 127.0.0.1:8001 |
| 5386 | } |
| 5387 | |
| 5388 | handle /health { |
| 5389 | reverse_proxy 127.0.0.1:8001 |
| 5390 | } |
| 5391 | |
| 5392 | handle /login { |
| 5393 | reverse_proxy 127.0.0.1:8001 |
| 5394 | } |
| 5395 | |
| 5396 | handle /logout { |
| 5397 | reverse_proxy 127.0.0.1:8001 |
| 5398 | } |
| 5399 | |
| 5400 | handle /admin/status { |
| 5401 | reverse_proxy 127.0.0.1:8001 |
| 5402 | } |
| 5403 | |
| 5404 | handle /admin/changes* { |
| 5405 | reverse_proxy 127.0.0.1:8001 |
| 5406 | } |
| 5407 | |
| 5408 | handle /admin/objects* { |
| 5409 | reverse_proxy 127.0.0.1:8001 |
| 5410 | } |
| 5411 | |
| 5412 | handle /admin/files* { |
| 5413 | reverse_proxy 127.0.0.1:8001 |
| 5414 | } |
| 5415 | |
| 5416 | handle /admin/collections* { |
| 5417 | reverse_proxy 127.0.0.1:8001 |
| 5418 | } |
| 5419 | |
| 5420 | handle /admin/schemas* { |
| 5421 | reverse_proxy 127.0.0.1:8001 |
| 5422 | } |
| 5423 | |
| 5424 | handle /admin/identity* { |
| 5425 | reverse_proxy 127.0.0.1:8001 |
| 5426 | } |
| 5427 | |
| 5428 | handle /permissions/status* { |
| 5429 | reverse_proxy 127.0.0.1:8001 |
| 5430 | } |
| 5431 | |
| 5432 | handle /permissions/policy* { |
| 5433 | reverse_proxy 127.0.0.1:8001 |
| 5434 | } |
| 5435 | |
| 5436 | handle /daemon/status { |
| 5437 | reverse_proxy 127.0.0.1:8001 |
| 5438 | } |
| 5439 | |
| 5440 | handle /daemon/scheduler/tasks* { |
| 5441 | reverse_proxy 127.0.0.1:8001 |
| 5442 | } |
| 5443 | |
| 5444 | handle /daemon/queue/messages* { |
| 5445 | reverse_proxy 127.0.0.1:8001 |
| 5446 | } |
| 5447 | |
| 5448 | handle /events* { |
| 5449 | reverse_proxy 127.0.0.1:8001 |
| 5450 | } |
| 5451 | |
| 5452 | handle /packages* { |
| 5453 | reverse_proxy 127.0.0.1:8001 |
| 5454 | } |
| 5455 | |
| 5456 | handle /api/mcp { |
| 5457 | reverse_proxy 127.0.0.1:8001 |
| 5458 | } |
| 5459 | |
| 5460 | handle_path /static/* { |
| 5461 | root * /var/lib/dbbasic-static |
| 5462 | file_server |
| 5463 | header Cache-Control ""public, max-age=3600"" |
| 5464 | } |
| 5465 | |
| 5466 | handle /permissions/audit* { |
| 5467 | reverse_proxy 127.0.0.1:8001 |
| 5468 | } |
| 5469 | |
| 5470 | handle /dashboard { |
| 5471 | rewrite * /objects/system_dashboard |
| 5472 | reverse_proxy 127.0.0.1:8001 |
| 5473 | } |
| 5474 | |
| 5475 | handle /admin/write-probe { |
| 5476 | rewrite * /objects/system_write_probe |
| 5477 | reverse_proxy 127.0.0.1:8001 |
| 5478 | } |
| 5479 | |
| 5480 | handle /collections/dbbasic_probe/records* { |
| 5481 | reverse_proxy 127.0.0.1:8001 |
| 5482 | } |
| 5483 | |
| 5484 | handle /collections/dbbasic_probe/changes { |
| 5485 | reverse_proxy 127.0.0.1:8001 |
| 5486 | } |
| 5487 | |
| 5488 | handle /schemas/dbbasic_probe { |
| 5489 | reverse_proxy 127.0.0.1:8001 |
| 5490 | } |
| 5491 | |
| 5492 | handle /identity/session { |
| 5493 | reverse_proxy 127.0.0.1:8001 |
| 5494 | } |
| 5495 | |
| 5496 | # This exposes GET/DELETE current-session checks. POST password login works |
| 5497 | # when DBBASIC_ENABLE_PASSWORD_LOGIN=true; the gateway session-mint POST |
| 5498 | # remains disabled unless DBBASIC_ENABLE_SESSION_LOGIN=true and |
| 5499 | # DBBASIC_SESSION_LOGIN_TOKEN are set. |
| 5500 | |
| 5501 | handle /objects/site_home* { |
| 5502 | reverse_proxy 127.0.0.1:8001 |
| 5503 | } |
| 5504 | |
| 5505 | handle { |
| 5506 | respond ""Not found"" 404 |
| 5507 | } |
| 5508 | } |
| 5509 | ``` |
| 5510 | |
| 5511 | Validate and reload: |
| 5512 | |
| 5513 | ```bash |
| 5514 | sudo caddy fmt --overwrite /etc/caddy/Caddyfile |
| 5515 | sudo caddy validate --config /etc/caddy/Caddyfile |
| 5516 | sudo systemctl reload caddy |
| 5517 | ``` |
| 5518 | |
| 5519 | Then test: |
| 5520 | |
| 5521 | ```bash |
| 5522 | curl https://dbbasic.example.com/ |
| 5523 | curl https://dbbasic.example.com/health |
| 5524 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/status |
| 5525 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/collections |
| 5526 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/schemas |
| 5527 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/identity/users |
| 5528 | curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/daemon/status |
| 5529 | curl https://dbbasic.example.com/objects/site_home |
| 5530 | curl https://dbbasic.example.com/admin/write-probe |
| 5531 | curl https://dbbasic.example.com/objects |
| 5532 | ``` |
| 5533 | |
| 5534 | The first six should return responses from the object server. The root and |
| 5535 | `site_home` responses may be JSON or HTML depending on the object. The full |
| 5536 | `/objects` route should stay blocked by Caddy in this early staging mode. |
| 5537 | |
| 5538 | The `admin-write-probe` package may also expose the narrow |
| 5539 | `/collections/dbbasic_probe/records*`, `/collections/dbbasic_probe/changes`, |
| 5540 | and `/schemas/dbbasic_probe` routes. Those routes are still protected by the |
| 5541 | server-side admin-token gate. They exist to prove collection create/update/ |
| 5542 | delete behavior, record changelog reads, and schema reads through the public |
| 5543 | reverse proxy without exposing every collection or schema route. Do not expose |
| 5544 | broad `/collections*`, `/schemas*`, `/objects*`, or source write routes on a |
| 5545 | public staging server. Use `/admin/collections*` and `/admin/schemas*` for |
| 5546 | operator inspection instead. Use `/admin/identity*` for read-only account, user, |
| 5547 | and session inspection instead of exposing broad identity management routes. |
| 5548 | |
| 5549 | ## Public Code Execution Controls |
| 5550 | |
| 5551 | In the early staging deployment, do not expose the full object server API |
| 5552 | directly to the internet. |
| 5553 | |
| 5554 | Use all of these controls together: |
| 5555 | |
| 5556 | - keep uvicorn bound to `127.0.0.1` |
| 5557 | - expose only specific routes through Caddy |
| 5558 | - keep `DBBASIC_ENABLE_SOURCE_WRITES=false` |
| 5559 | - do not add a public object-create route until permissions are enforced |
| 5560 | - keep live object source and data out of the git checkout |
| 5561 | |
| 5562 | With that shape, outside users can only run the explicitly proxied objects. They |
| 5563 | cannot list every object, read source, write source, roll back source, or create |
| 5564 | new runnable code through the public hostname. |
| 5565 | |
| 5566 | ## Backup |
| 5567 | |
| 5568 | The portable runtime backup set is: |
| 5569 | |
| 5570 | ```text |
| 5571 | /var/lib/dbbasic-object-server/objects |
| 5572 | /var/lib/dbbasic-object-server/data/state |
| 5573 | /var/lib/dbbasic-object-server/data/logs |
| 5574 | /var/lib/dbbasic-object-server/data/versions |
| 5575 | /var/lib/dbbasic-object-server/data/source_changes |
| 5576 | /var/lib/dbbasic-object-server/data/schema_versions |
| 5577 | /var/lib/dbbasic-object-server/data/record_changes |
| 5578 | /var/lib/dbbasic-object-server/data/package_changes |
| 5579 | /var/lib/dbbasic-object-server/data/file_changes |
| 5580 | /var/lib/dbbasic-object-server/data/files |
| 5581 | /var/lib/dbbasic-object-server/data/schemas |
| 5582 | /var/lib/dbbasic-object-server/data/collections |
| 5583 | ``` |
| 5584 | |
| 5585 | Create and verify a portable runtime backup with: |
| 5586 | |
| 5587 | ```bash |
| 5588 | python -m object_backup create \ |
| 5589 | /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \ |
| 5590 | --objects-dir /var/lib/dbbasic-object-server/objects \ |
| 5591 | --data-dir /var/lib/dbbasic-object-server/data |
| 5592 | |
| 5593 | python -m object_backup verify \ |
| 5594 | /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \ |
| 5595 | --json |
| 5596 | ``` |
| 5597 | |
| 5598 | Restore into clean directories first: |
| 5599 | |
| 5600 | ```bash |
| 5601 | python -m object_backup restore \ |
| 5602 | /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \ |
| 5603 | --objects-dir /tmp/dbbasic-restore/objects \ |
| 5604 | --data-dir /tmp/dbbasic-restore/data \ |
| 5605 | --json |
| 5606 | ``` |
| 5607 | |
| 5608 | The runtime archive deliberately excludes deployment secrets, environment files, |
| 5609 | systemd service files, git history, virtualenvs, caches, lock files, temp files, |
| 5610 | and ephemeral rate-limit files. |
| 5611 | |
| 5612 | The server environment file and systemd unit still need to be backed up by an |
| 5613 | operator-controlled process: |
| 5614 | |
| 5615 | ```text |
| 5616 | /etc/dbbasic-object-server.env |
| 5617 | /etc/systemd/system/dbbasic-object-server.service |
| 5618 | ``` |
| 5619 | |
| 5620 | Keep those secret-aware backups separate from portable runtime archives. Before |
| 5621 | calling this a production deployment, restore runtime backups on a second clean |
| 5622 | VM and run the health/object checks again. |
| 5623 | |
| 5624 | See [`backup-restore.md`](/docs/backup-restore) for the runtime archive contract and restore rules. |
| 5625 | |
| 5626 | ## Current Limits |
| 5627 | |
| 5628 | This deployment shape is for staging and dogfooding. |
| 5629 | |
| 5630 | Known limits: |
| 5631 | |
| 5632 | - the current direct Python loader is not a production sandbox |
| 5633 | - source writes still use an explicit admin gate and are disabled until |
| 5634 | `DBBASIC_ENABLE_SOURCE_WRITES=true` |
| 5635 | - admin-token gates can optionally accept active admin-role DBBASIC sessions |
| 5636 | with `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true` |
| 5637 | - route permission enforcement is available, but off until a deployment enables it |
| 5638 | - real user/session auth still needs to replace trusted staging headers |
| 5639 | - WebSocket/SSE runtime behavior is still design-stage |
| 5640 | |
| 5641 | That is still useful. A clean VM lets DBBASIC prove that install, restart, |
| 5642 | object lookup, execution, state, logs, and versions work away from a developer |
| 5643 | machine. |
| 5644 | " |
| 5645 | docker-deployment Docker / Coolify Deployment Deployment & Operations 710 How to deploy DBBASIC Object Server as a container with Docker Compose or Coolify, including the state/code volume split and package upgrade caveats. "This is the second deployment path for DBBASIC Object Server, alongside the |
| 5646 | bare-VM systemd installer in [`docs/single-vm-deployment.md`](/docs/single-vm-deployment). |
| 5647 | It packages the same server into a container image so it can run under |
| 5648 | Docker Compose directly, or be deployed with [Coolify](https://coolify.io/). |
| 5649 | |
| 5650 | The goal is the same as the bare-VM path: stay boring, stay conservative on |
| 5651 | first boot, and keep the object edit/run/inspect loop working the same way. |
| 5652 | This page does not repeat what is not Docker-specific -- permissions, |
| 5653 | source-write gates, package installs, first-auth setup, and traffic limits |
| 5654 | are covered in [`docs/single-vm-deployment.md`](/docs/single-vm-deployment) and |
| 5655 | the root `README.md`; read those too. |
| 5656 | |
| 5657 | ## What Ships in the Image vs. What Must Be a Volume |
| 5658 | |
| 5659 | This is the concept the rest of this page assumes. |
| 5660 | |
| 5661 | The image is stateless. It contains the server code and the installable |
| 5662 | package fixtures under `packages/` -- both of those are tracked in git and |
| 5663 | rebuilt into the image on every `docker build`. |
| 5664 | |
| 5665 | Everything the server writes at runtime is state, not code, and must live on |
| 5666 | a mounted volume outside the image: |
| 5667 | |
| 5668 | - object source under `DBBASIC_OBJECTS_DIR` |
| 5669 | - TSV-backed records, state, logs, versions, schemas, and changelogs under |
| 5670 | `DBBASIC_DATA_DIR` |
| 5671 | - identity (accounts, users, sessions, credential hashes) |
| 5672 | - the permission policy and audit log |
| 5673 | - backups under `DBBASIC_BACKUPS_DIR` |
| 5674 | - the auto-generated admin token (see ""First Boot"" below) |
| 5675 | |
| 5676 | `Dockerfile` sets `DBBASIC_OBJECTS_DIR=/data/objects` and |
| 5677 | `DBBASIC_DATA_DIR=/data/state` as the image's default paths, and |
| 5678 | `docker-compose.yml` mounts two named volumes at exactly those paths. This |
| 5679 | mirrors the same separation `docs/single-vm-deployment.md` keeps between |
| 5680 | `/opt/dbbasic-object-server` (the git checkout) and |
| 5681 | `/var/lib/dbbasic-object-server` (runtime data) on the bare-VM path -- code |
| 5682 | and state never share a lifecycle. |
| 5683 | |
| 5684 | That split is what makes the two halves of the deploy story work |
| 5685 | independently (see ""Deploy Cadence"" below). |
| 5686 | |
| 5687 | ## Quickstart (Docker Compose) |
| 5688 | |
| 5689 | From a checkout of this repository: |
| 5690 | |
| 5691 | ```bash |
| 5692 | cp .env.example .env |
| 5693 | # edit .env if you want to change any of the conservative defaults |
| 5694 | docker compose up --build |
| 5695 | ``` |
| 5696 | |
| 5697 | This builds the image, starts the `object-server` service bound to |
| 5698 | `0.0.0.0:8001` inside the container, and publishes it on `localhost:8001`. |
| 5699 | Watch the logs for the first-boot admin token message (see ""First Boot"" |
| 5700 | below), then: |
| 5701 | |
| 5702 | ```bash |
| 5703 | curl http://localhost:8001/health |
| 5704 | ``` |
| 5705 | |
| 5706 | Object source and runtime data persist on the `objects` and `data` named |
| 5707 | volumes declared in `docker-compose.yml`, so `docker compose up --build` |
| 5708 | after a code change keeps existing objects, records, and identity intact. |
| 5709 | |
| 5710 | `docker-compose.yml` includes a commented-out, clearly-labeled optional |
| 5711 | `object-daemon` service for `object_daemon.py` (the scheduler/queue/event |
| 5712 | background worker). It is not required for the app to run; uncomment it |
| 5713 | only if you need scheduled tasks, queue processing, or webhook event |
| 5714 | delivery running in the background. |
| 5715 | |
| 5716 | ## Coolify Deployment |
| 5717 | |
| 5718 | Coolify can deploy this repository two ways: |
| 5719 | |
| 5720 | - **Dockerfile application** -- Coolify detects the `Dockerfile` at the repo |
| 5721 | root and builds/runs it directly. |
| 5722 | - **Docker Compose resource** -- Coolify deploys `docker-compose.yml` |
| 5723 | directly. |
| 5724 | |
| 5725 | The Compose path is the cleaner one for this project, because it declares |
| 5726 | the two persistent volumes explicitly instead of leaving them implicit. |
| 5727 | Prefer it unless you have a reason to use the plain Dockerfile path. |
| 5728 | |
| 5729 | Either way, in Coolify: |
| 5730 | |
| 5731 | 1. **Create the application** from this repository, pointing at the |
| 5732 | `Dockerfile` or `docker-compose.yml` per the choice above. |
| 5733 | 2. **Configure persistent storage.** In Coolify's Storage / Persistent |
| 5734 | Storage settings, add two volumes (or reuse the ones declared in |
| 5735 | `docker-compose.yml` if you deployed it as a Compose resource) mapped to |
| 5736 | the same container paths the image expects: |
| 5737 | |
| 5738 | ```text |
| 5739 | /data/objects |
| 5740 | /data/state |
| 5741 | ``` |
| 5742 | |
| 5743 | These must match `DBBASIC_OBJECTS_DIR` and `DBBASIC_DATA_DIR`. If you |
| 5744 | change one, change the other. |
| 5745 | |
| 5746 | 3. **Set environment variables** through Coolify's Environment Variables UI, |
| 5747 | using `.env.example` as the list of what to set and why. At minimum, |
| 5748 | decide whether to set `DBBASIC_ADMIN_TOKEN` explicitly (recommended once |
| 5749 | this is a real deployment) or leave it unset for the container to |
| 5750 | generate one on first boot. |
| 5751 | 4. **Enable health-check-based deploys.** The image's `HEALTHCHECK` |
| 5752 | instruction (and `docker-compose.yml`'s `healthcheck:` block) call |
| 5753 | `GET /health`. Point Coolify's deployment health check at the same |
| 5754 | endpoint so it waits for a healthy container before cutting traffic over |
| 5755 | on redeploy, giving zero-downtime deploys for server-code changes. |
| 5756 | 5. **Put a domain in front of it.** Coolify's own proxy handles TLS and |
| 5757 | hostname routing to the container's published port, the same way Caddy |
| 5758 | sits in front of, not inside, the systemd unit on the bare-VM path. This |
| 5759 | repository's docs use `object.dbbasic.com` as the example public domain; |
| 5760 | use your own. |
| 5761 | |
| 5762 | ## Deploy Cadence |
| 5763 | |
| 5764 | There are two independent kinds of change here, and only one of them needs |
| 5765 | an image rebuild: |
| 5766 | |
| 5767 | - **Server code changes** (anything in this git repository) require |
| 5768 | rebuilding and redeploying the image. Under Coolify this is a normal |
| 5769 | git-push-to-deploy flow: push to the branch Coolify watches, it rebuilds |
| 5770 | the image, runs the health check, and cuts over. |
| 5771 | - **Object, schema, and record edits made live through the admin HTTP API** |
| 5772 | (the same API described in the root `README.md` and |
| 5773 | `docs/single-vm-deployment.md`) land directly on the persistent volumes. |
| 5774 | They take effect on the next request, with no image rebuild and no |
| 5775 | redeploy, exactly like the bare-VM deployment's edit/run/inspect loop. |
| 5776 | |
| 5777 | This is the reason the volume split above exists: redeploying the image |
| 5778 | never touches `/data/objects` or `/data/state`, so in-flight object work |
| 5779 | survives ordinary code deploys. |
| 5780 | |
| 5781 | **A `packages/<name>/` change (in git) needs BOTH steps, not either/or.** |
| 5782 | An image rebuild brings the new `packages/` source into the image, but |
| 5783 | `/data/objects` and `/data/state/schemas` are the persistent volumes the |
| 5784 | running container actually serves from -- the same separate-copy split |
| 5785 | `docs/single-vm-deployment.md`'s Upgrade section documents for the bare-VM |
| 5786 | case, and the entrypoint does not auto-sync it. After the new image is |
| 5787 | live, still call `POST /packages/{id}/install` (`{""allow_replace"": true}`, |
| 5788 | admin-token gated) for the changed package, confirm |
| 5789 | `install.objects[].status` says `""updated""`, and expect the object module |
| 5790 | cache to need the container's next natural restart (or force one) to pick |
| 5791 | it up. Skipping this step is a silent no-op exactly like the bare-VM case: |
| 5792 | new image, tests green, health check passes, and the container keeps |
| 5793 | serving the old package code with no error anywhere. |
| 5794 | |
| 5795 | ## First Boot: the Admin Token |
| 5796 | |
| 5797 | If `DBBASIC_ADMIN_TOKEN` is not set, `scripts/docker-entrypoint.sh` (the |
| 5798 | image's container entrypoint) generates one, writes it to a file under the |
| 5799 | persistent data volume so it survives restarts and rebuilds, and prints it |
| 5800 | once, clearly, to the container's logs: |
| 5801 | |
| 5802 | ```bash |
| 5803 | docker compose logs object-server | grep -A4 ""generated a new admin token"" |
| 5804 | ``` |
| 5805 | |
| 5806 | or, in Coolify, check the application's log viewer after first deploy. |
| 5807 | |
| 5808 | The token is never baked into the image and never written anywhere in the |
| 5809 | git checkout -- only into the runtime-only data volume. It is not |
| 5810 | regenerated or overwritten on later boots. |
| 5811 | |
| 5812 | Once you have confirmed the deployment works, set `DBBASIC_ADMIN_TOKEN` |
| 5813 | explicitly (in `.env` for Compose, or in Coolify's environment variables |
| 5814 | UI) rather than relying on the generated value. That gives you a token you |
| 5815 | control and can rotate deliberately, and it is required if you ever need to |
| 5816 | recreate the data volume from a backup. |
| 5817 | |
| 5818 | ## See Also |
| 5819 | |
| 5820 | For everything not specific to Docker -- write gates, package installs, |
| 5821 | permission audit/enforcement, traffic limits, backups, and bootstrapping the |
| 5822 | first admin user and password login -- see: |
| 5823 | |
| 5824 | - [`docs/single-vm-deployment.md`](/docs/single-vm-deployment) |
| 5825 | - the root `README.md`, especially ""First Auth Setup"" and |
| 5826 | ""Public Repository Safety"" |
| 5827 | |
| 5828 | ## Security Notes |
| 5829 | |
| 5830 | These mirror the root README's ""Public Repository Safety"" checklist: |
| 5831 | |
| 5832 | - Never commit a real `.env`. Only `.env.example` (placeholder values and |
| 5833 | comments) belongs in git; `.env` is git-ignored. |
| 5834 | - Never bake a real admin token, password hash, or session secret into an |
| 5835 | image layer. The entrypoint's generated token is written only to the |
| 5836 | mounted data volume, not to any path that gets committed or that survives |
| 5837 | outside that volume. |
| 5838 | - Treat the persistent data volume as containing credentials-equivalent |
| 5839 | data: it holds the admin token file, identity credential hashes, and |
| 5840 | session records. Back it up and restrict access to it the same way you |
| 5841 | would `/var/lib/dbbasic-object-server` on the bare-VM path. |
| 5842 | - Do not commit real IPs, hostnames, or deployment-specific details into |
| 5843 | this repository, including into Coolify configuration you might otherwise |
| 5844 | be tempted to paste into a doc or issue. |
| 5845 | " |
| 5846 | upgrade-and-customization Upgrades and Customization Deployment & Operations 720 The doctrine for letting a package upgrade in place while an operator's data and customizations survive, via provenance baselines, reconcile-not-replace, overlays, additive schemas, hooks, and feature flags. "This is the doctrine for the hardest problem in packaged app systems: **how a third-party package upgrades in place while the operator's data AND customizations survive**. Rails and Django never solved this as a platform property — apps/engines existed, but customizing meant forking and upgrading meant being your own integrator. The systems that solved it (Salesforce managed packages, Odoo modules, Debian conffiles, WordPress child themes) converged on the same handful of ideas. This document adapts them to DBBASIC so any future contributor — human or AI — builds *with* the upgrade system instead of fighting it. |
| 5847 | |
| 5848 | ## The three layers (never entangle them) |
| 5849 | |
| 5850 | Every installed app is three independent layers: |
| 5851 | |
| 5852 | 1. **Package baseline** — what the package shipped: objects, schemas, permission fragments, seed. Versioned, replaceable, owned by the package. |
| 5853 | 2. **Local customization** — what this instance changed on top: edited source, added fields, extra rules, user code. Owned by the operator. |
| 5854 | 3. **Data** — the records in `data/collections/<name>/records.tsv`. Owned by the users. Lives entirely outside the package. |
| 5855 | |
| 5856 | An upgrade replaces layer 1. It must never silently destroy layers 2 or 3. Layer 3 is solved: seed is install-once (see the package authoring guide). This document is about layer 2. |
| 5857 | |
| 5858 | ## Rule 0: provenance baselines |
| 5859 | |
| 5860 | Nothing else works without this. On install/upgrade, stamp every shipped artifact with its baseline: |
| 5861 | |
| 5862 | ```json |
| 5863 | {""package"": ""app-articles"", ""version"": ""0.2.0"", ""sha256"": ""<hash of shipped source>""} |
| 5864 | ``` |
| 5865 | |
| 5866 | stored per object and per schema (e.g. `data/package_baselines/<pkg>.json`). Then ""is this customized?"" is a computable question: hash the live source and compare it to the baseline. Without the stamp you cannot distinguish a human's edit from the shipped file, and every upgrade is a coin flip. |
| 5867 | |
| 5868 | ## Rule 1: reconcile, don't replace |
| 5869 | |
| 5870 | On upgrade, per object/schema, three-way compare `old_baseline` vs `live` vs `new_shipped` — exactly `git merge` / dpkg conffile semantics: |
| 5871 | |
| 5872 | | live vs baseline | shipped changed? | action | |
| 5873 | |---|---|---| |
| 5874 | | pristine | — | fast-forward to new version, update baseline (silent) | |
| 5875 | | customized | no | keep the customization (silent) | |
| 5876 | | customized | yes, hunks don't overlap | three-way merge, apply, flag ""merged"" | |
| 5877 | | customized | yes, conflict | **park it** — write a pending-reconcile record holding both versions, leave live untouched, surface for a human decision (keep mine / take theirs / view diff) | |
| 5878 | |
| 5879 | `allow_replace` means ""fast-forward the pristine ones."" Discarding a customization requires a separate explicit force. The existing version stores (`object_versions`, `object_schema_versions`) are the undo; the install restore point is the rollback. A ""reconcile inbox"" (admin UI screen + CLI list) holds parked conflicts — upgrades never block on them. |
| 5880 | |
| 5881 | ## Rule 2: customize by overlay, not by editing core |
| 5882 | |
| 5883 | The reconcile engine is the safety net. The *preferred* path makes conflicts structurally impossible: customizations live in different files than the package's, so upgrades and edits never touch the same bytes. |
| 5884 | |
| 5885 | - **System vs user objects.** Package-shipped objects are *system* objects (baseline-stamped, upgrade-managed). Operator-created objects are *user* objects (never touched by any upgrade). The namespace should make this visible: editing a system object warns ""this came from app-articles 0.2.0 — override instead?"" |
| 5886 | - **Override objects.** A user object can *shadow* a system object by id: the resolver checks the override root first (`objects/overrides/…` or an `overrides: {site_articles: my_articles}` map). The shipped object stays pristine and freely upgradeable; the override persists across upgrades. This is Odoo's `_inherit` / WordPress child themes done with plain files. |
| 5887 | - **Where an install writes.** A NEW package object is written to the base/system root, never the override root — `object_namespace.get_base_object_roots` is explicit that this is ""the root packages install into and reconcile against ... the pristine, upgradeable copy"". An object that ALREADY exists is rewritten wherever it already lives, so an operator's override survives every upgrade. Both halves matter: installing into the override root would make package content indistinguishable from customization, and it also silently disables event handlers (an override is invisible to the handler index — see Rule 4). |
| 5888 | - **Extension points.** Objects should invite extension instead of forking: named hooks (see Rule 4) and small overridable pieces rather than one monolith page. |
| 5889 | |
| 5890 | ## Rule 3: data fields that survive schema upgrades |
| 5891 | |
| 5892 | Schemas are the friendly layer because changes are mostly **additive**, and records are read *through* the schema — an old row simply reads empty for a new field (proven by the `created_at` rollout). |
| 5893 | |
| 5894 | - **Field union on upgrade.** Merge shipped schema changes with locally added fields rather than replacing the field list. Conflict = same field name with a different type/enum — park it like a source conflict. A locally added `priority` field must survive an app-notes upgrade. |
| 5895 | - **Namespace local fields** (`x_` prefix by convention, enforced gently) so a package can never ship a field that collides with an operator's. |
| 5896 | - **Extension data (`extra`).** Give every collection an optional JSON overflow field: user-defined per-record data that no schema upgrade can ever collide with, because it isn't in the schema's flat namespace at all. Schemas may declare typed *views* into it (`{""name"": ""x_mood"", ""store"": ""extra""}`) so forms/lists render extension fields with zero migration. TSV stays one column; the column holds JSON. |
| 5897 | - **Migrations** (the currently-stubbed manifest section) are for the rare non-additive change: a shipped script that backfills/transforms records, run once, marker-tracked, restore-point-first. Until implemented, packages must keep schema changes additive. |
| 5898 | |
| 5899 | ## Rule 4: hooks — behavior customization without forking |
| 5900 | |
| 5901 | Let operators attach code to the system's existing event stream instead of editing package objects. The substrate exists: every record write already publishes `collection.record.created/updated/deleted` (it drives the websocket layer). |
| 5902 | |
| 5903 | - **Event handler objects.** A user object subscribes declaratively: `HANDLES = [""notes.record.created""]`; the server invokes it on the event (same sandboxed execution as any object). Custom side effects — notify, enrich, sync — with zero edits to the shipped app. Handlers are user objects, so upgrades never touch them. |
| 5904 | - **Page hooks.** Generated pages expose named slots (`before_list`, `row_actions`) that resolve to user objects if present. The generative renderer makes this cheap: one hook added to the shared list-page generator appears in every app at once. |
| 5905 | |
| 5906 | ## Rule 5: user prefs and feature flags |
| 5907 | |
| 5908 | A `user_prefs` collection (`user_id`, `key`, `value`, owner-scoped rows) — schema-driven like everything else — gives: |
| 5909 | |
| 5910 | - **Per-user settings** for any app (theme, defaults, layout) with zero new endpoints; the generators can read/write it directly. |
| 5911 | - **Feature flags.** Packages ship features dark: `features.json` → `{""kanban_view"": {""default"": ""off""}}`. Instance-level toggles live in a `feature_flags` collection; per-user overrides in `user_prefs`. Pages check flags via one shared helper. This decouples *shipping* code from *enabling* it — upgrades can carry big features safely off, canary them per user, and roll back by flipping a flag rather than reinstalling. It also shrinks customization pressure: many ""edits"" are really ""turn this on/off,"" which should be config, not forked source. |
| 5912 | |
| 5913 | ## What future coders (AI or human) must do |
| 5914 | |
| 5915 | When touching packages or upgrade code: |
| 5916 | |
| 5917 | 1. Never write an upgrade path that overwrites a file without checking its baseline hash first. If the baseline system doesn't exist yet, build it before the feature that needs it. |
| 5918 | 2. Never block an install on something preservable — preserve and skip (like seed), or park and surface (like conflicts). Blocking teaches operators to bypass the installer, which is how systems rot. |
| 5919 | 3. Keep schema changes additive; put anything non-additive in a migration. |
| 5920 | 4. New customization mechanisms must be overlay-shaped: separate file, separate collection, separate namespace — never ""edit the shipped thing."" |
| 5921 | 5. Data lives outside the package, keyed by collection, always. |
| 5922 | |
| 5923 | ## Build order |
| 5924 | |
| 5925 | 1. **Provenance stamps** (small; unblocks everything) — record baselines at install, expose `customized: true/false` per object in package status. |
| 5926 | 2. **Reconcile-on-upgrade** — the table above, plus pending-reconcile records and a CLI list; the admin UI renders the reconcile inbox. |
| 5927 | 3. **Override objects** — resolver-level shadowing; the conflict-free path. |
| 5928 | 4. **Schema field-union + `extra` extension data.** |
| 5929 | 5. **Event handler objects** (`HANDLES`), then page hooks. |
| 5930 | 6. **`user_prefs` + `feature_flags` collections** and the shared flag helper. |
| 5931 | |
| 5932 | Each phase is independently shippable and independently valuable; together they make ""upgrade the app, keep your data and your customizations"" a platform guarantee — the property that separated the systems that lasted (Salesforce, Odoo, Debian, WordPress) from the frameworks that made every operator their own integrator. |
| 5933 | " |
| 5934 | http-api-contract HTTP API Contract API Reference 800 Reference for the HTTP shape the DBBASIC Object Server preserves for clients: auth, correlation IDs, permissions, identity, collections, objects, packages, schemas, and the compatibility guarantees that hold across releases. "This is the HTTP shape the DBBASIC Object Server preserves as its internals evolve. |
| 5935 | |
| 5936 | The goal is compatibility. Existing clients, MCP tools, Scroll, scripts, and |
| 5937 | automation should not need a new API shape just because the internals become |
| 5938 | cleaner. |
| 5939 | |
| 5940 | Examples throughout this guide use a local placeholder host: |
| 5941 | |
| 5942 | ```text |
| 5943 | http://127.0.0.1:8001 |
| 5944 | ``` |
| 5945 | |
| 5946 | Replace it with your own deployment's address — the shape of the request and |
| 5947 | response is what matters, not the host. |
| 5948 | |
| 5949 | ## Client Defaults |
| 5950 | |
| 5951 | Existing clients expect JSON by default and commonly send: |
| 5952 | |
| 5953 | ```http |
| 5954 | Accept: application/json |
| 5955 | Authorization: Token <token> |
| 5956 | ``` |
| 5957 | |
| 5958 | Authentication is required for source, state, logs, versions, object creation, |
| 5959 | source updates, rollback, and destructive deletes. Basic object execution may be |
| 5960 | public or authenticated depending on server policy and the object being called. |
| 5961 | |
| 5962 | The current public ASGI slice enforces that sensitive read surface with the |
| 5963 | temporary admin token from `DBBASIC_ADMIN_TOKEN` by default. A deployment can |
| 5964 | opt in to `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true` so active DBBASIC sessions |
| 5965 | with an admin role can pass the same gate. Source update and rollback also |
| 5966 | require `DBBASIC_ENABLE_SOURCE_WRITES=true`. The real role, object, and row |
| 5967 | permission system still needs to replace this temporary admin boundary before |
| 5968 | general use. |
| 5969 | |
| 5970 | ## Correlation IDs |
| 5971 | |
| 5972 | Every HTTP response includes: |
| 5973 | |
| 5974 | ```http |
| 5975 | X-DBBASIC-Correlation-ID: <uuid-v4> |
| 5976 | ``` |
| 5977 | |
| 5978 | Clients may send their own UUIDv4 correlation ID: |
| 5979 | |
| 5980 | ```http |
| 5981 | X-DBBASIC-Correlation-ID: 123e4567-e89b-42d3-a456-426614174000 |
| 5982 | ``` |
| 5983 | |
| 5984 | Missing, empty, or invalid values are replaced with a server-generated UUIDv4. |
| 5985 | The response header always contains the accepted value. Source update responses, |
| 5986 | rollback responses, and execution error bodies also include `correlation_id`. |
| 5987 | |
| 5988 | The same ID is written into source version metadata, source-change entries, |
| 5989 | object-owned runtime logs, execution error logs, and permission audit entries |
| 5990 | when those records are created inside the request. Scroll and AI tools can use |
| 5991 | that ID to connect one user action to the source version, logs, permission |
| 5992 | decision, and runtime error it produced. |
| 5993 | |
| 5994 | ## Resource IDs |
| 5995 | |
| 5996 | New DBBASIC-facing URL and API resources should use UUIDv4 IDs by default: |
| 5997 | |
| 5998 | - record IDs |
| 5999 | - account and user IDs |
| 6000 | - session/action/change IDs |
| 6001 | - package install and restore IDs |
| 6002 | - external links and imported resource handles |
| 6003 | |
| 6004 | Imported systems may keep legacy compatibility IDs. For example, a table with |
| 6005 | integer primary keys can be imported with its old ID preserved in a |
| 6006 | compatibility column, while the DBBASIC route-facing `id` should be UUIDv4 for |
| 6007 | new records. This keeps URLs non-enumerable, makes exports easier to merge, and |
| 6008 | avoids coupling object packages to one database sequence. |
| 6009 | |
| 6010 | The current identity registry generates UUIDv4 account IDs, user IDs, and |
| 6011 | session IDs when the server creates them. It still accepts existing string |
| 6012 | account/user IDs and legacy `sess_...` session rows so old clients, Scroll |
| 6013 | prototypes, and migration tools do not break while the public runtime is being |
| 6014 | extracted. |
| 6015 | |
| 6016 | ## Health |
| 6017 | |
| 6018 | Plain health is a public liveness check: |
| 6019 | |
| 6020 | ```http |
| 6021 | GET /health |
| 6022 | ``` |
| 6023 | |
| 6024 | Response: |
| 6025 | |
| 6026 | ```json |
| 6027 | { |
| 6028 | ""status"": ""ok"" |
| 6029 | } |
| 6030 | ``` |
| 6031 | |
| 6032 | Detailed health is for operators, Scroll, and future cluster routing: |
| 6033 | |
| 6034 | ```http |
| 6035 | GET /health?capacity=true |
| 6036 | Authorization: Token <token> |
| 6037 | ``` |
| 6038 | |
| 6039 | Response: |
| 6040 | |
| 6041 | ```json |
| 6042 | { |
| 6043 | ""status"": ""ok"", |
| 6044 | ""version"": ""0.0.1"", |
| 6045 | ""station_id"": ""standalone"", |
| 6046 | ""uptime"": ""2h 10m"", |
| 6047 | ""requests"": 120, |
| 6048 | ""errors"": 0, |
| 6049 | ""rps"": 1.4, |
| 6050 | ""objects"": { |
| 6051 | ""count"": 4 |
| 6052 | }, |
| 6053 | ""capacity"": { |
| 6054 | ""requests"": { |
| 6055 | ""in_flight"": 1, |
| 6056 | ""max"": 64, |
| 6057 | ""available"": 63, |
| 6058 | ""limited"": true |
| 6059 | }, |
| 6060 | ""object_executions"": { |
| 6061 | ""in_flight"": 0, |
| 6062 | ""max"": 8, |
| 6063 | ""available"": 8, |
| 6064 | ""limited"": true |
| 6065 | } |
| 6066 | }, |
| 6067 | ""checks"": { |
| 6068 | ""storage"": { |
| 6069 | ""status"": ""ok"" |
| 6070 | } |
| 6071 | } |
| 6072 | } |
| 6073 | ``` |
| 6074 | |
| 6075 | `GET /health?metrics=true` returns the same detailed health payload plus a |
| 6076 | `metrics` object with request counts, status counts, response timing, top paths, |
| 6077 | and recent HTTP errors. This preserves the older dashboard/Scroll direction |
| 6078 | without exposing those details through the public liveness route. |
| 6079 | |
| 6080 | ## Admin Status |
| 6081 | |
| 6082 | Admin status is the compact operator snapshot Scroll and staging dashboards can |
| 6083 | use instead of stitching together many routes on first load: |
| 6084 | |
| 6085 | ```http |
| 6086 | GET /admin/status |
| 6087 | Authorization: Token <token> |
| 6088 | ``` |
| 6089 | |
| 6090 | Response: |
| 6091 | |
| 6092 | ```json |
| 6093 | { |
| 6094 | ""status"": ""ok"", |
| 6095 | ""timestamp"": ""2026-01-01T00:00:00+00:00"", |
| 6096 | ""version"": ""0.0.1"", |
| 6097 | ""station_id"": ""standalone"", |
| 6098 | ""health"": { |
| 6099 | ""status"": ""ok"", |
| 6100 | ""metrics"": {} |
| 6101 | }, |
| 6102 | ""inventory"": { |
| 6103 | ""objects"": 4, |
| 6104 | ""collections"": 2, |
| 6105 | ""schemas"": 2, |
| 6106 | ""packages"": 3 |
| 6107 | }, |
| 6108 | ""capabilities"": { |
| 6109 | ""source_writes"": { |
| 6110 | ""enabled"": false, |
| 6111 | ""env"": ""DBBASIC_ENABLE_SOURCE_WRITES"" |
| 6112 | }, |
| 6113 | ""file_writes"": { |
| 6114 | ""enabled"": false, |
| 6115 | ""env"": ""DBBASIC_ENABLE_FILE_WRITES"", |
| 6116 | ""max_bytes"": 1048576, |
| 6117 | ""max_bytes_env"": ""DBBASIC_MAX_OBJECT_FILE_BYTES"" |
| 6118 | }, |
| 6119 | ""package_installs"": { |
| 6120 | ""enabled"": false, |
| 6121 | ""env"": ""DBBASIC_ENABLE_PACKAGE_INSTALLS"" |
| 6122 | }, |
| 6123 | ""permission_enforcement"": { |
| 6124 | ""enabled"": false, |
| 6125 | ""requested"": false, |
| 6126 | ""blocked"": true, |
| 6127 | ""env"": ""DBBASIC_ENABLE_PERMISSION_ENFORCEMENT"" |
| 6128 | }, |
| 6129 | ""identity"": { |
| 6130 | ""trusted_headers_enabled"": false, |
| 6131 | ""require_known_identity_users"": true, |
| 6132 | ""session_login_enabled"": false, |
| 6133 | ""session_login_token_configured"": false, |
| 6134 | ""session_admin_gates_enabled"": false, |
| 6135 | ""session_admin_gates_env"": ""DBBASIC_ENABLE_SESSION_ADMIN_GATES"" |
| 6136 | } |
| 6137 | }, |
| 6138 | ""packages"": [ |
| 6139 | { |
| 6140 | ""id"": ""system-dashboard"", |
| 6141 | ""name"": ""System Dashboard"", |
| 6142 | ""version"": ""0.1.0"", |
| 6143 | ""status"": ""installed"", |
| 6144 | ""install"": { |
| 6145 | ""installed_count"": 1, |
| 6146 | ""installable_count"": 1, |
| 6147 | ""safe_to_install"": true, |
| 6148 | ""install_enabled"": false, |
| 6149 | ""warnings"": [] |
| 6150 | }, |
| 6151 | ""changes"": { |
| 6152 | ""total"": 1, |
| 6153 | ""latest"": { |
| 6154 | ""action"": ""installed"" |
| 6155 | } |
| 6156 | } |
| 6157 | } |
| 6158 | ], |
| 6159 | ""permissions"": { |
| 6160 | ""enforcement_enabled"": false, |
| 6161 | ""audit_enabled"": false, |
| 6162 | ""readiness"": { |
| 6163 | ""ready"": false |
| 6164 | }, |
| 6165 | ""warnings"": [] |
| 6166 | } |
| 6167 | } |
| 6168 | ``` |
| 6169 | |
| 6170 | If the server is degraded, the route may return `503` with the same response |
| 6171 | shape and `""status"": ""degraded""`. This route is admin-token gated because it |
| 6172 | contains configuration, package, and capacity details. |
| 6173 | |
| 6174 | ## Daemon Status |
| 6175 | |
| 6176 | Daemon status is the read-only operator snapshot for the scheduler, queue, |
| 6177 | event delivery, and cleanup primitives: |
| 6178 | |
| 6179 | ```http |
| 6180 | GET /daemon/status |
| 6181 | Authorization: Token <token> |
| 6182 | ``` |
| 6183 | |
| 6184 | Response: |
| 6185 | |
| 6186 | ```json |
| 6187 | { |
| 6188 | ""status"": ""ok"", |
| 6189 | ""timestamp"": ""2026-01-01T00:00:00+00:00"", |
| 6190 | ""daemon"": { |
| 6191 | ""mode"": ""polling"", |
| 6192 | ""croniter_available"": true, |
| 6193 | ""object_roots"": { |
| 6194 | ""count"": 1 |
| 6195 | }, |
| 6196 | ""triggers"": { |
| 6197 | ""scheduler"": { |
| 6198 | ""object_id"": ""scheduler"", |
| 6199 | ""source_present"": true |
| 6200 | }, |
| 6201 | ""queue"": { |
| 6202 | ""object_id"": ""queue"", |
| 6203 | ""source_present"": true |
| 6204 | }, |
| 6205 | ""events"": { |
| 6206 | ""object_id"": ""events"", |
| 6207 | ""source_present"": true |
| 6208 | } |
| 6209 | } |
| 6210 | }, |
| 6211 | ""scheduler"": { |
| 6212 | ""object_id"": ""scheduler"", |
| 6213 | ""source_present"": true, |
| 6214 | ""tasks"": { |
| 6215 | ""total"": 2, |
| 6216 | ""active"": 1, |
| 6217 | ""due"": 1, |
| 6218 | ""future"": 0, |
| 6219 | ""invalid"": 0, |
| 6220 | ""next_run"": 1767225600, |
| 6221 | ""next_run_iso"": ""2026-01-01T00:00:00+00:00"" |
| 6222 | } |
| 6223 | }, |
| 6224 | ""queue"": { |
| 6225 | ""object_id"": ""queue"", |
| 6226 | ""source_present"": true, |
| 6227 | ""messages"": { |
| 6228 | ""total"": 3, |
| 6229 | ""pending_visible"": 1, |
| 6230 | ""pending_delayed"": 1, |
| 6231 | ""expired_pending"": 0, |
| 6232 | ""invalid"": 0 |
| 6233 | } |
| 6234 | }, |
| 6235 | ""events"": { |
| 6236 | ""object_id"": ""events"", |
| 6237 | ""source_present"": true, |
| 6238 | ""events"": { |
| 6239 | ""total"": 10, |
| 6240 | ""latest"": { |
| 6241 | ""id"": ""evt_123"", |
| 6242 | ""event_type"": ""collection.record.created"", |
| 6243 | ""timestamp"": 1767225600 |
| 6244 | } |
| 6245 | }, |
| 6246 | ""subscriptions"": { |
| 6247 | ""total"": 2, |
| 6248 | ""by_delivery_status"": { |
| 6249 | ""ok"": 1, |
| 6250 | ""failed"": 1 |
| 6251 | }, |
| 6252 | ""pending_deliveries"": 1 |
| 6253 | } |
| 6254 | }, |
| 6255 | ""cleanup"": { |
| 6256 | ""event_retention"": { |
| 6257 | ""keep_count"": 1000, |
| 6258 | ""keep_seconds"": 604800 |
| 6259 | }, |
| 6260 | ""rate_limit_files"": 0 |
| 6261 | } |
| 6262 | } |
| 6263 | ``` |
| 6264 | |
| 6265 | This endpoint does not execute daemon work or expose event payloads. It reads |
| 6266 | the same TSV state that the daemon uses so Scroll can show due scheduler tasks, |
| 6267 | visible/delayed queue messages, failed subscription delivery, event retention, |
| 6268 | and cleanup pressure without requiring a separate queue dashboard service. |
| 6269 | |
| 6270 | ## Daemon Scheduler And Queue Controls |
| 6271 | |
| 6272 | Scheduler and queue controls are admin-token gated. They operate on the same |
| 6273 | `data/state/scheduler/state.tsv` and `data/state/queue/state.tsv` files that |
| 6274 | `object_daemon.py` already polls, so the HTTP API can manage work without |
| 6275 | introducing Celery, Redis, Flower, or a second queue service. |
| 6276 | |
| 6277 | These routes redact task/message payloads by default. Pass |
| 6278 | `include_payload=true` only on trusted operator screens. |
| 6279 | |
| 6280 | ```http |
| 6281 | GET /daemon/scheduler/tasks?status=active&limit=100&offset=0 |
| 6282 | POST /daemon/scheduler/tasks |
| 6283 | PATCH /daemon/scheduler/tasks/{task_id} |
| 6284 | DELETE /daemon/scheduler/tasks/{task_id} |
| 6285 | |
| 6286 | GET /daemon/queue/messages?status=pending&queue_name=default&limit=100&offset=0 |
| 6287 | POST /daemon/queue/messages |
| 6288 | PATCH /daemon/queue/messages/{message_id} |
| 6289 | DELETE /daemon/queue/messages/{message_id} |
| 6290 | Authorization: Token <token> |
| 6291 | ``` |
| 6292 | |
| 6293 | Create one scheduled task: |
| 6294 | |
| 6295 | ```json |
| 6296 | { |
| 6297 | ""object_id"": ""system_dashboard"", |
| 6298 | ""method"": ""POST"", |
| 6299 | ""type"": ""onetime"", |
| 6300 | ""schedule"": ""2026-07-01T12:00:00Z"", |
| 6301 | ""payload"": { |
| 6302 | ""refresh"": true |
| 6303 | } |
| 6304 | } |
| 6305 | ``` |
| 6306 | |
| 6307 | Response: |
| 6308 | |
| 6309 | ```json |
| 6310 | { |
| 6311 | ""status"": ""ok"", |
| 6312 | ""task"": { |
| 6313 | ""id"": ""018ff3f4-5f80-4df0-8e25-bcf6ad6fda01"", |
| 6314 | ""object_id"": ""system_dashboard"", |
| 6315 | ""method"": ""POST"", |
| 6316 | ""type"": ""onetime"", |
| 6317 | ""schedule"": ""2026-07-01T12:00:00Z"", |
| 6318 | ""status"": ""active"", |
| 6319 | ""next_run"": 1782907200, |
| 6320 | ""payload_present"": true |
| 6321 | } |
| 6322 | } |
| 6323 | ``` |
| 6324 | |
| 6325 | Patch examples: |
| 6326 | |
| 6327 | ```json |
| 6328 | {""status"": ""paused""} |
| 6329 | ``` |
| 6330 | |
| 6331 | ```json |
| 6332 | {""next_run"": ""2026-07-01T13:00:00Z""} |
| 6333 | ``` |
| 6334 | |
| 6335 | Enqueue one message: |
| 6336 | |
| 6337 | ```json |
| 6338 | { |
| 6339 | ""object_id"": ""system_dashboard"", |
| 6340 | ""method"": ""POST"", |
| 6341 | ""queue_name"": ""default"", |
| 6342 | ""priority_level"": 5, |
| 6343 | ""payload"": { |
| 6344 | ""refresh"": true |
| 6345 | } |
| 6346 | } |
| 6347 | ``` |
| 6348 | |
| 6349 | Response: |
| 6350 | |
| 6351 | ```json |
| 6352 | { |
| 6353 | ""status"": ""ok"", |
| 6354 | ""message"": { |
| 6355 | ""id"": ""2de66c27-149b-475b-a6f0-0c08a8d850f5"", |
| 6356 | ""queue_name"": ""default"", |
| 6357 | ""message"": { |
| 6358 | ""object_id"": ""system_dashboard"", |
| 6359 | ""method"": ""POST"", |
| 6360 | ""payload_present"": true |
| 6361 | }, |
| 6362 | ""priority_level"": 5, |
| 6363 | ""status"": ""pending"" |
| 6364 | } |
| 6365 | } |
| 6366 | ``` |
| 6367 | |
| 6368 | Queue patch actions: |
| 6369 | |
| 6370 | ```json |
| 6371 | {""action"": ""cancel""} |
| 6372 | ``` |
| 6373 | |
| 6374 | ```json |
| 6375 | {""action"": ""retry""} |
| 6376 | ``` |
| 6377 | |
| 6378 | These controls are for trusted operators. They do not change the rule that |
| 6379 | untrusted object execution still needs a stronger worker boundary, quotas, and |
| 6380 | permission enforcement before public signup is safe. |
| 6381 | |
| 6382 | ## Rate Limits |
| 6383 | |
| 6384 | When `DBBASIC_RATE_LIMIT_REQUESTS` is set above zero, the server rate-limits |
| 6385 | non-health traffic before reading request bodies or running object code. Plain |
| 6386 | `GET /health` is excluded so load balancers and uptime checks keep working under |
| 6387 | pressure. |
| 6388 | |
| 6389 | The current public server uses a valid admin token as the rate-limit identity |
| 6390 | for admin requests. Other requests use the client IP address. Proxy headers such |
| 6391 | as `X-Forwarded-For` are ignored unless `DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS` |
| 6392 | is enabled on a server that is only reachable through a trusted reverse proxy. |
| 6393 | |
| 6394 | Limit response: |
| 6395 | |
| 6396 | ```http |
| 6397 | 429 Too Many Requests |
| 6398 | Retry-After: 30 |
| 6399 | ``` |
| 6400 | |
| 6401 | ```json |
| 6402 | { |
| 6403 | ""status"": ""error"", |
| 6404 | ""error"": ""Rate limit exceeded"", |
| 6405 | ""retry_after"": 30, |
| 6406 | ""limit"": 1000, |
| 6407 | ""window_seconds"": 60 |
| 6408 | } |
| 6409 | ``` |
| 6410 | |
| 6411 | ## Identity |
| 6412 | |
| 6413 | Scroll and gateway code can inspect the subject the object server will use for |
| 6414 | permission checks: |
| 6415 | |
| 6416 | ```http |
| 6417 | GET /identity |
| 6418 | Authorization: Token <token> |
| 6419 | ``` |
| 6420 | |
| 6421 | Response: |
| 6422 | |
| 6423 | ```json |
| 6424 | { |
| 6425 | ""status"": ""ok"", |
| 6426 | ""subject"": { |
| 6427 | ""user_id"": ""7"", |
| 6428 | ""account_id"": ""acme"", |
| 6429 | ""roles"": [""sales""], |
| 6430 | ""subscriptions"": [""pro""], |
| 6431 | ""authenticated"": true |
| 6432 | }, |
| 6433 | ""auth"": { |
| 6434 | ""method"": ""trusted_headers"", |
| 6435 | ""trusted_headers_enabled"": true, |
| 6436 | ""trusted_headers_present"": true |
| 6437 | }, |
| 6438 | ""permissions"": { |
| 6439 | ""enforcement_enabled"": true, |
| 6440 | ""enforcement_requested"": true, |
| 6441 | ""enforcement_blocked"": false, |
| 6442 | ""audit_enabled"": true |
| 6443 | } |
| 6444 | } |
| 6445 | ``` |
| 6446 | |
| 6447 | Current auth methods are: |
| 6448 | |
| 6449 | - `anonymous` - no trusted identity was accepted. |
| 6450 | - `admin_token` - `Authorization: Token <DBBASIC_ADMIN_TOKEN>` or Bearer matched |
| 6451 | the server admin token. |
| 6452 | - `session_token` - `Authorization: Token <session-token>` or Bearer matched an |
| 6453 | active file-backed DBBASIC identity session. |
| 6454 | - `trusted_headers` - `DBBASIC_PERMISSION_TRUST_HEADERS=true` and at least one |
| 6455 | `X-DBBASIC-*` identity header was present. |
| 6456 | |
| 6457 | Trusted identity headers are ignored unless explicitly enabled. This keeps the |
| 6458 | current staging server safe while giving future login/session gateways and |
| 6459 | Scroll one stable endpoint for the active account, user, roles, and |
| 6460 | subscriptions. |
| 6461 | |
| 6462 | ### Identity Accounts And Users |
| 6463 | |
| 6464 | The server has a small file-backed identity registry so sessions can be minted |
| 6465 | from known users instead of every client inventing its own subject payload. These |
| 6466 | routes are admin-gated. |
| 6467 | |
| 6468 | Accounts are stored under `data/identity/accounts.tsv`: |
| 6469 | |
| 6470 | ```http |
| 6471 | GET /identity/accounts |
| 6472 | Authorization: Token <admin-token> |
| 6473 | ``` |
| 6474 | |
| 6475 | ```http |
| 6476 | POST /identity/accounts |
| 6477 | Authorization: Token <admin-token> |
| 6478 | Content-Type: application/json |
| 6479 | ``` |
| 6480 | |
| 6481 | ```json |
| 6482 | { |
| 6483 | ""account_id"": ""acme"", |
| 6484 | ""name"": ""Acme Corp"", |
| 6485 | ""subscriptions"": [""pro""] |
| 6486 | } |
| 6487 | ``` |
| 6488 | |
| 6489 | Users are stored under `data/identity/users.tsv`: |
| 6490 | |
| 6491 | ```http |
| 6492 | GET /identity/users |
| 6493 | Authorization: Token <admin-token> |
| 6494 | ``` |
| 6495 | |
| 6496 | ```http |
| 6497 | GET /identity/users?account_id=acme |
| 6498 | Authorization: Token <admin-token> |
| 6499 | ``` |
| 6500 | |
| 6501 | ```http |
| 6502 | POST /identity/users |
| 6503 | Authorization: Token <admin-token> |
| 6504 | Content-Type: application/json |
| 6505 | ``` |
| 6506 | |
| 6507 | ```json |
| 6508 | { |
| 6509 | ""user_id"": ""u_7"", |
| 6510 | ""account_id"": ""acme"", |
| 6511 | ""email"": ""alice@example.com"", |
| 6512 | ""display_name"": ""Alice"", |
| 6513 | ""roles"": [""sales""], |
| 6514 | ""subscriptions"": [""team""] |
| 6515 | } |
| 6516 | ``` |
| 6517 | |
| 6518 | Single-record reads are: |
| 6519 | |
| 6520 | ```http |
| 6521 | GET /identity/accounts/acme |
| 6522 | GET /identity/users/u_7 |
| 6523 | ``` |
| 6524 | |
| 6525 | `status` can be `active` or `disabled`. Disabled accounts and users cannot mint |
| 6526 | new sessions. |
| 6527 | |
| 6528 | ### User Passwords |
| 6529 | |
| 6530 | Users can have one password credential for browser/password login. Hashes are |
| 6531 | stored separately from user records under `data/identity/credentials.tsv` |
| 6532 | (scrypt, owner-only file permissions), so user payloads never include |
| 6533 | credential material. Both routes are admin-token gated, and the admin alias |
| 6534 | form works through the reverse proxy for Scroll: |
| 6535 | |
| 6536 | ```http |
| 6537 | POST /identity/users/{user_id}/password |
| 6538 | POST /admin/identity/users/{user_id}/password |
| 6539 | Authorization: Token <admin-token> |
| 6540 | Content-Type: application/json |
| 6541 | ``` |
| 6542 | |
| 6543 | ```json |
| 6544 | {""password"": ""at least 8 characters""} |
| 6545 | ``` |
| 6546 | |
| 6547 | Response returns only `user_id`, `operation` (`created` or `replaced`), and |
| 6548 | `updated_at` — never the password or hash. `DELETE` on the same paths removes |
| 6549 | the credential: |
| 6550 | |
| 6551 | ```http |
| 6552 | DELETE /identity/users/{user_id}/password |
| 6553 | DELETE /admin/identity/users/{user_id}/password |
| 6554 | Authorization: Token <admin-token> |
| 6555 | ``` |
| 6556 | |
| 6557 | Unknown users return `404`. Passwords must be 8 to 1024 characters. |
| 6558 | |
| 6559 | ### Identity Sessions |
| 6560 | |
| 6561 | The server can mint scoped subject tokens for Scroll, gateways, or controlled |
| 6562 | apps. Sessions are stored under `data/identity/sessions.tsv`. Only token hashes |
| 6563 | are written to disk; the raw token is returned once at creation time. |
| 6564 | |
| 6565 | Create a session: |
| 6566 | |
| 6567 | ```http |
| 6568 | POST /identity/sessions |
| 6569 | Authorization: Token <admin-token> |
| 6570 | Content-Type: application/json |
| 6571 | ``` |
| 6572 | |
| 6573 | ```json |
| 6574 | { |
| 6575 | ""user_id"": ""7"", |
| 6576 | ""account_id"": ""acme"", |
| 6577 | ""roles"": [""sales""], |
| 6578 | ""subscriptions"": [""pro""], |
| 6579 | ""label"": ""scroll desktop"", |
| 6580 | ""ttl_seconds"": 86400 |
| 6581 | } |
| 6582 | ``` |
| 6583 | |
| 6584 | Response: |
| 6585 | |
| 6586 | ```json |
| 6587 | { |
| 6588 | ""status"": ""ok"", |
| 6589 | ""session"": { |
| 6590 | ""session_id"": ""9d4fe8bd-6859-43ef-8f83-f7063c54f7bc"", |
| 6591 | ""user_id"": ""7"", |
| 6592 | ""account_id"": ""acme"", |
| 6593 | ""roles"": [""sales""], |
| 6594 | ""subscriptions"": [""pro""], |
| 6595 | ""label"": ""scroll desktop"", |
| 6596 | ""created_at"": ""2026-06-30T18:00:00Z"", |
| 6597 | ""expires_at"": ""2026-07-01T18:00:00Z"", |
| 6598 | ""revoked_at"": null, |
| 6599 | ""active"": true |
| 6600 | }, |
| 6601 | ""token"": ""returned-once"" |
| 6602 | } |
| 6603 | ``` |
| 6604 | |
| 6605 | If `user_id` already exists in `data/identity/users.tsv`, omitted `account_id`, |
| 6606 | `roles`, and `subscriptions` are filled from the registered user and account. |
| 6607 | This lets Scroll or a login gateway mint a session without duplicating policy |
| 6608 | metadata in every request. If `DBBASIC_REQUIRE_KNOWN_IDENTITY_USERS=true`, the |
| 6609 | server rejects sessions for unknown users. |
| 6610 | |
| 6611 | For a non-admin login gateway or local trusted client, the current-session route |
| 6612 | can mint a session for an existing active user when |
| 6613 | `DBBASIC_ENABLE_SESSION_LOGIN=true`: |
| 6614 | |
| 6615 | ```http |
| 6616 | POST /identity/session |
| 6617 | Authorization: Token <session-login-token> |
| 6618 | Content-Type: application/json |
| 6619 | ``` |
| 6620 | |
| 6621 | ```json |
| 6622 | { |
| 6623 | ""user_id"": ""7"", |
| 6624 | ""label"": ""scroll desktop"", |
| 6625 | ""ttl_seconds"": 86400 |
| 6626 | } |
| 6627 | ``` |
| 6628 | |
| 6629 | This route always requires the user to exist in `data/identity/users.tsv`. |
| 6630 | It also requires `DBBASIC_SESSION_LOGIN_TOKEN`; this token is separate from the |
| 6631 | admin token so a login gateway can mint sessions without broad admin access. |
| 6632 | It accepts only `user_id`, `label`, and `ttl_seconds`; caller-supplied |
| 6633 | `account_id`, `roles`, or `subscriptions` are rejected. The session subject is |
| 6634 | loaded from the registered user and account so a login request cannot grant |
| 6635 | itself stronger permissions. This is a session-mint primitive, not password |
| 6636 | authentication. |
| 6637 | |
| 6638 | ### Password Login |
| 6639 | |
| 6640 | For direct password authentication, the same route accepts a credential login |
| 6641 | when `DBBASIC_ENABLE_PASSWORD_LOGIN=true`. No `Authorization` header is |
| 6642 | required; the password is the credential: |
| 6643 | |
| 6644 | ```http |
| 6645 | POST /identity/session |
| 6646 | Content-Type: application/json |
| 6647 | ``` |
| 6648 | |
| 6649 | ```json |
| 6650 | {""email"": ""alice@example.com"", ""password"": ""correct horse battery""} |
| 6651 | ``` |
| 6652 | |
| 6653 | The body must contain exactly one of `email` or `user_id`, plus `password`, and |
| 6654 | may include `label` and `ttl_seconds`. Role, account, and subscription |
| 6655 | overrides are rejected. The password is verified against the scrypt hash in |
| 6656 | `data/identity/credentials.tsv` set through the password routes above. Email |
| 6657 | matching is case-insensitive and must resolve to exactly one active user. |
| 6658 | |
| 6659 | Success returns `201` with the same session-plus-one-time-token shape as other |
| 6660 | session mints. Any failure — unknown user, wrong password, disabled user or |
| 6661 | account — returns a uniform `401 Invalid credentials` after a short fixed |
| 6662 | delay, so callers cannot probe which accounts exist. The global rate limiter |
| 6663 | also applies. Password login availability (enabled flag plus at least one |
| 6664 | active user) counts as a non-admin identity path for permission enforcement |
| 6665 | readiness. |
| 6666 | |
| 6667 | Repeated failures lock the identifier: after |
| 6668 | `DBBASIC_LOGIN_LOCKOUT_ATTEMPTS` failed attempts (default 5) within |
| 6669 | `DBBASIC_LOGIN_LOCKOUT_WINDOW_SECONDS` (default 900), further attempts — |
| 6670 | even with the correct password — return `429 Too many failed attempts` (the |
| 6671 | browser form redirects to `/login?error=locked`). A successful login resets |
| 6672 | the count; the lock clears on its own when the window passes. Locked attempts |
| 6673 | are recorded as `login_locked` ops events rather than new failures, so |
| 6674 | hammering a locked identifier does not extend the lock. Set attempts to `0` |
| 6675 | to disable. The counters are the `login_failed` rows in the ops event feed — |
| 6676 | `GET /admin/ops?kind=auth&identifier=...` shows exactly why an identifier is |
| 6677 | locked. |
| 6678 | |
| 6679 | List sessions: |
| 6680 | |
| 6681 | ```http |
| 6682 | GET /identity/sessions |
| 6683 | Authorization: Token <admin-token> |
| 6684 | ``` |
| 6685 | |
| 6686 | Revoke a session: |
| 6687 | |
| 6688 | ```http |
| 6689 | DELETE /identity/sessions/{session_id} |
| 6690 | Authorization: Token <admin-token> |
| 6691 | ``` |
| 6692 | |
| 6693 | Inspect the current session with a session token: |
| 6694 | |
| 6695 | ```http |
| 6696 | GET /identity/session |
| 6697 | Authorization: Token <session-token> |
| 6698 | ``` |
| 6699 | |
| 6700 | Revoke the current session with the same token: |
| 6701 | |
| 6702 | ```http |
| 6703 | DELETE /identity/session |
| 6704 | Authorization: Token <session-token> |
| 6705 | ``` |
| 6706 | |
| 6707 | The current-session route does not accept the admin token as a user session. |
| 6708 | It is for Scroll, app clients, and login gateways that need a stable non-admin |
| 6709 | session lifecycle. |
| 6710 | |
| 6711 | ### Session Cookies |
| 6712 | |
| 6713 | Browser clients can present the session token in a `dbbasic_session` cookie |
| 6714 | instead of an `Authorization` header. Cookie tokens resolve to the same |
| 6715 | session subject for permission checks, `/identity`, and the current-session |
| 6716 | routes. An `Authorization` header always takes precedence over the cookie, |
| 6717 | and the admin token is never accepted from a cookie. |
| 6718 | |
| 6719 | Cookie-authenticated non-GET requests are checked against the request origin: |
| 6720 | when an `Origin` (or `Referer`) header is present and its host does not match |
| 6721 | the request `Host`, the request is rejected with |
| 6722 | `403 Cross-origin cookie request rejected`. Together with `SameSite` cookie |
| 6723 | attributes set at login, this is the server's CSRF posture for cookie |
| 6724 | sessions. Header-authenticated requests are not origin-checked. |
| 6725 | |
| 6726 | ### Browser Login Pages |
| 6727 | |
| 6728 | When `DBBASIC_ENABLE_PASSWORD_LOGIN=true`, the server serves a built-in |
| 6729 | browser login flow: |
| 6730 | |
| 6731 | ```http |
| 6732 | GET /login |
| 6733 | GET /login?next=/dashboard |
| 6734 | POST /login (application/x-www-form-urlencoded: email, password, next) |
| 6735 | POST /logout |
| 6736 | ``` |
| 6737 | |
| 6738 | `GET /login` renders a minimal HTML sign-in form; an active session redirects |
| 6739 | away instead of re-rendering. `POST /login` verifies the password, mints a |
| 6740 | session, sets the `dbbasic_session` cookie (`HttpOnly`, `SameSite=Lax`, |
| 6741 | `Path=/`, `Secure` unless `DBBASIC_COOKIE_SECURE=false` for plain-HTTP local |
| 6742 | development), and redirects with `303` to the sanitized `next` path |
| 6743 | (same-origin absolute paths only; anything else falls back to `/`). Failures |
| 6744 | redirect back to `/login?error=1` after the same fixed delay as JSON password |
| 6745 | login, without revealing whether the email exists. JSON bodies are rejected; |
| 6746 | API clients should use `POST /identity/session`. `POST /logout` revokes the |
| 6747 | cookie session, clears the cookie, and redirects to `/login`. The login and |
| 6748 | logout routes never accept or mint admin tokens. |
| 6749 | |
| 6750 | Session tokens normally only supply the active subject used by permission |
| 6751 | checks. Admin-token gates remain admin-token-only by default. When a deployment |
| 6752 | sets `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`, active sessions whose subject |
| 6753 | has one of the policy admin roles can pass the same admin gate. The explicit |
| 6754 | source-write gate is still separate and must also be enabled for source update |
| 6755 | or rollback routes. |
| 6756 | |
| 6757 | ## Permissions Policy |
| 6758 | |
| 6759 | The public server now has a persisted permission policy shape. The endpoints are |
| 6760 | admin-gated while broader login and account management mature. |
| 6761 | |
| 6762 | ```http |
| 6763 | GET /permissions/policy |
| 6764 | Authorization: Token <token> |
| 6765 | ``` |
| 6766 | |
| 6767 | Response: |
| 6768 | |
| 6769 | ```json |
| 6770 | { |
| 6771 | ""status"": ""ok"", |
| 6772 | ""policy"": { |
| 6773 | ""access_mode"": ""role_based"", |
| 6774 | ""roles"": {}, |
| 6775 | ""user_roles"": {}, |
| 6776 | ""rules"": [], |
| 6777 | ""admin_roles"": [""admin"", ""superuser""] |
| 6778 | } |
| 6779 | } |
| 6780 | ``` |
| 6781 | |
| 6782 | Update: |
| 6783 | |
| 6784 | ```http |
| 6785 | PUT /permissions/policy |
| 6786 | Authorization: Token <token> |
| 6787 | Content-Type: application/json |
| 6788 | ``` |
| 6789 | |
| 6790 | ```json |
| 6791 | { |
| 6792 | ""policy"": { |
| 6793 | ""access_mode"": ""role_based"", |
| 6794 | ""roles"": {""sales"": {""label"": ""Sales""}}, |
| 6795 | ""user_roles"": {""7"": [""sales""]}, |
| 6796 | ""rules"": [ |
| 6797 | { |
| 6798 | ""effect"": ""allow"", |
| 6799 | ""principal"": ""role:sales"", |
| 6800 | ""actions"": [""read""], |
| 6801 | ""collection"": ""contacts"", |
| 6802 | ""row_filter"": {""owner_id"": ""$user_id""} |
| 6803 | } |
| 6804 | ], |
| 6805 | ""admin_roles"": [""admin""] |
| 6806 | } |
| 6807 | } |
| 6808 | ``` |
| 6809 | |
| 6810 | The policy is stored under `data/permissions/policy.json`. Missing policy files |
| 6811 | load as a conservative `role_based` policy with no grants. |
| 6812 | |
| 6813 | ## Permissions Status |
| 6814 | |
| 6815 | Scroll and operators can inspect whether the active identity and permission |
| 6816 | configuration is ready for enforcement: |
| 6817 | |
| 6818 | ```http |
| 6819 | GET /permissions/status |
| 6820 | Authorization: Token <admin-token> |
| 6821 | ``` |
| 6822 | |
| 6823 | Response: |
| 6824 | |
| 6825 | ```json |
| 6826 | { |
| 6827 | ""status"": ""ok"", |
| 6828 | ""permissions"": { |
| 6829 | ""enforcement_enabled"": false, |
| 6830 | ""enforcement_requested"": false, |
| 6831 | ""enforcement_blocked"": false, |
| 6832 | ""allow_unready_enforcement"": false, |
| 6833 | ""audit_enabled"": true, |
| 6834 | ""trusted_headers_enabled"": false, |
| 6835 | ""require_known_identity_users"": true, |
| 6836 | ""admin_token_configured"": true, |
| 6837 | ""session_login_enabled"": true, |
| 6838 | ""session_login_token_configured"": true |
| 6839 | }, |
| 6840 | ""identity"": { |
| 6841 | ""accounts"": {""count"": 1, ""active"": 1, ""disabled"": 0}, |
| 6842 | ""users"": {""count"": 3, ""active"": 3, ""disabled"": 0}, |
| 6843 | ""sessions"": {""count"": 2, ""active"": 1, ""revoked"": 1} |
| 6844 | }, |
| 6845 | ""policy"": { |
| 6846 | ""valid"": true, |
| 6847 | ""policy_file_exists"": true, |
| 6848 | ""access_mode"": ""role_based"", |
| 6849 | ""rules_count"": 4, |
| 6850 | ""allow_rules"": 3, |
| 6851 | ""deny_rules"": 1 |
| 6852 | }, |
| 6853 | ""readiness"": { |
| 6854 | ""can_enable_enforcement"": true, |
| 6855 | ""blockers"": [] |
| 6856 | }, |
| 6857 | ""warnings"": [] |
| 6858 | } |
| 6859 | ``` |
| 6860 | |
| 6861 | This endpoint is intentionally read-only. `DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true` |
| 6862 | requests enforcement, but the server only makes `enforcement_enabled` effective |
| 6863 | when `readiness.can_enable_enforcement` is true. If readiness is blocked, |
| 6864 | `enforcement_requested` is true, `enforcement_enabled` is false, and |
| 6865 | `enforcement_blocked` is true. The explicit recovery/test override is |
| 6866 | `DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true`. |
| 6867 | |
| 6868 | Readiness currently requires an admin recovery token, a valid policy, an |
| 6869 | available non-admin identity path for identity-gated modes, and at least one |
| 6870 | allow grant for `role_based` policy. The accepted identity paths are trusted |
| 6871 | proxy headers, guarded session login for existing users, or an already-active |
| 6872 | DBBASIC session. |
| 6873 | |
| 6874 | ## Permissions Check |
| 6875 | |
| 6876 | Scroll and admin tools can ask the server to evaluate one permission decision. |
| 6877 | This is useful for previewing draft rules and for future ""test as role"" screens. |
| 6878 | |
| 6879 | ```http |
| 6880 | POST /permissions/check |
| 6881 | Authorization: Token <token> |
| 6882 | Content-Type: application/json |
| 6883 | ``` |
| 6884 | |
| 6885 | ```json |
| 6886 | { |
| 6887 | ""subject"": { |
| 6888 | ""user_id"": ""7"", |
| 6889 | ""account_id"": ""customer-acme"", |
| 6890 | ""roles"": [""sales""], |
| 6891 | ""subscriptions"": [""pro""] |
| 6892 | }, |
| 6893 | ""action"": ""read"", |
| 6894 | ""collection"": ""contacts"", |
| 6895 | ""object_id"": null, |
| 6896 | ""record"": null |
| 6897 | } |
| 6898 | ``` |
| 6899 | |
| 6900 | If the request includes a `policy` object, the server evaluates against that |
| 6901 | inline policy without saving it. Otherwise it evaluates against the persisted |
| 6902 | policy. The optional `now` field accepts an ISO timestamp for previewing |
| 6903 | temporary access windows. |
| 6904 | |
| 6905 | Response: |
| 6906 | |
| 6907 | ```json |
| 6908 | { |
| 6909 | ""status"": ""ok"", |
| 6910 | ""decision"": { |
| 6911 | ""allowed"": true, |
| 6912 | ""reason"": ""sales reps only see own contacts"", |
| 6913 | ""code"": ""allowed"", |
| 6914 | ""http_status"": 200, |
| 6915 | ""row_filter"": {""owner_id"": ""$user_id""}, |
| 6916 | ""fields"": null, |
| 6917 | ""denied_fields"": [] |
| 6918 | } |
| 6919 | } |
| 6920 | ``` |
| 6921 | |
| 6922 | When a paid entitlement is missing, the decision can include: |
| 6923 | |
| 6924 | ```json |
| 6925 | { |
| 6926 | ""allowed"": false, |
| 6927 | ""reason"": ""subscription required"", |
| 6928 | ""code"": ""payment_required"", |
| 6929 | ""http_status"": 402, |
| 6930 | ""row_filter"": {}, |
| 6931 | ""fields"": null, |
| 6932 | ""denied_fields"": [] |
| 6933 | } |
| 6934 | ``` |
| 6935 | |
| 6936 | ## Permission Route Enforcement |
| 6937 | |
| 6938 | Permission policy checks can be applied to object and collection-record routes |
| 6939 | when explicitly requested: |
| 6940 | |
| 6941 | ```text |
| 6942 | DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true |
| 6943 | ``` |
| 6944 | |
| 6945 | The request becomes effective only when `/permissions/status` reports no |
| 6946 | readiness blockers, unless `DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true` |
| 6947 | is also set. When enforcement is effective, denied route checks return the |
| 6948 | `http_status`, `reason`, and `code` from the permission decision before object |
| 6949 | source, state, logs, versions, execution, or collection-record data work runs. |
| 6950 | |
| 6951 | Audit-only mode records the same decisions without changing responses: |
| 6952 | |
| 6953 | ```text |
| 6954 | DBBASIC_ENABLE_PERMISSION_AUDIT=true |
| 6955 | ``` |
| 6956 | |
| 6957 | Audit entries are written as JSON lines under: |
| 6958 | |
| 6959 | ```text |
| 6960 | data/permissions/audit.jsonl |
| 6961 | ``` |
| 6962 | |
| 6963 | Trusted identity headers are off by default and should only be enabled behind a |
| 6964 | trusted auth gateway or reverse proxy: |
| 6965 | |
| 6966 | ```text |
| 6967 | DBBASIC_PERMISSION_TRUST_HEADERS=true |
| 6968 | ``` |
| 6969 | |
| 6970 | Supported headers are `X-DBBASIC-User-Id`, `X-DBBASIC-Account-Id`, |
| 6971 | `X-DBBASIC-Roles`, and `X-DBBASIC-Subscriptions`. |
| 6972 | |
| 6973 | Collection record read routes use the `read` action. Collection record write |
| 6974 | routes use `create`, `update`, and `delete`. In enforcement mode the server |
| 6975 | applies row filters before pagination, evaluates detail and write requests |
| 6976 | against the selected record, blocks owner-changing updates that would escape the |
| 6977 | allowed row filter, and redacts `fields` / `denied_fields` from returned |
| 6978 | records. Audit-only mode can log write decisions, but it does not grant |
| 6979 | mutation access without the admin token. |
| 6980 | |
| 6981 | Operators and Scroll can read recent audit entries through an admin-gated |
| 6982 | endpoint: |
| 6983 | |
| 6984 | ```http |
| 6985 | GET /permissions/audit?limit=100 |
| 6986 | Authorization: Token <token> |
| 6987 | ``` |
| 6988 | |
| 6989 | Optional filters: |
| 6990 | |
| 6991 | - `action` |
| 6992 | - `object_id` |
| 6993 | - `collection` |
| 6994 | - `allowed=true|false` |
| 6995 | - `enforced=true|false` |
| 6996 | |
| 6997 | Response: |
| 6998 | |
| 6999 | ```json |
| 7000 | { |
| 7001 | ""status"": ""ok"", |
| 7002 | ""entries"": [ |
| 7003 | { |
| 7004 | ""timestamp"": ""2026-06-29T00:00:00Z"", |
| 7005 | ""method"": ""GET"", |
| 7006 | ""object_id"": ""site_home"", |
| 7007 | ""collection"": ""site"", |
| 7008 | ""action"": ""execute"", |
| 7009 | ""enforced"": false, |
| 7010 | ""decision"": {""allowed"": false, ""code"": ""forbidden""} |
| 7011 | } |
| 7012 | ], |
| 7013 | ""count"": 1 |
| 7014 | } |
| 7015 | ``` |
| 7016 | |
| 7017 | ## Optimistic Concurrency (`_rev` / `If-Match`) |
| 7018 | |
| 7019 | Record updates are last-write-wins by default. To make a read-then-write |
| 7020 | safe against a concurrent writer, send back the `_rev` you read. |
| 7021 | |
| 7022 | **Reading a rev.** A single-record read returns it as a **sibling** of |
| 7023 | `record`, never a field inside it: |
| 7024 | |
| 7025 | ```json |
| 7026 | {""status"": ""ok"", ""collection"": ""tasks"", ""record"": {""id"": ""t1"", ""status"": ""open""}, |
| 7027 | ""_rev"": ""a1b2c3…""} |
| 7028 | ``` |
| 7029 | |
| 7030 | A list read returns a parallel **`revs` map**, keyed by record id, covering |
| 7031 | exactly the rows in the returned window: |
| 7032 | |
| 7033 | ```json |
| 7034 | {""status"": ""ok"", ""collection"": ""tasks"", |
| 7035 | ""records"": [{""id"": ""t1"", ""status"": ""open""}, {""id"": ""t2"", ""status"": ""open""}], |
| 7036 | ""count"": 2, ""total"": 57, ""limit"": 100, ""offset"": 0, |
| 7037 | ""revs"": {""t1"": ""a1b2c3…"", ""t2"": ""d4e5f6…""}} |
| 7038 | ``` |
| 7039 | |
| 7040 | It is a map rather than a `_rev` on each row so the record shape stays a |
| 7041 | contract: a rev must never collide with a schema field, and must never be |
| 7042 | echoed back on write. |
| 7043 | |
| 7044 | **Using one.** Send `If-Match: <rev>` (or an `expected_rev` body field) on |
| 7045 | `PUT`. The rev is compared **inside the write lock**, before validation: |
| 7046 | |
| 7047 | - match → the write proceeds |
| 7048 | - mismatch → **409**, and nothing is written |
| 7049 | |
| 7050 | ``` |
| 7051 | PUT /collections/tasks/records/t1 |
| 7052 | If-Match: a1b2c3… |
| 7053 | |
| 7054 | {""status"": ""assigned"", ""assigned_to"": ""agent-A""} |
| 7055 | ``` |
| 7056 | |
| 7057 | **Why it exists — the claim.** Two agents both read a task as `open` and |
| 7058 | both write `assigned`. Without a precondition both succeed and whoever |
| 7059 | lands second silently overwrites the first; both believe they hold it and |
| 7060 | neither can tell. With `If-Match`, the second finds the row changed, gets a |
| 7061 | 409, re-reads, and moves on. The rev captures ""the record I read is still |
| 7062 | the record on disk"", which necessarily includes every field — so no |
| 7063 | per-field guard is needed. |
| 7064 | |
| 7065 | Because the `revs` map ships with the list, an agent can find work and |
| 7066 | claim it without re-reading each candidate first. |
| 7067 | |
| 7068 | **Notes.** |
| 7069 | |
| 7070 | - Omitting `If-Match` is still valid and still last-write-wins. The |
| 7071 | precondition is opt-in per write, never mandatory. |
| 7072 | - The rev is computed over the **full stored record**, before any |
| 7073 | field-level permission trimming. A caller who cannot see a field still |
| 7074 | gets a rev that depends on it — which only ever fails closed, and |
| 7075 | discloses nothing about the value. |
| 7076 | - `DBBASIC_ENABLE_CONCURRENCY=false` turns the feature off: preconditions |
| 7077 | are ignored (never a 409) and `revs` is omitted from list payloads. |
| 7078 | |
| 7079 | ## Object List |
| 7080 | |
| 7081 | ```http |
| 7082 | GET /objects?format=json |
| 7083 | Authorization: Token <token> |
| 7084 | ``` |
| 7085 | |
| 7086 | Response: |
| 7087 | |
| 7088 | ```json |
| 7089 | { |
| 7090 | ""status"": ""ok"", |
| 7091 | ""objects"": [ |
| 7092 | { |
| 7093 | ""object_id"": ""basics_counter"", |
| 7094 | ""path"": ""basics/counter.py"", |
| 7095 | ""owner"": ""system"" |
| 7096 | } |
| 7097 | ], |
| 7098 | ""count"": 1 |
| 7099 | } |
| 7100 | ``` |
| 7101 | |
| 7102 | Clients should accept either a top-level list or an object with an `objects` |
| 7103 | field. New server code should prefer the object form above. |
| 7104 | |
| 7105 | ## Admin Object Inspection |
| 7106 | |
| 7107 | `/objects` remains the runtime surface: object execution and public object |
| 7108 | routes live there. Scroll and operator dashboards should use the admin |
| 7109 | inspection surface so staging can expose read-only object metadata without |
| 7110 | opening broad object execution routes at the reverse proxy. |
| 7111 | |
| 7112 | ```http |
| 7113 | GET /admin/objects |
| 7114 | POST /admin/objects |
| 7115 | Authorization: Token <token> |
| 7116 | ``` |
| 7117 | |
| 7118 | `GET` returns the same response shape as `GET /objects?format=json`. |
| 7119 | `POST` is the admin/operator alias for `POST /objects`; it creates a new |
| 7120 | object without requiring the reverse proxy to expose the broader runtime |
| 7121 | `/objects` collection route. Creation still requires |
| 7122 | `DBBASIC_ENABLE_SOURCE_WRITES=true` and the admin gate. |
| 7123 | |
| 7124 | ```http |
| 7125 | GET /admin/objects/{object_id} |
| 7126 | GET /admin/objects/{object_id}?metadata=true |
| 7127 | GET /admin/objects/{object_id}?source=true&format=json |
| 7128 | PUT /admin/objects/{object_id}?source=true |
| 7129 | POST /admin/objects/{object_id}/execute |
| 7130 | GET /admin/objects/{object_id}?state=true |
| 7131 | GET /admin/objects/{object_id}?logs=true&format=json&limit=100 |
| 7132 | GET /admin/objects/{object_id}?versions=true&limit=10 |
| 7133 | GET /admin/objects/{object_id}?version=1 |
| 7134 | GET /admin/objects/{object_id}?source_changes=true&limit=100 |
| 7135 | GET /admin/objects/{object_id}?changes=true&limit=100 |
| 7136 | GET /admin/objects/{object_id}?files=true |
| 7137 | GET /admin/objects/{object_id}?file=report.txt |
| 7138 | Authorization: Token <token> |
| 7139 | ``` |
| 7140 | |
| 7141 | If no inspection query is supplied, the server returns metadata. The admin |
| 7142 | inspection surface never executes the object. `PUT ...?source=true` is the |
| 7143 | admin/operator alias for source edits and uses the same |
| 7144 | `DBBASIC_ENABLE_SOURCE_WRITES` plus admin gate as the runtime source update |
| 7145 | route. `POST .../execute` is the admin/operator execution route and requires |
| 7146 | the admin gate before running object code. It accepts a JSON body: |
| 7147 | |
| 7148 | ```json |
| 7149 | { |
| 7150 | ""method"": ""GET"", |
| 7151 | ""payload"": {} |
| 7152 | } |
| 7153 | ``` |
| 7154 | |
| 7155 | `method` may be `GET`, `POST`, `PUT`, or `DELETE`; `payload` must be a JSON |
| 7156 | object and defaults to `{}`. Unsupported query flags return a 400 response so a |
| 7157 | client cannot accidentally turn an operator view into an execution endpoint. |
| 7158 | |
| 7159 | ## Admin Changes |
| 7160 | |
| 7161 | Scroll and operator dashboards can read a single normalized activity stream |
| 7162 | instead of querying every changelog separately. |
| 7163 | |
| 7164 | ```http |
| 7165 | GET /admin/changes?kind=file&object_id=site_home&limit=100&offset=0 |
| 7166 | Authorization: Token <token> |
| 7167 | ``` |
| 7168 | |
| 7169 | Supported filters: |
| 7170 | |
| 7171 | - `kind`: `source`, `file`, `record`, or `package` |
| 7172 | - `object_id`: source and file changes for one object |
| 7173 | - `collection`: record changes for one collection |
| 7174 | - `record_id`: record changes for one record |
| 7175 | - `package_id`: package changes for one package |
| 7176 | - `file`: file changes for one object-owned file |
| 7177 | |
| 7178 | Response: |
| 7179 | |
| 7180 | ```json |
| 7181 | { |
| 7182 | ""status"": ""ok"", |
| 7183 | ""changes"": [ |
| 7184 | { |
| 7185 | ""kind"": ""file"", |
| 7186 | ""change_id"": ""123e4567-e89b-42d3-a456-426614174000"", |
| 7187 | ""timestamp"": ""2026-01-01T00:00:00+00:00"", |
| 7188 | ""action"": ""file_update"", |
| 7189 | ""actor"": ""api"", |
| 7190 | ""summary"": ""File updated: reports/out.csv"", |
| 7191 | ""correlation_id"": ""123e4567-e89b-42d3-a456-426614174001"", |
| 7192 | ""target"": { |
| 7193 | ""object_id"": ""site_home"", |
| 7194 | ""file_name"": ""reports/out.csv"", |
| 7195 | ""file_size"": 42 |
| 7196 | }, |
| 7197 | ""change"": {} |
| 7198 | } |
| 7199 | ], |
| 7200 | ""count"": 1, |
| 7201 | ""total"": 1, |
| 7202 | ""limit"": 100, |
| 7203 | ""offset"": 0, |
| 7204 | ""has_more"": false, |
| 7205 | ""filters"": { |
| 7206 | ""kind"": ""file"", |
| 7207 | ""object_id"": ""site_home"", |
| 7208 | ""collection"": null, |
| 7209 | ""record_id"": null, |
| 7210 | ""package_id"": null, |
| 7211 | ""file"": ""reports/out.csv"" |
| 7212 | } |
| 7213 | } |
| 7214 | ``` |
| 7215 | |
| 7216 | Each entry keeps the original changelog payload under `change`. Clients should |
| 7217 | render `kind`, `timestamp`, `actor`, `summary`, `target`, and |
| 7218 | `correlation_id` first, then expand `change` for detail views. |
| 7219 | |
| 7220 | ## Admin File Inspection |
| 7221 | |
| 7222 | Object-owned files live under the local runtime data directory: |
| 7223 | |
| 7224 | ```text |
| 7225 | data/files/{object_id}/ |
| 7226 | ``` |
| 7227 | |
| 7228 | The compatibility object route can still list or download one object's files, |
| 7229 | but Scroll and operator dashboards should prefer the admin file inspection |
| 7230 | surface so staging can expose read-only file posture without exposing broad |
| 7231 | object execution routes. |
| 7232 | |
| 7233 | ```http |
| 7234 | GET /admin/files |
| 7235 | GET /admin/files?object_id=site_home |
| 7236 | GET /admin/files?object_id=site_home&file=report.txt |
| 7237 | GET /admin/files/{object_id} |
| 7238 | GET /admin/files/{object_id}?file=report.txt |
| 7239 | Authorization: Token <token> |
| 7240 | ``` |
| 7241 | |
| 7242 | `GET /admin/files` returns a paginated cross-object inventory: |
| 7243 | |
| 7244 | ```json |
| 7245 | { |
| 7246 | ""status"": ""ok"", |
| 7247 | ""files"": [ |
| 7248 | { |
| 7249 | ""object_id"": ""site_home"", |
| 7250 | ""name"": ""report.txt"", |
| 7251 | ""size"": 1200, |
| 7252 | ""modified"": 1760000000.0 |
| 7253 | } |
| 7254 | ], |
| 7255 | ""count"": 1, |
| 7256 | ""total"": 1 |
| 7257 | } |
| 7258 | ``` |
| 7259 | |
| 7260 | Supported query parameters: |
| 7261 | |
| 7262 | - `object_id` narrows the inventory to one object. |
| 7263 | - `limit` defaults to `100` and is capped at `1000`. |
| 7264 | - `offset` defaults to `0`. |
| 7265 | - `file` downloads one object-owned file and requires an object id. |
| 7266 | |
| 7267 | Read routes are admin-token gated. Write routes are additionally disabled unless |
| 7268 | `DBBASIC_ENABLE_FILE_WRITES=true` is set. Public staging should leave file writes |
| 7269 | off until permissions, quota, and review policy are intentionally enabled. |
| 7270 | |
| 7271 | Create one object-owned file: |
| 7272 | |
| 7273 | ```http |
| 7274 | POST /admin/files/{object_id} |
| 7275 | Authorization: Token <token> |
| 7276 | Content-Type: application/json |
| 7277 | ``` |
| 7278 | |
| 7279 | ```json |
| 7280 | { |
| 7281 | ""name"": ""reports/monthly.txt"", |
| 7282 | ""content_base64"": ""aGVsbG8="" |
| 7283 | } |
| 7284 | ``` |
| 7285 | |
| 7286 | Overwrite an existing object-owned file: |
| 7287 | |
| 7288 | ```http |
| 7289 | PUT /admin/files/{object_id} |
| 7290 | Authorization: Token <token> |
| 7291 | Content-Type: application/json |
| 7292 | ``` |
| 7293 | |
| 7294 | Delete one object-owned file: |
| 7295 | |
| 7296 | ```http |
| 7297 | DELETE /admin/files/{object_id}?file=reports/monthly.txt |
| 7298 | Authorization: Token <token> |
| 7299 | ``` |
| 7300 | |
| 7301 | Successful write responses return the file metadata, operation, and correlation |
| 7302 | id: |
| 7303 | |
| 7304 | ```json |
| 7305 | { |
| 7306 | ""status"": ""ok"", |
| 7307 | ""message"": ""File created: reports/monthly.txt"", |
| 7308 | ""object_id"": ""site_home"", |
| 7309 | ""operation"": ""created"", |
| 7310 | ""file"": { |
| 7311 | ""object_id"": ""site_home"", |
| 7312 | ""name"": ""reports/monthly.txt"", |
| 7313 | ""size"": 5, |
| 7314 | ""modified"": 1760000000.0 |
| 7315 | }, |
| 7316 | ""correlation_id"": ""07fb51ef-1a0d-43dd-a82f-2ec756b7cd90"" |
| 7317 | } |
| 7318 | ``` |
| 7319 | |
| 7320 | `POST` rejects existing files with `409`. `PUT` overwrites. Empty names, |
| 7321 | absolute paths, null bytes, and `..` traversal are rejected with `400`. File |
| 7322 | content is capped by `DBBASIC_MAX_OBJECT_FILE_BYTES` and the request body is |
| 7323 | still capped by `DBBASIC_MAX_REQUEST_BYTES`. |
| 7324 | |
| 7325 | ## Admin Ops Events |
| 7326 | |
| 7327 | The operational event feed records object execution failures and auth |
| 7328 | activity in one admin-gated stream: |
| 7329 | |
| 7330 | ```http |
| 7331 | GET /admin/ops |
| 7332 | GET /admin/ops?limit=100 |
| 7333 | GET /admin/ops?kind=execution_error |
| 7334 | GET /admin/ops?kind=auth&event=login_failed |
| 7335 | GET /admin/ops?identifier=alice@example.com |
| 7336 | Authorization: Token <token> |
| 7337 | ``` |
| 7338 | |
| 7339 | Events are newest-first. `execution_error` entries carry `object_id`, |
| 7340 | `method`, `error_type`, a truncated `message`, the caller's `user_id`, and |
| 7341 | the `correlation_id` linking back to source versions and object logs. `auth` |
| 7342 | entries carry `event` (`login_succeeded`, `login_failed`, `logout`, |
| 7343 | `session_minted`), the attempted `identifier` on failures, `user_id` on |
| 7344 | successes, `auth_method`, and session `label`. Rows never contain passwords, |
| 7345 | tokens, or hashes. The feed lives in `data/ops/events.jsonl`, capped to the |
| 7346 | most recent 5000 rows. Failed-login rows are queryable per identifier, which |
| 7347 | is the data the future login-lockout gate will read. |
| 7348 | |
| 7349 | ## MCP Endpoint |
| 7350 | |
| 7351 | AI agents can operate the server through MCP (Model Context Protocol) instead |
| 7352 | of raw HTTP. The endpoint speaks JSON-RPC 2.0 over HTTP POST (the Streamable |
| 7353 | HTTP transport): |
| 7354 | |
| 7355 | ```http |
| 7356 | POST /api/mcp |
| 7357 | Authorization: Token <admin-token-or-admin-role-session-token> |
| 7358 | Content-Type: application/json |
| 7359 | ``` |
| 7360 | |
| 7361 | Supported methods: `initialize` (protocol negotiation, returns an |
| 7362 | `mcp-session-id` header), `tools/list`, `tools/call`, and |
| 7363 | `notifications/*` (acknowledged with `202`). The gate is the same admin gate |
| 7364 | as the rest of the admin surface: the deployment token always works, and an |
| 7365 | admin-role session token works when `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`. |
| 7366 | |
| 7367 | Register with Claude Code: |
| 7368 | |
| 7369 | ```bash |
| 7370 | claude mcp add dbbasic --transport http \ |
| 7371 | --url https://your-host/api/mcp \ |
| 7372 | --header ""Authorization: Bearer YOUR_SESSION_TOKEN"" |
| 7373 | ``` |
| 7374 | |
| 7375 | The tool catalog wraps the existing admin surface — objects |
| 7376 | (list/read/create/update-source/execute, state, logs, metadata, changes), |
| 7377 | collections and records (list/get/create/update/delete), schemas |
| 7378 | (get/update/rollback), cross-collection search (`global_search`), the |
| 7379 | unified change history, and admin status. Every |
| 7380 | tool call is dispatched back through the server's own routing with the |
| 7381 | caller's credentials, so tools inherit the same gates, capability flags |
| 7382 | (source writes, package installs), permission checks, audit trail, and |
| 7383 | correlation ids as external HTTP callers. Tool results carry the underlying |
| 7384 | `http_status` plus the route's JSON response, and failures set `isError`. |
| 7385 | |
| 7386 | Give each agent its own identity instead of the deployment token: create a |
| 7387 | user (for example `agent-claude`) with the admin role, mint a labeled session, |
| 7388 | and connect with that bearer token, so `/admin/changes` distinguishes agent |
| 7389 | actions from operator actions. |
| 7390 | |
| 7391 | ## Service Keys (Per-User AI Provider Keys) |
| 7392 | |
| 7393 | Users store API keys for AI providers so the server can call providers on |
| 7394 | their behalf. The contract is write-only: set a key, list which services |
| 7395 | have one, delete one — key material never appears in any response, never |
| 7396 | lives in record collections, and stays out of portable backups (it sits |
| 7397 | beside `credentials.tsv` in the owner-only identity directory). |
| 7398 | |
| 7399 | ```http |
| 7400 | GET /identity/users/{user_id}/service-keys -> {""services"": [{""service"", ""created_at"", ""updated_at""}]} |
| 7401 | PUT /identity/users/{user_id}/service-keys {""service"": ""anthropic"", ""key"": ""sk-...""} |
| 7402 | DELETE /identity/users/{user_id}/service-keys/{service} |
| 7403 | ``` |
| 7404 | |
| 7405 | A signed-in user manages their own keys (session token or cookie; cookie |
| 7406 | writes require a same-origin request); the admin gate covers operator |
| 7407 | use. Set/remove events land in the ops feed without key material. |
| 7408 | |
| 7409 | ## AI Chat (The Shell's Brain) |
| 7410 | |
| 7411 | One conversation turn against an AI provider, using the caller's stored |
| 7412 | service key, with optional MCP tool calling: |
| 7413 | |
| 7414 | ```http |
| 7415 | POST /api/ai/chat |
| 7416 | Authorization: Bearer <session-token> (or the session cookie) |
| 7417 | |
| 7418 | {""message"": ""find my notes about the flywheel"", |
| 7419 | ""model"": ""anthropic:claude-haiku-4-5"", |
| 7420 | ""tools"": [""global_search"", ""get_record""], |
| 7421 | ""max_rounds"": 6} |
| 7422 | ``` |
| 7423 | |
| 7424 | Models are named `service:model`; the service picks which stored key is |
| 7425 | used (`anthropic`, `openai`). `tools` names a subset of the MCP tool |
| 7426 | catalog — handing small models only a few tools keeps their context |
| 7427 | small, and the subset is per-request configuration. When the model calls |
| 7428 | a tool, the server dispatches it through its own routing **with the |
| 7429 | caller's credentials**, so the AI can do exactly what the caller could do |
| 7430 | directly — permission checks, row filters, and the audit trail all apply. |
| 7431 | |
| 7432 | Response: `{""reply"", ""model"", ""rounds"", ""tool_calls"": [{name, arguments, |
| 7433 | http_status}], ""usage"": {input_tokens, output_tokens}}`, plus |
| 7434 | `""truncated"": true` when the round limit stopped a tool loop. |
| 7435 | |
| 7436 | Requires `DBBASIC_ENABLE_AI_CHAT=true` and a signed-in session; callers |
| 7437 | without a stored key for the model's service get a pointed 400. Timeout |
| 7438 | per provider round: `DBBASIC_AI_TIMEOUT_SECONDS` (default 60); default |
| 7439 | model: `DBBASIC_AI_DEFAULT_MODEL`. |
| 7440 | |
| 7441 | ## Text-to-Speech |
| 7442 | |
| 7443 | ```http |
| 7444 | POST /api/tts |
| 7445 | Authorization: Bearer <session-token> (or the session cookie) |
| 7446 | |
| 7447 | {""text"": ""one note matches"", ""voice"": ""en-us"", ""engine"": ""local""} |
| 7448 | ``` |
| 7449 | |
| 7450 | Response: audio bytes (`audio/wav` for `engine: ""local""`, `audio/mpeg` for |
| 7451 | `engine: ""openai""`) on success, or `{""status"": ""error"", ""error""}` on |
| 7452 | failure. `engine` is optional (`""local""` default, or `""openai""` — uses the |
| 7453 | caller's stored `openai` service key, no engine ever silently falls back |
| 7454 | to the other). Requires `DBBASIC_ENABLE_TTS=true` and a signed-in session. |
| 7455 | `text` capped at 800 characters (413 beyond). `DBBASIC_TTS_CLOUD_TIMEOUT_SECONDS` |
| 7456 | bounds the cloud path (default 30). See `docs/shell-and-ai.md`'s ""Voice"" |
| 7457 | section for engine details and caching. |
| 7458 | |
| 7459 | ## Speech-to-Text |
| 7460 | |
| 7461 | ```http |
| 7462 | POST /api/stt |
| 7463 | Authorization: Bearer <session-token> (or the session cookie) |
| 7464 | Content-Type: audio/webm |
| 7465 | |
| 7466 | <raw audio bytes> |
| 7467 | ``` |
| 7468 | |
| 7469 | Response: `{""status"": ""ok"", ""text"": ""...""}` or the usual error shape. Body |
| 7470 | is raw audio bytes (not JSON) in whatever format `Content-Type` says. |
| 7471 | Cloud-only — no local engine — calls OpenAI's transcription API with the |
| 7472 | caller's stored `openai` service key. Requires `DBBASIC_ENABLE_STT=true` |
| 7473 | (off by default even when TTS is on: this costs money per utterance, not |
| 7474 | per occasional reply) and a signed-in session. Capped at 10MB per clip |
| 7475 | (413 beyond). `DBBASIC_STT_TIMEOUT_SECONDS` bounds the call (default 30). |
| 7476 | |
| 7477 | ## Reader |
| 7478 | |
| 7479 | ```http |
| 7480 | POST /api/read |
| 7481 | Authorization: Bearer <session-token> (or the session cookie) |
| 7482 | |
| 7483 | {""url"": ""https://example.com/article""} |
| 7484 | ``` |
| 7485 | |
| 7486 | Response: `{""status"": ""ok"", ""title"", ""text"", ""links"": [{n, label, href}], |
| 7487 | ""final_url"", ""truncated""}` or the usual error shape. Fetches server-side, |
| 7488 | strips HTML to readable text, numbers links in document order. SSRF-hardened |
| 7489 | (every hostname, including redirect hops, resolved and checked before the |
| 7490 | socket opens; `http`/`https` only; fails closed on any resolution |
| 7491 | ambiguity) — see `object_reader.py`. Requires `DBBASIC_ENABLE_READER=true` |
| 7492 | and a signed-in session. Also offered as the `read_page` MCP tool for |
| 7493 | `/api/ai/chat` — see `docs/shell-and-ai.md`'s ""Reading the Web"" section. |
| 7494 | |
| 7495 | ## Backups |
| 7496 | |
| 7497 | Runtime backups are tar/gzip archives of the whole data directory. They |
| 7498 | contain everything — records, identity, credentials, service keys — so |
| 7499 | the surface is **strictly admin-gated** (admin token or admin-role |
| 7500 | session; never public): |
| 7501 | |
| 7502 | ```http |
| 7503 | GET /admin/backups -> {backups: [{id, created_at, size, kind, scope}], count, schedule} |
| 7504 | POST /admin/backups -> {backup: {id, created_at, size, kind, scope}} (creates one now, 201) |
| 7505 | GET /admin/backups/{id}/download -> the archive bytes (application/gzip, attachment) |
| 7506 | ``` |
| 7507 | |
| 7508 | `kind` is `manual` (created on demand), `package` (auto restore point |
| 7509 | before a package install; `scope` is the package), or `restore-point`. |
| 7510 | The `/admin/status` `capabilities.backups` block reports |
| 7511 | `{available, can_create, can_download, can_restore, scheduled, schedule}` |
| 7512 | so clients gate their buttons off real capability. Restore over HTTP is |
| 7513 | not exposed yet (`can_restore: false`); use the `object_backup.py restore` |
| 7514 | CLI, which is deliberately a deliberate, on-box operation. |
| 7515 | |
| 7516 | **Automatic backups are a config option, not on by default.** A scheduled |
| 7517 | run is an external timer (systemd/cron) calling `object_backup.py create`; |
| 7518 | `DBBASIC_BACKUP_SCHEDULE` records the operator's intent so clients can show |
| 7519 | it. On-demand create and download always work regardless. |
| 7520 | |
| 7521 | ## Realtime Push (Websocket) |
| 7522 | |
| 7523 | Clients get live record-change signals over a websocket at `/ws`, so open |
| 7524 | pages auto-update the instant something changes — no polling. The durable |
| 7525 | event log (see the runtime contract) remains as the poll-based fallback. |
| 7526 | |
| 7527 | ``` |
| 7528 | WebSocket wss://your-host/ws |
| 7529 | ``` |
| 7530 | |
| 7531 | Connect with your session (the `dbbasic_session` cookie is sent |
| 7532 | automatically; a bearer/admin token also works). Anonymous connections |
| 7533 | are closed. Then speak a tiny JSON protocol: |
| 7534 | |
| 7535 | ```json |
| 7536 | // client -> server |
| 7537 | {""action"": ""subscribe"", ""collections"": [""notifications"", ""notes""]} |
| 7538 | {""action"": ""unsubscribe"", ""collections"": [""notes""]} |
| 7539 | {""action"": ""ping""} |
| 7540 | |
| 7541 | // server -> client |
| 7542 | {""type"": ""welcome"", ""user"": ""dan""} |
| 7543 | {""type"": ""subscribed"", ""collections"": [""notifications"", ""notes""]} |
| 7544 | {""type"": ""record"", ""collection"": ""notes"", ""record_id"": ""…"", ""action"": ""created""} |
| 7545 | {""type"": ""pong""} |
| 7546 | ``` |
| 7547 | |
| 7548 | The `record` message is a **signal, not data** — only the collection, |
| 7549 | record id, and action travel, never the record body. The client refetches |
| 7550 | through the normal permission-enforced API. Two guarantees follow the same |
| 7551 | rules as reads: you can only `subscribe` to a collection you may read, and |
| 7552 | with enforcement on you only receive events for records your row filters |
| 7553 | would let you see (a private note's change is never signalled to another |
| 7554 | user). Requires `DBBASIC_ENABLE_REALTIME` (default on); the reverse proxy |
| 7555 | must forward websocket upgrades (Caddy does by default). Single-process |
| 7556 | deployment assumption applies — see |
| 7557 | [asgi-realtime-direction](/docs/asgi-realtime-direction). |
| 7558 | |
| 7559 | The shared nav (`/nav`) and every app page already use this. The |
| 7560 | notification bell subscribes to `notifications` and updates live (with a |
| 7561 | 20s poll as backup); each list page (`/notes`, `/tasks`, `/contacts`, …) |
| 7562 | subscribes to its own collection and re-renders when a record it can see |
| 7563 | changes — in another tab, by another user, or by an agent — over the same |
| 7564 | single shared socket. Any page opts in with one line: |
| 7565 | `window.dbbasicSubscribe(collection, handler)`. |
| 7566 | |
| 7567 | ## Global Search |
| 7568 | |
| 7569 | Records can be searched across every collection whose schema declares |
| 7570 | searchable fields (see the schema field contract's `search` section): |
| 7571 | |
| 7572 | ```http |
| 7573 | GET /api/search?q=flywheel+growth&limit=10 |
| 7574 | GET /api/search?q=flywheel&collections=notes,tasks |
| 7575 | Authorization: Token <token-or-session> (or the dbbasic_session cookie) |
| 7576 | ``` |
| 7577 | |
| 7578 | The query is split on whitespace; every term must match at least one |
| 7579 | searchable field (case-insensitive substring), and a short hex-ish query |
| 7580 | also matches record ids by prefix. Results come back per collection with |
| 7581 | no ranking: |
| 7582 | |
| 7583 | ```json |
| 7584 | { |
| 7585 | ""status"": ""ok"", |
| 7586 | ""query"": ""flywheel growth"", |
| 7587 | ""limit"": 10, |
| 7588 | ""results"": { |
| 7589 | ""notes"": [{""id"": ""n1"", ""content"": ""flywheel growth loop""}], |
| 7590 | ""tasks"": [] |
| 7591 | }, |
| 7592 | ""total_count"": 1 |
| 7593 | } |
| 7594 | ``` |
| 7595 | |
| 7596 | Each result record is trimmed to the schema's `search.result_fields` |
| 7597 | (default: `id` plus the schema's `views.list_fields`, falling back to the |
| 7598 | searchable fields). Search runs inside the permission engine: with |
| 7599 | enforcement on, collections the caller cannot read are skipped entirely, |
| 7600 | row filters hide other users' records, and field permissions redact |
| 7601 | fields before results are returned — the same subject resolution as |
| 7602 | record reads (admin token, session token, or session cookie). Without |
| 7603 | permission checks the endpoint stays behind the admin gate. Malformed |
| 7604 | `search` sections surface in a `warnings` list instead of failing the |
| 7605 | whole request. |
| 7606 | |
| 7607 | ## Admin Collection And Schema Inspection |
| 7608 | |
| 7609 | `/collections` and `/schemas` remain the data and schema API surfaces. Scroll |
| 7610 | and operator dashboards should use the admin inspection aliases for read-only |
| 7611 | collection, record, changelog, schema, and schema-version views. This lets a |
| 7612 | public staging reverse proxy expose inspection routes without exposing the broad |
| 7613 | write-capable collection and schema routes. |
| 7614 | |
| 7615 | ```http |
| 7616 | GET /admin/collections |
| 7617 | GET /admin/collections/{collection} |
| 7618 | GET /admin/collections/{collection}/records |
| 7619 | GET /admin/collections/{collection}/records/{record_id} |
| 7620 | GET /admin/collections/{collection}/changes |
| 7621 | GET /admin/collections/{collection}/records/{record_id}/changes |
| 7622 | GET /admin/schemas |
| 7623 | GET /admin/schemas/{collection} |
| 7624 | GET /admin/schemas/{collection}?versions=true&limit=10 |
| 7625 | GET /admin/schemas/{collection}?version=1 |
| 7626 | Authorization: Token <token> |
| 7627 | ``` |
| 7628 | |
| 7629 | The responses match the underlying read-only `GET /collections*` and |
| 7630 | `GET /schemas*` responses. |
| 7631 | |
| 7632 | ## Admin Collection Record Writes |
| 7633 | |
| 7634 | Operator screens can create, update, and delete collection records through |
| 7635 | narrow admin aliases without exposing the broad write-capable |
| 7636 | `/collections/*` routes through the reverse proxy: |
| 7637 | |
| 7638 | ```http |
| 7639 | POST /admin/collections/{collection}/records |
| 7640 | PUT /admin/collections/{collection}/records/{record_id} |
| 7641 | DELETE /admin/collections/{collection}/records/{record_id} |
| 7642 | Authorization: Token <token> |
| 7643 | ``` |
| 7644 | |
| 7645 | These aliases route to the same handlers as `POST /collections/{collection}/records`, |
| 7646 | `PUT /collections/{collection}/records/{record_id}`, and |
| 7647 | `DELETE /collections/{collection}/records/{record_id}`, so request bodies, |
| 7648 | responses, schema validation, field permissions, record change history, and |
| 7649 | permission enforcement behave identically. Writes require the admin token (or |
| 7650 | enforced permission policy when enforcement is enabled). |
| 7651 | |
| 7652 | Other write methods on `/admin/collections*` paths — including collection |
| 7653 | create or delete — remain unsupported and return `405`. |
| 7654 | |
| 7655 | ## Admin Storage And Compaction |
| 7656 | |
| 7657 | Compaction observability and control for append-mode collections (see |
| 7658 | [storage-modes.md](/docs/storage-modes)): |
| 7659 | |
| 7660 | ```http |
| 7661 | GET /admin/storage |
| 7662 | POST /admin/collections/{collection}/compact |
| 7663 | Authorization: Token <token> |
| 7664 | ``` |
| 7665 | |
| 7666 | `GET /admin/storage` returns stats for every collection whose schema |
| 7667 | currently declares `""storage"": ""append""`: |
| 7668 | |
| 7669 | ```json |
| 7670 | { |
| 7671 | ""status"": ""ok"", |
| 7672 | ""append_collections"": [ |
| 7673 | { |
| 7674 | ""collection"": ""events_log"", |
| 7675 | ""storage"": ""append"", |
| 7676 | ""physical_rows"": 41200, |
| 7677 | ""live_rows"": 9800, |
| 7678 | ""dead_rows"": 31400, |
| 7679 | ""bloat_ratio"": 3.204, |
| 7680 | ""file_bytes"": 5872000, |
| 7681 | ""sidecar_present"": true, |
| 7682 | ""compaction_flagged"": false |
| 7683 | } |
| 7684 | ] |
| 7685 | } |
| 7686 | ``` |
| 7687 | |
| 7688 | This route is deliberately cheap: it never folds a collection's |
| 7689 | `records.tsv` to answer, preferring a warm in-process cache entry or the |
| 7690 | id→offset sidecar's own row count. A collection neither can answer comes |
| 7691 | back as an **estimated** entry instead of paying an O(file) parse on a |
| 7692 | status call — `file_bytes` is still real, but `physical_rows`, |
| 7693 | `live_rows`, `dead_rows`, and `bloat_ratio` are `null` and |
| 7694 | `""estimated"": true` is set. `bloat_ratio` is `dead_rows / max(live_rows, |
| 7695 | 1)`, rounded to 3 decimal places. A classic-mode collection is never |
| 7696 | included. |
| 7697 | |
| 7698 | `POST /admin/collections/{collection}/compact` runs |
| 7699 | `object_records.compact_collection` under the same gate that protects |
| 7700 | other mutating admin record routes (admin token, or enforced permission |
| 7701 | policy when enforcement is enabled — there is no single record for this |
| 7702 | operation, so the collection-level write check is the whole check): |
| 7703 | |
| 7704 | ```json |
| 7705 | { |
| 7706 | ""status"": ""ok"", |
| 7707 | ""collection"": ""events_log"", |
| 7708 | ""rows_before"": 41200, |
| 7709 | ""rows_after"": 9800, |
| 7710 | ""bytes_before"": 5872000, |
| 7711 | ""bytes_after"": 1398600, |
| 7712 | ""duration_ms"": 812.4 |
| 7713 | } |
| 7714 | ``` |
| 7715 | |
| 7716 | Returns `400` for a collection not currently in append storage mode |
| 7717 | (nothing to compact) and `404` for an unknown collection. Compaction also |
| 7718 | runs automatically — on a write, once a collection's dead-row ratio and |
| 7719 | size cross their thresholds, and on a timer if the optional daemon |
| 7720 | (`object_daemon.py`) is running — see storage-modes.md for the envs |
| 7721 | (`DBBASIC_APPEND_COMPACT_MIN_ROWS`, `DBBASIC_COMPACTION_INTERVAL_SECONDS`, |
| 7722 | `DBBASIC_COMPACTION_BLOAT_RATIO`). The `/admin/status` `capabilities.storage` |
| 7723 | block reports `{""append_collections"": <count>}`. |
| 7724 | |
| 7725 | ## Admin Schema Writes |
| 7726 | |
| 7727 | Operator screens can replace and roll back collection schemas through narrow |
| 7728 | admin aliases without exposing the broad write-capable `/schemas/*` routes |
| 7729 | through the reverse proxy: |
| 7730 | |
| 7731 | ```http |
| 7732 | PUT /admin/schemas/{collection} |
| 7733 | POST /admin/schemas/{collection} |
| 7734 | Authorization: Token <token> |
| 7735 | ``` |
| 7736 | |
| 7737 | These aliases route to the same handlers as `PUT /schemas/{collection}` and |
| 7738 | `POST /schemas/{collection}`, so request bodies, responses, schema |
| 7739 | normalization, and schema version history behave identically. `PUT` replaces |
| 7740 | the schema and records a new schema version. `POST` supports action-style |
| 7741 | operations; the only supported action is `{""action"": ""rollback"", |
| 7742 | ""version_id"": <id>}`. Both writes require the admin token. |
| 7743 | |
| 7744 | Other write methods on `/admin/schemas*` paths — including schema delete — |
| 7745 | remain unsupported and return `405`. The admin aliases still do not install |
| 7746 | packages or expose broad `/objects` execution routes. |
| 7747 | |
| 7748 | ## Admin Identity Inspection |
| 7749 | |
| 7750 | `/identity/accounts`, `/identity/users`, and `/identity/sessions` remain the |
| 7751 | identity management API surfaces. Scroll and operator dashboards should use the |
| 7752 | admin inspection aliases for read-only account, user, and session views. This |
| 7753 | lets public staging expose identity posture without exposing identity creation, |
| 7754 | arbitrary session minting, or session revocation. |
| 7755 | |
| 7756 | ```http |
| 7757 | GET /admin/identity/accounts |
| 7758 | GET /admin/identity/accounts/{account_id} |
| 7759 | GET /admin/identity/users |
| 7760 | GET /admin/identity/users?account_id=acme |
| 7761 | GET /admin/identity/users/{user_id} |
| 7762 | GET /admin/identity/sessions |
| 7763 | GET /admin/identity/sessions/{session_id} |
| 7764 | Authorization: Token <token> |
| 7765 | ``` |
| 7766 | |
| 7767 | The responses match the underlying read-only `GET /identity/accounts*`, |
| 7768 | `GET /identity/users*`, and `GET /identity/sessions*` responses. These admin |
| 7769 | aliases are admin-token gated and read-only, with one write exception: the |
| 7770 | password set/reset aliases (`POST/DELETE /admin/identity/users/{user_id}/password`, |
| 7771 | documented under User Passwords). They still do not create accounts or users, |
| 7772 | mint sessions, or revoke sessions. |
| 7773 | |
| 7774 | ## Collections |
| 7775 | |
| 7776 | ```http |
| 7777 | GET /collections |
| 7778 | Authorization: Token <token> |
| 7779 | ``` |
| 7780 | |
| 7781 | Response: |
| 7782 | |
| 7783 | ```json |
| 7784 | { |
| 7785 | ""status"": ""ok"", |
| 7786 | ""collections"": [ |
| 7787 | { |
| 7788 | ""name"": ""site"", |
| 7789 | ""object_count"": 2, |
| 7790 | ""file_count"": 1, |
| 7791 | ""state_object_count"": 1, |
| 7792 | ""log_object_count"": 1, |
| 7793 | ""has_records"": true, |
| 7794 | ""owners"": [""system""], |
| 7795 | ""kinds"": {""system"": 2}, |
| 7796 | ""permission"": { |
| 7797 | ""access_mode"": ""role_based"", |
| 7798 | ""rule_count"": 1, |
| 7799 | ""allow_count"": 1, |
| 7800 | ""deny_count"": 0, |
| 7801 | ""actions"": [""execute"", ""read""], |
| 7802 | ""principals"": [""role:admin""] |
| 7803 | } |
| 7804 | } |
| 7805 | ], |
| 7806 | ""count"": 1 |
| 7807 | } |
| 7808 | ``` |
| 7809 | |
| 7810 | Read one collection with object details: |
| 7811 | |
| 7812 | ```http |
| 7813 | GET /collections/{collection} |
| 7814 | Authorization: Token <token> |
| 7815 | ``` |
| 7816 | |
| 7817 | Response: |
| 7818 | |
| 7819 | ```json |
| 7820 | { |
| 7821 | ""status"": ""ok"", |
| 7822 | ""collection"": { |
| 7823 | ""name"": ""site"", |
| 7824 | ""object_count"": 1, |
| 7825 | ""file_count"": 0, |
| 7826 | ""state_object_count"": 0, |
| 7827 | ""log_object_count"": 0, |
| 7828 | ""has_records"": false, |
| 7829 | ""owners"": [""system""], |
| 7830 | ""kinds"": {""system"": 1}, |
| 7831 | ""permission"": { |
| 7832 | ""access_mode"": ""role_based"", |
| 7833 | ""rule_count"": 0, |
| 7834 | ""allow_count"": 0, |
| 7835 | ""deny_count"": 0, |
| 7836 | ""actions"": [], |
| 7837 | ""principals"": [] |
| 7838 | }, |
| 7839 | ""objects"": [ |
| 7840 | { |
| 7841 | ""object_id"": ""site_home"", |
| 7842 | ""path"": ""site/home.py"", |
| 7843 | ""owner"": ""system"", |
| 7844 | ""kind"": ""system"", |
| 7845 | ""state_count"": 0, |
| 7846 | ""has_logs"": false, |
| 7847 | ""file_count"": 0 |
| 7848 | } |
| 7849 | ] |
| 7850 | } |
| 7851 | } |
| 7852 | ``` |
| 7853 | |
| 7854 | Collections are a derived view over object source IDs, source folders, object |
| 7855 | state/files/log presence, record-file presence, and permission rules. The server |
| 7856 | does not store a separate collection table yet. This keeps the existing |
| 7857 | `/objects` contract stable while giving tools such as Scroll a cleaner grouping |
| 7858 | API. |
| 7859 | |
| 7860 | The public server keeps collection summary routes read-only and admin-token |
| 7861 | gated for now. Missing collections return `404`; unsafe collection names return |
| 7862 | `400`. |
| 7863 | |
| 7864 | ## Collection Records |
| 7865 | |
| 7866 | List records: |
| 7867 | |
| 7868 | ```http |
| 7869 | GET /collections/{collection}/records?limit=100&offset=0 |
| 7870 | Authorization: Token <token> |
| 7871 | ``` |
| 7872 | |
| 7873 | By default this route is admin-token gated. If |
| 7874 | `DBBASIC_ENABLE_PERMISSION_AUDIT=true` or |
| 7875 | `DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true` is set, the route uses the |
| 7876 | persisted permission policy instead. Enforcement uses the `read` action with the |
| 7877 | collection name, applies row filters before `limit` / `offset`, and redacts |
| 7878 | fields according to the matching decision. |
| 7879 | |
| 7880 | Response: |
| 7881 | |
| 7882 | ```json |
| 7883 | { |
| 7884 | ""status"": ""ok"", |
| 7885 | ""collection"": ""contacts"", |
| 7886 | ""records"": [ |
| 7887 | { |
| 7888 | ""id"": ""c1"", |
| 7889 | ""first_name"": ""Ada"", |
| 7890 | ""last_name"": ""Lovelace"" |
| 7891 | } |
| 7892 | ], |
| 7893 | ""count"": 1, |
| 7894 | ""total"": 1, |
| 7895 | ""limit"": 100, |
| 7896 | ""offset"": 0, |
| 7897 | ""has_more"": false |
| 7898 | } |
| 7899 | ``` |
| 7900 | |
| 7901 | Read one record: |
| 7902 | |
| 7903 | ```http |
| 7904 | GET /collections/{collection}/records/{record_id} |
| 7905 | Authorization: Token <token> |
| 7906 | ``` |
| 7907 | |
| 7908 | The detail route is also admin-token gated by default. With permission |
| 7909 | enforcement enabled, the server loads the record, checks the same `read` |
| 7910 | permission against that record, and then applies field allow/deny rules before |
| 7911 | returning it. |
| 7912 | |
| 7913 | Response: |
| 7914 | |
| 7915 | ```json |
| 7916 | { |
| 7917 | ""status"": ""ok"", |
| 7918 | ""collection"": ""contacts"", |
| 7919 | ""record"": { |
| 7920 | ""id"": ""c1"", |
| 7921 | ""first_name"": ""Ada"", |
| 7922 | ""last_name"": ""Lovelace"" |
| 7923 | } |
| 7924 | } |
| 7925 | ``` |
| 7926 | |
| 7927 | Record files live under: |
| 7928 | |
| 7929 | ```text |
| 7930 | data/collections/{collection}/records.tsv |
| 7931 | ``` |
| 7932 | |
| 7933 | The TSV file must have a header row and an `id` column. Values are returned as |
| 7934 | strings. |
| 7935 | |
| 7936 | For new DBBASIC-created records, the server generates a UUIDv4 `id` when the |
| 7937 | request omits one. During migrations, imported rows may preserve legacy integer |
| 7938 | or slug IDs for compatibility, but new public routes and generated UI should |
| 7939 | not assume sequential IDs. |
| 7940 | |
| 7941 | Create one record: |
| 7942 | |
| 7943 | ```http |
| 7944 | POST /collections/{collection}/records |
| 7945 | Authorization: Token <token> |
| 7946 | Content-Type: application/json |
| 7947 | ``` |
| 7948 | |
| 7949 | ```json |
| 7950 | { |
| 7951 | ""first_name"": ""Grace"", |
| 7952 | ""last_name"": ""Hopper"" |
| 7953 | } |
| 7954 | ``` |
| 7955 | |
| 7956 | Response: |
| 7957 | |
| 7958 | ```json |
| 7959 | { |
| 7960 | ""status"": ""ok"", |
| 7961 | ""collection"": ""contacts"", |
| 7962 | ""record"": { |
| 7963 | ""id"": ""8c2d4de7-89fe-45f7-9cf9-f0f42610e7be"", |
| 7964 | ""first_name"": ""Grace"", |
| 7965 | ""last_name"": ""Hopper"" |
| 7966 | } |
| 7967 | } |
| 7968 | ``` |
| 7969 | |
| 7970 | Successful creates return `201`. Explicit record IDs are still accepted for |
| 7971 | imports and compatibility. Duplicate record IDs return `409`. |
| 7972 | |
| 7973 | Update one record: |
| 7974 | |
| 7975 | ```http |
| 7976 | PUT /collections/{collection}/records/{record_id} |
| 7977 | Authorization: Token <token> |
| 7978 | Content-Type: application/json |
| 7979 | ``` |
| 7980 | |
| 7981 | ```json |
| 7982 | { |
| 7983 | ""last_name"": ""Lovelace"", |
| 7984 | ""email"": ""ada@example.com"" |
| 7985 | } |
| 7986 | ``` |
| 7987 | |
| 7988 | The record `id` cannot be changed. |
| 7989 | |
| 7990 | Delete one record: |
| 7991 | |
| 7992 | ```http |
| 7993 | DELETE /collections/{collection}/records/{record_id} |
| 7994 | Authorization: Token <token> |
| 7995 | ``` |
| 7996 | |
| 7997 | Response: |
| 7998 | |
| 7999 | ```json |
| 8000 | { |
| 8001 | ""status"": ""ok"", |
| 8002 | ""collection"": ""contacts"", |
| 8003 | ""record"": { |
| 8004 | ""id"": ""c1"", |
| 8005 | ""first_name"": ""Ada"", |
| 8006 | ""last_name"": ""Lovelace"" |
| 8007 | }, |
| 8008 | ""deleted"": true |
| 8009 | } |
| 8010 | ``` |
| 8011 | |
| 8012 | Missing collections or records return `404`; unsafe collection, record, or field |
| 8013 | names return `400`. Missing subscription entitlements can return |
| 8014 | `402 Payment Required` when the active policy requires them. By default all |
| 8015 | record mutations require the admin token. With |
| 8016 | `DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true`, mutations can also be authorized |
| 8017 | by persisted policy rules using `create`, `update`, and `delete`. |
| 8018 | |
| 8019 | If `data/schemas/{collection}.json` exists, `POST` and `PUT` mutations validate |
| 8020 | known fields before the TSV write. Required fields, create defaults, basic |
| 8021 | scalar types, enum values, length/numeric/pattern rules, and computed/read-only |
| 8022 | fields are enforced by the server. Unknown fields are still accepted so tools |
| 8023 | can add data before a schema is complete. |
| 8024 | |
| 8025 | When `DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true`, schema field permissions are |
| 8026 | also enforced for collection records. Fields resolved as `hidden` are removed |
| 8027 | from `GET` responses. Submitted fields resolved as `read` or `hidden` are |
| 8028 | rejected with `403`: |
| 8029 | |
| 8030 | ```json |
| 8031 | { |
| 8032 | ""status"": ""error"", |
| 8033 | ""error"": ""Record field 'margin' is not editable for this subject"", |
| 8034 | ""code"": ""forbidden"", |
| 8035 | ""denied_fields"": [""margin""] |
| 8036 | } |
| 8037 | ``` |
| 8038 | |
| 8039 | Read collection record change history: |
| 8040 | |
| 8041 | ```http |
| 8042 | GET /collections/{collection}/changes?limit=100&offset=0 |
| 8043 | Authorization: Token <token> |
| 8044 | ``` |
| 8045 | |
| 8046 | Read change history for one record: |
| 8047 | |
| 8048 | ```http |
| 8049 | GET /collections/{collection}/records/{record_id}/changes?limit=100&offset=0 |
| 8050 | Authorization: Token <token> |
| 8051 | ``` |
| 8052 | |
| 8053 | Response: |
| 8054 | |
| 8055 | ```json |
| 8056 | { |
| 8057 | ""status"": ""ok"", |
| 8058 | ""collection"": ""contacts"", |
| 8059 | ""record_id"": ""c1"", |
| 8060 | ""changes"": [ |
| 8061 | { |
| 8062 | ""change_id"": ""0f8b5e96-d8b5-4a0d-8b7d-c3c2e2092d24"", |
| 8063 | ""timestamp"": ""2026-06-29T12:00:00+00:00"", |
| 8064 | ""collection"": ""contacts"", |
| 8065 | ""record_id"": ""c1"", |
| 8066 | ""action"": ""update"", |
| 8067 | ""actor"": ""admin"", |
| 8068 | ""message"": ""Updated record"", |
| 8069 | ""changed_fields"": [""name""], |
| 8070 | ""before"": {""id"": ""c1"", ""name"": ""Ada""}, |
| 8071 | ""after"": {""id"": ""c1"", ""name"": ""Ada Lovelace""} |
| 8072 | } |
| 8073 | ], |
| 8074 | ""count"": 1, |
| 8075 | ""total"": 1, |
| 8076 | ""limit"": 100, |
| 8077 | ""offset"": 0, |
| 8078 | ""has_more"": false |
| 8079 | } |
| 8080 | ``` |
| 8081 | |
| 8082 | Successful `POST`, `PUT`, and `DELETE` record mutations append JSONL entries |
| 8083 | under: |
| 8084 | |
| 8085 | ```text |
| 8086 | data/record_changes/{collection}/changes.jsonl |
| 8087 | ``` |
| 8088 | |
| 8089 | This is the durable audit and admin-history surface. When record events are |
| 8090 | enabled, successful `POST`, `PUT`, and `DELETE` mutations also publish |
| 8091 | `collection.record.created`, `collection.record.updated`, and |
| 8092 | `collection.record.deleted` events from the same change entry. |
| 8093 | |
| 8094 | ## Events |
| 8095 | |
| 8096 | Events are the daemon-compatible notification surface for triggers, listeners, |
| 8097 | webhooks, and future worker-style objects. They are intentionally separate from |
| 8098 | record change history. Change history is the durable audit trail; events are the |
| 8099 | delivery queue. |
| 8100 | |
| 8101 | Collection record mutations publish metadata-only events by default. Set |
| 8102 | `DBBASIC_ENABLE_RECORD_EVENTS=false` to turn that off. Event payloads include |
| 8103 | the change id, collection, record id, action, actor, timestamp, and changed |
| 8104 | field names; they intentionally do not copy full `before` or `after` snapshots |
| 8105 | until subscriber permissions are enforced. |
| 8106 | |
| 8107 | The current endpoints are admin-gated: |
| 8108 | |
| 8109 | ```http |
| 8110 | GET /events?event_type=collection.record.created&limit=100&offset=0 |
| 8111 | DELETE /events?keep_count=1000&keep_seconds=604800 |
| 8112 | Authorization: Token <token> |
| 8113 | ``` |
| 8114 | |
| 8115 | Response: |
| 8116 | |
| 8117 | ```json |
| 8118 | { |
| 8119 | ""status"": ""ok"", |
| 8120 | ""events"": [ |
| 8121 | { |
| 8122 | ""id"": ""789f0c18-7845-4620-a2b0-8bbad752f91b"", |
| 8123 | ""event_type"": ""collection.record.created"", |
| 8124 | ""payload"": { |
| 8125 | ""change_id"": ""0f8b5e96-d8b5-4a0d-8b7d-c3c2e2092d24"", |
| 8126 | ""collection"": ""contacts"", |
| 8127 | ""record_id"": ""c1"", |
| 8128 | ""action"": ""create"", |
| 8129 | ""actor"": ""admin"", |
| 8130 | ""timestamp"": ""2026-06-29T12:00:00Z"", |
| 8131 | ""changed_fields"": [""id"", ""name""] |
| 8132 | }, |
| 8133 | ""source"": ""record_changes"", |
| 8134 | ""actor"": ""admin"", |
| 8135 | ""timestamp"": 1782734400, |
| 8136 | ""created_at"": ""2026-06-29T12:00:00Z"" |
| 8137 | } |
| 8138 | ], |
| 8139 | ""count"": 1, |
| 8140 | ""total"": 1, |
| 8141 | ""limit"": 100, |
| 8142 | ""offset"": 0, |
| 8143 | ""has_more"": false |
| 8144 | } |
| 8145 | ``` |
| 8146 | |
| 8147 | Publish one event: |
| 8148 | |
| 8149 | ```http |
| 8150 | POST /events |
| 8151 | Authorization: Token <token> |
| 8152 | Content-Type: application/json |
| 8153 | ``` |
| 8154 | |
| 8155 | ```json |
| 8156 | { |
| 8157 | ""event_type"": ""collection.record.created"", |
| 8158 | ""source"": ""record_changes"", |
| 8159 | ""payload"": {""collection"": ""contacts"", ""record_id"": ""c1""} |
| 8160 | } |
| 8161 | ``` |
| 8162 | |
| 8163 | Successful publishes return `201`. |
| 8164 | |
| 8165 | Prune the delivery queue: |
| 8166 | |
| 8167 | ```http |
| 8168 | DELETE /events?keep_count=1000&keep_seconds=604800 |
| 8169 | Authorization: Token <token> |
| 8170 | ``` |
| 8171 | |
| 8172 | Response: |
| 8173 | |
| 8174 | ```json |
| 8175 | { |
| 8176 | ""status"": ""ok"", |
| 8177 | ""retention"": { |
| 8178 | ""deleted"": 12, |
| 8179 | ""kept"": 1000, |
| 8180 | ""scanned"": 1012, |
| 8181 | ""protected"": 1, |
| 8182 | ""corrupt_deleted"": 0, |
| 8183 | ""keep_count"": 1000, |
| 8184 | ""keep_seconds"": 604800 |
| 8185 | } |
| 8186 | } |
| 8187 | ``` |
| 8188 | |
| 8189 | `keep_count=0` disables count-based pruning. `keep_seconds=0` disables |
| 8190 | age-based pruning. Subscriptions are never deleted by event pruning, and the |
| 8191 | event referenced by a subscription `last_event_id` is protected to avoid |
| 8192 | surprise replay. Failed delivery attempts are protected too, so the daemon and |
| 8193 | Scroll can still see the pending retry event. Configure the default |
| 8194 | publish/daemon cleanup policy with `DBBASIC_EVENT_KEEP_COUNT` and |
| 8195 | `DBBASIC_EVENT_KEEP_SECONDS`. |
| 8196 | |
| 8197 | Subscriptions are stored in the same daemon-compatible `events` object state: |
| 8198 | |
| 8199 | ```http |
| 8200 | GET /events/subscriptions?event_type=collection.record.created |
| 8201 | POST /events/subscriptions |
| 8202 | DELETE /events/subscriptions?event_type=collection.record.created&subscriber_id=scroll |
| 8203 | Authorization: Token <token> |
| 8204 | ``` |
| 8205 | |
| 8206 | Create or replace a subscription: |
| 8207 | |
| 8208 | ```json |
| 8209 | { |
| 8210 | ""event_type"": ""collection.record.created"", |
| 8211 | ""subscriber_id"": ""scroll"", |
| 8212 | ""callback_url"": ""https://example.com/hooks/dbbasic"" |
| 8213 | } |
| 8214 | ``` |
| 8215 | |
| 8216 | Response: |
| 8217 | |
| 8218 | ```json |
| 8219 | { |
| 8220 | ""status"": ""ok"", |
| 8221 | ""subscription"": { |
| 8222 | ""id"": ""scroll"", |
| 8223 | ""event_type"": ""collection.record.created"", |
| 8224 | ""callback_url"": ""https://example.com/hooks/dbbasic"", |
| 8225 | ""created_at"": 1782734400, |
| 8226 | ""created_at_iso"": ""2026-06-29T12:00:00Z"", |
| 8227 | ""created_by"": ""admin"", |
| 8228 | ""last_event_id"": null, |
| 8229 | ""delivery"": { |
| 8230 | ""status"": ""idle"", |
| 8231 | ""attempts"": 0, |
| 8232 | ""successes"": 0, |
| 8233 | ""failures"": 0, |
| 8234 | ""last_attempted_event_id"": null, |
| 8235 | ""last_attempt_at"": null, |
| 8236 | ""last_attempt_at_iso"": null, |
| 8237 | ""last_success_event_id"": null, |
| 8238 | ""last_success_at"": null, |
| 8239 | ""last_success_at_iso"": null, |
| 8240 | ""last_failure_event_id"": null, |
| 8241 | ""last_failure_at"": null, |
| 8242 | ""last_failure_at_iso"": null, |
| 8243 | ""last_status_code"": null, |
| 8244 | ""last_error"": null |
| 8245 | } |
| 8246 | } |
| 8247 | } |
| 8248 | ``` |
| 8249 | |
| 8250 | The daemon updates `delivery` after each callback attempt. Successful delivery |
| 8251 | sets `status=ok` and advances `last_event_id`; failed delivery sets |
| 8252 | `status=failed`, records the last error/status, and leaves `last_event_id` |
| 8253 | unchanged so the event can be retried. |
| 8254 | |
| 8255 | Delivery status is exposed separately so Scroll and operator dashboards can |
| 8256 | show pending, delivered, and failed work without loading event payloads or |
| 8257 | callback secrets: |
| 8258 | |
| 8259 | ```http |
| 8260 | GET /events/deliveries?event_type=collection.record.created&pending=true |
| 8261 | GET /events/deliveries?delivery_status=failed |
| 8262 | Authorization: Token <token> |
| 8263 | ``` |
| 8264 | |
| 8265 | Response: |
| 8266 | |
| 8267 | ```json |
| 8268 | { |
| 8269 | ""status"": ""ok"", |
| 8270 | ""deliveries"": [ |
| 8271 | { |
| 8272 | ""id"": ""scroll"", |
| 8273 | ""subscriber_id"": ""scroll"", |
| 8274 | ""event_type"": ""collection.record.created"", |
| 8275 | ""callback_url_present"": true, |
| 8276 | ""pending"": true, |
| 8277 | ""pending_count"": 2, |
| 8278 | ""last_event_id"": null, |
| 8279 | ""next_pending_event"": { |
| 8280 | ""id"": ""789f0c18-7845-4620-a2b0-8bbad752f91b"", |
| 8281 | ""event_type"": ""collection.record.created"", |
| 8282 | ""source"": ""record_changes"", |
| 8283 | ""actor"": ""admin"", |
| 8284 | ""timestamp"": 1782734400, |
| 8285 | ""created_at"": ""2026-06-29T12:00:00Z"" |
| 8286 | }, |
| 8287 | ""latest_pending_event"": { |
| 8288 | ""id"": ""98b9fd7c-9e38-48a6-9134-153473c0172a"", |
| 8289 | ""event_type"": ""collection.record.created"", |
| 8290 | ""source"": ""record_changes"", |
| 8291 | ""actor"": ""admin"", |
| 8292 | ""timestamp"": 1782734460, |
| 8293 | ""created_at"": ""2026-06-29T12:01:00Z"" |
| 8294 | }, |
| 8295 | ""delivery"": { |
| 8296 | ""status"": ""failed"", |
| 8297 | ""attempts"": 1, |
| 8298 | ""successes"": 0, |
| 8299 | ""failures"": 1, |
| 8300 | ""last_attempted_event_id"": ""789f0c18-7845-4620-a2b0-8bbad752f91b"", |
| 8301 | ""last_attempt_at"": 1782734401, |
| 8302 | ""last_attempt_at_iso"": ""2026-06-29T12:00:01Z"", |
| 8303 | ""last_status_code"": 500, |
| 8304 | ""last_error"": ""callback failed"" |
| 8305 | } |
| 8306 | } |
| 8307 | ], |
| 8308 | ""count"": 1, |
| 8309 | ""total"": 1, |
| 8310 | ""limit"": 100, |
| 8311 | ""offset"": 0, |
| 8312 | ""has_more"": false |
| 8313 | } |
| 8314 | ``` |
| 8315 | |
| 8316 | `callback_url` is redacted by default. Trusted detail views may request |
| 8317 | `include_callback_url=true`. Event payloads are also excluded by default; use |
| 8318 | `include_events=true&event_limit=10` only for trusted operator/detail views. |
| 8319 | |
| 8320 | ## Packages |
| 8321 | |
| 8322 | Packages are DBBASIC bundles: objects, schemas, permissions, seed data, and |
| 8323 | migrations collected under one package directory. GitHub can host the source, |
| 8324 | but Scroll/Object Server should inspect and later install packages through the |
| 8325 | DBBASIC API so backups, dry-runs, changelogs, and rollback points stay part of |
| 8326 | the object loop. |
| 8327 | |
| 8328 | The first public package surface can list packages, return dry-run plans, and |
| 8329 | perform conservative installs when both admin auth and the explicit package |
| 8330 | install flag are enabled. Dry-runs and installs append compact package changelog |
| 8331 | rows so operators can see which packages were reviewed, installed, or rejected. |
| 8332 | |
| 8333 | Package directory shape: |
| 8334 | |
| 8335 | ```text |
| 8336 | packages/{package_id}/ |
| 8337 | dbbasic-package.json |
| 8338 | objects/ |
| 8339 | schemas/ |
| 8340 | permissions/ |
| 8341 | seed/ |
| 8342 | migrations/ |
| 8343 | ``` |
| 8344 | |
| 8345 | List packages: |
| 8346 | |
| 8347 | ```http |
| 8348 | GET /packages |
| 8349 | Authorization: Token <token> |
| 8350 | ``` |
| 8351 | |
| 8352 | Response: |
| 8353 | |
| 8354 | ```json |
| 8355 | { |
| 8356 | ""status"": ""ok"", |
| 8357 | ""packages"": [ |
| 8358 | { |
| 8359 | ""id"": ""hello-world"", |
| 8360 | ""name"": ""Hello World"", |
| 8361 | ""version"": ""0.1.0"", |
| 8362 | ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."", |
| 8363 | ""status"": ""available"", |
| 8364 | ""object_count"": 1, |
| 8365 | ""schema_count"": 0, |
| 8366 | ""permission_count"": 0, |
| 8367 | ""seed_count"": 0, |
| 8368 | ""migration_count"": 0, |
| 8369 | ""dependency_count"": 0 |
| 8370 | } |
| 8371 | ], |
| 8372 | ""count"": 1 |
| 8373 | } |
| 8374 | ``` |
| 8375 | |
| 8376 | Read one manifest: |
| 8377 | |
| 8378 | ```http |
| 8379 | GET /packages/{package_id} |
| 8380 | Authorization: Token <token> |
| 8381 | ``` |
| 8382 | |
| 8383 | Response: |
| 8384 | |
| 8385 | ```json |
| 8386 | { |
| 8387 | ""status"": ""ok"", |
| 8388 | ""package"": { |
| 8389 | ""id"": ""hello-world"", |
| 8390 | ""name"": ""Hello World"", |
| 8391 | ""version"": ""0.1.0"", |
| 8392 | ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."", |
| 8393 | ""compatibility"": {""object_server"": "">=0.0.1""}, |
| 8394 | ""dependencies"": [], |
| 8395 | ""objects"": [ |
| 8396 | {""id"": ""hello_world"", ""path"": ""objects/hello/world.py""} |
| 8397 | ], |
| 8398 | ""schemas"": [], |
| 8399 | ""permissions"": [], |
| 8400 | ""seed"": [], |
| 8401 | ""migrations"": [] |
| 8402 | } |
| 8403 | } |
| 8404 | ``` |
| 8405 | |
| 8406 | Dry-run a future install/update: |
| 8407 | |
| 8408 | ```http |
| 8409 | GET /packages/{package_id}?dry_run=true |
| 8410 | Authorization: Token <token> |
| 8411 | ``` |
| 8412 | |
| 8413 | Response: |
| 8414 | |
| 8415 | ```json |
| 8416 | { |
| 8417 | ""status"": ""ok"", |
| 8418 | ""dry_run"": { |
| 8419 | ""package"": { |
| 8420 | ""id"": ""hello-world"", |
| 8421 | ""name"": ""Hello World"", |
| 8422 | ""version"": ""0.1.0"", |
| 8423 | ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."", |
| 8424 | ""status"": ""available"", |
| 8425 | ""object_count"": 1, |
| 8426 | ""schema_count"": 0, |
| 8427 | ""permission_count"": 0, |
| 8428 | ""seed_count"": 0, |
| 8429 | ""migration_count"": 0, |
| 8430 | ""dependency_count"": 0 |
| 8431 | }, |
| 8432 | ""mode"": ""dry_run"", |
| 8433 | ""install_enabled"": false, |
| 8434 | ""safe_to_install"": true, |
| 8435 | ""objects"": [ |
| 8436 | { |
| 8437 | ""id"": ""hello_world"", |
| 8438 | ""path"": ""objects/hello/world.py"", |
| 8439 | ""exists"": true, |
| 8440 | ""action"": ""create"", |
| 8441 | ""installed"": false |
| 8442 | } |
| 8443 | ], |
| 8444 | ""schemas"": [], |
| 8445 | ""permissions"": [], |
| 8446 | ""seed"": [], |
| 8447 | ""migrations"": [], |
| 8448 | ""warnings"": [] |
| 8449 | }, |
| 8450 | ""change"": { |
| 8451 | ""change_id"": ""0a34c2e6-3144-4900-8f99-fd7c7ec77c61"", |
| 8452 | ""timestamp"": ""2026-06-30T12:00:00+00:00"", |
| 8453 | ""package_id"": ""hello-world"", |
| 8454 | ""package_version"": ""0.1.0"", |
| 8455 | ""action"": ""dry_run"", |
| 8456 | ""actor"": ""admin"", |
| 8457 | ""message"": ""Dry run package install"", |
| 8458 | ""details"": { |
| 8459 | ""safe_to_install"": true, |
| 8460 | ""install_enabled"": false, |
| 8461 | ""objects"": {""create"": 1}, |
| 8462 | ""schemas"": {}, |
| 8463 | ""permissions"": {}, |
| 8464 | ""seed"": {}, |
| 8465 | ""migrations"": {}, |
| 8466 | ""warnings"": [] |
| 8467 | } |
| 8468 | } |
| 8469 | } |
| 8470 | ``` |
| 8471 | |
| 8472 | Install a reviewed package: |
| 8473 | |
| 8474 | ```http |
| 8475 | POST /packages/{package_id}/install |
| 8476 | Authorization: Token <token> |
| 8477 | Content-Type: application/json |
| 8478 | |
| 8479 | {""allow_replace"": false} |
| 8480 | ``` |
| 8481 | |
| 8482 | Package installs require: |
| 8483 | |
| 8484 | ```text |
| 8485 | DBBASIC_ADMIN_TOKEN=... |
| 8486 | DBBASIC_ENABLE_PACKAGE_INSTALLS=true |
| 8487 | ``` |
| 8488 | |
| 8489 | The first install implementation is deliberately narrow: |
| 8490 | |
| 8491 | - object files are written under the configured object root |
| 8492 | - schema JSON is validated and written under `data/schemas/` |
| 8493 | - seed TSV is written only when `data/collections/{collection}/records.tsv` |
| 8494 | does not already exist |
| 8495 | - replacing objects or schemas requires `{""allow_replace"": true}` |
| 8496 | - package permissions and migrations are rejected until merge/run semantics are |
| 8497 | explicit |
| 8498 | - a runtime restore point is created before any live object, schema, or seed |
| 8499 | file is changed |
| 8500 | |
| 8501 | Response: |
| 8502 | |
| 8503 | ```json |
| 8504 | { |
| 8505 | ""status"": ""ok"", |
| 8506 | ""install"": { |
| 8507 | ""package"": { |
| 8508 | ""id"": ""hello-world"", |
| 8509 | ""name"": ""Hello World"", |
| 8510 | ""version"": ""0.1.0"", |
| 8511 | ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."", |
| 8512 | ""status"": ""available"", |
| 8513 | ""object_count"": 1, |
| 8514 | ""schema_count"": 0, |
| 8515 | ""permission_count"": 0, |
| 8516 | ""seed_count"": 0, |
| 8517 | ""migration_count"": 0, |
| 8518 | ""dependency_count"": 0 |
| 8519 | }, |
| 8520 | ""mode"": ""install"", |
| 8521 | ""install_enabled"": true, |
| 8522 | ""allow_replace"": false, |
| 8523 | ""safe_to_install"": true, |
| 8524 | ""objects"": [ |
| 8525 | { |
| 8526 | ""id"": ""hello_world"", |
| 8527 | ""path"": ""objects/hello/world.py"", |
| 8528 | ""exists"": true, |
| 8529 | ""action"": ""create"", |
| 8530 | ""installed"": false, |
| 8531 | ""status"": ""written"", |
| 8532 | ""destination"": ""hello/world.py"" |
| 8533 | } |
| 8534 | ], |
| 8535 | ""schemas"": [], |
| 8536 | ""permissions"": [], |
| 8537 | ""seed"": [], |
| 8538 | ""migrations"": [], |
| 8539 | ""warnings"": [], |
| 8540 | ""restore_point"": { |
| 8541 | ""path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"", |
| 8542 | ""format_version"": 1, |
| 8543 | ""created_at"": ""2026-06-30T12:01:00Z"", |
| 8544 | ""files"": 14, |
| 8545 | ""bytes"": 12345, |
| 8546 | ""warnings"": [] |
| 8547 | } |
| 8548 | }, |
| 8549 | ""changes"": { |
| 8550 | ""requested"": { |
| 8551 | ""change_id"": ""c0c3f4a6-8b10-47cb-8309-b381354f6cc0"", |
| 8552 | ""action"": ""install_requested"" |
| 8553 | }, |
| 8554 | ""installed"": { |
| 8555 | ""change_id"": ""c77681de-5b1e-4a5e-adc9-16701ca230f5"", |
| 8556 | ""action"": ""installed"" |
| 8557 | } |
| 8558 | }, |
| 8559 | ""restore_point"": { |
| 8560 | ""path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"", |
| 8561 | ""format_version"": 1, |
| 8562 | ""created_at"": ""2026-06-30T12:01:00Z"", |
| 8563 | ""files"": 14, |
| 8564 | ""bytes"": 12345, |
| 8565 | ""warnings"": [] |
| 8566 | } |
| 8567 | } |
| 8568 | ``` |
| 8569 | |
| 8570 | Restore a package install restore point: |
| 8571 | |
| 8572 | ```http |
| 8573 | POST /packages/{package_id}/restore |
| 8574 | Authorization: Token <token> |
| 8575 | Content-Type: application/json |
| 8576 | |
| 8577 | {""change_id"": ""c77681de-5b1e-4a5e-adc9-16701ca230f5"", ""confirm"": ""restore-runtime""} |
| 8578 | ``` |
| 8579 | |
| 8580 | Package restore requires: |
| 8581 | |
| 8582 | ```text |
| 8583 | DBBASIC_ADMIN_TOKEN=... |
| 8584 | DBBASIC_ENABLE_PACKAGE_RESTORE=true |
| 8585 | ``` |
| 8586 | |
| 8587 | The change id must point at a recorded package change with restore-point |
| 8588 | metadata. The server restores that runtime snapshot with overwrite and |
| 8589 | prune-extra enabled, then appends `restore_requested` and `rolled_back` package |
| 8590 | change rows. This is intentionally a whole-runtime rollback, not a selective |
| 8591 | package uninstall, and the restore path must live under the configured backup |
| 8592 | directory. |
| 8593 | |
| 8594 | Response: |
| 8595 | |
| 8596 | ```json |
| 8597 | { |
| 8598 | ""status"": ""ok"", |
| 8599 | ""restore"": { |
| 8600 | ""backup_path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"", |
| 8601 | ""objects_dir"": ""objects"", |
| 8602 | ""data_dir"": ""data"", |
| 8603 | ""files"": 14, |
| 8604 | ""bytes"": 12345, |
| 8605 | ""overwritten"": true, |
| 8606 | ""pruned_files"": 3, |
| 8607 | ""pruned_dirs"": 2 |
| 8608 | }, |
| 8609 | ""restore_point"": { |
| 8610 | ""path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"", |
| 8611 | ""format_version"": 1, |
| 8612 | ""created_at"": ""2026-06-30T12:01:00Z"", |
| 8613 | ""files"": 14, |
| 8614 | ""bytes"": 12345, |
| 8615 | ""warnings"": [] |
| 8616 | }, |
| 8617 | ""from_change"": { |
| 8618 | ""change_id"": ""c77681de-5b1e-4a5e-adc9-16701ca230f5"", |
| 8619 | ""action"": ""installed"" |
| 8620 | }, |
| 8621 | ""changes"": { |
| 8622 | ""requested"": { |
| 8623 | ""change_id"": ""9fd9b09c-22bf-41c2-b6ee-3a96ca195fef"", |
| 8624 | ""action"": ""restore_requested"" |
| 8625 | }, |
| 8626 | ""rolled_back"": { |
| 8627 | ""change_id"": ""9a63dcde-ec78-426b-8787-f1e8ec401190"", |
| 8628 | ""action"": ""rolled_back"" |
| 8629 | } |
| 8630 | } |
| 8631 | } |
| 8632 | ``` |
| 8633 | |
| 8634 | Read package changelog: |
| 8635 | |
| 8636 | ```http |
| 8637 | GET /packages/{package_id}/changes?limit=100&offset=0 |
| 8638 | Authorization: Token <token> |
| 8639 | ``` |
| 8640 | |
| 8641 | Response: |
| 8642 | |
| 8643 | ```json |
| 8644 | { |
| 8645 | ""status"": ""ok"", |
| 8646 | ""package_id"": ""hello-world"", |
| 8647 | ""changes"": [ |
| 8648 | { |
| 8649 | ""change_id"": ""0a34c2e6-3144-4900-8f99-fd7c7ec77c61"", |
| 8650 | ""timestamp"": ""2026-06-30T12:00:00+00:00"", |
| 8651 | ""package_id"": ""hello-world"", |
| 8652 | ""package_version"": ""0.1.0"", |
| 8653 | ""action"": ""dry_run"", |
| 8654 | ""actor"": ""admin"", |
| 8655 | ""message"": ""Dry run package install"", |
| 8656 | ""details"": { |
| 8657 | ""safe_to_install"": true, |
| 8658 | ""install_enabled"": false, |
| 8659 | ""objects"": {""create"": 1}, |
| 8660 | ""schemas"": {}, |
| 8661 | ""permissions"": {}, |
| 8662 | ""seed"": {}, |
| 8663 | ""migrations"": {}, |
| 8664 | ""warnings"": [] |
| 8665 | } |
| 8666 | } |
| 8667 | ], |
| 8668 | ""count"": 1, |
| 8669 | ""total"": 1, |
| 8670 | ""limit"": 100, |
| 8671 | ""offset"": 0, |
| 8672 | ""has_more"": false |
| 8673 | } |
| 8674 | ``` |
| 8675 | |
| 8676 | Package ids are route-safe lowercase names such as `hello-world` or |
| 8677 | `crm-starter`. Package manifests use relative paths only. Absolute paths, |
| 8678 | `..`, null bytes, unsupported permission/migration writes, and unsafe |
| 8679 | source/data destinations are rejected at this layer. |
| 8680 | |
| 8681 | ## Schemas |
| 8682 | |
| 8683 | ```http |
| 8684 | GET /schemas |
| 8685 | Authorization: Token <token> |
| 8686 | ``` |
| 8687 | |
| 8688 | Response: |
| 8689 | |
| 8690 | ```json |
| 8691 | { |
| 8692 | ""status"": ""ok"", |
| 8693 | ""schemas"": [ |
| 8694 | { |
| 8695 | ""name"": ""invoices"", |
| 8696 | ""title"": ""Invoices"", |
| 8697 | ""source"": ""manual"", |
| 8698 | ""version"": 1, |
| 8699 | ""field_count"": 4 |
| 8700 | } |
| 8701 | ], |
| 8702 | ""count"": 1 |
| 8703 | } |
| 8704 | ``` |
| 8705 | |
| 8706 | Read one schema: |
| 8707 | |
| 8708 | ```http |
| 8709 | GET /schemas/{collection} |
| 8710 | Authorization: Token <token> |
| 8711 | ``` |
| 8712 | |
| 8713 | Response: |
| 8714 | |
| 8715 | ```json |
| 8716 | { |
| 8717 | ""status"": ""ok"", |
| 8718 | ""message"": ""Schema updated to version 1"", |
| 8719 | ""version_id"": 1, |
| 8720 | ""collection"": ""invoices"", |
| 8721 | ""schema"": { |
| 8722 | ""name"": ""invoices"", |
| 8723 | ""title"": ""Invoices"", |
| 8724 | ""source"": ""manual"", |
| 8725 | ""version"": 1, |
| 8726 | ""fields"": [ |
| 8727 | { |
| 8728 | ""name"": ""customer_id"", |
| 8729 | ""type"": ""relation"", |
| 8730 | ""required"": true, |
| 8731 | ""relation"": {""collection"": ""contacts""}, |
| 8732 | ""validation"": {""not_null"": true} |
| 8733 | }, |
| 8734 | { |
| 8735 | ""name"": ""total"", |
| 8736 | ""type"": ""computed"", |
| 8737 | ""required"": false, |
| 8738 | ""computed"": ""sum(line_items)"" |
| 8739 | }, |
| 8740 | { |
| 8741 | ""name"": ""cost_price"", |
| 8742 | ""type"": ""currency"", |
| 8743 | ""required"": false, |
| 8744 | ""permissions"": {""admin"": ""edit"", ""sales"": ""hidden""}, |
| 8745 | ""ui"": {""section"": ""totals""} |
| 8746 | } |
| 8747 | ], |
| 8748 | ""field_count"": 3 |
| 8749 | } |
| 8750 | } |
| 8751 | ``` |
| 8752 | |
| 8753 | Replace one manual schema: |
| 8754 | |
| 8755 | ```http |
| 8756 | PUT /schemas/{collection} |
| 8757 | Content-Type: application/json |
| 8758 | Authorization: Token <token> |
| 8759 | ``` |
| 8760 | |
| 8761 | Request: |
| 8762 | |
| 8763 | ```json |
| 8764 | { |
| 8765 | ""schema"": { |
| 8766 | ""title"": ""Invoices"", |
| 8767 | ""ui"": {""default_view"": ""form""}, |
| 8768 | ""views"": [{""name"": ""invoice_admin"", ""type"": ""form""}], |
| 8769 | ""fields"": [ |
| 8770 | { |
| 8771 | ""name"": ""invoice_date"", |
| 8772 | ""type"": ""date"", |
| 8773 | ""required"": true, |
| 8774 | ""layout"": {""column"": 1} |
| 8775 | }, |
| 8776 | { |
| 8777 | ""name"": ""margin"", |
| 8778 | ""type"": ""currency"", |
| 8779 | ""permissions"": {""admin"": ""edit"", ""sales"": ""hidden""} |
| 8780 | } |
| 8781 | ] |
| 8782 | } |
| 8783 | } |
| 8784 | ``` |
| 8785 | |
| 8786 | Response: |
| 8787 | |
| 8788 | ```json |
| 8789 | { |
| 8790 | ""status"": ""ok"", |
| 8791 | ""schema"": { |
| 8792 | ""name"": ""invoices"", |
| 8793 | ""title"": ""Invoices"", |
| 8794 | ""source"": ""manual"", |
| 8795 | ""version"": 1, |
| 8796 | ""fields"": [ |
| 8797 | {""name"": ""invoice_date"", ""type"": ""date"", ""required"": true}, |
| 8798 | { |
| 8799 | ""name"": ""margin"", |
| 8800 | ""type"": ""currency"", |
| 8801 | ""required"": false, |
| 8802 | ""permissions"": {""admin"": ""edit"", ""sales"": ""hidden""} |
| 8803 | } |
| 8804 | ], |
| 8805 | ""field_count"": 2, |
| 8806 | ""ui"": {""default_view"": ""form""}, |
| 8807 | ""views"": [{""name"": ""invoice_admin"", ""type"": ""form""}] |
| 8808 | } |
| 8809 | } |
| 8810 | ``` |
| 8811 | |
| 8812 | List schema versions: |
| 8813 | |
| 8814 | ```http |
| 8815 | GET /schemas/{collection}?versions=true&limit=10 |
| 8816 | Authorization: Token <token> |
| 8817 | ``` |
| 8818 | |
| 8819 | Response: |
| 8820 | |
| 8821 | ```json |
| 8822 | { |
| 8823 | ""status"": ""ok"", |
| 8824 | ""collection"": ""invoices"", |
| 8825 | ""versions"": [ |
| 8826 | { |
| 8827 | ""version_id"": 2, |
| 8828 | ""timestamp"": ""2026-06-29T12:00:00"", |
| 8829 | ""author"": ""admin"", |
| 8830 | ""message"": ""Add margin field"", |
| 8831 | ""hash"": ""..."" |
| 8832 | }, |
| 8833 | { |
| 8834 | ""version_id"": 1, |
| 8835 | ""timestamp"": ""2026-06-29T11:50:00"", |
| 8836 | ""author"": ""admin"", |
| 8837 | ""message"": ""Initial schema"", |
| 8838 | ""hash"": ""..."" |
| 8839 | } |
| 8840 | ], |
| 8841 | ""count"": 2 |
| 8842 | } |
| 8843 | ``` |
| 8844 | |
| 8845 | Read one schema version: |
| 8846 | |
| 8847 | ```http |
| 8848 | GET /schemas/{collection}?version=1 |
| 8849 | Authorization: Token <token> |
| 8850 | ``` |
| 8851 | |
| 8852 | Response: |
| 8853 | |
| 8854 | ```json |
| 8855 | { |
| 8856 | ""status"": ""ok"", |
| 8857 | ""collection"": ""invoices"", |
| 8858 | ""version"": { |
| 8859 | ""version_id"": 1, |
| 8860 | ""timestamp"": ""2026-06-29T11:50:00"", |
| 8861 | ""author"": ""admin"", |
| 8862 | ""message"": ""Initial schema"", |
| 8863 | ""hash"": ""..."", |
| 8864 | ""content"": ""{...}"", |
| 8865 | ""schema"": { |
| 8866 | ""name"": ""invoices"", |
| 8867 | ""title"": ""Invoices"", |
| 8868 | ""source"": ""manual"", |
| 8869 | ""version"": 1, |
| 8870 | ""fields"": [], |
| 8871 | ""field_count"": 0 |
| 8872 | } |
| 8873 | } |
| 8874 | } |
| 8875 | ``` |
| 8876 | |
| 8877 | Rollback is non-destructive. It creates a new latest version containing the old |
| 8878 | schema content and then replaces the live schema file: |
| 8879 | |
| 8880 | ```http |
| 8881 | POST /schemas/{collection} |
| 8882 | Content-Type: application/json |
| 8883 | Authorization: Token <token> |
| 8884 | ``` |
| 8885 | |
| 8886 | Request: |
| 8887 | |
| 8888 | ```json |
| 8889 | { |
| 8890 | ""action"": ""rollback"", |
| 8891 | ""version_id"": 1, |
| 8892 | ""author"": ""admin"", |
| 8893 | ""message"": ""Restore first invoice form"" |
| 8894 | } |
| 8895 | ``` |
| 8896 | |
| 8897 | Response: |
| 8898 | |
| 8899 | ```json |
| 8900 | { |
| 8901 | ""status"": ""ok"", |
| 8902 | ""message"": ""Rolled back schema to version 1"", |
| 8903 | ""version_id"": 1, |
| 8904 | ""new_version_id"": 3, |
| 8905 | ""collection"": ""invoices"", |
| 8906 | ""schema"": { |
| 8907 | ""name"": ""invoices"", |
| 8908 | ""title"": ""Invoices"", |
| 8909 | ""source"": ""manual"", |
| 8910 | ""version"": 1, |
| 8911 | ""fields"": [], |
| 8912 | ""field_count"": 0 |
| 8913 | } |
| 8914 | } |
| 8915 | ``` |
| 8916 | |
| 8917 | Schema files live under: |
| 8918 | |
| 8919 | ```text |
| 8920 | data/schemas/{collection}.json |
| 8921 | data/schema_versions/{collection}/metadata.tsv |
| 8922 | data/schema_versions/{collection}/vN.json |
| 8923 | ``` |
| 8924 | |
| 8925 | Manual schemas can be replaced through the admin-token gated `PUT` route. Writes |
| 8926 | are atomic file replacements under `data/schemas/`, and each write records a |
| 8927 | schema version. If a collection has no manual schema, the server may return an |
| 8928 | empty derived schema for that collection so Scroll can still show the collection |
| 8929 | and later attach fields. Missing schemas return `404`; unsafe schema names |
| 8930 | return `400`. |
| 8931 | |
| 8932 | Schema `permissions` and `ui` fields are preserved for generated admin screens. |
| 8933 | With permission enforcement enabled, schema `permissions` also refine record |
| 8934 | reads and writes after the broader server policy has allowed the row. |
| 8935 | |
| 8936 | ## Create Object |
| 8937 | |
| 8938 | ```http |
| 8939 | POST /objects |
| 8940 | Content-Type: application/json |
| 8941 | Authorization: Token <token> |
| 8942 | ``` |
| 8943 | |
| 8944 | Request: |
| 8945 | |
| 8946 | ```json |
| 8947 | { |
| 8948 | ""object_id"": ""site_home"", |
| 8949 | ""code"": ""def GET(request):\n return {\""status\"": \""ok\""}\n"", |
| 8950 | ""author"": ""admin"", |
| 8951 | ""message"": ""Create home object"", |
| 8952 | ""description"": ""Optional client-provided description"" |
| 8953 | } |
| 8954 | ``` |
| 8955 | |
| 8956 | Response: |
| 8957 | |
| 8958 | ```json |
| 8959 | { |
| 8960 | ""status"": ""ok"", |
| 8961 | ""object_id"": ""site_home"", |
| 8962 | ""message"": ""Object created: site_home"", |
| 8963 | ""version_id"": 1, |
| 8964 | ""correlation_id"": ""123e4567-e89b-42d3-a456-426614174001"" |
| 8965 | } |
| 8966 | ``` |
| 8967 | |
| 8968 | Creation is intentionally gated. It requires `DBBASIC_ENABLE_SOURCE_WRITES=true` |
| 8969 | and the same admin gate used by source updates. A configured |
| 8970 | `DBBASIC_ADMIN_TOKEN` always works; an active DBBASIC session with an admin role |
| 8971 | can also pass the gate only when `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`. |
| 8972 | |
| 8973 | Clients can choose one of these creation shapes: |
| 8974 | |
| 8975 | - `object_id` creates that exact object ID, such as `site_home`. |
| 8976 | - `name` plus `owner_user_id` creates `u_<owner_user_id>_<name>`, such as |
| 8977 | `u_42_deals`. |
| 8978 | - `name` with an admin session whose `user_id` is numeric also creates |
| 8979 | `u_<session_user_id>_<name>`. |
| 8980 | - `name` without a user owner creates a system object using `name` as the object |
| 8981 | ID. |
| 8982 | |
| 8983 | The first source write creates version `1`, appends a `source_create` change |
| 8984 | entry, and the object can be executed immediately through |
| 8985 | `GET/POST/PUT/DELETE /objects/{object_id}`. Existing object IDs return `409`. |
| 8986 | The `description` field is stored in source-change details for tools like |
| 8987 | Scroll. |
| 8988 | |
| 8989 | ## Execute Object |
| 8990 | |
| 8991 | ```http |
| 8992 | GET /objects/{object_id} |
| 8993 | POST /objects/{object_id} |
| 8994 | PUT /objects/{object_id} |
| 8995 | DELETE /objects/{object_id} |
| 8996 | ``` |
| 8997 | |
| 8998 | `GET` passes query parameters to the object. `POST`, `PUT`, and `DELETE` pass a |
| 8999 | JSON body when present. |
| 9000 | |
| 9001 | Compatibility details from the working prototype: |
| 9002 | |
| 9003 | - `POST` merges query parameters into the JSON body without overriding body keys. |
| 9004 | - `POST` with a non-JSON body passes raw bytes as the `body` field plus query |
| 9005 | parameters. |
| 9006 | - `PUT` and `DELETE` reject invalid JSON bodies with `400`. |
| 9007 | - request bodies over `DBBASIC_MAX_REQUEST_BYTES` are rejected with `413`. |
| 9008 | - rate-limited requests are rejected with `429`. |
| 9009 | - full request or object execution capacity is rejected with `503`. |
| 9010 | - object executions over `DBBASIC_OBJECT_TIMEOUT_SECONDS` are rejected with `504`. |
| 9011 | - `POST` with `{""action"": ""rollback""}` is reserved for rollback. |
| 9012 | - `PUT /objects/{object_id}?source=true` is reserved for source updates. |
| 9013 | |
| 9014 | Example: |
| 9015 | |
| 9016 | ```http |
| 9017 | POST /objects/basics_counter |
| 9018 | Content-Type: application/json |
| 9019 | ``` |
| 9020 | |
| 9021 | ```json |
| 9022 | { |
| 9023 | ""action"": ""increment"" |
| 9024 | } |
| 9025 | ``` |
| 9026 | |
| 9027 | Execution responses are object-defined. The server should pass successful object |
| 9028 | results through without wrapping them in a new envelope. |
| 9029 | |
| 9030 | Compatibility response shapes: |
| 9031 | |
| 9032 | - normal dicts return JSON |
| 9033 | - dicts with `content_type` and `body` return raw HTTP responses |
| 9034 | - `(status, headers, body)` tuples return low-level HTTP responses |
| 9035 | - strings return `text/html; charset=utf-8` |
| 9036 | - bytes return `application/octet-stream` |
| 9037 | |
| 9038 | Example HTML object response: |
| 9039 | |
| 9040 | ```python |
| 9041 | def GET(request): |
| 9042 | return { |
| 9043 | ""content_type"": ""text/html; charset=utf-8"", |
| 9044 | ""body"": ""<!doctype html><h1>DBBASIC</h1>"", |
| 9045 | } |
| 9046 | ``` |
| 9047 | |
| 9048 | Example tuple response: |
| 9049 | |
| 9050 | ```python |
| 9051 | def POST(request): |
| 9052 | return (201, [(""Content-Type"", ""text/plain"")], [b""created""]) |
| 9053 | ``` |
| 9054 | |
| 9055 | On server-side execution failure, return a non-2xx status with: |
| 9056 | |
| 9057 | ```json |
| 9058 | { |
| 9059 | ""status"": ""error"", |
| 9060 | ""error"": ""Execution failed: ..."" |
| 9061 | } |
| 9062 | ``` |
| 9063 | |
| 9064 | Future structured error fields may be added, but existing clients must still be |
| 9065 | able to read the `error` field. |
| 9066 | |
| 9067 | If `DBBASIC_OBJECT_TIMEOUT_SECONDS` is set above zero, the public ASGI server |
| 9068 | runs object methods in a subprocess. When the timeout expires, the worker is |
| 9069 | terminated and the client receives: |
| 9070 | |
| 9071 | ```http |
| 9072 | 504 Gateway Timeout |
| 9073 | ``` |
| 9074 | |
| 9075 | ```json |
| 9076 | { |
| 9077 | ""status"": ""error"", |
| 9078 | ""error"": ""Execution failed: GET timed out for object basics_slow after 5 seconds"" |
| 9079 | } |
| 9080 | ``` |
| 9081 | |
| 9082 | The timeout is a wall-clock boundary. It is not a complete CPU or memory |
| 9083 | sandbox. |
| 9084 | |
| 9085 | Objects named in `DBBASIC_TRUSTED_IN_PROCESS_OBJECTS` keep the fast in-process |
| 9086 | execution path even when timeouts are enabled. This is for reviewed |
| 9087 | server-owned objects, such as a high-traffic public homepage. Do not use it for |
| 9088 | unreviewed user code. |
| 9089 | |
| 9090 | ## Source |
| 9091 | |
| 9092 | ```http |
| 9093 | GET /objects/{object_id}?source=true&format=json |
| 9094 | Authorization: Token <token> |
| 9095 | ``` |
| 9096 | |
| 9097 | Response: |
| 9098 | |
| 9099 | ```json |
| 9100 | { |
| 9101 | ""status"": ""ok"", |
| 9102 | ""object_id"": ""basics_counter"", |
| 9103 | ""source"": ""def GET(request):\n return {\""status\"": \""ok\""}\n"" |
| 9104 | } |
| 9105 | ``` |
| 9106 | |
| 9107 | HTML source rendering can exist for browsers, but JSON must stay available with |
| 9108 | `format=json` or an `Accept: application/json` request. |
| 9109 | |
| 9110 | ## Update Source |
| 9111 | |
| 9112 | ```http |
| 9113 | PUT /objects/{object_id}?source=true |
| 9114 | Content-Type: application/json |
| 9115 | Authorization: Token <token> |
| 9116 | ``` |
| 9117 | |
| 9118 | Request: |
| 9119 | |
| 9120 | ```json |
| 9121 | { |
| 9122 | ""code"": ""def GET(request):\n return {\""status\"": \""updated\""}\n"", |
| 9123 | ""author"": ""api"", |
| 9124 | ""message"": ""Updated via client"" |
| 9125 | } |
| 9126 | ``` |
| 9127 | |
| 9128 | Response: |
| 9129 | |
| 9130 | ```json |
| 9131 | { |
| 9132 | ""status"": ""ok"", |
| 9133 | ""message"": ""Code updated to version 2"", |
| 9134 | ""version_id"": 2, |
| 9135 | ""object_id"": ""u_42_deals"" |
| 9136 | } |
| 9137 | ``` |
| 9138 | |
| 9139 | The update must save a source version, write the source file, and leave enough |
| 9140 | information for clients to show the new version number. |
| 9141 | |
| 9142 | The current public server keeps this endpoint closed by default. Local |
| 9143 | development source writes require: |
| 9144 | |
| 9145 | ```bash |
| 9146 | DBBASIC_ENABLE_SOURCE_WRITES=true |
| 9147 | DBBASIC_ADMIN_TOKEN=replace-with-a-local-dev-token |
| 9148 | ``` |
| 9149 | |
| 9150 | The token value shown here is a placeholder. Real deployments must generate a |
| 9151 | server-specific secret outside the source tree. Production code should replace |
| 9152 | that temporary gate with the real auth and permission system before this |
| 9153 | endpoint is exposed to users. |
| 9154 | |
| 9155 | ## State |
| 9156 | |
| 9157 | ```http |
| 9158 | GET /objects/{object_id}?state=true |
| 9159 | Authorization: Token <token> |
| 9160 | ``` |
| 9161 | |
| 9162 | Response: |
| 9163 | |
| 9164 | ```json |
| 9165 | { |
| 9166 | ""status"": ""ok"", |
| 9167 | ""object_id"": ""basics_counter"", |
| 9168 | ""state"": { |
| 9169 | ""count"": 3 |
| 9170 | } |
| 9171 | } |
| 9172 | ``` |
| 9173 | |
| 9174 | State is currently read from: |
| 9175 | |
| 9176 | ```text |
| 9177 | data/state/{object_id}/state.tsv |
| 9178 | ``` |
| 9179 | |
| 9180 | Rows may be `key<TAB>value` or `key<TAB>value<TAB>timestamp`. The public server |
| 9181 | currently exposes state read-only. |
| 9182 | |
| 9183 | ## Metadata |
| 9184 | |
| 9185 | ```http |
| 9186 | GET /objects/{object_id}?metadata=true |
| 9187 | Authorization: Token <token> |
| 9188 | ``` |
| 9189 | |
| 9190 | Response: |
| 9191 | |
| 9192 | ```json |
| 9193 | { |
| 9194 | ""status"": ""ok"", |
| 9195 | ""object_id"": ""basics_counter"", |
| 9196 | ""metadata"": { |
| 9197 | ""object_id"": ""basics_counter"", |
| 9198 | ""source_path"": ""basics/counter.py"", |
| 9199 | ""owner"": ""system"", |
| 9200 | ""kind"": ""system"", |
| 9201 | ""last_modified"": 1760000000.0, |
| 9202 | ""state_count"": 1, |
| 9203 | ""state_keys"": [""count""], |
| 9204 | ""log_count"": 3, |
| 9205 | ""file_count"": 1, |
| 9206 | ""version_count"": 2 |
| 9207 | } |
| 9208 | } |
| 9209 | ``` |
| 9210 | |
| 9211 | The public server reports source paths relative to the object source root, not |
| 9212 | absolute local filesystem paths. Metadata may grow over time. Existing clients |
| 9213 | expect the top-level `metadata` field. |
| 9214 | |
| 9215 | ## Files |
| 9216 | |
| 9217 | ```http |
| 9218 | GET /objects/{object_id}?files=true |
| 9219 | Authorization: Token <token> |
| 9220 | ``` |
| 9221 | |
| 9222 | Response: |
| 9223 | |
| 9224 | ```json |
| 9225 | { |
| 9226 | ""status"": ""ok"", |
| 9227 | ""object_id"": ""site_home"", |
| 9228 | ""files"": [ |
| 9229 | { |
| 9230 | ""name"": ""report.txt"", |
| 9231 | ""size"": 1200, |
| 9232 | ""modified"": 1760000000.0 |
| 9233 | } |
| 9234 | ], |
| 9235 | ""count"": 1 |
| 9236 | } |
| 9237 | ``` |
| 9238 | |
| 9239 | Download one object-owned file: |
| 9240 | |
| 9241 | ```http |
| 9242 | GET /objects/{object_id}?file=report.txt |
| 9243 | Authorization: Token <token> |
| 9244 | ``` |
| 9245 | |
| 9246 | The server reads files from: |
| 9247 | |
| 9248 | ```text |
| 9249 | data/files/{object_id}/ |
| 9250 | ``` |
| 9251 | |
| 9252 | Filenames are validated before filesystem access. Empty names, absolute paths, |
| 9253 | null bytes, and `..` traversal are rejected with `400`. Missing files return |
| 9254 | `404`. |
| 9255 | |
| 9256 | This public object route stays read-only for files. Scroll and operator |
| 9257 | dashboards should use `/admin/files*` for file inventory, downloads, and the |
| 9258 | separately gated admin write workflow. |
| 9259 | |
| 9260 | ## Logs |
| 9261 | |
| 9262 | ```http |
| 9263 | GET /objects/{object_id}?logs=true&format=json&limit=100 |
| 9264 | Authorization: Token <token> |
| 9265 | ``` |
| 9266 | |
| 9267 | Response: |
| 9268 | |
| 9269 | ```json |
| 9270 | { |
| 9271 | ""status"": ""ok"", |
| 9272 | ""object_id"": ""basics_counter"", |
| 9273 | ""logs"": [ |
| 9274 | { |
| 9275 | ""entry_id"": ""a1"", |
| 9276 | ""timestamp"": ""2026-01-01T00:00:00"", |
| 9277 | ""level"": ""DEBUG"", |
| 9278 | ""message"": ""GET completed successfully"", |
| 9279 | ""method"": ""GET"", |
| 9280 | ""status"": ""success"", |
| 9281 | ""duration_ms"": ""1.25"" |
| 9282 | } |
| 9283 | ], |
| 9284 | ""count"": 1 |
| 9285 | } |
| 9286 | ``` |
| 9287 | |
| 9288 | Optional query parameters: |
| 9289 | |
| 9290 | - `level` filters by exact log level, such as `INFO` or `ERROR`. |
| 9291 | - `limit` defaults to `100` in the public ASGI server. |
| 9292 | |
| 9293 | Logs are currently read from: |
| 9294 | |
| 9295 | ```text |
| 9296 | data/logs/{object_id}/log.tsv |
| 9297 | data/logs/{object_id}/log-*.tsv |
| 9298 | data/logs/{object_id}/log-*.tsv.gz |
| 9299 | ``` |
| 9300 | |
| 9301 | The TSV header defines the fields. The normal fields are `entry_id`, |
| 9302 | `timestamp`, `level`, and `message`; object code may add extra columns such as |
| 9303 | `method`, `status`, `duration_ms`, `error_type`, `error`, or `user_id`. |
| 9304 | |
| 9305 | The public ASGI server appends one execution log entry after each object method |
| 9306 | run. Successful runs use `level=DEBUG` and `status=success`; failed runs use |
| 9307 | `level=ERROR`, `status=error`, and include error fields. The endpoint itself |
| 9308 | remains read-only. |
| 9309 | |
| 9310 | Log storage may rotate and gzip old logs on disk. The HTTP response shape does |
| 9311 | not change; clients still receive JSON log entries. Transport compression for |
| 9312 | large log responses should be handled through normal HTTP compression, such as a |
| 9313 | reverse proxy honoring `Accept-Encoding`, rather than changing this JSON shape. |
| 9314 | |
| 9315 | ## Source Changes |
| 9316 | |
| 9317 | ```http |
| 9318 | GET /objects/{object_id}?source_changes=true&limit=100&offset=0 |
| 9319 | Authorization: Token <token> |
| 9320 | ``` |
| 9321 | |
| 9322 | Response: |
| 9323 | |
| 9324 | ```json |
| 9325 | { |
| 9326 | ""status"": ""ok"", |
| 9327 | ""object_id"": ""basics_counter"", |
| 9328 | ""changes"": [ |
| 9329 | { |
| 9330 | ""change_id"": ""123e4567-e89b-42d3-a456-426614174000"", |
| 9331 | ""timestamp"": ""2026-01-01T00:00:00"", |
| 9332 | ""object_id"": ""basics_counter"", |
| 9333 | ""action"": ""source_update"", |
| 9334 | ""version_id"": 2, |
| 9335 | ""from_version_id"": null, |
| 9336 | ""actor"": ""api"", |
| 9337 | ""message"": ""Updated via client"", |
| 9338 | ""correlation_id"": ""123e4567-e89b-42d3-a456-426614174001"", |
| 9339 | ""details"": {} |
| 9340 | } |
| 9341 | ], |
| 9342 | ""count"": 1, |
| 9343 | ""total"": 1, |
| 9344 | ""limit"": 100, |
| 9345 | ""offset"": 0, |
| 9346 | ""has_more"": false |
| 9347 | } |
| 9348 | ``` |
| 9349 | |
| 9350 | History is newest first and does not include source content. Use the versions |
| 9351 | endpoint to inspect or restore source snapshots. Source changes are the operator |
| 9352 | activity timeline for source edits and rollbacks. |
| 9353 | |
| 9354 | ## Object Changes |
| 9355 | |
| 9356 | ```http |
| 9357 | GET /objects/{object_id}?changes=true&kind=source&limit=100&offset=0 |
| 9358 | Authorization: Token <token> |
| 9359 | ``` |
| 9360 | |
| 9361 | This returns the normalized source/file timeline for one object. The admin |
| 9362 | inspection equivalent is: |
| 9363 | |
| 9364 | ```http |
| 9365 | GET /admin/objects/{object_id}?changes=true&limit=100&offset=0 |
| 9366 | Authorization: Token <token> |
| 9367 | ``` |
| 9368 | |
| 9369 | Response: |
| 9370 | |
| 9371 | ```json |
| 9372 | { |
| 9373 | ""status"": ""ok"", |
| 9374 | ""object_id"": ""basics_counter"", |
| 9375 | ""changes"": [], |
| 9376 | ""count"": 0, |
| 9377 | ""total"": 0, |
| 9378 | ""limit"": 100, |
| 9379 | ""offset"": 0, |
| 9380 | ""has_more"": false |
| 9381 | } |
| 9382 | ``` |
| 9383 | |
| 9384 | `kind` can be omitted or set to `source` or `file`. Use `/admin/changes` when a |
| 9385 | global activity stream needs record and package changes too. |
| 9386 | |
| 9387 | ## Versions |
| 9388 | |
| 9389 | ```http |
| 9390 | GET /objects/{object_id}?versions=true&limit=10 |
| 9391 | Authorization: Token <token> |
| 9392 | ``` |
| 9393 | |
| 9394 | Response: |
| 9395 | |
| 9396 | ```json |
| 9397 | { |
| 9398 | ""status"": ""ok"", |
| 9399 | ""object_id"": ""basics_counter"", |
| 9400 | ""versions"": [ |
| 9401 | { |
| 9402 | ""version_id"": 2, |
| 9403 | ""timestamp"": ""2026-01-01T00:00:00"", |
| 9404 | ""author"": ""api"", |
| 9405 | ""message"": ""Updated via client"", |
| 9406 | ""hash"": ""..."" |
| 9407 | } |
| 9408 | ], |
| 9409 | ""count"": 1 |
| 9410 | } |
| 9411 | ``` |
| 9412 | |
| 9413 | History is newest first and does not include source content. |
| 9414 | |
| 9415 | ## Specific Version |
| 9416 | |
| 9417 | ```http |
| 9418 | GET /objects/{object_id}?version=2 |
| 9419 | Authorization: Token <token> |
| 9420 | ``` |
| 9421 | |
| 9422 | Response: |
| 9423 | |
| 9424 | ```json |
| 9425 | { |
| 9426 | ""status"": ""ok"", |
| 9427 | ""object_id"": ""basics_counter"", |
| 9428 | ""version"": { |
| 9429 | ""version_id"": 2, |
| 9430 | ""timestamp"": ""2026-01-01T00:00:00"", |
| 9431 | ""author"": ""api"", |
| 9432 | ""message"": ""Updated via client"", |
| 9433 | ""hash"": ""..."", |
| 9434 | ""content"": ""def GET(request):\n return {\""status\"": \""ok\""}\n"" |
| 9435 | } |
| 9436 | } |
| 9437 | ``` |
| 9438 | |
| 9439 | ## Rollback |
| 9440 | |
| 9441 | ```http |
| 9442 | POST /objects/{object_id} |
| 9443 | Content-Type: application/json |
| 9444 | Authorization: Token <token> |
| 9445 | ``` |
| 9446 | |
| 9447 | Request: |
| 9448 | |
| 9449 | ```json |
| 9450 | { |
| 9451 | ""action"": ""rollback"", |
| 9452 | ""version_id"": 1, |
| 9453 | ""author"": ""api"", |
| 9454 | ""message"": ""Rollback to version 1"" |
| 9455 | } |
| 9456 | ``` |
| 9457 | |
| 9458 | Response: |
| 9459 | |
| 9460 | ```json |
| 9461 | { |
| 9462 | ""status"": ""ok"", |
| 9463 | ""message"": ""Rolled back to version 1"", |
| 9464 | ""version_id"": 1, |
| 9465 | ""new_version_id"": 3, |
| 9466 | ""object_id"": ""u_42_deals"" |
| 9467 | } |
| 9468 | ``` |
| 9469 | |
| 9470 | Rollback is non-destructive. The server should create a new latest version from |
| 9471 | the old source, write that source to the object file, and preserve history. |
| 9472 | |
| 9473 | The historical response reports the requested rollback version in `version_id`. |
| 9474 | Future responses may add `new_version_id`, but must not remove `version_id` |
| 9475 | without a client migration. |
| 9476 | |
| 9477 | The current public server supports `new_version_id` and uses the same temporary |
| 9478 | source-write gate as source updates: |
| 9479 | |
| 9480 | ```bash |
| 9481 | DBBASIC_ENABLE_SOURCE_WRITES=true |
| 9482 | DBBASIC_ADMIN_TOKEN=replace-with-a-local-dev-token |
| 9483 | ``` |
| 9484 | |
| 9485 | The token value shown here is a placeholder. Real deployments must generate a |
| 9486 | server-specific secret outside the source tree. Production code should replace |
| 9487 | that temporary gate with the real auth and permission system before rollback is |
| 9488 | exposed to users. |
| 9489 | |
| 9490 | ## Destroy Object |
| 9491 | |
| 9492 | ```http |
| 9493 | DELETE /objects/{object_id}?destroy=true |
| 9494 | Authorization: Token <token> |
| 9495 | ``` |
| 9496 | |
| 9497 | Response: |
| 9498 | |
| 9499 | ```json |
| 9500 | { |
| 9501 | ""status"": ""ok"", |
| 9502 | ""message"": ""Object destroyed: u_42_deals"", |
| 9503 | ""object_id"": ""u_42_deals"" |
| 9504 | } |
| 9505 | ``` |
| 9506 | |
| 9507 | Destructive delete must be authenticated and authorization-checked. |
| 9508 | |
| 9509 | ## Routing Compatibility |
| 9510 | |
| 9511 | The historical server supports explicit station routing: |
| 9512 | |
| 9513 | ```http |
| 9514 | GET /objects/{object_id}@{station_id} |
| 9515 | ``` |
| 9516 | |
| 9517 | The public v1 server does not need to promise distributed correctness, but code |
| 9518 | should not accidentally treat `@` routing as a normal object ID. If station |
| 9519 | routing is disabled, return a clear error instead of executing the wrong object. |
| 9520 | |
| 9521 | ## Compatibility Rules |
| 9522 | |
| 9523 | - Keep `/objects` and `/objects/{object_id}` as the main HTTP surface. |
| 9524 | - Keep query flags such as `source=true`, `state=true`, and `versions=true`. |
| 9525 | - Keep `PUT /objects/{object_id}?source=true` for source updates. |
| 9526 | - Keep rollback as `POST /objects/{object_id}` with `action=rollback`. |
| 9527 | - Keep top-level fields existing clients read: `status`, `error`, `object_id`, |
| 9528 | `objects`, `source`, `state`, `metadata`, `logs`, `versions`, `version`, |
| 9529 | `version_id`, `message`, and `count`. |
| 9530 | - Add fields only when old clients can ignore them safely. |
| 9531 | - Do not require clients to move to a new path such as `/api/v1` without a |
| 9532 | compatibility layer. |
| 9533 | " |
| 9534 | app-packages The App Suite — Applications as Installable Packages Platform & Extras 900 How DBBASIC's fifty-two built-in applications are built entirely from schema, permission rules, and an optional page object — with no server code required. "DBBASIC ships a suite of everyday applications. None of them required |
| 9535 | server code: each app is a package of **schema + permission rules + |
| 9536 | (optionally) one page object**. The server provides the primitives — |
| 9537 | records, permissions, search, sharing, files, AI — and apps are data |
| 9538 | that composes them. |
| 9539 | |
| 9540 | ```mermaid |
| 9541 | flowchart LR |
| 9542 | P[""Package""] --> S[""Schema<br/>fields, forms, views,<br/>search, transitions""] |
| 9543 | P --> R[""Permission rules<br/>owner rows, public flags,<br/>project sharing""] |
| 9544 | P --> O[""Page object<br/>(optional; talks to the<br/>records API as the visitor)""] |
| 9545 | S --> C[""Collection records""] |
| 9546 | R --> C |
| 9547 | O --> C |
| 9548 | C --> API[""Records API""] |
| 9549 | C --> MCP[""MCP tools""] |
| 9550 | C --> SRCH[""Global search""] |
| 9551 | C --> AI[""AI chat tools""] |
| 9552 | C --> UI[""Generated forms/lists<br/>(Scroll)""] |
| 9553 | ``` |
| 9554 | |
| 9555 | Because every surface reads the same schema and passes the same |
| 9556 | permission engine, installing an app instantly gives it: a validated |
| 9557 | records API, generated forms and lists in Scroll, global search, MCP |
| 9558 | tools for agents, AI chat access, and a live web page — with row-level |
| 9559 | visibility rules applied identically everywhere. |
| 9560 | |
| 9561 | Each app's page is also **realtime**: it subscribes to its collection |
| 9562 | over the shared `/ws` websocket and re-renders when a record it can see |
| 9563 | changes, so a note added in one tab (or by an agent) appears in another |
| 9564 | with no reload — one line per page, permission-filtered like everything |
| 9565 | else. See [design-system](/docs/design-system) and the |
| 9566 | [realtime contract](/docs/http-api-contract#realtime-push-websocket). |
| 9567 | |
| 9568 | ## The Suite |
| 9569 | |
| 9570 | Fifty-two packages. They are grouped here by what they are FOR rather than |
| 9571 | alphabetically, because the grouping is the argument: the money apps share |
| 9572 | one ledger and one composer, the commerce apps share one catalogue and one |
| 9573 | stock log, and nothing in either column needed server code to exist. |
| 9574 | |
| 9575 | ### The original slice — records, sharing, capture |
| 9576 | |
| 9577 | | Package | Collections | Page | Notable | |
| 9578 | |---|---|---|---| |
| 9579 | | `app-projects` | projects, project_access | `/projects` | The hub other apps relate to; self-serve sharing grants | |
| 9580 | | `app-notes` | notes | `/notes` | Public/private per note; permalinks; quick capture | |
| 9581 | | `app-tasks` | tasks | `/tasks` | Status lifecycle enforced by a `transitions` map; assignee access | |
| 9582 | | `app-contacts` | contacts, organizations, interactions, tags | `/contacts` | CRM; interactions use the `feed` list mode | |
| 9583 | | `app-articles` | articles | `/articles` | Publish flips one boolean; anonymous-readable blog | |
| 9584 | | `app-links` | links | `/links` | Bookmarks with tags | |
| 9585 | | `app-events` | events | `/calendar` | Calendar; purpose enum; project sharing | |
| 9586 | | `app-files` | files | `/files` | Uploads with quotas; downloads governed by the metadata record | |
| 9587 | | `app-templates` | templates | `/templates` | Reusable bodies; the thing `app-runner` executes | |
| 9588 | | `app-timers` | time_logs, rate_cards | `/time-logs` | Time tracking against tasks, and what an hour costs | |
| 9589 | | `app-forum` | forum_categories, forum_topics, forum_replies | `/forum` | Threaded discussion on the same permission engine | |
| 9590 | | `app-worker` | profiles, follows, profile_comments | `/profile/edit` | Public profiles and a follow graph | |
| 9591 | |
| 9592 | ### Money — one ledger, one composer, one set of books |
| 9593 | |
| 9594 | | Package | Collections | Page | Notable | |
| 9595 | |---|---|---|---| |
| 9596 | | `app-finance` | denominations, rates, fin_accounts, fin_journals, fin_journal_lines, fin_recurring, fin_closed_periods, expenses | `/accounts`, `/journals`, `/trial-balance`, `/statements`, `/expenses` | Double-entry: every other money app composes journals through one idempotent composer, provenance-stamped | |
| 9597 | | `app-invoices` | invoices, invoice_lines | `/invoices` | Line discounts and tax through the shared `object_lines` arithmetic | |
| 9598 | | `app-payments` | payments, refunds | `/payments` | Cash/accrual basis is configuration; bounces reverse rather than delete | |
| 9599 | | `app-billing` | wallets, wallet_entries, billing_plans, subscriptions, usage_events, usage_summaries | `/customer-funds` | Prepaid balances as a LIABILITY: a top-up is not revenue until the work is done, and the reconciliation proves it | |
| 9600 | | `app-banking` | value_accounts, value_account_counts, bank_import_profiles, bank_statement_imports, bank_lines | `/reconcile` | Statement import and matching — the anti-fraud keystone | |
| 9601 | | `app-entities` | entities | `/entities` | Multi-entity: separate books under one login | |
| 9602 | | `app-promotions` | promotions, promotion_redemptions | `/promotions` | Basket-level codes, rounded once, with every blocker returned at once | |
| 9603 | | `app-disputes` | disputes | `/disputes` | Chargebacks and what was said about them | |
| 9604 | | `app-integrity` | anchors | `/ledger-integrity` | A daily digest per money ledger, so a break can be bracketed to a day | |
| 9605 | | `app-notary` | notarizations | `/notary` | Store a digest, check it later — no custody of the document | |
| 9606 | |
| 9607 | ### Commerce — one catalogue, one stock ledger |
| 9608 | |
| 9609 | | Package | Collections | Page | Notable | |
| 9610 | |---|---|---|---| |
| 9611 | | `app-catalog` | products, locations, stock_moves, backorders, reorder_suggestions | `/products`, `/stock`, `/locations`, `/reorder`, `/backorders` | Stock levels are never stored, only folded from an immutable move log; a variant is a product | |
| 9612 | | `app-orders` | orders, order_lines | `/orders` | Line discounts, modifiers and tax through `object_lines` | |
| 9613 | | `app-shop` | carts, cart_items | `/shop` | Storefront and checkout: stock, price and slot are all re-checked at commit | |
| 9614 | | `app-shipping` | shipments, shipment_lines | `/shipments`, `/pick-list` | Partial fulfilment is ordinary business | |
| 9615 | | `app-receiving` | receipts, receipt_lines | — | Goods in, at invoice cost, feeding valuation | |
| 9616 | | `app-returns` | return_authorizations | `/returns` | The reverse trip, authorized before it arrives | |
| 9617 | | `app-pickup` | pickup_slots | `/pickup` | Collection windows generated before dawn, so the picker is never empty | |
| 9618 | | `app-kitchen` | — | `/kitchen` | The make-line view of open orders | |
| 9619 | |
| 9620 | ### AI, agents and jobs |
| 9621 | |
| 9622 | | Package | Collections | Page | Notable | |
| 9623 | |---|---|---|---| |
| 9624 | | `app-runner` | template_runs | `/runs` | The job engine: hold money at submission, execute or submit-and-poll, settle once. A run a provider still holds is never abandoned | |
| 9625 | | `app-agents` | agent_registry | `/agents` | Who is operating this server, and whether it and its own scheduled work are still answering | |
| 9626 | | `app-shell` | shell_preferences, shell_commands, ai_usage | `/shell`, `/talk` | Talk to everything; drop a file in and the model can read it | |
| 9627 | | `app-intake` | scans | `/scans` | A photographed receipt becomes a SUGGESTION, never a posting | |
| 9628 | | `app-documents` | sent_documents | `/documents` | One renderer for every business document | |
| 9629 | |
| 9630 | ### Platform surfaces — the uniform layers |
| 9631 | |
| 9632 | | Package | Collections | Page | Notable | |
| 9633 | |---|---|---|---| |
| 9634 | | `app-theme` | — | `/appearance` | The generative renderer: list, detail and form from any schema | |
| 9635 | | `app-views` | views | `/flow`, `/urls` | Pages as records — a dashboard is data, not code | |
| 9636 | | `app-nav` | nav_entries, attention_sources, attention_counts | `/` | The home page and every app's ""needs a human"" count | |
| 9637 | | `app-settings` | user_prefs, feature_flags, ai_prices, app_settings, unit_prices | — | Server config, per-user prefs, and the price tables | |
| 9638 | | `app-share` | record_shares | — | Per-record grants that can only ever narrow | |
| 9639 | | `app-notify` | notify_rules | — | Declarative event → notification | |
| 9640 | | `app-rollup` | rollup_definitions | — | Derived aggregates, defined as data | |
| 9641 | | `app-materialize` | materialize_definitions | — | Scheduled generation of records from a definition | |
| 9642 | | `app-thread` | thread_comments | — | The comments capability, for any collection | |
| 9643 | | `app-collab` | task_comments, feed_posts, notifications | — | Comments, the human+agent coordination feed, notifications | |
| 9644 | | `app-feed` | — | `/feed` | The composed activity stream | |
| 9645 | | `app-activity` | — | `/activity` | The change log, rendered | |
| 9646 | | `app-dashboard` | — | `/dashboard` | Counts and queues at a glance | |
| 9647 | | `app-messaging` | message_threads, messages, message_recipients, message_drafts | `/inbox` | Internal mail on the same permission engine | |
| 9648 | | `app-email` | email_outbox | — | Outbound mail as records, sent by a pass | |
| 9649 | | `app-analytics` | page_views, conversions | `/analytics`, `/visitors` | Visitors, bots and us, counted apart | |
| 9650 | | `app-privacy` | — | `/privacy` | What is collected and why, as a page | |
| 9651 | |
| 9652 | Packages without a page are managed entirely through generated UIs, |
| 9653 | the shell, and MCP — proof that the schema contract carries a whole |
| 9654 | app's interface. |
| 9655 | |
| 9656 | ## The Pattern |
| 9657 | |
| 9658 | Every app repeats the same shape. To build a new one, copy `app-notes` |
| 9659 | and follow [package authoring](/docs/package-authoring): |
| 9660 | |
| 9661 | 1. **Schema** — fields with semantics (`docs/schema-forms.md`): |
| 9662 | types, enums, relations, validation, `search.fields`, |
| 9663 | `views.list_mode`, and `transitions` for lifecycle fields. |
| 9664 | 2. **Rules** — the standard grants, composed per record: |
| 9665 | - owner rows: `{""row_filter"": {""owner_id"": ""$user_id""}}` |
| 9666 | - public rows: `{""principal"": ""public"", ""row_filter"": {""is_public"": ""true""}}` |
| 9667 | - project sharing: `{""row_filter"": {""project_id"": ""$accessible_projects""}}` |
| 9668 | - a public `execute` grant for the page object, which renders a |
| 9669 | sign-in prompt to anonymous visitors |
| 9670 | 3. **Page object** — one Python file returning HTML; its JavaScript |
| 9671 | talks to `/collections/{name}/records` and `/api/search` with the |
| 9672 | visitor's session cookie, so the page holds no data access of its |
| 9673 | own. |
| 9674 | |
| 9675 | Visibility rules compose per record: a note can be readable because |
| 9676 | you own it, because it is public, or because it sits in a project |
| 9677 | shared with you — three rules, no visibility code in the app. |
| 9678 | |
| 9679 | ## Relations Between Apps |
| 9680 | |
| 9681 | Apps point at each other with validated `relation` fields, not an |
| 9682 | association framework: tasks and notes relate to projects, interactions |
| 9683 | to contacts, time logs to tasks, comments to tasks. A relation is a |
| 9684 | pointer plus a display hint; writes validate that the target record |
| 9685 | exists, and nothing joins, cascades, or lazy-loads. |
| 9686 | " |
| 9687 | analytics Analytics — First-Party, Server-Side, and Honest About It Platform & Extras 910 How the object server's built-in, first-party analytics works, what it can see that JavaScript tags can't, and exactly what it costs in accuracy and consent. "Most platforms forgot analytics. You build a site, and to find out |
| 9688 | whether anyone came you paste in someone else's JavaScript, which reports |
| 9689 | a number you cannot check, derived from data you no longer solely hold. |
| 9690 | |
| 9691 | This server records its own traffic. Every request appends one row to |
| 9692 | `page_views`; `/visitors` folds those rows into people, bots and your own |
| 9693 | operations, counted apart. No third party is involved, no script is |
| 9694 | served to the browser, and the data is a TSV on your disk that you can |
| 9695 | open with `less`. |
| 9696 | |
| 9697 | This page argues why that is better, and states plainly what it costs — |
| 9698 | because it does cost something, and a page that only listed advantages |
| 9699 | would be exactly the kind of analytics writing this system exists in |
| 9700 | opposition to. |
| 9701 | |
| 9702 | ## The structural argument: JavaScript can only see what ran JavaScript |
| 9703 | |
| 9704 | This is the whole thing, and it is not a matter of degree. |
| 9705 | |
| 9706 | A tag-based analytics product observes a page view when a script loads |
| 9707 | and executes. It therefore cannot see: |
| 9708 | |
| 9709 | - **Ad-blocked visitors.** The most technical part of any audience blocks |
| 9710 | the most common tags by default. For developer tools, that is precisely |
| 9711 | the people evaluating you. |
| 9712 | - **Clients with no JavaScript** — curl, RSS readers, link previewers, |
| 9713 | text browsers, accessibility tooling, anything scripted. |
| 9714 | - **Bots and scanners.** Invisible to the tag, and they are a large share |
| 9715 | of what actually arrives. |
| 9716 | - **Errors.** A 404, a 500, a page that failed before the tag ran. The |
| 9717 | requests most worth knowing about are the ones least likely to report |
| 9718 | themselves. |
| 9719 | - **Anything that is not a page.** API calls, webhooks, feeds, downloads. |
| 9720 | |
| 9721 | The server, by definition, sees every request it answered. It cannot be |
| 9722 | blocked from knowing what it did. |
| 9723 | |
| 9724 | ## The ownership argument |
| 9725 | |
| 9726 | A hosted analytics vendor holds a complete copy of your traffic and |
| 9727 | returns you a summary, subject to sampling, thresholds, and categories |
| 9728 | like ""not provided"". You are shown a fraction of your own data, and the |
| 9729 | fraction is chosen by the party who kept all of it. |
| 9730 | |
| 9731 | Here the raw rows are yours, on your disk, in a format `grep` can read. |
| 9732 | Nothing is sampled. Nothing is withheld. No third party receives a copy |
| 9733 | of your visitors' requests as a side effect of you wanting to count them, |
| 9734 | which is also the part your visitors would care about if anybody asked |
| 9735 | them. |
| 9736 | |
| 9737 | ## What it captures that a tag cannot |
| 9738 | |
| 9739 | Because it sits at the server, the same data answers questions analytics |
| 9740 | products usually cannot: |
| 9741 | |
| 9742 | - **Attack traffic.** Scanners walking a list of URLs that do not exist |
| 9743 | here, credential-stuffing attempts, payloads arriving in headers. One |
| 9744 | Log4Shell probe with an AWS-credential exfiltration template in its |
| 9745 | `Referer` was found this way — inert against a Python server, but worth |
| 9746 | seeing. |
| 9747 | - **Which of your own machines did what.** Deploys and scripted calls are |
| 9748 | labelled `operator`, not hidden, so you can tell your traffic from |
| 9749 | everyone else's rather than guessing. |
| 9750 | - **Failure shape.** A flood of 404s from one address is a security |
| 9751 | signal that a page-view counter would never show you. |
| 9752 | - **Multi-site truth.** One process may answer for several hostnames; the |
| 9753 | `host` is recorded, so ""did anyone read the pitch"" and ""did anyone try |
| 9754 | the product"" are different questions with different answers. |
| 9755 | |
| 9756 | ## Who counts as a person |
| 9757 | |
| 9758 | The hard part of first-party analytics is not collection, it is |
| 9759 | classification. Traffic is sorted into three kinds and never merged: |
| 9760 | |
| 9761 | **operator** — yours. An owner IP, or a request carrying the admin token. |
| 9762 | Labelled rather than hidden: seeing your own visit is how you confirm the |
| 9763 | page is recording anything at all, and a counter that silently drops the |
| 9764 | only traffic you can verify is a counter you cannot trust. |
| 9765 | |
| 9766 | **bot** — a client that never behaved like a person. The test is |
| 9767 | behavioural, not a user-agent blocklist, because the blocklist is always |
| 9768 | out of date and the behaviour is not. An address must have loaded a real |
| 9769 | page **and** done one thing a prober does not: opened a second distinct |
| 9770 | page, arrived from a referrer, or carried a session. A declared crawler |
| 9771 | stays a bot even when it fetches something real — Googlebot is honest |
| 9772 | about what it is, and honesty should not promote it. So does anything |
| 9773 | sending an attack payload in a header. |
| 9774 | |
| 9775 | **visitor** — everything else. A person, probably. |
| 9776 | |
| 9777 | The rule earned its strictness. A first version counted any address that |
| 9778 | loaded a real page, and `/` answers 200 to everyone: on a real week that |
| 9779 | promoted nearly half the ""visitors"" — 187 fell to 92 — from probers who |
| 9780 | touched the front page and left. A metric that turns a quiet day into a |
| 9781 | busy one is worse than no metric, because it misleads exactly when |
| 9782 | somebody is relying on it. |
| 9783 | |
| 9784 | ## What it costs — read this part |
| 9785 | |
| 9786 | **""Unique"" means a distinct IP address, and an IP is not a person.** An |
| 9787 | office behind one connection counts once. A phone moving between wifi and |
| 9788 | cellular counts twice. A CGNAT range counts thousands of people as a |
| 9789 | handful. |
| 9790 | |
| 9791 | That is the honest unit available to a server that sets no *cross-site* |
| 9792 | cookie and fingerprints nobody, and the alternative — a durable |
| 9793 | cross-site identifier — is a thing this system deliberately does not |
| 9794 | have. The `/visitors` page says so on the page, in words, rather than |
| 9795 | presenting a confident number. |
| 9796 | |
| 9797 | A first-party visitor cookie (below) improves this for anyone who accepts |
| 9798 | one, and does not change the ceiling: it makes returning visitors |
| 9799 | countable, and it still cannot tell a phone from a laptop. |
| 9800 | |
| 9801 | Also absent, and not coming: |
| 9802 | |
| 9803 | - **No cross-device identity.** The same person on a laptop and a phone |
| 9804 | is two visitors. |
| 9805 | - **No demographics, interests, or inferred attributes.** Those come from |
| 9806 | a surveillance network; there isn't one. |
| 9807 | - **No client-side interaction data** — scroll depth, rage clicks, time |
| 9808 | on page — unless an app deliberately posts an event. The server knows |
| 9809 | what it served, not what someone did with it afterwards. |
| 9810 | - **Bounded history.** `page_views` is capped by age *and* row count |
| 9811 | (`DBBASIC_ANALYTICS_RETENTION_DAYS`, `DBBASIC_ANALYTICS_MAX_ROWS`), |
| 9812 | because a log nobody told how large it may get is a log that fails on |
| 9813 | the worst day rather than an ordinary one. Long-horizon trends need |
| 9814 | rolling those up before they age out. |
| 9815 | |
| 9816 | If you need cross-device attribution and demographic segments, this will |
| 9817 | not give them to you, and no amount of configuration will. |
| 9818 | |
| 9819 | ## Operating it |
| 9820 | |
| 9821 | - **Off by default.** `DBBASIC_ANALYTICS=on` is an operator choice, |
| 9822 | because it adds a write to every request. |
| 9823 | - **The visitor cookie is off by default and switched on separately** |
| 9824 | (`DBBASIC_ANALYTICS_VISITOR_COOKIE=on`), because it is the one thing |
| 9825 | here that stores an identifier on a visitor's device. See the cookie |
| 9826 | section below for what that commits you to; `/privacy` states it for |
| 9827 | your visitors, folded from these same settings. |
| 9828 | - Asset, health and polling paths are skipped; 4xx and 5xx are |
| 9829 | deliberately **kept**, because a 404 flood is the signal. |
| 9830 | - The write is offloaded off the event loop and is best-effort: |
| 9831 | analytics must never break a response. |
| 9832 | - Retention runs as a scheduled daemon pass with both bounds. |
| 9833 | - Rollups (`rollup_definitions`) group the raw rows into top paths, top |
| 9834 | addresses, status codes and daily counts. Set `min_group_size` on the |
| 9835 | path and address rollups — without it, every one-hit scanner probe |
| 9836 | becomes its own row and the ""top paths"" report becomes a copy of the |
| 9837 | log rather than a summary of it. |
| 9838 | |
| 9839 | ## Goals and funnels |
| 9840 | |
| 9841 | `conversions` is a collection: `event_type`, optional |
| 9842 | `session_id`/`user_id`, and a JSON `metadata` blob, append storage, |
| 9843 | written server-side by whichever app owns the goal. A rollup can group it |
| 9844 | by `event_type`. |
| 9845 | |
| 9846 | **It is written now, and the implementation was as small as promised** — |
| 9847 | because this system is full of transitions and a conversion is simply a |
| 9848 | transition worth counting. `system_record_conversion` (app-analytics) |
| 9849 | declares `HANDLES` on the changes that already happen and records four |
| 9850 | goals: |
| 9851 | |
| 9852 | | event_type | the transition | |
| 9853 | |---|---| |
| 9854 | | `order_confirmed` | a sale order reached `confirmed` or anywhere past it | |
| 9855 | | `order_collected` | a pickup order was actually handed over | |
| 9856 | | `payment_received` | money arrived against an invoice, and did not bounce | |
| 9857 | | `scan_confirmed` | a human confirmed a scanned document into the books | |
| 9858 | |
| 9859 | Nothing new is observed. Four facts the server already knew simply never |
| 9860 | landed anywhere a report could see them. |
| 9861 | |
| 9862 | It is a **reaction**, in the same posture as `system_order_email`: |
| 9863 | post-commit, best-effort, never a gate, and it cannot fail the write that |
| 9864 | confirmed the order. On a box with no app-analytics installed it counts |
| 9865 | nothing and says which — because silence would read as ""already counted"", |
| 9866 | and the shop would never find out it had no numbers. |
| 9867 | |
| 9868 | **Idempotent by provenance.** The change dispatcher is at-least-once, and |
| 9869 | `status is confirmed` is a state an order *sits in* rather than an edge it |
| 9870 | crosses, so the handler sees the same order on every later write. Each |
| 9871 | conversion carries its source in the metadata blob |
| 9872 | (`{""source"": ""orders/ord-1""}`) and one whose `(event_type, source)` pair |
| 9873 | already exists is not written again. A double-counted conversion is a |
| 9874 | permanent overcount in a report nothing downstream will ever correct. |
| 9875 | |
| 9876 | **These four rows carry no `session_id`, and that is stated rather than |
| 9877 | hidden.** A back-office transition has no browser anywhere near it — an |
| 9878 | order confirmed by staff the next morning was not a page view — and the |
| 9879 | basket's `carts.session_token` is a different identifier in a different |
| 9880 | namespace, so stamping it here would merge two populations and produce a |
| 9881 | funnel that looks stitched and is not. The funnel counts them in |
| 9882 | `unthreaded_conversions` and says so on the page. The field is there for |
| 9883 | the day an app records a goal from a request that did carry the cookie. |
| 9884 | |
| 9885 | ### The folds |
| 9886 | |
| 9887 | `object_conversions.py` is the pure half — rows in, numbers out, no I/O, |
| 9888 | the same posture as `object_visitors.py`. Three folds, each of which |
| 9889 | returns its own caveat **as data** so a surface cannot render the number |
| 9890 | without being handed the qualification to render beside it: |
| 9891 | |
| 9892 | - **`funnel(page_views, conversions, steps)`** — ordered steps, each a |
| 9893 | path prefix (`/shop`) or an `event_type`. Per step: distinct visitors |
| 9894 | who reached it and drop-off from the step before. It is a *sequence* — |
| 9895 | a visitor counts at step N only if they reached N−1 and then did N |
| 9896 | afterwards, because somebody who bought last week and browsed today did |
| 9897 | not convert today. Threaded by `session_id`, falling back to the IP |
| 9898 | address for rows that carry none, and the result reports |
| 9899 | `ip_stitched` / `ip_stitched_pct` per step and overall. A funnel that |
| 9900 | hides how much of itself is guesswork is the kind of analytics this |
| 9901 | page exists in opposition to. |
| 9902 | - **`returning_visitors(page_views, window_days)`** — new versus |
| 9903 | returning by token, returning `floor: True` and the caveat text itself. |
| 9904 | See the ceiling below. |
| 9905 | - **`time_to_conversion(page_views, conversions)`** — median and spread of |
| 9906 | days between a visitor's first recorded page and their conversion, with |
| 9907 | the two ways of not knowing reported apart: `unthreaded` (no token at |
| 9908 | all) and `no_first_view` (the first visit aged out of a bounded log). |
| 9909 | Dropping the second silently would bias the median toward fast |
| 9910 | conversions — the number would improve every time retention was |
| 9911 | shortened, which is the most dangerous shape a metric can have. |
| 9912 | |
| 9913 | ### Describing your own funnel |
| 9914 | |
| 9915 | A funnel is a settings row, not code. `analytics.funnel_steps` in |
| 9916 | `app_settings` is a JSON list: |
| 9917 | |
| 9918 | ```json |
| 9919 | [""/shop"", ""/checkout"", ""order_confirmed""] |
| 9920 | ``` |
| 9921 | |
| 9922 | A step beginning with `/` is a path prefix; anything else is a conversion |
| 9923 | `event_type`. The long form takes labels: |
| 9924 | `[{""label"": ""Browsed"", ""path"": ""/shop""}, …]`. With nothing configured, |
| 9925 | `/visitors` shows what to configure — built from the paths and event types |
| 9926 | this server has actually recorded, so it can be copied rather than |
| 9927 | invented — instead of an empty table that reads as ""nobody converted"". A |
| 9928 | malformed setting is reported as a malformed setting, because a |
| 9929 | misconfigured funnel and a funnel nobody entered look identical and only |
| 9930 | one of them is your fault. |
| 9931 | |
| 9932 | ## The visitor cookie — the one addition worth making, and its limits |
| 9933 | |
| 9934 | Everything above is what a server can know without asking the browser to |
| 9935 | remember anything. There is exactly one addition that changes the class |
| 9936 | of question this can answer, and it deserves stating carefully, because |
| 9937 | it is also the point at which analytics products usually stop being |
| 9938 | honest. |
| 9939 | |
| 9940 | Before it, `page_views.session_id` read a cookie that nothing ever set, |
| 9941 | so every visitor looked new on every visit and a journey could only be |
| 9942 | stitched by address — precisely wrong for the multi-visit paths that |
| 9943 | matter. Somebody reads the pitch on Monday, comes back Thursday, and |
| 9944 | tries the product: that was three strangers. |
| 9945 | |
| 9946 | **A first-party cookie with a long-but-not-indefinite expiry fixes it.** |
| 9947 | It is `dbbasic_visitor`, set by `object_server` on a page response when |
| 9948 | the cookie is switched on, analytics is on, and the request carried none; |
| 9949 | its value is stamped into `page_views.session_id` on the same request that |
| 9950 | mints it — so the first page of a visit is threaded rather than the |
| 9951 | second. It unlocks the questions a site actually has: |
| 9952 | |
| 9953 | - **New versus returning** — the most useful ratio a site has, and |
| 9954 | invisible without it. |
| 9955 | - **Return paths** — what somebody read first, and what brought them |
| 9956 | back. |
| 9957 | - **Time to conversion** — days between first visit and purchase, which |
| 9958 | is what tells you whether your funnel is a funnel or a queue. |
| 9959 | - **Real funnels**, stitched across visits rather than within one. |
| 9960 | |
| 9961 | ### It is OFF by default, and that is the whole consent posture |
| 9962 | |
| 9963 | `DBBASIC_ANALYTICS_VISITOR_COOKIE` gates it, and unset — the default — |
| 9964 | **no visitor cookie is set at all**. The server behaves exactly as it did |
| 9965 | before the cookie existed: visitors counted by address, nothing written |
| 9966 | to anybody's device, no ePrivacy consent trigger, no banner needed |
| 9967 | anywhere in the world. |
| 9968 | |
| 9969 | That is deliberate, and the reason is worth stating because it is the |
| 9970 | whole design. The consent rule people call ""the cookie law"" is the |
| 9971 | ePrivacy Directive, not GDPR, and it fires on *storing or reading |
| 9972 | something on a device that is not strictly necessary for the service the |
| 9973 | user asked for*. Everything else this server puts in a browser — |
| 9974 | `dbbasic_session`, the shop's `cart` — is strictly necessary for a thing |
| 9975 | the person actively asked for, and `page_views` is server-side logging |
| 9976 | rather than device storage. **One cookie is the entire consent problem**, |
| 9977 | so the default that needs the least law is the one where it does not |
| 9978 | exist. |
| 9979 | |
| 9980 | Turning it on is a deliberate operator act, and *that* is the moment the |
| 9981 | obligation appears. So the obligation travels with the setting rather |
| 9982 | than living only here: the daemon's and the server's boot lines say, in |
| 9983 | words, that it ""stores an identifier on a visitor's device; in the EU/UK |
| 9984 | that requires consent"" (`object_analytics.OBLIGATION`), and `/privacy` |
| 9985 | grows a row in its cookie table with the real `Max-Age` the moment it is |
| 9986 | switched on. |
| 9987 | |
| 9988 | It is a separate switch from `DBBASIC_ANALYTICS` on purpose. Recording |
| 9989 | what the server did and asking a browser to remember who it is are |
| 9990 | different acts under different instruments, and one flag for both would |
| 9991 | mean an operator who wanted traffic numbers could not have them without |
| 9992 | also taking on a consent obligation. With the cookie off, everything on |
| 9993 | `/visitors` still works except new-versus-returning — which is what |
| 9994 | Plausible and Fathom sell as a feature. |
| 9995 | |
| 9996 | DNT and Sec-GPC suppression is unchanged and applies on top: switching |
| 9997 | the cookie on does not switch those off. |
| 9998 | |
| 9999 | ### The five rules, for a box that switches it on |
| 10000 | |
| 10001 | The rules that keep it honest are not optional, and none of them is left |
| 10002 | to judgement: each is enforced in code and pinned by a test in |
| 10003 | `tests/test_conversions.py`, because a rule that lives only in a document |
| 10004 | is a rule the next person breaks in one line without noticing. |
| 10005 | |
| 10006 | 1. **First-party and same-site only.** Set by this server, sent only to |
| 10007 | this server, `Path=/`, `SameSite=Lax`, `HttpOnly`. Not readable by |
| 10008 | JavaScript, not a cross-site identifier — the same posture the shop's |
| 10009 | basket cookie already takes, and the reason page objects are never |
| 10010 | handed the identity session cookie. It is also withheld from page |
| 10011 | objects themselves (`_object_visible_cookies` drops it alongside |
| 10012 | `dbbasic_session`): they have no use for it, since the server stamps |
| 10013 | it into `page_views` itself, and a token only the capture hook can see |
| 10014 | cannot be joined to anything by a package the operator installed. |
| 10015 | 2. **An opaque random token.** `secrets.token_urlsafe`. It says *this |
| 10016 | browser has been here before*. It carries no name, no email, no |
| 10017 | account, and no meaning outside this site's own logs — and nothing |
| 10018 | derived from the request, because a token hashed from IP and user |
| 10019 | agent would be a stable identifier nobody could ever clear. |
| 10020 | 3. **A stated expiry.** `DBBASIC_ANALYTICS_VISITOR_DAYS`, default 180 — |
| 10021 | long enough to see a return visit, short enough to lapse rather than |
| 10022 | follow somebody for years. Junk and non-positive values fall back to |
| 10023 | the default rather than being read as unbounded: ""indefinite"" is how a |
| 10024 | session identifier quietly becomes a permanent one. |
| 10025 | 4. **It never becomes identity.** A signed-in `user_id` is already |
| 10026 | recorded separately, and the visitor token must not be joined to it to |
| 10027 | retroactively de-anonymise earlier browsing. That join is the exact |
| 10028 | move that turns analytics into surveillance, and it is one line of |
| 10029 | code away at all times — so `build_page_view` and `build_conversion` |
| 10030 | write **at most one of the two columns**, preferring the anonymous |
| 10031 | one. A signed-in member's traffic is still counted; it simply is not |
| 10032 | labelled with who they are. Where there is no token (a DNT visitor, an |
| 10033 | API call) `user_id` is written as before, because there is nothing for |
| 10034 | it to be correlated with. |
| 10035 | 5. **Honoured refusal.** `DNT: 1` and `Sec-GPC: 1` mean no cookie is set — |
| 10036 | not a shorter one, not an anonymised one. Those visitors are still |
| 10037 | **counted** by the IP rule, because refusing to be remembered is not |
| 10038 | refusing to be counted; they simply appear new each visit, which is the |
| 10039 | correct consequence of having asked not to be remembered. `DNT: 0` is |
| 10040 | consent and an absent header is no statement at all; reading either as |
| 10041 | a refusal would mean this server never remembered anybody. |
| 10042 | |
| 10043 | It is also never set on a path a browser did not choose to visit: assets, |
| 10044 | health checks and polling endpoints (`should_capture`) and the API and |
| 10045 | collection surfaces a script talks to (`is_page_path`). An API client is |
| 10046 | not a browser and cannot be a returning visitor; handing it a cookie only |
| 10047 | writes an identifier into somebody's cron job. A 404 *is* a page — often |
| 10048 | the first one a visitor sees — so those are threaded like any other. |
| 10049 | |
| 10050 | The ceiling worth knowing: a cleared cookie is a new visitor, a second |
| 10051 | browser is a second visitor, and a phone is a third. Returning-visitor |
| 10052 | counts are a **floor, never a census** — they undercount, which is the |
| 10053 | right direction for a number to be wrong in. `/visitors` says so on the |
| 10054 | page, next to the number, and reports how many addresses had no cookie at |
| 10055 | all so the size of the blind spot is on the same screen as the ratio it |
| 10056 | is missing from. |
| 10057 | |
| 10058 | Funnels are then an ordinary fold over `page_views` grouped by that |
| 10059 | token — `object_conversions.funnel` — and `conversions.session_id` is the |
| 10060 | field waiting for the goals that are recorded from a real request. |
| 10061 | |
| 10062 | ## Where to look |
| 10063 | |
| 10064 | - `/visitors` — people, bots and your own traffic, by hour and by day, |
| 10065 | with referrers, landing pages and a per-host split; then new versus |
| 10066 | returning, goals by `event_type`, your configured funnel and time to |
| 10067 | conversion. Deliberately one page rather than a second `/funnels`: ""how |
| 10068 | many came"", ""how many had been before"" and ""how many bought"" are one |
| 10069 | question asked at increasing depth, and a conversion rate whose |
| 10070 | denominator lives on another screen is a conversion rate nobody |
| 10071 | computes. |
| 10072 | - `/analytics` — path, status and referrer detail from the rollups. |
| 10073 | - `/privacy` — what visitors are told, folded from these same settings by |
| 10074 | `app-privacy`, so the retention on this page and the retention on that |
| 10075 | one are the same number read from the same place. `/cookies` is its |
| 10076 | cookie table alone. |
| 10077 | - `object_analytics.py` — capture, configuration and the visitor-cookie |
| 10078 | decision. |
| 10079 | - `object_visitors.py` — the classifier and the traffic fold, pure and |
| 10080 | tested. |
| 10081 | - `object_conversions.py` — the goal record, the funnel, new-versus- |
| 10082 | returning and time-to-conversion, pure and tested. |
| 10083 | - `packages/app-analytics/objects/system/record_conversion.py` — the |
| 10084 | handler that turns transitions into `conversions` rows, idempotently." |
| 10085 | comparisons Comparisons: What DBBASIC Deletes, and What That Costs Platform & Extras 920 An honest comparison of DBBASIC against Django/Rails, Kubernetes, JS meta-frameworks, SaaS suites, and metered AI platforms — what each one makes you carry, and what deleting it costs. "Rails announced itself by giving developers an enemy: enterprise |
| 10086 | ceremony. It was right, and it won. Twenty years later the ceremony |
| 10087 | grew back in new clothes — deploy pipelines for one-line changes, |
| 10088 | migration rituals, SaaS rent on your own data, and metered AI where |
| 10089 | every question pays a vendor. |
| 10090 | |
| 10091 | This page compares honestly. Every framework here is good software |
| 10092 | built by serious people; the differences are about what each one makes |
| 10093 | you carry. Where DBBASIC deletes something, the cost of the deletion is |
| 10094 | stated too. Claims about speed and effort come with receipts from this |
| 10095 | project's own history. |
| 10096 | |
| 10097 | ## The one-line version of each |
| 10098 | |
| 10099 | | Stack | What you carry | What DBBASIC deletes | What deleting costs | |
| 10100 | |---|---|---|---| |
| 10101 | | Django / Rails | ORM, migrations, deploys, middleware, signals, app server + DB | All of it: records are files, schemas are live data, changes serve on the next request | No relational joins; big-data queries need indexes you don't have yet | |
| 10102 | | Enterprise / k8s | Clusters, YAML, service meshes, CI/CD, staging ladders | One VM, one process, systemd | No horizontal scaling story; deliberate single-box boundary | |
| 10103 | | JS meta-frameworks | Build steps, bundlers, hydration, node_modules | Pages are one Python file emitting HTML + small fetch calls | No component ecosystem; you write your own widgets or generate them from schemas | |
| 10104 | | SaaS suites (Basecamp, Notion, Airtable) | Monthly rent, their servers, their data model, their API limits | Self-hosted; your disk, your TSVs, greppable | You are the operator: backups and TLS are your (automated) chores | |
| 10105 | | No-code platforms | A ceiling: when the builder can't express it, you're stuck | Schemas generate the easy 90%; real Python objects catch the rest | Objects are code; someone (or some AI) still writes them | |
| 10106 | | Metered AI platforms | Per-seat AI pricing, vendor lock, their tools only | Bring your own key, pick any model, hand it your tools, audited as you | You pay your provider directly; no bundled ""free"" AI | |
| 10107 | | Hosted analytics (GA & friends) | A tag in every page, a vendor's complete copy of your traffic, and a sampled summary back | The server counts what it answered; raw rows are a TSV on your disk | No cross-device identity, no demographics, no client-side interaction data; ""unique"" means a distinct IP | |
| 10108 | |
| 10109 | ## Against the ceremony, specifically |
| 10110 | |
| 10111 | **Migrations.** Django asks: edit models.py, generate a migration, |
| 10112 | review it, run it in each environment, in order, forever. DBBASIC asks: |
| 10113 | PUT the schema; it versions itself; rollback is one call. The cost: |
| 10114 | no schema-enforced relational integrity beyond `relation` existence |
| 10115 | checks — the engine validates pointers, not cascades. That is a |
| 10116 | stated design position, not an accident. |
| 10117 | |
| 10118 | **Deploys.** The receipt: during one working day this project |
| 10119 | shipped twelve application packages, an AI runtime, file storage, and |
| 10120 | a sharing model to a live server — the home page changed four times, |
| 10121 | one page object shipped five times, and the only restarts were for the |
| 10122 | core server file, each a few seconds. Every change was versioned and |
| 10123 | reversible. The equivalent Django day is spent watching pipelines. |
| 10124 | |
| 10125 | **Middleware and signals.** A large Django application at scale can |
| 10126 | easily run six or more middleware layers and hundreds of lines of |
| 10127 | signal handlers on every write. Here a request is a router match, a |
| 10128 | cached file read, permission string checks, JSON out. That is most of |
| 10129 | why a small Python file server outruns a ""more powerful"" stack: the |
| 10130 | fastest layer is the one that is not there. |
| 10131 | |
| 10132 | **""But will it scale?""** Rails answered this correctly in 2005 and we |
| 10133 | repeat it: optimize for the loop first. DBBASIC adds a number — TSV |
| 10134 | scans in page cache are instant to roughly hundreds of thousands of |
| 10135 | rows per collection, which is beyond most personal and team software, |
| 10136 | and per-collection files mean a hot collection can gain an index later |
| 10137 | without a platform rewrite. |
| 10138 | |
| 10139 | ## What Rails-era slogans become here |
| 10140 | |
| 10141 | - *Convention over configuration* → **Schemas declare semantics, |
| 10142 | never widgets.** One declaration drives forms, tables, validation, |
| 10143 | search, AI tools, and permissions on every surface. |
| 10144 | - *Don't repeat yourself* → **One permission engine, every surface.** |
| 10145 | A row filter written once binds the web page, the API, search, |
| 10146 | files, MCP agents, and AI tool calls identically. |
| 10147 | - *Less software* → **Apps are data.** Eleven of twelve suite apps |
| 10148 | contain zero server code; several contain no code at all. |
| 10149 | - *Optimize for programmer happiness* → **Optimize for the loop.** |
| 10150 | Edit → live → inspect → undo, in seconds, with history. For humans |
| 10151 | and for AI agents, which compounds it: the machine that writes the |
| 10152 | code is also freed from the ceremony. |
| 10153 | - *Opinionated software* → unchanged, and still true. If you want |
| 10154 | microservices, joins, or a component framework, this is honestly not |
| 10155 | your server. |
| 10156 | |
| 10157 | ## The new argument Rails could not make |
| 10158 | |
| 10159 | Rails freed the developer from ceremony. DBBASIC also frees the |
| 10160 | *software* from rent. A person with a small VM gets the suite, the |
| 10161 | data in files they can read, an AI operator that uses their own key |
| 10162 | and obeys their own permissions, and an MCP surface any agent can |
| 10163 | drive — MIT licensed, small enough to read end to end. The enemy is |
| 10164 | not another framework. It is the idea that your own tools should bill |
| 10165 | you monthly to change." |
| 10166 | logic-decisions Business Logic Decisions Design Decisions 1000 A living log of where business-logic rules go and why, covering doctrine on stamping vs. deriving values, time-driven state, immutable money, and integer-denomination amounts. "The living log of *where rules go and why* — the companion to `business-logic-patterns.md` (the placement guide) the same way `ui-decisions.md` companions the generative renderers. Because the primitives are shared, a placement decision here is not a per-module choice — it applies to every module at once, and an inconsistent placement is a bug, not a style preference. |
| 10167 | |
| 10168 | Format per decision: **Decision · Rationale · Applies to · Status.** |
| 10169 | |
| 10170 | ## Decided |
| 10171 | |
| 10172 | ### 1. Stamp point-in-time facts; derive live facts. |
| 10173 | |
| 10174 | - **Decision:** A value that is a fact about a *moment* (the tax rate an invoice was issued under, the price a line was sold at, the owner who created a record) is **stamped into the record at write time** — by the server or a pre-write hook — and never recomputed from today's reference data. A value that should always reflect *current* stored state (an invoice total, a payment balance, a full name) is **derived** by a formula/rollup and recomputes when inputs change. |
| 10175 | - **Rationale:** Reference data changes; history must not. Deriving yesterday's invoice tax from today's rate table silently rewrites the past — the classic sales-tax bug. Conversely, stamping a total that should track its lines guarantees drift. The question ""moment-fact or live-fact?"" decides stamp vs derive every time. |
| 10176 | - **Applies to:** tax rates, sold-at prices, FX rates on transaction dates, ownership; vs. totals, balances, counts, display names. |
| 10177 | - **Status:** doctrine. First stamping precedent: server-side `owner_id`; first derivation adopters: fin_journals totals, contacts.full_name. |
| 10178 | |
| 10179 | ### 2. Time-driven state belongs to the daemon; never to read-time checks. |
| 10180 | |
| 10181 | - **Decision:** When the passage of time changes a record's meaning (an invoice becomes *overdue*, a task becomes *stale*, an approval expires), a **daemon scheduled pass** makes the transition through the ordinary write/transition machinery — the state is *stored*. No surface ever computes ""is it overdue?"" at read time. |
| 10182 | - **Rationale:** Nothing writes when time passes, so materialize-on-write can't see it — and read-time checks multiply into disagreeing copies (list says overdue, report says current). A stored transition is permission-checked, transition-guarded, realtime-pushed, notify-able, and audited — one truth for every surface. The daemon's `after_hours` auto-transitions already established the pattern. |
| 10183 | - **Applies to:** aging/overdue states, expiries, escalation steps, SLA timers. |
| 10184 | - **Status:** doctrine; aging/dunning are the next adopters (need the payments collection first). |
| 10185 | |
| 10186 | ### 3. Money never mutates; it moves. |
| 10187 | |
| 10188 | - **Decision:** Corrections to money (and inventory) are **new, append-only compensating records** — a refund pointing at its payment, a reversing journal, a stock move — never edits to the original. Invariants on the movement (""refund ≤ payment − prior refunds"", ""journal balances"") are pre-write **hooks**; displayed positions (""refunded_cents"", ""on-hand"") are **rollups**. |
| 10189 | - **Rationale:** Auditability is structural, not procedural: if the original can't change, the history is the ledger. Every mature money system converges here (double-entry is 700 years of this doctrine); systems that edit payment rows in place cannot answer ""was this refund correctly credited?"" because the question has no record to point at. |
| 10190 | - **Applies to:** payments/refunds, journals (posted = immutable + reversals), stock_moves; `storage: append` is the substrate. |
| 10191 | - **Status:** doctrine. Journals + stock_moves live it today; refunds adopt it when payments land. |
| 10192 | |
| 10193 | ### 4. Extract the primitive after the pattern repeats — never before. |
| 10194 | |
| 10195 | - **Decision:** New business logic is first written **concretely** in the cheapest correct primitive for that one module — usually a hook (three lines of real code) or a plain object. Only after substantially the same shape appears in **two or three real modules** is it promoted to a declarative schema key, and the concrete versions are then deleted onto it. |
| 10196 | - **Rationale:** This is how every good layer here was actually built: four hand-rolled comment tables → `capabilities.comments`; eight bespoke view pages → one detail renderer; one balance rule → the hook primitive; the hook's repeated shapes → formulas and rollups. The graveyard of ""business operating systems"" is full of the opposite order: a workflow/rules engine designed first, with reality forced through it — and the first rule that doesn't fit becomes the wall. An abstraction extracted from three working instances fits by construction. (UI precedent: ui-decisions #10.) |
| 10197 | - **Applies to:** every ""should this be declarative?"" debate. The hook is the staging area; schema keys are the graduation. |
| 10198 | - **Status:** doctrine — the house method, now named. |
| 10199 | |
| 10200 | ### 5. State freezes fields — in hooks now, a schema key when it repeats. |
| 10201 | |
| 10202 | - **Decision:** ""A posted invoice's amounts are immutable"" (and every rule of the shape *fields X are locked when status = Y*) is expressed in each module's pre-write hook for now: reject when `existing.status` is in the frozen set and `changes` touches a locked field. When the third module writes that same hook, promote it to a declarative field/schema key (working name `locked_when`) and delete the hooks onto it, per #4. |
| 10203 | - **Rationale:** It's clearly a repeating shape, but guessing the declarative form before seeing real instances is how bad keys get designed — does it lock per-field or per-schema? Does admin bypass? Does it compose with transitions? The hooks will tell us. |
| 10204 | - **Applies to:** posted journals/invoices, completed orders, archived records. |
| 10205 | - **Status:** pattern named; first hook instances pending (journals are the natural first). |
| 10206 | |
| 10207 | ### 6. Gates before the write; reactions after — never crossed. |
| 10208 | |
| 10209 | - **Decision:** A pre-write hook may reject or stamp but never notify, email, or call external systems. Notifications and side effects run **after** the write via notify_rules / trigger objects / connectors, and never block or undo it. |
| 10210 | - **Rationale:** A gate with side effects leaks that a denied write was attempted, couples save latency to SMTP, and can half-happen. A reaction that gates would mean the write's fate depends on a mail server. The two postures also fail differently by design: gates fail **closed** (a broken gate rejects), reactions and derived values fail **soft** (a broken formula stores empty; a failed notify retries). |
| 10211 | - **Applies to:** every hook, every notify rule, every connector. |
| 10212 | - **Status:** doctrine (established in `event-hooks-decisions.md`; restated here as the load-bearing boundary of the whole map). |
| 10213 | |
| 10214 | ### 7. Generated documents carry provenance and face the same gates. |
| 10215 | |
| 10216 | - **Decision:** Every machine-composed record (a journal from a payment, a reversal, a recurring adjusting entry) stamps `generated_from` with its source (""payments/{id}"", ""reversal:{id}"", ""fin_recurring/{id}:{period}""), composers check that stamp before composing (replays are no-ops), and a generated entry passes the same balance verification a human entry would — composers re-verify because storage-layer writes bypass the HTTP-only hook. |
| 10217 | - **Rationale:** Idempotency by provenance makes event redelivery, daemon restarts, and manual re-runs safe by construction — no dedup tables, no distributed locks; the ledger itself remembers what was generated. And a machine allowed to skip the gates would make generated books less trustworthy than hand-kept ones — the opposite of the point. |
| 10218 | - **Applies to:** system_books composers, action_reverse_journal, fin_recurring_runner; every future composer. |
| 10219 | - **Status:** doctrine, shipped with the books spine. |
| 10220 | |
| 10221 | ### 8. Independent evidence is never edited to agree with our records |
| 10222 | |
| 10223 | - **Decision:** Records authored by an outside party — imported bank statement lines today, provider payout feeds and scanned documents next — are stored append-only with the original text preserved verbatim, and their factual fields are immutable after the write. What an operator may add is their *own* work about that evidence: what it matched, how it was resolved, a correction recorded as a new record. Corrections to the evidence itself come from re-importing a corrected file. |
| 10224 | - **Rationale:** The reason comparing our books to a bank statement is the strongest practical control is that the bank's copy was written by someone with no stake in our numbers. The moment a line can be edited, that independence evaporates and the check becomes self-certification — reconciliation degenerates into making the two sides match rather than explaining why they differ. This is also why the *act* of reconciling is an ordinary attributed update rather than a silent background match: reconciliation performed invisibly by the same hands that move the money is the classic embezzlement pattern, and an attributed change log makes who-decided-what structural instead of procedural. |
| 10225 | - **Applies to:** `bank_lines` (hook-enforced immutability of posted_on/amount/description/raw/identity fields), imported statement provenance (`bank_statement_imports`); the same posture is required of scanned-document intake and provider payout imports when they land. |
| 10226 | - **Corollary — a check that cannot run must say so:** an import records which self-checks *ran*, not only which passed, so a statement that carried no balances never reads as trustworthy as one whose arithmetic was verified. Silent skipping is how controls rot. |
| 10227 | - **Status:** doctrine, shipped with bank import. |
| 10228 | |
| 10229 | ### 9. Reactions belong to the write path; storage-level writers must say so |
| 10230 | |
| 10231 | - **Decision:** Event handlers dispatch on the HTTP write path only. An object that writes through `object_records` directly (a runner, an action) does **not** trigger `HANDLES` reactions, so if such a write needs its reaction it must perform it explicitly — using the *shared* primitive with the *same* provenance marker the handler would have used, never a private copy of the logic. |
| 10232 | - **Rationale:** Sharing the marker makes the two paths idempotent against each other: whichever runs first wins, the other is a no-op. Copying the logic instead would fork the rule and guarantee eventual divergence. |
| 10233 | - **Instances so far (counting toward #4):** (1) `action_apply_count` composes its own inventory-variance journal; (2) `action_resolve_bank_line` composes the NSF bounce reversal under `payments/{id}:bounced`, the marker `system_books` also uses. |
| 10234 | - **At the third instance, extract instead of working around:** the real fix is a daemon pass that dispatches handlers from the record-change log — the same shape `process_notifications` already uses to avoid depending on synchronous dispatch. Do not write a third workaround. |
| 10235 | - **Status:** observed twice; extraction pending the third. |
| 10236 | |
| 10237 | ### 10. An amount is an integer in its denomination's smallest unit |
| 10238 | |
| 10239 | - **Decision:** Money is stored as a whole number of the smallest unit of its own denomination, and a `denominations` record declares what that unit is (USD scale 2, JPY scale 0, BTC scale 8, ETH scale 18, XAU scale 4). Floats never appear; `Decimal` is used only at the edges to convert human text to and from integers. Amounts in different denominations are never added — combining them requires converting through a rate **stamped at the transaction moment** (#1), which is a deliberate act. |
| 10240 | - **Rationale:** This is the generalization of the founding ""money is integer cents"" rule rather than a replacement for it — cents were always *USD minor units*. A business holds value in several denominations at once (a bank balance, bitcoin, a stablecoin, a box of cash, gift cards, maybe a weight of metal), and they divide differently. The alternative — a per-denomination `decimal_places` field beside one fixed-scale decimal column — is what a predecessor system did, and it lets a schema *declare* more precision than it can *hold*: the first 18-decimal asset truncates silently, while every ordinary dollar carries six meaningless digits where rounding dust collects. Integer minor units are exact by construction, have no ceiling (Python ints are unbounded), and are what payment processors and chains already use. |
| 10241 | - **Corollary — rounding is a decision, not a parse.** Converting text that does not fit the scale (0.005 USD) raises; a caller that genuinely wants rounding calls `quantize_minor` and says so. Half a cent silently becoming a cent is how a per-row error turns into a real number at scale. |
| 10242 | - **Applies to:** `object_money.py`, the `denominations` collection; new value-bearing collections use `*_minor` + a denomination. Existing USD-only collections keep `*_cents`, which is honest for them — a big-bang rename buys nothing. |
| 10243 | - **Status:** doctrine, shipped with denominations; the value-account layer it founds is specified in `plan/value-accounts-and-denominations-spec.md`. |
| 10244 | |
| 10245 | ## How to use this |
| 10246 | |
| 10247 | - Placing a new rule? Walk `business-logic-patterns.md`'s flowchart; if the placement needed a judgment call, record it here. |
| 10248 | - If two modules place the same kind of rule differently, that's a bug: pick one, log it, and fix the outlier. |
| 10249 | - Counting instances for #4: when you write the second concrete copy of a shape, note it here so the third triggers extraction. |
| 10250 | " |
| 10251 | ui-decisions UI Interaction Decisions Design Decisions 1010 A living log of the small, repeated interaction decisions the generative UI makes — detail vs edit, page vs in-place, when a list caps, when a row links. "A living log of the small, repeated interaction decisions the generative UI |
| 10252 | makes — detail vs edit, page vs in-place, when a list caps, when a row links. |
| 10253 | The companion to the design-system guide (`design-system.md`): that doc is the *architecture* (semantics → |
| 10254 | tokens → renderers, ""define once, project everywhere""); this doc is the |
| 10255 | *behavior* those renderers agree on. |
| 10256 | |
| 10257 | **Why write these down.** Because the UI is one generative renderer, a decision |
| 10258 | here is not a per-screen choice — it applies to *every* app at once. The value |
| 10259 | is not picking the clever option on each screen; it's that the same rule holds |
| 10260 | everywhere, so nothing surprises you. When a decision is inconsistent across |
| 10261 | screens, that is a bug (a copy leaked), not a style preference. Most of these |
| 10262 | were found by screenshotting the running product and noticing the inconsistency. |
| 10263 | |
| 10264 | Format per decision: **Decision · Rationale · Applies to · Status.** Add new |
| 10265 | ones as they're settled; move Open Questions up as they're decided. |
| 10266 | |
| 10267 | --- |
| 10268 | |
| 10269 | ## Decided |
| 10270 | |
| 10271 | ### 1. Detail is a page (its own URL); edit and create are in-place. |
| 10272 | - **Decision:** Opening a record's *detail* navigates to `/{collection}/{id}`. |
| 10273 | *Editing* or *creating* a record happens in-place (an inline form on the |
| 10274 | current page), never a separate page. |
| 10275 | - **Rationale:** Detail is a *destination* — it must be linkable, bookmarkable, |
| 10276 | back-button-able (""everything has a url""), and it can hold a **composed view** |
| 10277 | (detail block + related records + aggregate) a modal can't. Edit/create is a |
| 10278 | transient *action* — you must not lose your list position or scroll to do it. |
| 10279 | - **Applies to:** every list/board row and card (→ detail page); every `+ New X` |
| 10280 | button and every row `✎` (→ in-place form). |
| 10281 | - **Status:** shipped. Rows/cards link to `/{collection}/{id}`; `+ New`/`✎` |
| 10282 | reveal the generated form in place. |
| 10283 | |
| 10284 | ### 2. Reports and logs have no detail link and no row actions. |
| 10285 | - **Decision:** A list over a generated/read-only collection (a rollup target, |
| 10286 | a log) shows neither edit/delete buttons nor a click-through to detail. |
| 10287 | - **Rationale:** A rollup row (top IP → hits) has no detail page and can't be |
| 10288 | edited — the next recompute overwrites it. Offering either is a dead end. |
| 10289 | - **Applies to:** any list block with `row_actions: false` (which also implies |
| 10290 | `link: false`). |
| 10291 | - **Status:** shipped (analytics reports). |
| 10292 | |
| 10293 | ### 3. Lists render-cap at 50 rows, with Show-all. |
| 10294 | - **Decision:** A list renders at most 50 rows (overridable per block via |
| 10295 | `row_limit`) with a **Show all / Show fewer** toggle; the data is already |
| 10296 | fetched, so expanding is free. |
| 10297 | - **Rationale:** Without a cap, a busy collection (a real-traffic rollup, a big |
| 10298 | browse) renders a 50,000-px page. Found by screenshotting live `/analytics`. |
| 10299 | - **Applies to:** every list surface in every app, automatically. |
| 10300 | - **Status:** shipped. |
| 10301 | |
| 10302 | ### 4. Every record has a reachable URL; detail routes accept any id. |
| 10303 | - **Decision:** A detail route is `/{collection}/{id}` with **no `:uuid` |
| 10304 | constraint** — any id segment routes to the detail view. Records still get |
| 10305 | UUIDs on create; the route simply doesn't *reject* other ids. |
| 10306 | - **Rationale:** ""Everything has a url"" must hold for *every* record, including |
| 10307 | ones with friendly/seed ids. Once rows and cards link (decision 1), a |
| 10308 | `:uuid`-only route 404s exactly the records a demo shows first. A missing |
| 10309 | record is the view's own not-found state, not a routing 404. |
| 10310 | - **Applies to:** every detail route in every package. (The `:uuid` matcher |
| 10311 | still exists for a route that deliberately wants to enforce shape.) |
| 10312 | - **Status:** shipped. |
| 10313 | |
| 10314 | ### 5. Board columns flex to fit; scroll only when genuinely too many. |
| 10315 | - **Decision:** Board columns are `flex: 1 1 200px` (min 200, max 340), not a |
| 10316 | fixed width. A few columns fill the container; the board scrolls **internally** |
| 10317 | (never the page) only when columns would go below their min width. |
| 10318 | - **Rationale:** Fixed-width columns overflowed the container and forced a |
| 10319 | *page-level* horizontal scrollbar for a mostly-empty board. The page must |
| 10320 | never scroll sideways. |
| 10321 | - **Applies to:** every board (kanban / lead-pipeline). |
| 10322 | - **Status:** shipped. |
| 10323 | |
| 10324 | ### 6. A report renders as a sorted top-N. |
| 10325 | - **Decision:** A list block can declare `sort_by` (+ `sort_dir`) and |
| 10326 | `row_limit` so it renders a true top-N by a real field (top IPs by hits desc), |
| 10327 | not insertion order. |
| 10328 | - **Rationale:** ""Top"" must mean *ranked*, or the report is noise. |
| 10329 | - **Applies to:** analytics and any report-shaped list. |
| 10330 | - **Status:** shipped. |
| 10331 | |
| 10332 | ### 7. A data-dense collection renders as a live table (`list_mode: ""table""`). |
| 10333 | - **Decision:** A schema (or a view's list block) can declare `list_mode: |
| 10334 | ""table""` to render its `list_fields` as a dense HTML table — sortable column |
| 10335 | headers, one row per record, cells formatted by field type (money `_cents` → |
| 10336 | `$`, boolean → Yes/No, datetime → relative). It reuses the same |
| 10337 | `startRowList` engine as the rich-row list, so it **inherits for free**: the |
| 10338 | search box, the 50-row cap + Show-all (#3), client-side sort (#6), |
| 10339 | row→detail navigation (#1), and — the differentiator — **realtime |
| 10340 | re-render** on the change log. A table here is not a static snapshot the way |
| 10341 | a conventional server-rendered table library produces — it updates live as records change. |
| 10342 | - **Rationale:** Rich rows (title + snippet) suit content/feeds; they waste |
| 10343 | space and hide structure for inherently tabular data (invoices, contacts, |
| 10344 | links, a rollup). One renderer, chosen per collection by a single schema key, |
| 10345 | keeps every table consistent instead of each app hand-rolling one. |
| 10346 | - **Applies to:** any collection whose index goes through `dbbasicList`; opt in |
| 10347 | with `views.list_mode: ""table""` + `list_fields`. Header click toggles |
| 10348 | sort field/dir; the whole row is the detail link. |
| 10349 | - **Status:** shipped. Live on `/articles` (Title · Published On · Published). |
| 10350 | |
| 10351 | ### 8. The generated form matches a hand-built one (spacing + themed selects). |
| 10352 | - **Decision:** The generative form uses **one** spacing mechanism (grid |
| 10353 | `gap`), collapses a field's error line until there's an actual message |
| 10354 | (`.err:empty { display: none }`), and themes every `<select>` itself |
| 10355 | (`appearance: none` + our own chevron). |
| 10356 | - **Rationale:** The generated form was looser than a bespoke one for two |
| 10357 | invisible reasons — it inherited *both* the grid gap and the global |
| 10358 | `.stack > * + *` margin (doubled space), and reserved a blank 1rem error |
| 10359 | line under every field (~44px between fields vs ~14px). And an unstyled |
| 10360 | `<select>` renders as Safari's beveled native control but Chrome's flat one, |
| 10361 | so the same page looked different per browser. A generic renderer has to be |
| 10362 | *at least as good* as the hand-built page it replaces, or migrating onto it |
| 10363 | (decision below) is a regression. Found by comparing the bespoke Projects |
| 10364 | form to the generated Notes form side by side. |
| 10365 | - **Applies to:** every generated form (`window.dbbasicForm`) and every |
| 10366 | `<select>` in the product. |
| 10367 | - **Status:** shipped. Measured 14px inter-field gap, no reserved error space. |
| 10368 | |
| 10369 | ### 9. A board collection can toggle to a table (persisted per-collection). |
| 10370 | - **Decision:** A collection declared `list_mode: ""board""` that also has |
| 10371 | `list_fields` shows a small **Board / Table** segmented control; the table is |
| 10372 | the same generative table from decision #7. The choice is stored per |
| 10373 | collection in `localStorage` and survives navigation. |
| 10374 | - **Rationale:** A kanban is the right default for a workflow surface, but the |
| 10375 | same records are often better *scanned* as a dense sortable table — the |
| 10376 | classic kanban⇄list toggle. Both renderers already exist behind one |
| 10377 | `resolveListMode`, so it's a switch, not new rendering. Switching **reloads** |
| 10378 | rather than swapping in place: `dbbasicSubscribe` has no unsubscribe, so an |
| 10379 | in-place swap would leave the previous mode's change-log handler live and |
| 10380 | double-render. One page load = one subscription. |
| 10381 | - **Applies to:** any board-declared collection with `list_fields` (tasks |
| 10382 | today). Tree/calendar toggles are a possible follow-on, same mechanism. |
| 10383 | - **Status:** shipped (tasks Board/Table). |
| 10384 | |
| 10385 | ### 10. Make the generic layer ≥ the bespoke page before deleting the bespoke. |
| 10386 | - **Decision:** When a hand-built page does something the generic renderer |
| 10387 | doesn't (or does *better*), fix the generic renderer to that bar **first**, |
| 10388 | then delete the bespoke page onto it — never the reverse order. |
| 10389 | - **Rationale:** A bespoke page that looks better than the generic one isn't |
| 10390 | just debt to delete; it's a spec for a quality gap in the shared layer. The |
| 10391 | Projects form (tight) was showing what the generated form (loose, #8) should |
| 10392 | be. Delete-first would have made Projects *worse*; fix-first made every |
| 10393 | form better and Projects free. |
| 10394 | - **Applies to:** every ""migrate a bespoke page onto the generator"" task |
| 10395 | (Projects done; analytics/commerce pages next). |
| 10396 | - **Status:** doctrine. First applied migrating `site/projects.py` (~60 lines |
| 10397 | of hand-rolled table/form/fetch deleted) onto `dbbasicList` + `dbbasicForm`. |
| 10398 | |
| 10399 | ### 11. A collection filters by declaring `filter_fields` (works on every mode). |
| 10400 | - **Decision:** A schema's `views.filter_fields` renders a **filter bar** above |
| 10401 | the list — an enum field → a select of its options, a boolean → Yes/No. A |
| 10402 | pick ANDs into the fetch `where` and reloads the active mode. Filters compose |
| 10403 | with the caller's own `where` (entity scope, a parent FK) and with the |
| 10404 | client-side search box; a **Clear** button appears once anything is set. |
| 10405 | - **Rationale:** The list renderer had sort + live but not filters — the third |
| 10406 | leg of ""a table, but generative and alive."" Filtering is server-side (the |
| 10407 | same `field=value` path the board already uses, applied after the permission |
| 10408 | row filter), so a filtered list can only ever *narrow* what the viewer may |
| 10409 | already see — never widen it. One schema key, filters on list, table, and |
| 10410 | board at once. |
| 10411 | - **Applies to:** any collection that declares `filter_fields` (tasks: |
| 10412 | status + urgency today). Text is left to the search box; a date-range control |
| 10413 | is the next increment (needs the dotted-operator `gte/lte` query path). |
| 10414 | - **Status:** shipped. Verified server-side narrowing (6 tasks → 2 on |
| 10415 | status=open) and that the bar renders in board and table. |
| 10416 | |
| 10417 | ### 12. A collection opts into behaviors via `capabilities` (not just display). |
| 10418 | - **Decision:** Beyond the display keys (`list_mode`/`filter_fields`/…), a |
| 10419 | schema declares generic *behaviors* under a root-level `capabilities` key, |
| 10420 | wired by the platform rather than hand-built per app. First one: |
| 10421 | `capabilities.comments: true` → the detail page grows a comment thread |
| 10422 | (`window.dbbasicThread`, backed by the polymorphic `thread_comments` |
| 10423 | collection), no per-view block, no per-app comment table. |
| 10424 | - **Rationale:** Comments existed **four times** (task_comments, thread_comments, |
| 10425 | profile_comments, interactions) — the same ""a record has a thread hung off |
| 10426 | it"" copied per app. That's a workaround for a layer that should be uniform. |
| 10427 | One capability flag collapses them: the widget is polymorphic |
| 10428 | (parent_collection + parent_id), so any collection gets comments by declaring |
| 10429 | one key. This is the *behavior/connection* layer sitting on top of the |
| 10430 | display layer — the same ""define once, project everywhere,"" for what a |
| 10431 | collection *does* and *connects to*, not just how it looks. |
| 10432 | - **Applies to:** any collection (tasks today). `capabilities` is whitelisted in |
| 10433 | schema normalization and surfaced in `/api/schema` so the client can wire it. |
| 10434 | - **Status:** shipped (two capabilities now). `comments`: task_comments |
| 10435 | migrated onto thread_comments, related block removed, live widget post |
| 10436 | verified. `attachments`: `window.dbbasicAttachments` (served at |
| 10437 | `/attachments`) uploads/lists/downloads/deletes files for a record via the |
| 10438 | polymorphic `files.(parent_collection, parent_id)` pair — no per-collection |
| 10439 | FK; tasks adopts both, the old `related:files` block removed, a live |
| 10440 | widget upload verified. `renderDetail` mounts them via one |
| 10441 | `maybeMountCapabilities`. `shareable` is the foundational one — it changes |
| 10442 | the permission check, not just a widget. `capabilities.shareable` |
| 10443 | + a `{id: $shared_records}` read rule let a record's OWNER grant other users |
| 10444 | access via the `record_shares` collection. `window.dbbasicShare` (owner-only, |
| 10445 | renders nothing for non-owners) writes grants only through the owner-checked |
| 10446 | `/api/share` endpoint (authorized against the TARGET record's ownership — a |
| 10447 | cross-collection fact a row_filter can't express, and the thing that stops |
| 10448 | self-serve escalation). A new collection-aware `$shared_records` row-filter |
| 10449 | variable (the collection is threaded into resolution) keeps a share on a task |
| 10450 | from leaking a same-id row in another collection. Security boundary verified |
| 10451 | live: grantee reads (200), non-grantee denied (403), non-owner can't grant |
| 10452 | (403). All three capabilities now mount in one detail page from flags alone. |
| 10453 | - **Gotchas found building it:** a field that is `required` **and** `read_only` |
| 10454 | can never be created through the HTTP write path (only a server-side |
| 10455 | `preserve_read_only` bypass) — `thread_comments.parent_*` had to drop |
| 10456 | `read_only` to be settable on create. And `owner_id` (a `public:hidden` field) |
| 10457 | is server-set from the session; a client value is rejected, and it's redacted |
| 10458 | from other readers — so comments stamp a separate `author_name` for |
| 10459 | attribution. |
| 10460 | |
| 10461 | ### 13. Every data collection carries created_at; detail always shows it. |
| 10462 | - **Decision:** `created_at` (datetime, read-only, server-owned) is a baseline |
| 10463 | field on every user-facing data collection, and the detail generator always |
| 10464 | surfaces `created_at`/`updated_at` as record metadata — even when |
| 10465 | `detail_fields` curates the main fields — the way the list row already shows |
| 10466 | a timestamp. |
| 10467 | - **Rationale:** ""When was this created"" is baseline info every record should |
| 10468 | carry and every detail should show; `detail_fields` is for curating the |
| 10469 | *content* fields, not for suppressing metadata. Found by spotting records |
| 10470 | with no timestamp (a collection missing the field entirely — e.g. projects) |
| 10471 | and detail pages hiding it (tasks/contacts/forum_topics curated it out). |
| 10472 | - **Applies to:** all user-facing collections (config/system ones — |
| 10473 | feature_flags, user_prefs, shell_*, *_definitions, project_access, |
| 10474 | ai_prices, dbbasic_probe — intentionally omit it). Detail metadata append is |
| 10475 | automatic in the shared renderer. |
| 10476 | - **Status:** shipped. Added the field to projects + organizations/files/ |
| 10477 | events/interactions/time_logs/tags/templates; detail appends the timestamps. |
| 10478 | (Old records predate the field and read it back empty — new ones get a |
| 10479 | server-set value.) |
| 10480 | |
| 10481 | ### 14. A relation field shows its target's name, not the raw FK id. |
| 10482 | - **Decision:** A field with `relation:{collection, display_field}` renders the |
| 10483 | referenced record's display value (""Acme Corp"") in the table cell and board |
| 10484 | card — never the raw id (""o-acme""). Each referenced collection is fetched once |
| 10485 | per render surface to build an id→label map; fails soft to the id when |
| 10486 | unmapped. Board cards additionally drop any card field whose value is already |
| 10487 | part of the title (so a full-name title doesn't repeat first/last below). |
| 10488 | - **Rationale:** The detail page already resolves relations (form.py's |
| 10489 | `control`); the list surfaces showed raw ids, which read as broken data. |
| 10490 | ""Same field, same display, everywhere"" — a relation is a name to a human, |
| 10491 | not a UUID. Found by screenshotting the contacts board. |
| 10492 | - **Applies to:** every table and board over a collection with a relation field. |
| 10493 | - **Status:** shipped. Verified contacts board + table show ""Acme Corp"". |
| 10494 | |
| 10495 | ### 15. Derived values are stored values (formulas & rollups materialize). |
| 10496 | - **Decision:** A computed field's `formula`/`rollup` value is **written into |
| 10497 | the record** whenever its inputs change — never computed on read, never |
| 10498 | client-side. |
| 10499 | - **Rationale:** Stored values mean the entire generative stack (lists, |
| 10500 | tables, boards, detail, search, server-side filters, sort, CSV, backups, |
| 10501 | grep) works with zero special cases, and realtime is free: a child-line |
| 10502 | edit rewrites the parent's stored total → change log fires → every open |
| 10503 | browser updates. Staleness is confined to definition changes, handled by an |
| 10504 | explicit `recompute_computed_fields` backfill — never a runtime loop. |
| 10505 | - **Applies to:** every `type: computed` field with a `formula` or `rollup` |
| 10506 | (fin_journals debit/credit totals; contacts full_name). |
| 10507 | - **Status:** shipped. |
| 10508 | |
| 10509 | --- |
| 10510 | |
| 10511 | ## Open questions (decide, then move up) |
| 10512 | |
| 10513 | ### A. Create UX on data-heavy pages. |
| 10514 | `+ New X` currently reveals a full-width inline form that pushes the list/board |
| 10515 | down and out of view. Fine when empty; disruptive when the page has data. Options: |
| 10516 | a collapsible/side form, a modal, or navigating to a dedicated create route. |
| 10517 | Leaning: keep in-place (decision #1) but render the form as a **right-side drawer |
| 10518 | or a modal** on data-heavy index pages, so the list stays visible. Undecided. |
| 10519 | |
| 10520 | ### B. Triage lists → master-detail split-pane. |
| 10521 | An inbox / review-queue / messaging thread list processes many items fast; |
| 10522 | navigating away per item (decision #1) is wrong there. Proposed: a |
| 10523 | `list_mode: ""split""` — click a row, preview in a right pane, never navigate. |
| 10524 | A per-app opt-in exception to decision #1, not the default. Not built. |
| 10525 | |
| 10526 | ### C. Detail view de-emphasizes the raw id. |
| 10527 | The generated detail renders the record's `id` (a UUID) as the first field row — |
| 10528 | noise. Proposal: hide `id`/computed fields from the default detail field table |
| 10529 | (they're still reachable), the way `list_fields` already curates the list. |
| 10530 | |
| 10531 | ### D. ~~Seed data should use UUIDs.~~ RESOLVED (decision 4). |
| 10532 | Was: friendly seed ids (`c-grace`, `t-demo-1`) 404'd against `{id:uuid}` routes |
| 10533 | once rows linked. Resolved by relaxing detail routes to accept any id (decision |
| 10534 | 4) rather than rewriting seed ids and their FK references — every record is |
| 10535 | reachable whatever its id shape. |
| 10536 | |
| 10537 | --- |
| 10538 | |
| 10539 | ## How to use this |
| 10540 | |
| 10541 | - Before adding a per-screen interaction, check here — the decision may already |
| 10542 | exist and just needs wiring. |
| 10543 | - If you find two screens behaving differently for the same interaction, that's |
| 10544 | a bug: pick the rule (add it here) and fix it in the shared renderer, not the |
| 10545 | screen. |
| 10546 | - These decisions live in `packages/app-theme/objects/site/` (list/form/detail |
| 10547 | generators) and `packages/app-views/objects/site/view_render.py` (block → |
| 10548 | generator config), the single places that project them everywhere. |
| 10549 | " |
| 10550 | event-hooks-decisions Event Hooks: Decisions and Why Design Decisions 1020 Design rationale for DBBASIC's post-commit event handlers, why they shipped after feature flags, and a real production trap around override-root objects. "This page records the design decisions behind event handler objects (Phase 5) and how they were sequenced against `user_prefs` + `feature_flags` (Phase 6). It's written the way the decisions were made — before implementation — so you understand *why* the system works this way, not just what it does. |
| 10551 | |
| 10552 | ## Decision 1: Post-commit, best-effort dispatch |
| 10553 | |
| 10554 | Handlers run **after** a record write commits, isolated in the normal object sandbox; a handler failure is logged and never breaks or rolls back the write. There is no pre-write mutation or rejection in this phase. |
| 10555 | |
| 10556 | **Why.** The write path is the platform's most load-bearing promise: a valid write succeeds fast. Synchronous in-path handlers make every write's latency and reliability hostage to the worst installed handler — the exact footgun that made in-transaction trigger systems (classic SQL triggers, early Salesforce Apex) notorious, and the reason the systems that aged well converged on post-commit dispatch: Stripe/GitHub webhooks, Django's `on_commit` signals, Rails' `after_commit`. Post-commit covers the real 90% of use cases — notify, sync, enrich other records, log, kick off follow-on work. Validation and derivation already have a home (schema rules, computed fields); pre-write hooks could become a later, separately-gated phase if genuinely needed. |
| 10557 | |
| 10558 | This mirrors the shape of the platform's realtime push mechanism: `_publish_record_change_event` fires post-commit, and handlers are just one more subscriber to that stream. The hook point already existed, so adding handlers can't destabilize writes. |
| 10559 | |
| 10560 | Safety posture: |
| 10561 | |
| 10562 | - Gated off by default (`DBBASIC_ENABLE_EVENT_HANDLERS`) |
| 10563 | - Depth/reentry guard — a handler's own writes fire events too, but dispatch depth is capped, so there are no infinite loops |
| 10564 | - Standard sandbox timeout per handler |
| 10565 | - Failures land in the object's log tagged with the triggering event's correlation ID |
| 10566 | |
| 10567 | ## Decision 2: System/package objects only (this phase) |
| 10568 | |
| 10569 | Only operator-installed objects may declare `HANDLES = [...]`; they run in a defined service context. User-authored (`u_*`) handlers are deferred. |
| 10570 | |
| 10571 | **Why.** ""User code runs when *other people's* data changes"" is a privilege question with real teeth — whose rows can the handler read? who is the actor on its writes? The operator case needs none of that resolved, because the operator already owns the instance. Deferring user handlers keeps this phase's blast radius limited to code the operator explicitly installed, and lets the identity-threading design mature before it's exposed to untrusted authors. This is the same staging pattern used for overrides generally: opt-in, dormant by default. |
| 10572 | |
| 10573 | ### The trap this creates, found in production |
| 10574 | |
| 10575 | `build_index` scans object sources and skips anything whose `kind` is not `""system""` — that check is how this decision gets enforced. An object living in the override root has `kind=""override""`, so a handler installed there declares `HANDLES`, installs cleanly, reports `""written""`, and then receives no events at all. Nothing errors; the dispatcher simply never learns the object exists. |
| 10576 | |
| 10577 | This bit a real deploy: `install_package` sent every new object to `roots[0]`, and `get_object_roots()` returns `[override, base]` in lookup order, so new package objects landed in the override root by default. It's fixed now — new objects go to the base root, and an object that already exists stays wherever it lives, so a customization is never yanked out from under an upgrade — but the underlying sharp edge remains by design: **an override cannot handle events.** |
| 10578 | |
| 10579 | If a handler is silent, check its `kind` before checking its code: |
| 10580 | |
| 10581 | ```python |
| 10582 | import object_handlers, object_namespace |
| 10583 | print(object_handlers.build_index().get(""your_collection.record.created"")) |
| 10584 | print([(s.object_id, s.kind) for s in object_namespace.iter_object_sources() |
| 10585 | if s.object_id == ""system_your_handler""]) |
| 10586 | ``` |
| 10587 | |
| 10588 | ## Decision 3: Phase 6 (prefs + flags) ships first |
| 10589 | |
| 10590 | **Why.** |
| 10591 | |
| 10592 | 1. It's data plus permission rules layered on machinery that already exists — collections, owner-scoped row filters, the generators — near-zero new engine code, so it's the fast, clean win. |
| 10593 | 2. Feature flags are *how event handlers should ship*: handlers land dark behind a flag, get canaried, and roll back by flipping a flag instead of uninstalling code. Building flags first means the riskier phase ships the safe way. |
| 10594 | 3. Flags convert a whole class of would-be code customizations into configuration, shrinking the reconcile surface — the same motive behind the rest of the upgrade system. |
| 10595 | |
| 10596 | Phase 6 shape: a `user_prefs` collection (owner-scoped `user_id`/`key`/`value`), a `feature_flags` collection for instance-wide toggles, and package manifests that may declare `features` with defaults stamped at install time. The effective value resolves as user override → instance toggle → package default, through one shared helper (a `/flags` object, exposed client-side as `window.dbbasicFlags`). |
| 10597 | |
| 10598 | ## Shipping order |
| 10599 | |
| 10600 | 1. Phase 6: `user_prefs` + `feature_flags`. |
| 10601 | 2. Phase 5a: `HANDLES` dispatch (post-commit, system objects only, flag-gated). |
| 10602 | 3. Phase 5b: generator page slots (`before_list`, `row_actions`) — separable, UI-only, can ship alongside either of the above. |
| 10603 | 4. Later, only if pulled by real need: pre-write hooks, and user-authored handlers running as their own author. |
| 10604 | |
| 10605 | ## Update: pre-write hooks landed |
| 10606 | |
| 10607 | The ""later, only if pulled by real need"" condition in step 4 above was eventually met — the need being custom cross-field/cross-collection validation on generatively-formed collections, which is the extensibility question every schema-driven system eventually has to answer. |
| 10608 | |
| 10609 | `hooks.before_write` now runs a declared object synchronously in the generic write path: after permission checks, before persist; it can reject or transform the write; it **fails closed**; and it's opt-in per collection, so the latency concern from Decision 1 stays bounded — collections with no hook pay one cached dict lookup and nothing more. After-write work remains the event system's job; nothing about pre-write hooks changes that split. |
| 10610 | |
| 10611 | See the validation and logic guide for the full contract on `before_write` hooks. |
| 10612 | " |