Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
doc_pages.tsv · 450.1 KB · 10612 lines
1id title category nav_order summary content
2why Why DBBASIC — The Advantages, Honestly Stated Start Here 100 A plain-spoken case for DBBASIC's design — live changes, apps as data, one permission engine — with its current boundaries stated just as plainly. "DBBASIC is a small server with strong opinions. This page states the
3advantages plainly, and the boundaries next to them, because a claim
4without its boundary is marketing.
5
6## Change is live
7
8Objects load per execution: edit an object and the next request serves
9the new code. Schemas, permission rules, routes, and site pages are
10data, changed over HTTP. Only changes to the server file itself need a
11restart (seconds). There is no build, no deploy pipeline, no migration
12ceremony.
13
14Safety comes from **reversibility instead of ceremony**: object source
15is versioned with rollback, schemas keep history, records keep
16changelogs, permission changes and admin actions are audited. Ship in
17steps of one; undo in steps of one.
18
19## Apps are data
20
21An application is a package: schema + permission rules + optionally one
22page object ([the app suite](/docs/app-packages)). The server grows only
23when a genuinely new *kind* of capability appears — search, sharing,
24file storage, AI — and each lands once, generically, then every app
25uses it by declaring so in its schema.
26
27The consequence: **capability grows with data, not code.** Add a
28schema for any obscure domain and it immediately has validated writes,
29generated forms, search, sharing, MCP tools, and AI conversation about
30it. Nobody ships function-calling for your beekeeping records; here it
31is free.
32
33## One permission engine, every surface
34
35Browser pages, the records API, MCP agents, global search, file
36downloads, and AI tool calls all pass the same policy: role rules, row
37filters (`$user_id`, `$accessible_projects`, `$owned_projects`), field
38redaction, audit. A rule written once binds everywhere — no surface can
39leak what another surface hides. Sharing is records (`project_access`
40rows), so granting and revoking are audited record writes.
41
42## Fast because it does less
43
44A request here is: a dict-router match, a TSV read the OS page cache
45already holds, permission string-checks, JSON out. No ORM, no
46connection pool, no query planner, no middleware stack, no signal
47cascade. At personal-and-team data sizes (thousands of rows per
48collection), scanning a hot file in-process beats an indexed query
49across a socket.
50
51**Boundary:** this inverts somewhere past hundreds of thousands of rows
52in one collection. Collections are per-file, so a hot one can gain an
53index later without changing the platform; the search contract promises
54no ranking, so an index can slot in behind it.
55
56## AI-native without AI lock-in
57
58The whole admin surface is MCP; agents operate the server with their
59own identities and audited sessions. Users store their own provider
60keys (write-only), pick their own models, and hand the AI a chosen
61subset of tools ([the shell](/docs/shell-and-ai)). The server never meters
62your tokens or requires a vendor. An AI acting for a user is exactly as
63powerful as the user — no more.
64
65## Yours
66
67Stdlib-only Python plus uvicorn; one VM; data in human-readable TSV and
68JSON files you can grep, back up, and take elsewhere. Small enough for
69one person to read. MIT licensed, with the operator console (Scroll)
70open source too. Multi-domain hosting means one instance serves many
71sites with one identity store and one audit trail.
72
73## Current boundaries, plainly
74
75- One VM; no cluster correctness claims.
76- Realtime push is live over one permission-filtered websocket (`/ws`);
77 the durable event log remains as the poll fallback.
78- A background-job runtime exists (`app-runner`, plus the daemon's
79 scheduler and queue); what it lacks is handlers for long media work
80 such as PDF extraction and thumbnails.
81- Untrusted arbitrary code execution is not offered; object writes are
82 operator/admin actions, and the worker pool isolates processes without
83 CPU or memory limits.
84- Open signup and password reset flows are not built; identities are
85 operator-created today.
86- Permission enforcement is opt-in per deployment rather than the shipped
87 default (staging runs with it on).
88
89The direction on each is recorded in status. The rule for
90new capability is unchanged: it must pass the same permission engine,
91audit trail, and reversibility bar as everything before it.
92"
93quickstart Quickstart — From a Fresh VM to Your First App Start Here 110 A linear, thirty-minute path from a bare Debian/Ubuntu VM to a running DBBASIC Object Server with login, HTTPS, and your first app installed. "This is the linear path: a bare Debian/Ubuntu VM to a running object
94server, a login, a public domain, and a first app installed — in one
95sitting, about thirty minutes, most of it waiting on `apt`. Every step
96is a real command you can paste.
97
98For the full reference (systemd hardening, journald caps, backups,
99permission rollout, per-route proxy allowlists), see
100[single-vm deployment](/docs/single-vm-deployment). This page is the happy
101path; that one is the operator's manual.
102
103```mermaid
104flowchart LR
105 A[""Fresh VM<br/>+ a domain""] --> B[""Run the installer""]
106 B --> C[""Create your admin user<br/>+ enable login""]
107 C --> D[""Caddy: domain + TLS""]
108 D --> E[""Enable package installs""]
109 E --> F[""Install an app<br/>→ live at /notes""]
110```
111
112## 0. What you need
113
114- A small VM (a $5–$7 droplet is plenty) running Debian 12 or Ubuntu
115 22.04+, with root or `sudo`.
116- Optional but recommended: a domain name with an `A` record pointing at
117 the VM's IP, so Caddy can get you automatic HTTPS.
118
119## 1. Run the installer
120
121The installer is a readable script — download it, read it, run it. It
122installs dependencies, creates the service user and layout, builds a
123virtualenv, generates an admin token, and starts a systemd service on
124`127.0.0.1:8001`. It is idempotent, so re-running it is safe.
125
126```bash
127git clone <repository-url> dbbasic-object-server
128cd dbbasic-object-server
129less scripts/install.sh # read it first
130sudo bash scripts/install.sh
131```
132
133When it finishes you have a running server. Confirm:
134
135```bash
136curl http://127.0.0.1:8001/health
137```
138
139Your admin token lives in `/etc/dbbasic-object-server.env` as
140`DBBASIC_ADMIN_TOKEN` (readable by root and the service group only).
141Everything the installer did on purpose *not* to do — login, a public
142domain, app installs — is below, as short deliberate steps.
143
144## 2. Create your admin user and turn on login
145
146```bash
147cd /opt/dbbasic-object-server
148sudo -u dbbasic DBBASIC_DATA_DIR=/var/lib/dbbasic-object-server/data \
149 .venv/bin/python object_identity_cli.py create-superuser \
150 --user-id you --email you@example.com
151```
152
153It prompts for a password twice (no echo) and stores only a scrypt hash
154under the runtime data directory. Now enable password login and restart:
155
156```bash
157sudo sed -i 's/^DBBASIC_ENABLE_PASSWORD_LOGIN=false/DBBASIC_ENABLE_PASSWORD_LOGIN=true/' \
158 /etc/dbbasic-object-server.env
159sudo systemctl restart dbbasic-object-server
160```
161
162## 3. Put a domain and HTTPS in front
163
164Keep uvicorn bound to localhost and let Caddy terminate TLS. If a domain
165points at the VM, this is all it takes:
166
167```bash
168sudo apt install -y caddy
169sudo tee /etc/caddy/Caddyfile >/dev/null <<'EOF'
170your-domain.com {
171 handle / {
172 rewrite * /objects/site_home
173 reverse_proxy 127.0.0.1:8001
174 }
175 handle {
176 reverse_proxy 127.0.0.1:8001
177 }
178}
179EOF
180sudo systemctl reload caddy
181```
182
183Caddy fetches a certificate automatically. Visit `https://your-domain.com`
184— you should see the home object, and `https://your-domain.com/login`
185should accept the user you just created.
186
187> The catch-all `handle` above proxies every path to the server, which
188> relies on the permission policy as the gate. For a locked-down public
189> staging box that instead exposes only named routes, use the per-route
190> allowlist in [single-vm deployment](/docs/single-vm-deployment#https-proxy).
191
192## 4. Enable app installs and install your first app
193
194Apps are packages that ship in the repo under `packages/`. Turn on
195installs, then ask the server to install one. (Installs are gated by the
196admin token, so this stays an operator action.)
197
198```bash
199sudo sed -i 's/^DBBASIC_ENABLE_PACKAGE_INSTALLS=false/DBBASIC_ENABLE_PACKAGE_INSTALLS=true/' \
200 /etc/dbbasic-object-server.env
201sudo systemctl restart dbbasic-object-server
202
203TOKEN=$(sudo grep -oP 'DBBASIC_ADMIN_TOKEN=\K.*' /etc/dbbasic-object-server.env)
204
205# Projects is the hub other apps relate to; install it first, then notes.
206for app in app-projects app-notes; do
207 curl -s -X POST http://127.0.0.1:8001/packages/$app/install \
208 -H ""Authorization: Token $TOKEN"" -H 'Content-Type: application/json' -d '{}'
209 echo
210done
211```
212
213`app-notes` includes a page object. To turn permission enforcement on so
214each user sees only their own notes, follow the audit-then-enforce
215rollout in [permissions](/docs/permissions-model#route-enforcement); on a
216single-operator box you can leave it open to start.
217
218Visit `https://your-domain.com/notes`, sign in, and you have a working
219notes app — quick capture, projects, search, public sharing — that you
220never wrote a line of server code for. Browse
221[the app suite](/docs/app-packages) for the rest (tasks, contacts,
222articles, calendar, files, the shell) and install the ones you want.
223
224## 5. Optional: the talk-to-everything shell
225
226Install `app-shell`, then store an AI provider key and chat with your
227data — see [the shell and AI guide](/docs/shell-and-ai). Storing a key is
228one command in the shell (`/key anthropic sk-...`), and the AI operates
229the server with your permissions.
230
231## What to do next
232
233- **Back it up.** All state is files under
234 `/var/lib/dbbasic-object-server/data`; see [backup and restore](/docs/backup-restore).
235- **Harden it.** systemd sandboxing, journald caps, and the readiness
236 gates are in [single-vm deployment](/docs/single-vm-deployment).
237- **Understand the model.** [Why DBBASIC](/docs/why) and
238 [comparisons](/docs/comparisons) explain what you are and are not
239 signing up for.
240- **Build.** Copy `app-notes` to make your own app
241 ([package authoring](/docs/package-authoring)), or add a system-tool
242 object ([capability objects](/docs/capability-objects)).
243
244## Upgrading later
245
246```bash
247cd /opt/dbbasic-object-server
248sudo -u dbbasic git pull --ff-only
249sudo -u dbbasic .venv/bin/python -m pip install -e '.[server]'
250sudo systemctl restart dbbasic-object-server
251```
252
253Changes to live objects and schemas need no deploy at all — that is the
254point. Only server-code upgrades need the pull-and-restart above.
255"
256architecture Architecture — The Layers, and Where Each Doc Fits Start Here 120 A map of the DBBASIC Object Server stack, from storage through access surfaces, with pointers to the focused doc for each layer. "DBBASIC is a small stack. Everything is a record or an object in a
257human-readable store; each layer above adds capability without hiding the one
258below. This page is the map — read it first, then branch into the focused doc
259for whichever layer you care about.
260
261```mermaid
262flowchart TB
263 A[""Access surfaces — Web · HTTP API · Shell / AI / MCP · Files""]
264 P[""Permission engine — one policy enforced on every surface""]
265 G[""Generative UI — list / table / board / tree / calendar · forms · detail · filters · search""]
266 C[""Capabilities — comments · attachments · sharing""]
267 D[""Data — schemas (types · validation · relations · transitions) + collections""]
268 R[""Runtime — ASGI object execution · realtime change log · daemon (triggers · schedules)""]
269 S[""Storage — human-readable TSV + object source / state / logs / files / versions""]
270
271 A --> P --> G
272 G --> C
273 G --> D
274 C --> D
275 D --> R --> S
276 R -. live updates .-> G
277```
278
279Every app is a **package** installed onto this stack — a schema, permission
280rules, and at most one page object, with no app-specific server code. The same
281stack serves the web page, the API, search, files, and the AI.
282
283## The layers, bottom to top
284
285**Storage.** Collections are plain TSV you can `grep`, in one of two modes
286(classic overwrite, or append-only with tombstones + compaction). Each object
287keeps its source, state, logs, files, and version trail next to it.
288See `storage-modes.md`, `append-only-storage-design.md`,
289`durability-and-recovery.md`, `backup-restore.md`.
290
291**Runtime.** A long-running ASGI server executes objects in-process, pooled, or
292in a subprocess. A daemon runs scheduled passes, compaction, and
293trigger→target automation. A websocket change log lets any surface re-render
294live.
295See `runtime-contract.md`, `realtime.md`, `asgi-realtime-direction.md`,
296`object-authoring.md`, `capability-objects.md`.
297
298**Data & schema.** A collection's schema declares its fields — types, enums,
299relations (validated pointers), bounds, defaults, and status transitions — and
300those are enforced on every write, not just in the form.
301See `schema-forms.md`, `validation-and-logic.md`,
302`business-logic-patterns.md`, `logic-decisions.md`.
303
304**Permissions.** One engine decides access for every surface: access modes,
305role/object/action rules, ownership, row and field filters, temporary paid
306access, and record sharing. Routing maps URLs; the policy decides what a
307caller may see or write.
308See `permissions-model.md`, `site-routing.md`, `secrets-and-credentials.md`.
309
310**Generative UI.** One renderer turns a schema into a live
311list/table/board/tree/calendar, a form, and a composed detail page — with
312filters, sort, search, relation labels, and realtime updates — no per-app UI
313code. The design system is a semantic contract, not prescribed widgets.
314See `generative-ui.md`, `design-system.md`, `ui-decisions.md`.
315
316**Capabilities.** Generic per-collection *behaviors* on top of the display
317layer: a schema flag grows a comment thread, attachment list, or owner-checked
318sharing, with no per-app table.
319See `capabilities.md`.
320
321**Access surfaces.** The same records reach a web page, the HTTP API,
322cross-collection search, file up/download, MCP agents, and a talk-to-everything
323shell — all behind the one permission engine.
324See `http-api-contract.md`, `rest-and-object-messages.md`, `shell-and-ai.md`.
325
326**Packages.** Apps are installable packages with a data-preserving upgrade
327system (provenance baselines, three-way reconcile, override objects, feature
328flags).
329See `app-packages.md`, `package-authoring.md`, `upgrade-and-customization.md`.
330
331## Operating it
332
333See `quickstart.md` (fresh VM to first app), `single-vm-deployment.md`,
334`docker-deployment.md`, `traffic-limits.md`, `status.md`.
335
336## Why it's shaped this way
337
338See `why-dbbasic.md` (the advantages, honestly stated) and `comparisons.md`
339(what DBBASIC deletes compared to Django/Rails/no-code tools, and the cost of
340each deletion).
341"
342zero-to-code From Zero Code to Some Code Start Here 130 The staged path every app on the DBBASIC Object Server follows, from a schema with no code at all through hooks, event handlers, verbs, scheduled runners, and hand-written pages. "The question every generative system eventually gets asked: *fine, the
343metadata builds the app — but when I need real code, where does it go?*
344This page is the known path. Every app on this server grows through the
345same stages, in the same order, and the order is the point: **code enters
346through the write path first and the page last** — the exact reverse of
347the classic web framework, where code entered through the controller
348first and that is where the logic rotted.
349
350## Stage 0 — an app with no code at all
351
352A schema (`schemas/notes.json`), permission rules, and two seeded view
353records (an index at `/notes`, a detail at `/notes/{id}`). From those
354alone: list, search, sort, create/edit forms, a detail page, realtime
355updates, the JSON API, MCP tools for agents, and global search. All of it
356reads the schema; none of it is emitted as code
357([`generative-ui.md`](/docs/generative-ui), including why this is *not*
358scaffolding).
359
360`app-notes` ships this way today: a package registering zero objects.
361
362## Stage 1 — logic as declarations (still no code)
363
364Most business rules are not code and never should be:
365
366- shape, bounds, required, enums, FK-exists → **schema validation**
367- who may do what, whose rows are whose → **permission rules**
368- legal state moves and who may make them → **`transitions` + `when`**
369- same-record derivations → **`formula` fields**
370- sums over children → **`rollup` fields**
371
372Every surface enforces these because they live on the shared write path,
373not in a page. A large fraction of real apps end here.
374
375## Stage 2 — the first line of Python is a gate
376
377The first rule that *cannot* be declared is almost always a cross-record
378check: ""a payment may not exceed the invoice balance"", ""a journal must
379balance before posting"", ""a loss move cannot have a destination"". That is
380a **pre-write hook** — one file, one function:
381
382```python
383# objects/hook/payments.py schema: ""hooks"": {""before_write"": ""hook_payments""}
384def BEFORE_WRITE(request):
385 if too_much(request[""record""]):
386 return {""error"": ""..."", ""status"": 409} # reject
387 record = dict(request[""record""]); record[""rate""] = looked_up # or stamp
388 return {""record"": record}
389```
390
391Hooks run on the shared write path, so the form, the API, and an AI agent
392all hit the same gate. They fail closed. This is where code enters the
393system, and it enters *enforcing*, not rendering.
394
395## Stage 3 — reactions
396
397Something should *happen* after a write: a status flips, a journal is
398composed, someone is notified. Notifications are data (`notify_rules`);
399follow-on writes are an **event handler** — a `system/` object with
400`HANDLES = [""payments.record.created""]` and an `EVENT(request)` function.
401Reactions are post-commit and best-effort: they may never block the write
402that already happened ([`logic-decisions.md`](/docs/logic-decisions) #6), and
403anything they generate is idempotent by provenance (#7).
404
405## Stage 4 — verbs
406
407Users (and agents) do things that are not CRUD: *reverse this journal*,
408*apply this count*, *import this statement*, *resolve this bank line*.
409Each is an **action object** — `objects/action/<verb>.py` with a `POST`
410that checks identity, validates, and writes through the same records
411layer. A verb is the honest home for anything a button should do that a
412form cannot.
413
414## Stage 5 — time
415
416Aging, dunning, recurring entries, escalation: time-driven state belongs
417to the **daemon** (#2), as a runner object registered on the scheduler
418(the `/scheduler` board shows every run's outcome) or a message on the
419queue (delay, retry, backoff — already built). A runner is just an object
420with a `POST`; it is callable by hand for testing and by the clock in
421production.
422
423## Stage 6 — presentation code, last of all
424
425Only when a page must *show* something no generic block can express:
426
427- first an **`object` block** — one hand-written panel inside an otherwise
428 generated page (`{""kind"": ""object"", ""object_id"": ""site_my_panel""}`);
429- then, rarely, a **full page object** — `site/notes.py` overrides the
430 generated `/notes` by convention the moment the file exists, and
431 deleting it reverts. Reports are the legitimate case: the trial balance,
432 the statements, the reconciliation tie-out are folds across collections
433 with audit-grade warnings, and they stay hand-written on purpose.
434
435The rule that keeps stage 6 safe: **a page may never be the only place a
436rule exists.** A gate written into a page does not gate the API or an
437agent — it provably does not hold — so enforcement stays in stages 1–2
438where every surface shares it. Pages show; they never decide.
439
440## The whole path at a glance
441
442| stage | you add | it lives in | enforced for |
443|---|---|---|---|
444| 0 | schema + rules + 2 view rows | data | every surface |
445| 1 | declarations (transitions, formulas, rollups) | schema | every surface |
446| 2 | a gate | `hook/` on the write path | every surface |
447| 3 | a reaction | `system/` handler, `notify_rules` | post-commit |
448| 4 | a verb | `action/` | its callers |
449| 5 | a schedule | runner + scheduler/queue | the clock |
450| 6 | a panel, then a page | `object` block → `site/` | presentation only |
451
452## Where this page came from
453
454This path was not designed and then implemented — it was **extracted**. A
455week of building real modules concretely (payments, the books spine, bank
456reconciliation, inventory losses, cash counts) kept producing the same
457placement decisions, and once every kind of business rule had exactly one
458home, the homes sorted themselves into the order an app acquires them.
459The taxonomy is the load-bearing thing; the path is its corollary.
460
461That is also why earlier frameworks never had a page like this. They
462documented their *mechanisms* (models, controllers, middleware, signals)
463but never fixed where *logic* lives — any rule could go anywhere, so there
464was no gradient from zero code to some code, only a cliff. ""Fat models,
465skinny controllers"" was a slogan, not a placement table with enforcement
466behind it. A path only exists where placement is decided.
467
468The runtime twin of this page is [`write-pipeline.md`](/docs/write-pipeline):
469once logic is placed, that page says exactly when it runs, in what order,
470and what can stop a write.
471
472Placement questions beyond this page: the flowchart in
473[`business-logic-patterns.md`](/docs/business-logic-patterns) decides where
474any individual rule goes, and [`logic-decisions.md`](/docs/logic-decisions)
475records why.
476"
477object-authoring Object Authoring Building on DBBASIC 200 How to write DBBASIC objects: source layout, HTTP-style methods, request identity, runtime helpers, response shapes, and public deployment safety. "This guide describes the object shape supported by the current public server release. It's intentionally small, and is designed to stay compatible with the hardened runtime as more of that runtime's functionality moves into this repository over time.
478
479## Source Location
480
481Object source lives under `objects/` by default:
482
483```text
484objects/site/home.py
485objects/basics/counter.py
486objects/users/42/deals.py
487```
488
489The source root can be changed for deployments:
490
491```text
492DBBASIC_OBJECTS_DIR=/var/lib/dbbasic-object-server/objects
493```
494
495Object IDs are derived from paths:
496
497- `objects/site/home.py` becomes `site_home`
498- `objects/basics/counter.py` becomes `basics_counter`
499- `objects/users/42/deals.py` becomes `u_42_deals`
500
501## Methods
502
503An object is a Python file that can define HTTP-style methods:
504
505```python
506def GET(request):
507 return {""status"": ""ok""}
508
509def POST(request):
510 return {""status"": ""created"", ""payload"": request}
511```
512
513`GET` receives query parameters. `POST`, `PUT`, and `DELETE` receive the parsed
514request body plus query parameters where the HTTP contract allows it.
515
516## Request Identity
517
518Every request payload includes a server-controlled `_identity` key describing
519the resolved caller, so objects can render per-user pages and make per-user
520decisions without hand-rolling auth:
521
522```python
523def GET(request):
524 identity = request[""_identity""]
525 if identity[""user_id""] is None:
526 return {""status"": 401, ""error"": ""Sign in first""}
527 return {""hello"": identity[""user_id""], ""roles"": identity[""roles""]}
528```
529
530`_identity` contains `user_id`, `account_id`, `roles`, `subscriptions`, and
531`auth_method` (`admin_token`, `session_token`, `session_cookie`,
532`trusted_headers`, or `anonymous`). The server overwrites any client-supplied
533`_identity` value before the object runs, so it cannot be spoofed through the
534request body or query string. Anonymous requests still receive the key with
535`user_id` set to `None`.
536
537## Runtime Helpers
538
539Loaded object modules receive a small set of runtime helpers. The current public
540runtime injects:
541
542```python
543_state_manager.get(key, default=None)
544_state_manager.set(key, value)
545_state_manager.get_all()
546_state_manager.reload()
547
548_logger.debug(message, **fields)
549_logger.info(message, **fields)
550_logger.warning(message, **fields)
551_logger.error(message, **fields)
552```
553
554State is stored under `data/state/{object_id}/state.tsv`. Logs are stored under
555`data/logs/{object_id}/log.tsv`. The active log is plain TSV. Rotated logs are
556gzip-compressed by default and can still be read by the server.
557
558## Default Storage And Integrations
559
560The default DBBASIC object should not need a separate database. Source, state,
561logs, files, and versions are file-backed so a useful app can start on one VM
562with normal Unix paths and backups.
563
564Objects can still connect to SQL databases, SQLite files, HTTP APIs, AI APIs, or
565other services when a workload needs them. Those integrations should be explicit
566object behavior or package behavior, not hidden platform requirements.
567
568This keeps the base stack small while leaving room for:
569
570- SQL/reporting objects
571- schema and relationship inspectors
572- API connector objects
573- migration objects
574- AI-assisted query and data tools
575
576DBBASIC Scroll can expose SQL, schema, and diagram screens, but those should be
577capabilities of the object system rather than proof that every app needs a
578database tier by default.
579
580Schema still matters. The difference is that DBBASIC should let schemas,
581validation rules, relationships, and generated admin screens live close to the
582objects and packages that use them. That avoids forcing every small app through
583an external database, migration, ORM, admin, and deployment loop before the app
584has proved it needs that weight.
585
586```mermaid
587flowchart LR
588 subgraph DB[""Database-first loop""]
589 D1[""Design schema""] --> D2[""Run migration""]
590 D2 --> D3[""Update model/ORM""]
591 D3 --> D4[""Update route/view/controller""]
592 D4 --> D5[""Deploy""]
593 D5 --> D6[""Inspect errors""]
594 D6 --> D1
595 end
596
597 subgraph OBJ[""DBBASIC object loop""]
598 O1[""Edit object or package metadata""] --> O2[""Run it""]
599 O2 --> O3[""Read output, logs, state, errors""]
600 O3 --> O4[""Fix source or schema metadata""]
601 O4 --> O5[""Keep versions""]
602 O5 --> O1
603 end
604
605 O3 -.-> I[""Optional integrations: SQL, SQLite, HTTP APIs, AI APIs""]
606```
607
608## JSON Object
609
610```python
611def GET(request):
612 count = int(_state_manager.get(""count"", 0)) + 1
613 _state_manager.set(""count"", count)
614 _logger.info(""counter served"", count=count)
615 return {
616 ""status"": ""ok"",
617 ""count"": count,
618 }
619```
620
621Normal dictionaries return JSON.
622
623## HTML Object
624
625```python
626def GET(request):
627 count = int(_state_manager.get(""count"", 0)) + 1
628 _state_manager.set(""count"", count)
629 _logger.info(""home served"", count=count, response_type=""html"")
630
631 html = f""""""<!doctype html>
632<html lang=""en"">
633<head>
634 <meta charset=""utf-8"">
635 <meta name=""viewport"" content=""width=device-width, initial-scale=1"">
636 <title>DBBASIC Object</title>
637</head>
638<body>
639 <h1>DBBASIC Object</h1>
640 <p>This object has been served {count} times.</p>
641</body>
642</html>
643""""""
644
645 return {
646 ""content_type"": ""text/html; charset=utf-8"",
647 ""body"": html,
648 }
649```
650
651Dicts with `content_type` and `body` become raw HTTP responses. This is the
652shape used for pages, generated views, images, and other non-JSON object
653responses.
654
655Plain strings also return `text/html; charset=utf-8`. Plain bytes return
656`application/octet-stream`.
657
658## HTML Form Object
659
660Forms are how design changes ship without a deploy: the page, the handler, and
661the state live in one object. `GET` renders the form; `POST` receives the
662submitted fields (form posts from a signed-in browser carry the session cookie,
663so `request[""_identity""]` identifies the user in both methods):
664
665```python
666def GET(request):
667 identity = request[""_identity""]
668 if identity[""user_id""] is None:
669 return (303, [(""Location"", ""/login?next=/objects/notes_quick"")], b"""")
670
671 saved = _state_manager.get(f""note:{identity['user_id']}"", """")
672 html = f""""""<!doctype html>
673<html lang=""en"">
674<body>
675 <h1>Quick note for {identity['user_id']}</h1>
676 <form method=""post"">
677 <textarea name=""note"">{saved}</textarea>
678 <button type=""submit"">Save</button>
679 </form>
680</body>
681</html>""""""
682 return {""content_type"": ""text/html; charset=utf-8"", ""body"": html}
683
684
685def POST(request):
686 identity = request[""_identity""]
687 if identity[""user_id""] is None:
688 return {""status"": 401, ""error"": ""Sign in first""}
689
690 _state_manager.set(f""note:{identity['user_id']}"", str(request.get(""note"", """")))
691 _logger.info(""note saved"", user_id=identity[""user_id""])
692 return (303, [(""Location"", ""/objects/notes_quick"")], b"""")
693```
694
695Cookie-authenticated form posts are origin-checked by the server (same-site
696only), so objects do not need their own CSRF plumbing for basic forms. Records
697and schemas are the step up from object state when the data needs listing,
698validation, and change history — see the collections sections of
699[`http-api-contract.md`](/docs/http-api-contract).
700
701## Upload Forms
702
703`multipart/form-data` posts (forms with `enctype=""multipart/form-data""`) are
704parsed for you: text fields arrive as plain payload keys, and each uploaded
705file arrives under `request[""_files""]` with its content base64-encoded so the
706payload survives the subprocess execution boundary:
707
708```python
709import base64
710
711def POST(request):
712 upload = request[""_files""][""document""]
713 content = base64.b64decode(upload[""content_base64""])
714 _logger.info(""upload received"", filename=upload[""filename""], size=upload[""size""])
715 return {""saved"": upload[""filename""], ""bytes"": len(content)}
716```
717
718Each `_files` entry has `filename`, `content_type`, `size`, and
719`content_base64`. Upload size is bounded by `DBBASIC_MAX_REQUEST_BYTES`.
720
721An object that receives an upload can process it with system tools —
722OCR, transcode, thumbnail, extract — by shelling out to binaries like
723`tesseract` or `ffmpeg`. See [capability objects](/docs/capability-objects)
724for the subprocess model, worked examples, and the trust boundary.
725Store durable uploads through the object file APIs so they inherit
726permissions and change history.
727
728## Low-Level Response
729
730When an object needs exact status and headers, return a tuple:
731
732```python
733def POST(request):
734 return (201, [(""Content-Type"", ""text/plain"")], [b""created""])
735```
736
737The tuple shape is:
738
739```text
740(status, headers, body)
741```
742
743`body` may be bytes, text, or a list of bytes/text parts.
744
745## Authoring Rule
746
747Keep each object small enough to inspect, run, log, fix, and version quickly.
748The point of DBBASIC is the short loop:
749
750```mermaid
751flowchart LR
752 A[""edit object""] --> B[""run""]
753 B --> C[""read output, logs, state, errors""]
754 C --> D[""fix""]
755 D --> E[""keep version""]
756 E --> A
757```
758
759Objects can call other objects, emit events, handle jobs, serve pages, or expose
760APIs, but the first rule is the same: make the running object easy to inspect.
761
762## Public Deployment Safety
763
764Until role, object, and row-level permissions are enforced, do not expose the
765full `/objects` API on a public hostname.
766
767For public staging:
768
769- keep `DBBASIC_ENABLE_SOURCE_WRITES=false`
770- generate a deployment-specific `DBBASIC_ADMIN_TOKEN` outside the source tree
771- bind uvicorn to `127.0.0.1`
772- allowlist explicit public object routes in a reverse proxy
773- keep object source, state, logs, versions, and object listing private from
774 normal public users
775"
776package-authoring Package Authoring Building on DBBASIC 210 Practical guide to building and installing a DBBASIC package: manifest structure, install semantics, navigation and attention registries, and the authoring workflow. "This is the practical guide for building a DBBASIC package: one installable
777directory that ships an app piece — objects, schemas, and seed data — without
778a deploy pipeline. It is written for humans and AI tools that generate
779packages. The runtime contract details live in `runtime-contract.md#packages`;
780this page is the authoring workflow.
781
782## What A Package Is
783
784A package is a directory under `packages/` (or `DBBASIC_PACKAGES_DIR`) with a
785manifest and package-relative content:
786
787```text
788packages/{package_id}/
789 dbbasic-package.json
790 objects/ # object source files
791 schemas/ # collection schema JSON files
792 seed/ # initial records TSV files
793 permissions/ # declared, not yet installed (merge semantics pending)
794 migrations/ # declared, not yet run (run semantics pending)
795```
796
797A good package is one app primitive: contacts, notes, tasks, articles,
798invoices. Small enough to review, complete enough to install and use.
799
800## The Manifest
801
802`dbbasic-package.json` is plain JSON:
803
804```json
805{
806 ""id"": ""contacts"",
807 ""name"": ""Contacts"",
808 ""version"": ""0.1.0"",
809 ""description"": ""Contact records with a browse/edit page"",
810 ""compatibility"": {""dbbasic_object_server"": "">=0.1.0""},
811 ""dependencies"": [],
812 ""objects"": [
813 {""id"": ""contacts_directory"", ""path"": ""objects/contacts/directory.py""}
814 ],
815 ""schemas"": [
816 {""collection"": ""contacts"", ""path"": ""schemas/contacts.json""}
817 ],
818 ""seed"": [
819 {""collection"": ""contacts"", ""path"": ""seed/contacts.tsv""}
820 ],
821 ""permissions"": [],
822 ""migrations"": [],
823 ""schedules"": [
824 {""id"": ""contacts_dedupe_nightly"",
825 ""object_id"": ""contacts_directory"",
826 ""schedule"": ""10 6 * * *"",
827 ""description"": ""Flag likely duplicate contacts overnight.""}
828 ],
829 ""nav"": [
830 {""id"": ""contacts"", ""label"": ""Contacts"", ""path"": ""/contacts"",
831 ""blurb"": ""People, the organizations they belong to, and interactions"",
832 ""surface"": ""member"", ""group"": ""Work"", ""order"": 60}
833 ]
834}
835```
836
837Rules the server enforces:
838
839- `id` must be a safe package identifier; object ids and collection names are
840 validated with the same rules as the rest of the server.
841- All paths are package-relative. Absolute paths, null bytes, and `..`
842 traversal are rejected at parse time.
843- Unknown manifest sections are rejected rather than ignored.
844
845## What Install Actually Does Today
846
847Installs are deliberately conservative:
848
849- `objects` entries are written under the configured objects root. Existing
850 objects require `allow_replace`.
851- `schemas` entries are validated and written under `data/schemas/`. Existing
852 schemas require `allow_replace`.
853- `seed` TSV is **install-once**: it is written only when the target
854 collection has no records file yet. On a reinstall or upgrade of a package
855 whose collection already holds data, seeding is skipped (reported as
856 `status: ""skipped""`, `action: ""skip""`) — it never overwrites live data, and
857 it never blocks the install. This is what makes an in-place upgrade safe:
858 reinstall with `allow_replace` to ship new object code, a migrated schema,
859 and re-merged permissions while every existing record is preserved. Live
860 records live outside the package, in `data/collections/<name>/records.tsv`
861 keyed by collection, so an upgrade has no reason to touch them. (Evolving a
862 schema across an upgrade — backfilling a newly added field on old rows — is
863 the separate `migrations` story below, not yet implemented; adding a
864 read-only/server-set field like `created_at` is safe because old rows simply
865 read as empty for it.)
866- `permissions` entries MERGE: each fragment file is
867 `{""rules"": [ ... ]}` using the same rule shape as the policy document
868 (`effect`, `principal`, `actions`, optional `collection`/`object_id`,
869 row filters, field lists). On install the rules are validated, stamped
870 with `""package"": ""<package_id>""` provenance, and appended to the
871 deployment policy — skipping any rule that already exists, so reinstalls
872 are idempotent. This is how a site package makes its own pages public:
873 ship the grant with the code instead of hand-editing policy after install.
874 Invalid fragments block the whole install.
875- `migrations` are accepted in the manifest and reported in dry-runs, but
876 install rejects them until explicit run semantics land.
877- `schedules` become `task_<id>` rows in the scheduler trigger's state,
878 which is where `object_daemon.process_scheduler` reads the board from.
879 **If your app needs a recurring pass, declare it here.** A schedule that
880 exists only in a running server's state is invisible to review, absent
881 from a fresh install, and gone the next time the box is rebuilt — that
882 is precisely how a demo server ended up with four nightly passes that
883 appeared nowhere in the package sources. Fields: `id` (becomes the state
884 key, `[a-z][a-z0-9_]*`), `object_id`, `schedule`, and optional `type`
885 (`cron` — the default — or `onetime`), `method` (default `POST`),
886 `payload`, `description`.
887
888 Three properties make declaring a schedule safe to do repeatedly:
889
890 - **Run history survives.** `last_run`, `run_count` and `next_run`
891 belong to the daemon and are never reset by an install. An upgrade
892 that forgot when a pass last worked would make ""is this actually
893 running?"" unanswerable.
894 - **A pause is honoured.** The package declares what *should* run; an
895 operator decides what *does* right now. A reinstall never restarts a
896 task somebody deliberately paused.
897 - **A changed expression reschedules.** Editing the cron clears
898 `next_run` so the daemon recomputes it; leaving it alone keeps
899 tonight's firing exactly where it was, so shipping a patch does not
900 skip a run.
901
902 Two things are refused outright, at install time rather than at 3am:
903 an unparseable cron (the daemon reads that as ""no next run"" and moves
904 on in silence — a pass that never runs and never complains), and a
905 schedule aimed at an object neither this package nor the server
906 provides. Removing a schedule from a manifest does **not** delete the
907 task, matching the rest of install: deregistering never destroys.
908- `nav` entries become rows in the `nav_entries` collection, which every
909 navigation surface folds over. **If your app serves a page, declare its
910 door here.** See the section below.
911- `attention` entries become rows in the `attention_sources` collection,
912 which the daemon's rollup pass reads its list of providers from.
913 **If your app has a queue that waits on a person, declare it here.**
914 See the section below.
915- The HTTP install route creates a restore point first and appends changelog
916 rows under `data/package_changes/{package_id}/changes.jsonl`.
917
918## Navigation (`nav`)
919
920A menu maintained by hand rots the moment somebody ships without editing
921it, and the rot is invisible — nothing fails, the front door just
922advertises a server that no longer exists. One deployment of this system
923accumulated three separate lists of the same apps: an app switcher's JS
924array (25 entries), a home page's tile grid (21 entries), and a
925collection-to-URL map for search hits. They disagreed with each other and
926between them named none of the eight newest apps. Worse, the home page's
927tile grid existed in no package at all — it lived only on a running
928server, so a rebuilt box came back with no front door and said nothing
929about it, which is precisely the failure the `schedules` section was
930built to end for cron.
931
932So a door is declared by the app that owns it, and every navigation
933surface becomes a fold over one registry:
934
935```json
936""nav"": [
937 {""id"": ""shop"", ""label"": ""Shop"", ""path"": ""/shop"",
938 ""blurb"": ""Browse what is for sale and buy it"",
939 ""surface"": ""public"", ""group"": ""Commerce"", ""order"": 400}
940]
941```
942
943- `id` becomes the `nav_entries` record id, so `[a-z][a-z0-9_]*`. It must
944 be unique across every installed package: two packages writing one row
945 is two packages fighting over one door.
946- `label` and `path` are required. `path` starts with `/` and holds no
947 whitespace.
948- `blurb` is one honest line in the present tense saying what the app
949 does — it is rendered under the label on the home grid. No marketing.
950- `surface` is `public` | `member` | `operator` | `hidden`, default
951 `member`. This governs the MENU and is not access control: permissions
952 decide who may open a page, and leaving a door off the menu hides
953 nothing that was not already protected. `/shop` and the public
954 product/pay/portal pages are `public`; ordinary app pages are `member`;
955 `/scheduler`, `/flow`, `/urls`, `/dashboard` and other admin or debug
956 pages are `operator`.
957- `group` is free text, default `Apps`. The house groups are **Work**,
958 **Publishing**, **Money**, **Commerce**, **Warehouse** and **System**.
959- `order` is an integer, default `100`. Packages band it by group (Work
960 in the 0s, Publishing the 200s, Money the 300s, Commerce the 400s,
961 Warehouse the 500s, System the 600s) because a group sorts where its
962 earliest entry sorts — that way the reading order of the groups falls
963 out of the same number, instead of a seventh list holding it.
964
965Three properties make declaring a door safe to do repeatedly:
966
967- **The operator's `hidden` survives.** `operator_hidden` is a separate
968 boolean column and an install never writes it on an existing row. The
969 package says what the app IS; the operator says what they want to SEE.
970 An upgrade that silently put back a page somebody deliberately removed
971 would be the same class of incident as one that restarts a paused
972 nightly pass — so an operator flips `operator_hidden` (a `role:manager`
973 update on `nav_entries`) instead of editing somebody else's manifest.
974- **Unchanged means unwritten.** Every package-owned field — label, path,
975 blurb, group, surface, order — is compared before anything is written,
976 so a plan that reports `unchanged` is one where nothing at all is about
977 to happen. A dry run that under-reports is worse than no dry run.
978- **Removing an entry does not delete the row.** Same as the rest of
979 install: deregistering never destroys.
980
981Two things are refused, and one deliberately is not:
982
983- A **duplicate id inside one manifest**, and an **invalid surface, path
984 or order**, fail at parse time.
985- A **collision with another package** blocks the install: an id already
986 registered by a different package, or a path a different package
987 already claims. Whichever installed last would otherwise silently win,
988 and the loser's menu entry would point somewhere its own package never
989 chose.
990- Whether a path is actually **served** is NOT checked. Routing has three
991 sources (convention, `site_routes` records, `views` records) and two of
992 them are data the same install may be about to seed, so resolving a
993 route here would produce false blockers. The registry records the door
994 the package intends; `/urls` is where you go to see what the server
995 actually answers.
996
997If the `nav_entries` collection does not exist yet (app-nav is not
998installed), nav entries are reported as `skipped` with a reason and the
999install proceeds. A package must never fail because the navigation app is
1000absent. The corollary is an ordering note for a fresh box: **install
1001`app-nav` first**, or reinstall afterwards, since entries declared before
1002the registry exists are skipped rather than queued.
1003
1004### The opt-out
1005
1006`tests/test_package_nav.py` asserts that every package shipping a `site_*`
1007object or a `site_routes` seed declares at least one nav entry. That test
1008is the thing that stops the drift coming back. Some packages legitimately
1009serve a URL that is not a door — a mounted widget (`site_thread`,
1010`site_share`), a fragment inside another page
1011(`site_materialize_run_button`), a POST endpoint (`site_timer_actions`),
1012or a per-record document reached from the record itself
1013(`site_packing_slip`, `site_receiving_sheet`, `site_return_form`). Those
1014declare:
1015
1016```json
1017""nav_optional"": true
1018```
1019
1020which makes ""this app has no front page"" a reviewable claim in the
1021manifest rather than an omission nobody notices.
1022
1023## Attention (`attention`)
1024
1025A menu answers ""where can I go"". The only question a home screen is
1026actually asked is **does anything need me?**, and this server is unusually
1027well placed to answer it, because it is built almost entirely out of gates
1028and derived states. Every one of them produces a queue of things a machine
1029deliberately refused to decide: scans sitting at `extracted` waiting to be
1030confirmed into expenses, time entries `submitted` waiting for an approver
1031who is by design not the person who logged them, invoices past their due
1032date, bank lines nothing matched, POs a supplier shorted, orders committed
1033and not yet in a box, returns that arrived and were never judged.
1034
1035The system already computes all of it, on a schedule, and throws it away
1036at the end of each pass. `system_scan_processor` has been returning a
1037field literally named `needing_a_human` since the day it was written and
1038no surface has ever read it. The arithmetic was never missing; a place to
1039declare that the arithmetic *means* something was — and that declaration
1040can only honestly live in the package that owns the domain, because
1041nobody else knows that `extracted` is a waiting room and `confirmed` is
1042not.
1043
1044So it is declared, third use of the pattern that already fixed cron
1045(`schedules`) and navigation (`nav`):
1046
1047```json
1048""attention"": [
1049 {""id"": ""scans_to_confirm"", ""object_id"": ""system_scan_attention"",
1050 ""label"": ""Receipts to confirm"", ""path"": ""/scans?status=extracted"",
1051 ""nav_id"": ""scans"", ""group"": ""Money"", ""severity"": ""normal""}
1052]
1053```
1054
1055- `id` becomes the `attention_sources` record id, so `[a-z][a-z0-9_]*`,
1056 and it must be unique across every installed package.
1057- `object_id` names the provider (below). Required, and checked: a
1058 counter aimed at an object neither this package nor the server provides
1059 **blocks the install**, exactly as a schedule does. A counter pointing
1060 at nothing reads zero forever, and a queue that reads zero forever is
1061 indistinguishable from a queue that is empty — the silent failure this
1062 whole section exists to end.
1063- `label` is what a person calls the queue, in their words: ""Receipts to
1064 confirm"", not ""scans.status=extracted"".
1065- `path` is where the count clicks through to. Required, starts with `/`,
1066 no whitespace. **A number that cannot be acted on in one click does not
1067 belong on a home page.** Whether anything actually serves the path is
1068 not checked, for the same reason `nav` does not check it.
1069- `nav_id` is optional: the `nav_entries` row this count decorates, so the
1070 app list itself reads ""Invoices — 5 overdue"" with no second surface to
1071 maintain. Omit it when the package ships no door at all (app-intake).
1072- `group` is free text, default `Apps`, and uses the same house groups as
1073 `nav`.
1074- `severity` is `normal` | `warning` | `urgent`, default `normal`, and is
1075 the first sort key on the home band. `normal` is work waiting;
1076 `warning` is work that is already late (an overdue invoice); `urgent`
1077 is something broken (a scheduler pass that failed). Three steps,
1078 because a scale with more is a scale nobody calibrates.
1079
1080### The COUNT contract
1081
1082A provider is an ordinary object exposing:
1083
1084```python
1085def COUNT(request):
1086 return {""count"": 3, ""detail"": ""1 over a week old""}
1087```
1088
1089`COUNT` and not `GET` or `POST`, deliberately: a distinct verb means the
1090same object stays free to answer a human on the other methods without
1091either use accidentally serving the other. `count` must be a real
1092non-negative integer — the rollup refuses to coerce a string or a float,
1093because every coercion here is a way for a broken provider to look like
1094an empty queue. `detail` is optional and is the sentence a bare number
1095cannot say (""62 hours awaiting approval"", ""18 units still owed"").
1096
1097A provider reads its own collections and **degrades to `{""count"": 0}`
1098when they are absent** — a deployment that never installed your app
1099should not produce an error every five minutes. A provider that genuinely
1100fails should raise; see below for what happens then.
1101
1102### Counts are a rollup, not a live fold
1103
1104`object_daemon.process_attention` executes every declared provider on an
1105interval (`DBBASIC_ATTENTION_INTERVAL_SECONDS`, default 300) and writes
1106the answers to the `attention_counts` collection. The home page and the
1107app list read that one small table.
1108
1109This is not an optimisation, it is a rule. Folding a dozen collections on
1110every home render is how a home page becomes the slowest page, and worse:
1111one box went to 675MB resident and started swapping because something
1112folded a big collection on a timer. A home screen that folds a dozen
1113collections per render is that same mistake with a nicer name, and it
1114makes the front page's cost grow with the number of installed packages —
1115the one place cost must not grow. Staleness is bounded by the interval
1116and **stated on the page** (""as of 6 minutes ago"") rather than pretended
1117away.
1118
1119Two consequences worth knowing when you write a provider:
1120
1121- **A broken provider must look broken, never look calm.** One that
1122 raises, or answers with a non-integer count, has its `error` written
1123 and its *previous* count kept, with `computed_at` left where it was.
1124 Zeroing it would report an empty queue when what happened is that
1125 nobody looked.
1126- **Do not do expensive work per row.** The pass runs every five minutes
1127 on the same box that serves requests. A full scan of your own
1128 collections is the intended cost; calling out to a network is not.
1129
1130### Zeros disappear
1131
1132A count of zero is stored (the absence of a row means the provider has
1133never run, which is a different fact) and **rendered nowhere**. When
1134every queue is empty the home page's ""Needs you"" band is absent entirely
1135— not ""0 pending"", not an empty panel with a heading: gone.
1136
1137This is the rule that separates an inbox from a dashboard. A board that
1138says ""0 pending approvals"" every day trains people to stop reading it,
1139and once they have stopped they do not start again on the day it says
1140three. A page that is EMPTY when nothing needs them, and short when
1141something does, gets read every time. ""Nothing needs you"" is a complete
1142and welcome answer, and blank space gives it better than any sentence
1143would.
1144
1145### Idempotency and the operator
1146
1147The same three properties `nav` has:
1148
1149- **The operator's `operator_muted` survives.** A deployment that does
1150 not work a queue mutes it (a `role:manager` update on
1151 `attention_sources`) instead of editing somebody else's manifest, and
1152 an install never writes that column. A muted source is not executed at
1153 all and its stored count is removed, so it costs nothing and shows
1154 nothing.
1155- **Unchanged means unwritten.** Every package-owned field —
1156 `object_id`, `label`, `path`, `nav_id`, `group`, `severity` — is
1157 compared before anything is written.
1158- **Removing an entry does not delete the row.** Deregistering never
1159 destroys.
1160
1161A **collision with another package** on an `id` blocks the install: two
1162packages writing one row is two definitions of what needs a human, and
1163whichever installed last would silently win. Unlike `nav` there is no
1164path-collision rule — two queues legitimately point at one list, and
1165blocking that would be a false blocker.
1166
1167If the `attention_sources` collection does not exist (app-nav is not
1168installed) the entries are reported as `skipped` with a reason and the
1169install proceeds, exactly as `nav` entries are.
1170
1171## Authoring Workflow
1172
11731. Create the package directory and manifest.
11742. Write the objects (see [`object-authoring.md`](/docs/object-authoring) for method shape, state, logs,
1175 `request[""_identity""]`, and HTML form patterns).
11763. Write each schema JSON (`{""fields"": [{""name"": ""id""}, ...]}` — see the
1177 schemas section of [`http-api-contract.md`](/docs/http-api-contract) for validation and field
1178 permission options).
11794. Seed TSV: first row is the header, tab-separated, and must include `id`.
11805. Dry-run before any install:
1181
1182 ```http
1183 GET /packages/{package_id}?dry_run=true
1184 Authorization: Token <token>
1185 ```
1186
1187 The plan reports what would be created, replaced, merged, applied, or
1188 skipped, with warnings. Nothing is written.
11896. Install (requires `DBBASIC_ENABLE_PACKAGE_INSTALLS=true` plus the admin
1190 gate):
1191
1192 ```http
1193 POST /packages/{package_id}/install
1194 Authorization: Token <token>
1195 ```
1196
11977. Verify: run the objects, read the records, check
1198 `GET /packages/{package_id}/changes` for the install changelog, and use
1199 `POST /packages/{package_id}/restore` if the install needs to be rolled
1200 back (requires `DBBASIC_ENABLE_PACKAGE_RESTORE=true`).
1201
1202## A Minimal Working Example
1203
1204`packages/hello-world/` in this repository is the smallest complete package.
1205`packages/admin-write-probe/` is the smallest complete *app-shaped* package:
1206one HTML page object, one schema, one seed file, exercising records end to
1207end. `packages/app-projects/` and `packages/app-notes/` are the reference
1208*user apps*: an owner-scoped schema with `search` metadata, permission
1209rules granting signed-in users their own rows (`row_filter` on
1210`$user_id`), and a signed-in page that reads and writes records with the
1211visitor's session cookie. Start a new app package from `app-notes`.
1212
1213`app-notes` also shows the public/private pattern: a boolean `is_public`
1214field plus a second permission rule (`principal: public`, `row_filter:
1215{""is_public"": ""true""}`) makes shared records readable by anyone across
1216every surface — pages, records API, MCP, and search — with no
1217visibility code in the app itself. Its permalink page (`site_note_view`)
1218expects a `site_routes` record mapping `/notes/{note_id:uuid}` to it;
1219route records are site data, so each deployment adds them next to its
1220other routes.
1221
1222Object files inside packages are normal DBBASIC objects. A package page that
1223renders per-user content reads `request[""_identity""]`; a package form posts to
1224its own object id and writes records through the object runtime. Signed-in
1225browser users carry their session cookie automatically.
1226
1227## Rules For Generated Packages
1228
1229- Never put secrets, tokens, real hostnames, or private paths in package
1230 content — packages are meant to be shareable source.
1231- Keep one package per app primitive; use `dependencies` to declare (not yet
1232 resolve) cross-package needs.
1233- Bump `version` on any content change so install changelogs and restore
1234 points stay meaningful.
1235- Prefer seed data that demonstrates the schema without pretending to be real
1236 data.
1237- Dry-run output is the review artifact: a package that cannot explain itself
1238 in a dry-run plan is not ready to install.
1239"
1240schema-forms Schema Field Contract: Generated Forms and Views Building on DBBASIC 220 How one schema document drives desktop forms, web forms, MCP agents, and record-write validation from the same declared field semantics. "This is the contract that lets user interfaces generate themselves from schemas. One schema drives every surface at once: desktop forms, a server-rendered web form, an agent reading constraints over MCP, and the record-write APIs enforcing the same rules for all of them.
1241
1242The governing principle: **schemas declare semantics, never widgets.** A field says ""one of these values"" or ""points at a projects record"" — each interface maps that meaning onto its own idiom (a dropdown, a picker, a JSON constraint, a spoken prompt). New kinds of interface plug in later without touching the data.
1243
1244Everything here is enforced by the server on record writes today. None of it is a DSL — it is keys in the schema JSON document, checked by explicit code, versioned and rolled back through the normal schema history.
1245
1246## Field Keys
1247
1248```json
1249{
1250 ""name"": ""urgency"",
1251 ""type"": ""enum"",
1252 ""required"": true,
1253 ""label"": ""Urgency"",
1254 ""help"": ""How soon this needs attention"",
1255 ""placeholder"": ""Choose…"",
1256 ""default"": ""normal"",
1257 ""enum"": [""low"", ""normal"", ""high"", ""critical""]
1258}
1259```
1260
1261- `name` — required; letters/digits/underscores.
1262- `type` — `text` (default), `textarea`, `integer`, `number`, `boolean`, `date`, `datetime`, `enum`, `computed`. Types are validated on write (integers must parse, dates must be ISO, enums must match, and so on). Boolean fields accept several spellings on input (`True`, `1`, `yes`) but are stored canonically as `true`/`false`, so permission row filters like `{""is_public"": ""true""}` match reliably.
1263- `required` — empty values rejected on write.
1264- `label`, `help`, `placeholder` — presentation text for generated forms.
1265- `default` — filled in when the field is not submitted.
1266- `enum` — allowed values; writes outside the list are rejected. Renders as a dropdown/segment control in form UIs.
1267- `computed` / `read_only` — the server rejects client writes to these fields; forms render them display-only.
1268- `transitions` — for lifecycle fields, which values each current value may move to. Enforced on record update; values missing from the map are terminal. This is data plus one check, not a state machine framework — no hooks, no callbacks, no side effects:
1269
1270 ```json
1271 {""name"": ""status"", ""type"": ""enum"",
1272 ""enum"": [""open"", ""assigned"", ""done""],
1273 ""transitions"": {""open"": [""assigned""], ""assigned"": [""done"", ""open""]}}
1274 ```
1275
1276 Form UIs can offer only the moves allowed from the record's current value; the server rejects everything else either way.
1277
1278 A list entry can also be a guarded object instead of a plain string, adding *who* on top of *whether*:
1279
1280 ```json
1281 {""transitions"": {
1282 ""assigned"": [{""to"": ""open"", ""when"": {""assigned_to"": ""$user_id""}}]
1283 }}
1284 ```
1285
1286 The move is allowed only once every `when` clause matches the record's current stored values against the resolved subject variable (`$user_id`, `$account_id`, `$accessible_projects`, `$owned_projects`, `$writable_projects`) or a literal string — the same closed set and matching rules row filters use (see `permissions-model.md`). Guards are only evaluated where a request subject is available (the HTTP update path); direct library callers still get the plain validity check (the move's `to` must be in the list) but not the guard.
1287- `relation` — a validated pointer to another collection:
1288
1289 ```json
1290 {""name"": ""project_id"", ""relation"": {""collection"": ""projects"", ""display_field"": ""name""}}
1291 ```
1292
1293 On write, the value must be an existing record id in that collection (`projects/p999` missing → 400). `display_field` tells form UIs which field of the target record to show in pickers. A bare string (`""relation"": ""projects""`) is shorthand. This is deliberately a pointer plus a display hint, not an association framework — no joins, no lazy loading, no cascades.
1294- `validation` — explicit bounds, all enforced on write:
1295
1296 ```json
1297 {""validation"": {""min_length"": 3, ""max_length"": 80, ""pattern"": ""^[a-z-]+$"", ""min"": 0, ""max"": 100}}
1298 ```
1299
1300 `pattern` is **full-matched** against the whole value (`re.fullmatch`), not searched — so it must describe the entire field, not a prefix. A URL check is `^https?://\S+$`, not `^https?://` (that would only match the literal scheme and reject every real URL). The `^`/`$` anchors are therefore redundant but fine to keep for readability.
1301
1302## Schema Root Keys
1303
1304Presentation metadata lives beside the fields and versions with them — a ""Save Layout"" action in a desktop client persists here, so form design has history and rollback like source code:
1305
1306```json
1307{
1308 ""fields"": [...],
1309 ""forms"": {
1310 ""default"": {""fields"": [""title"", ""description"", ""project_id"", ""urgency"", ""due_date"", ""assigned_to""]}
1311 },
1312 ""views"": {
1313 ""list_mode"": ""table"",
1314 ""list_fields"": [""title"", ""urgency"", ""due_date"", ""assigned_to""]
1315 }
1316}
1317```
1318
1319- `forms.default.fields` — field order for the generated record form. Additional named forms can define alternate layouts.
1320- `views.list_mode` — how the generated list renders. The shared list renderer (`window.dbbasicList`) resolves it from the schema:
1321 - `table` — a dense, sortable HTML table over `list_fields`; cells format by type (money `_cents`, boolean, relative dates, enum badges), and relation columns show the target's label, not the raw id.
1322 - `board` — a kanban grouped by an enum field, drag-to-transition (the drag issues the ordinary status write, so `transitions` rules still gate it). A board collection that also has `list_fields` gets a **Board ⇄ Table** toggle (the choice persists per collection).
1323 - `tree` — nests a self-relation (`parent_id`) into a hierarchy.
1324 - `calendar` — buckets by a date field.
1325 - absent / `cards` / `feed` — the plain rich-row list.
1326
1327 Every mode shares one fetch/sort/cap/search/realtime pipeline, so all of them inherit filtering, the 50-row cap, and live updates. A mode whose required field can't be derived falls back to the row list with a visible notice — never a blank page.
1328- `views.list_fields` — columns (table), card fields (board), or summary fields (rows). Relation fields resolve to the referenced record's label.
1329- `views.filter_fields` — fields to expose as a **filter bar** above the list (enum → a select of its options, boolean → Yes/No). A pick narrows the fetch server-side (`field=value`, after the permission row filter) and composes with the search box; works in every mode.
1330- `search` — opts the collection into global search (`GET /api/search` and the `global_search` MCP tool):
1331
1332 ```json
1333 {""search"": {""fields"": [""title"", ""content""], ""result_fields"": [""id"", ""title""]}}
1334 ```
1335
1336 `fields` are the searchable fields; `result_fields` (optional) trims what each search hit returns, defaulting to `id` plus `views.list_fields`. Collections without a `search` section never appear in search results. Search runs inside the permission engine, so row filters and field permissions bound what any caller can find.
1337- `formula` / `rollup` (on a `type: computed` field) — derived values, materialized on write: a formula computes from sibling fields (`""first_name + \"" \"" + last_name""`); a rollup aggregates a child collection (`{""collection"": ""fin_journal_lines"", ""fk_field"": ""journal_id"", ""op"": ""sum"", ""field"": ""debit_cents""}`). Stored like any field, so every view/filter/sort sees them; kept current by the write path. See `validation-and-logic.md`.
1338- `capabilities` — generic per-collection *behaviors* on top of this display layer: `{""comments"": true}`, `{""attachments"": true}`, `{""shareable"": true}` grow a comment thread, attachment list, and owner-checked sharing on the detail page from one flag each. See `capabilities.md`.
1339
1340## Worked Example: tasks
1341
1342The acceptance test for this contract is a ""New Task"" screen, expressed entirely as data:
1343
1344```json
1345{
1346 ""fields"": [
1347 {""name"": ""id""},
1348 {""name"": ""title"", ""type"": ""text"", ""required"": true, ""label"": ""Title"",
1349 ""validation"": {""min_length"": 1, ""max_length"": 200}},
1350 {""name"": ""description"", ""type"": ""textarea"", ""label"": ""Description""},
1351 {""name"": ""project_id"", ""label"": ""Project"",
1352 ""relation"": {""collection"": ""projects"", ""display_field"": ""name""}},
1353 {""name"": ""urgency"", ""type"": ""enum"", ""label"": ""Urgency"", ""default"": ""normal"",
1354 ""enum"": [""low"", ""normal"", ""high"", ""critical""]},
1355 {""name"": ""due_date"", ""type"": ""date"", ""label"": ""Due Date""},
1356 {""name"": ""assigned_to"", ""label"": ""Assigned To"",
1357 ""relation"": {""collection"": ""users"", ""display_field"": ""display_name""}}
1358 ],
1359 ""forms"": {""default"": {""fields"": [""title"", ""description"", ""project_id"",
1360 ""urgency"", ""due_date"", ""assigned_to""]}},
1361 ""views"": {""list_mode"": ""table"",
1362 ""list_fields"": [""title"", ""urgency"", ""due_date"", ""assigned_to""]}
1363}
1364```
1365
1366A form renderer walks `forms.default.fields`, looks up each field's semantics, and emits the right control. The record POST enforces every rule the form promised — and the same rules bind a record created over MCP or raw HTTP, so no surface can bypass what another surface displays."
1367business-logic-patterns Business Logic Patterns: Where Every Rule Goes Building on DBBASIC 230 A placement guide and decision procedure for where each kind of business rule belongs in the primitive stack, illustrated with worked examples from tax, aging, dunning, refunds, inventory, and bank reconciliation. "Every business rule has one correct home in the primitive stack. This page is the placement guide: the decision procedure, the map, and worked examples (sales tax, payment aging, dunning, refunds, inventory shrinkage, bank reconciliation) that most systems get wrong.
1368
1369The method behind all of it: build each module's rules **concretely** in the cheapest correct primitive — a hook is three lines of real code — and only promote a pattern to a declarative schema key after it has repeated in two or three real modules. Concrete → pattern → primitive, never primitive-first. Systems that build the workflow/rules engine first die when reality doesn't fit it; this one extracts engines from reality.
1370
1371## Where does this rule go?
1372
1373```mermaid
1374flowchart TD
1375 Q[""a business rule""] --> D1{""Derivable from<br/>stored data?""}
1376 D1 -->|same record| F[""formula field""]
1377 D1 -->|child records| R[""rollup field""]
1378 D1 -->|no| D2{""Does it GATE<br/>a write?""}
1379 D2 -->|shape / bounds / FK| V[""schema validation""]
1380 D2 -->|legal state moves + who| T[""transitions + when guards""]
1381 D2 -->|cross-record or computed gate| H[""pre-write hook<br/>(reject or stamp)""]
1382 D2 -->|no| D3{""Does TIME<br/>cause it?""}
1383 D3 -->|yes| DM[""daemon: scheduled pass /<br/>auto-transition""]
1384 D3 -->|no| D4{""Is it a REACTION<br/>to a change?""}
1385 D4 -->|notify / email| N[""notify_rules (after write)""]
1386 D4 -->|run code| TR[""trigger → target object""]
1387 D4 -->|external system| C[""connector<br/>(desired state + reconcile)""]
1388 D4 -->|no| O[""plain object<br/>(page / API / worker)""]
1389```
1390
1391## The placement map
1392
1393| Kind of rule | Primitive | Lives in | Example |
1394|---|---|---|---|
1395| Shape, type, bounds, enum, FK-exists | schema validation | `object_records` write path | amount is integer cents; project must exist |
1396| Same-record derivation | `formula` (materialized) | `object_computed` via `object_records` | `tax_cents = amount_cents * tax_rate` |
1397| Aggregation over children | `rollup` (materialized, live) | same | `invoice.paid_cents = sum(payments)` |
1398| Legal state moves, and who may move | `transitions` + `when` | `object_records` | `draft → posted`, owner only |
1399| Cross-record gate, computed gate, point-in-time stamp | **pre-write hook** | `hooks.before_write` → `object_server` write path | journal must balance; stamp `tax_rate` |
1400| Time-driven state | daemon scheduled pass / `after_hours` auto-transition | `object_daemon` | `sent → overdue` when past due |
1401| Reaction: notify, email | `notify_rules` | daemon, **after** the write | dunning email on `overdue` |
1402| Reaction: run code | trigger → target object | daemon | recompute, follow-on records |
1403| External side effect | connector | desired-state records + reconcile | payment-provider refund call |
1404| Correction to money / inventory | append-only compensating record | `storage: append` collections | refunds, stock moves, journal reversals |
1405| Everything else | a plain object | anywhere | bespoke report, integration, one-off page |
1406
1407Two hard boundaries hold the map together:
1408
1409- **Gates before the write; reactions after.** A hook may reject or stamp but never send email or call out — a gate with side effects leaks that a write was attempted and couples saves to SMTP. A reaction may notify or spawn work but never block the write that already happened.
1410- **Derived values are non-authoritative; gates are authoritative.** A broken formula stores `""""` and the write succeeds; a broken hook fails closed and the write is rejected. Never let a gate read only a derived value it could recompute — the journal-balance hook sums the lines itself even though rollup totals exist.
1411
1412## Worked example 1: sales tax (the one everyone leaves out)
1413
1414Why it's left out: it hides a doctrine most systems never articulate — **stamp point-in-time facts; derive live facts**. The tax *rate* is a fact about the transaction moment: rates change, and the invoice must forever carry the rate it was issued under.
1415
14161. **Rates are data**: a `tax_rates` collection — `region`, `category`, `rate`, `valid_from`. Editable like any records; history visible.
14172. **Stamping is a hook**: `hooks.before_write` on invoice lines looks up the applicable rate and *stamps* `tax_rate` onto the line — same posture as the server stamping `owner_id`. The client never supplies it; later rate changes never rewrite history.
14183. **Amounts are formulas/rollups**: `line.tax_cents = amount_cents * tax_rate` (formula); `invoice.tax_total_cents` (rollup). Materialized, so they appear on every list/detail/report and update live.
1419
1420## Worked example 2: payment aging (time makes the state)
1421
1422The trap: an invoice becomes overdue with **no write happening** — time did it. Any system that computes ""is it late?"" ad hoc in the UI grows a dozen disagreeing copies of that check.
1423
14241. `balance_cents` is a **rollup** (`total_cents` minus `sum(payments)`).
14252. `sent → overdue` is a **daemon scheduled pass** through the ordinary transition machinery, when `due_date < today` and `balance_cents > 0` — so the state is *stored*, permission-checked, realtime-pushed, and every surface agrees (time-driven state belongs to the daemon, never to read-time checks).
14263. Aging buckets (30/60/90) are a rollup/report over stored state.
1427
1428## Worked example 3: dunning (reactions compose on state)
1429
1430Dunning is nothing new — it is the composition of the two layers above:
1431
14321. The daemon flips the state (`overdue`, later `overdue_30`, …).
14332. `notify_rules` reacts: state change → email template to the invoice's contact. The rule is data; the daemon delivers after the write.
14343. Escalation (""again at 30 days unless paid"") is *more time-driven state*, not a special notification feature — each escalation step is a transition the daemon makes, each with its own notify rule.
1435
1436## Worked example 4: refunds (money moves, never mutates)
1437
1438**Money never mutates; it moves.** A refund is not an edit to a payment — it is a new, append-only compensating record, exactly like a stock move or a journal reversal.
1439
14401. `refunds` is a `storage: append` collection pointing at the payment.
14412. The invariant is a **hook**: `refund ≤ payment − prior refunds` — a cross-record gate, summed from the records themselves.
14423. `payment.refunded_cents` / `net_cents` are **rollup/formula** — display truth, derived.
14434. ""Correctly credited"" is the balanced journal entry the refund composes — enforced by the journal-balance hook that already ships. The chain closes: append-only record → hook invariant → rollup display → balanced books.
1444
1445## Worked example 5: inventory shrinkage (the events nobody codes for)
1446
1447Waste, breakage, theft, expiry, damage, disaster: systems are easy to build when quantities just flow buy → stock → sell; real usage is losses. Naming them is the point — ""adjustment"" is the euphemism that hides exactly what operators and auditors need to see.
1448
14491. **Losses are movements**: a loss is an append-only stock move with a loss `reason`, out of a location, to nowhere — never an edit to an on-hand number (levels stay derived).
14502. **The hook stamps and gates**: `unit_cost_cents` is stamped from the product at the moment of movement (tomorrow's cost edit must not reprice yesterday's loss); a loss must leave a real location and can have no destination.
14513. **The composer books the shadow**: a loss move composes a posted journal — DR shrinkage-expense / CR inventory-asset — with per-reason account overrides as settings (theft to its own account for the insurance claim).
14524. **Count reconciliation is the same machinery**: a physical count writes one compensating adjustment move for the variance (out for shortage, in for overage) plus the mirror journal. The count is an *event*, not an edit.
1453
1454This was also the composer pattern's third repeat, which triggered the extraction of a shared composer: every generated journal now goes through `object_finance.compose_posted_journal` (idempotent by `generated_from`, verify-then-post), and the payment/refund/recurring/reversal composers shrank onto it.
1455
1456## Worked example 6: bank reconciliation (evidence, not data entry)
1457
1458Comparing the books to a bank statement is the strongest practical control a small business has, and the reason is not arithmetic — it is that the bank's copy was written by someone with no stake in the numbers. Every design choice follows from protecting that independence.
1459
14601. **Imported lines are append-only and keep their original text** in `raw`; parsed fields are a view of it. A pre-write hook makes the factual fields immutable on update — corrections come from re-importing a corrected statement, never from editing a line to agree with the books.
14612. **The statement checks itself.** Opening + lines = closing (tie-out), and this opening = the last statement's closing (continuity). Both run at import and *set* the import's status; a truncated or edited file fails arithmetic. The flags record which checks **ran**, so a balance-less CSV never reads as trustworthy as a verified one.
14623. **Reading a bank's format is data**, not code: a saved column map, authored once with AI help and executed deterministically thereafter. AI at configuration time, code at run time — a control cannot be nondeterministic.
14634. **The matcher proposes; a person disposes.** Suggestions are derived and non-authoritative; confirming is an ordinary attributed update, so the change log names who reconciled what. Only a two-signal match (reference *and* amount) is auto-confirmable, and even that is configurable to off.
14645. **The unmatched tail is where the business logic lives**: fees, interest, own-account transfers, NSF bounces — each a resolution verb composing a posted journal through the shared composer, stamped with the line's id so resolving twice books once. These are exactly the entries small books never record, which is why their cash balance drifts until they stop reconciling at all.
1465
1466## State-dependent behavior generally
1467
1468`transitions` + `when` guards define the machine; the board's drag is its UI; hooks gate specific moves with computed conditions (the balance check gates `draft → posted`); the daemon makes time-based moves. The known missing declarative is **""state freezes fields""** (a posted invoice's amounts are immutable). Per the method: express it in each module's hook first; promote it to a schema key (`locked_when`) only after the shape repeats.
1469
1470## What this implies we build next
1471
1472The examples above want one substrate piece that doesn't exist yet: a **payments collection** (plain records — provider integration is a connector later). It unblocks aging, dunning, and refunds at once. Then: `tax_rates` + the stamping hook; then a delay/repeat notion for `notify_rules` if escalation schedules outgrow ""each step is its own transition.""
1473"
1474validation-and-logic Validation and Business Logic Building on DBBASIC 240 How DBBASIC Object Server enforces field validation and runs business rules from the schema alone, with no per-app code required. "Two things a schema-driven system has to get right without per-app code:
1475rejecting bad writes, and running the business rules a record's lifecycle
1476implies. DBBASIC does both from the schema, enforced on the server — the form
1477is a convenience, not the gate.
1478
1479## Validation — declared once, enforced on every write
1480
1481The write path (create and update) validates each field against the schema,
1482server-side, regardless of which surface issued the write (web form, API, MCP
1483agent, AI). A form's client-side checks are a nicety on top; they are not what
1484protects the data.
1485
1486What is enforced today:
1487
1488| Check | Declared as | Enforced |
1489|-------|-------------|----------|
1490| Type parses | `type` (integer, number, date, datetime, boolean, enum) | Yes |
1491| Required present | `required: true` | Yes |
1492| Enum membership | `enum: [...]` | Yes (value must be in the list) |
1493| Length bounds | `validation.min_length` / `max_length` | Yes |
1494| Pattern | `validation.pattern` (regex) | Yes |
1495| Referential integrity | `relation: {collection}` | Yes (id must exist in the target) |
1496| State transition | field `transitions` map / `flow` | Yes (status can only move where allowed) |
1497| Read-only / computed | `read_only` / `type: computed` | Yes (client value rejected) |
1498| Storable in TSV | (substrate) | Yes (size, no NUL/control chars) |
1499
1500Referential integrity and enforced state transitions are the notable ones —
1501most no-code tools skip both. A relation is a *validated pointer*: you cannot
1502point a `project_id` at a project that doesn't exist. A status field with a
1503`transitions` map is a state machine: an order can go `open → cancelled` only
1504if the schema says so.
1505
1506### Known gaps
1507
1508- **Numeric range** — we bound string length, not numeric value (a price can go
1509 negative). A `min`/`max` value rule is the fix.
1510- **Uniqueness** — no unique constraint (e.g. unique email).
1511- **Declarative cross-field rules** — no formula language yet for
1512 ""end_date > start_date"" (`visible_when` handles conditional *visibility*,
1513 not validation). The general mechanism exists, though: a **pre-write hook**
1514 (below) expresses any cross-field or cross-collection rule in real code. A
1515 Salesforce-style declarative rule (formula + message) remains the ergonomic
1516 follow-on for the common cases.
1517- **Inline client feedback** — the form pre-checks `required`; length/pattern
1518 errors currently surface after a server round-trip, and messages are
1519 technical (""longer than max_length""). Friendlier, inline feedback is a
1520 UX polish item, not a safety gap.
1521
1522## Pre-write hooks — code inside the write path
1523
1524For rules the schema can't express, a collection declares a **hook object**:
1525
1526```json
1527{ ""name"": ""fin_journals"", ""hooks"": { ""before_write"": ""hook_fin_journals"" } }
1528```
1529
1530The object's `BEFORE_WRITE(request)` runs **synchronously inside the generic
1531HTTP write path** — after permission checks, before persist — for every public
1532create/update on that collection. `request` carries `{collection, action,
1533record, existing, changes, subject}`. It returns `None` to allow,
1534`{""error"": ""..."", ""status"": 4xx}` to reject with its own message, or
1535`{""record"": {...}}` to transform (the transformed record still passes full
1536schema validation, and can never touch `id`, `owner_id`, or read-only/computed
1537fields).
1538
1539The contract is deliberately strict:
1540
1541- **Fail closed** — a declared hook that is missing, raises, or returns a
1542 non-contract shape rejects the write. A gate that silently passes on error
1543 is not a gate.
1544- **After permissions** — a denied subject never reaches the hook.
1545- **Opt-in per collection** — collections without `hooks` pay one cached dict
1546 lookup; the original latency concern that deferred this feature stays
1547 bounded.
1548- **Public surface only** — trusted server-side writers (the daemon,
1549 migrations, seed-merge) call the storage layer directly and bypass hooks.
1550
1551This is what keeps the generative form working when custom logic arrives: the
1552form still POSTs to `/collections/{c}/records`; the rule lives server-side.
1553First adopter: `fin_journals` refuses to post a journal whose lines don't
1554balance — a cross-collection rule (it sums `fin_journal_lines`) no schema key
1555could express.
1556
1557## Business logic — the automation substrate
1558
1559Business rules run without per-app code through a set of declarative
1560primitives, driven by a background daemon and the change log:
1561
1562```mermaid
1563flowchart LR
1564 W[""record write""] --> CL[""change log""]
1565 CL --> DAEMON[""daemon: poll triggers""]
1566 DAEMON --> NOTIFY[""notify_rules → notification / email""]
1567 DAEMON --> TARGET[""trigger object → target object""]
1568 W --> TRANS[""transition guards (flow: when clauses)""]
1569 SCHED[""schedule""] --> DAEMON
1570 DAEMON --> COMPUTE[""computed / materialize fields""]
1571 DAEMON --> CONN[""connectors → external systems + reconcile""]
1572```
1573
1574- **Transitions with guards** — a `flow`/`transitions` map is a conditional
1575 state machine; a `when` clause gates whether a move is allowed. The board's
1576 drag and every status write run through it.
1577- **notify_rules** — declarative ""a record changed → send a notification /
1578 email."" Data, not code.
1579- **Trigger → target objects** — the daemon polls trigger objects and executes
1580 target objects: an event-handler path built on top of the daemon and its
1581 scheduled passes.
1582- **Computed / materialized fields** — derived values (`type: computed`,
1583 `materialize_definitions`).
1584- **Connectors** — external side effects expressed as desired-state records
1585 with a reconcile pass.
1586- **Scheduled passes** — the daemon runs periodic work (compaction,
1587 auto-transitions).
1588
1589This is the same family as Salesforce flows/validation-rules, Hasura event
1590triggers, and Directus Flows — declarative automation over a data store, here
1591on plain-text records.
1592
1593### Formula & rollup fields — shipped
1594
1595The spreadsheet feature, on collections. A `type: computed` field declares
1596either a **formula** over sibling fields or a **rollup** over a child
1597collection:
1598
1599```json
1600{""name"": ""full_name"", ""type"": ""computed"",
1601 ""formula"": ""first_name + \"" \"" + last_name""}
1602
1603{""name"": ""debit_total_cents"", ""type"": ""computed"",
1604 ""rollup"": {""collection"": ""fin_journal_lines"", ""fk_field"": ""journal_id"",
1605 ""op"": ""sum"", ""field"": ""debit_cents""}}
1606```
1607
1608Values are **materialized on write at the storage layer** — stored in the TSV,
1609recomputed whenever inputs change (formulas on writes of the record itself;
1610rollups when source-collection records are created, updated, or deleted —
1611single-hop, loop-guarded). Because the value is stored, every surface works
1612with zero changes: lists, tables, detail, search, `field=value` filters, sort,
1613CSV, backups — and **realtime falls out free**: edit a journal line and the
1614journal's totals update live in every open browser. `rollup.op` is
1615`sum | count | min | max | avg`, with an optional equality `where`. The
1616formula language is a tiny safe expression grammar (fields, strings, numbers,
1617`+ - * /`, parens — never a general-purpose `eval`). A broken formula stores an
1618empty value and never fails the write — derived values are non-authoritative,
1619the opposite posture of hooks. Definition changes are backfilled explicitly
1620with `object_records.recompute_computed_fields(collection)` (install step,
1621admin action, or cron — never a runtime loop).
1622
1623**Declarative validation rules** (cross-field formula + message) remain the
1624natural next sugar over the hook.
1625
1626## Related
1627
1628- **Placement of rules** — a companion guide walks through *where* each kind
1629 of rule goes (the placement flowchart) with worked examples, and the
1630 doctrine behind those placements.
1631- **Field contract** — the schema field reference these checks read against.
1632- **Runtime contract** — the daemon, triggers, schedules, and event contracts
1633 in full.
1634- **Storage modes** — append-only collections, the immutable substrate for
1635 audit-shaped data (journals, stock moves).
1636"
1637capabilities Capabilities: Per-Collection Behaviors, Declared Not Coded Building on DBBASIC 250 How the platform's capability layer lets any collection opt into comments, attachments, and sharing by declaring a flag instead of hand-building each behavior per app. "The schema-driven UI has two layers. The **display** layer turns a schema into
1638a list, table, board, form, and detail page (see
1639[`schema-forms.md`](/docs/schema-forms)). The **capability** layer sits on top of
1640it: generic *behaviors* a collection opts into by declaring a flag, wired by the
1641platform instead of hand-built per app.
1642
1643A collection declares them under a root-level `capabilities` key:
1644
1645```json
1646{
1647 ""name"": ""tasks"",
1648 ""capabilities"": { ""comments"": true, ""attachments"": true, ""shareable"": true },
1649 ""fields"": [ ... ]
1650}
1651```
1652
1653and its detail page grows a comment thread, an attachment list, and a share
1654control — with no per-view block and no per-app table. The same three flags on
1655`contacts`, `invoices`, or any other collection give the same behaviors.
1656
1657This is the ""define once, project everywhere"" rule applied to what a record
1658*does* and *connects to*, not just how it looks. It replaces the pattern where
1659every app hand-rolled its own version of the same idea: comments existed four
1660separate times (`task_comments`, `thread_comments`, `profile_comments`,
1661`interactions`) before one `capabilities.comments` flag collapsed them.
1662
1663## The model
1664
1665Each capability is one **polymorphic collection** (a row attaches to a record in
1666*any* collection via a `parent_collection` + `parent_id` pair), one **widget**
1667served as a static script like `/list` and `/form`, and one line in the detail
1668renderer that mounts the widget when the flag is present.
1669
1670```mermaid
1671flowchart TD
1672 S[""schema.capabilities flag""] --> API[""/api/schema exposes it""]
1673 API --> VR[""view_render: maybeMountCapabilities()""]
1674 VR -->|flag present| W[""window.dbbasicThread / Attachments / Share .mount()""]
1675 W --> C[""polymorphic collection<br/>parent_collection + parent_id""]
1676 C -->|permission-checked read and write| STORE[""object store""]
1677 D[""detail block""] --> VR
1678```
1679
1680The wiring is one function, `maybeMountCapabilities` in
1681`app-views/objects/site/view_render.py`: it fetches the record's schema, reads
1682`capabilities`, and mounts each declared widget into its own sub-element under
1683the detail. Absent flag or unloaded widget → nothing renders, never an error.
1684
1685`capabilities` is whitelisted in schema normalization
1686(`object_schemas.py`) and surfaced by `/api/schema` so the client can wire it —
1687the same ""carried, not interpreted"" posture as `flow` and `blocks`.
1688
1689## The three capabilities
1690
1691| Flag | Widget (route) | Backing collection | Package |
1692|------|----------------|--------------------|---------|
1693| `comments` | `window.dbbasicThread` (`/thread`) | `thread_comments` | `app-thread` |
1694| `attachments` | `window.dbbasicAttachments` (`/attachments`) | `files` (+`parent_*`) | `app-files` |
1695| `shareable` | `window.dbbasicShare` (`/share`) | `record_shares` | `app-share` |
1696
1697### comments
1698
1699A comment thread for one record. Reads/writes `thread_comments` filtered to the
1700`(parent_collection, parent_id)` pair, posts with a minimal payload (the server
1701owns `created_at`; `owner_id` is set from the session and redacted from other
1702readers, so each comment also stamps a display `author_name`), owner-only
1703delete, and re-renders over the change log like every other surface.
1704
1705### attachments
1706
1707An upload/list/download/delete surface for files attached to one record.
1708Uploads go multipart to `/api/files` — which owns `owner_id`/`size`/
1709`content_type` and saves the blob — carrying `parent_collection`/`parent_id` so
1710the file lands on this record. Delete is blob-aware (`DELETE /api/files/{id}`
1711removes the bytes *and* the metadata row; a plain record delete would orphan the
1712blob). No per-collection foreign key on `files` is needed.
1713
1714### shareable
1715
1716The foundational one — it changes the **permission check**, not just a widget.
1717A record's owner grants other users read (or write) access to that individual
1718record. See the security model below.
1719
1720## Sharing: the security model
1721
1722Sharing is the one capability that grants access, so it is designed so that a
1723grant can only ever *narrow* what a viewer could already reach, never widen it
1724for the granter's own benefit.
1725
1726```mermaid
1727sequenceDiagram
1728 participant Owner
1729 participant API as /api/share
1730 participant Store as record_shares
1731 participant Grantee
1732 participant Engine as permission check
1733
1734 Owner->>API: POST {collection, record_id, user_id}
1735 API->>API: does Owner own the TARGET record?
1736 Note right of API: cross-collection ownership check<br/>a row_filter cannot express
1737 API-->>Owner: 403 if not owner
1738 API->>Store: create grant (owner-only path)
1739 Grantee->>Engine: read record
1740 Engine->>Store: subject.shared_records[collection]
1741 Engine->>Engine: {id: $shared_records} row filter matches
1742 Engine-->>Grantee: allowed (only this record, only this collection)
1743```
1744
1745Three properties make it safe:
1746
17471. **Grants are minted only by `/api/share`.** `record_shares` has *no*
1748 create/update/delete permission rule, so a raw collection write can't forge
1749 one. The endpoint authorizes each grant against the **target record's**
1750 `owner_id` — a cross-collection fact a row filter can't check. A non-owner
1751 gets `403 ""Only the record's owner can share it.""`
17522. **Reads resolve through the subject.** Before a check runs, the server builds
1753 `subject.shared_records` (per collection) from `record_shares`, the same way
1754 it builds `project_ids` from `project_access`. A collection opts in with a
1755 `{""id"": ""$shared_records""}` read rule.
17563. **`$shared_records` is collection-aware.** The collection being checked is
1757 threaded through `_record_matches_filter → _resolve_filter_value`, so a share
1758 on a *task* never matches a same-id row in another collection. (Covered by
1759 `test_shared_records_filter_is_collection_scoped`.)
1760
1761Verified end to end: a grantee reads a shared record (`200`), a non-grantee is
1762denied (`403`), and a non-owner cannot create a grant (`403`).
1763
1764## Adding a new capability
1765
1766The recipe, in order:
1767
17681. **A polymorphic collection** with `parent_collection` + `parent_id` (both
1769 *settable on create* — a field that is `required` **and** `read_only` can
1770 never be created through the HTTP write path, only a server-side
1771 `preserve_read_only` bypass).
17722. **A widget** object at `site_{name}` serving `window.dbbasic{Name}` (public
1773 `execute` rule — it's a static script; the data it reads stays row-gated).
17743. **Wire it** in `view_render`'s `maybeMountCapabilities` and add
1775 `<script src=""/{name}"">` to the view page.
17764. **Adopt it** on a collection with `capabilities.{name}: true`. If the
1777 behavior grants access (like sharing), also add the permission rule.
1778
1779Leave `owner_id` to the server on create — a field with a `public: hidden`
1780policy is server-set from the session and a client value is rejected.
1781
1782## Related
1783
1784- [`schema-forms.md`](/docs/schema-forms) — the display layer (forms, views, list
1785 modes) capabilities sit on top of.
1786- [`permissions-model.md`](/docs/permissions-model) — row/field filters, the
1787 `$`-variable row-filter language `$shared_records` extends, and sharing.
1788- [`ui-decisions.md`](/docs/ui-decisions) — decisions #7–#14, the living log of the
1789 generative-UI and capability behaviors (with the gotchas found building them).
1790"
1791capability-objects Capability Objects: ffmpeg, OCR, and Other System Tools Building on DBBASIC 260 How to write DBBASIC objects that shell out to system binaries like tesseract and ffmpeg safely, with worked examples and a security checklist. "A DBBASIC object is plain Python running in a subprocess. There is no import sandbox and no call whitelist, so an object can do anything Python on that machine can do — including shell out to system binaries. That makes real capability work a normal object: OCR a scanned receipt with `tesseract`, transcode a clip with `ffmpeg`, render a thumbnail with ImageMagick, extract text from a PDF, call a local model.
1792
1793An earlier prototype of this platform shipped a pack of capability objects like these. The current runtime supports the same capability; this page documents how to write them safely.
1794
1795## The execution model
1796
1797Each object call runs in a **spawned subprocess** with a wall-clock timeout (`DBBASIC_OBJECT_TIMEOUT_SECONDS`). Practical consequences:
1798
1799- The object has the full standard library, the filesystem, and the ability to launch its own subprocesses. `subprocess.run([...])` works.
1800- A runaway object is killed at the timeout, so a hung `ffmpeg` cannot wedge the server. Set the timeout to fit the workload.
1801- It is isolated from the parent's memory, not from the machine. There is no CPU or memory cap yet, so capability objects are **operator/admin code** — your own tools on your own box, the same trust level as a cron job. Writing object source is gated; visitors run these objects, they do not author them.
1802
1803## Prerequisite: install the tool
1804
1805Capability objects call binaries that must exist on the VM. This is a normal operator step, explicit rather than hidden:
1806
1807```bash
1808sudo apt install tesseract-ocr ffmpeg imagemagick poppler-utils
1809```
1810
1811If a binary is missing, the object's `subprocess.run` raises `FileNotFoundError`, which surfaces as a normal object execution error in the logs and change feed — no silent failure.
1812
1813## Worked example: OCR an uploaded image
1814
1815A browser form posting `multipart/form-data` delivers files under `request[""_files""]` with base64 content. The object writes the bytes to a temp file, runs `tesseract`, and returns the text.
1816
1817```python
1818import base64
1819import subprocess
1820import tempfile
1821from pathlib import Path
1822
1823
1824def POST(request):
1825 upload = request.get(""_files"", {}).get(""image"")
1826 if upload is None:
1827 return {""status"": ""error"", ""error"": ""post an image field""}
1828
1829 content = base64.b64decode(upload[""content_base64""])
1830 with tempfile.TemporaryDirectory() as work:
1831 source = Path(work) / ""in.png""
1832 source.write_bytes(content)
1833 result = subprocess.run(
1834 [""tesseract"", str(source), ""stdout""],
1835 capture_output=True, text=True, timeout=30,
1836 )
1837
1838 if result.returncode != 0:
1839 _logger.error(""ocr failed"", stderr=result.stderr[:500])
1840 return {""status"": ""error"", ""error"": ""OCR failed""}
1841
1842 text = result.stdout.strip()
1843 _logger.info(""ocr done"", chars=len(text))
1844 return {""status"": ""ok"", ""text"": text}
1845```
1846
1847Set that object's timeout generously (`DBBASIC_OBJECT_TIMEOUT_SECONDS`) so a large scan has room, and grant it a public `execute` rule if the form is public.
1848
1849## Worked example: transcode with ffmpeg
1850
1851```python
1852import base64
1853import subprocess
1854import tempfile
1855from pathlib import Path
1856
1857
1858def POST(request):
1859 upload = request.get(""_files"", {}).get(""clip"")
1860 content = base64.b64decode(upload[""content_base64""])
1861 with tempfile.TemporaryDirectory() as work:
1862 src = Path(work) / ""in""
1863 out = Path(work) / ""out.mp3""
1864 src.write_bytes(content)
1865 proc = subprocess.run(
1866 [""ffmpeg"", ""-y"", ""-i"", str(src), ""-vn"", ""-b:a"", ""128k"", str(out)],
1867 capture_output=True, timeout=120,
1868 )
1869 if proc.returncode != 0 or not out.exists():
1870 return {""status"": ""error"", ""error"": ""transcode failed""}
1871 audio = out.read_bytes()
1872
1873 return {
1874 ""content_type"": ""audio/mpeg"",
1875 ""body"": base64.b64encode(audio).decode(""ascii""),
1876 ""encoding"": ""base64"",
1877 }
1878```
1879
1880Small media (receipts, short clips, single images) fits inside a request. **Long media does not** — a two-second timeout is fine for a receipt but a 2 GB video transcode belongs in a background job, not a web request. Background jobs are a planned platform slice; until then, keep capability objects to work that finishes in seconds.
1881
1882## Composing with files, records, and AI
1883
1884Capability objects get more useful combined with the rest of the platform:
1885
1886- **Files** — instead of accepting an upload each time, an object can read a stored file's bytes and OCR or transcode it, then write the result back as a new file or a record. (The clean helper for an object to read from the `files` collection by id is a small planned addition; today an object works from the request payload.)
1887- **Records** — an OCR object can `create` a record with the extracted text so it becomes searchable through global search.
1888- **AI + shell** — because the object is exposed as an `execute_object` MCP tool, the shell AI can call it. ""OCR my latest upload and note the total"" becomes: the AI reads the file, calls the OCR object, and creates a note — each step permission-checked and audited, no glue code.
1889
1890## Safety checklist for a capability object
1891
1892- Set a timeout that fits the job; never rely on the default.
1893- Validate the upload before spending CPU on it (size, content type).
1894- Write to a `tempfile.TemporaryDirectory()` so nothing leaks between calls; never build a shell command by string-concatenating user input — pass argument lists to `subprocess.run`, never `shell=True`.
1895- Log failures with `_logger` so they land in the change feed.
1896- Treat these as operator-authored: source writes are gated, and a per-family ""builder role"" for scoped, non-admin object creation is a deliberate open design item, not a shipped capability.
1897"
1898storage-modes Collection Storage Modes Data & Storage 300 How collections choose between classic (rewrite-on-write) and append (log-structured) storage, with real numbers on write cost, compaction, and current limits. "A collection stores its records one of two ways. Both live in the same
1899single `records.tsv`, both are served by the same API, permissions, cache,
1900and generators, and both satisfy the same core invariant: the file alone
1901is the complete truth.
1902
1903## Classic (the default)
1904
1905Every write atomically rewrites the whole file (temp file + rename). Every
1906row appears exactly once. This is the right mode for what collections
1907usually are — hundreds to tens of thousands of records — and it is what
1908every collection uses unless its schema says otherwise. Nothing existing
1909changes modes on its own; there is no migration.
1910
1911- Write cost grows with collection size (measured: ~4ms at 1k rows,
1912 ~284ms at 100k, ~2.8s at 1M — per write).
1913- The file never contains history; `cat` shows current state only.
1914
1915## Append (opt-in per collection)
1916
1917Declare it in the schema:
1918
1919```json
1920{ ""name"": ""events_log"", ""storage"": ""append"", ""fields"": [ ... ] }
1921```
1922
1923The collection becomes a last-wins log inside the same file: create,
1924update, and delete each append one row instead of rewriting. An internal
1925first column `_op` marks ordinary rows (empty) and deletions (`del`) — it
1926is never visible in schemas, API responses, or returned records, and no
1927schema may declare a field named `_op`. Reads fold the log newest-wins;
1928list order matches classic mode exactly.
1929
1930- Write cost is near-constant (measured: create ~4ms at 100k rows, ~34ms
1931 at 1M; update ~1–9ms — versus seconds in classic mode at that scale).
1932- Until compaction, `cat` shows the row's full history and deletions — an
1933 inspectable audit trail, by design.
1934- Mode transitions are just writes: adding `""storage"": ""append""` takes
1935 effect on the next write (one final rewrite adds `_op`); removing it
1936 compacts back to a classic file on the next write.
1937
1938### Compaction (the VACUUM analog)
1939
1940Superseded rows and tombstones accumulate until compaction folds the file
1941back to live rows only (atomic rewrite, under the writer lock):
1942
1943- Manually: `object_records.compact_collection(""name"", base_dir=...)`
1944 (returns rows/bytes before and after).
1945- Automatically: when a read observes more dead rows than live ones and
1946 the file exceeds `DBBASIC_APPEND_COMPACT_MIN_ROWS` (default 10000)
1947 physical rows, the **next write** compacts instead of appending. Reads
1948 never rewrite anything.
1949- Via HTTP (admin-gated, same posture as other mutating admin record
1950 routes): `POST /admin/collections/{collection}/compact` runs
1951 `compact_collection` and returns its rows/bytes-before/after summary
1952 plus `duration_ms`. 404 for an unknown collection, 400 for a collection
1953 not currently in append storage mode. `GET /admin/storage` reports
1954 per-collection compaction observability — `{""physical_rows"",
1955 ""live_rows"", ""dead_rows"", ""bloat_ratio"", ""file_bytes"",
1956 ""sidecar_present"", ""compaction_flagged""}` for every append-mode
1957 collection (`object_records.append_collection_stats` /
1958 `list_append_collection_stats`). This endpoint is cheap by
1959 construction: it never folds records.tsv to answer, preferring a warm
1960 in-process cache entry or the id→offset sidecar's own row count; a
1961 collection neither can answer comes back with `""estimated"": true` and
1962 only `file_bytes` populated (`physical_rows`/`live_rows` null) rather
1963 than paying an O(file) parse on a status call.
1964- Via the daemon (`object_daemon.py`, optional — nothing else depends on
1965 it running): `process_compactions` polls every
1966 `DBBASIC_COMPACTION_INTERVAL_SECONDS` (default 3600, tracked by a
1967 `.compaction_last_run` marker file under the data dir rather than an
1968 in-process timer, so the interval survives a daemon restart) and
1969 compacts any append-mode collection whose `bloat_ratio` is at or above
1970 `DBBASIC_COMPACTION_BLOAT_RATIO` (default 1.0 — dead rows at least
1971 matching live rows) AND whose physical row count is at or above
1972 `DBBASIC_APPEND_COMPACT_MIN_ROWS`. A failure compacting one collection
1973 is logged and skipped; it never stops the rest of the pass.
1974
1975### Crash behavior
1976
1977A write interrupted mid-row leaves a torn final line. Readers ignore it;
1978the next writer truncates it away before appending. Compaction and classic
1979mode keep full temp+rename atomicity.
1980
1981### Current limits (honest)
1982
1983- Collections larger than `DBBASIC_RECORDS_CACHE_MAX_ROWS` (default
1984 500k) fall out of the in-process cache. Point operations (create's
1985 duplicate check, get, update, delete) no longer pay a full fold-parse
1986 above that size: an id→offset sidecar (`.records.oidx` next to
1987 `records.tsv`, derived and disposable) answers them by seek instead
1988 (measured: create/update ~1ms, get ~0.3ms at 1M rows, versus ~2.8s for
1989 a full fold-parse before the sidecar existed). The sidecar builds
1990 lazily on first use past the threshold (one sequential scan, ~2s at 1M
1991 rows) and self-heals from any inconsistency by rebuilding. **List and
1992 read-all-records stay unchanged** above the threshold: they still fold
1993 the whole file, since ordering and windowing need the full picture — a
1994 secondary sidecar for that is future work. At or below the threshold,
1995 nothing changed: the in-process cache still serves everything, sidecar
1996 or not.
1997- At or below the threshold, a warm cache entry for an append-mode
1998 collection that grows in place (another process, the CLI, or a sibling
1999 worker appending — same inode, size only) is caught up with a
2000 tail-delta fold of just the new rows instead of a full re-parse of the
2001 whole file: when a stat signature changes by growth alone, only the
2002 tail delta gets parsed. Measured: reading a 400,000-row collection
2003 right after an external +100-row append dropped from ~742–815ms (full
2004 re-fold every time, regardless of how small the growth was) to
2005 ~74–100ms (fold just the 100 new rows) — roughly 8x at this size, and
2006 the gap widens with collection size since the old cost was O(file) and
2007 the new one is O(delta). A delta containing any deletion falls back to
2008 a full fold (deletes shift every later row's position, which the cheap
2009 merge doesn't model); growth past the cache threshold is still evicted
2010 per the existing row-count rule above.
2011- Cold reads of an append file cost ~1.6x a classic parse (fold
2012 overhead). Compaction restores parity.
2013- Backup **preview/diff** interprets `_op`: for append-mode collections
2014 (backup side, live side, or both — a backup taken before a mode switch
2015 compares fine against a now-classic or now-append live file) it folds
2016 the log last-wins-by-id before diffing, so `_op` never appears as a
2017 field and a tombstoned id is treated as absent, not live. Backup/restore
2018 of the file itself is unaffected either way (the file is opaque bytes to
2019 backup).
2020
2021## Choosing
2022
2023Classic for ordinary app data (the default is correct). Append for
2024write-hot or log-shaped collections: event logs, impressions, telemetry,
2025game/world state, anything updated far more often than it is read cold.
2026When in doubt, stay classic — switching later is one schema edit."
2027append-only-storage Append-Only Storage: Design Decisions Data & Storage 310 Design decisions for making collection writes O(1) via append-only TSV storage, while preserving the single-file-of-truth invariant. "This document lays out the design decisions for the next phase of the storage engine: making collection writes O(1) while preserving the system's defining constraint. As with other design work in this project, it's written before implementation, so the direction and its reasoning are on record before any code exists.
2028
2029## The Invariant (Decided First, Everything Else Follows)
2030
2031**A collection either IS one `records.tsv` file, or can deterministically emit one.** The single TSV is the source of truth: complete on its own, greppable, diffable, backed up with `cp`. Anything clever — indexes, caches, offsets — is a *sidecar*: derived, disposable, and rebuildable from the TSV alone. No storage feature may ever be added that cannot be flattened back to the single file. (This also binds any future alternate backend: its contract includes materializing the canonical TSV on demand.)
2032
2033## Why Change Anything
2034
2035Measured (2026-07-12, on an M1 machine, local SSD, after the read-cache work landed): `create` and `update` rewrite the whole file per record — 10.5s at 1M rows, 2.5min at 5M. Reads scale fine (O(1) indexed `get` holds at millions of rows); writes are the wall. Motivating workloads: game-server state (write-hot large collections), ad-exchange impression logs, price history — all append-shaped.
2036
2037## The Design
2038
2039Append-only *within the single file*:
2040
2041- **Create** appends one row. O(1).
2042- **Update** appends a superseding row with the same id. Reads resolve **last-wins** per id. This is a deliberate change from today's duplicate-id behavior, which is first-occurrence-wins.
2043- **Delete** appends a tombstone row. The exact representation is decided at implementation time, but it must be a reserved marker that cannot collide with user data, must survive the CSV layer, and must be visible and obvious in a raw `cat`.
2044- **Compaction** is the only remaining whole-file rewrite: collapse to live rows via the existing atomic temp+rename, run on a schedule or when the superseded-row ratio passes a threshold — never inline in a request. Until compaction, the file contains its own row history: grep shows every version a record went through. That is a feature (an inspectable audit trail), not merely tolerated.
2045- **New-column handling**: today a new field triggers a full-file header rewrite. Append-only tolerates rows shorter than the current header (they read as empty — the semantics that already exist), so a header change can ride the next compaction rather than forcing an immediate rewrite.
2046
2047## Sidecars
2048
2049- **id → byte-offset index** file per collection: maps each id to the offset of its *latest* row. Enables point reads by seek without parsing the file. Derived and disposable — rebuild is one sequential scan. Never required for correctness; readers without the sidecar fall back to scan.
2050- **The in-process read cache** (already shipped) composes into incremental reads: cache entries remember the byte offset they consumed; when a stat signature changes by growth alone, parse only the tail delta — O(new rows), not O(file). This dissolves the ""big hot collection vs. cache eviction"" tension for write-hot workloads.
2051- The read cache's staleness check currently relies on every write producing a fresh inode via atomic rename. Appends mutate in place instead (same inode; mtime/size change). The stat-signature check still detects the change correctly, but the reasoning behind it needs updating for the append case at implementation time, including the torn-tail rule described below.
2052
2053## Crash Safety
2054
2055Rename-atomicity is replaced (for appends) by the classic log rule: a torn final line — no trailing newline, or a row that fails to parse at EOF — is ignored by readers and truncated/overwritten by the next writer. Appends are flushed per write; fsync policy is a deliberate knob (default: flush, no fsync per row — same durability class as today's writes; an env var enables fsync-per-append for workloads that want it). Compaction keeps full temp+rename atomicity.
2056
2057## Concurrency
2058
2059Writers still serialize under the existing per-collection file lock — but the critical section shrinks from ""rewrite everything"" to ""append one line,"" so effective write throughput rises by orders of magnitude without new machinery. Readers remain lock-free; the stat-signature cache covers cross-process coherence, as today.
2060
2061## What This Buys, Honestly
2062
2063- The measured interactive ceiling moves from tens of thousands of rows to **tens of millions** for ordinary use, and far beyond for append-heavy, read-by-id workloads (a 15GB file appends in microseconds and point-reads by seek).
2064- What stays hard: rich secondary queries (by customer, by date range, by status) still scan unless a sidecar index for that field exists. Secondary sidecar indexes are future work, following the same rules: derived, disposable.
2065- What is out of scope on purpose: cross-collection transactions, replication, multi-writer scaling. Single VM, one honest file, is still the product.
2066
2067## Migration and Compatibility
2068
2069Opt-in per collection (schema flag or storage marker), default unchanged — existing collections keep their exact current behavior until switched. A compacted append-only file is byte-compatible with a classic one (header + unique rows), so switching a collection back is: compact, remove the marker. Package installs, backups, and restore already treat `records.tsv` as opaque content and keep working; the backup process gains nothing new to learn (the sidecar index is rebuildable, so backups may ignore it entirely).
2070
2071## The Escape Hatch (Documented Now, Built Never — Until Real Need)
2072
2073If a collection someday truly outgrows one file (billions of rows, heavy secondary queries), the answer is a **pluggable per-collection backend** under the identical schema/permission/API surface — SQLite being the obvious candidate (single file, stdlib, public domain). The invariant above still binds it: such a backend must emit the canonical `records.tsv` on demand. This paragraph exists so nobody ever ""improves"" the TSV engine into a half-database; past the envelope, graft a real one underneath instead.
2074
2075## Prior Art
2076
2077The delta-plus-merge shape has flat-file ancestry: Strozzi NoSQL (1998), the system that coined ""NoSQL,"" documented exactly this pattern for big tables — extract a small working table, edit it, background-merge it into the sorted master, keeping the old table and the delta ""for backup and/or journaling purposes."" This design automates that workflow inside one file. Its lock strategy (a `.lock` sidecar next to the table, cleaned on interrupt) is also the mechanism this codebase independently uses. The offset sidecar planned above is the same trick genomics' `tabix` proved on TSV at petabyte scale.
2078
2079## Build Order
2080
20811. Tail-delta read cache (offset-tracking) — useful even before append-only lands, zero format change.
20822. Append-only create/update/delete + torn-tail recovery + last-wins reads, behind the per-collection opt-in.
20833. Compaction (manual/admin trigger first, then scheduled via the daemon).
20844. id → offset sidecar for seek-based point reads on large collections.
20855. Secondary-field sidecars — only when a named workload demands one.
2086"
2087write-pipeline The Write Pipeline: What Happens, In What Order, And What Can Stop It Data & Storage 320 A stage-by-stage reference for every synchronous, fan-out, and asynchronous step a record write goes through, with each stage's failure semantics. "[`zero-to-code.md`](/docs/zero-to-code) says where each kind of logic lives.
2088This page is its runtime twin: when a record is written, **every stage that
2089runs, in order, with its failure semantics** — because ""code working with
2090code, signals going around"" is exactly where frameworks went dark. Django
2091signals and Rails callbacks earned their reputation not because reactions
2092are wrong but because nobody could answer *why did my record change?*
2093without a debugger. Here the pipeline is fixed, short, and this page is its
2094single description. Verified against `object_server.py` and
2095`object_records.py`; if this page and the code disagree, fix one of them —
2096that disagreement is a bug.
2097
2098## The synchronous path (inside the request)
2099
2100Everything here happens before the client gets its response, in this order:
2101
2102| # | stage | on failure | lives in |
2103|---|---|---|---|
2104| 1 | **permission check** (collection + row rules) | 403, nothing ran | policy |
2105| 2 | **field-write denial** (schema field permissions) | 403, names the fields | schema |
2106| 3 | **owner stamp** — sessions get `owner_id` written server-side; clients cannot spoof it (admin *tokens* skip this: seeding and migration set ownership deliberately) | — | server |
2107| 4 | **pre-write hook** (`hooks.before_write`) — may reject (its own 4xx) or transform (stamp fields; id/owner/read-only/computed protected) | **fails CLOSED**: a broken hook is a 500 and no write | `hook/` object |
2108| 5 | **auto `created_at`**, then **computed fields materialize** — rollups first, then formulas, so a formula may read a rollup | a broken formula stores `""""` — **fails SOFT**, the write proceeds | `object_computed` |
2109| 6 | **schema validation** — types, required, enums, relations-exist, integer-cents; runs on the *final* record, so a hook's transform is validated too | 400, no write | schema |
2110| 7 | **transition check** — is this status move declared, and may *this* subject make it (`when` guards) | 403/409, no write | schema |
2111| 8 | **persist under the file lock**, then the **change-log entry** (actor, before/after, changed fields) | write errors surface as 4xx/5xx | `object_records` |
2112
2113Two asymmetries in there are doctrine, not accident: **gates fail closed,
2114derivations fail soft** (a validation gate that silently passes on error is
2115not a gate; a caption that blanks on error is just a blank caption), and
2116**hooks run before validation** so that what a hook stamps is still checked
2117like any client input.
2118
2119## The fan-out (after the write, same request, best-effort)
2120
2121Once the record is durably written, the server fans out — in this order,
2122each stage isolated so one failing never blocks the next, and none of them
2123can un-write the record:
2124
21251. **rollup re-triggers** — a payment write recomputes its invoice's rollup
2126 fields (single-hop by design: a thread-local guard stops A→B→A cascades;
2127 a chain that needs more hops is a design smell, not a setting)
21282. **realtime push** — permission-filtered signal (collection, id, action);
2129 subscribers re-fetch through the enforced API, so the socket never
2130 leaks record bodies
21313. **event handlers** (`HANDLES` objects) — the reaction layer: composers,
2132 status flips. Payload carries the raw verb (`create`) while the event
2133 name uses the participle (`…created`) — accept both
21344. **durable event log** (when enabled) — for pollers and webhooks
2135
2136## The asynchronous tail (the daemon, seconds later)
2137
2138Independent passes over the change log and collections, each with its own
2139cursor, each at-least-once: **notify** (rules → in-app/email intents),
2140**email outbox** drain, **change dispatch** (off by default — replays
2141handlers for storage-level writes so reactions stop depending on which
2142path wrote), **auto-transitions** (the 48h approve), **scheduled runners**
2143(aging, dunning, recurring journals, matcher, escalation — every run's
2144outcome a `scheduler_runs` record), **queue messages** (delay, retry,
2145backoff).
2146
2147## How chains work without becoming Django signals
2148
2149The reaction graph is real: a payment triggers the books composer, which
2150writes a journal, whose lines re-trigger the journal's rollups. What keeps
2151it legible instead of haunted:
2152
2153- **One hop is synchronous, everything else is a record.** A handler's
2154 write does not re-enter the dispatcher (storage-level writes don't
2155 dispatch), so there is no unbounded cascade — a deeper chain must go
2156 through the change-dispatch pass, which has a cursor, markers, and a log.
2157- **Everything generated says who generated it**: `generated_from`
2158 provenance on composed records, `actor` on every change-log entry.
2159 ""Why did my record change?"" is a query, not a debugging session:
2160 `record_changes` names the actor; a composed journal names its source.
2161- **Idempotency by provenance** means redelivery is safe by construction —
2162 a replayed event composes nothing, so at-least-once is a guarantee, not
2163 a hazard.
2164- **Reactions may not gate and gates may not react**
2165 ([`logic-decisions.md`](/docs/logic-decisions) #6). The moment a reaction
2166 can block a write, or a gate can send an email, the pipeline above stops
2167 being a truthful description — that boundary is what makes this page
2168 possible.
2169
2170## Multi-collection workflows
2171
2172An order becomes an invoice becomes a payment becomes a journal; a task is
2173claimed, submitted, approved, settled. There is deliberately **no workflow
2174engine** — a business process here is a chain of *(transition on one
2175record) → (reaction that writes the next record)*, and each link is
2176individually visible in the table above. The process's state is not hidden
2177in an engine: it IS the records, their statuses, and their provenance
2178stamps, which is why an AI operator can answer ""where is this order stuck?""
2179by reading collections. If a real orchestration need repeats (sagas,
2180compensation across many collections), that is a doctrine-#4 moment —
2181extract it *then*, from the concrete chains that exist.
2182
2183## Reading a live system
2184
2185- `record_changes` — who did what, before/after, per record
2186- `generated_from` — what machine wrote this, from what source
2187- `/scheduler` — what the clock did and what each run returned
2188- execution logs — every hook and handler run, observable like any object
2189- boot lines — the daemon prints every pass's posture (`Books: NOT READY…`,
2190 `Queue: enabled`, `Change dispatch: disabled`) so a disabled layer is
2191 announced, not discovered
2192</content>
2193</StructuredOutput>
2194</invoke>
2195"
2196durability-and-recovery Durability & Recovery Data & Storage 330 How the object server keeps your data safe and readable in plain text, with crash-safe writes, replayable change logs, and a dedicated conformance suite. "How the object server keeps your data safe, and why you can still read it in fifty years.
2197
2198## The design principle: plain text is the source of truth
2199
2200Every record lives in a tab-separated-values file on disk —
2201`data/collections/<name>/records.tsv` — that you can open with `cat`, `grep`,
2202a spreadsheet, or any programming language's CSV reader, on any operating
2203system, with no special software. There is no proprietary binary format your
2204data can be trapped inside.
2205
2206Everything that makes the store *fast* — the id→offset index, the in-memory
2207read cache — is **derived from the TSV and rebuildable from it**. Delete every
2208index and cache and the database is byte-for-byte identical; you lose speed,
2209never a byte. The canonical data is exactly the set of `.tsv` files, and
2210nothing else is load-bearing.
2211
2212> **Why it matters:** a binary database entombs your bytes in a format that
2213> needs *its own software* to read. Plain text means your data outlives the
2214> software that wrote it.
2215
2216## Graceful degradation
2217
2218Because the store is plain text, corruption **degrades gracefully**. A damaged
2219`.tsv` — or a truncated backup of one — can be opened in any editor and
2220salvaged by hand: at worst a single line is garbled; the file never becomes
2221""won't open."" A corrupted binary database page, by contrast, can render the
2222whole database unreadable without vendor-specific recovery tools, and a torn
2223write can damage internal tree structure catastrophically. Plain text has no
2224structure to catastrophically damage.
2225
2226## Crash safety
2227
2228- **Classic collections** are rewritten with a temp-file-plus-atomic-rename.
2229 A reader always sees either the entire previous version or the entire new
2230 one — never a half-written file.
2231- **Append-only collections** (used for high-write logs) treat an interrupted
2232 write as a torn tail: only the last, incomplete entry is ignored on read;
2233 every fully-written entry before it survives — including rows whose quoted
2234 fields span multiple physical lines (the write-side repair is quote-aware;
2235 see [History: the torn-tail write gap](#history-the-torn-tail-write-gap-fixed)).
2236- **Concurrency** is serialized with a kernel file lock (`flock`) that applies
2237 across both threads *and* separate processes, so concurrent writers on a
2238 multi-worker deployment cannot corrupt or lose each other's writes.
2239
2240## Recovery by replay: backup + log
2241
2242Alongside each collection, every mutation is also appended to an independent
2243**change log** — `record_changes/<collection>/changes.jsonl` — recording, for
2244each create/update/delete: a timestamp, the actor, and full *before* and
2245*after* snapshots of the record.
2246
2247This gives you the classic ""restore a backup and replay the log"" recovery
2248model, in plain text:
2249
2250- **Reconstruct a collection from its change log alone.** Replaying the log in
2251 chronological order — apply each `after` snapshot on create/update, drop the
2252 record on delete — reproduces the collection's exact current state. If a
2253 collection file is ever lost or damaged, the change log rebuilds it.
2254- **Point-in-time recovery.** Restore a plain-text backup, then replay the
2255 change log *forward* to a chosen timestamp.
2256- **The change log is itself crash-safe by construction.** It is line-based
2257 (JSON-per-line); a read skips any incomplete final line, so a crash
2258 mid-append costs only that one in-flight entry.
2259
2260The result is **two independent plain-text records of every change** — the
2261collection file and the change log — either of which can reconstruct the
2262other. Your recovery, like your data, needs no special software: a short
2263script replays the log.
2264
2265## How we test durability
2266
2267Durability is verified by a dedicated conformance suite, run deliberately
2268(like a benchmark) rather than on every commit, covering:
2269
2270- **Character-set fidelity** — ASCII and control bytes, multibyte UTF-8,
2271 emoji and grapheme clusters, adversarial Unicode, and arbitrary JSON, each
2272 round-tripped byte-exact through every storage and read path, with no silent
2273 normalization.
2274- **Crash recovery** — a write interrupted at *every byte offset* is simulated
2275 and the store is checked to recover to a valid state with no loss of
2276 committed data — including multi-line quoted rows, whose every-offset sweep
2277 was held as strict xfails until the write-side repair was fixed and is now
2278 an ordinary regression battery.
2279- **Concurrency** — real concurrent processes writing the same collection,
2280 checked for lost updates and corruption.
2281- **Atomicity** — single-record writes are all-or-nothing, and a reader never
2282 observes a torn intermediate.
2283- **Backup, restore, and compaction** — verified byte-exact, including exotic
2284 content, and crash-safe (an interrupted compaction never damages the
2285 original).
2286- **Plain-text rebuild and replay** — deleting every index leaves the data
2287 fully intact and queryable from the TSV alone; a collection is reconstructed
2288 exactly from its change log.
2289
2290The philosophy follows the databases that made testing their reputation: a
2291visible, exhaustive test suite is itself the assurance that the data is safe.
2292
2293## History: the torn-tail write gap (fixed)
2294
2295**Status: fixed (2026-07-30).** This section replaces a ""Known gap"" that
2296stood here for one day — kept as history because the shape of the bug and of
2297its first, reverted fix are both instructive.
2298
2299**The bug.** Before an append lands, `_repair_torn_tail` trims any partial
2300row left by an interrupted write. It found ""the last row boundary"" by
2301scanning backwards for a newline — quote-blind, so a newline *inside* a
2302quoted multi-line field looked like a boundary. Worse, its fast path
2303returned untouched any file whose final byte was `\n`, including a newline
2304inside an unclosed quote — skipping repair in exactly the case that needed
2305it. A crash mid-multi-line-row could then swallow the next appended row into
2306the open quote. Exposure: ~97% of such a row's write window.
2307
2308**The first fix, and why it was reverted.** It gated the repair on the
2309records cache's `covered_bytes`. Two flaws, found the second time around:
2310that value lives only in process memory (gone precisely when a crash makes
2311repair matter), and — the actual killer — a full parse of a *torn* file
2312stores `covered_bytes` equal to the whole file size, because the read fold
2313""accounts for"" the torn bytes by knowing they are torn. So a warm cache
2314reported clean on exactly the files needing repair. Under pytest the cache
2315was warm from other tests; standalone it was cold. ""Fails in pytest, passes
2316standalone"" was that, nothing more.
2317
2318**The fix that stands.** The repair scans forward with the same quote-aware
2319`committed_prefix_len` the read path and the backup index use — one
2320implementation, three users, no copies to drift. A cache may only ever
2321*skip* the scan, never choose a truncation point, and the only cache
2322consulted is the id-sidecar's, whose `covered_bytes` values are genuine row
2323boundaries by construction. Cold (post-crash) the repair pays one full scan,
2324which is the honest price of the one moment a torn tail can exist. Two
2325neighbouring holes were closed in the same pass: an intact-but-unterminated
2326*header* no longer truncates to a headerless file that reports success and
2327then fails every read (the header is restored from the schema), and a
2328sidecar corrupted with non-UTF-8 bytes now rebuilds instead of raising on
2329every read.
2330
2331**Verification.** The strict-xfail acceptance tests that mapped the trigger
2332surface — every byte offset from a row's first embedded newline to its last
2333— were rewritten into the regression guards they were designed to become,
2334and the every-offset sweeps now pass as ordinary tests.
2335"
2336backup-restore Backup and Restore Data & Storage 340 How to create, verify, and restore runtime backups of a DBBASIC Object Server instance, including scheduled backups and the deliberately CLI-only restore path. "DBBASIC runtime backups protect the object loop by keeping source, state, logs, files, versions, schemas, and collection records recoverable together.
2337
2338The public backup helper is:
2339
2340```bash
2341python -m object_backup
2342```
2343
2344It creates a tar/gzip archive with a JSON manifest and a narrow runtime backup set.
2345
2346## Included
2347
2348Runtime backups include:
2349
2350```text
2351objects/
2352data/state/
2353data/logs/
2354data/versions/
2355data/source_changes/
2356data/schema_versions/
2357data/record_changes/
2358data/package_changes/
2359data/file_changes/
2360data/files/
2361data/schemas/
2362data/collections/
2363```
2364
2365That covers object source, object state, current logs, rotated compressed logs, source versions, source change history, schema change history, record change history, package change history, object file change history, object-owned files, schema metadata, and TSV-backed collection records.
2366
2367## Excluded
2368
2369Runtime backups deliberately exclude:
2370
2371- deployment secrets
2372- environment files
2373- systemd service files
2374- git history
2375- virtualenvs
2376- caches
2377- lock files
2378- temp files
2379- ephemeral rate-limit files
2380
2381The environment file and service unit still need operator-managed backups, but they should be stored through a secret-aware server backup process, not inside the portable runtime archive.
2382
2383## Create
2384
2385```bash
2386mkdir -p /var/backups/dbbasic-object-server
2387
2388python -m object_backup create \
2389 /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \
2390 --objects-dir /var/lib/dbbasic-object-server/objects \
2391 --data-dir /var/lib/dbbasic-object-server/data
2392```
2393
2394The command writes `dbbasic-backup-manifest.json` inside the archive. The manifest records the format version, timestamp, included roots, file count, byte count, and warnings.
2395
2396## Restore points
2397
2398Server-side mutations that can rewrite multiple runtime files should create a restore point before writing. Package installs use:
2399
2400```python
2401object_backup.create_runtime_restore_point(""package-hello-world"")
2402```
2403
2404By default, restore points are written under:
2405
2406```text
2407data/backups/
2408```
2409
2410Set `DBBASIC_BACKUPS_DIR` to move those archives to a mounted backup volume or another operator-managed path. Runtime backup archives do not include `data/backups/`, so restore points do not recursively back up other archives.
2411
2412Package restore uses the recorded restore point from a package install change, not an arbitrary package path. That package-scoped HTTP route (`POST /packages/{id}/restore`, gated by `DBBASIC_ENABLE_PACKAGE_RESTORE`) is separate from full-runtime restore, which stays CLI-only. It calls `restore_runtime_backup(..., overwrite=True, prune_extra=True)` so files created by the package are removed if they were not present in the snapshot. Pruning is limited to known runtime roots such as `objects/`, `data/state/`, `data/logs/`, `data/collections/`, `data/schemas/`, and `data/source_changes/`, `data/package_changes/`, `data/file_changes/`; `data/backups/` is left alone.
2413
2414## Verify
2415
2416```bash
2417python -m object_backup verify \
2418 /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \
2419 --json
2420```
2421
2422Verification checks that:
2423
2424- the archive can be read
2425- the manifest is present
2426- the manifest count matches the archive contents
2427- archive paths stay under `objects/` or supported `data/` roots
2428- symlinks, hard links, device files, and other unsafe member types are rejected
2429
2430## Restore
2431
2432Restore is a **deliberate CLI operation**, not an HTTP verb. It is destructive and whole-instance, so it is intentionally not exposed on the admin surface (`/admin/status` reports `capabilities.backups.can_restore: false`); a ""restore"" button one misclick from wiping live data is a footgun. The pairing is: **download a backup from Scroll or `GET /admin/backups/{id}/download` to get it, and this CLI to apply it.**
2433
2434Runtime backups live under `data/backups/` (or `DBBASIC_BACKUPS_DIR`) and are named `YYYYMMDDTHHMMSSffffffZ-<label>.tar.gz` (label `manual` for on-demand, `package-<id>` for install restore points). Always verify first (see above).
2435
2436### Dry-run into clean directories first
2437
2438```bash
2439mkdir -p /tmp/dbbasic-restore/objects /tmp/dbbasic-restore/data
2440
2441python -m object_backup restore \
2442 /var/lib/dbbasic-object-server/data/backups/20260708T145041382734Z-manual.tar.gz \
2443 --objects-dir /tmp/dbbasic-restore/objects \
2444 --data-dir /tmp/dbbasic-restore/data \
2445 --json
2446```
2447
2448By default restore refuses to overwrite existing files, so this dry run never touches a live runtime. Start the server against the restored directories and check it before trusting the archive:
2449
2450```bash
2451DBBASIC_OBJECTS_DIR=/tmp/dbbasic-restore/objects \
2452DBBASIC_DATA_DIR=/tmp/dbbasic-restore/data \
2453uvicorn object_server:app --host 127.0.0.1 --port 8001
2454```
2455
2456Then verify `/health` and representative object routes. A real recovery drill should do this on a second clean VM.
2457
2458### Recovering the live runtime
2459
2460Stop the service first so nothing writes during the restore. Use `--overwrite` to replace existing files, and `--prune-extra` (which requires `--overwrite`) only when you want an exact match to the archive, removing files added since — a powerful, opt-in flag:
2461
2462```bash
2463sudo systemctl stop dbbasic-object-server
2464
2465sudo -u dbbasic /opt/dbbasic-object-server/.venv/bin/python \
2466 -m object_backup restore \
2467 /var/lib/dbbasic-object-server/data/backups/<id>.tar.gz \
2468 --objects-dir /var/lib/dbbasic-object-server/objects \
2469 --data-dir /var/lib/dbbasic-object-server/data \
2470 --overwrite --prune-extra --json
2471
2472sudo systemctl start dbbasic-object-server
2473curl http://127.0.0.1:8001/health
2474```
2475
2476Pruning is limited to known runtime roots (`objects/`, `data/state/`, `data/logs/`, `data/collections/`, `data/schemas/`, `data/source_changes/`, `data/package_changes/`, `data/file_changes/`); `data/backups/` is left alone, so restoring never destroys your other archives.
2477
2478## HTTP surface
2479
2480On-demand backup and download are exposed on the admin API (and used by Scroll's Backup screen) — see [the HTTP contract](/docs/http-api-contract#backups):
2481
2482- `GET /admin/backups` — inventory (`id`, `created_at`, `size`, `kind`, `scope`).
2483- `POST /admin/backups` — create a full-runtime backup now.
2484- `GET /admin/backups/{id}/download` — stream the archive.
2485
2486All admin-gated (a backup contains credentials and service keys, so never public). Restore is deliberately absent from this surface — use the CLI above.
2487
2488## Scheduling
2489
2490Automatic backups are a **config option, off by default**. The run itself is an external timer (systemd or cron) invoking `object_backup.py create` on a cadence with retention; `DBBASIC_BACKUP_SCHEDULE` records the operator's intent so `/admin/status` and Scroll can display it. A sample daily systemd timer:
2491
2492```ini
2493# /etc/systemd/system/dbbasic-backup.service
2494[Service]
2495Type=oneshot
2496User=dbbasic
2497EnvironmentFile=/etc/dbbasic-object-server.env
2498WorkingDirectory=/opt/dbbasic-object-server
2499ExecStart=/opt/dbbasic-object-server/.venv/bin/python -m object_backup create \
2500 /var/lib/dbbasic-object-server/data/backups/scheduled-%%i.tar.gz
2501```
2502
2503```ini
2504# /etc/systemd/system/dbbasic-backup.timer
2505[Timer]
2506OnCalendar=daily
2507Persistent=true
2508[Install]
2509WantedBy=timers.target
2510```
2511
2512Note that on-box backups do not survive losing the VM — pair them with provider-level disk backups (or copy archives to a bucket) for disaster recovery. On-demand create and download always work regardless of any schedule.
2513"
2514per-tenant-storage Per-Tenant Storage: The SaaS Decomposition Data & Storage 350 A direction note on decomposing one shared collection into per-tenant storage roots — what it buys, what it costs, and why the industry is already headed there. "This is a direction note, not a build plan. It's recorded alongside the [append-only storage design guide](append-only-storage-design) because the two together answer ""how far can one TSV go, and what happens at SaaS scale"" — and the answer to the second question is not ""a bigger table.""
2515
2516## The observation
2517
2518The measured storage envelope (see the append-only storage design guide: tens of thousands of rows interactive today, tens of millions after append-only) looks like a limit under the one-big-table model. Per-tenant storage inverts it: a SaaS with 100,000 customers and ""a billion orders"" does not have a billion-row problem — it has 100,000 ten-thousand-row problems, each of which sits in the comfortable center of the measured envelope. The giant commingled table is an artifact of shared-database design, not a law of SaaS.
2519
2520## What decomposition buys
2521
2522- **Physical isolation.** Row-level security (`WHERE tenant_id = ?`, one missing predicate from a breach) becomes filesystem separation. The permission engine's row filters harden into file boundaries.
2523- **GDPR and export for free.** Right-to-be-forgotten is removing the tenant's directory. ""Export my data"" is handing over the tenant's own TSV files — human-readable, Excel-openable, no export pipeline.
2524- **Per-customer backup/restore already exists.** Scoped restore plus a read-only preview already ship as platform features, and they apply per tenant with zero new code: ""restore customer X's notes to yesterday"" is the same operation as any other scoped restore.
2525- **Per-tenant migrations.** Package upgrades ride the provenance/reconcile system tenant-by-tenant: incremental rollout, conflicts parked per tenant, no all-customers-at-once schema locks.
2526- **Noisy neighbors decompose.** One tenant's huge collection cannot slow another's small one; the records-cache LRU bound doubles as per-tenant cache fairness.
2527- **Tenant mobility is file movement.** Rebalancing tenants across VMs, or graduating one outlier to its own instance or to the per-collection backend escape hatch, is `mv` of a directory. Shared-nothing per tenant; horizontal scale without a distributed database.
2528- **Metering is `du`.** Storage billing per tenant is a directory size.
2529
2530## Honest costs
2531
2532- **Cross-tenant queries become scatter-gather.** Admin analytics (""orders today, all customers"") need rollup collections maintained by the daemon (aggregates are just another collection), or a scan across tenant directories. This is the standard sharding trade-off and should be designed as rollups, not ad-hoc scans.
2533- **File-count hygiene.** 100k tenants times N collections is roughly 1M files — fine for modern filesystems, but the directory layout should shard (e.g. `tenants/ab/abcd.../collections/...`) from day one.
2534- **The single-TSV invariant holds per collection per tenant** — and the whole-instance view (""all tenants' notes as one TSV"") is itself just a deterministic concatenation with a tenant column, satisfying the emit-one-file rule at the aggregate level whenever it's needed.
2535
2536## Prior art
2537
2538The industry is arriving at the same place from several directions: Cloudflare Durable Objects (SQLite per entity, storage co-located with logic), Turso (database-per-tenant as a product), and the SQLite-per-user pattern generally. Per-tenant single-file storage next to per-tenant code is the current direction of travel; this system's version is the flavor you can `cat`.
2539
2540There's also older advice in the flat-file lineage worth citing directly. Strozzi NoSQL's ""Big tables"" documentation opens with *""big tables should be avoided in the first place""* — use the Unix filesystem to pre-organize relations and keep each one small. Per-tenant decomposition is that doctrine, systematized.
2541
2542## Current state and gap
2543
2544Today the platform is multi-tenant by *rows* (`owner_id` row filters, one shared collection per name) and multi-site by *hosts* (`site_hosts`). The gap this note names: collections themselves are instance-global. The eventual shape is tenant-scoped collection roots under one instance — same schemas, same permission engine, same generators, same API surface, with the tenant resolved from identity the way hosts are resolved from requests today.
2545
2546This isn't scheduled work. It's recorded here so the append-only and cache work keep it reachable — nothing in either conflicts with tenant-scoped roots, since both operate per file.
2547"
2548site-routing Site Routing Sites, UI & Realtime 400 How to turn the object server into a webserver with clean public URLs, a route table, multi-domain hosting, and static asset handling. "Site routing turns the object server into a webserver: clean public URLs
2549(`/about`, `/articles/{uuid}`) resolve to objects, so a whole website can be
2550built, edited, and rolled back through the normal object loop with no deploy
2551step and no per-page reverse-proxy rewrites.
2552
2553It is off by default and enabled per deployment:
2554
2555```text
2556DBBASIC_ENABLE_SITE_ROUTES=true
2557```
2558
2559## Resolution Order
2560
2561Site routing runs only after every built-in route family has declined the
2562path, so reserved surfaces (`/objects`, `/admin`, `/api`, `/identity`,
2563`/login`, `/health`, `/collections`, `/schemas`, `/events`, `/packages`,
2564`/daemon`, `/permissions`, `/logout`) can never be shadowed by a site page.
2565
2566For everything else:
2567
25681. **Convention.** `/` maps to `site_home`, `/about` to `site_about`,
2569 `/docs/install` to `site_docs_install`. Hyphens become underscores
2570 (`/getting-started` → `site_getting_started`). Creating a page object
2571 creates its URL — the filesystem-routing feel of the original prototype,
2572 kept because it makes the common case zero-config.
25732. **The `site_routes` collection.** Records with `pattern`, `object_id`, and
2574 optional `priority` handle what conventions cannot express:
2575
2576 | id | pattern | object_id | priority |
2577 |----|---------|-----------|----------|
2578 | r1 | `/articles/{article_id:uuid}` | `articles_view` | 10 |
2579 | r2 | `/blog/{slug}` | `blog_post` | 20 |
2580
2581 Pattern segments are literal text, `{name}` (any single segment), or
2582 `{name:uuid}` (segment must be UUID-shaped — the common case for record
2583 ids, which are UUIDv4 by default). Captured params are merged into the
2584 object's request payload, so `articles_view` reads
2585 `request[""article_id""]`. Lower `priority` wins; ties go to the pattern
2586 with more literal segments.
25873. **`site_404`.** If nothing resolves and a `site_404` object exists, it
2588 runs with the missed `path` in its payload and should return
2589 `status_code: 404`. Without it, the plain JSON 404 stays.
2590
2591## Design Decisions
2592
2593These choices came out of comparing the original prototype's pure filesystem
2594router, the route-table-in-object-state design from the early dbbasic.com
2595plan, and what the public server has grown since:
2596
2597- **Routing only maps URLs; authorization stays in the permission policy.**
2598 The resolved object executes through the normal execution path, so
2599 enforcement, audit entries, execution logs, timeouts, concurrency limits,
2600 and correlation ids all apply. Publishing a page is two audited steps:
2601 create the object, grant `execute` to `public` (or whichever principal).
2602 A file appearing on disk never silently becomes public under enforcement.
2603- **The route table lives in records, not object state.** Records give the
2604 table schema validation, change history, rollback, Scroll editing, and MCP
2605 tools for free. Editing the sitemap is a data change with an audit trail.
2606- **Convention for the 90%, data for the 10%.** Plain pages need no table
2607 entry. The table exists for parameterized routes, renames, and pointing
2608 multiple URLs at one object.
2609- **TLS and static assets stay in the reverse proxy.** Caddy terminates TLS
2610 and should serve `/static/*` from disk with cache headers; site routing
2611 replaces per-page rewrites with one catch-all proxy block.
2612
2613## Deployment
2614
2615With site routing enabled, the Caddy shape becomes: explicit handles for the
2616reserved admin/API surfaces you want public, a `/static/*` file server, and a
2617catch-all `reverse_proxy` instead of a catch-all 404. Under permission
2618enforcement, unrouted or unauthorized paths still return controlled 403/404
2619responses from the server.
2620
2621## Multiple Domains
2622
2623One object server can serve many websites. The `site_hosts` collection maps
2624request hosts to object prefixes:
2625
2626| id | host | prefix | home_object | not_found_object |
2627|----|------|--------|-------------|-------------------|
2628| h1 | dbbasic.com | db | | |
2629| h2 | example.org | eg | | |
2630
2631With those records, `/` on dbbasic.com serves `db_home` and `/pricing`
2632serves `db_pricing`, while `/` on example.org serves `eg_home` — same
2633server, same data directory, fully separate page namespaces. `home_object`
2634and `not_found_object` default to `{prefix}_home` and `{prefix}_404`
2635(falling back to the generic `site_404`). Hosts without a record keep the
2636default `site_*` behavior, so single-site deployments need no
2637configuration. `site_routes` pattern records take an optional `host`
2638column; host-specific rows only match their domain and beat host-agnostic
2639rows.
2640
2641Pointing a new domain at the server is then:
2642
26431. DNS: point the domain's A record at the VM.
26442. Caddy: add a site block for it and reload — Caddy only answers (and only
2645 fetches certificates) for hostnames it has a block for, and DNS must
2646 already point at the VM for the certificate challenge to succeed. With
2647 the `public_site` snippet defined once at the top of the Caddyfile, each
2648 new domain is three lines:
2649
2650 ```caddyfile
2651 (public_site) {
2652 handle /login { reverse_proxy 127.0.0.1:8001 }
2653 handle /logout { reverse_proxy 127.0.0.1:8001 }
2654 handle_path /static/* {
2655 root * /var/lib/dbbasic-static/{args[0]}
2656 file_server
2657 header Cache-Control ""public, max-age=3600""
2658 }
2659 handle { reverse_proxy 127.0.0.1:8001 }
2660 }
2661
2662 newsite.example {
2663 import public_site newsite
2664 }
2665 ```
2666
2667 Public domains expose only the site surface; keep the admin/API/MCP
2668 handles only on the operations domain, so public sites never route to
2669 the operator surface at all.
26703. Add the `site_hosts` record and create `{prefix}_home` — the site is live.
2671
2672Identity, sessions, records, policy, and audit are shared across domains:
2673one operator account works everywhere, and browsers scope session cookies
2674per domain automatically. Domains that must not share those belong on their
2675own server instance instead (one process per trust boundary).
2676
2677## Static Assets
2678
2679Site CSS, JavaScript, images, and fonts are served by the reverse proxy from
2680a plain directory — never through object execution — so they get file-server
2681performance, gzip, and cache headers:
2682
2683```caddyfile
2684handle_path /static/* {
2685 root * /var/lib/dbbasic-static
2686 file_server
2687 header Cache-Control ""public, max-age=3600""
2688}
2689```
2690
2691Pages reference them normally (`<link href=""/static/site.css"">` in the
2692rendered HTML body). The directory is deploy-time content owned by the
2693service user; treat it like object source (versioned in the site package),
2694not like runtime data. User-uploaded files are different: keep those in the
2695object file APIs so they inherit permissions, audit, and backups.
2696
2697## Writing Site Pages
2698
2699Site pages are normal objects (see the object authoring guide): return a
2700dict for JSON, a `content_type`/`body` dict for HTML, use
2701`request[""_identity""]` for per-user rendering, and handle `POST` for forms —
2702browser form posts arrive as parsed fields with the session cookie applied.
2703Ship a site as a package (see the package authoring guide) so it installs
2704with its schema, seed records, and route table rows in one reviewed unit.
2705"
2706generative-ui Generative UI — One Renderer, Every App Sites, UI & Realtime 410 How a collection's schema alone drives list, form, and detail rendering across every app, and why customizing a generated page never means forking generated code. "There is no per-app UI code. A collection's schema drives a single set of
2707generative renderers, served as static scripts (`/list`, `/form`, `/detail`),
2708that turn records into live, permission-checked, realtime surfaces. This page
2709is the overview; the field-level contract is in
2710[`schema-forms.md`](/docs/schema-forms), the behavior decisions in
2711[`ui-decisions.md`](/docs/ui-decisions), and the visual system in
2712[`design-system.md`](/docs/design-system).
2713
2714```mermaid
2715flowchart LR
2716 SCH[""schema<br/>fields · views · capabilities""] --> LIST[""window.dbbasicList""]
2717 SCH --> FORM[""window.dbbasicForm""]
2718 SCH --> DET[""window.dbbasicDetail""]
2719 LIST --> MODES[""list · table · board · tree · calendar""]
2720 LIST --> FSR[""filters · search · sort · row cap""]
2721 FORM --> CTRL[""type-aware controls · validation""]
2722 DET --> COMP[""detail + related + capabilities""]
2723 CL[""change log (websocket)""] -. re-render .-> LIST
2724 CL -. re-render .-> DET
2725```
2726
2727## The list, in five modes
2728
2729`window.dbbasicList(collection, cfg)` resolves `views.list_mode` from the
2730schema and renders one of:
2731
2732- **list** (default) — rich rows: avatar, title, subtitle, meta, tag pills,
2733 owner edit/delete.
2734- **table** — a dense, sortable HTML table over `list_fields`; cells format by
2735 type (money `_cents` → `$`, boolean → Yes/No, datetime → relative, enum →
2736 colored badge), and **relation columns show the target's label, not the raw
2737 id**. Owned rows get the same edit/delete the list has.
2738- **board** — a kanban grouped by an enum field; dragging a card issues the
2739 ordinary status write, so `flow` transitions still gate the move. A board
2740 collection that also has `list_fields` shows a **Board ⇄ Table** toggle whose
2741 choice persists per collection.
2742- **tree** — nests a self-relation (`parent_id`) into a hierarchy.
2743- **calendar** — buckets records by a date field.
2744
2745Every mode shares **one** fetch/sort/cap/search/subscribe pipeline, so all of
2746them inherit:
2747
2748- **filters** — `views.filter_fields` renders a filter bar (enum → select,
2749 boolean → Yes/No); a pick narrows the fetch server-side (`field=value`, after
2750 the permission row filter) and composes with the search box.
2751- **search** — the toolbar box, server-side over the collection's `search`
2752 fields.
2753- **the 50-row cap** with a Show-all toggle (no surface ever renders a
2754 50,000px page).
2755- **realtime** — each surface subscribes to the change log and re-renders when
2756 a record changes, from any tab, user, or agent.
2757
2758A mode whose required field can't be derived (no enum for board, no
2759self-relation for tree, no date for calendar) falls back to the row list with a
2760visible notice — never a blank page.
2761
2762## The form
2763
2764`window.dbbasicForm(collection)` builds a record form from the schema: field
2765order from `forms.default.fields`, controls from field semantics (enum →
2766select, relation → picker, boolean → checkbox, date → date input, textarea,
2767number, text), and labels/help/required/max-length from the field. It handles
2768create (POST) and edit (PUT), sets `id`/`owner_id`/`created_at` automatically,
2769never writes computed/read-only fields, and supports conditional field
2770visibility (`visible_when`). See [`schema-forms.md`](/docs/schema-forms).
2771
2772## The detail page
2773
2774`window.dbbasicDetail` renders one record read-only by reusing the form's own
2775field renderer in read-only mode — no second field renderer to keep in sync. It
2776adds owner-only Edit/Delete (reusing the form's edit pipeline), always shows
2777record metadata (`created_at`/`updated_at`) even when `detail_fields` curates
2778the main fields, and auto-mounts any declared **capabilities** (comments,
2779attachments, sharing — see [`capabilities.md`](/docs/capabilities)) below the
2780detail. Composed detail pages are `views` records with a `detail` block plus
2781`related` child blocks; the view renderer
2782(`app-views/objects/site/view_render.py`) assembles them.
2783
2784A `textarea`-type field shown read-only gets a safe, minimal markdown
2785subset (bold, italic, links, paragraph breaks) — the same transform the
2786standalone `markdown` block below applies — instead of plain escaped text,
2787which used to collapse multi-line content into one run-together line.
2788
2789## Other view block kinds
2790
2791Beyond `list`/`form`/`detail`/`related`, a view's `blocks` array can also
2792declare:
2793
2794- **`markdown`** — a block of static prose baked directly into the view
2795 definition (`{kind: ""markdown"", text: ""...""}`), the same escape-first,
2796 safe-subset renderer as the textarea treatment above.
2797- **`reader`** — `{kind: ""reader"", url: ""...""}` fetches a URL through the
2798 flag-gated, signed-in-only `POST /api/read` and renders title + body text
2799 + a numbered link list. See [`shell-and-ai.md`](/docs/shell-and-ai)'s
2800 ""Reading the Web"" section for the SSRF posture and the voice-navigation
2801 loop this backs.
2802- **`count`** / **`aggregate`** — live stat cards over a collection (row
2803 count, or a sum of numeric/`_cents` fields, with an optional
2804 balanced/not-balanced badge for double-entry-style checks).
2805- **`thread`** — sugar for a comment/reply thread capability.
2806- **`object`** — mounts an arbitrary object's own rendered output inline.
2807
2808Every kind subscribes to its source collection and re-renders live, the
2809same as `list`/`related`. An unrecognized kind (or one missing a required
2810field) renders a small ""unsupported"" placeholder card rather than breaking
2811the page.
2812
2813## Why this is the point
2814
2815Adding an app is a schema file. The list, the table, the board, the toggle,
2816the filters, the form, the detail page, badges, the row cap, relation labels,
2817realtime, and owner actions all fall out of it — the same code for every app.
2818The measure isn't the feature list (anyone can claim those); it's the amount of
2819per-app code, which is a schema and at most one page object.
2820
2821## This is not scaffolding
2822
2823Rails scaffolds and Django's startapp share one failure that defined a
2824generation of frameworks: the generator emits **code**, you edit it, and from
2825that moment you own all of it. The generator can never help that screen again
2826— template improvements don't flow, upgrades don't apply, and nothing records
2827which parts you changed. The fork happens at generation time and diverges
2828silently forever. ""Eject"" is the same failure with a button on it.
2829
2830The rule that keeps this system on the other side of that line:
2831
2832> **Generated output is never materialized, so it can never be edited.**
2833> A page is a view record *interpreted at request time* by a renderer that is
2834> itself an ordinary object. Customizing never means editing emitted code,
2835> because there is no emitted code. You either change **data** (and keep
2836> receiving renderer improvements) or you write **your own object** (which was
2837> never generated, so nothing was forked).
2838
2839### The customization ladder
2840
2841Each rung says exactly what you take ownership of and what keeps flowing:
2842
28431. **Edit the block config** (filters, `title_field`, sort, `row_limit`) — you
2844 own the config; every renderer improvement still flows.
28452. **Add blocks** — a view is a list of blocks; put an `aggregate` summary or
2846 `markdown` intro above the generated list. Same ownership as (1).
28473. **Customize a shipped view record** — the upgrade system detects it against
2848 the package baseline and parks a reconcile instead of clobbering: the system
2849 *records* what you own and shows you when upstream moved. Scaffolding's
2850 fork is silent; this one has a ledger.
28514. **Drop an `object` block into a generated page** — one hand-written panel
2852 among generated ones (`{""kind"": ""object"", ""object_id"": ""site_my_panel""}`).
2853 You own that panel; the rest of the page keeps flowing. It runs through the
2854 ordinary execution path, so permissions and audit apply — the block widens
2855 what a page can *show*, never what a viewer may *see*.
28565. **Write the whole page object.** Convention routing means creating
2857 `site/notes.py` instantly overrides the generated `/notes` — and deleting
2858 the file reverts to it. The override is total, explicit, and reversible,
2859 and it is visible in the objects listing rather than buried in a diff
2860 against generator output.
2861
2862### Why replacing a page is safe here and wasn't in Rails
2863
2864The deep failure of scaffolds wasn't the HTML — it was that the scaffold
2865contained **rules** (validations in the controller, logic in the view), so
2866editing it forked the rules. Here the doctrine is that *a page may never be
2867the only place a rule exists*: enforcement lives in the schema, hooks,
2868transitions, and permissions, on the shared write path that the JSON API, MCP
2869agents, and the websocket all go through.
2870
2871That doctrine is not just policy — the architecture makes the wrong place
2872**fail loudly**. A rule written into a page object simply does not gate API or
2873MCP writes, so the mistake shows up the first time anything that isn't the
2874page touches the collection. In Rails, logic in the controller *worked* for
2875web users, which is exactly why the fork stayed invisible for years. The
2876residual risk here is someone treating rung 4–5 as a place to put gates; the
2877system's answer is that gates there provably don't hold, and the hook is one
2878file away.
2879
2880## Related
2881
2882- [`schema-forms.md`](/docs/schema-forms) — the field-level contract these
2883 renderers read.
2884- [`ui-decisions.md`](/docs/ui-decisions) — the living log of interaction
2885 decisions (detail-vs-edit, the row cap, board flex, filters, relation labels,
2886 capabilities) and the reasons behind them.
2887- [`design-system.md`](/docs/design-system) — semantic tokens, themes as data, and
2888 the stylesheet served as an object.
2889- [`capabilities.md`](/docs/capabilities) — the behavior layer mounted on detail.
2890"
2891design-system The Design System Sites, UI & Realtime 420 How DBBASIC's design system works as a semantic contract with a thin, themeable renderer, delivered as a live object rather than a static component library. "DBBASIC's design system is not a component library you depend on — it is
2892a **semantic contract with a thin, themeable renderer, delivered as an
2893object.** It has three layers, and only the top one is pixels.
2894
2895```mermaid
2896flowchart TD
2897 SEM[""Layer 1 — Semantics<br/>schema: types, enum, relation,<br/>list_mode, transitions, permissions""] --> REND
2898 TOK[""Layer 2 — Tokens<br/>semantic roles: surface, text,<br/>accent, positive/warning/danger""] --> REND
2899 REND[""Layer 3 — Renderers<br/>web pages · Scroll · shell · AI""]
2900 TOK -.theme = a set of role values.-> THEMES[""Themes as data:<br/>base · paper · terminal · yours""]
2901```
2902
2903## The governing idea: define once, project everywhere, own no copies
2904
2905Everything below is one principle seen from different angles. State the
2906design once — as tokens, as a shared stylesheet object, as a nav object,
2907as a schema — and let every surface *project* it. No page owns a copy of
2908the theme; it references `/style`. No page owns a copy of the nav; it
2909includes `/nav`. No app hardcodes a widget; it declares a field's
2910meaning. Because nothing downstream owns a copy, changing the source
2911changes the whole product at once, with nothing to chase.
2912
2913This is why rebranding the entire twelve-app suite and giving it a shared
2914navigation bar was **two objects edited live** — no deploy, no migration,
2915no per-page rework. If a future change to the look or the chrome ever
2916requires editing many pages, that is the signal that a copy leaked in and
2917should be pulled back into a shared object. Guard this property; it is the
2918whole value.
2919
2920## Layer 1 — Semantics (the source of truth)
2921
2922Design decisions live in the schema, not in markup. A field that declares
2923`enum` *means* ""one of these values""; each surface picks its own control
2924(dropdown on the web, segmented control in Scroll, a spoken list to the
2925AI). `views.list_mode` picks table / cards / feed. `transitions` says
2926which state moves are legal. Permissions decide which controls exist at
2927all. See [schema forms](/docs/schema-forms) and [permissions](/docs/permissions-model).
2928
2929**We deliberately do not use a widget-prescriptive `ui_schema`.** A
2930predecessor system paired each schema with a React-JSON-Schema-Form
2931`ui_schema` that said things like `""ui:widget"": ""radio""`. That only ever
2932renders on one surface — a `radio` hint is meaningless to an AI, the
2933shell, a voice interface, or a native app. Semantics render everywhere;
2934widget instructions render once. This is the single most important choice
2935in the system, and the reason a design here can span surfaces that do not
2936exist yet.
2937
2938**Permissions are affordances.** A control exists because the caller can
2939perform the action, not because markup drew it and then hid it. A delete
2940button appears when the policy allows delete; a project filter appears
2941because there is a `project_id` relation; a column appears because
2942`list_fields` lists it. The design system renders *capability* — which is
2943why an AI acting for a user, or a shared component, can never show a
2944control the user could not actually use.
2945
2946## Layer 2 — Tokens (roles, not colors)
2947
2948The visual vocabulary is a small set of **semantic roles** — surface
2949(`bg`, `panel`, `line`), text (`text`, `muted`), `accent`, and status
2950(`positive`, `warning`, `danger`) — plus spacing, radii, shadow, and two
2951type families. Components are written against the roles, never against
2952raw colors, so a retheme changes values in one place and nothing in any
2953page moves.
2954
2955## Layer 3 — Renderers, and the design system *as an object*
2956
2957The stylesheet is itself a DBBASIC object: **`site_style`, served at
2958`/style`.** Every page links to it (`<link rel=""stylesheet"" href=""/style"">`)
2959instead of carrying its own CSS. Because it is an object, the whole visual
2960system is versioned, rollback-able, and live-editable like everything
2961else — retheming is one reversible object change, not a deploy.
2962
2963Other renderers consume the same two layers: Scroll's generated Flutter
2964forms and lists, the shell's terminal, and the AI (whose ""UI"" is the
2965schema's `label`/`help` text — good microcopy is design work that serves
2966the form, the accessibility label, and the AI tool description at once).
2967
2968## The app shell — chrome contributed, not copied
2969
2970Navigation is part of the design system, and it follows the same rule.
2971The **`site_nav` object, served at `/nav`,** injects one shared top bar
2972into every page that includes `<script src=""/nav""></script>`: brand, an
2973app switcher across installed apps, global search (`⌘K` / `Ctrl-K`) over
2974`/api/search`, an Ask-AI link to the shell, a notification bell reading
2975the `notifications` collection, and a user menu (appearance, sign out).
2976Its styling lives in `/style`, so it themes with everything else.
2977
2978The point is that a *package* (`app-theme`) contributes this bar to every
2979other app without those apps knowing — a thing a Django app could never do
2980to its siblings. Adding a nav item is one edit to one object; every app
2981gets it. If you find yourself pasting nav markup into a page, stop: add it
2982to `site_nav`.
2983
2984The bell is **live**: it subscribes to the `notifications` collection over
2985the `/ws` websocket and updates the instant something happens, with a 20s
2986poll as backup. Every app page is live the same way — each subscribes to
2987its collection and auto-refreshes (150ms debounced) when a record it can
2988see changes, over the one shared socket, via
2989`window.dbbasicSubscribe(collection, handler)`. Add a live list to a new
2990page in one line. Auto-update on events is the thing older
2991dashboard/Channels-based stacks never did cleanly — see the
2992[realtime contract](/docs/http-api-contract#realtime-push-websocket).
2993
2994## Themes are data — and packages
2995
2996A **theme is just a set of values for the token roles.** `site_style`
2997ships three built in:
2998
2999- **base** — the DBBASIC identity: a warm dark (per the earth-theme rule
3000 ""warm, not cold blue-black"") with a terracotta/ember accent, carried
3001 forward from an earlier internal palette.
3002- **paper** — a warm light theme.
3003- **terminal** — high-contrast green-on-black.
3004
3005The active theme lives in the object's state. There is a visual chooser
3006at **`/appearance`** (the `site_appearance` object): it reads the
3007available themes and their preview swatches from `/style?info=true` and
3008lets an admin click one to reskin the whole instance; non-admins see the
3009active theme read-only. Or switch over HTTP:
3010
3011```http
3012POST /style {""theme"": ""terminal""} # admin session only
3013POST /style {""tokens"": {""accent"": ""#5aa7ff""}} # custom role overrides
3014GET /style?info=true # {active, available, previews, tokens}
3015```
3016
3017`site_style` self-gates writes to admin sessions (using the unspoofable
3018injected identity), so a public `execute` grant can serve the CSS to
3019everyone while only operators change the look.
3020
3021**A theme is a package.** Installing a package that ships its own
3022`site_style` (or that sets custom tokens) reskins the whole instance —
3023design distributed the same way apps are. A community can publish themes;
3024you install the one you like; you can fork it because it is one small,
3025readable object. Per-user theming (each account choosing a theme) is a
3026planned extension built on the same roles plus the `shell_preferences`
3027pattern.
3028
3029## Forms, tables, search, and filters
3030
3031The base stylesheet carries a genuinely complete component vocabulary so
3032generated UIs and hand-written pages share one look: form fields with
3033labels/help/required markers and validation states; tables with sortable
3034headers, hover, numeric alignment, and pagination; a toolbar with a search
3035input and filter chips; buttons as **intents** (primary / neutral /
3036danger) rather than colors; badges; and first-class empty / loading /
3037denied / error states.
3038
3039That is the visual layer. The **generative renderer** that reads a schema
3040and emits these — the successor to the declarative table/filter libraries
3041(`django_tables2` + `django_filters`) used in an earlier internal system,
3042which worked because they were declarative from a single source — unified
3043across table, cards, and feed with search and filters wired to
3044`search.fields` — is the next build. The honest lesson from that earlier
3045system is that this substrate must be *one* renderer, not the four
3046disconnected implementations it grew (`django_tables2` for tables, two
3047incompatible form engines, hand-written cards and feeds). The semantics
3048and the token vocabulary are now in place for it.
3049
3050## Building a page against the design system
3051
3052```html
3053<link rel=""stylesheet"" href=""/style"">
3054...
3055<div class=""wrap"">
3056 <header class=""app""><h1>Notes</h1><div class=""who"">…</div></header>
3057 <div class=""toolbar"">
3058 <input class=""search grow"" placeholder=""Search…"">
3059 <div class=""filters""><button class=""chip"" aria-pressed=""true"">All</button></div>
3060 </div>
3061 <div class=""cards"">…</div>
3062</div>
3063```
3064
3065No inline CSS, no framework, no build step — just the shared classes. When
3066the instance's theme changes, this page changes with it. Page-unique
3067layout (a form's column grid, an edit-mode toggle) may stay in a small
3068`<style>` block, but it must use the token variables (`var(--panel)`,
3069`var(--line)`, `var(--accent)`) — never hardcoded colors — so it reskins
3070too.
3071
3072## Why this is a design system, not a style guide
3073
3074The test that separates the two: **change the source, and does the whole
3075product change with no copies to chase?** Here it does — edit `/style`,
3076every page reskins; edit `/nav`, every page's chrome updates; edit a
3077schema, every surface re-renders that field. A folder of CSS classes and a
3078component catalog is a style guide; you still hand-apply it and it drifts.
3079A design system propagates from one place. Protect that difference: the
3080day a change stops propagating is the day it quietly became a style guide.
3081
3082## Why not a dashboard template
3083
3084The web-template industry (Bootstrap admin themes, and the marketplaces
3085that sell them — an earlier internal project shipped on the ""Volt""
3086dashboard) assumed **everything is a dashboard**: a fixed chrome of
3087sidebar, topbar, cards, and a data table, into which you pour your app.
3088That is widgets-before-meaning — the shape is the product, and the only
3089app you can express is the one the widgets already draw. It is the same
3090mistake as a widget-prescriptive `ui_schema`, at the scale of a whole
3091product. It is also why that earlier project accumulated 110 KB of
3092override CSS *fighting* its template: when the chrome is copied into
3093every page, changing it means fighting it everywhere.
3094
3095Reading modes (`table` / `cards` / `feed`) are the direct rejection of the
3096one-shape assumption: a note is read individually (cards), a task is
3097compared to others (table), a message is sequential (feed). The dashboard
3098world only had the table.
3099
3100The reason no one built it this way earlier is not taste, it is
3101arithmetic. Projecting one schema into a web form, a table, a Flutter
3102widget, an AI tool description, and an accessibility label meant
3103hand-writing five renderers per app — insane in 2015, so buying the frozen
3104template was rational. What changed is that the schema is now legible to a
3105model, a model can render a surface, and the runtime hot-loads it with no
3106deploy — so the marginal cost of a surface collapsed, and the correct
3107architecture (declare meaning once, project everywhere) finally pays. Build
3108for that, not for the dashboard era.
3109
3110## Current edges (honest)
3111
3112- **Theme-complete, component-incomplete.** The token/theme/app-shell
3113 layers are done and propagate. The component *vocabulary* exists in
3114 `/style`, but pages still assemble it by hand — the **generative
3115 renderer** (schema → dense list with row actions, filters, and relative
3116 timestamps) is not built yet. It is the next big design build, and it
3117 needs a `created_at` field added to schemas first (nothing captures
3118 creation time today, so there are no relative timestamps to show).
3119- **Web + tokens today.** Scroll shares the token *roles* but the web and
3120 Flutter renderers are not unified against one contract yet.
3121- **Instance-level theming.** Per-user theme choice is a planned extension
3122 on the same roles plus `shell_preferences`.
3123- **The shell page** (`/shell`) deliberately keeps its own terminal palette
3124 and does not link `/style` or `/nav` — a focused full-screen mode. The
3125 admin dashboard is not yet on the shared shell either.
3126
3127## Extending it correctly — rules for future builds
3128
3129When adding to the design system, keep the propagation property intact:
3130
31311. **Contribute, don't copy.** New chrome goes in a shared object
3132 (`site_style`, `site_nav`), never pasted into pages. If it must live in
3133 a page, it is page-unique and must use tokens.
31342. **Tokens are roles, never hex.** Add a role (e.g. `--info`) if you need
3135 a new meaning; never a raw color in a component or a page.
31363. **Semantics, not widgets.** Express new capability as schema meaning
3137 (a new field key, a new `views` mode) that every surface can interpret
3138 — resist any per-surface ""render as X"" instruction; that is `ui_schema`
3139 returning, and it collapses you back to one surface.
31404. **Let permissions draw the controls.** Show an action because the
3141 policy allows it, not because you wrote the button.
31425. **One renderer per concern.** When the generative renderer lands, route
3143 all lists/forms through it — do not grow a second form engine. A
3144 previous system grew four; that fragmentation is the failure mode to
3145 avoid.
31466. **No framework to fight.** Do not adopt a CSS framework you will then
3147 override. The system is small and authoritative on purpose.
31487. **If a change stops propagating, fix the leak** — pull the copy back
3149 into the shared object — rather than editing many pages.
3150
3151Follow these and the next person (or the next model, or a stranger who
3152forks this) can retheme, re-chrome, or extend the whole product from a
3153handful of objects — which is the point.
3154"
3155rest-and-object-messages REST and Object Messages Sites, UI & Realtime 430 How the DBBASIC Object Server splits its HTTP API into RESTful resources, object messages, and realtime streams — and why that split matters. "DBBASIC should be RESTful where it exposes resources and direct where it executes behavior.
3156
3157The goal is not to pretend every operation is pure REST. The goal is to keep a clean split. The HTTP API breaks down into three kinds of surface:
3158
3159- **Resources** — `source`, `state`, `logs`, `files`, `versions`, `metadata`
3160- **Object messages** — execute method, run job, handle webhook, perform admin action
3161- **Streams** — WebSocket events, Server-Sent Events
3162
3163## Why This Matters
3164
3165Many APIs called REST are really RPC over HTTP:
3166
3167```text
3168POST /doThing
3169POST /executeAction
3170POST /api/run
3171```
3172
3173That shape is easy to add, but it hides what the system actually owns. DBBASIC objects own real resources: source, state, logs, files, versions, and metadata. Those should be exposed as resources with stable paths, predictable HTTP methods, status codes, and JSON representations.
3174
3175Object behavior is different. Executing an object is message passing. Jobs, events, webhooks, and realtime commands are also messages. They should be documented as messages instead of disguised as fake CRUD resources.
3176
3177## RESTful Resource Surface
3178
3179DBBASIC should keep these resource-style endpoints boring and predictable:
3180
3181```text
3182GET /objects
3183POST /objects
3184GET /objects/{object_id}?source=true
3185PUT /objects/{object_id}?source=true
3186GET /objects/{object_id}?state=true
3187GET /objects/{object_id}?logs=true
3188GET /objects/{object_id}?versions=true
3189GET /objects/{object_id}?version=N
3190DELETE /objects/{object_id}?destroy=true
3191```
3192
3193These endpoints expose things the object server owns.
3194
3195They should preserve:
3196
3197- stable object identity
3198- standard HTTP status codes
3199- JSON response shapes existing clients can keep using
3200- explicit authorization checks
3201- useful cache behavior where safe
3202- compatibility with the HTTP API contract
3203
3204## Object Message Surface
3205
3206Object execution is not only CRUD. It is behavior:
3207
3208```text
3209GET /objects/{object_id}
3210POST /objects/{object_id}
3211PUT /objects/{object_id}
3212DELETE /objects/{object_id}
3213```
3214
3215Those calls deliver a method and payload to the object.
3216
3217That model is closer to:
3218
3219```text
3220object_id + method + payload -> result
3221```
3222
3223than:
3224
3225```text
3226table + row + CRUD operation
3227```
3228
3229This is intentional. A DBBASIC object can represent a report, worker, webhook, view, business process, queue handler, generated UI, or admin tool.
3230
3231## Rollback
3232
3233Rollback is the awkward case:
3234
3235```text
3236POST /objects/{object_id}
3237{""action"": ""rollback"", ""version_id"": 1}
3238```
3239
3240That is a message, not a pure REST resource update. It is kept for compatibility with existing clients.
3241
3242Future server code may add a cleaner resource-style alias, but it should not break the existing message shape without a migration:
3243
3244```text
3245POST /objects/{object_id}/versions/{version_id}/rollback
3246```
3247
3248## Design Rule
3249
3250Do not turn every operation into `/api/doThing`.
3251
3252Use resources for object-owned data:
3253
3254```text
3255source, state, logs, files, versions, metadata
3256```
3257
3258Use messages for object-owned behavior:
3259
3260```text
3261execute, run, handle, repair, migrate, broadcast
3262```
3263
3264Use streams for object-owned realtime feedback:
3265
3266```text
3267source changed, state changed, log appended, version saved, execution failed
3268```
3269
3270That is the honest shape of DBBASIC: REST where resources are real, object messages where behavior is real, and streams where live feedback is real.
3271"
3272realtime Realtime: The Change Log and Live Surfaces Sites, UI & Realtime 440 How DBBASIC Object Server pushes change notifications over a websocket without ever putting record data on the wire, and why that design choice matters. "Every generative surface — every list, table, board, detail page, comment thread, and attachment list — subscribes to the change log and re-renders when a record it can see changes, from any tab, user, or agent. This falls out of one small websocket contract with a deliberate rule: **the wire carries a signal, never the record body.**
3273
3274```mermaid
3275sequenceDiagram
3276 participant Writer as any write (web/API/agent)
3277 participant Server
3278 participant Hub as realtime hub
3279 participant Client as a live surface
3280
3281 Client->>Server: connect /ws (session), subscribe [collections]
3282 Server->>Server: may this subject READ each collection?
3283 Server-->>Client: {type: subscribed, collections}
3284 Writer->>Server: create/update/delete a record
3285 Server->>Hub: publish {collection, record_id, action}
3286 Hub->>Hub: per subscriber, re-run the READ decision
3287 Hub-->>Client: {type: record, collection, record_id, action}
3288 Client->>Server: refetch through the normal permission-enforced API
3289 Client->>Client: re-render
3290```
3291
3292## Transport
3293
3294`GET /ws` (upgraded to a websocket). The connection is **authenticated** — a session cookie or bearer token — and anonymous connections are refused (`1008`). Realtime is behind a flag; when off, surfaces fall back to polling.
3295
3296## Client protocol
3297
3298Messages are JSON. On accept, the server sends `{""type"": ""welcome"", ""user"": ...}`. Then:
3299
3300| Client sends | Server replies |
3301|--------------|----------------|
3302| `{""action"": ""subscribe"", ""collections"": [""tasks"", ""notes""]}` | `{""type"": ""subscribed"", ""collections"": [...]}` — only the collections the subject may read are kept |
3303| `{""action"": ""unsubscribe"", ""collections"": [...]}` | `{""type"": ""subscribed"", ""collections"": [...]}` |
3304| `{""action"": ""ping""}` | `{""type"": ""pong""}` |
3305
3306Subscriptions are capped per connection, and each collection name is validated.
3307
3308In the browser this is wrapped as one helper, `window.dbbasicSubscribe(collection, handler)` (served by the `nav` object): it opens the socket once, subscribes on demand, reconnects with backoff, and runs a ~20-second poll as a fallback for missed events and multi-worker deployments. A page never talks to `/ws` directly — it calls `dbbasicSubscribe` and re-renders in the handler.
3309
3310## The event
3311
3312A change pushes exactly:
3313
3314```json
3315{ ""type"": ""record"", ""collection"": ""tasks"", ""record_id"": ""…"", ""action"": ""update"" }
3316```
3317
3318`action` is `create` / `update` / `delete`. **No field values travel** — only which record changed. The client reacts by *refetching* through the ordinary API, which re-applies the full permission policy and field redaction. So a live surface can never show a field the API wouldn't have served.
3319
3320## Permission filtering — twice
3321
3322Realtime is inside the permission engine, not beside it:
3323
33241. **On subscribe** — a subject may follow a collection only if it may `READ` it (`_ws_can_subscribe` runs the same check a `GET` would).
33252. **On publish** — for each subscriber, the server re-runs the READ decision *against the changed record*, so a row-filtered subscriber only hears about rows its filter would let it see. If the policy can't be loaded to filter safely, nothing is pushed.
3326
3327Combined with the signal-only payload, this means the change log leaks neither the existence of records you can't see (filtered per-subscriber) nor their contents (never sent).
3328
3329## Why signal, not body
3330
3331- **Security** — the wire can't leak a field, because it carries none. All reads go back through the one enforced path.
3332- **Simplicity** — there is no second serialization/redaction path to keep in sync with the API; the client just refetches.
3333- **Correctness under races** — the refetch always returns the current row, so a burst of edits collapses to one correct re-render rather than replaying stale bodies.
3334
3335The durable, poll-based event log remains the fallback for missed pushes and for deployments with more than one worker process.
3336
3337## Related reading
3338
3339- **ASGI and realtime direction** — why the server uses plain ASGI and how websocket/SSE object events fit that direction.
3340- **Generative UI** — the surfaces that subscribe to the change log.
3341- **Permissions model** — the READ decision reused on subscribe and on every push.
3342- **Runtime contract** — the daemon and event contracts behind the durable log.
3343"
3344asgi-realtime-direction ASGI and Realtime Direction Sites, UI & Realtime 450 Why DBBASIC's ASGI layer stays close to persistent CGI, what it borrows from Phoenix/Erlang, and how the websocket realtime overlay sits on top of the durable event log. "The working prototype uses plain ASGI with Uvicorn, not Django ASGI, Channels, FastAPI, or Starlette.
3345
3346The shape is simple:
3347
3348```mermaid
3349flowchart LR
3350 A[""Uvicorn""] --> B[""ASGI app""]
3351 B --> C[""Filesystem router""]
3352 C --> D[""api/*.py handlers""]
3353```
3354
3355That keeps the useful part of classic CGI:
3356
3357```mermaid
3358flowchart LR
3359 A[""URL""] --> B[""File""]
3360 B --> C[""Function""]
3361```
3362
3363but without classic CGI's process-per-request cost.
3364
3365That is a better comparison than saying DBBASIC is a Rails-like app structure. The goal is closer to persistent CGI with a reusable object runtime:
3366
3367- filesystem paths can map to runnable code
3368- the server stays warm between requests
3369- routing is derived from object source placement
3370- common helpers provide state, logs, files, versions, and scheduling
3371- object boundaries stay small enough for humans and AI tools to inspect
3372
3373Classic CGI made deployment easy but left too much work to each script. DBBASIC keeps the direct path-to-code idea and adds the shared runtime pieces that make objects reusable as application building blocks.
3374
3375## Why ASGI Matters
3376
3377ASGI gives DBBASIC one server surface for:
3378
3379- normal HTTP requests
3380- long-lived WebSocket connections
3381- Server-Sent Events
3382- startup and shutdown lifecycle hooks
3383
3384That matters because DBBASIC objects are not only pages or REST handlers. They can also be jobs, event handlers, admin actions, realtime screens, data feeds, and AI-edited live application pieces.
3385
3386The important loop is:
3387
3388```mermaid
3389flowchart LR
3390 A[""Object changes""] --> B[""Version saved""]
3391 B --> C[""Object event""]
3392 C --> D[""UI/client updates""]
3393```
3394
3395HTTP can handle the command. WebSockets or SSE can carry the feedback.
3396
3397## Phoenix and Erlang
3398
3399Phoenix and Erlang/BEAM show why realtime application systems can feel so much better than request/response-only web frameworks:
3400
3401- long-lived connections are normal
3402- PubSub is a core idea
3403- server-side events can update clients quickly
3404- stateful processes can model active parts of the application
3405- failures are isolated and supervised
3406
3407DBBASIC should learn from that direction.
3408
3409The DBBASIC version is not Erlang and does not get BEAM supervision for free. Python ASGI does not automatically give per-object process isolation, hot code replacement, distributed process registries, or Phoenix PubSub semantics.
3410
3411What DBBASIC can still use well is the product shape:
3412
3413```mermaid
3414flowchart TD
3415 A[""Object source""]
3416 B[""Object state""]
3417 C[""Object logs""]
3418 D[""Object versions""]
3419 E[""Realtime object events""]
3420 A --> E
3421 B --> E
3422 C --> E
3423 D --> E
3424```
3425
3426That can support live admin screens, Scroll object inspectors, AI repair loops, dashboards, collaborative editing, queues, and business app interfaces without forcing everything through a Rails/Django-style request/deploy cycle.
3427
3428## Public v1 Direction
3429
3430The public v1 server stays single-node first and keeps clustering separate until replication and conflict behavior are proven.
3431
3432The first public ASGI layer is small and runnable. It includes:
3433
3434- `GET /health`
3435- `GET /objects?format=json`
3436- `GET /objects/{object_id}?source=true&format=json`
3437- object execution
3438- source update and rollback
3439- state/log/version endpoints
3440- metadata endpoints
3441- temporary admin-token gate for object listing and introspection reads
3442
3443Role, object, and row-level permissions, resource limits, and backup/restore checks have all shipped. The websocket event stream is now implemented too (see below).
3444
3445## Realtime: Implemented
3446
3447Realtime arrived as an event system, not a decorative feature, and it reuses the durable event log rather than inventing a parallel path:
3448
3449- **Durable log (source of truth).** Every record write already publishes `collection.record.created/updated/deleted` to `object_events`, which is persisted, capped, and pollable with a cursor. This is the reliable, cross-restart, cross-worker path.
3450- **Live overlay (`/ws`).** An in-process pub/sub hub (`object_realtime`) pushes those same events to connected websockets the instant they happen. See the [HTTP API contract](/docs/http-api-contract#realtime-push-websocket) for the connect + subscribe protocol.
3451
3452Events are boring JSON signals — collection, record id, action — never the record body, so a push can never leak a field. Delivery obeys the permission engine: you can only subscribe to a collection you may read, and with enforcement on you only receive events for records your row filters would show you. The client refetches through the normal enforced API.
3453
3454**Single-process assumption.** The live hub lives in one process; with one Uvicorn worker every write and every socket share the event loop, so push is immediate and consistent. Across multiple workers each has its own hub, and clients fall back to polling the durable log — which is why the durable log, not the socket, is the source of truth. Keep the public v1 single-worker (or add a shared broker before scaling workers) so live push stays whole; either way correctness never depends on the socket.
3455
3456An object-scoped room model (`/ws/objects/{object_id}` for source/state/log changes) is a natural next addition on the same hub, using the existing source/state/version change events.
3457
3458## Design Rule
3459
3460Keep the ASGI layer thin.
3461
3462The server should route requests and streams to object operations. It should not become a second framework with its own competing object model.
3463</content>
3464</StructuredOutput>
3465
3466"
3467shell-and-ai The Shell and AI — Talk to the Whole System AI 500 How the DBBASIC shell routes instant record commands, search, and AI chat through the same MCP tools and permission engine, plus voice and web-reading features. "The shell (`/shell`) is one input that reaches everything: instant
3468record commands, global search, and an AI that operates the server
3469through the same MCP tools any agent uses — with **your key, your
3470model choice, and your permissions**.
3471
3472```mermaid
3473flowchart TD
3474 U[""You type""] --> D{""prefix?""}
3475 D -->|""$ . ^""| REC[""Create note / task / link<br/>(records API, your session)""]
3476 D -->|""~""| SRCH[""GET /api/search""]
3477 D -->|""/""| CMD[""Built-ins: /help /key /model /tools /voice""]
3478 D -->|""anything else""| CHAT[""POST /api/ai/chat""]
3479 CHAT --> PROV[""AI provider<br/>(your stored key)""]
3480 PROV -->|""tool calls""| MCP[""MCP tool subset<br/>dispatched with YOUR credentials""]
3481 MCP --> PERM[""Permission engine + audit""]
3482 PERM --> PROV
3483 PROV --> REPLY[""Reply + tool log + token usage + cost""]
3484```
3485
3486## Setup: Your Key, Your Model
3487
3488AI features use each user's own provider key — the server stores it
3489write-only and calls providers on your behalf. Key material never
3490appears in any response, record, or backup. Two providers are supported
3491today (`object_ai.SUPPORTED_SERVICES`): `anthropic` and `openai` — no
3492Gemini, no local/on-device inference. An `openai` key, once stored, also
3493covers the cloud voice paths below (`POST /api/tts` with `engine: ""openai""`,
3494`POST /api/stt`) — one key, no separate signup for voice.
3495
3496In the shell:
3497
3498```
3499/key anthropic sk-ant-... store a key (masked, never logged)
3500/keys which services have keys
3501/model anthropic:claude-haiku-4-5 pick a model (service:model)
3502/model openai:gpt-5-mini or the other provider
3503```
3504
3505Or over HTTP: `PUT /identity/users/{you}/service-keys` with your
3506session (see the [HTTP API contract](/docs/http-api-contract)).
3507
3508## Tool Subsets: Small Context, Configurable Power
3509
3510The AI is offered a **named subset** of the MCP tool catalog, not the
3511whole thing. Small subsets keep the context small enough for fast,
3512inexpensive models — and the subset is your safety dial:
3513
3514```
3515/tools global_search,list_records,get_record,create_record conversation mode
3516/tools global_search,list_objects,get_object_source,create_object,update_object_source,execute_object builder mode
3517```
3518
3519Every tool call the model makes is dispatched through the server's own
3520routing **with your credentials**: the AI can do exactly what you could
3521do yourself — row filters, field redaction, and the audit trail apply.
3522An AI acting for a user is never more powerful than the user.
3523
3524## Conversation Memory
3525
3526The shell logs every exchange to the `shell_commands` collection (your
3527history is just records — searchable and owner-scoped). A new browser
3528session replays recent history and sends prior AI turns back with each
3529message, so conversations resume. The server stays stateless about
3530chats: `POST /api/ai/chat` accepts a `history` list, and what to
3531remember is the client's choice.
3532
3533## Cost Recording
3534
3535Every chat turn is priced server-side, not by the caller. `POST
3536/api/ai/chat` reads token counts straight from the provider's own response,
3537looks up the model's price in the `ai_prices` collection (editable records,
3538never a hardcoded table -- so a price change is a data edit, not a
3539deploy), and computes the cost in integer cents: `tokens *
3540per_million_cents // 1_000_000`, input and output added separately, no
3541float division on money. The result -- `tokens_in`, `tokens_out`,
3542`cost_cents`, `model`, `provider` -- lands in the `ai_usage` collection,
3543written by the chat handler itself with the caller's user id as actor, so
3544the record can't be skipped or forged the way a client-written log could
3545be. A model with no matching price row still gets its tokens recorded;
3546only the cost is null. The same numbers come back in the chat response's
3547`usage` object so a surface can show them immediately.
3548
3549## Coding Without Coding
3550
3551With builder-mode tools, the AI can create and edit live objects:
3552
3553> make an object called site_dice that renders a page rolling two dice
3554
3555The model writes the source, calls `create_object`, and the page is
3556live at `/dice` immediately — objects load per execution, so there is
3557no deploy step between the AI writing code and the code serving
3558traffic. Edits use `update_object_source`; every version lands in
3559source history with rollback; and create/update responses report the
3560methods the code actually exposes, so the model can self-correct.
3561
3562Object writes ride the admin gate (an admin-role session today), and
3563source writes require `DBBASIC_ENABLE_SOURCE_WRITES=true` — the same
3564boundaries that govern humans.
3565
3566## Voice
3567
3568The shell doubles as a push-to-talk terminal. Speech in and speech out each
3569have two paths: browser-native (free, on-device, always available if the
3570browser supports it) and cloud (a real per-user OpenAI key, real cost, but
3571""not on device"" — works on browsers/hardware the native APIs don't cover,
3572and sounds noticeably better than the local synthesis engines).
3573
3574- **Speech in, browser path**: the mic button uses `window.SpeechRecognition`
3575 (or the `webkit` prefix); interim words show live in the input, and the
3576 final transcript submits through the same form-submit path a typed line
3577 does — no separate send code. Neither Chrome nor Safari expose the API
3578 outside a secure context (`https://` or `localhost`), and without it the
3579 button simply stays hidden. This is the default (`talk_stt_engine: ""browser""`).
3580- **Speech in, cloud path**: when `talk_stt_engine` is `""openai""`, the mic
3581 becomes push-to-talk instead of continuous listening — tap to start a
3582 real `MediaRecorder` capture, tap again to stop and upload the clip to
3583 `POST /api/stt`, which transcribes it via the caller's own stored OpenAI
3584 key and returns text. No wake word, no VAD silence endpoint: raw audio has
3585 no equivalent to `SpeechRecognition`'s live partial-result signal, so
3586 there is no live signal to drive one. See `POST /api/stt` below.
3587- **Speech out, browser path**: `speechSynthesis.speak` — free, on-device,
3588 no server round-trip.
3589- **Speech out, cloud path**: when `talk_tts_engine` is `""openai""`, assistant
3590 replies (markdown and code fences stripped first — only prose is spoken,
3591 never tool-call noise) POST to `POST /api/tts` with `""engine"": ""openai""`
3592 and play the returned `audio/mpeg`, via the caller's own stored OpenAI key.
3593 If the request fails for any reason, the browser's own `speechSynthesis`
3594 says the line instead, so voice mode never goes silent because of a
3595 server-side gap.
3596
3597`/voice`, `/voice on`, `/voice off` toggle voice mode inline (mirrors
3598`/model`); the setting persists on `shell_preferences.voice_enabled` and is
3599honored on page load like `ai_model` and `tools`. `talk_tts_engine` and
3600`talk_stt_engine` (both default to the free/local path) live on the same
3601`shell_preferences` record — the Talk page (`/talk`) additionally has a
3602single in-page toggle that flips both together, since ""cloud voice"" is one
3603combined mode to remember, not two settings.
3604
3605### `POST /api/tts`
3606
3607```http
3608POST /api/tts
3609Authorization: Bearer <session-token> (or the session cookie)
3610
3611{""text"": ""one note matches"", ""voice"": ""en-us"", ""engine"": ""local""}
3612```
3613
3614Response: audio bytes (`audio/wav` for `engine: ""local""`, `audio/mpeg` for
3615`engine: ""openai""`), or the usual `{""status"": ""error"", ""error""}` JSON shape
3616on failure. Requires `DBBASIC_ENABLE_TTS=true` (off by default) and a
3617signed-in session — same posture as `/api/ai/chat`. `text` is capped at 800
3618characters (413 beyond that). `engine` is optional, `""local""` or `""openai""`,
3619defaulting to `""local""` — the two paths never silently fall back to each
3620other; a caller who asked for cloud quality and hits a provider error sees
3621that error.
3622
3623**`engine: ""local""`** (default): discovered at call time, first match wins:
3624`espeak-ng`, `espeak`, then macOS `say` (development convenience — `say`
3625writes AIFF and the also-stock `afconvert` turns it into WAV; if `afconvert`
3626isn't present the endpoint returns 501 rather than growing a bespoke audio
3627pipeline). No engine on `PATH` at all is a 503 with a clear message.
3628
3629**`engine: ""openai""`**: calls OpenAI's TTS API (`gpt-4o-mini-tts` by
3630default) using the caller's own key, stored via `PUT
3631/identity/users/{user_id}/service-keys` (`service: ""openai""`) — no key
3632stored is a 400 with the exact PUT to fix it. A provider-side failure (bad
3633key, rate limit) is a 502. `DBBASIC_TTS_CLOUD_TIMEOUT_SECONDS` bounds the
3634outbound call (default 30s).
3635
3636Successful audio is cached at
3637`data/tts-cache/{sha256(engine|voice|text[|model])}.wav` regardless of
3638engine — a repeat of the same line is a cache read, not a re-synthesis.
3639There's no eviction in v1; operators who want a bound should prune that
3640directory on their own schedule.
3641
3642### `POST /api/stt`
3643
3644```http
3645POST /api/stt
3646Authorization: Bearer <session-token> (or the session cookie)
3647Content-Type: audio/webm
3648
3649<raw audio bytes>
3650```
3651
3652Response: `{""status"": ""ok"", ""text"": ""...""}`, or the usual
3653`{""status"": ""error"", ""error""}` shape. The request body is the raw audio
3654bytes themselves (whatever `MediaRecorder` produced client-side — typically
3655`audio/webm` on Chrome/Firefox, `audio/mp4`-ish on Safari), not JSON; the
3656`Content-Type` header says what format it is. Requires
3657`DBBASIC_ENABLE_STT=true` (off by default, unlike TTS which many operators
3658turn on immediately — STT costs money on every utterance, not just the
3659occasional reply) and a signed-in session. Capped at 10MB per clip (413
3660beyond that).
3661
3662Unlike TTS, there is **no local-engine path here at all** — every call goes
3663to OpenAI's transcription API (`gpt-4o-mini-transcribe` by default), using
3664the caller's own stored key exactly like the TTS cloud path. No key stored
3665is a 400; a provider-side failure is a 502.
3666`DBBASIC_STT_TIMEOUT_SECONDS` bounds the outbound call (default 30s).
3667Nothing is persisted server-side beyond the ordinary `ai_usage`/cost
3668accounting every provider call gets — the audio itself is uploaded,
3669transcribed, and discarded within the request.
3670
3671## Reading the Web
3672
3673The AI can fetch and read a real web page — the lynx/w3m move for an AI +
3674voice consumer: strip a URL down to `{title, text, links}`, links numbered
3675in document order so they double as speakable navigation targets (""open
3676link three""). It is a real, tested loop: read a page, get a short spoken
3677summary with numbered links, say ""open link one,"" and it resolves against
3678the previous turn's link list, fetches that page, and summarizes it too —
3679follow a chain of pages by voice alone.
3680
3681The `read_page` tool is offered like any other MCP tool (add it to a
3682`/tools` subset to use it) and calls `object_reader.py` server-side, gated
3683behind `DBBASIC_ENABLE_READER` (off by default). Fetching is SSRF-hardened:
3684every hostname — the original and every redirect hop — is resolved and
3685checked before the socket opens, refusing loopback/link-local/private
3686ranges and failing closed on any resolution ambiguity; `http`/`https` only,
3687a small redirect cap, and hard timeout/size limits. HTML is stripped with
3688`html.parser` (no regex scraping), never a headless browser — JS-rendered
3689pages, images, and login-walled pages are out of scope by design.
3690
3691A fetched page can also be shown on screen as a `{""kind"": ""reader"", ""url"":
3692...}` view block (title + body text + the numbered link list), fetched
3693client-side through `POST /api/read` — the same flag and SSRF posture
3694apply.
3695
3696## The Instant Commands
3697
3698| Input | Effect |
3699|---|---|
3700| `$ pay the hosting bill` | quick note |
3701| `. fix the header` | quick task |
3702| `^https://example.com docs` | save a link |
3703| `~flywheel` | global search across collections |
3704| `/help` | list commands |
3705
3706These never touch the AI — they are one permission-checked record
3707write each, which is why they feel instant.
3708
3709## For Agents
3710
3711Everything above is equally available to AI agents connecting over MCP
3712(`POST /api/mcp`) with their own identities and labeled sessions, and
3713to headless callers hitting `POST /api/ai/chat` directly. One surface,
3714many kinds of operator.
3715"
3716permissions-model Permissions Model Security & Limits 600 A reference guide to DBBASIC's server-side permissions primitive: access modes, rules, row/field filters, sharing, subscriptions, and route enforcement. "DBBASIC permissions are a server-side primitive. A visual editor like Scroll can edit, preview, and explain rules, but the object server must make the final allow/deny decision.
3717
3718Many web frameworks treat permissions as something each app has to rediscover: blog visibility, admin-only pages, intranet roles, or social-app follows. Those cases matter, but they are not enough for a general business platform.
3719
3720DBBASIC needs object-level permissions that can model SaaS, customer portals, employee roles, contractors, paid content, shared workspaces, internal admin tools, and AI-generated app surfaces with the same small set of primitives. Getting this right means most apps need less custom permission code.
3721
3722The model needs to cover the common app cases without forcing every object into one framework shape:
3723
3724- public visitors
3725- signed-in users
3726- paid subscribers
3727- temporary paid access
3728- admins and employees
3729- customer accounts with their own employees
3730- object owners
3731- users or accounts an owner shared with
3732- per-row and per-field restrictions
3733
3734## Decision shape
3735
3736The core API is:
3737
3738```python
3739check_permission(subject, action, policy, collection=None, object_id=None, record=None)
3740```
3741
3742It returns:
3743
3744```python
3745PermissionDecision(
3746 allowed=True/False,
3747 reason=""..."",
3748 code=""allowed|authentication_required|payment_required|forbidden"",
3749 http_status=200|401|402|403,
3750 row_filter={...},
3751)
3752```
3753
3754When a list/query request has a row filter but no specific record, the decision can return `allowed=True` with the row filter attached. The caller then applies that filter before returning rows.
3755
3756```mermaid
3757flowchart TD
3758 A[""request""] --> B[""identify subject""]
3759 B --> C[""load policy""]
3760 C --> D{""admin role?""}
3761 D -- yes --> Z[""allow""]
3762 D -- no --> E{""explicit deny matches?""}
3763 E -- yes --> Y[""deny with reason""]
3764 E -- no --> F{""access mode allows?""}
3765 F -- yes --> Z
3766 F -- no --> G{""allow rule matches?""}
3767 G -- yes --> H[""allow with optional row/field filters""]
3768 G -- no --> I{""owner fallback?""}
3769 I -- yes --> Z
3770 I -- no --> Y
3771```
3772
3773## Persistence and API
3774
3775The current public server stores the active policy as:
3776
3777```text
3778data/permissions/policy.json
3779```
3780
3781If the file does not exist, the server uses a conservative default:
3782
3783```json
3784{
3785 ""access_mode"": ""role_based"",
3786 ""roles"": {},
3787 ""user_roles"": {},
3788 ""rules"": [],
3789 ""admin_roles"": [""admin"", ""superuser""]
3790}
3791```
3792
3793That default has no allow grants, so enforcement readiness will block until a real policy is written. `object_permission_store.starter_policy_payload()` returns the documented starter policy for a fresh deployment: anonymous visitors can execute the public pages (`site_home`, `system_dashboard`, `system_write_probe`) and read the `dbbasic_probe` demo records; signed-in users can read and execute objects and write probe records; admin-role subjects bypass rules. Install it with the policy endpoint:
3794
3795```bash
3796python3 -c ""import json, object_permission_store as s; print(json.dumps({'policy': s.starter_policy_payload()}))"" \
3797 | curl -s -X PUT https://your-host/permissions/policy \
3798 -H ""Authorization: Token <admin-token>"" \
3799 -H ""Content-Type: application/json"" --data-binary @-
3800```
3801
3802Then edit grants per app instead of widening the starter rules in place. Run audit mode before enforcement to see what live traffic would be denied.
3803
3804The policy endpoints are still admin-gated while broader login, account management, and permission editing mature:
3805
3806```http
3807GET /permissions/policy
3808Authorization: Token <token>
3809```
3810
3811```http
3812PUT /permissions/policy
3813Authorization: Token <token>
3814Content-Type: application/json
3815```
3816
3817```json
3818{
3819 ""policy"": {
3820 ""access_mode"": ""role_based"",
3821 ""roles"": {""sales"": {""label"": ""Sales""}},
3822 ""user_roles"": {""7"": [""sales""]},
3823 ""rules"": []
3824 }
3825}
3826```
3827
3828Identity subjects can come from trusted headers, the admin token, or DBBASIC session tokens. The session tokens are backed by a small local registry:
3829
3830```text
3831data/identity/accounts.tsv
3832data/identity/users.tsv
3833data/identity/sessions.tsv
3834```
3835
3836Accounts carry tenant/subscription defaults. Users carry account membership, roles, and user-level subscriptions. A session minted for a registered user can inherit that account, role, and subscription data, giving permission checks the same subject shape whether the request came from a visual editor, a gateway, or a future DBBASIC login object.
3837
3838New DBBASIC-created accounts, users, sessions, shared links, record IDs, and change/audit IDs should use UUIDv4 values. Imported systems can keep their old integer or slug IDs as compatibility columns, but new routes and generated UI should not depend on sequential IDs. This keeps SaaS-style tenant data, temporary links, subscriptions, shared records, and future package exports from being tied to one database sequence. The identity registry still accepts string IDs so relational-database data and older prototypes can migrate without a flag day.
3839
3840Editors and tests can preview decisions without persisting draft rules:
3841
3842```http
3843POST /permissions/check
3844Authorization: Token <token>
3845Content-Type: application/json
3846```
3847
3848```json
3849{
3850 ""policy"": {
3851 ""access_mode"": ""role_based"",
3852 ""rules"": [
3853 {
3854 ""effect"": ""allow"",
3855 ""principal"": ""role:sales"",
3856 ""actions"": [""read""],
3857 ""collection"": ""contacts"",
3858 ""row_filter"": {""owner_id"": ""$user_id""}
3859 }
3860 ]
3861 },
3862 ""subject"": {""user_id"": ""7"", ""roles"": [""sales""]},
3863 ""action"": ""read"",
3864 ""collection"": ""contacts""
3865}
3866```
3867
3868If `policy` is omitted, `/permissions/check` uses the persisted policy file. The optional `now` field accepts an ISO timestamp for testing temporary access windows.
3869
3870## Route enforcement
3871
3872The public server can enforce the persisted policy against object routes and collection-record routes, but it is intentionally opt-in while auth and the editing tools mature.
3873
3874```text
3875DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true
3876```
3877
3878That flag requests enforcement. The server only makes it effective when the readiness checks from `/permissions/status` pass. If recovery, identity, or policy readiness is incomplete, the request stays in audit/shadow mode and responses continue to follow the pre-enforcement route behavior. Current blockers include:
3879
3880- no admin recovery token
3881- unreadable identity store
3882- invalid permission policy
3883- `password` mode before a password verifier exists
3884- `role_based` mode with no allow grants
3885- private, registered, subscription, or role-based modes with no non-admin identity path
3886
3887Operators can see this through:
3888
3889```json
3890{
3891 ""permissions"": {
3892 ""enforcement_requested"": true,
3893 ""enforcement_enabled"": false,
3894 ""enforcement_blocked"": true
3895 }
3896}
3897```
3898
3899The explicit recovery/test override is:
3900
3901```text
3902DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true
3903```
3904
3905Do not use that override as a normal production setting.
3906
3907When enforcement is effective, protected routes call the same decision engine before serving or executing data:
3908
3909```mermaid
3910flowchart LR
3911 A[""HTTP request""] --> B[""map subject""]
3912 B --> C[""map route to action""]
3913 C --> D[""load data/permissions/policy.json""]
3914 D --> E[""check_permission""]
3915 E --> F{""allowed?""}
3916 F -- yes --> G[""serve route""]
3917 F -- no --> H[""return 401 / 402 / 403""]
3918 E --> I[""append data/permissions/audit.jsonl""]
3919```
3920
3921Route actions currently map like this:
3922
3923- `GET /objects/{id}` -> `execute`
3924- `POST /objects/{id}` -> `execute`
3925- `PUT /objects/{id}` -> `update`
3926- `DELETE /objects/{id}` -> `delete`
3927- `?source=true` -> `source`
3928- `?state=true` -> `state`
3929- `?logs=true` -> `logs`
3930- `?versions=true` or `?version=N` -> `versions`
3931- `?metadata=true` -> `read`
3932- `GET /collections/{collection}/records` -> `read`
3933- `POST /collections/{collection}/records` -> `create`
3934- `GET /collections/{collection}/records/{record_id}` -> `read`
3935- `PUT /collections/{collection}/records/{record_id}` -> `update`
3936- `DELETE /collections/{collection}/records/{record_id}` -> `delete`
3937
3938Collection record lists apply row filters before pagination. Record detail checks include the selected record, so owner, account, subscription, temporary access, row-filter, and field-redaction rules can all be enforced by the server. Record writes check the affected record. Updates also check the candidate row after applying the submitted changes, so a user cannot pass a row filter on the old row and then move the row into another owner/account by changing fields.
3939
3940By default the collection record routes are admin-token gated. Read routes can use audit or enforcement mode with the active policy. Write routes require the admin token unless permission enforcement is enabled and the active policy allows the matching `create`, `update`, or `delete` action. Audit-only mode logs write decisions, but it does not authorize mutations by itself.
3941
3942Audit-only mode logs decisions without blocking the request:
3943
3944```text
3945DBBASIC_ENABLE_PERMISSION_AUDIT=true
3946```
3947
3948That is useful before turning enforcement on for a live app. Enforcement also writes audit entries.
3949
3950Operators and editing tools can read recent audit entries through the admin-gated audit endpoint:
3951
3952```http
3953GET /permissions/audit?limit=100&object_id=site_home
3954Authorization: Token <token>
3955```
3956
3957Supported filters are `action`, `object_id`, `collection`, `allowed`, and `enforced`.
3958
3959By default, route checks only trust the admin token and anonymous public traffic. If a reverse proxy or auth gateway has already authenticated the request, trusted identity headers can be enabled explicitly:
3960
3961```text
3962DBBASIC_PERMISSION_TRUST_HEADERS=true
3963```
3964
3965Supported headers:
3966
3967- `X-DBBASIC-User-Id`
3968- `X-DBBASIC-Account-Id`
3969- `X-DBBASIC-Roles`
3970- `X-DBBASIC-Subscriptions`
3971
3972Only enable trusted headers behind infrastructure that strips or overwrites client-supplied copies.
3973
3974The server can also mint DBBASIC identity sessions:
3975
3976```http
3977POST /identity/sessions
3978Authorization: Token <admin-token>
3979```
3980
3981The admin route can mint arbitrary operator-controlled sessions. For a login gateway or local trusted client, `POST /identity/session` can mint a session for an existing active user when `DBBASIC_ENABLE_SESSION_LOGIN=true` and the caller presents `DBBASIC_SESSION_LOGIN_TOKEN`. That token is separate from the admin token. The route accepts only `user_id`, `label`, and `ttl_seconds`; role, account, and subscription overrides are rejected so the subject comes from the registered user and account.
3982
3983Sessions store only a token hash under `data/identity/sessions.tsv`. Requests can then send `Authorization: Token <session-token>` or `Authorization: Bearer <session-token>`, and the server resolves the subject directly from its local identity store. This is the first built-in replacement for each app hand-rolling the same ""current user, current account, current roles, current subscription"" lookup.
3984
3985Clients can inspect or revoke their own active session without the admin token:
3986
3987```http
3988GET /identity/session
3989DELETE /identity/session
3990Authorization: Token <session-token>
3991```
3992
3993Those routes require an active session token and do not treat the admin token as a user session.
3994
3995The active subject can be inspected through:
3996
3997```http
3998GET /identity
3999Authorization: Token <token>
4000```
4001
4002Admin-token-gated operator routes still require `DBBASIC_ADMIN_TOKEN` by default. A deployment can opt in to `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`; when enabled, an active DBBASIC session whose subject has one of the policy admin roles can pass the same gate. This is intended for operator clients that should stop carrying the raw deployment token after login exists.
4003
4004That endpoint returns the normalized `user_id`, `account_id`, roles, subscriptions, auth method, and permission-mode flags the server will use for route checks.
4005
4006Operators and editing tools can inspect permission rollout readiness through:
4007
4008```http
4009GET /permissions/status
4010Authorization: Token <admin-token>
4011```
4012
4013That endpoint summarizes the active mode flags, identity counts, persisted policy shape, covered route groups, blockers, and warnings. It is meant to be used before enabling enforcement on a live app. For identity-gated modes, one of these non-admin identity paths must exist before enforcement becomes effective: trusted proxy headers, guarded session login for existing users, or an active DBBASIC session.
4014
4015## Access modes
4016
4017Access modes answer who gets through the front door.
4018
4019- `public` - anyone can read or execute public objects.
4020- `password` - a shared gate authenticates before policy checks.
4021- `registered` - any signed-in user can read or execute.
4022- `subscription` - signed-in users with an active subscription can read or execute.
4023- `role_based` - rules decide by role, user, account, owner, row, field, and action.
4024- `private` - owner-only fallback unless explicit rules grant access.
4025
4026## Rules
4027
4028Rules answer what the subject can do once they are known.
4029
4030Actions should stay small and explicit:
4031
4032- `create`
4033- `read`
4034- `update`
4035- `delete`
4036- `execute`
4037- `source`
4038- `state`
4039- `logs`
4040- `files`
4041- `versions`
4042- `share`
4043- `admin`
4044
4045Principal strings are portable on purpose:
4046
4047- `public`
4048- `registered`
4049- `owner`
4050- `role:admin`
4051- `role:sales`
4052- `user:42`
4053- `account:customer-acme`
4054- `subscription:pro`
4055
4056Explicit denies are checked before allow rules.
4057
4058## Ownership
4059
4060The old object convention still matters:
4061
4062```text
4063u_42_report
4064```
4065
4066That object belongs to user `42`. Owners can read, execute, update, inspect, share, and delete their own user objects unless a future stricter policy blocks that behavior.
4067
4068Records can also carry ownership fields:
4069
4070```json
4071{
4072 ""owner_id"": ""42"",
4073 ""customer_account_id"": ""customer-acme""
4074}
4075```
4076
4077The server can use `owner_id`, `user_id`, or `created_by` as owner fields for the basic fallback. Application objects can add richer policy around account, department, project, team, or customer ownership.
4078
4079## Sharing
4080
4081Sharing is just a rule:
4082
4083```python
4084PermissionRule.allow(
4085 ""user:99"",
4086 [""read"", ""update""],
4087 object_id=""u_42_shared_report"",
4088 reason=""shared by owner"",
4089)
4090```
4091
4092Account sharing uses the same shape:
4093
4094```python
4095PermissionRule.allow(
4096 ""account:customer-acme"",
4097 [""read""],
4098 collection=""invoices"",
4099 row_filter={""customer_account_id"": ""$account_id""},
4100)
4101```
4102
4103That covers the common case where a customer has multiple employees and they all need access to the customer portal, invoices, files, or support tickets.
4104
4105Project sharing — ""records I own OR records in a project shared with me"" — uses grant records plus one filter value. Grants are rows in the plain `project_access` collection (`project_id`, `user_id`), so sharing is data: browseable, audited, versioned. Before checks run, the server resolves the subject's grants; rules opt in with `$accessible_projects`, which matches when the record's field value is any granted project id:
4106
4107```python
4108PermissionRule.allow(
4109 ""registered"",
4110 [""read""],
4111 collection=""notes"",
4112 row_filter={""project_id"": ""$accessible_projects""},
4113 reason=""notes in shared projects are readable"",
4114)
4115```
4116
4117Pair it with an owner rule on the same collection: matching stays first-rule-wins at the collection level, but list, single-record, and search reads evaluate rules per record, so ""own"" and ""shared"" rules combine naturally. The engine itself stays pure — grant resolution is IO the server does once per request, and the subject carries the result.
4118
4119Granting is self-serve through the companion value `$owned_projects`, which resolves to the projects whose `owner_id` is the subject. ""Only a project's owner may share it"" — a cross-record condition — becomes a plain row filter on the grant collection itself:
4120
4121```python
4122PermissionRule.allow(
4123 ""registered"",
4124 [""create"", ""read"", ""delete""],
4125 collection=""project_access"",
4126 row_filter={""project_id"": ""$owned_projects""},
4127 reason=""project owners share and revoke access to their own projects"",
4128)
4129```
4130
4131Creating a grant is creating a record; revoking is deleting it; both are audited and appear in the change history like every other write.
4132
4133Grants carry a `permission` of `read` or `write`. Most rules only need membership (`$accessible_projects`), but some need to tell read grants apart from write grants — ""only project writers may publish"" — and that narrower set is `$writable_projects`, resolved from the same `project_access` rows filtered to `permission == ""write""`:
4134
4135```python
4136PermissionRule.allow(
4137 ""registered"",
4138 [""update""],
4139 collection=""docs"",
4140 row_filter={""project_id"": ""$writable_projects""},
4141 reason=""only writers may edit docs in a shared project"",
4142)
4143```
4144
4145It resolves the same way and is available everywhere `$accessible_projects` and `$owned_projects` are: row filters and, for schema-declared lifecycle fields, transition guards (see the `transitions` field key in the schema-forms guide).
4146
4147## Row and field rules
4148
4149Row filters model rules like:
4150
4151```text
4152sales reps only see their own leads
4153```
4154
4155As data:
4156
4157```python
4158PermissionRule.allow(
4159 ""role:sales"",
4160 [""read""],
4161 collection=""contacts"",
4162 row_filter={""owner_id"": ""$user_id""},
4163)
4164```
4165
4166Field rules model cases like hiding salary, internal notes, cost, or private metadata:
4167
4168```python
4169PermissionRule.allow(
4170 ""account:customer-acme"",
4171 [""read""],
4172 collection=""invoices"",
4173 fields=[""invoice_id"", ""status"", ""total""],
4174 denied_fields=[""internal_notes""],
4175)
4176```
4177
4178Schema field permissions add the generated-form version of the same idea. They live next to field metadata so a visual editor can render the right controls and the server can enforce the same decision:
4179
4180```json
4181{
4182 ""name"": ""margin"",
4183 ""type"": ""currency"",
4184 ""permissions"": {
4185 ""admin"": ""edit"",
4186 ""sales"": ""hidden"",
4187 ""viewer"": ""hidden""
4188 }
4189}
4190```
4191
4192The access levels are:
4193
4194- `edit` - visible and writable
4195- `read` - visible but not writable
4196- `hidden` - removed from reads and not writable
4197
4198Principal keys can be role names such as `sales`, or full principals such as `role:sales`, `user:42`, `account:customer-acme`, `subscription:pro`, `registered`, `public`, or `owner`. Grouped access lists are also accepted:
4199
4200```json
4201{
4202 ""name"": ""notes"",
4203 ""permissions"": {
4204 ""edit"": [""owner"", ""role:admin""],
4205 ""read"": [""role:support""],
4206 ""default"": ""hidden""
4207 }
4208}
4209```
4210
4211When multiple principals match one subject, the most restrictive field access wins. Schema field permissions only refine fields after the broader policy has already allowed access to the row.
4212
4213## Subscription and temporary access
4214
4215Subscriptions can be broad:
4216
4217```python
4218PermissionPolicy(access_mode=""subscription"")
4219```
4220
4221Or plan-specific:
4222
4223```python
4224PermissionRule.allow(
4225 ""subscription:pro"",
4226 [""read""],
4227 collection=""premium_reports"",
4228)
4229```
4230
4231Temporary pay-per-view style access is a time-boxed grant:
4232
4233```python
4234PermissionRule.allow(
4235 ""user:42"",
4236 [""read""],
4237 object_id=""reports_market_snapshot"",
4238 valid_from=""2026-06-01T00:00:00Z"",
4239 expires_at=""2026-07-01T00:00:00Z"",
4240)
4241```
4242
4243If access should be consumed after one or more views, that should be tracked by an entitlement or ledger object. The permission rule says access is possible; the ledger object records purchases, consumption, refunds, and audit history.
4244
4245For HTTP APIs, missing or expired paid entitlement should map to:
4246
4247```text
4248402 Payment Required
4249```
4250
4251That is different from:
4252
4253- `401 Unauthorized` - the server does not know who the user is yet.
4254- `403 Forbidden` - the server knows the user, but the user is not allowed.
4255- `402 Payment Required` - the user may be allowed after subscription, purchase, credit, or another payment entitlement.
4256
4257## Editor contract
4258
4259A visual editing tool should eventually call a server endpoint that exposes the same decision shape:
4260
4261```json
4262{
4263 ""allowed"": true,
4264 ""reason"": ""sales reps only see own contacts"",
4265 ""code"": ""allowed"",
4266 ""http_status"": 200,
4267 ""row_filter"": {""owner_id"": ""$user_id""},
4268 ""fields"": null,
4269 ""denied_fields"": []
4270}
4271```
4272
4273That lets an editor show permission previews, generated matrices, ""test as role"", and AI-generated rules without becoming the authority.
4274"
4275secrets-and-credentials Secrets, Credentials, and the Trust Boundary Security & Limits 610 Where secrets live in DBBASIC, why a secret is never a record field, and the honest threat model behind the write-only identity vault. "This is the doctrine for **where secrets live** in DBBASIC and why. It matters
4276to anyone writing a package that touches a credential — a user's third-party
4277API key, an outbound-mail relay password, an email-hosting mailbox password, a
4278signing token. Get this wrong and a secret ends up in a record collection,
4279which means it ends up in an export, a backup, or an API read. Get it right and
4280secrets stay in one place, with one honest boundary, that every other part of
4281the system already trusts.
4282
4283The one-line rule: **a secret is never a record field.** Records are readable,
4284listable, exportable, and included in portable backups by design. A secret must
4285be none of those things. So secrets go somewhere else, through a different door.
4286
4287## Where secrets live
4288
4289There are exactly two homes, by lifetime and ownership:
4290
4291### 1. Deploy-time, server-wide secrets → the environment file
4292
4293A credential the *whole deployment* shares — the SMTP relay password
4294(`DBBASIC_SMTP_PASSWORD`), a connector's API key (`DBBASIC_MAILCOW_API_KEY`),
4295the admin token (`DBBASIC_ADMIN_TOKEN`) — lives in
4296`/etc/dbbasic-object-server.env`, `640 root:dbbasic` (see
4297[`single-vm-deployment.md`](/docs/single-vm-deployment)). It is set by the operator, read by the server
4298process, and never appears in a schema field, a record, an API response, or a
4299backup archive. A package that needs one **declares the env var name in its
4300docs and reads it at runtime** — it never ships the value and never stores it.
4301
4302### 2. Per-user recoverable secrets → the identity vault
4303
4304A secret that belongs to *one user* and must be *recoverable to be used* (a BYO
4305API key the server calls a provider with; a mailbox password handed to the mail
4306server) lives in the **write-only identity vault**, `object_service_keys`
4307(`identity/service_keys.tsv`). Its contract:
4308
4309- **Owner-only file**, `0600`, next to `credentials.tsv` — same filesystem
4310 trust boundary as the admin token.
4311- **Write-only**: a caller can *set* a key, *list which services have one*, and
4312 *delete* one. **No surface ever reads key material back** — not the HTTP API,
4313 not the shell, not an agent, not a package. The server uses a stored key
4314 internally (to call the provider) so it never travels to a browser.
4315- **Stored as-provided, not encrypted, not hashed.** Passwords are hashed
4316 because they only need to be *verified*; these must be *used*, so they must be
4317 recoverable. On-record encryption with a key derived from a server secret
4318 (the pattern many frameworks reach for) is *theater* here — the ciphertext and
4319 the key it's decrypted with live in the same blast radius, so they leak
4320 together (see the Trust Boundary section). The real protection is the `0600`
4321 file and the write-only contract, not a cipher.
4322- **Excluded from portable backups and source control** — so a leaked backup
4323 archive, the most common accidental exposure, contains no secret material.
4324 (See [`backup-restore.md`](/docs/backup-restore); a package must never undo this by copying a secret
4325 into a record it *does* back up.)
4326
4327**Concrete example**: the AI chat, cloud TTS, and cloud STT features
4328(`docs/shell-and-ai.md`) all read from this vault, not the env file — a
4329user's own `anthropic` and/or `openai` key, set via `PUT
4330/identity/users/{you}/service-keys`. One stored `openai` key covers chat,
4331`POST /api/tts` with `engine: ""openai""`, and `POST /api/stt` — no separate
4332signup per feature. This is also why those features are per-user opt-in
4333rather than deployment-wide: `DBBASIC_ENABLE_AI_CHAT`/`_TTS`/`_STT` in the
4334env file only unlock the *capability*; each user still needs their own
4335stored key before any of it does anything for them.
4336
4337## How a secret gets written (it is not a record write)
4338
4339The vault has its own door, deliberately separate from the generic record API:
4340
4341```
4342GET /identity/users/{user_id}/service-keys → which services are set (status only)
4343PUT /identity/users/{user_id}/service-keys → { ""service"": ""..."", ""key"": ""..."" }
4344DELETE /identity/users/{user_id}/service-keys/{service} → clear one
4345```
4346
4347- **Owner-scoped**: only the account owner's own session may write their vault
4348 (or the operator admin gate). Identity is resolved server-side from an opaque
4349 session token — it cannot be spoofed by a client-supplied user id.
4350- **CSRF-guarded**: cross-origin cookie writes are refused.
4351- **Never echoes material**: `PUT` returns metadata (`service`, timestamps)
4352 only; `GET` returns presence only.
4353
4354`POST /collections/{c}/records` — the generic record write — **cannot reach the
4355vault at all.** That is the point: there is no record surface to misconfigure
4356into leaking a secret.
4357
4358**In a generated UI**, a secret is a *write-only field*: it renders as ""set /
4359not set"" with an input that only ever POSTs to the vault endpoint above and
4360never displays a stored value. The form still looks uniform to the user — one
4361field just writes to a different door than the record.
4362
4363## Reserved namespaces: user-writable vs platform-owned
4364
4365A vault entry is keyed by a **service name**. Two kinds:
4366
4367- **User-writable** (a person's own BYO credential): a plain name — `openai`,
4368 `anthropic`, `stripe`. The self-service endpoint above writes these.
4369- **Platform-owned** (a secret the *platform* provisions on the user's behalf —
4370 a generated mailbox password): a **reserved name**, prefixed `sys-`
4371 (e.g. `sys-mailbox-<mailbox_id>`). **The self-service HTTP endpoint refuses to
4372 write or delete a reserved name.** Only in-process server code (a connector, a
4373 provisioning verb) may set one.
4374
4375This is the control that stops a user from **overwriting a secret the platform
4376owns**. Without it, a user could `PUT` their own `sys-mailbox-<id>` and clobber
4377the password the mail server actually uses. `object_service_keys.is_reserved_service()`
4378is the single source of truth for the rule; the HTTP handler enforces it.
4379
4380Two more rules for platform-owned secrets a package must follow:
4381
4382- **Scope every read by the record's owner**, never by caller input. Resolve a
4383 mailbox's secret from *the mailbox record's `owner_id`* — never from a
4384 `mailbox_id` a request supplied — or a user could plant an entry in their own
4385 vault and hope a global lookup reads it.
4386- **Generate, don't collect, when you can.** A provisioned password should be
4387 generated server-side (`secrets.token_urlsafe`), stored in the vault, and
4388 revealed to the user *exactly once* in the provisioning response — never read
4389 back afterward.
4390
4391## The trust boundary (threat model)
4392
4393Be honest about what actually protects the vault, because it changes what's
4394worth building. A secret is reachable only by one of these, and **every one of
4395them also compromises the entire system** — so the vault is not a weaker link,
4396it inherits the system's security:
4397
4398| Vector | What it takes | Does encryption-at-rest help? |
4399|---|---|---|
4400| **A logged-in user** reaching another user's secret | Impossible via the API: owner-scoped, server-derived identity, reserved-namespace guard | N/A — blocked at the app layer |
4401| **Code execution as the service user** | An RCE in the server, or an installed package/connector (both run as the service user and legitimately read the vault) | **No** — the running process holds the key too |
4402| **Root on the host** | Kernel/sudo/other-service exploit; `0600` is nothing to root | **No** — same blast radius |
4403| **The raw disk** | A VM snapshot, an attached volume, a stolen *unencrypted* backup | **Only here** — and only if the key lives off-box (a KMS) |
4404| **The deploy path** | A dev-box SSH key or a push-to-prod pipeline → arbitrary code as the service user | **No** — it's code execution with a login |
4405
4406Conclusions that follow:
4407
4408- **A plain user cannot reach it.** They are boxed into their own namespace by a
4409 server-resolved identity and the reserved-namespace guard.
4410- **On-box encryption of the vault is theater** against every vector except a
4411 raw-disk/snapshot theft — because for all the others the decryption key rides
4412 in the same boundary. The genuinely stronger option is **envelope encryption
4413 with a KMS-held root key**, and only when you have a KMS (not on a single VM).
4414 Until then, the higher-leverage move is to **encrypt backups/snapshots with an
4415 off-box key and lock the hosting account with 2FA** — the deploy credential
4416 and the hosting account are, in effect, vault keys.
4417- **""Installed package = code execution as the service user""** is a stated trust
4418 fact, which is why package installs (and any dynamically loaded connector
4419 code) are admin-gated. Fine while packages are first-party; it needs a
4420 stronger answer before any third-party marketplace.
4421
4422## Checklist for package authors
4423
4424- **Never** put a secret in a schema field or a record.
4425- Server-wide credential → read from a documented `DBBASIC_*` env var; never ship or store the value.
4426- Per-user recoverable secret → the identity vault (`object_service_keys`), write-only.
4427- Platform-provisioned secret → a `sys-`-prefixed reserved service name; write it only from server-side code.
4428- Resolve a secret by the owning record's `owner_id`, never by caller-supplied input.
4429- Prefer generating a secret and revealing it once over collecting and storing it.
4430- In a UI, a secret field is write-only (set/not-set), posting to the vault endpoint, never rendering the value.
4431- Never copy a secret into anything that lands in a backup or an export.
4432
4433## See also
4434
4435- [`single-vm-deployment.md`](/docs/single-vm-deployment) — the env-file trust boundary and admin-route posture.
4436- [`permissions-model.md`](/docs/permissions-model) — owner-scoping and the admin bypass.
4437- [`backup-restore.md`](/docs/backup-restore) — what is and isn't in a portable backup.
4438- [`package-authoring.md`](/docs/package-authoring) — how a package ships schemas, permissions, and (soon) connectors.
4439"
4440traffic-limits Traffic Limits Security & Limits 620 How the DBBASIC Object Server rejects oversized, overloaded, or runaway traffic before object code runs, and the env vars that control each limit. "DBBASIC should reject bad or oversized traffic before object code runs. The
4441runtime keeps the object loop fast by failing early when a request is too large
4442or when the server is under pressure.
4443
4444## Request Body Size
4445
4446The ASGI server enforces a request body cap:
4447
4448```text
4449DBBASIC_MAX_REQUEST_BYTES=1048576
4450```
4451
4452The default is `1048576` bytes, or 1 MiB. If a request body is larger than the
4453configured value, the server returns:
4454
4455```http
4456413 Payload Too Large
4457```
4458
4459```json
4460{
4461 ""status"": ""error"",
4462 ""error"": ""Request body too large"",
4463 ""max_bytes"": 1048576
4464}
4465```
4466
4467This applies before JSON parsing and before object execution. It checks
4468`Content-Length` when present and also counts streamed ASGI body chunks.
4469
4470Large file upload paths should get their own file API and their own explicit
4471limits. Normal object `POST`, `PUT`, and `DELETE` requests should stay small.
4472
4473## Concurrency
4474
4475The ASGI server also enforces per-process concurrency caps:
4476
4477```text
4478DBBASIC_MAX_CONCURRENT_REQUESTS=64
4479DBBASIC_MAX_CONCURRENT_EXECUTIONS=8
4480```
4481
4482`DBBASIC_MAX_CONCURRENT_REQUESTS` limits non-health HTTP requests that are in
4483flight inside one server process. `/health` bypasses this limit so a monitor can
4484still check whether the process is alive under load.
4485
4486`DBBASIC_MAX_CONCURRENT_EXECUTIONS` limits object method executions inside one
4487server process. Introspection routes, health checks, and other non-execution
4488requests do not consume object execution slots.
4489
4490When either limit is full, the server returns:
4491
4492```http
4493503 Service Unavailable
4494```
4495
4496```json
4497{
4498 ""status"": ""error"",
4499 ""error"": ""Server is busy"",
4500 ""limit"": ""object_executions"",
4501 ""max_concurrent"": 8
4502}
4503```
4504
4505Set either concurrency value to `0` to disable that app-level limit for local
4506experiments. Production and staging deployments should keep explicit limits.
4507
4508These caps are per process. If uvicorn runs multiple workers, each worker has
4509its own counters. A reverse proxy or process manager still controls the total
4510machine-level shape.
4511
4512## Rate Limits
4513
4514Rate limiting is opt-in:
4515
4516```text
4517DBBASIC_RATE_LIMIT_REQUESTS=1000
4518DBBASIC_RATE_LIMIT_WINDOW_SECONDS=60
4519```
4520
4521The default `DBBASIC_RATE_LIMIT_REQUESTS=0` leaves the app-level limiter
4522disabled for local development. Public staging and production deployments should
4523set an explicit value.
4524
4525When enabled, the server checks the rate limit before reading the request body
4526or running object code. Plain `GET /health` is excluded so external monitoring
4527can still tell whether the process is alive. Detailed health via
4528`capacity=true` or `metrics=true` is rate-limited and still requires the admin
4529token.
4530
4531The limiter stores one timestamp bucket per identity under:
4532
4533```text
4534data/ratelimit/
4535```
4536
4537Those files are runtime guardrails, not application data. Backups exclude them.
4538
4539Valid admin-token requests use a token bucket. Other requests use the client IP.
4540By default the server ignores `X-Forwarded-For` and `X-Real-IP`; enable proxy
4541header trust only when uvicorn is bound privately behind a trusted reverse
4542proxy:
4543
4544```text
4545DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS=true
4546```
4547
4548When the limit is full, the server returns:
4549
4550```http
4551429 Too Many Requests
4552Retry-After: 30
4553```
4554
4555```json
4556{
4557 ""status"": ""error"",
4558 ""error"": ""Rate limit exceeded"",
4559 ""retry_after"": 30,
4560 ""limit"": 1000,
4561 ""window_seconds"": 60
4562}
4563```
4564
4565## Object Execution Timeout
4566
4567Object execution timeout is opt-in:
4568
4569```text
4570DBBASIC_OBJECT_TIMEOUT_SECONDS=5
4571```
4572
4573The default `0` keeps local development on the direct in-process execution path.
4574When the value is above zero, object methods run in a short-lived worker process.
4575If the object does not finish before the wall-clock timeout, the server
4576terminates the worker and returns:
4577
4578```http
4579504 Gateway Timeout
4580```
4581
4582```json
4583{
4584 ""status"": ""error"",
4585 ""error"": ""Execution failed: GET timed out for object basics_slow after 5 seconds""
4586}
4587```
4588
4589Timeout failures are written to the object log with
4590`ObjectExecutionTimeoutError`, so Scroll and operators can see which object
4591timed out.
4592
4593Trusted server-owned objects can opt back into the fast in-process path even
4594when timeouts are enabled:
4595
4596```text
4597DBBASIC_TRUSTED_IN_PROCESS_OBJECTS=site_home,system_health
4598```
4599
4600Use that only for reviewed objects shipped or controlled by the server operator,
4601such as a public homepage that receives frequent hits. Untrusted user objects
4602should stay on the timeout-enabled worker path.
4603
4604This boundary is intentionally narrow. It stops hung object work and releases
4605the execution slot, but it is not the final sandbox. CPU quotas, memory limits,
4606and a longer-lived worker pool are still future hardening steps.
4607
4608## Capacity Health
4609
4610`GET /health` stays public and cheap:
4611
4612```json
4613{
4614 ""status"": ""ok""
4615}
4616```
4617
4618Detailed capacity is an operator surface and requires the admin token:
4619
4620```http
4621GET /health?capacity=true
4622Authorization: Token <token>
4623```
4624
4625It reports the configured request and object execution limits, current
4626in-flight slot counts, object count, storage status, version, uptime, request
4627counts, recent response timing, and basic process/system information. Scroll and
4628future station routers should use this detailed shape instead of guessing from
4629logs.
4630
4631`GET /health?metrics=true` includes the same capacity payload plus detailed
4632request metrics such as status counts, top paths, and recent HTTP errors.
4633
4634## High-Traffic Shape
4635
4636Use layered limits:
4637
4638- reverse proxy body limit before Python receives the request
4639- app body limit through `DBBASIC_MAX_REQUEST_BYTES`
4640- app request and object execution limits through the concurrency env vars
4641- per-IP and per-token rate limits before write or execute paths
4642- object execution wall-clock timeouts
4643- later CPU and memory isolation around worker processes
4644
4645When the server is overloaded, it should return `429 Too Many Requests` for rate
4646limits or `503 Service Unavailable` for temporary capacity pressure. That is
4647better than letting a small VM run out of memory or accumulate unbounded object
4648work.
4649
4650## Staging Defaults
4651
4652For the first public staging VM:
4653
4654```text
4655DBBASIC_MAX_REQUEST_BYTES=1048576
4656DBBASIC_MAX_CONCURRENT_REQUESTS=64
4657DBBASIC_MAX_CONCURRENT_EXECUTIONS=8
4658DBBASIC_OBJECT_TIMEOUT_SECONDS=5
4659DBBASIC_RATE_LIMIT_REQUESTS=1000
4660DBBASIC_RATE_LIMIT_WINDOW_SECONDS=60
4661DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS=true
4662DBBASIC_ENABLE_SOURCE_WRITES=false
4663```
4664
4665Keep source writes closed until auth, permissions, backups, and rollback checks
4666are working. Keep public routes narrow. Let the object server prove the loop
4667without allowing arbitrary public code changes.
4668
4669## Cluster Direction
4670
4671Local capacity should become a station signal. A single station can already
4672return `503` when it is full. Later, station heartbeat or health data should
4673include capacity fields so a cluster router can avoid saturated stations before
4674forwarding object work.
4675
4676That signal should include request slots, execution slots, recent error rate,
4677and basic machine load such as CPU and memory pressure. That keeps the cluster
4678model simple: each station knows its own limits, and the router uses those
4679limits to choose where work should go.
4680
4681## Logs And Garbage Collection
4682
4683Traffic controls also need disk controls:
4684
4685- rotate object logs before they grow forever
4686- gzip rotated logs at rest
4687- keep a bounded number of rotated logs per object
4688- disable noisy uvicorn access logs in systemd and rely on object logs,
4689 request metrics, and deliberate analytics objects
4690- cap journald retention for process logs
4691- keep backup retention explicit
4692- delete temp files and stale lock files only through known runtime paths
4693
4694The active object log stays plain TSV so tools can tail it. Rotated logs can be
4695read in place with normal Unix tools such as `gzip -cd`.
4696
4697## Next Limits
4698
4699The request, concurrency, rate-limit, and timeout caps are only the first
4700boundaries. The next production-hardening steps are:
4701
4702- CPU and memory isolation for untrusted object code
4703- a longer-lived worker pool so timeout-enabled execution does not need a new
4704 process per request
4705"
4706single-vm-deployment Single VM Deployment Deployment & Operations 700 How to deploy DBBASIC Object Server on a single VM: layout, install, environment, systemd, HTTPS proxy, backups, and upgrade procedure. "This is the first deployment target for DBBASIC Object Server: one small VM,
4707one ASGI process, one object source directory, and one data directory.
4708
4709The goal is not to claim the public runtime is production-ready. The goal is to
4710keep a clean staging server running so install, restart, logs, object storage,
4711and the edit/run/inspect loop are tested on a real machine.
4712
4713This shape should stay cheap and understandable: one small VM can run the
4714server, objects, TSV-backed state, logs, versions, HTTPS proxy, and monitoring
4715without requiring a separate database tier for the first useful app.
4716
4717## Deployment Goal
4718
4719The first deployment path should stay closer to a simple Unix/PHP-style upload
4720than a large multi-service app stack:
4721
4722- one VM is enough for the first useful business app
4723- no container runtime is required for the base deployment
4724- no separate database server is required before the app proves itself
4725- server code, live object source, runtime data, and secrets live in separate
4726 Unix paths
4727- normal upgrades are `git pull`, package install, service restart, checks
4728- object edits do not require a full server redeploy
4729- every manual step should become scriptable after it stays boring
4730
4731This does not mean skipping safety. The simple path still uses a service user,
4732systemd, a reverse proxy, HTTPS, private config, filesystem checks, backups, and
4733explicit public route allowlists.
4734
4735## Verified Baseline
4736
4737The first staging install was verified on:
4738
4739- Ubuntu 24.04.4 LTS
4740- Python 3.12.3
4741- Git 2.43.0
4742- Caddy 2.11.4
4743- systemd with the DBBASIC service and Caddy running as separate units
4744
4745Record the baseline on each new VM. Run the Caddy commands after Caddy is
4746installed, if the VM does not already have it.
4747
4748```bash
4749lsb_release -a
4750python3 --version
4751git --version
4752caddy version
4753systemctl is-active dbbasic-object-server caddy
4754cd /opt/dbbasic-object-server
4755sudo -u dbbasic git rev-parse --short HEAD
4756```
4757
4758Do not commit the VM's real IP address, hostname, or provider-specific details
4759to this repository.
4760
4761## System Updates
4762
4763Before using a new VM for staging, apply normal OS updates and reboot if Ubuntu
4764requests it:
4765
4766```bash
4767sudo apt update
4768apt list --upgradable
4769sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade -y
4770sudo DEBIAN_FRONTEND=noninteractive apt-get autoremove -y
4771test -f /var/run/reboot-required && cat /var/run/reboot-required || true
4772```
4773
4774If packages are kept back because they need new dependencies, install them
4775explicitly when they are normal platform packages:
4776
4777```bash
4778sudo DEBIAN_FRONTEND=noninteractive apt-get install -y fwupd
4779```
4780
4781After a reboot, verify the services and endpoint again:
4782
4783```bash
4784sudo systemctl is-active dbbasic-object-server caddy
4785curl https://dbbasic.example.com/
4786```
4787
4788## Provider Monitoring
4789
4790For DigitalOcean Droplets, the metrics agent is useful on staging VMs because it
4791adds CPU, memory, disk, and network graphs plus alerting while the server is
4792under active development. DigitalOcean describes this as a free opt-in service.
4793It sends system telemetry for monitoring, not customer content.
4794
4795Install it after normal OS updates:
4796
4797```bash
4798curl -fsSL https://repos.insights.digitalocean.com/install.sh -o /tmp/do-agent-install.sh
4799sudo bash /tmp/do-agent-install.sh
4800```
4801
4802Verify it:
4803
4804```bash
4805systemctl is-active do-agent
4806systemctl is-enabled do-agent
4807dpkg-query -W do-agent
4808systemctl --no-pager --full status do-agent
4809```
4810
4811Metrics usually appear in the Droplet's Insights tab after a few minutes. This
4812is provider-specific VM monitoring; it does not replace DBBASIC's own object
4813logs, state, versions, or future application/runtime metrics.
4814
4815## Layout
4816
4817Use separate paths for server code, object source, and runtime data:
4818
4819```text
4820/opt/dbbasic-object-server
4821/var/lib/dbbasic-object-server/objects
4822/var/lib/dbbasic-object-server/data
4823```
4824
4825- `/opt/dbbasic-object-server` is the git checkout.
4826- `/var/lib/dbbasic-object-server/objects` contains live object source files.
4827- `/var/lib/dbbasic-object-server/data` contains state, logs, and versions.
4828
4829Keeping objects and data outside the checkout lets the server be upgraded
4830without mixing application data into git history.
4831
4832## Install
4833
4834On a fresh Debian or Ubuntu VM:
4835
4836```bash
4837sudo apt update
4838sudo apt install -y python3 python3-venv python3-pip git
4839sudo useradd --system --home /var/lib/dbbasic-object-server --shell /usr/sbin/nologin dbbasic
4840sudo mkdir -p /opt/dbbasic-object-server
4841sudo mkdir -p /var/lib/dbbasic-object-server/objects
4842sudo mkdir -p /var/lib/dbbasic-object-server/data
4843sudo chown -R dbbasic:dbbasic /opt/dbbasic-object-server /var/lib/dbbasic-object-server
4844sudo chmod 755 /opt/dbbasic-object-server
4845sudo chmod 750 /var/lib/dbbasic-object-server
4846sudo chmod 750 /var/lib/dbbasic-object-server/objects
4847sudo chmod 750 /var/lib/dbbasic-object-server/data
4848```
4849
4850Clone and install the server:
4851
4852```bash
4853sudo -u dbbasic git clone <repository-url> /opt/dbbasic-object-server
4854cd /opt/dbbasic-object-server
4855sudo -u dbbasic python3 -m venv .venv
4856sudo -u dbbasic .venv/bin/python -m pip install -e '.[server]'
4857```
4858
4859## Environment
4860
4861Create `/etc/dbbasic-object-server.env`:
4862
4863```text
4864DBBASIC_OBJECTS_DIR=/var/lib/dbbasic-object-server/objects
4865DBBASIC_DATA_DIR=/var/lib/dbbasic-object-server/data
4866DBBASIC_BACKUPS_DIR=/var/lib/dbbasic-object-server/data/backups
4867DBBASIC_PACKAGES_DIR=/opt/dbbasic-object-server/packages
4868DBBASIC_ENABLE_SOURCE_WRITES=false
4869DBBASIC_ENABLE_FILE_WRITES=false
4870DBBASIC_ENABLE_PACKAGE_INSTALLS=false
4871DBBASIC_ENABLE_PACKAGE_RESTORE=false
4872# AI chat, voice, and reader -- each is its own flag; enabling AI chat does
4873# NOT enable TTS/STT/reader. Each also needs a per-user provider key stored
4874# via PUT /identity/users/{id}/service-keys before it does anything -- see
4875# docs/shell-and-ai.md and docs/secrets-and-credentials.md.
4876DBBASIC_ENABLE_AI_CHAT=false
4877DBBASIC_AI_TIMEOUT_SECONDS=60
4878DBBASIC_AI_DEFAULT_MODEL=anthropic:claude-haiku-4-5
4879DBBASIC_ENABLE_TTS=false
4880DBBASIC_TTS_CLOUD_TIMEOUT_SECONDS=30
4881# STT costs real money on every utterance (not just occasional replies
4882# like TTS) -- leave this off even where TTS is on unless you mean it.
4883DBBASIC_ENABLE_STT=false
4884DBBASIC_STT_TIMEOUT_SECONDS=30
4885DBBASIC_ENABLE_READER=false
4886DBBASIC_ADMIN_TOKEN=replace-with-a-generated-token
4887DBBASIC_MAX_REQUEST_BYTES=1048576
4888DBBASIC_MAX_OBJECT_FILE_BYTES=1048576
4889DBBASIC_MAX_CONCURRENT_REQUESTS=64
4890DBBASIC_MAX_CONCURRENT_EXECUTIONS=8
4891DBBASIC_OBJECT_TIMEOUT_SECONDS=5
4892DBBASIC_TRUSTED_IN_PROCESS_OBJECTS=site_home
4893DBBASIC_RATE_LIMIT_REQUESTS=1000
4894DBBASIC_RATE_LIMIT_WINDOW_SECONDS=60
4895DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS=true
4896DBBASIC_ENABLE_PERMISSION_AUDIT=false
4897DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=false
4898DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=false
4899DBBASIC_PERMISSION_TRUST_HEADERS=false
4900DBBASIC_ENABLE_SESSION_LOGIN=false
4901DBBASIC_SESSION_LOGIN_TOKEN=
4902DBBASIC_ENABLE_PASSWORD_LOGIN=false
4903DBBASIC_COOKIE_SECURE=true
4904DBBASIC_LOGIN_LOCKOUT_ATTEMPTS=5
4905DBBASIC_LOGIN_LOCKOUT_WINDOW_SECONDS=900
4906DBBASIC_ENABLE_RECORD_EVENTS=true
4907DBBASIC_EVENT_KEEP_COUNT=1000
4908DBBASIC_EVENT_KEEP_SECONDS=604800
4909DBBASIC_LOG_MAX_BYTES=10485760
4910DBBASIC_LOG_COMPRESS_ROTATED=true
4911DBBASIC_LOG_KEEP_ROTATED=32
4912# Other feature gates worth knowing exist -- off by default, turn on what
4913# you actually use rather than everything at once.
4914DBBASIC_ENABLE_USER_FILES=false
4915DBBASIC_ENABLE_REALTIME=false
4916DBBASIC_ENABLE_SITE_ROUTES=false
4917DBBASIC_ENABLE_FEED=false
4918DBBASIC_ENABLE_FILTERING=false
4919DBBASIC_ENABLE_CONCURRENCY=false
4920DBBASIC_ENABLE_SESSION_ADMIN_GATES=false
4921```
4922
4923Generate a local token on the VM:
4924
4925```bash
4926python3 -c ""import secrets; print(secrets.token_urlsafe(32))""
4927```
4928
4929Use that generated value for `DBBASIC_ADMIN_TOKEN`. Do not reuse test tokens,
4930README placeholders, or tokens from another server. It is required for local or
4931admin object listing and introspection requests such as source, state, logs,
4932metadata, and versions.
4933
4934## Bootstrap Browser Login
4935
4936To enable browser login on a fresh VM, create the first admin user from the
4937shell and turn on password login:
4938
4939```bash
4940cd /opt/dbbasic-object-server
4941sudo -u dbbasic DBBASIC_DATA_DIR=/var/lib/dbbasic-object-server/data \
4942 .venv/bin/python object_identity_cli.py create-superuser \
4943 --user-id operator --email you@example.com
4944```
4945
4946The password is prompted twice without echo and stored only as a scrypt hash
4947under `data/identity/credentials.tsv` (owner-only permissions, inside the
4948runtime data directory that never enters source control). Then set
4949`DBBASIC_ENABLE_PASSWORD_LOGIN=true` in the environment file, restart the
4950service, and allow `/login` and `/logout` through the reverse proxy. Keep
4951`DBBASIC_COOKIE_SECURE=true` behind HTTPS; set it to `false` only for
4952plain-HTTP local development. Additional users can be managed with the same
4953CLI (`create-user`, `set-password`, `list-users`) or through the admin HTTP
4954surface from Scroll.
4955
4956For the first VM boot, leave source writes disabled. After health checks,
4957backups, and proxy access are working, a staging server can enable source
4958writes:
4959
4960```text
4961DBBASIC_ENABLE_SOURCE_WRITES=true
4962```
4963
4964Leave file writes disabled on public staging until quotas, permissions, and
4965content policy are ready for the use case. A controlled operator VM can enable
4966them separately:
4967
4968```text
4969DBBASIC_ENABLE_FILE_WRITES=true
4970```
4971
4972Permission audit and enforcement should also stay closed on first boot. Turn on
4973audit mode first to watch route decisions without blocking users:
4974
4975```text
4976DBBASIC_ENABLE_PERMISSION_AUDIT=true
4977```
4978
4979Request blocking only after the persisted policy and auth gateway are confirmed:
4980
4981```text
4982DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true
4983```
4984
4985That flag is a request, not a blind switch. The server checks the same
4986readiness blockers exposed by `/permissions/status`; if the policy is invalid or
4987the default role-based policy has no grants, the route stays in audit/shadow
4988mode and `permissions.enforcement_blocked` reports `true`.
4989
4990For manual recovery or a narrow test, the readiness guard can be bypassed:
4991
4992```text
4993DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true
4994```
4995
4996Do not leave that override enabled on public staging.
4997
4998`DBBASIC_PERMISSION_TRUST_HEADERS=true` is only appropriate when a trusted proxy
4999or auth gateway strips client-supplied identity headers and writes fresh
5000`X-DBBASIC-*` headers itself.
5001
5002Do not commit real tokens, VM hostnames, private URLs, or deployment-specific
5003paths to this repository.
5004
5005After creating the file, keep it readable by root and the service group only:
5006
5007```bash
5008sudo chown root:dbbasic /etc/dbbasic-object-server.env
5009sudo chmod 640 /etc/dbbasic-object-server.env
5010```
5011
5012## Filesystem Check
5013
5014The public package includes a small single-VM layout checker. Run it after
5015installing the server and after changing ownership or deployment paths:
5016
5017```bash
5018cd /opt/dbbasic-object-server
5019set -a
5020. /etc/dbbasic-object-server.env
5021set +a
5022.venv/bin/python -m deployment_checks
5023```
5024
5025The checker validates the normal Unix boundary:
5026
5027- server code under `/opt/dbbasic-object-server`
5028- live object source under `/var/lib/dbbasic-object-server/objects`
5029- runtime data under `/var/lib/dbbasic-object-server/data`
5030- deployment secrets under `/etc/dbbasic-object-server.env`
5031- systemd unit under `/etc/systemd/system/dbbasic-object-server.service`
5032
5033Errors should be fixed before exposing routes publicly. Warnings usually mean a
5034runtime path is visible to other local users. On a single-purpose staging VM
5035that may not break anything, but the safer default is `750` for object and data
5036directories.
5037
5038## Traffic Limits
5039
5040Keep the public server narrow while it is staging.
5041
5042`DBBASIC_MAX_REQUEST_BYTES` limits inbound HTTP request bodies before JSON
5043parsing or object execution. The default is 1 MiB:
5044
5045```text
5046DBBASIC_MAX_REQUEST_BYTES=1048576
5047DBBASIC_MAX_CONCURRENT_REQUESTS=64
5048DBBASIC_MAX_CONCURRENT_EXECUTIONS=8
5049```
5050
5051Configure the same or smaller request body limit in the reverse proxy so Caddy
5052or nginx can reject oversized traffic before Python handles it. The app-level
5053limit stays in place because proxy configuration can drift.
5054
5055The concurrency limits are per process. Under overload, the server returns
5056`503` rather than queueing unlimited work on a small VM. Future production
5057hardening still needs CPU/memory isolation and a longer-lived worker pool.
5058`DBBASIC_OBJECT_TIMEOUT_SECONDS` runs object methods in a worker process and
5059returns `504` if the wall-clock timeout is exceeded.
5060`DBBASIC_TRUSTED_IN_PROCESS_OBJECTS` is a comma-separated allowlist for
5061reviewed server-owned objects that should keep the fast in-process path, such as
5062the public homepage. Do not add user-created objects to this list.
5063The rate limit values above return `429` with `Retry-After` when one IP or
5064valid admin token exceeds the configured window.
5065`DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS` is appropriate only because uvicorn is
5066bound to `127.0.0.1` behind Caddy.
5067
5068See [`traffic-limits.md`](/docs/traffic-limits) for the operating model.
5069
5070## Log Maintenance
5071
5072Object logs are part of the runtime feedback loop, so staging servers should not
5073let them grow forever.
5074
5075The active log for an object stays plain TSV:
5076
5077```text
5078/var/lib/dbbasic-object-server/data/logs/{object_id}/log.tsv
5079```
5080
5081When it reaches `DBBASIC_LOG_MAX_BYTES`, the server rotates it. New rotated logs
5082are gzip-compressed by default:
5083
5084```text
5085/var/lib/dbbasic-object-server/data/logs/{object_id}/log-YYYYMMDD-HHMMSS-ffffff.tsv.gz
5086```
5087
5088Inspect compressed logs without expanding a second copy:
5089
5090```bash
5091gzip -cd /var/lib/dbbasic-object-server/data/logs/site_home/log-*.tsv.gz
5092```
5093
5094Garbage collection is controlled by `DBBASIC_LOG_KEEP_ROTATED`. The default
5095keeps the newest 32 rotated files per object and deletes older rotated logs
5096after a successful rotation. Set it to `0` to keep all rotated logs.
5097
5098This is at-rest compression. For in-motion compression, start with reverse-proxy
5099HTTP compression for large log responses. Later station replication should
5100compress batches, not individual tiny events.
5101
5102Event rows are a delivery queue, not the durable audit log. The server and
5103daemon prune old `data/state/events/state.tsv` event rows with:
5104
5105```bash
5106DBBASIC_EVENT_KEEP_COUNT=1000
5107DBBASIC_EVENT_KEEP_SECONDS=604800
5108```
5109
5110Set either value to `0` to disable that retention rule. Subscriptions stay in
5111place, and any event referenced by a subscription `last_event_id` is protected so
5112the daemon keeps its delivery cursor. Failed delivery attempts are protected too,
5113so callback failures stay visible and retryable until delivery succeeds or the
5114subscription is repaired.
5115
5116## Packages
5117
5118Packages are read from `DBBASIC_PACKAGES_DIR`. The public server currently
5119supports package listing, dry-run planning, and package installs only when
5120`DBBASIC_ENABLE_PACKAGE_INSTALLS=true`. Dry-run and install requests append
5121compact audit rows under
5122`/var/lib/dbbasic-object-server/data/package_changes/{package_id}/changes.jsonl`.
5123Package installs create a runtime restore point before writing source, schema,
5124or seed files. Restore points default to
5125`/var/lib/dbbasic-object-server/data/backups/`; set `DBBASIC_BACKUPS_DIR` if
5126they should live on another mounted volume.
5127
5128Package restore is separate and requires `DBBASIC_ENABLE_PACKAGE_RESTORE=true`.
5129It restores a recorded install restore point with overwrite and snapshot
5130pruning, so package-created runtime files are removed if they were not in the
5131archive.
5132
5133Keep `DBBASIC_ENABLE_PACKAGE_INSTALLS=false` and
5134`DBBASIC_ENABLE_PACKAGE_RESTORE=false` on public staging unless `/packages` is
5135reachable only from a private admin surface. The first install path writes
5136objects and schemas, creates seed TSV files only when no data exists, and
5137rejects permissions/migrations until merge/run semantics are explicit.
5138
5139## Minimal Object
5140
5141Create one object so the VM can prove execution, state, and logs:
5142
5143```bash
5144sudo -u dbbasic mkdir -p /var/lib/dbbasic-object-server/objects/site
5145sudo -u dbbasic tee /var/lib/dbbasic-object-server/objects/site/home.py >/dev/null <<'PY'
5146def GET(request):
5147 count = _state_manager.get(""count"", 0) + 1
5148 _state_manager.set(""count"", count)
5149 return {
5150 ""status"": ""ok"",
5151 ""message"": ""DBBASIC Object Server is running"",
5152 ""count"": count,
5153 }
5154PY
5155```
5156
5157This object resolves as `site_home`.
5158
5159For a public staging page, the same object can return HTML instead of JSON:
5160
5161```python
5162def GET(request):
5163 count = int(_state_manager.get(""count"", 0)) + 1
5164 _state_manager.set(""count"", count)
5165 _logger.info(""site_home served"", count=count, response_type=""html"")
5166
5167 return {
5168 ""content_type"": ""text/html; charset=utf-8"",
5169 ""body"": f""<!doctype html><h1>DBBASIC Object Server</h1><p>Served {count} times.</p>"",
5170 }
5171```
5172
5173That keeps the staging page honest: it is a live object with state and logs, not
5174a static file.
5175
5176## Run Manually
5177
5178Before creating a service, run the server by hand:
5179
5180```bash
5181cd /opt/dbbasic-object-server
5182set -a
5183. /etc/dbbasic-object-server.env
5184set +a
5185.venv/bin/uvicorn object_server:app --host 127.0.0.1 --port 8001 --no-access-log
5186```
5187
5188In another shell:
5189
5190```bash
5191curl http://127.0.0.1:8001/health
5192curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/health?capacity=true'
5193curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" http://127.0.0.1:8001/objects
5194curl http://127.0.0.1:8001/objects/site_home
5195curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/objects/site_home?state=true'
5196curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/objects/site_home?logs=true'
5197```
5198
5199## systemd
5200
5201Create `/etc/systemd/system/dbbasic-object-server.service`:
5202
5203```ini
5204[Unit]
5205Description=DBBASIC Object Server
5206After=network-online.target
5207Wants=network-online.target
5208
5209[Service]
5210User=dbbasic
5211Group=dbbasic
5212WorkingDirectory=/opt/dbbasic-object-server
5213EnvironmentFile=/etc/dbbasic-object-server.env
5214ExecStart=/opt/dbbasic-object-server/.venv/bin/uvicorn object_server:app --host 127.0.0.1 --port 8001 --no-access-log
5215Restart=on-failure
5216RestartSec=2
5217NoNewPrivileges=true
5218PrivateTmp=true
5219ProtectSystem=full
5220ReadWritePaths=/var/lib/dbbasic-object-server
5221
5222[Install]
5223WantedBy=multi-user.target
5224```
5225
5226Start it:
5227
5228```bash
5229sudo systemctl daemon-reload
5230sudo systemctl enable --now dbbasic-object-server
5231sudo systemctl status dbbasic-object-server
5232curl http://127.0.0.1:8001/health
5233curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" 'http://127.0.0.1:8001/health?capacity=true'
5234```
5235
5236Watch logs:
5237
5238```bash
5239sudo journalctl -u dbbasic-object-server -f
5240```
5241
5242`--no-access-log` keeps successful request lines out of journald. DBBASIC keeps
5243the useful application trail in object logs and `/health?metrics=true`, which
5244are the surfaces Scroll should read. If access analytics are needed later,
5245build them as an explicit app/object instead of relying on raw process logs.
5246
5247Cap journald so process logs cannot consume the VM:
5248
5249```bash
5250sudo mkdir -p /etc/systemd/journald.conf.d
5251sudo tee /etc/systemd/journald.conf.d/99-dbbasic.conf >/dev/null <<'EOF'
5252[Journal]
5253SystemMaxUse=128M
5254SystemKeepFree=1G
5255MaxRetentionSec=7day
5256Compress=yes
5257EOF
5258sudo systemctl restart systemd-journald
5259sudo journalctl --vacuum-size=128M
5260sudo journalctl --disk-usage
5261```
5262
5263## Upgrade
5264
5265**Two different things live at `/opt` and two different things get served
5266at `/var/lib` — know which one you changed before declaring a deploy done.**
5267
5268Core server modules (`object_server.py`, `object_ai.py`, and every other
5269`.py` at the repo root) are imported directly from
5270`/opt/dbbasic-object-server` — the steps below are complete for these:
5271
5272```bash
5273cd /opt/dbbasic-object-server
5274sudo -u dbbasic git pull --ff-only
5275sudo -u dbbasic .venv/bin/python -m pip install -e '.[server]'
5276sudo systemctl restart dbbasic-object-server
5277```
5278
5279**Anything under `packages/<name>/` (objects, schemas, seed data) is
5280different.** The running server reads these from a separately *installed*
5281copy at `/var/lib/dbbasic-object-server/objects/` and
5282`/var/lib/dbbasic-object-server/data/schemas/` — `git pull` alone never
5283syncs into it, no matter how many times you restart. A `git pull`-only
5284""deploy"" of a `packages/` change is a silent no-op: the commit lands, the
5285tests pass, the service restarts clean, and the live server keeps running
5286the old code with no error anywhere. After the `git pull` above, also run:
5287
5288```bash
5289curl -s -X POST https://dbbasic.example.com/packages/app-shell/install \
5290 -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" -H 'Content-Type: application/json' \
5291 -d '{""allow_replace"": true}'
5292```
5293
5294(swap `app-shell` for whichever package changed; `DBBASIC_ENABLE_PACKAGE_INSTALLS`
5295must be `true`). Check the response's `install.objects[].status` — `""updated""`
5296means a real change landed, `""unchanged""` means nothing to sync. A schema-only
5297change can also go straight to `PUT /admin/schemas/{collection}` with the
5298updated schema JSON, same admin-token gate. Either way, **restart the
5299service afterward** — object modules are cached in the running process, so
5300even a correctly-installed file has no effect until the next restart clears
5301that cache:
5302
5303```bash
5304sudo systemctl restart dbbasic-object-server
5305```
5306
5307To confirm a `packages/` deploy actually landed rather than trusting the
5308install response alone, diff the live file against the source directly:
5309
5310```bash
5311diff /var/lib/dbbasic-object-server/objects/site/<name>.py \
5312 /opt/dbbasic-object-server/packages/<pkg>/objects/site/<name>.py
5313```
5314
5315A non-empty diff means the install step (or the restart) didn't happen yet.
5316
5317Then verify the local service, filesystem layout, public proxy, and deployed
5318commit:
5319
5320```bash
5321cd /opt/dbbasic-object-server
5322set -a
5323. /etc/dbbasic-object-server.env
5324set +a
5325.venv/bin/python -m deployment_checks
5326curl http://127.0.0.1:8001/health
5327curl https://dbbasic.example.com/
5328sudo systemctl is-active dbbasic-object-server caddy
5329sudo -u dbbasic git rev-parse --short HEAD
5330```
5331
5332If the upgrade only changes live objects that were edited **directly**
5333under `/var/lib/dbbasic-object-server/objects` (not via a `packages/` git
5334change), the server should not need a git deploy at all. That is the
5335object loop: edit one object, run it, inspect state/logs, and keep the
5336version trail. This is a third, separate path from both cases above — it
5337never touches `/opt` or git at all.
5338
5339## HTTPS Proxy
5340
5341For a public or staging hostname, put Caddy or nginx in front of uvicorn and keep
5342uvicorn bound to `127.0.0.1`.
5343
5344Install Caddy if the VM does not already have a reverse proxy:
5345
5346```bash
5347sudo apt install -y caddy
5348```
5349
5350Example Caddyfile using documentation domains:
5351
5352```caddyfile
5353dbbasic.example.com {
5354 reverse_proxy 127.0.0.1:8001
5355}
5356```
5357
5358Replace `dbbasic.example.com` only on the VM. Do not commit real deployment
5359hostnames into this repository.
5360
5361If Caddy was already installed, inspect what it is serving before taking over
5362the hostname:
5363
5364```bash
5365sudo systemctl status caddy --no-pager
5366sudo sed -n '1,220p' /etc/caddy/Caddyfile
5367sudo find /usr/share/caddy -maxdepth 3 -type f -printf '%p\t%s bytes\n'
5368```
5369
5370Back up the old config before changing it:
5371
5372```bash
5373sudo cp -a /etc/caddy/Caddyfile /etc/caddy/Caddyfile.before-dbbasic
5374```
5375
5376For the earliest public staging endpoint, expose only the hello object, system
5377dashboard object, admin status and admin inspection routes, health check, and
5378current-session self-service route until auth, permissions, and source
5379visibility are ready. Keep the broad `/objects` listing blocked:
5380
5381```caddyfile
5382dbbasic.example.com {
5383 handle / {
5384 rewrite * /objects/site_home
5385 reverse_proxy 127.0.0.1:8001
5386 }
5387
5388 handle /health {
5389 reverse_proxy 127.0.0.1:8001
5390 }
5391
5392 handle /login {
5393 reverse_proxy 127.0.0.1:8001
5394 }
5395
5396 handle /logout {
5397 reverse_proxy 127.0.0.1:8001
5398 }
5399
5400 handle /admin/status {
5401 reverse_proxy 127.0.0.1:8001
5402 }
5403
5404 handle /admin/changes* {
5405 reverse_proxy 127.0.0.1:8001
5406 }
5407
5408 handle /admin/objects* {
5409 reverse_proxy 127.0.0.1:8001
5410 }
5411
5412 handle /admin/files* {
5413 reverse_proxy 127.0.0.1:8001
5414 }
5415
5416 handle /admin/collections* {
5417 reverse_proxy 127.0.0.1:8001
5418 }
5419
5420 handle /admin/schemas* {
5421 reverse_proxy 127.0.0.1:8001
5422 }
5423
5424 handle /admin/identity* {
5425 reverse_proxy 127.0.0.1:8001
5426 }
5427
5428 handle /permissions/status* {
5429 reverse_proxy 127.0.0.1:8001
5430 }
5431
5432 handle /permissions/policy* {
5433 reverse_proxy 127.0.0.1:8001
5434 }
5435
5436 handle /daemon/status {
5437 reverse_proxy 127.0.0.1:8001
5438 }
5439
5440 handle /daemon/scheduler/tasks* {
5441 reverse_proxy 127.0.0.1:8001
5442 }
5443
5444 handle /daemon/queue/messages* {
5445 reverse_proxy 127.0.0.1:8001
5446 }
5447
5448 handle /events* {
5449 reverse_proxy 127.0.0.1:8001
5450 }
5451
5452 handle /packages* {
5453 reverse_proxy 127.0.0.1:8001
5454 }
5455
5456 handle /api/mcp {
5457 reverse_proxy 127.0.0.1:8001
5458 }
5459
5460 handle_path /static/* {
5461 root * /var/lib/dbbasic-static
5462 file_server
5463 header Cache-Control ""public, max-age=3600""
5464 }
5465
5466 handle /permissions/audit* {
5467 reverse_proxy 127.0.0.1:8001
5468 }
5469
5470 handle /dashboard {
5471 rewrite * /objects/system_dashboard
5472 reverse_proxy 127.0.0.1:8001
5473 }
5474
5475 handle /admin/write-probe {
5476 rewrite * /objects/system_write_probe
5477 reverse_proxy 127.0.0.1:8001
5478 }
5479
5480 handle /collections/dbbasic_probe/records* {
5481 reverse_proxy 127.0.0.1:8001
5482 }
5483
5484 handle /collections/dbbasic_probe/changes {
5485 reverse_proxy 127.0.0.1:8001
5486 }
5487
5488 handle /schemas/dbbasic_probe {
5489 reverse_proxy 127.0.0.1:8001
5490 }
5491
5492 handle /identity/session {
5493 reverse_proxy 127.0.0.1:8001
5494 }
5495
5496 # This exposes GET/DELETE current-session checks. POST password login works
5497 # when DBBASIC_ENABLE_PASSWORD_LOGIN=true; the gateway session-mint POST
5498 # remains disabled unless DBBASIC_ENABLE_SESSION_LOGIN=true and
5499 # DBBASIC_SESSION_LOGIN_TOKEN are set.
5500
5501 handle /objects/site_home* {
5502 reverse_proxy 127.0.0.1:8001
5503 }
5504
5505 handle {
5506 respond ""Not found"" 404
5507 }
5508}
5509```
5510
5511Validate and reload:
5512
5513```bash
5514sudo caddy fmt --overwrite /etc/caddy/Caddyfile
5515sudo caddy validate --config /etc/caddy/Caddyfile
5516sudo systemctl reload caddy
5517```
5518
5519Then test:
5520
5521```bash
5522curl https://dbbasic.example.com/
5523curl https://dbbasic.example.com/health
5524curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/status
5525curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/collections
5526curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/schemas
5527curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/admin/identity/users
5528curl -H ""Authorization: Token $DBBASIC_ADMIN_TOKEN"" https://dbbasic.example.com/daemon/status
5529curl https://dbbasic.example.com/objects/site_home
5530curl https://dbbasic.example.com/admin/write-probe
5531curl https://dbbasic.example.com/objects
5532```
5533
5534The first six should return responses from the object server. The root and
5535`site_home` responses may be JSON or HTML depending on the object. The full
5536`/objects` route should stay blocked by Caddy in this early staging mode.
5537
5538The `admin-write-probe` package may also expose the narrow
5539`/collections/dbbasic_probe/records*`, `/collections/dbbasic_probe/changes`,
5540and `/schemas/dbbasic_probe` routes. Those routes are still protected by the
5541server-side admin-token gate. They exist to prove collection create/update/
5542delete behavior, record changelog reads, and schema reads through the public
5543reverse proxy without exposing every collection or schema route. Do not expose
5544broad `/collections*`, `/schemas*`, `/objects*`, or source write routes on a
5545public staging server. Use `/admin/collections*` and `/admin/schemas*` for
5546operator inspection instead. Use `/admin/identity*` for read-only account, user,
5547and session inspection instead of exposing broad identity management routes.
5548
5549## Public Code Execution Controls
5550
5551In the early staging deployment, do not expose the full object server API
5552directly to the internet.
5553
5554Use all of these controls together:
5555
5556- keep uvicorn bound to `127.0.0.1`
5557- expose only specific routes through Caddy
5558- keep `DBBASIC_ENABLE_SOURCE_WRITES=false`
5559- do not add a public object-create route until permissions are enforced
5560- keep live object source and data out of the git checkout
5561
5562With that shape, outside users can only run the explicitly proxied objects. They
5563cannot list every object, read source, write source, roll back source, or create
5564new runnable code through the public hostname.
5565
5566## Backup
5567
5568The portable runtime backup set is:
5569
5570```text
5571/var/lib/dbbasic-object-server/objects
5572/var/lib/dbbasic-object-server/data/state
5573/var/lib/dbbasic-object-server/data/logs
5574/var/lib/dbbasic-object-server/data/versions
5575/var/lib/dbbasic-object-server/data/source_changes
5576/var/lib/dbbasic-object-server/data/schema_versions
5577/var/lib/dbbasic-object-server/data/record_changes
5578/var/lib/dbbasic-object-server/data/package_changes
5579/var/lib/dbbasic-object-server/data/file_changes
5580/var/lib/dbbasic-object-server/data/files
5581/var/lib/dbbasic-object-server/data/schemas
5582/var/lib/dbbasic-object-server/data/collections
5583```
5584
5585Create and verify a portable runtime backup with:
5586
5587```bash
5588python -m object_backup create \
5589 /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \
5590 --objects-dir /var/lib/dbbasic-object-server/objects \
5591 --data-dir /var/lib/dbbasic-object-server/data
5592
5593python -m object_backup verify \
5594 /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \
5595 --json
5596```
5597
5598Restore into clean directories first:
5599
5600```bash
5601python -m object_backup restore \
5602 /var/backups/dbbasic-object-server/runtime-YYYYMMDD-HHMMSS.tar.gz \
5603 --objects-dir /tmp/dbbasic-restore/objects \
5604 --data-dir /tmp/dbbasic-restore/data \
5605 --json
5606```
5607
5608The runtime archive deliberately excludes deployment secrets, environment files,
5609systemd service files, git history, virtualenvs, caches, lock files, temp files,
5610and ephemeral rate-limit files.
5611
5612The server environment file and systemd unit still need to be backed up by an
5613operator-controlled process:
5614
5615```text
5616/etc/dbbasic-object-server.env
5617/etc/systemd/system/dbbasic-object-server.service
5618```
5619
5620Keep those secret-aware backups separate from portable runtime archives. Before
5621calling this a production deployment, restore runtime backups on a second clean
5622VM and run the health/object checks again.
5623
5624See [`backup-restore.md`](/docs/backup-restore) for the runtime archive contract and restore rules.
5625
5626## Current Limits
5627
5628This deployment shape is for staging and dogfooding.
5629
5630Known limits:
5631
5632- the current direct Python loader is not a production sandbox
5633- source writes still use an explicit admin gate and are disabled until
5634 `DBBASIC_ENABLE_SOURCE_WRITES=true`
5635- admin-token gates can optionally accept active admin-role DBBASIC sessions
5636 with `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`
5637- route permission enforcement is available, but off until a deployment enables it
5638- real user/session auth still needs to replace trusted staging headers
5639- WebSocket/SSE runtime behavior is still design-stage
5640
5641That is still useful. A clean VM lets DBBASIC prove that install, restart,
5642object lookup, execution, state, logs, and versions work away from a developer
5643machine.
5644"
5645docker-deployment Docker / Coolify Deployment Deployment & Operations 710 How to deploy DBBASIC Object Server as a container with Docker Compose or Coolify, including the state/code volume split and package upgrade caveats. "This is the second deployment path for DBBASIC Object Server, alongside the
5646bare-VM systemd installer in [`docs/single-vm-deployment.md`](/docs/single-vm-deployment).
5647It packages the same server into a container image so it can run under
5648Docker Compose directly, or be deployed with [Coolify](https://coolify.io/).
5649
5650The goal is the same as the bare-VM path: stay boring, stay conservative on
5651first boot, and keep the object edit/run/inspect loop working the same way.
5652This page does not repeat what is not Docker-specific -- permissions,
5653source-write gates, package installs, first-auth setup, and traffic limits
5654are covered in [`docs/single-vm-deployment.md`](/docs/single-vm-deployment) and
5655the root `README.md`; read those too.
5656
5657## What Ships in the Image vs. What Must Be a Volume
5658
5659This is the concept the rest of this page assumes.
5660
5661The image is stateless. It contains the server code and the installable
5662package fixtures under `packages/` -- both of those are tracked in git and
5663rebuilt into the image on every `docker build`.
5664
5665Everything the server writes at runtime is state, not code, and must live on
5666a mounted volume outside the image:
5667
5668- object source under `DBBASIC_OBJECTS_DIR`
5669- TSV-backed records, state, logs, versions, schemas, and changelogs under
5670 `DBBASIC_DATA_DIR`
5671- identity (accounts, users, sessions, credential hashes)
5672- the permission policy and audit log
5673- backups under `DBBASIC_BACKUPS_DIR`
5674- the auto-generated admin token (see ""First Boot"" below)
5675
5676`Dockerfile` sets `DBBASIC_OBJECTS_DIR=/data/objects` and
5677`DBBASIC_DATA_DIR=/data/state` as the image's default paths, and
5678`docker-compose.yml` mounts two named volumes at exactly those paths. This
5679mirrors the same separation `docs/single-vm-deployment.md` keeps between
5680`/opt/dbbasic-object-server` (the git checkout) and
5681`/var/lib/dbbasic-object-server` (runtime data) on the bare-VM path -- code
5682and state never share a lifecycle.
5683
5684That split is what makes the two halves of the deploy story work
5685independently (see ""Deploy Cadence"" below).
5686
5687## Quickstart (Docker Compose)
5688
5689From a checkout of this repository:
5690
5691```bash
5692cp .env.example .env
5693# edit .env if you want to change any of the conservative defaults
5694docker compose up --build
5695```
5696
5697This builds the image, starts the `object-server` service bound to
5698`0.0.0.0:8001` inside the container, and publishes it on `localhost:8001`.
5699Watch the logs for the first-boot admin token message (see ""First Boot""
5700below), then:
5701
5702```bash
5703curl http://localhost:8001/health
5704```
5705
5706Object source and runtime data persist on the `objects` and `data` named
5707volumes declared in `docker-compose.yml`, so `docker compose up --build`
5708after a code change keeps existing objects, records, and identity intact.
5709
5710`docker-compose.yml` includes a commented-out, clearly-labeled optional
5711`object-daemon` service for `object_daemon.py` (the scheduler/queue/event
5712background worker). It is not required for the app to run; uncomment it
5713only if you need scheduled tasks, queue processing, or webhook event
5714delivery running in the background.
5715
5716## Coolify Deployment
5717
5718Coolify can deploy this repository two ways:
5719
5720- **Dockerfile application** -- Coolify detects the `Dockerfile` at the repo
5721 root and builds/runs it directly.
5722- **Docker Compose resource** -- Coolify deploys `docker-compose.yml`
5723 directly.
5724
5725The Compose path is the cleaner one for this project, because it declares
5726the two persistent volumes explicitly instead of leaving them implicit.
5727Prefer it unless you have a reason to use the plain Dockerfile path.
5728
5729Either way, in Coolify:
5730
57311. **Create the application** from this repository, pointing at the
5732 `Dockerfile` or `docker-compose.yml` per the choice above.
57332. **Configure persistent storage.** In Coolify's Storage / Persistent
5734 Storage settings, add two volumes (or reuse the ones declared in
5735 `docker-compose.yml` if you deployed it as a Compose resource) mapped to
5736 the same container paths the image expects:
5737
5738 ```text
5739 /data/objects
5740 /data/state
5741 ```
5742
5743 These must match `DBBASIC_OBJECTS_DIR` and `DBBASIC_DATA_DIR`. If you
5744 change one, change the other.
5745
57463. **Set environment variables** through Coolify's Environment Variables UI,
5747 using `.env.example` as the list of what to set and why. At minimum,
5748 decide whether to set `DBBASIC_ADMIN_TOKEN` explicitly (recommended once
5749 this is a real deployment) or leave it unset for the container to
5750 generate one on first boot.
57514. **Enable health-check-based deploys.** The image's `HEALTHCHECK`
5752 instruction (and `docker-compose.yml`'s `healthcheck:` block) call
5753 `GET /health`. Point Coolify's deployment health check at the same
5754 endpoint so it waits for a healthy container before cutting traffic over
5755 on redeploy, giving zero-downtime deploys for server-code changes.
57565. **Put a domain in front of it.** Coolify's own proxy handles TLS and
5757 hostname routing to the container's published port, the same way Caddy
5758 sits in front of, not inside, the systemd unit on the bare-VM path. This
5759 repository's docs use `object.dbbasic.com` as the example public domain;
5760 use your own.
5761
5762## Deploy Cadence
5763
5764There are two independent kinds of change here, and only one of them needs
5765an image rebuild:
5766
5767- **Server code changes** (anything in this git repository) require
5768 rebuilding and redeploying the image. Under Coolify this is a normal
5769 git-push-to-deploy flow: push to the branch Coolify watches, it rebuilds
5770 the image, runs the health check, and cuts over.
5771- **Object, schema, and record edits made live through the admin HTTP API**
5772 (the same API described in the root `README.md` and
5773 `docs/single-vm-deployment.md`) land directly on the persistent volumes.
5774 They take effect on the next request, with no image rebuild and no
5775 redeploy, exactly like the bare-VM deployment's edit/run/inspect loop.
5776
5777This is the reason the volume split above exists: redeploying the image
5778never touches `/data/objects` or `/data/state`, so in-flight object work
5779survives ordinary code deploys.
5780
5781**A `packages/<name>/` change (in git) needs BOTH steps, not either/or.**
5782An image rebuild brings the new `packages/` source into the image, but
5783`/data/objects` and `/data/state/schemas` are the persistent volumes the
5784running container actually serves from -- the same separate-copy split
5785`docs/single-vm-deployment.md`'s Upgrade section documents for the bare-VM
5786case, and the entrypoint does not auto-sync it. After the new image is
5787live, still call `POST /packages/{id}/install` (`{""allow_replace"": true}`,
5788admin-token gated) for the changed package, confirm
5789`install.objects[].status` says `""updated""`, and expect the object module
5790cache to need the container's next natural restart (or force one) to pick
5791it up. Skipping this step is a silent no-op exactly like the bare-VM case:
5792new image, tests green, health check passes, and the container keeps
5793serving the old package code with no error anywhere.
5794
5795## First Boot: the Admin Token
5796
5797If `DBBASIC_ADMIN_TOKEN` is not set, `scripts/docker-entrypoint.sh` (the
5798image's container entrypoint) generates one, writes it to a file under the
5799persistent data volume so it survives restarts and rebuilds, and prints it
5800once, clearly, to the container's logs:
5801
5802```bash
5803docker compose logs object-server | grep -A4 ""generated a new admin token""
5804```
5805
5806or, in Coolify, check the application's log viewer after first deploy.
5807
5808The token is never baked into the image and never written anywhere in the
5809git checkout -- only into the runtime-only data volume. It is not
5810regenerated or overwritten on later boots.
5811
5812Once you have confirmed the deployment works, set `DBBASIC_ADMIN_TOKEN`
5813explicitly (in `.env` for Compose, or in Coolify's environment variables
5814UI) rather than relying on the generated value. That gives you a token you
5815control and can rotate deliberately, and it is required if you ever need to
5816recreate the data volume from a backup.
5817
5818## See Also
5819
5820For everything not specific to Docker -- write gates, package installs,
5821permission audit/enforcement, traffic limits, backups, and bootstrapping the
5822first admin user and password login -- see:
5823
5824- [`docs/single-vm-deployment.md`](/docs/single-vm-deployment)
5825- the root `README.md`, especially ""First Auth Setup"" and
5826 ""Public Repository Safety""
5827
5828## Security Notes
5829
5830These mirror the root README's ""Public Repository Safety"" checklist:
5831
5832- Never commit a real `.env`. Only `.env.example` (placeholder values and
5833 comments) belongs in git; `.env` is git-ignored.
5834- Never bake a real admin token, password hash, or session secret into an
5835 image layer. The entrypoint's generated token is written only to the
5836 mounted data volume, not to any path that gets committed or that survives
5837 outside that volume.
5838- Treat the persistent data volume as containing credentials-equivalent
5839 data: it holds the admin token file, identity credential hashes, and
5840 session records. Back it up and restrict access to it the same way you
5841 would `/var/lib/dbbasic-object-server` on the bare-VM path.
5842- Do not commit real IPs, hostnames, or deployment-specific details into
5843 this repository, including into Coolify configuration you might otherwise
5844 be tempted to paste into a doc or issue.
5845"
5846upgrade-and-customization Upgrades and Customization Deployment & Operations 720 The doctrine for letting a package upgrade in place while an operator's data and customizations survive, via provenance baselines, reconcile-not-replace, overlays, additive schemas, hooks, and feature flags. "This is the doctrine for the hardest problem in packaged app systems: **how a third-party package upgrades in place while the operator's data AND customizations survive**. Rails and Django never solved this as a platform property — apps/engines existed, but customizing meant forking and upgrading meant being your own integrator. The systems that solved it (Salesforce managed packages, Odoo modules, Debian conffiles, WordPress child themes) converged on the same handful of ideas. This document adapts them to DBBASIC so any future contributor — human or AI — builds *with* the upgrade system instead of fighting it.
5847
5848## The three layers (never entangle them)
5849
5850Every installed app is three independent layers:
5851
58521. **Package baseline** — what the package shipped: objects, schemas, permission fragments, seed. Versioned, replaceable, owned by the package.
58532. **Local customization** — what this instance changed on top: edited source, added fields, extra rules, user code. Owned by the operator.
58543. **Data** — the records in `data/collections/<name>/records.tsv`. Owned by the users. Lives entirely outside the package.
5855
5856An upgrade replaces layer 1. It must never silently destroy layers 2 or 3. Layer 3 is solved: seed is install-once (see the package authoring guide). This document is about layer 2.
5857
5858## Rule 0: provenance baselines
5859
5860Nothing else works without this. On install/upgrade, stamp every shipped artifact with its baseline:
5861
5862```json
5863{""package"": ""app-articles"", ""version"": ""0.2.0"", ""sha256"": ""<hash of shipped source>""}
5864```
5865
5866stored per object and per schema (e.g. `data/package_baselines/<pkg>.json`). Then ""is this customized?"" is a computable question: hash the live source and compare it to the baseline. Without the stamp you cannot distinguish a human's edit from the shipped file, and every upgrade is a coin flip.
5867
5868## Rule 1: reconcile, don't replace
5869
5870On upgrade, per object/schema, three-way compare `old_baseline` vs `live` vs `new_shipped` — exactly `git merge` / dpkg conffile semantics:
5871
5872| live vs baseline | shipped changed? | action |
5873|---|---|---|
5874| pristine | — | fast-forward to new version, update baseline (silent) |
5875| customized | no | keep the customization (silent) |
5876| customized | yes, hunks don't overlap | three-way merge, apply, flag ""merged"" |
5877| customized | yes, conflict | **park it** — write a pending-reconcile record holding both versions, leave live untouched, surface for a human decision (keep mine / take theirs / view diff) |
5878
5879`allow_replace` means ""fast-forward the pristine ones."" Discarding a customization requires a separate explicit force. The existing version stores (`object_versions`, `object_schema_versions`) are the undo; the install restore point is the rollback. A ""reconcile inbox"" (admin UI screen + CLI list) holds parked conflicts — upgrades never block on them.
5880
5881## Rule 2: customize by overlay, not by editing core
5882
5883The reconcile engine is the safety net. The *preferred* path makes conflicts structurally impossible: customizations live in different files than the package's, so upgrades and edits never touch the same bytes.
5884
5885- **System vs user objects.** Package-shipped objects are *system* objects (baseline-stamped, upgrade-managed). Operator-created objects are *user* objects (never touched by any upgrade). The namespace should make this visible: editing a system object warns ""this came from app-articles 0.2.0 — override instead?""
5886- **Override objects.** A user object can *shadow* a system object by id: the resolver checks the override root first (`objects/overrides/…` or an `overrides: {site_articles: my_articles}` map). The shipped object stays pristine and freely upgradeable; the override persists across upgrades. This is Odoo's `_inherit` / WordPress child themes done with plain files.
5887- **Where an install writes.** A NEW package object is written to the base/system root, never the override root — `object_namespace.get_base_object_roots` is explicit that this is ""the root packages install into and reconcile against ... the pristine, upgradeable copy"". An object that ALREADY exists is rewritten wherever it already lives, so an operator's override survives every upgrade. Both halves matter: installing into the override root would make package content indistinguishable from customization, and it also silently disables event handlers (an override is invisible to the handler index — see Rule 4).
5888- **Extension points.** Objects should invite extension instead of forking: named hooks (see Rule 4) and small overridable pieces rather than one monolith page.
5889
5890## Rule 3: data fields that survive schema upgrades
5891
5892Schemas are the friendly layer because changes are mostly **additive**, and records are read *through* the schema — an old row simply reads empty for a new field (proven by the `created_at` rollout).
5893
5894- **Field union on upgrade.** Merge shipped schema changes with locally added fields rather than replacing the field list. Conflict = same field name with a different type/enum — park it like a source conflict. A locally added `priority` field must survive an app-notes upgrade.
5895- **Namespace local fields** (`x_` prefix by convention, enforced gently) so a package can never ship a field that collides with an operator's.
5896- **Extension data (`extra`).** Give every collection an optional JSON overflow field: user-defined per-record data that no schema upgrade can ever collide with, because it isn't in the schema's flat namespace at all. Schemas may declare typed *views* into it (`{""name"": ""x_mood"", ""store"": ""extra""}`) so forms/lists render extension fields with zero migration. TSV stays one column; the column holds JSON.
5897- **Migrations** (the currently-stubbed manifest section) are for the rare non-additive change: a shipped script that backfills/transforms records, run once, marker-tracked, restore-point-first. Until implemented, packages must keep schema changes additive.
5898
5899## Rule 4: hooks — behavior customization without forking
5900
5901Let operators attach code to the system's existing event stream instead of editing package objects. The substrate exists: every record write already publishes `collection.record.created/updated/deleted` (it drives the websocket layer).
5902
5903- **Event handler objects.** A user object subscribes declaratively: `HANDLES = [""notes.record.created""]`; the server invokes it on the event (same sandboxed execution as any object). Custom side effects — notify, enrich, sync — with zero edits to the shipped app. Handlers are user objects, so upgrades never touch them.
5904- **Page hooks.** Generated pages expose named slots (`before_list`, `row_actions`) that resolve to user objects if present. The generative renderer makes this cheap: one hook added to the shared list-page generator appears in every app at once.
5905
5906## Rule 5: user prefs and feature flags
5907
5908A `user_prefs` collection (`user_id`, `key`, `value`, owner-scoped rows) — schema-driven like everything else — gives:
5909
5910- **Per-user settings** for any app (theme, defaults, layout) with zero new endpoints; the generators can read/write it directly.
5911- **Feature flags.** Packages ship features dark: `features.json` → `{""kanban_view"": {""default"": ""off""}}`. Instance-level toggles live in a `feature_flags` collection; per-user overrides in `user_prefs`. Pages check flags via one shared helper. This decouples *shipping* code from *enabling* it — upgrades can carry big features safely off, canary them per user, and roll back by flipping a flag rather than reinstalling. It also shrinks customization pressure: many ""edits"" are really ""turn this on/off,"" which should be config, not forked source.
5912
5913## What future coders (AI or human) must do
5914
5915When touching packages or upgrade code:
5916
59171. Never write an upgrade path that overwrites a file without checking its baseline hash first. If the baseline system doesn't exist yet, build it before the feature that needs it.
59182. Never block an install on something preservable — preserve and skip (like seed), or park and surface (like conflicts). Blocking teaches operators to bypass the installer, which is how systems rot.
59193. Keep schema changes additive; put anything non-additive in a migration.
59204. New customization mechanisms must be overlay-shaped: separate file, separate collection, separate namespace — never ""edit the shipped thing.""
59215. Data lives outside the package, keyed by collection, always.
5922
5923## Build order
5924
59251. **Provenance stamps** (small; unblocks everything) — record baselines at install, expose `customized: true/false` per object in package status.
59262. **Reconcile-on-upgrade** — the table above, plus pending-reconcile records and a CLI list; the admin UI renders the reconcile inbox.
59273. **Override objects** — resolver-level shadowing; the conflict-free path.
59284. **Schema field-union + `extra` extension data.**
59295. **Event handler objects** (`HANDLES`), then page hooks.
59306. **`user_prefs` + `feature_flags` collections** and the shared flag helper.
5931
5932Each phase is independently shippable and independently valuable; together they make ""upgrade the app, keep your data and your customizations"" a platform guarantee — the property that separated the systems that lasted (Salesforce, Odoo, Debian, WordPress) from the frameworks that made every operator their own integrator.
5933"
5934http-api-contract HTTP API Contract API Reference 800 Reference for the HTTP shape the DBBASIC Object Server preserves for clients: auth, correlation IDs, permissions, identity, collections, objects, packages, schemas, and the compatibility guarantees that hold across releases. "This is the HTTP shape the DBBASIC Object Server preserves as its internals evolve.
5935
5936The goal is compatibility. Existing clients, MCP tools, Scroll, scripts, and
5937automation should not need a new API shape just because the internals become
5938cleaner.
5939
5940Examples throughout this guide use a local placeholder host:
5941
5942```text
5943http://127.0.0.1:8001
5944```
5945
5946Replace it with your own deployment's address — the shape of the request and
5947response is what matters, not the host.
5948
5949## Client Defaults
5950
5951Existing clients expect JSON by default and commonly send:
5952
5953```http
5954Accept: application/json
5955Authorization: Token <token>
5956```
5957
5958Authentication is required for source, state, logs, versions, object creation,
5959source updates, rollback, and destructive deletes. Basic object execution may be
5960public or authenticated depending on server policy and the object being called.
5961
5962The current public ASGI slice enforces that sensitive read surface with the
5963temporary admin token from `DBBASIC_ADMIN_TOKEN` by default. A deployment can
5964opt in to `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true` so active DBBASIC sessions
5965with an admin role can pass the same gate. Source update and rollback also
5966require `DBBASIC_ENABLE_SOURCE_WRITES=true`. The real role, object, and row
5967permission system still needs to replace this temporary admin boundary before
5968general use.
5969
5970## Correlation IDs
5971
5972Every HTTP response includes:
5973
5974```http
5975X-DBBASIC-Correlation-ID: <uuid-v4>
5976```
5977
5978Clients may send their own UUIDv4 correlation ID:
5979
5980```http
5981X-DBBASIC-Correlation-ID: 123e4567-e89b-42d3-a456-426614174000
5982```
5983
5984Missing, empty, or invalid values are replaced with a server-generated UUIDv4.
5985The response header always contains the accepted value. Source update responses,
5986rollback responses, and execution error bodies also include `correlation_id`.
5987
5988The same ID is written into source version metadata, source-change entries,
5989object-owned runtime logs, execution error logs, and permission audit entries
5990when those records are created inside the request. Scroll and AI tools can use
5991that ID to connect one user action to the source version, logs, permission
5992decision, and runtime error it produced.
5993
5994## Resource IDs
5995
5996New DBBASIC-facing URL and API resources should use UUIDv4 IDs by default:
5997
5998- record IDs
5999- account and user IDs
6000- session/action/change IDs
6001- package install and restore IDs
6002- external links and imported resource handles
6003
6004Imported systems may keep legacy compatibility IDs. For example, a table with
6005integer primary keys can be imported with its old ID preserved in a
6006compatibility column, while the DBBASIC route-facing `id` should be UUIDv4 for
6007new records. This keeps URLs non-enumerable, makes exports easier to merge, and
6008avoids coupling object packages to one database sequence.
6009
6010The current identity registry generates UUIDv4 account IDs, user IDs, and
6011session IDs when the server creates them. It still accepts existing string
6012account/user IDs and legacy `sess_...` session rows so old clients, Scroll
6013prototypes, and migration tools do not break while the public runtime is being
6014extracted.
6015
6016## Health
6017
6018Plain health is a public liveness check:
6019
6020```http
6021GET /health
6022```
6023
6024Response:
6025
6026```json
6027{
6028 ""status"": ""ok""
6029}
6030```
6031
6032Detailed health is for operators, Scroll, and future cluster routing:
6033
6034```http
6035GET /health?capacity=true
6036Authorization: Token <token>
6037```
6038
6039Response:
6040
6041```json
6042{
6043 ""status"": ""ok"",
6044 ""version"": ""0.0.1"",
6045 ""station_id"": ""standalone"",
6046 ""uptime"": ""2h 10m"",
6047 ""requests"": 120,
6048 ""errors"": 0,
6049 ""rps"": 1.4,
6050 ""objects"": {
6051 ""count"": 4
6052 },
6053 ""capacity"": {
6054 ""requests"": {
6055 ""in_flight"": 1,
6056 ""max"": 64,
6057 ""available"": 63,
6058 ""limited"": true
6059 },
6060 ""object_executions"": {
6061 ""in_flight"": 0,
6062 ""max"": 8,
6063 ""available"": 8,
6064 ""limited"": true
6065 }
6066 },
6067 ""checks"": {
6068 ""storage"": {
6069 ""status"": ""ok""
6070 }
6071 }
6072}
6073```
6074
6075`GET /health?metrics=true` returns the same detailed health payload plus a
6076`metrics` object with request counts, status counts, response timing, top paths,
6077and recent HTTP errors. This preserves the older dashboard/Scroll direction
6078without exposing those details through the public liveness route.
6079
6080## Admin Status
6081
6082Admin status is the compact operator snapshot Scroll and staging dashboards can
6083use instead of stitching together many routes on first load:
6084
6085```http
6086GET /admin/status
6087Authorization: Token <token>
6088```
6089
6090Response:
6091
6092```json
6093{
6094 ""status"": ""ok"",
6095 ""timestamp"": ""2026-01-01T00:00:00+00:00"",
6096 ""version"": ""0.0.1"",
6097 ""station_id"": ""standalone"",
6098 ""health"": {
6099 ""status"": ""ok"",
6100 ""metrics"": {}
6101 },
6102 ""inventory"": {
6103 ""objects"": 4,
6104 ""collections"": 2,
6105 ""schemas"": 2,
6106 ""packages"": 3
6107 },
6108 ""capabilities"": {
6109 ""source_writes"": {
6110 ""enabled"": false,
6111 ""env"": ""DBBASIC_ENABLE_SOURCE_WRITES""
6112 },
6113 ""file_writes"": {
6114 ""enabled"": false,
6115 ""env"": ""DBBASIC_ENABLE_FILE_WRITES"",
6116 ""max_bytes"": 1048576,
6117 ""max_bytes_env"": ""DBBASIC_MAX_OBJECT_FILE_BYTES""
6118 },
6119 ""package_installs"": {
6120 ""enabled"": false,
6121 ""env"": ""DBBASIC_ENABLE_PACKAGE_INSTALLS""
6122 },
6123 ""permission_enforcement"": {
6124 ""enabled"": false,
6125 ""requested"": false,
6126 ""blocked"": true,
6127 ""env"": ""DBBASIC_ENABLE_PERMISSION_ENFORCEMENT""
6128 },
6129 ""identity"": {
6130 ""trusted_headers_enabled"": false,
6131 ""require_known_identity_users"": true,
6132 ""session_login_enabled"": false,
6133 ""session_login_token_configured"": false,
6134 ""session_admin_gates_enabled"": false,
6135 ""session_admin_gates_env"": ""DBBASIC_ENABLE_SESSION_ADMIN_GATES""
6136 }
6137 },
6138 ""packages"": [
6139 {
6140 ""id"": ""system-dashboard"",
6141 ""name"": ""System Dashboard"",
6142 ""version"": ""0.1.0"",
6143 ""status"": ""installed"",
6144 ""install"": {
6145 ""installed_count"": 1,
6146 ""installable_count"": 1,
6147 ""safe_to_install"": true,
6148 ""install_enabled"": false,
6149 ""warnings"": []
6150 },
6151 ""changes"": {
6152 ""total"": 1,
6153 ""latest"": {
6154 ""action"": ""installed""
6155 }
6156 }
6157 }
6158 ],
6159 ""permissions"": {
6160 ""enforcement_enabled"": false,
6161 ""audit_enabled"": false,
6162 ""readiness"": {
6163 ""ready"": false
6164 },
6165 ""warnings"": []
6166 }
6167}
6168```
6169
6170If the server is degraded, the route may return `503` with the same response
6171shape and `""status"": ""degraded""`. This route is admin-token gated because it
6172contains configuration, package, and capacity details.
6173
6174## Daemon Status
6175
6176Daemon status is the read-only operator snapshot for the scheduler, queue,
6177event delivery, and cleanup primitives:
6178
6179```http
6180GET /daemon/status
6181Authorization: Token <token>
6182```
6183
6184Response:
6185
6186```json
6187{
6188 ""status"": ""ok"",
6189 ""timestamp"": ""2026-01-01T00:00:00+00:00"",
6190 ""daemon"": {
6191 ""mode"": ""polling"",
6192 ""croniter_available"": true,
6193 ""object_roots"": {
6194 ""count"": 1
6195 },
6196 ""triggers"": {
6197 ""scheduler"": {
6198 ""object_id"": ""scheduler"",
6199 ""source_present"": true
6200 },
6201 ""queue"": {
6202 ""object_id"": ""queue"",
6203 ""source_present"": true
6204 },
6205 ""events"": {
6206 ""object_id"": ""events"",
6207 ""source_present"": true
6208 }
6209 }
6210 },
6211 ""scheduler"": {
6212 ""object_id"": ""scheduler"",
6213 ""source_present"": true,
6214 ""tasks"": {
6215 ""total"": 2,
6216 ""active"": 1,
6217 ""due"": 1,
6218 ""future"": 0,
6219 ""invalid"": 0,
6220 ""next_run"": 1767225600,
6221 ""next_run_iso"": ""2026-01-01T00:00:00+00:00""
6222 }
6223 },
6224 ""queue"": {
6225 ""object_id"": ""queue"",
6226 ""source_present"": true,
6227 ""messages"": {
6228 ""total"": 3,
6229 ""pending_visible"": 1,
6230 ""pending_delayed"": 1,
6231 ""expired_pending"": 0,
6232 ""invalid"": 0
6233 }
6234 },
6235 ""events"": {
6236 ""object_id"": ""events"",
6237 ""source_present"": true,
6238 ""events"": {
6239 ""total"": 10,
6240 ""latest"": {
6241 ""id"": ""evt_123"",
6242 ""event_type"": ""collection.record.created"",
6243 ""timestamp"": 1767225600
6244 }
6245 },
6246 ""subscriptions"": {
6247 ""total"": 2,
6248 ""by_delivery_status"": {
6249 ""ok"": 1,
6250 ""failed"": 1
6251 },
6252 ""pending_deliveries"": 1
6253 }
6254 },
6255 ""cleanup"": {
6256 ""event_retention"": {
6257 ""keep_count"": 1000,
6258 ""keep_seconds"": 604800
6259 },
6260 ""rate_limit_files"": 0
6261 }
6262}
6263```
6264
6265This endpoint does not execute daemon work or expose event payloads. It reads
6266the same TSV state that the daemon uses so Scroll can show due scheduler tasks,
6267visible/delayed queue messages, failed subscription delivery, event retention,
6268and cleanup pressure without requiring a separate queue dashboard service.
6269
6270## Daemon Scheduler And Queue Controls
6271
6272Scheduler and queue controls are admin-token gated. They operate on the same
6273`data/state/scheduler/state.tsv` and `data/state/queue/state.tsv` files that
6274`object_daemon.py` already polls, so the HTTP API can manage work without
6275introducing Celery, Redis, Flower, or a second queue service.
6276
6277These routes redact task/message payloads by default. Pass
6278`include_payload=true` only on trusted operator screens.
6279
6280```http
6281GET /daemon/scheduler/tasks?status=active&limit=100&offset=0
6282POST /daemon/scheduler/tasks
6283PATCH /daemon/scheduler/tasks/{task_id}
6284DELETE /daemon/scheduler/tasks/{task_id}
6285
6286GET /daemon/queue/messages?status=pending&queue_name=default&limit=100&offset=0
6287POST /daemon/queue/messages
6288PATCH /daemon/queue/messages/{message_id}
6289DELETE /daemon/queue/messages/{message_id}
6290Authorization: Token <token>
6291```
6292
6293Create one scheduled task:
6294
6295```json
6296{
6297 ""object_id"": ""system_dashboard"",
6298 ""method"": ""POST"",
6299 ""type"": ""onetime"",
6300 ""schedule"": ""2026-07-01T12:00:00Z"",
6301 ""payload"": {
6302 ""refresh"": true
6303 }
6304}
6305```
6306
6307Response:
6308
6309```json
6310{
6311 ""status"": ""ok"",
6312 ""task"": {
6313 ""id"": ""018ff3f4-5f80-4df0-8e25-bcf6ad6fda01"",
6314 ""object_id"": ""system_dashboard"",
6315 ""method"": ""POST"",
6316 ""type"": ""onetime"",
6317 ""schedule"": ""2026-07-01T12:00:00Z"",
6318 ""status"": ""active"",
6319 ""next_run"": 1782907200,
6320 ""payload_present"": true
6321 }
6322}
6323```
6324
6325Patch examples:
6326
6327```json
6328{""status"": ""paused""}
6329```
6330
6331```json
6332{""next_run"": ""2026-07-01T13:00:00Z""}
6333```
6334
6335Enqueue one message:
6336
6337```json
6338{
6339 ""object_id"": ""system_dashboard"",
6340 ""method"": ""POST"",
6341 ""queue_name"": ""default"",
6342 ""priority_level"": 5,
6343 ""payload"": {
6344 ""refresh"": true
6345 }
6346}
6347```
6348
6349Response:
6350
6351```json
6352{
6353 ""status"": ""ok"",
6354 ""message"": {
6355 ""id"": ""2de66c27-149b-475b-a6f0-0c08a8d850f5"",
6356 ""queue_name"": ""default"",
6357 ""message"": {
6358 ""object_id"": ""system_dashboard"",
6359 ""method"": ""POST"",
6360 ""payload_present"": true
6361 },
6362 ""priority_level"": 5,
6363 ""status"": ""pending""
6364 }
6365}
6366```
6367
6368Queue patch actions:
6369
6370```json
6371{""action"": ""cancel""}
6372```
6373
6374```json
6375{""action"": ""retry""}
6376```
6377
6378These controls are for trusted operators. They do not change the rule that
6379untrusted object execution still needs a stronger worker boundary, quotas, and
6380permission enforcement before public signup is safe.
6381
6382## Rate Limits
6383
6384When `DBBASIC_RATE_LIMIT_REQUESTS` is set above zero, the server rate-limits
6385non-health traffic before reading request bodies or running object code. Plain
6386`GET /health` is excluded so load balancers and uptime checks keep working under
6387pressure.
6388
6389The current public server uses a valid admin token as the rate-limit identity
6390for admin requests. Other requests use the client IP address. Proxy headers such
6391as `X-Forwarded-For` are ignored unless `DBBASIC_RATE_LIMIT_TRUST_PROXY_HEADERS`
6392is enabled on a server that is only reachable through a trusted reverse proxy.
6393
6394Limit response:
6395
6396```http
6397429 Too Many Requests
6398Retry-After: 30
6399```
6400
6401```json
6402{
6403 ""status"": ""error"",
6404 ""error"": ""Rate limit exceeded"",
6405 ""retry_after"": 30,
6406 ""limit"": 1000,
6407 ""window_seconds"": 60
6408}
6409```
6410
6411## Identity
6412
6413Scroll and gateway code can inspect the subject the object server will use for
6414permission checks:
6415
6416```http
6417GET /identity
6418Authorization: Token <token>
6419```
6420
6421Response:
6422
6423```json
6424{
6425 ""status"": ""ok"",
6426 ""subject"": {
6427 ""user_id"": ""7"",
6428 ""account_id"": ""acme"",
6429 ""roles"": [""sales""],
6430 ""subscriptions"": [""pro""],
6431 ""authenticated"": true
6432 },
6433 ""auth"": {
6434 ""method"": ""trusted_headers"",
6435 ""trusted_headers_enabled"": true,
6436 ""trusted_headers_present"": true
6437 },
6438 ""permissions"": {
6439 ""enforcement_enabled"": true,
6440 ""enforcement_requested"": true,
6441 ""enforcement_blocked"": false,
6442 ""audit_enabled"": true
6443 }
6444}
6445```
6446
6447Current auth methods are:
6448
6449- `anonymous` - no trusted identity was accepted.
6450- `admin_token` - `Authorization: Token <DBBASIC_ADMIN_TOKEN>` or Bearer matched
6451 the server admin token.
6452- `session_token` - `Authorization: Token <session-token>` or Bearer matched an
6453 active file-backed DBBASIC identity session.
6454- `trusted_headers` - `DBBASIC_PERMISSION_TRUST_HEADERS=true` and at least one
6455 `X-DBBASIC-*` identity header was present.
6456
6457Trusted identity headers are ignored unless explicitly enabled. This keeps the
6458current staging server safe while giving future login/session gateways and
6459Scroll one stable endpoint for the active account, user, roles, and
6460subscriptions.
6461
6462### Identity Accounts And Users
6463
6464The server has a small file-backed identity registry so sessions can be minted
6465from known users instead of every client inventing its own subject payload. These
6466routes are admin-gated.
6467
6468Accounts are stored under `data/identity/accounts.tsv`:
6469
6470```http
6471GET /identity/accounts
6472Authorization: Token <admin-token>
6473```
6474
6475```http
6476POST /identity/accounts
6477Authorization: Token <admin-token>
6478Content-Type: application/json
6479```
6480
6481```json
6482{
6483 ""account_id"": ""acme"",
6484 ""name"": ""Acme Corp"",
6485 ""subscriptions"": [""pro""]
6486}
6487```
6488
6489Users are stored under `data/identity/users.tsv`:
6490
6491```http
6492GET /identity/users
6493Authorization: Token <admin-token>
6494```
6495
6496```http
6497GET /identity/users?account_id=acme
6498Authorization: Token <admin-token>
6499```
6500
6501```http
6502POST /identity/users
6503Authorization: Token <admin-token>
6504Content-Type: application/json
6505```
6506
6507```json
6508{
6509 ""user_id"": ""u_7"",
6510 ""account_id"": ""acme"",
6511 ""email"": ""alice@example.com"",
6512 ""display_name"": ""Alice"",
6513 ""roles"": [""sales""],
6514 ""subscriptions"": [""team""]
6515}
6516```
6517
6518Single-record reads are:
6519
6520```http
6521GET /identity/accounts/acme
6522GET /identity/users/u_7
6523```
6524
6525`status` can be `active` or `disabled`. Disabled accounts and users cannot mint
6526new sessions.
6527
6528### User Passwords
6529
6530Users can have one password credential for browser/password login. Hashes are
6531stored separately from user records under `data/identity/credentials.tsv`
6532(scrypt, owner-only file permissions), so user payloads never include
6533credential material. Both routes are admin-token gated, and the admin alias
6534form works through the reverse proxy for Scroll:
6535
6536```http
6537POST /identity/users/{user_id}/password
6538POST /admin/identity/users/{user_id}/password
6539Authorization: Token <admin-token>
6540Content-Type: application/json
6541```
6542
6543```json
6544{""password"": ""at least 8 characters""}
6545```
6546
6547Response returns only `user_id`, `operation` (`created` or `replaced`), and
6548`updated_at` — never the password or hash. `DELETE` on the same paths removes
6549the credential:
6550
6551```http
6552DELETE /identity/users/{user_id}/password
6553DELETE /admin/identity/users/{user_id}/password
6554Authorization: Token <admin-token>
6555```
6556
6557Unknown users return `404`. Passwords must be 8 to 1024 characters.
6558
6559### Identity Sessions
6560
6561The server can mint scoped subject tokens for Scroll, gateways, or controlled
6562apps. Sessions are stored under `data/identity/sessions.tsv`. Only token hashes
6563are written to disk; the raw token is returned once at creation time.
6564
6565Create a session:
6566
6567```http
6568POST /identity/sessions
6569Authorization: Token <admin-token>
6570Content-Type: application/json
6571```
6572
6573```json
6574{
6575 ""user_id"": ""7"",
6576 ""account_id"": ""acme"",
6577 ""roles"": [""sales""],
6578 ""subscriptions"": [""pro""],
6579 ""label"": ""scroll desktop"",
6580 ""ttl_seconds"": 86400
6581}
6582```
6583
6584Response:
6585
6586```json
6587{
6588 ""status"": ""ok"",
6589 ""session"": {
6590 ""session_id"": ""9d4fe8bd-6859-43ef-8f83-f7063c54f7bc"",
6591 ""user_id"": ""7"",
6592 ""account_id"": ""acme"",
6593 ""roles"": [""sales""],
6594 ""subscriptions"": [""pro""],
6595 ""label"": ""scroll desktop"",
6596 ""created_at"": ""2026-06-30T18:00:00Z"",
6597 ""expires_at"": ""2026-07-01T18:00:00Z"",
6598 ""revoked_at"": null,
6599 ""active"": true
6600 },
6601 ""token"": ""returned-once""
6602}
6603```
6604
6605If `user_id` already exists in `data/identity/users.tsv`, omitted `account_id`,
6606`roles`, and `subscriptions` are filled from the registered user and account.
6607This lets Scroll or a login gateway mint a session without duplicating policy
6608metadata in every request. If `DBBASIC_REQUIRE_KNOWN_IDENTITY_USERS=true`, the
6609server rejects sessions for unknown users.
6610
6611For a non-admin login gateway or local trusted client, the current-session route
6612can mint a session for an existing active user when
6613`DBBASIC_ENABLE_SESSION_LOGIN=true`:
6614
6615```http
6616POST /identity/session
6617Authorization: Token <session-login-token>
6618Content-Type: application/json
6619```
6620
6621```json
6622{
6623 ""user_id"": ""7"",
6624 ""label"": ""scroll desktop"",
6625 ""ttl_seconds"": 86400
6626}
6627```
6628
6629This route always requires the user to exist in `data/identity/users.tsv`.
6630It also requires `DBBASIC_SESSION_LOGIN_TOKEN`; this token is separate from the
6631admin token so a login gateway can mint sessions without broad admin access.
6632It accepts only `user_id`, `label`, and `ttl_seconds`; caller-supplied
6633`account_id`, `roles`, or `subscriptions` are rejected. The session subject is
6634loaded from the registered user and account so a login request cannot grant
6635itself stronger permissions. This is a session-mint primitive, not password
6636authentication.
6637
6638### Password Login
6639
6640For direct password authentication, the same route accepts a credential login
6641when `DBBASIC_ENABLE_PASSWORD_LOGIN=true`. No `Authorization` header is
6642required; the password is the credential:
6643
6644```http
6645POST /identity/session
6646Content-Type: application/json
6647```
6648
6649```json
6650{""email"": ""alice@example.com"", ""password"": ""correct horse battery""}
6651```
6652
6653The body must contain exactly one of `email` or `user_id`, plus `password`, and
6654may include `label` and `ttl_seconds`. Role, account, and subscription
6655overrides are rejected. The password is verified against the scrypt hash in
6656`data/identity/credentials.tsv` set through the password routes above. Email
6657matching is case-insensitive and must resolve to exactly one active user.
6658
6659Success returns `201` with the same session-plus-one-time-token shape as other
6660session mints. Any failure — unknown user, wrong password, disabled user or
6661account — returns a uniform `401 Invalid credentials` after a short fixed
6662delay, so callers cannot probe which accounts exist. The global rate limiter
6663also applies. Password login availability (enabled flag plus at least one
6664active user) counts as a non-admin identity path for permission enforcement
6665readiness.
6666
6667Repeated failures lock the identifier: after
6668`DBBASIC_LOGIN_LOCKOUT_ATTEMPTS` failed attempts (default 5) within
6669`DBBASIC_LOGIN_LOCKOUT_WINDOW_SECONDS` (default 900), further attempts —
6670even with the correct password — return `429 Too many failed attempts` (the
6671browser form redirects to `/login?error=locked`). A successful login resets
6672the count; the lock clears on its own when the window passes. Locked attempts
6673are recorded as `login_locked` ops events rather than new failures, so
6674hammering a locked identifier does not extend the lock. Set attempts to `0`
6675to disable. The counters are the `login_failed` rows in the ops event feed —
6676`GET /admin/ops?kind=auth&identifier=...` shows exactly why an identifier is
6677locked.
6678
6679List sessions:
6680
6681```http
6682GET /identity/sessions
6683Authorization: Token <admin-token>
6684```
6685
6686Revoke a session:
6687
6688```http
6689DELETE /identity/sessions/{session_id}
6690Authorization: Token <admin-token>
6691```
6692
6693Inspect the current session with a session token:
6694
6695```http
6696GET /identity/session
6697Authorization: Token <session-token>
6698```
6699
6700Revoke the current session with the same token:
6701
6702```http
6703DELETE /identity/session
6704Authorization: Token <session-token>
6705```
6706
6707The current-session route does not accept the admin token as a user session.
6708It is for Scroll, app clients, and login gateways that need a stable non-admin
6709session lifecycle.
6710
6711### Session Cookies
6712
6713Browser clients can present the session token in a `dbbasic_session` cookie
6714instead of an `Authorization` header. Cookie tokens resolve to the same
6715session subject for permission checks, `/identity`, and the current-session
6716routes. An `Authorization` header always takes precedence over the cookie,
6717and the admin token is never accepted from a cookie.
6718
6719Cookie-authenticated non-GET requests are checked against the request origin:
6720when an `Origin` (or `Referer`) header is present and its host does not match
6721the request `Host`, the request is rejected with
6722`403 Cross-origin cookie request rejected`. Together with `SameSite` cookie
6723attributes set at login, this is the server's CSRF posture for cookie
6724sessions. Header-authenticated requests are not origin-checked.
6725
6726### Browser Login Pages
6727
6728When `DBBASIC_ENABLE_PASSWORD_LOGIN=true`, the server serves a built-in
6729browser login flow:
6730
6731```http
6732GET /login
6733GET /login?next=/dashboard
6734POST /login (application/x-www-form-urlencoded: email, password, next)
6735POST /logout
6736```
6737
6738`GET /login` renders a minimal HTML sign-in form; an active session redirects
6739away instead of re-rendering. `POST /login` verifies the password, mints a
6740session, sets the `dbbasic_session` cookie (`HttpOnly`, `SameSite=Lax`,
6741`Path=/`, `Secure` unless `DBBASIC_COOKIE_SECURE=false` for plain-HTTP local
6742development), and redirects with `303` to the sanitized `next` path
6743(same-origin absolute paths only; anything else falls back to `/`). Failures
6744redirect back to `/login?error=1` after the same fixed delay as JSON password
6745login, without revealing whether the email exists. JSON bodies are rejected;
6746API clients should use `POST /identity/session`. `POST /logout` revokes the
6747cookie session, clears the cookie, and redirects to `/login`. The login and
6748logout routes never accept or mint admin tokens.
6749
6750Session tokens normally only supply the active subject used by permission
6751checks. Admin-token gates remain admin-token-only by default. When a deployment
6752sets `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`, active sessions whose subject
6753has one of the policy admin roles can pass the same admin gate. The explicit
6754source-write gate is still separate and must also be enabled for source update
6755or rollback routes.
6756
6757## Permissions Policy
6758
6759The public server now has a persisted permission policy shape. The endpoints are
6760admin-gated while broader login and account management mature.
6761
6762```http
6763GET /permissions/policy
6764Authorization: Token <token>
6765```
6766
6767Response:
6768
6769```json
6770{
6771 ""status"": ""ok"",
6772 ""policy"": {
6773 ""access_mode"": ""role_based"",
6774 ""roles"": {},
6775 ""user_roles"": {},
6776 ""rules"": [],
6777 ""admin_roles"": [""admin"", ""superuser""]
6778 }
6779}
6780```
6781
6782Update:
6783
6784```http
6785PUT /permissions/policy
6786Authorization: Token <token>
6787Content-Type: application/json
6788```
6789
6790```json
6791{
6792 ""policy"": {
6793 ""access_mode"": ""role_based"",
6794 ""roles"": {""sales"": {""label"": ""Sales""}},
6795 ""user_roles"": {""7"": [""sales""]},
6796 ""rules"": [
6797 {
6798 ""effect"": ""allow"",
6799 ""principal"": ""role:sales"",
6800 ""actions"": [""read""],
6801 ""collection"": ""contacts"",
6802 ""row_filter"": {""owner_id"": ""$user_id""}
6803 }
6804 ],
6805 ""admin_roles"": [""admin""]
6806 }
6807}
6808```
6809
6810The policy is stored under `data/permissions/policy.json`. Missing policy files
6811load as a conservative `role_based` policy with no grants.
6812
6813## Permissions Status
6814
6815Scroll and operators can inspect whether the active identity and permission
6816configuration is ready for enforcement:
6817
6818```http
6819GET /permissions/status
6820Authorization: Token <admin-token>
6821```
6822
6823Response:
6824
6825```json
6826{
6827 ""status"": ""ok"",
6828 ""permissions"": {
6829 ""enforcement_enabled"": false,
6830 ""enforcement_requested"": false,
6831 ""enforcement_blocked"": false,
6832 ""allow_unready_enforcement"": false,
6833 ""audit_enabled"": true,
6834 ""trusted_headers_enabled"": false,
6835 ""require_known_identity_users"": true,
6836 ""admin_token_configured"": true,
6837 ""session_login_enabled"": true,
6838 ""session_login_token_configured"": true
6839 },
6840 ""identity"": {
6841 ""accounts"": {""count"": 1, ""active"": 1, ""disabled"": 0},
6842 ""users"": {""count"": 3, ""active"": 3, ""disabled"": 0},
6843 ""sessions"": {""count"": 2, ""active"": 1, ""revoked"": 1}
6844 },
6845 ""policy"": {
6846 ""valid"": true,
6847 ""policy_file_exists"": true,
6848 ""access_mode"": ""role_based"",
6849 ""rules_count"": 4,
6850 ""allow_rules"": 3,
6851 ""deny_rules"": 1
6852 },
6853 ""readiness"": {
6854 ""can_enable_enforcement"": true,
6855 ""blockers"": []
6856 },
6857 ""warnings"": []
6858}
6859```
6860
6861This endpoint is intentionally read-only. `DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true`
6862requests enforcement, but the server only makes `enforcement_enabled` effective
6863when `readiness.can_enable_enforcement` is true. If readiness is blocked,
6864`enforcement_requested` is true, `enforcement_enabled` is false, and
6865`enforcement_blocked` is true. The explicit recovery/test override is
6866`DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true`.
6867
6868Readiness currently requires an admin recovery token, a valid policy, an
6869available non-admin identity path for identity-gated modes, and at least one
6870allow grant for `role_based` policy. The accepted identity paths are trusted
6871proxy headers, guarded session login for existing users, or an already-active
6872DBBASIC session.
6873
6874## Permissions Check
6875
6876Scroll and admin tools can ask the server to evaluate one permission decision.
6877This is useful for previewing draft rules and for future ""test as role"" screens.
6878
6879```http
6880POST /permissions/check
6881Authorization: Token <token>
6882Content-Type: application/json
6883```
6884
6885```json
6886{
6887 ""subject"": {
6888 ""user_id"": ""7"",
6889 ""account_id"": ""customer-acme"",
6890 ""roles"": [""sales""],
6891 ""subscriptions"": [""pro""]
6892 },
6893 ""action"": ""read"",
6894 ""collection"": ""contacts"",
6895 ""object_id"": null,
6896 ""record"": null
6897}
6898```
6899
6900If the request includes a `policy` object, the server evaluates against that
6901inline policy without saving it. Otherwise it evaluates against the persisted
6902policy. The optional `now` field accepts an ISO timestamp for previewing
6903temporary access windows.
6904
6905Response:
6906
6907```json
6908{
6909 ""status"": ""ok"",
6910 ""decision"": {
6911 ""allowed"": true,
6912 ""reason"": ""sales reps only see own contacts"",
6913 ""code"": ""allowed"",
6914 ""http_status"": 200,
6915 ""row_filter"": {""owner_id"": ""$user_id""},
6916 ""fields"": null,
6917 ""denied_fields"": []
6918 }
6919}
6920```
6921
6922When a paid entitlement is missing, the decision can include:
6923
6924```json
6925{
6926 ""allowed"": false,
6927 ""reason"": ""subscription required"",
6928 ""code"": ""payment_required"",
6929 ""http_status"": 402,
6930 ""row_filter"": {},
6931 ""fields"": null,
6932 ""denied_fields"": []
6933}
6934```
6935
6936## Permission Route Enforcement
6937
6938Permission policy checks can be applied to object and collection-record routes
6939when explicitly requested:
6940
6941```text
6942DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true
6943```
6944
6945The request becomes effective only when `/permissions/status` reports no
6946readiness blockers, unless `DBBASIC_ALLOW_UNREADY_PERMISSION_ENFORCEMENT=true`
6947is also set. When enforcement is effective, denied route checks return the
6948`http_status`, `reason`, and `code` from the permission decision before object
6949source, state, logs, versions, execution, or collection-record data work runs.
6950
6951Audit-only mode records the same decisions without changing responses:
6952
6953```text
6954DBBASIC_ENABLE_PERMISSION_AUDIT=true
6955```
6956
6957Audit entries are written as JSON lines under:
6958
6959```text
6960data/permissions/audit.jsonl
6961```
6962
6963Trusted identity headers are off by default and should only be enabled behind a
6964trusted auth gateway or reverse proxy:
6965
6966```text
6967DBBASIC_PERMISSION_TRUST_HEADERS=true
6968```
6969
6970Supported headers are `X-DBBASIC-User-Id`, `X-DBBASIC-Account-Id`,
6971`X-DBBASIC-Roles`, and `X-DBBASIC-Subscriptions`.
6972
6973Collection record read routes use the `read` action. Collection record write
6974routes use `create`, `update`, and `delete`. In enforcement mode the server
6975applies row filters before pagination, evaluates detail and write requests
6976against the selected record, blocks owner-changing updates that would escape the
6977allowed row filter, and redacts `fields` / `denied_fields` from returned
6978records. Audit-only mode can log write decisions, but it does not grant
6979mutation access without the admin token.
6980
6981Operators and Scroll can read recent audit entries through an admin-gated
6982endpoint:
6983
6984```http
6985GET /permissions/audit?limit=100
6986Authorization: Token <token>
6987```
6988
6989Optional filters:
6990
6991- `action`
6992- `object_id`
6993- `collection`
6994- `allowed=true|false`
6995- `enforced=true|false`
6996
6997Response:
6998
6999```json
7000{
7001 ""status"": ""ok"",
7002 ""entries"": [
7003 {
7004 ""timestamp"": ""2026-06-29T00:00:00Z"",
7005 ""method"": ""GET"",
7006 ""object_id"": ""site_home"",
7007 ""collection"": ""site"",
7008 ""action"": ""execute"",
7009 ""enforced"": false,
7010 ""decision"": {""allowed"": false, ""code"": ""forbidden""}
7011 }
7012 ],
7013 ""count"": 1
7014}
7015```
7016
7017## Optimistic Concurrency (`_rev` / `If-Match`)
7018
7019Record updates are last-write-wins by default. To make a read-then-write
7020safe against a concurrent writer, send back the `_rev` you read.
7021
7022**Reading a rev.** A single-record read returns it as a **sibling** of
7023`record`, never a field inside it:
7024
7025```json
7026{""status"": ""ok"", ""collection"": ""tasks"", ""record"": {""id"": ""t1"", ""status"": ""open""},
7027 ""_rev"": ""a1b2c3…""}
7028```
7029
7030A list read returns a parallel **`revs` map**, keyed by record id, covering
7031exactly the rows in the returned window:
7032
7033```json
7034{""status"": ""ok"", ""collection"": ""tasks"",
7035 ""records"": [{""id"": ""t1"", ""status"": ""open""}, {""id"": ""t2"", ""status"": ""open""}],
7036 ""count"": 2, ""total"": 57, ""limit"": 100, ""offset"": 0,
7037 ""revs"": {""t1"": ""a1b2c3…"", ""t2"": ""d4e5f6…""}}
7038```
7039
7040It is a map rather than a `_rev` on each row so the record shape stays a
7041contract: a rev must never collide with a schema field, and must never be
7042echoed back on write.
7043
7044**Using one.** Send `If-Match: <rev>` (or an `expected_rev` body field) on
7045`PUT`. The rev is compared **inside the write lock**, before validation:
7046
7047- match → the write proceeds
7048- mismatch → **409**, and nothing is written
7049
7050```
7051PUT /collections/tasks/records/t1
7052If-Match: a1b2c3…
7053
7054{""status"": ""assigned"", ""assigned_to"": ""agent-A""}
7055```
7056
7057**Why it exists — the claim.** Two agents both read a task as `open` and
7058both write `assigned`. Without a precondition both succeed and whoever
7059lands second silently overwrites the first; both believe they hold it and
7060neither can tell. With `If-Match`, the second finds the row changed, gets a
7061409, re-reads, and moves on. The rev captures ""the record I read is still
7062the record on disk"", which necessarily includes every field — so no
7063per-field guard is needed.
7064
7065Because the `revs` map ships with the list, an agent can find work and
7066claim it without re-reading each candidate first.
7067
7068**Notes.**
7069
7070- Omitting `If-Match` is still valid and still last-write-wins. The
7071 precondition is opt-in per write, never mandatory.
7072- The rev is computed over the **full stored record**, before any
7073 field-level permission trimming. A caller who cannot see a field still
7074 gets a rev that depends on it — which only ever fails closed, and
7075 discloses nothing about the value.
7076- `DBBASIC_ENABLE_CONCURRENCY=false` turns the feature off: preconditions
7077 are ignored (never a 409) and `revs` is omitted from list payloads.
7078
7079## Object List
7080
7081```http
7082GET /objects?format=json
7083Authorization: Token <token>
7084```
7085
7086Response:
7087
7088```json
7089{
7090 ""status"": ""ok"",
7091 ""objects"": [
7092 {
7093 ""object_id"": ""basics_counter"",
7094 ""path"": ""basics/counter.py"",
7095 ""owner"": ""system""
7096 }
7097 ],
7098 ""count"": 1
7099}
7100```
7101
7102Clients should accept either a top-level list or an object with an `objects`
7103field. New server code should prefer the object form above.
7104
7105## Admin Object Inspection
7106
7107`/objects` remains the runtime surface: object execution and public object
7108routes live there. Scroll and operator dashboards should use the admin
7109inspection surface so staging can expose read-only object metadata without
7110opening broad object execution routes at the reverse proxy.
7111
7112```http
7113GET /admin/objects
7114POST /admin/objects
7115Authorization: Token <token>
7116```
7117
7118`GET` returns the same response shape as `GET /objects?format=json`.
7119`POST` is the admin/operator alias for `POST /objects`; it creates a new
7120object without requiring the reverse proxy to expose the broader runtime
7121`/objects` collection route. Creation still requires
7122`DBBASIC_ENABLE_SOURCE_WRITES=true` and the admin gate.
7123
7124```http
7125GET /admin/objects/{object_id}
7126GET /admin/objects/{object_id}?metadata=true
7127GET /admin/objects/{object_id}?source=true&format=json
7128PUT /admin/objects/{object_id}?source=true
7129POST /admin/objects/{object_id}/execute
7130GET /admin/objects/{object_id}?state=true
7131GET /admin/objects/{object_id}?logs=true&format=json&limit=100
7132GET /admin/objects/{object_id}?versions=true&limit=10
7133GET /admin/objects/{object_id}?version=1
7134GET /admin/objects/{object_id}?source_changes=true&limit=100
7135GET /admin/objects/{object_id}?changes=true&limit=100
7136GET /admin/objects/{object_id}?files=true
7137GET /admin/objects/{object_id}?file=report.txt
7138Authorization: Token <token>
7139```
7140
7141If no inspection query is supplied, the server returns metadata. The admin
7142inspection surface never executes the object. `PUT ...?source=true` is the
7143admin/operator alias for source edits and uses the same
7144`DBBASIC_ENABLE_SOURCE_WRITES` plus admin gate as the runtime source update
7145route. `POST .../execute` is the admin/operator execution route and requires
7146the admin gate before running object code. It accepts a JSON body:
7147
7148```json
7149{
7150 ""method"": ""GET"",
7151 ""payload"": {}
7152}
7153```
7154
7155`method` may be `GET`, `POST`, `PUT`, or `DELETE`; `payload` must be a JSON
7156object and defaults to `{}`. Unsupported query flags return a 400 response so a
7157client cannot accidentally turn an operator view into an execution endpoint.
7158
7159## Admin Changes
7160
7161Scroll and operator dashboards can read a single normalized activity stream
7162instead of querying every changelog separately.
7163
7164```http
7165GET /admin/changes?kind=file&object_id=site_home&limit=100&offset=0
7166Authorization: Token <token>
7167```
7168
7169Supported filters:
7170
7171- `kind`: `source`, `file`, `record`, or `package`
7172- `object_id`: source and file changes for one object
7173- `collection`: record changes for one collection
7174- `record_id`: record changes for one record
7175- `package_id`: package changes for one package
7176- `file`: file changes for one object-owned file
7177
7178Response:
7179
7180```json
7181{
7182 ""status"": ""ok"",
7183 ""changes"": [
7184 {
7185 ""kind"": ""file"",
7186 ""change_id"": ""123e4567-e89b-42d3-a456-426614174000"",
7187 ""timestamp"": ""2026-01-01T00:00:00+00:00"",
7188 ""action"": ""file_update"",
7189 ""actor"": ""api"",
7190 ""summary"": ""File updated: reports/out.csv"",
7191 ""correlation_id"": ""123e4567-e89b-42d3-a456-426614174001"",
7192 ""target"": {
7193 ""object_id"": ""site_home"",
7194 ""file_name"": ""reports/out.csv"",
7195 ""file_size"": 42
7196 },
7197 ""change"": {}
7198 }
7199 ],
7200 ""count"": 1,
7201 ""total"": 1,
7202 ""limit"": 100,
7203 ""offset"": 0,
7204 ""has_more"": false,
7205 ""filters"": {
7206 ""kind"": ""file"",
7207 ""object_id"": ""site_home"",
7208 ""collection"": null,
7209 ""record_id"": null,
7210 ""package_id"": null,
7211 ""file"": ""reports/out.csv""
7212 }
7213}
7214```
7215
7216Each entry keeps the original changelog payload under `change`. Clients should
7217render `kind`, `timestamp`, `actor`, `summary`, `target`, and
7218`correlation_id` first, then expand `change` for detail views.
7219
7220## Admin File Inspection
7221
7222Object-owned files live under the local runtime data directory:
7223
7224```text
7225data/files/{object_id}/
7226```
7227
7228The compatibility object route can still list or download one object's files,
7229but Scroll and operator dashboards should prefer the admin file inspection
7230surface so staging can expose read-only file posture without exposing broad
7231object execution routes.
7232
7233```http
7234GET /admin/files
7235GET /admin/files?object_id=site_home
7236GET /admin/files?object_id=site_home&file=report.txt
7237GET /admin/files/{object_id}
7238GET /admin/files/{object_id}?file=report.txt
7239Authorization: Token <token>
7240```
7241
7242`GET /admin/files` returns a paginated cross-object inventory:
7243
7244```json
7245{
7246 ""status"": ""ok"",
7247 ""files"": [
7248 {
7249 ""object_id"": ""site_home"",
7250 ""name"": ""report.txt"",
7251 ""size"": 1200,
7252 ""modified"": 1760000000.0
7253 }
7254 ],
7255 ""count"": 1,
7256 ""total"": 1
7257}
7258```
7259
7260Supported query parameters:
7261
7262- `object_id` narrows the inventory to one object.
7263- `limit` defaults to `100` and is capped at `1000`.
7264- `offset` defaults to `0`.
7265- `file` downloads one object-owned file and requires an object id.
7266
7267Read routes are admin-token gated. Write routes are additionally disabled unless
7268`DBBASIC_ENABLE_FILE_WRITES=true` is set. Public staging should leave file writes
7269off until permissions, quota, and review policy are intentionally enabled.
7270
7271Create one object-owned file:
7272
7273```http
7274POST /admin/files/{object_id}
7275Authorization: Token <token>
7276Content-Type: application/json
7277```
7278
7279```json
7280{
7281 ""name"": ""reports/monthly.txt"",
7282 ""content_base64"": ""aGVsbG8=""
7283}
7284```
7285
7286Overwrite an existing object-owned file:
7287
7288```http
7289PUT /admin/files/{object_id}
7290Authorization: Token <token>
7291Content-Type: application/json
7292```
7293
7294Delete one object-owned file:
7295
7296```http
7297DELETE /admin/files/{object_id}?file=reports/monthly.txt
7298Authorization: Token <token>
7299```
7300
7301Successful write responses return the file metadata, operation, and correlation
7302id:
7303
7304```json
7305{
7306 ""status"": ""ok"",
7307 ""message"": ""File created: reports/monthly.txt"",
7308 ""object_id"": ""site_home"",
7309 ""operation"": ""created"",
7310 ""file"": {
7311 ""object_id"": ""site_home"",
7312 ""name"": ""reports/monthly.txt"",
7313 ""size"": 5,
7314 ""modified"": 1760000000.0
7315 },
7316 ""correlation_id"": ""07fb51ef-1a0d-43dd-a82f-2ec756b7cd90""
7317}
7318```
7319
7320`POST` rejects existing files with `409`. `PUT` overwrites. Empty names,
7321absolute paths, null bytes, and `..` traversal are rejected with `400`. File
7322content is capped by `DBBASIC_MAX_OBJECT_FILE_BYTES` and the request body is
7323still capped by `DBBASIC_MAX_REQUEST_BYTES`.
7324
7325## Admin Ops Events
7326
7327The operational event feed records object execution failures and auth
7328activity in one admin-gated stream:
7329
7330```http
7331GET /admin/ops
7332GET /admin/ops?limit=100
7333GET /admin/ops?kind=execution_error
7334GET /admin/ops?kind=auth&event=login_failed
7335GET /admin/ops?identifier=alice@example.com
7336Authorization: Token <token>
7337```
7338
7339Events are newest-first. `execution_error` entries carry `object_id`,
7340`method`, `error_type`, a truncated `message`, the caller's `user_id`, and
7341the `correlation_id` linking back to source versions and object logs. `auth`
7342entries carry `event` (`login_succeeded`, `login_failed`, `logout`,
7343`session_minted`), the attempted `identifier` on failures, `user_id` on
7344successes, `auth_method`, and session `label`. Rows never contain passwords,
7345tokens, or hashes. The feed lives in `data/ops/events.jsonl`, capped to the
7346most recent 5000 rows. Failed-login rows are queryable per identifier, which
7347is the data the future login-lockout gate will read.
7348
7349## MCP Endpoint
7350
7351AI agents can operate the server through MCP (Model Context Protocol) instead
7352of raw HTTP. The endpoint speaks JSON-RPC 2.0 over HTTP POST (the Streamable
7353HTTP transport):
7354
7355```http
7356POST /api/mcp
7357Authorization: Token <admin-token-or-admin-role-session-token>
7358Content-Type: application/json
7359```
7360
7361Supported methods: `initialize` (protocol negotiation, returns an
7362`mcp-session-id` header), `tools/list`, `tools/call`, and
7363`notifications/*` (acknowledged with `202`). The gate is the same admin gate
7364as the rest of the admin surface: the deployment token always works, and an
7365admin-role session token works when `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`.
7366
7367Register with Claude Code:
7368
7369```bash
7370claude mcp add dbbasic --transport http \
7371 --url https://your-host/api/mcp \
7372 --header ""Authorization: Bearer YOUR_SESSION_TOKEN""
7373```
7374
7375The tool catalog wraps the existing admin surface — objects
7376(list/read/create/update-source/execute, state, logs, metadata, changes),
7377collections and records (list/get/create/update/delete), schemas
7378(get/update/rollback), cross-collection search (`global_search`), the
7379unified change history, and admin status. Every
7380tool call is dispatched back through the server's own routing with the
7381caller's credentials, so tools inherit the same gates, capability flags
7382(source writes, package installs), permission checks, audit trail, and
7383correlation ids as external HTTP callers. Tool results carry the underlying
7384`http_status` plus the route's JSON response, and failures set `isError`.
7385
7386Give each agent its own identity instead of the deployment token: create a
7387user (for example `agent-claude`) with the admin role, mint a labeled session,
7388and connect with that bearer token, so `/admin/changes` distinguishes agent
7389actions from operator actions.
7390
7391## Service Keys (Per-User AI Provider Keys)
7392
7393Users store API keys for AI providers so the server can call providers on
7394their behalf. The contract is write-only: set a key, list which services
7395have one, delete one — key material never appears in any response, never
7396lives in record collections, and stays out of portable backups (it sits
7397beside `credentials.tsv` in the owner-only identity directory).
7398
7399```http
7400GET /identity/users/{user_id}/service-keys -> {""services"": [{""service"", ""created_at"", ""updated_at""}]}
7401PUT /identity/users/{user_id}/service-keys {""service"": ""anthropic"", ""key"": ""sk-...""}
7402DELETE /identity/users/{user_id}/service-keys/{service}
7403```
7404
7405A signed-in user manages their own keys (session token or cookie; cookie
7406writes require a same-origin request); the admin gate covers operator
7407use. Set/remove events land in the ops feed without key material.
7408
7409## AI Chat (The Shell's Brain)
7410
7411One conversation turn against an AI provider, using the caller's stored
7412service key, with optional MCP tool calling:
7413
7414```http
7415POST /api/ai/chat
7416Authorization: Bearer <session-token> (or the session cookie)
7417
7418{""message"": ""find my notes about the flywheel"",
7419 ""model"": ""anthropic:claude-haiku-4-5"",
7420 ""tools"": [""global_search"", ""get_record""],
7421 ""max_rounds"": 6}
7422```
7423
7424Models are named `service:model`; the service picks which stored key is
7425used (`anthropic`, `openai`). `tools` names a subset of the MCP tool
7426catalog — handing small models only a few tools keeps their context
7427small, and the subset is per-request configuration. When the model calls
7428a tool, the server dispatches it through its own routing **with the
7429caller's credentials**, so the AI can do exactly what the caller could do
7430directly — permission checks, row filters, and the audit trail all apply.
7431
7432Response: `{""reply"", ""model"", ""rounds"", ""tool_calls"": [{name, arguments,
7433http_status}], ""usage"": {input_tokens, output_tokens}}`, plus
7434`""truncated"": true` when the round limit stopped a tool loop.
7435
7436Requires `DBBASIC_ENABLE_AI_CHAT=true` and a signed-in session; callers
7437without a stored key for the model's service get a pointed 400. Timeout
7438per provider round: `DBBASIC_AI_TIMEOUT_SECONDS` (default 60); default
7439model: `DBBASIC_AI_DEFAULT_MODEL`.
7440
7441## Text-to-Speech
7442
7443```http
7444POST /api/tts
7445Authorization: Bearer <session-token> (or the session cookie)
7446
7447{""text"": ""one note matches"", ""voice"": ""en-us"", ""engine"": ""local""}
7448```
7449
7450Response: audio bytes (`audio/wav` for `engine: ""local""`, `audio/mpeg` for
7451`engine: ""openai""`) on success, or `{""status"": ""error"", ""error""}` on
7452failure. `engine` is optional (`""local""` default, or `""openai""` — uses the
7453caller's stored `openai` service key, no engine ever silently falls back
7454to the other). Requires `DBBASIC_ENABLE_TTS=true` and a signed-in session.
7455`text` capped at 800 characters (413 beyond). `DBBASIC_TTS_CLOUD_TIMEOUT_SECONDS`
7456bounds the cloud path (default 30). See `docs/shell-and-ai.md`'s ""Voice""
7457section for engine details and caching.
7458
7459## Speech-to-Text
7460
7461```http
7462POST /api/stt
7463Authorization: Bearer <session-token> (or the session cookie)
7464Content-Type: audio/webm
7465
7466<raw audio bytes>
7467```
7468
7469Response: `{""status"": ""ok"", ""text"": ""...""}` or the usual error shape. Body
7470is raw audio bytes (not JSON) in whatever format `Content-Type` says.
7471Cloud-only — no local engine — calls OpenAI's transcription API with the
7472caller's stored `openai` service key. Requires `DBBASIC_ENABLE_STT=true`
7473(off by default even when TTS is on: this costs money per utterance, not
7474per occasional reply) and a signed-in session. Capped at 10MB per clip
7475(413 beyond). `DBBASIC_STT_TIMEOUT_SECONDS` bounds the call (default 30).
7476
7477## Reader
7478
7479```http
7480POST /api/read
7481Authorization: Bearer <session-token> (or the session cookie)
7482
7483{""url"": ""https://example.com/article""}
7484```
7485
7486Response: `{""status"": ""ok"", ""title"", ""text"", ""links"": [{n, label, href}],
7487""final_url"", ""truncated""}` or the usual error shape. Fetches server-side,
7488strips HTML to readable text, numbers links in document order. SSRF-hardened
7489(every hostname, including redirect hops, resolved and checked before the
7490socket opens; `http`/`https` only; fails closed on any resolution
7491ambiguity) — see `object_reader.py`. Requires `DBBASIC_ENABLE_READER=true`
7492and a signed-in session. Also offered as the `read_page` MCP tool for
7493`/api/ai/chat` — see `docs/shell-and-ai.md`'s ""Reading the Web"" section.
7494
7495## Backups
7496
7497Runtime backups are tar/gzip archives of the whole data directory. They
7498contain everything — records, identity, credentials, service keys — so
7499the surface is **strictly admin-gated** (admin token or admin-role
7500session; never public):
7501
7502```http
7503GET /admin/backups -> {backups: [{id, created_at, size, kind, scope}], count, schedule}
7504POST /admin/backups -> {backup: {id, created_at, size, kind, scope}} (creates one now, 201)
7505GET /admin/backups/{id}/download -> the archive bytes (application/gzip, attachment)
7506```
7507
7508`kind` is `manual` (created on demand), `package` (auto restore point
7509before a package install; `scope` is the package), or `restore-point`.
7510The `/admin/status` `capabilities.backups` block reports
7511`{available, can_create, can_download, can_restore, scheduled, schedule}`
7512so clients gate their buttons off real capability. Restore over HTTP is
7513not exposed yet (`can_restore: false`); use the `object_backup.py restore`
7514CLI, which is deliberately a deliberate, on-box operation.
7515
7516**Automatic backups are a config option, not on by default.** A scheduled
7517run is an external timer (systemd/cron) calling `object_backup.py create`;
7518`DBBASIC_BACKUP_SCHEDULE` records the operator's intent so clients can show
7519it. On-demand create and download always work regardless.
7520
7521## Realtime Push (Websocket)
7522
7523Clients get live record-change signals over a websocket at `/ws`, so open
7524pages auto-update the instant something changes — no polling. The durable
7525event log (see the runtime contract) remains as the poll-based fallback.
7526
7527```
7528WebSocket wss://your-host/ws
7529```
7530
7531Connect with your session (the `dbbasic_session` cookie is sent
7532automatically; a bearer/admin token also works). Anonymous connections
7533are closed. Then speak a tiny JSON protocol:
7534
7535```json
7536// client -> server
7537{""action"": ""subscribe"", ""collections"": [""notifications"", ""notes""]}
7538{""action"": ""unsubscribe"", ""collections"": [""notes""]}
7539{""action"": ""ping""}
7540
7541// server -> client
7542{""type"": ""welcome"", ""user"": ""dan""}
7543{""type"": ""subscribed"", ""collections"": [""notifications"", ""notes""]}
7544{""type"": ""record"", ""collection"": ""notes"", ""record_id"": ""…"", ""action"": ""created""}
7545{""type"": ""pong""}
7546```
7547
7548The `record` message is a **signal, not data** — only the collection,
7549record id, and action travel, never the record body. The client refetches
7550through the normal permission-enforced API. Two guarantees follow the same
7551rules as reads: you can only `subscribe` to a collection you may read, and
7552with enforcement on you only receive events for records your row filters
7553would let you see (a private note's change is never signalled to another
7554user). Requires `DBBASIC_ENABLE_REALTIME` (default on); the reverse proxy
7555must forward websocket upgrades (Caddy does by default). Single-process
7556deployment assumption applies — see
7557[asgi-realtime-direction](/docs/asgi-realtime-direction).
7558
7559The shared nav (`/nav`) and every app page already use this. The
7560notification bell subscribes to `notifications` and updates live (with a
756120s poll as backup); each list page (`/notes`, `/tasks`, `/contacts`, …)
7562subscribes to its own collection and re-renders when a record it can see
7563changes — in another tab, by another user, or by an agent — over the same
7564single shared socket. Any page opts in with one line:
7565`window.dbbasicSubscribe(collection, handler)`.
7566
7567## Global Search
7568
7569Records can be searched across every collection whose schema declares
7570searchable fields (see the schema field contract's `search` section):
7571
7572```http
7573GET /api/search?q=flywheel+growth&limit=10
7574GET /api/search?q=flywheel&collections=notes,tasks
7575Authorization: Token <token-or-session> (or the dbbasic_session cookie)
7576```
7577
7578The query is split on whitespace; every term must match at least one
7579searchable field (case-insensitive substring), and a short hex-ish query
7580also matches record ids by prefix. Results come back per collection with
7581no ranking:
7582
7583```json
7584{
7585 ""status"": ""ok"",
7586 ""query"": ""flywheel growth"",
7587 ""limit"": 10,
7588 ""results"": {
7589 ""notes"": [{""id"": ""n1"", ""content"": ""flywheel growth loop""}],
7590 ""tasks"": []
7591 },
7592 ""total_count"": 1
7593}
7594```
7595
7596Each result record is trimmed to the schema's `search.result_fields`
7597(default: `id` plus the schema's `views.list_fields`, falling back to the
7598searchable fields). Search runs inside the permission engine: with
7599enforcement on, collections the caller cannot read are skipped entirely,
7600row filters hide other users' records, and field permissions redact
7601fields before results are returned — the same subject resolution as
7602record reads (admin token, session token, or session cookie). Without
7603permission checks the endpoint stays behind the admin gate. Malformed
7604`search` sections surface in a `warnings` list instead of failing the
7605whole request.
7606
7607## Admin Collection And Schema Inspection
7608
7609`/collections` and `/schemas` remain the data and schema API surfaces. Scroll
7610and operator dashboards should use the admin inspection aliases for read-only
7611collection, record, changelog, schema, and schema-version views. This lets a
7612public staging reverse proxy expose inspection routes without exposing the broad
7613write-capable collection and schema routes.
7614
7615```http
7616GET /admin/collections
7617GET /admin/collections/{collection}
7618GET /admin/collections/{collection}/records
7619GET /admin/collections/{collection}/records/{record_id}
7620GET /admin/collections/{collection}/changes
7621GET /admin/collections/{collection}/records/{record_id}/changes
7622GET /admin/schemas
7623GET /admin/schemas/{collection}
7624GET /admin/schemas/{collection}?versions=true&limit=10
7625GET /admin/schemas/{collection}?version=1
7626Authorization: Token <token>
7627```
7628
7629The responses match the underlying read-only `GET /collections*` and
7630`GET /schemas*` responses.
7631
7632## Admin Collection Record Writes
7633
7634Operator screens can create, update, and delete collection records through
7635narrow admin aliases without exposing the broad write-capable
7636`/collections/*` routes through the reverse proxy:
7637
7638```http
7639POST /admin/collections/{collection}/records
7640PUT /admin/collections/{collection}/records/{record_id}
7641DELETE /admin/collections/{collection}/records/{record_id}
7642Authorization: Token <token>
7643```
7644
7645These aliases route to the same handlers as `POST /collections/{collection}/records`,
7646`PUT /collections/{collection}/records/{record_id}`, and
7647`DELETE /collections/{collection}/records/{record_id}`, so request bodies,
7648responses, schema validation, field permissions, record change history, and
7649permission enforcement behave identically. Writes require the admin token (or
7650enforced permission policy when enforcement is enabled).
7651
7652Other write methods on `/admin/collections*` paths — including collection
7653create or delete — remain unsupported and return `405`.
7654
7655## Admin Storage And Compaction
7656
7657Compaction observability and control for append-mode collections (see
7658[storage-modes.md](/docs/storage-modes)):
7659
7660```http
7661GET /admin/storage
7662POST /admin/collections/{collection}/compact
7663Authorization: Token <token>
7664```
7665
7666`GET /admin/storage` returns stats for every collection whose schema
7667currently declares `""storage"": ""append""`:
7668
7669```json
7670{
7671 ""status"": ""ok"",
7672 ""append_collections"": [
7673 {
7674 ""collection"": ""events_log"",
7675 ""storage"": ""append"",
7676 ""physical_rows"": 41200,
7677 ""live_rows"": 9800,
7678 ""dead_rows"": 31400,
7679 ""bloat_ratio"": 3.204,
7680 ""file_bytes"": 5872000,
7681 ""sidecar_present"": true,
7682 ""compaction_flagged"": false
7683 }
7684 ]
7685}
7686```
7687
7688This route is deliberately cheap: it never folds a collection's
7689`records.tsv` to answer, preferring a warm in-process cache entry or the
7690id→offset sidecar's own row count. A collection neither can answer comes
7691back as an **estimated** entry instead of paying an O(file) parse on a
7692status call — `file_bytes` is still real, but `physical_rows`,
7693`live_rows`, `dead_rows`, and `bloat_ratio` are `null` and
7694`""estimated"": true` is set. `bloat_ratio` is `dead_rows / max(live_rows,
76951)`, rounded to 3 decimal places. A classic-mode collection is never
7696included.
7697
7698`POST /admin/collections/{collection}/compact` runs
7699`object_records.compact_collection` under the same gate that protects
7700other mutating admin record routes (admin token, or enforced permission
7701policy when enforcement is enabled — there is no single record for this
7702operation, so the collection-level write check is the whole check):
7703
7704```json
7705{
7706 ""status"": ""ok"",
7707 ""collection"": ""events_log"",
7708 ""rows_before"": 41200,
7709 ""rows_after"": 9800,
7710 ""bytes_before"": 5872000,
7711 ""bytes_after"": 1398600,
7712 ""duration_ms"": 812.4
7713}
7714```
7715
7716Returns `400` for a collection not currently in append storage mode
7717(nothing to compact) and `404` for an unknown collection. Compaction also
7718runs automatically — on a write, once a collection's dead-row ratio and
7719size cross their thresholds, and on a timer if the optional daemon
7720(`object_daemon.py`) is running — see storage-modes.md for the envs
7721(`DBBASIC_APPEND_COMPACT_MIN_ROWS`, `DBBASIC_COMPACTION_INTERVAL_SECONDS`,
7722`DBBASIC_COMPACTION_BLOAT_RATIO`). The `/admin/status` `capabilities.storage`
7723block reports `{""append_collections"": <count>}`.
7724
7725## Admin Schema Writes
7726
7727Operator screens can replace and roll back collection schemas through narrow
7728admin aliases without exposing the broad write-capable `/schemas/*` routes
7729through the reverse proxy:
7730
7731```http
7732PUT /admin/schemas/{collection}
7733POST /admin/schemas/{collection}
7734Authorization: Token <token>
7735```
7736
7737These aliases route to the same handlers as `PUT /schemas/{collection}` and
7738`POST /schemas/{collection}`, so request bodies, responses, schema
7739normalization, and schema version history behave identically. `PUT` replaces
7740the schema and records a new schema version. `POST` supports action-style
7741operations; the only supported action is `{""action"": ""rollback"",
7742""version_id"": <id>}`. Both writes require the admin token.
7743
7744Other write methods on `/admin/schemas*` paths — including schema delete —
7745remain unsupported and return `405`. The admin aliases still do not install
7746packages or expose broad `/objects` execution routes.
7747
7748## Admin Identity Inspection
7749
7750`/identity/accounts`, `/identity/users`, and `/identity/sessions` remain the
7751identity management API surfaces. Scroll and operator dashboards should use the
7752admin inspection aliases for read-only account, user, and session views. This
7753lets public staging expose identity posture without exposing identity creation,
7754arbitrary session minting, or session revocation.
7755
7756```http
7757GET /admin/identity/accounts
7758GET /admin/identity/accounts/{account_id}
7759GET /admin/identity/users
7760GET /admin/identity/users?account_id=acme
7761GET /admin/identity/users/{user_id}
7762GET /admin/identity/sessions
7763GET /admin/identity/sessions/{session_id}
7764Authorization: Token <token>
7765```
7766
7767The responses match the underlying read-only `GET /identity/accounts*`,
7768`GET /identity/users*`, and `GET /identity/sessions*` responses. These admin
7769aliases are admin-token gated and read-only, with one write exception: the
7770password set/reset aliases (`POST/DELETE /admin/identity/users/{user_id}/password`,
7771documented under User Passwords). They still do not create accounts or users,
7772mint sessions, or revoke sessions.
7773
7774## Collections
7775
7776```http
7777GET /collections
7778Authorization: Token <token>
7779```
7780
7781Response:
7782
7783```json
7784{
7785 ""status"": ""ok"",
7786 ""collections"": [
7787 {
7788 ""name"": ""site"",
7789 ""object_count"": 2,
7790 ""file_count"": 1,
7791 ""state_object_count"": 1,
7792 ""log_object_count"": 1,
7793 ""has_records"": true,
7794 ""owners"": [""system""],
7795 ""kinds"": {""system"": 2},
7796 ""permission"": {
7797 ""access_mode"": ""role_based"",
7798 ""rule_count"": 1,
7799 ""allow_count"": 1,
7800 ""deny_count"": 0,
7801 ""actions"": [""execute"", ""read""],
7802 ""principals"": [""role:admin""]
7803 }
7804 }
7805 ],
7806 ""count"": 1
7807}
7808```
7809
7810Read one collection with object details:
7811
7812```http
7813GET /collections/{collection}
7814Authorization: Token <token>
7815```
7816
7817Response:
7818
7819```json
7820{
7821 ""status"": ""ok"",
7822 ""collection"": {
7823 ""name"": ""site"",
7824 ""object_count"": 1,
7825 ""file_count"": 0,
7826 ""state_object_count"": 0,
7827 ""log_object_count"": 0,
7828 ""has_records"": false,
7829 ""owners"": [""system""],
7830 ""kinds"": {""system"": 1},
7831 ""permission"": {
7832 ""access_mode"": ""role_based"",
7833 ""rule_count"": 0,
7834 ""allow_count"": 0,
7835 ""deny_count"": 0,
7836 ""actions"": [],
7837 ""principals"": []
7838 },
7839 ""objects"": [
7840 {
7841 ""object_id"": ""site_home"",
7842 ""path"": ""site/home.py"",
7843 ""owner"": ""system"",
7844 ""kind"": ""system"",
7845 ""state_count"": 0,
7846 ""has_logs"": false,
7847 ""file_count"": 0
7848 }
7849 ]
7850 }
7851}
7852```
7853
7854Collections are a derived view over object source IDs, source folders, object
7855state/files/log presence, record-file presence, and permission rules. The server
7856does not store a separate collection table yet. This keeps the existing
7857`/objects` contract stable while giving tools such as Scroll a cleaner grouping
7858API.
7859
7860The public server keeps collection summary routes read-only and admin-token
7861gated for now. Missing collections return `404`; unsafe collection names return
7862`400`.
7863
7864## Collection Records
7865
7866List records:
7867
7868```http
7869GET /collections/{collection}/records?limit=100&offset=0
7870Authorization: Token <token>
7871```
7872
7873By default this route is admin-token gated. If
7874`DBBASIC_ENABLE_PERMISSION_AUDIT=true` or
7875`DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true` is set, the route uses the
7876persisted permission policy instead. Enforcement uses the `read` action with the
7877collection name, applies row filters before `limit` / `offset`, and redacts
7878fields according to the matching decision.
7879
7880Response:
7881
7882```json
7883{
7884 ""status"": ""ok"",
7885 ""collection"": ""contacts"",
7886 ""records"": [
7887 {
7888 ""id"": ""c1"",
7889 ""first_name"": ""Ada"",
7890 ""last_name"": ""Lovelace""
7891 }
7892 ],
7893 ""count"": 1,
7894 ""total"": 1,
7895 ""limit"": 100,
7896 ""offset"": 0,
7897 ""has_more"": false
7898}
7899```
7900
7901Read one record:
7902
7903```http
7904GET /collections/{collection}/records/{record_id}
7905Authorization: Token <token>
7906```
7907
7908The detail route is also admin-token gated by default. With permission
7909enforcement enabled, the server loads the record, checks the same `read`
7910permission against that record, and then applies field allow/deny rules before
7911returning it.
7912
7913Response:
7914
7915```json
7916{
7917 ""status"": ""ok"",
7918 ""collection"": ""contacts"",
7919 ""record"": {
7920 ""id"": ""c1"",
7921 ""first_name"": ""Ada"",
7922 ""last_name"": ""Lovelace""
7923 }
7924}
7925```
7926
7927Record files live under:
7928
7929```text
7930data/collections/{collection}/records.tsv
7931```
7932
7933The TSV file must have a header row and an `id` column. Values are returned as
7934strings.
7935
7936For new DBBASIC-created records, the server generates a UUIDv4 `id` when the
7937request omits one. During migrations, imported rows may preserve legacy integer
7938or slug IDs for compatibility, but new public routes and generated UI should
7939not assume sequential IDs.
7940
7941Create one record:
7942
7943```http
7944POST /collections/{collection}/records
7945Authorization: Token <token>
7946Content-Type: application/json
7947```
7948
7949```json
7950{
7951 ""first_name"": ""Grace"",
7952 ""last_name"": ""Hopper""
7953}
7954```
7955
7956Response:
7957
7958```json
7959{
7960 ""status"": ""ok"",
7961 ""collection"": ""contacts"",
7962 ""record"": {
7963 ""id"": ""8c2d4de7-89fe-45f7-9cf9-f0f42610e7be"",
7964 ""first_name"": ""Grace"",
7965 ""last_name"": ""Hopper""
7966 }
7967}
7968```
7969
7970Successful creates return `201`. Explicit record IDs are still accepted for
7971imports and compatibility. Duplicate record IDs return `409`.
7972
7973Update one record:
7974
7975```http
7976PUT /collections/{collection}/records/{record_id}
7977Authorization: Token <token>
7978Content-Type: application/json
7979```
7980
7981```json
7982{
7983 ""last_name"": ""Lovelace"",
7984 ""email"": ""ada@example.com""
7985}
7986```
7987
7988The record `id` cannot be changed.
7989
7990Delete one record:
7991
7992```http
7993DELETE /collections/{collection}/records/{record_id}
7994Authorization: Token <token>
7995```
7996
7997Response:
7998
7999```json
8000{
8001 ""status"": ""ok"",
8002 ""collection"": ""contacts"",
8003 ""record"": {
8004 ""id"": ""c1"",
8005 ""first_name"": ""Ada"",
8006 ""last_name"": ""Lovelace""
8007 },
8008 ""deleted"": true
8009}
8010```
8011
8012Missing collections or records return `404`; unsafe collection, record, or field
8013names return `400`. Missing subscription entitlements can return
8014`402 Payment Required` when the active policy requires them. By default all
8015record mutations require the admin token. With
8016`DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true`, mutations can also be authorized
8017by persisted policy rules using `create`, `update`, and `delete`.
8018
8019If `data/schemas/{collection}.json` exists, `POST` and `PUT` mutations validate
8020known fields before the TSV write. Required fields, create defaults, basic
8021scalar types, enum values, length/numeric/pattern rules, and computed/read-only
8022fields are enforced by the server. Unknown fields are still accepted so tools
8023can add data before a schema is complete.
8024
8025When `DBBASIC_ENABLE_PERMISSION_ENFORCEMENT=true`, schema field permissions are
8026also enforced for collection records. Fields resolved as `hidden` are removed
8027from `GET` responses. Submitted fields resolved as `read` or `hidden` are
8028rejected with `403`:
8029
8030```json
8031{
8032 ""status"": ""error"",
8033 ""error"": ""Record field 'margin' is not editable for this subject"",
8034 ""code"": ""forbidden"",
8035 ""denied_fields"": [""margin""]
8036}
8037```
8038
8039Read collection record change history:
8040
8041```http
8042GET /collections/{collection}/changes?limit=100&offset=0
8043Authorization: Token <token>
8044```
8045
8046Read change history for one record:
8047
8048```http
8049GET /collections/{collection}/records/{record_id}/changes?limit=100&offset=0
8050Authorization: Token <token>
8051```
8052
8053Response:
8054
8055```json
8056{
8057 ""status"": ""ok"",
8058 ""collection"": ""contacts"",
8059 ""record_id"": ""c1"",
8060 ""changes"": [
8061 {
8062 ""change_id"": ""0f8b5e96-d8b5-4a0d-8b7d-c3c2e2092d24"",
8063 ""timestamp"": ""2026-06-29T12:00:00+00:00"",
8064 ""collection"": ""contacts"",
8065 ""record_id"": ""c1"",
8066 ""action"": ""update"",
8067 ""actor"": ""admin"",
8068 ""message"": ""Updated record"",
8069 ""changed_fields"": [""name""],
8070 ""before"": {""id"": ""c1"", ""name"": ""Ada""},
8071 ""after"": {""id"": ""c1"", ""name"": ""Ada Lovelace""}
8072 }
8073 ],
8074 ""count"": 1,
8075 ""total"": 1,
8076 ""limit"": 100,
8077 ""offset"": 0,
8078 ""has_more"": false
8079}
8080```
8081
8082Successful `POST`, `PUT`, and `DELETE` record mutations append JSONL entries
8083under:
8084
8085```text
8086data/record_changes/{collection}/changes.jsonl
8087```
8088
8089This is the durable audit and admin-history surface. When record events are
8090enabled, successful `POST`, `PUT`, and `DELETE` mutations also publish
8091`collection.record.created`, `collection.record.updated`, and
8092`collection.record.deleted` events from the same change entry.
8093
8094## Events
8095
8096Events are the daemon-compatible notification surface for triggers, listeners,
8097webhooks, and future worker-style objects. They are intentionally separate from
8098record change history. Change history is the durable audit trail; events are the
8099delivery queue.
8100
8101Collection record mutations publish metadata-only events by default. Set
8102`DBBASIC_ENABLE_RECORD_EVENTS=false` to turn that off. Event payloads include
8103the change id, collection, record id, action, actor, timestamp, and changed
8104field names; they intentionally do not copy full `before` or `after` snapshots
8105until subscriber permissions are enforced.
8106
8107The current endpoints are admin-gated:
8108
8109```http
8110GET /events?event_type=collection.record.created&limit=100&offset=0
8111DELETE /events?keep_count=1000&keep_seconds=604800
8112Authorization: Token <token>
8113```
8114
8115Response:
8116
8117```json
8118{
8119 ""status"": ""ok"",
8120 ""events"": [
8121 {
8122 ""id"": ""789f0c18-7845-4620-a2b0-8bbad752f91b"",
8123 ""event_type"": ""collection.record.created"",
8124 ""payload"": {
8125 ""change_id"": ""0f8b5e96-d8b5-4a0d-8b7d-c3c2e2092d24"",
8126 ""collection"": ""contacts"",
8127 ""record_id"": ""c1"",
8128 ""action"": ""create"",
8129 ""actor"": ""admin"",
8130 ""timestamp"": ""2026-06-29T12:00:00Z"",
8131 ""changed_fields"": [""id"", ""name""]
8132 },
8133 ""source"": ""record_changes"",
8134 ""actor"": ""admin"",
8135 ""timestamp"": 1782734400,
8136 ""created_at"": ""2026-06-29T12:00:00Z""
8137 }
8138 ],
8139 ""count"": 1,
8140 ""total"": 1,
8141 ""limit"": 100,
8142 ""offset"": 0,
8143 ""has_more"": false
8144}
8145```
8146
8147Publish one event:
8148
8149```http
8150POST /events
8151Authorization: Token <token>
8152Content-Type: application/json
8153```
8154
8155```json
8156{
8157 ""event_type"": ""collection.record.created"",
8158 ""source"": ""record_changes"",
8159 ""payload"": {""collection"": ""contacts"", ""record_id"": ""c1""}
8160}
8161```
8162
8163Successful publishes return `201`.
8164
8165Prune the delivery queue:
8166
8167```http
8168DELETE /events?keep_count=1000&keep_seconds=604800
8169Authorization: Token <token>
8170```
8171
8172Response:
8173
8174```json
8175{
8176 ""status"": ""ok"",
8177 ""retention"": {
8178 ""deleted"": 12,
8179 ""kept"": 1000,
8180 ""scanned"": 1012,
8181 ""protected"": 1,
8182 ""corrupt_deleted"": 0,
8183 ""keep_count"": 1000,
8184 ""keep_seconds"": 604800
8185 }
8186}
8187```
8188
8189`keep_count=0` disables count-based pruning. `keep_seconds=0` disables
8190age-based pruning. Subscriptions are never deleted by event pruning, and the
8191event referenced by a subscription `last_event_id` is protected to avoid
8192surprise replay. Failed delivery attempts are protected too, so the daemon and
8193Scroll can still see the pending retry event. Configure the default
8194publish/daemon cleanup policy with `DBBASIC_EVENT_KEEP_COUNT` and
8195`DBBASIC_EVENT_KEEP_SECONDS`.
8196
8197Subscriptions are stored in the same daemon-compatible `events` object state:
8198
8199```http
8200GET /events/subscriptions?event_type=collection.record.created
8201POST /events/subscriptions
8202DELETE /events/subscriptions?event_type=collection.record.created&subscriber_id=scroll
8203Authorization: Token <token>
8204```
8205
8206Create or replace a subscription:
8207
8208```json
8209{
8210 ""event_type"": ""collection.record.created"",
8211 ""subscriber_id"": ""scroll"",
8212 ""callback_url"": ""https://example.com/hooks/dbbasic""
8213}
8214```
8215
8216Response:
8217
8218```json
8219{
8220 ""status"": ""ok"",
8221 ""subscription"": {
8222 ""id"": ""scroll"",
8223 ""event_type"": ""collection.record.created"",
8224 ""callback_url"": ""https://example.com/hooks/dbbasic"",
8225 ""created_at"": 1782734400,
8226 ""created_at_iso"": ""2026-06-29T12:00:00Z"",
8227 ""created_by"": ""admin"",
8228 ""last_event_id"": null,
8229 ""delivery"": {
8230 ""status"": ""idle"",
8231 ""attempts"": 0,
8232 ""successes"": 0,
8233 ""failures"": 0,
8234 ""last_attempted_event_id"": null,
8235 ""last_attempt_at"": null,
8236 ""last_attempt_at_iso"": null,
8237 ""last_success_event_id"": null,
8238 ""last_success_at"": null,
8239 ""last_success_at_iso"": null,
8240 ""last_failure_event_id"": null,
8241 ""last_failure_at"": null,
8242 ""last_failure_at_iso"": null,
8243 ""last_status_code"": null,
8244 ""last_error"": null
8245 }
8246 }
8247}
8248```
8249
8250The daemon updates `delivery` after each callback attempt. Successful delivery
8251sets `status=ok` and advances `last_event_id`; failed delivery sets
8252`status=failed`, records the last error/status, and leaves `last_event_id`
8253unchanged so the event can be retried.
8254
8255Delivery status is exposed separately so Scroll and operator dashboards can
8256show pending, delivered, and failed work without loading event payloads or
8257callback secrets:
8258
8259```http
8260GET /events/deliveries?event_type=collection.record.created&pending=true
8261GET /events/deliveries?delivery_status=failed
8262Authorization: Token <token>
8263```
8264
8265Response:
8266
8267```json
8268{
8269 ""status"": ""ok"",
8270 ""deliveries"": [
8271 {
8272 ""id"": ""scroll"",
8273 ""subscriber_id"": ""scroll"",
8274 ""event_type"": ""collection.record.created"",
8275 ""callback_url_present"": true,
8276 ""pending"": true,
8277 ""pending_count"": 2,
8278 ""last_event_id"": null,
8279 ""next_pending_event"": {
8280 ""id"": ""789f0c18-7845-4620-a2b0-8bbad752f91b"",
8281 ""event_type"": ""collection.record.created"",
8282 ""source"": ""record_changes"",
8283 ""actor"": ""admin"",
8284 ""timestamp"": 1782734400,
8285 ""created_at"": ""2026-06-29T12:00:00Z""
8286 },
8287 ""latest_pending_event"": {
8288 ""id"": ""98b9fd7c-9e38-48a6-9134-153473c0172a"",
8289 ""event_type"": ""collection.record.created"",
8290 ""source"": ""record_changes"",
8291 ""actor"": ""admin"",
8292 ""timestamp"": 1782734460,
8293 ""created_at"": ""2026-06-29T12:01:00Z""
8294 },
8295 ""delivery"": {
8296 ""status"": ""failed"",
8297 ""attempts"": 1,
8298 ""successes"": 0,
8299 ""failures"": 1,
8300 ""last_attempted_event_id"": ""789f0c18-7845-4620-a2b0-8bbad752f91b"",
8301 ""last_attempt_at"": 1782734401,
8302 ""last_attempt_at_iso"": ""2026-06-29T12:00:01Z"",
8303 ""last_status_code"": 500,
8304 ""last_error"": ""callback failed""
8305 }
8306 }
8307 ],
8308 ""count"": 1,
8309 ""total"": 1,
8310 ""limit"": 100,
8311 ""offset"": 0,
8312 ""has_more"": false
8313}
8314```
8315
8316`callback_url` is redacted by default. Trusted detail views may request
8317`include_callback_url=true`. Event payloads are also excluded by default; use
8318`include_events=true&event_limit=10` only for trusted operator/detail views.
8319
8320## Packages
8321
8322Packages are DBBASIC bundles: objects, schemas, permissions, seed data, and
8323migrations collected under one package directory. GitHub can host the source,
8324but Scroll/Object Server should inspect and later install packages through the
8325DBBASIC API so backups, dry-runs, changelogs, and rollback points stay part of
8326the object loop.
8327
8328The first public package surface can list packages, return dry-run plans, and
8329perform conservative installs when both admin auth and the explicit package
8330install flag are enabled. Dry-runs and installs append compact package changelog
8331rows so operators can see which packages were reviewed, installed, or rejected.
8332
8333Package directory shape:
8334
8335```text
8336packages/{package_id}/
8337 dbbasic-package.json
8338 objects/
8339 schemas/
8340 permissions/
8341 seed/
8342 migrations/
8343```
8344
8345List packages:
8346
8347```http
8348GET /packages
8349Authorization: Token <token>
8350```
8351
8352Response:
8353
8354```json
8355{
8356 ""status"": ""ok"",
8357 ""packages"": [
8358 {
8359 ""id"": ""hello-world"",
8360 ""name"": ""Hello World"",
8361 ""version"": ""0.1.0"",
8362 ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."",
8363 ""status"": ""available"",
8364 ""object_count"": 1,
8365 ""schema_count"": 0,
8366 ""permission_count"": 0,
8367 ""seed_count"": 0,
8368 ""migration_count"": 0,
8369 ""dependency_count"": 0
8370 }
8371 ],
8372 ""count"": 1
8373}
8374```
8375
8376Read one manifest:
8377
8378```http
8379GET /packages/{package_id}
8380Authorization: Token <token>
8381```
8382
8383Response:
8384
8385```json
8386{
8387 ""status"": ""ok"",
8388 ""package"": {
8389 ""id"": ""hello-world"",
8390 ""name"": ""Hello World"",
8391 ""version"": ""0.1.0"",
8392 ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."",
8393 ""compatibility"": {""object_server"": "">=0.0.1""},
8394 ""dependencies"": [],
8395 ""objects"": [
8396 {""id"": ""hello_world"", ""path"": ""objects/hello/world.py""}
8397 ],
8398 ""schemas"": [],
8399 ""permissions"": [],
8400 ""seed"": [],
8401 ""migrations"": []
8402 }
8403}
8404```
8405
8406Dry-run a future install/update:
8407
8408```http
8409GET /packages/{package_id}?dry_run=true
8410Authorization: Token <token>
8411```
8412
8413Response:
8414
8415```json
8416{
8417 ""status"": ""ok"",
8418 ""dry_run"": {
8419 ""package"": {
8420 ""id"": ""hello-world"",
8421 ""name"": ""Hello World"",
8422 ""version"": ""0.1.0"",
8423 ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."",
8424 ""status"": ""available"",
8425 ""object_count"": 1,
8426 ""schema_count"": 0,
8427 ""permission_count"": 0,
8428 ""seed_count"": 0,
8429 ""migration_count"": 0,
8430 ""dependency_count"": 0
8431 },
8432 ""mode"": ""dry_run"",
8433 ""install_enabled"": false,
8434 ""safe_to_install"": true,
8435 ""objects"": [
8436 {
8437 ""id"": ""hello_world"",
8438 ""path"": ""objects/hello/world.py"",
8439 ""exists"": true,
8440 ""action"": ""create"",
8441 ""installed"": false
8442 }
8443 ],
8444 ""schemas"": [],
8445 ""permissions"": [],
8446 ""seed"": [],
8447 ""migrations"": [],
8448 ""warnings"": []
8449 },
8450 ""change"": {
8451 ""change_id"": ""0a34c2e6-3144-4900-8f99-fd7c7ec77c61"",
8452 ""timestamp"": ""2026-06-30T12:00:00+00:00"",
8453 ""package_id"": ""hello-world"",
8454 ""package_version"": ""0.1.0"",
8455 ""action"": ""dry_run"",
8456 ""actor"": ""admin"",
8457 ""message"": ""Dry run package install"",
8458 ""details"": {
8459 ""safe_to_install"": true,
8460 ""install_enabled"": false,
8461 ""objects"": {""create"": 1},
8462 ""schemas"": {},
8463 ""permissions"": {},
8464 ""seed"": {},
8465 ""migrations"": {},
8466 ""warnings"": []
8467 }
8468 }
8469}
8470```
8471
8472Install a reviewed package:
8473
8474```http
8475POST /packages/{package_id}/install
8476Authorization: Token <token>
8477Content-Type: application/json
8478
8479{""allow_replace"": false}
8480```
8481
8482Package installs require:
8483
8484```text
8485DBBASIC_ADMIN_TOKEN=...
8486DBBASIC_ENABLE_PACKAGE_INSTALLS=true
8487```
8488
8489The first install implementation is deliberately narrow:
8490
8491- object files are written under the configured object root
8492- schema JSON is validated and written under `data/schemas/`
8493- seed TSV is written only when `data/collections/{collection}/records.tsv`
8494 does not already exist
8495- replacing objects or schemas requires `{""allow_replace"": true}`
8496- package permissions and migrations are rejected until merge/run semantics are
8497 explicit
8498- a runtime restore point is created before any live object, schema, or seed
8499 file is changed
8500
8501Response:
8502
8503```json
8504{
8505 ""status"": ""ok"",
8506 ""install"": {
8507 ""package"": {
8508 ""id"": ""hello-world"",
8509 ""name"": ""Hello World"",
8510 ""version"": ""0.1.0"",
8511 ""description"": ""Small package proving DBBASIC package discovery and dry-run planning."",
8512 ""status"": ""available"",
8513 ""object_count"": 1,
8514 ""schema_count"": 0,
8515 ""permission_count"": 0,
8516 ""seed_count"": 0,
8517 ""migration_count"": 0,
8518 ""dependency_count"": 0
8519 },
8520 ""mode"": ""install"",
8521 ""install_enabled"": true,
8522 ""allow_replace"": false,
8523 ""safe_to_install"": true,
8524 ""objects"": [
8525 {
8526 ""id"": ""hello_world"",
8527 ""path"": ""objects/hello/world.py"",
8528 ""exists"": true,
8529 ""action"": ""create"",
8530 ""installed"": false,
8531 ""status"": ""written"",
8532 ""destination"": ""hello/world.py""
8533 }
8534 ],
8535 ""schemas"": [],
8536 ""permissions"": [],
8537 ""seed"": [],
8538 ""migrations"": [],
8539 ""warnings"": [],
8540 ""restore_point"": {
8541 ""path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"",
8542 ""format_version"": 1,
8543 ""created_at"": ""2026-06-30T12:01:00Z"",
8544 ""files"": 14,
8545 ""bytes"": 12345,
8546 ""warnings"": []
8547 }
8548 },
8549 ""changes"": {
8550 ""requested"": {
8551 ""change_id"": ""c0c3f4a6-8b10-47cb-8309-b381354f6cc0"",
8552 ""action"": ""install_requested""
8553 },
8554 ""installed"": {
8555 ""change_id"": ""c77681de-5b1e-4a5e-adc9-16701ca230f5"",
8556 ""action"": ""installed""
8557 }
8558 },
8559 ""restore_point"": {
8560 ""path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"",
8561 ""format_version"": 1,
8562 ""created_at"": ""2026-06-30T12:01:00Z"",
8563 ""files"": 14,
8564 ""bytes"": 12345,
8565 ""warnings"": []
8566 }
8567}
8568```
8569
8570Restore a package install restore point:
8571
8572```http
8573POST /packages/{package_id}/restore
8574Authorization: Token <token>
8575Content-Type: application/json
8576
8577{""change_id"": ""c77681de-5b1e-4a5e-adc9-16701ca230f5"", ""confirm"": ""restore-runtime""}
8578```
8579
8580Package restore requires:
8581
8582```text
8583DBBASIC_ADMIN_TOKEN=...
8584DBBASIC_ENABLE_PACKAGE_RESTORE=true
8585```
8586
8587The change id must point at a recorded package change with restore-point
8588metadata. The server restores that runtime snapshot with overwrite and
8589prune-extra enabled, then appends `restore_requested` and `rolled_back` package
8590change rows. This is intentionally a whole-runtime rollback, not a selective
8591package uninstall, and the restore path must live under the configured backup
8592directory.
8593
8594Response:
8595
8596```json
8597{
8598 ""status"": ""ok"",
8599 ""restore"": {
8600 ""backup_path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"",
8601 ""objects_dir"": ""objects"",
8602 ""data_dir"": ""data"",
8603 ""files"": 14,
8604 ""bytes"": 12345,
8605 ""overwritten"": true,
8606 ""pruned_files"": 3,
8607 ""pruned_dirs"": 2
8608 },
8609 ""restore_point"": {
8610 ""path"": ""data/backups/20260630T120100Z-package-hello-world.tar.gz"",
8611 ""format_version"": 1,
8612 ""created_at"": ""2026-06-30T12:01:00Z"",
8613 ""files"": 14,
8614 ""bytes"": 12345,
8615 ""warnings"": []
8616 },
8617 ""from_change"": {
8618 ""change_id"": ""c77681de-5b1e-4a5e-adc9-16701ca230f5"",
8619 ""action"": ""installed""
8620 },
8621 ""changes"": {
8622 ""requested"": {
8623 ""change_id"": ""9fd9b09c-22bf-41c2-b6ee-3a96ca195fef"",
8624 ""action"": ""restore_requested""
8625 },
8626 ""rolled_back"": {
8627 ""change_id"": ""9a63dcde-ec78-426b-8787-f1e8ec401190"",
8628 ""action"": ""rolled_back""
8629 }
8630 }
8631}
8632```
8633
8634Read package changelog:
8635
8636```http
8637GET /packages/{package_id}/changes?limit=100&offset=0
8638Authorization: Token <token>
8639```
8640
8641Response:
8642
8643```json
8644{
8645 ""status"": ""ok"",
8646 ""package_id"": ""hello-world"",
8647 ""changes"": [
8648 {
8649 ""change_id"": ""0a34c2e6-3144-4900-8f99-fd7c7ec77c61"",
8650 ""timestamp"": ""2026-06-30T12:00:00+00:00"",
8651 ""package_id"": ""hello-world"",
8652 ""package_version"": ""0.1.0"",
8653 ""action"": ""dry_run"",
8654 ""actor"": ""admin"",
8655 ""message"": ""Dry run package install"",
8656 ""details"": {
8657 ""safe_to_install"": true,
8658 ""install_enabled"": false,
8659 ""objects"": {""create"": 1},
8660 ""schemas"": {},
8661 ""permissions"": {},
8662 ""seed"": {},
8663 ""migrations"": {},
8664 ""warnings"": []
8665 }
8666 }
8667 ],
8668 ""count"": 1,
8669 ""total"": 1,
8670 ""limit"": 100,
8671 ""offset"": 0,
8672 ""has_more"": false
8673}
8674```
8675
8676Package ids are route-safe lowercase names such as `hello-world` or
8677`crm-starter`. Package manifests use relative paths only. Absolute paths,
8678`..`, null bytes, unsupported permission/migration writes, and unsafe
8679source/data destinations are rejected at this layer.
8680
8681## Schemas
8682
8683```http
8684GET /schemas
8685Authorization: Token <token>
8686```
8687
8688Response:
8689
8690```json
8691{
8692 ""status"": ""ok"",
8693 ""schemas"": [
8694 {
8695 ""name"": ""invoices"",
8696 ""title"": ""Invoices"",
8697 ""source"": ""manual"",
8698 ""version"": 1,
8699 ""field_count"": 4
8700 }
8701 ],
8702 ""count"": 1
8703}
8704```
8705
8706Read one schema:
8707
8708```http
8709GET /schemas/{collection}
8710Authorization: Token <token>
8711```
8712
8713Response:
8714
8715```json
8716{
8717 ""status"": ""ok"",
8718 ""message"": ""Schema updated to version 1"",
8719 ""version_id"": 1,
8720 ""collection"": ""invoices"",
8721 ""schema"": {
8722 ""name"": ""invoices"",
8723 ""title"": ""Invoices"",
8724 ""source"": ""manual"",
8725 ""version"": 1,
8726 ""fields"": [
8727 {
8728 ""name"": ""customer_id"",
8729 ""type"": ""relation"",
8730 ""required"": true,
8731 ""relation"": {""collection"": ""contacts""},
8732 ""validation"": {""not_null"": true}
8733 },
8734 {
8735 ""name"": ""total"",
8736 ""type"": ""computed"",
8737 ""required"": false,
8738 ""computed"": ""sum(line_items)""
8739 },
8740 {
8741 ""name"": ""cost_price"",
8742 ""type"": ""currency"",
8743 ""required"": false,
8744 ""permissions"": {""admin"": ""edit"", ""sales"": ""hidden""},
8745 ""ui"": {""section"": ""totals""}
8746 }
8747 ],
8748 ""field_count"": 3
8749 }
8750}
8751```
8752
8753Replace one manual schema:
8754
8755```http
8756PUT /schemas/{collection}
8757Content-Type: application/json
8758Authorization: Token <token>
8759```
8760
8761Request:
8762
8763```json
8764{
8765 ""schema"": {
8766 ""title"": ""Invoices"",
8767 ""ui"": {""default_view"": ""form""},
8768 ""views"": [{""name"": ""invoice_admin"", ""type"": ""form""}],
8769 ""fields"": [
8770 {
8771 ""name"": ""invoice_date"",
8772 ""type"": ""date"",
8773 ""required"": true,
8774 ""layout"": {""column"": 1}
8775 },
8776 {
8777 ""name"": ""margin"",
8778 ""type"": ""currency"",
8779 ""permissions"": {""admin"": ""edit"", ""sales"": ""hidden""}
8780 }
8781 ]
8782 }
8783}
8784```
8785
8786Response:
8787
8788```json
8789{
8790 ""status"": ""ok"",
8791 ""schema"": {
8792 ""name"": ""invoices"",
8793 ""title"": ""Invoices"",
8794 ""source"": ""manual"",
8795 ""version"": 1,
8796 ""fields"": [
8797 {""name"": ""invoice_date"", ""type"": ""date"", ""required"": true},
8798 {
8799 ""name"": ""margin"",
8800 ""type"": ""currency"",
8801 ""required"": false,
8802 ""permissions"": {""admin"": ""edit"", ""sales"": ""hidden""}
8803 }
8804 ],
8805 ""field_count"": 2,
8806 ""ui"": {""default_view"": ""form""},
8807 ""views"": [{""name"": ""invoice_admin"", ""type"": ""form""}]
8808 }
8809}
8810```
8811
8812List schema versions:
8813
8814```http
8815GET /schemas/{collection}?versions=true&limit=10
8816Authorization: Token <token>
8817```
8818
8819Response:
8820
8821```json
8822{
8823 ""status"": ""ok"",
8824 ""collection"": ""invoices"",
8825 ""versions"": [
8826 {
8827 ""version_id"": 2,
8828 ""timestamp"": ""2026-06-29T12:00:00"",
8829 ""author"": ""admin"",
8830 ""message"": ""Add margin field"",
8831 ""hash"": ""...""
8832 },
8833 {
8834 ""version_id"": 1,
8835 ""timestamp"": ""2026-06-29T11:50:00"",
8836 ""author"": ""admin"",
8837 ""message"": ""Initial schema"",
8838 ""hash"": ""...""
8839 }
8840 ],
8841 ""count"": 2
8842}
8843```
8844
8845Read one schema version:
8846
8847```http
8848GET /schemas/{collection}?version=1
8849Authorization: Token <token>
8850```
8851
8852Response:
8853
8854```json
8855{
8856 ""status"": ""ok"",
8857 ""collection"": ""invoices"",
8858 ""version"": {
8859 ""version_id"": 1,
8860 ""timestamp"": ""2026-06-29T11:50:00"",
8861 ""author"": ""admin"",
8862 ""message"": ""Initial schema"",
8863 ""hash"": ""..."",
8864 ""content"": ""{...}"",
8865 ""schema"": {
8866 ""name"": ""invoices"",
8867 ""title"": ""Invoices"",
8868 ""source"": ""manual"",
8869 ""version"": 1,
8870 ""fields"": [],
8871 ""field_count"": 0
8872 }
8873 }
8874}
8875```
8876
8877Rollback is non-destructive. It creates a new latest version containing the old
8878schema content and then replaces the live schema file:
8879
8880```http
8881POST /schemas/{collection}
8882Content-Type: application/json
8883Authorization: Token <token>
8884```
8885
8886Request:
8887
8888```json
8889{
8890 ""action"": ""rollback"",
8891 ""version_id"": 1,
8892 ""author"": ""admin"",
8893 ""message"": ""Restore first invoice form""
8894}
8895```
8896
8897Response:
8898
8899```json
8900{
8901 ""status"": ""ok"",
8902 ""message"": ""Rolled back schema to version 1"",
8903 ""version_id"": 1,
8904 ""new_version_id"": 3,
8905 ""collection"": ""invoices"",
8906 ""schema"": {
8907 ""name"": ""invoices"",
8908 ""title"": ""Invoices"",
8909 ""source"": ""manual"",
8910 ""version"": 1,
8911 ""fields"": [],
8912 ""field_count"": 0
8913 }
8914}
8915```
8916
8917Schema files live under:
8918
8919```text
8920data/schemas/{collection}.json
8921data/schema_versions/{collection}/metadata.tsv
8922data/schema_versions/{collection}/vN.json
8923```
8924
8925Manual schemas can be replaced through the admin-token gated `PUT` route. Writes
8926are atomic file replacements under `data/schemas/`, and each write records a
8927schema version. If a collection has no manual schema, the server may return an
8928empty derived schema for that collection so Scroll can still show the collection
8929and later attach fields. Missing schemas return `404`; unsafe schema names
8930return `400`.
8931
8932Schema `permissions` and `ui` fields are preserved for generated admin screens.
8933With permission enforcement enabled, schema `permissions` also refine record
8934reads and writes after the broader server policy has allowed the row.
8935
8936## Create Object
8937
8938```http
8939POST /objects
8940Content-Type: application/json
8941Authorization: Token <token>
8942```
8943
8944Request:
8945
8946```json
8947{
8948 ""object_id"": ""site_home"",
8949 ""code"": ""def GET(request):\n return {\""status\"": \""ok\""}\n"",
8950 ""author"": ""admin"",
8951 ""message"": ""Create home object"",
8952 ""description"": ""Optional client-provided description""
8953}
8954```
8955
8956Response:
8957
8958```json
8959{
8960 ""status"": ""ok"",
8961 ""object_id"": ""site_home"",
8962 ""message"": ""Object created: site_home"",
8963 ""version_id"": 1,
8964 ""correlation_id"": ""123e4567-e89b-42d3-a456-426614174001""
8965}
8966```
8967
8968Creation is intentionally gated. It requires `DBBASIC_ENABLE_SOURCE_WRITES=true`
8969and the same admin gate used by source updates. A configured
8970`DBBASIC_ADMIN_TOKEN` always works; an active DBBASIC session with an admin role
8971can also pass the gate only when `DBBASIC_ENABLE_SESSION_ADMIN_GATES=true`.
8972
8973Clients can choose one of these creation shapes:
8974
8975- `object_id` creates that exact object ID, such as `site_home`.
8976- `name` plus `owner_user_id` creates `u_<owner_user_id>_<name>`, such as
8977 `u_42_deals`.
8978- `name` with an admin session whose `user_id` is numeric also creates
8979 `u_<session_user_id>_<name>`.
8980- `name` without a user owner creates a system object using `name` as the object
8981 ID.
8982
8983The first source write creates version `1`, appends a `source_create` change
8984entry, and the object can be executed immediately through
8985`GET/POST/PUT/DELETE /objects/{object_id}`. Existing object IDs return `409`.
8986The `description` field is stored in source-change details for tools like
8987Scroll.
8988
8989## Execute Object
8990
8991```http
8992GET /objects/{object_id}
8993POST /objects/{object_id}
8994PUT /objects/{object_id}
8995DELETE /objects/{object_id}
8996```
8997
8998`GET` passes query parameters to the object. `POST`, `PUT`, and `DELETE` pass a
8999JSON body when present.
9000
9001Compatibility details from the working prototype:
9002
9003- `POST` merges query parameters into the JSON body without overriding body keys.
9004- `POST` with a non-JSON body passes raw bytes as the `body` field plus query
9005 parameters.
9006- `PUT` and `DELETE` reject invalid JSON bodies with `400`.
9007- request bodies over `DBBASIC_MAX_REQUEST_BYTES` are rejected with `413`.
9008- rate-limited requests are rejected with `429`.
9009- full request or object execution capacity is rejected with `503`.
9010- object executions over `DBBASIC_OBJECT_TIMEOUT_SECONDS` are rejected with `504`.
9011- `POST` with `{""action"": ""rollback""}` is reserved for rollback.
9012- `PUT /objects/{object_id}?source=true` is reserved for source updates.
9013
9014Example:
9015
9016```http
9017POST /objects/basics_counter
9018Content-Type: application/json
9019```
9020
9021```json
9022{
9023 ""action"": ""increment""
9024}
9025```
9026
9027Execution responses are object-defined. The server should pass successful object
9028results through without wrapping them in a new envelope.
9029
9030Compatibility response shapes:
9031
9032- normal dicts return JSON
9033- dicts with `content_type` and `body` return raw HTTP responses
9034- `(status, headers, body)` tuples return low-level HTTP responses
9035- strings return `text/html; charset=utf-8`
9036- bytes return `application/octet-stream`
9037
9038Example HTML object response:
9039
9040```python
9041def GET(request):
9042 return {
9043 ""content_type"": ""text/html; charset=utf-8"",
9044 ""body"": ""<!doctype html><h1>DBBASIC</h1>"",
9045 }
9046```
9047
9048Example tuple response:
9049
9050```python
9051def POST(request):
9052 return (201, [(""Content-Type"", ""text/plain"")], [b""created""])
9053```
9054
9055On server-side execution failure, return a non-2xx status with:
9056
9057```json
9058{
9059 ""status"": ""error"",
9060 ""error"": ""Execution failed: ...""
9061}
9062```
9063
9064Future structured error fields may be added, but existing clients must still be
9065able to read the `error` field.
9066
9067If `DBBASIC_OBJECT_TIMEOUT_SECONDS` is set above zero, the public ASGI server
9068runs object methods in a subprocess. When the timeout expires, the worker is
9069terminated and the client receives:
9070
9071```http
9072504 Gateway Timeout
9073```
9074
9075```json
9076{
9077 ""status"": ""error"",
9078 ""error"": ""Execution failed: GET timed out for object basics_slow after 5 seconds""
9079}
9080```
9081
9082The timeout is a wall-clock boundary. It is not a complete CPU or memory
9083sandbox.
9084
9085Objects named in `DBBASIC_TRUSTED_IN_PROCESS_OBJECTS` keep the fast in-process
9086execution path even when timeouts are enabled. This is for reviewed
9087server-owned objects, such as a high-traffic public homepage. Do not use it for
9088unreviewed user code.
9089
9090## Source
9091
9092```http
9093GET /objects/{object_id}?source=true&format=json
9094Authorization: Token <token>
9095```
9096
9097Response:
9098
9099```json
9100{
9101 ""status"": ""ok"",
9102 ""object_id"": ""basics_counter"",
9103 ""source"": ""def GET(request):\n return {\""status\"": \""ok\""}\n""
9104}
9105```
9106
9107HTML source rendering can exist for browsers, but JSON must stay available with
9108`format=json` or an `Accept: application/json` request.
9109
9110## Update Source
9111
9112```http
9113PUT /objects/{object_id}?source=true
9114Content-Type: application/json
9115Authorization: Token <token>
9116```
9117
9118Request:
9119
9120```json
9121{
9122 ""code"": ""def GET(request):\n return {\""status\"": \""updated\""}\n"",
9123 ""author"": ""api"",
9124 ""message"": ""Updated via client""
9125}
9126```
9127
9128Response:
9129
9130```json
9131{
9132 ""status"": ""ok"",
9133 ""message"": ""Code updated to version 2"",
9134 ""version_id"": 2,
9135 ""object_id"": ""u_42_deals""
9136}
9137```
9138
9139The update must save a source version, write the source file, and leave enough
9140information for clients to show the new version number.
9141
9142The current public server keeps this endpoint closed by default. Local
9143development source writes require:
9144
9145```bash
9146DBBASIC_ENABLE_SOURCE_WRITES=true
9147DBBASIC_ADMIN_TOKEN=replace-with-a-local-dev-token
9148```
9149
9150The token value shown here is a placeholder. Real deployments must generate a
9151server-specific secret outside the source tree. Production code should replace
9152that temporary gate with the real auth and permission system before this
9153endpoint is exposed to users.
9154
9155## State
9156
9157```http
9158GET /objects/{object_id}?state=true
9159Authorization: Token <token>
9160```
9161
9162Response:
9163
9164```json
9165{
9166 ""status"": ""ok"",
9167 ""object_id"": ""basics_counter"",
9168 ""state"": {
9169 ""count"": 3
9170 }
9171}
9172```
9173
9174State is currently read from:
9175
9176```text
9177data/state/{object_id}/state.tsv
9178```
9179
9180Rows may be `key<TAB>value` or `key<TAB>value<TAB>timestamp`. The public server
9181currently exposes state read-only.
9182
9183## Metadata
9184
9185```http
9186GET /objects/{object_id}?metadata=true
9187Authorization: Token <token>
9188```
9189
9190Response:
9191
9192```json
9193{
9194 ""status"": ""ok"",
9195 ""object_id"": ""basics_counter"",
9196 ""metadata"": {
9197 ""object_id"": ""basics_counter"",
9198 ""source_path"": ""basics/counter.py"",
9199 ""owner"": ""system"",
9200 ""kind"": ""system"",
9201 ""last_modified"": 1760000000.0,
9202 ""state_count"": 1,
9203 ""state_keys"": [""count""],
9204 ""log_count"": 3,
9205 ""file_count"": 1,
9206 ""version_count"": 2
9207 }
9208}
9209```
9210
9211The public server reports source paths relative to the object source root, not
9212absolute local filesystem paths. Metadata may grow over time. Existing clients
9213expect the top-level `metadata` field.
9214
9215## Files
9216
9217```http
9218GET /objects/{object_id}?files=true
9219Authorization: Token <token>
9220```
9221
9222Response:
9223
9224```json
9225{
9226 ""status"": ""ok"",
9227 ""object_id"": ""site_home"",
9228 ""files"": [
9229 {
9230 ""name"": ""report.txt"",
9231 ""size"": 1200,
9232 ""modified"": 1760000000.0
9233 }
9234 ],
9235 ""count"": 1
9236}
9237```
9238
9239Download one object-owned file:
9240
9241```http
9242GET /objects/{object_id}?file=report.txt
9243Authorization: Token <token>
9244```
9245
9246The server reads files from:
9247
9248```text
9249data/files/{object_id}/
9250```
9251
9252Filenames are validated before filesystem access. Empty names, absolute paths,
9253null bytes, and `..` traversal are rejected with `400`. Missing files return
9254`404`.
9255
9256This public object route stays read-only for files. Scroll and operator
9257dashboards should use `/admin/files*` for file inventory, downloads, and the
9258separately gated admin write workflow.
9259
9260## Logs
9261
9262```http
9263GET /objects/{object_id}?logs=true&format=json&limit=100
9264Authorization: Token <token>
9265```
9266
9267Response:
9268
9269```json
9270{
9271 ""status"": ""ok"",
9272 ""object_id"": ""basics_counter"",
9273 ""logs"": [
9274 {
9275 ""entry_id"": ""a1"",
9276 ""timestamp"": ""2026-01-01T00:00:00"",
9277 ""level"": ""DEBUG"",
9278 ""message"": ""GET completed successfully"",
9279 ""method"": ""GET"",
9280 ""status"": ""success"",
9281 ""duration_ms"": ""1.25""
9282 }
9283 ],
9284 ""count"": 1
9285}
9286```
9287
9288Optional query parameters:
9289
9290- `level` filters by exact log level, such as `INFO` or `ERROR`.
9291- `limit` defaults to `100` in the public ASGI server.
9292
9293Logs are currently read from:
9294
9295```text
9296data/logs/{object_id}/log.tsv
9297data/logs/{object_id}/log-*.tsv
9298data/logs/{object_id}/log-*.tsv.gz
9299```
9300
9301The TSV header defines the fields. The normal fields are `entry_id`,
9302`timestamp`, `level`, and `message`; object code may add extra columns such as
9303`method`, `status`, `duration_ms`, `error_type`, `error`, or `user_id`.
9304
9305The public ASGI server appends one execution log entry after each object method
9306run. Successful runs use `level=DEBUG` and `status=success`; failed runs use
9307`level=ERROR`, `status=error`, and include error fields. The endpoint itself
9308remains read-only.
9309
9310Log storage may rotate and gzip old logs on disk. The HTTP response shape does
9311not change; clients still receive JSON log entries. Transport compression for
9312large log responses should be handled through normal HTTP compression, such as a
9313reverse proxy honoring `Accept-Encoding`, rather than changing this JSON shape.
9314
9315## Source Changes
9316
9317```http
9318GET /objects/{object_id}?source_changes=true&limit=100&offset=0
9319Authorization: Token <token>
9320```
9321
9322Response:
9323
9324```json
9325{
9326 ""status"": ""ok"",
9327 ""object_id"": ""basics_counter"",
9328 ""changes"": [
9329 {
9330 ""change_id"": ""123e4567-e89b-42d3-a456-426614174000"",
9331 ""timestamp"": ""2026-01-01T00:00:00"",
9332 ""object_id"": ""basics_counter"",
9333 ""action"": ""source_update"",
9334 ""version_id"": 2,
9335 ""from_version_id"": null,
9336 ""actor"": ""api"",
9337 ""message"": ""Updated via client"",
9338 ""correlation_id"": ""123e4567-e89b-42d3-a456-426614174001"",
9339 ""details"": {}
9340 }
9341 ],
9342 ""count"": 1,
9343 ""total"": 1,
9344 ""limit"": 100,
9345 ""offset"": 0,
9346 ""has_more"": false
9347}
9348```
9349
9350History is newest first and does not include source content. Use the versions
9351endpoint to inspect or restore source snapshots. Source changes are the operator
9352activity timeline for source edits and rollbacks.
9353
9354## Object Changes
9355
9356```http
9357GET /objects/{object_id}?changes=true&kind=source&limit=100&offset=0
9358Authorization: Token <token>
9359```
9360
9361This returns the normalized source/file timeline for one object. The admin
9362inspection equivalent is:
9363
9364```http
9365GET /admin/objects/{object_id}?changes=true&limit=100&offset=0
9366Authorization: Token <token>
9367```
9368
9369Response:
9370
9371```json
9372{
9373 ""status"": ""ok"",
9374 ""object_id"": ""basics_counter"",
9375 ""changes"": [],
9376 ""count"": 0,
9377 ""total"": 0,
9378 ""limit"": 100,
9379 ""offset"": 0,
9380 ""has_more"": false
9381}
9382```
9383
9384`kind` can be omitted or set to `source` or `file`. Use `/admin/changes` when a
9385global activity stream needs record and package changes too.
9386
9387## Versions
9388
9389```http
9390GET /objects/{object_id}?versions=true&limit=10
9391Authorization: Token <token>
9392```
9393
9394Response:
9395
9396```json
9397{
9398 ""status"": ""ok"",
9399 ""object_id"": ""basics_counter"",
9400 ""versions"": [
9401 {
9402 ""version_id"": 2,
9403 ""timestamp"": ""2026-01-01T00:00:00"",
9404 ""author"": ""api"",
9405 ""message"": ""Updated via client"",
9406 ""hash"": ""...""
9407 }
9408 ],
9409 ""count"": 1
9410}
9411```
9412
9413History is newest first and does not include source content.
9414
9415## Specific Version
9416
9417```http
9418GET /objects/{object_id}?version=2
9419Authorization: Token <token>
9420```
9421
9422Response:
9423
9424```json
9425{
9426 ""status"": ""ok"",
9427 ""object_id"": ""basics_counter"",
9428 ""version"": {
9429 ""version_id"": 2,
9430 ""timestamp"": ""2026-01-01T00:00:00"",
9431 ""author"": ""api"",
9432 ""message"": ""Updated via client"",
9433 ""hash"": ""..."",
9434 ""content"": ""def GET(request):\n return {\""status\"": \""ok\""}\n""
9435 }
9436}
9437```
9438
9439## Rollback
9440
9441```http
9442POST /objects/{object_id}
9443Content-Type: application/json
9444Authorization: Token <token>
9445```
9446
9447Request:
9448
9449```json
9450{
9451 ""action"": ""rollback"",
9452 ""version_id"": 1,
9453 ""author"": ""api"",
9454 ""message"": ""Rollback to version 1""
9455}
9456```
9457
9458Response:
9459
9460```json
9461{
9462 ""status"": ""ok"",
9463 ""message"": ""Rolled back to version 1"",
9464 ""version_id"": 1,
9465 ""new_version_id"": 3,
9466 ""object_id"": ""u_42_deals""
9467}
9468```
9469
9470Rollback is non-destructive. The server should create a new latest version from
9471the old source, write that source to the object file, and preserve history.
9472
9473The historical response reports the requested rollback version in `version_id`.
9474Future responses may add `new_version_id`, but must not remove `version_id`
9475without a client migration.
9476
9477The current public server supports `new_version_id` and uses the same temporary
9478source-write gate as source updates:
9479
9480```bash
9481DBBASIC_ENABLE_SOURCE_WRITES=true
9482DBBASIC_ADMIN_TOKEN=replace-with-a-local-dev-token
9483```
9484
9485The token value shown here is a placeholder. Real deployments must generate a
9486server-specific secret outside the source tree. Production code should replace
9487that temporary gate with the real auth and permission system before rollback is
9488exposed to users.
9489
9490## Destroy Object
9491
9492```http
9493DELETE /objects/{object_id}?destroy=true
9494Authorization: Token <token>
9495```
9496
9497Response:
9498
9499```json
9500{
9501 ""status"": ""ok"",
9502 ""message"": ""Object destroyed: u_42_deals"",
9503 ""object_id"": ""u_42_deals""
9504}
9505```
9506
9507Destructive delete must be authenticated and authorization-checked.
9508
9509## Routing Compatibility
9510
9511The historical server supports explicit station routing:
9512
9513```http
9514GET /objects/{object_id}@{station_id}
9515```
9516
9517The public v1 server does not need to promise distributed correctness, but code
9518should not accidentally treat `@` routing as a normal object ID. If station
9519routing is disabled, return a clear error instead of executing the wrong object.
9520
9521## Compatibility Rules
9522
9523- Keep `/objects` and `/objects/{object_id}` as the main HTTP surface.
9524- Keep query flags such as `source=true`, `state=true`, and `versions=true`.
9525- Keep `PUT /objects/{object_id}?source=true` for source updates.
9526- Keep rollback as `POST /objects/{object_id}` with `action=rollback`.
9527- Keep top-level fields existing clients read: `status`, `error`, `object_id`,
9528 `objects`, `source`, `state`, `metadata`, `logs`, `versions`, `version`,
9529 `version_id`, `message`, and `count`.
9530- Add fields only when old clients can ignore them safely.
9531- Do not require clients to move to a new path such as `/api/v1` without a
9532 compatibility layer.
9533"
9534app-packages The App Suite — Applications as Installable Packages Platform & Extras 900 How DBBASIC's fifty-two built-in applications are built entirely from schema, permission rules, and an optional page object — with no server code required. "DBBASIC ships a suite of everyday applications. None of them required
9535server code: each app is a package of **schema + permission rules +
9536(optionally) one page object**. The server provides the primitives —
9537records, permissions, search, sharing, files, AI — and apps are data
9538that composes them.
9539
9540```mermaid
9541flowchart LR
9542 P[""Package""] --> S[""Schema<br/>fields, forms, views,<br/>search, transitions""]
9543 P --> R[""Permission rules<br/>owner rows, public flags,<br/>project sharing""]
9544 P --> O[""Page object<br/>(optional; talks to the<br/>records API as the visitor)""]
9545 S --> C[""Collection records""]
9546 R --> C
9547 O --> C
9548 C --> API[""Records API""]
9549 C --> MCP[""MCP tools""]
9550 C --> SRCH[""Global search""]
9551 C --> AI[""AI chat tools""]
9552 C --> UI[""Generated forms/lists<br/>(Scroll)""]
9553```
9554
9555Because every surface reads the same schema and passes the same
9556permission engine, installing an app instantly gives it: a validated
9557records API, generated forms and lists in Scroll, global search, MCP
9558tools for agents, AI chat access, and a live web page — with row-level
9559visibility rules applied identically everywhere.
9560
9561Each app's page is also **realtime**: it subscribes to its collection
9562over the shared `/ws` websocket and re-renders when a record it can see
9563changes, so a note added in one tab (or by an agent) appears in another
9564with no reload — one line per page, permission-filtered like everything
9565else. See [design-system](/docs/design-system) and the
9566[realtime contract](/docs/http-api-contract#realtime-push-websocket).
9567
9568## The Suite
9569
9570Fifty-two packages. They are grouped here by what they are FOR rather than
9571alphabetically, because the grouping is the argument: the money apps share
9572one ledger and one composer, the commerce apps share one catalogue and one
9573stock log, and nothing in either column needed server code to exist.
9574
9575### The original slice — records, sharing, capture
9576
9577| Package | Collections | Page | Notable |
9578|---|---|---|---|
9579| `app-projects` | projects, project_access | `/projects` | The hub other apps relate to; self-serve sharing grants |
9580| `app-notes` | notes | `/notes` | Public/private per note; permalinks; quick capture |
9581| `app-tasks` | tasks | `/tasks` | Status lifecycle enforced by a `transitions` map; assignee access |
9582| `app-contacts` | contacts, organizations, interactions, tags | `/contacts` | CRM; interactions use the `feed` list mode |
9583| `app-articles` | articles | `/articles` | Publish flips one boolean; anonymous-readable blog |
9584| `app-links` | links | `/links` | Bookmarks with tags |
9585| `app-events` | events | `/calendar` | Calendar; purpose enum; project sharing |
9586| `app-files` | files | `/files` | Uploads with quotas; downloads governed by the metadata record |
9587| `app-templates` | templates | `/templates` | Reusable bodies; the thing `app-runner` executes |
9588| `app-timers` | time_logs, rate_cards | `/time-logs` | Time tracking against tasks, and what an hour costs |
9589| `app-forum` | forum_categories, forum_topics, forum_replies | `/forum` | Threaded discussion on the same permission engine |
9590| `app-worker` | profiles, follows, profile_comments | `/profile/edit` | Public profiles and a follow graph |
9591
9592### Money — one ledger, one composer, one set of books
9593
9594| Package | Collections | Page | Notable |
9595|---|---|---|---|
9596| `app-finance` | denominations, rates, fin_accounts, fin_journals, fin_journal_lines, fin_recurring, fin_closed_periods, expenses | `/accounts`, `/journals`, `/trial-balance`, `/statements`, `/expenses` | Double-entry: every other money app composes journals through one idempotent composer, provenance-stamped |
9597| `app-invoices` | invoices, invoice_lines | `/invoices` | Line discounts and tax through the shared `object_lines` arithmetic |
9598| `app-payments` | payments, refunds | `/payments` | Cash/accrual basis is configuration; bounces reverse rather than delete |
9599| `app-billing` | wallets, wallet_entries, billing_plans, subscriptions, usage_events, usage_summaries | `/customer-funds` | Prepaid balances as a LIABILITY: a top-up is not revenue until the work is done, and the reconciliation proves it |
9600| `app-banking` | value_accounts, value_account_counts, bank_import_profiles, bank_statement_imports, bank_lines | `/reconcile` | Statement import and matching — the anti-fraud keystone |
9601| `app-entities` | entities | `/entities` | Multi-entity: separate books under one login |
9602| `app-promotions` | promotions, promotion_redemptions | `/promotions` | Basket-level codes, rounded once, with every blocker returned at once |
9603| `app-disputes` | disputes | `/disputes` | Chargebacks and what was said about them |
9604| `app-integrity` | anchors | `/ledger-integrity` | A daily digest per money ledger, so a break can be bracketed to a day |
9605| `app-notary` | notarizations | `/notary` | Store a digest, check it later — no custody of the document |
9606
9607### Commerce — one catalogue, one stock ledger
9608
9609| Package | Collections | Page | Notable |
9610|---|---|---|---|
9611| `app-catalog` | products, locations, stock_moves, backorders, reorder_suggestions | `/products`, `/stock`, `/locations`, `/reorder`, `/backorders` | Stock levels are never stored, only folded from an immutable move log; a variant is a product |
9612| `app-orders` | orders, order_lines | `/orders` | Line discounts, modifiers and tax through `object_lines` |
9613| `app-shop` | carts, cart_items | `/shop` | Storefront and checkout: stock, price and slot are all re-checked at commit |
9614| `app-shipping` | shipments, shipment_lines | `/shipments`, `/pick-list` | Partial fulfilment is ordinary business |
9615| `app-receiving` | receipts, receipt_lines | — | Goods in, at invoice cost, feeding valuation |
9616| `app-returns` | return_authorizations | `/returns` | The reverse trip, authorized before it arrives |
9617| `app-pickup` | pickup_slots | `/pickup` | Collection windows generated before dawn, so the picker is never empty |
9618| `app-kitchen` | — | `/kitchen` | The make-line view of open orders |
9619
9620### AI, agents and jobs
9621
9622| Package | Collections | Page | Notable |
9623|---|---|---|---|
9624| `app-runner` | template_runs | `/runs` | The job engine: hold money at submission, execute or submit-and-poll, settle once. A run a provider still holds is never abandoned |
9625| `app-agents` | agent_registry | `/agents` | Who is operating this server, and whether it and its own scheduled work are still answering |
9626| `app-shell` | shell_preferences, shell_commands, ai_usage | `/shell`, `/talk` | Talk to everything; drop a file in and the model can read it |
9627| `app-intake` | scans | `/scans` | A photographed receipt becomes a SUGGESTION, never a posting |
9628| `app-documents` | sent_documents | `/documents` | One renderer for every business document |
9629
9630### Platform surfaces — the uniform layers
9631
9632| Package | Collections | Page | Notable |
9633|---|---|---|---|
9634| `app-theme` | — | `/appearance` | The generative renderer: list, detail and form from any schema |
9635| `app-views` | views | `/flow`, `/urls` | Pages as records — a dashboard is data, not code |
9636| `app-nav` | nav_entries, attention_sources, attention_counts | `/` | The home page and every app's ""needs a human"" count |
9637| `app-settings` | user_prefs, feature_flags, ai_prices, app_settings, unit_prices | — | Server config, per-user prefs, and the price tables |
9638| `app-share` | record_shares | — | Per-record grants that can only ever narrow |
9639| `app-notify` | notify_rules | — | Declarative event → notification |
9640| `app-rollup` | rollup_definitions | — | Derived aggregates, defined as data |
9641| `app-materialize` | materialize_definitions | — | Scheduled generation of records from a definition |
9642| `app-thread` | thread_comments | — | The comments capability, for any collection |
9643| `app-collab` | task_comments, feed_posts, notifications | — | Comments, the human+agent coordination feed, notifications |
9644| `app-feed` | — | `/feed` | The composed activity stream |
9645| `app-activity` | — | `/activity` | The change log, rendered |
9646| `app-dashboard` | — | `/dashboard` | Counts and queues at a glance |
9647| `app-messaging` | message_threads, messages, message_recipients, message_drafts | `/inbox` | Internal mail on the same permission engine |
9648| `app-email` | email_outbox | — | Outbound mail as records, sent by a pass |
9649| `app-analytics` | page_views, conversions | `/analytics`, `/visitors` | Visitors, bots and us, counted apart |
9650| `app-privacy` | — | `/privacy` | What is collected and why, as a page |
9651
9652Packages without a page are managed entirely through generated UIs,
9653the shell, and MCP — proof that the schema contract carries a whole
9654app's interface.
9655
9656## The Pattern
9657
9658Every app repeats the same shape. To build a new one, copy `app-notes`
9659and follow [package authoring](/docs/package-authoring):
9660
96611. **Schema** — fields with semantics (`docs/schema-forms.md`):
9662 types, enums, relations, validation, `search.fields`,
9663 `views.list_mode`, and `transitions` for lifecycle fields.
96642. **Rules** — the standard grants, composed per record:
9665 - owner rows: `{""row_filter"": {""owner_id"": ""$user_id""}}`
9666 - public rows: `{""principal"": ""public"", ""row_filter"": {""is_public"": ""true""}}`
9667 - project sharing: `{""row_filter"": {""project_id"": ""$accessible_projects""}}`
9668 - a public `execute` grant for the page object, which renders a
9669 sign-in prompt to anonymous visitors
96703. **Page object** — one Python file returning HTML; its JavaScript
9671 talks to `/collections/{name}/records` and `/api/search` with the
9672 visitor's session cookie, so the page holds no data access of its
9673 own.
9674
9675Visibility rules compose per record: a note can be readable because
9676you own it, because it is public, or because it sits in a project
9677shared with you — three rules, no visibility code in the app.
9678
9679## Relations Between Apps
9680
9681Apps point at each other with validated `relation` fields, not an
9682association framework: tasks and notes relate to projects, interactions
9683to contacts, time logs to tasks, comments to tasks. A relation is a
9684pointer plus a display hint; writes validate that the target record
9685exists, and nothing joins, cascades, or lazy-loads.
9686"
9687analytics Analytics — First-Party, Server-Side, and Honest About It Platform & Extras 910 How the object server's built-in, first-party analytics works, what it can see that JavaScript tags can't, and exactly what it costs in accuracy and consent. "Most platforms forgot analytics. You build a site, and to find out
9688whether anyone came you paste in someone else's JavaScript, which reports
9689a number you cannot check, derived from data you no longer solely hold.
9690
9691This server records its own traffic. Every request appends one row to
9692`page_views`; `/visitors` folds those rows into people, bots and your own
9693operations, counted apart. No third party is involved, no script is
9694served to the browser, and the data is a TSV on your disk that you can
9695open with `less`.
9696
9697This page argues why that is better, and states plainly what it costs —
9698because it does cost something, and a page that only listed advantages
9699would be exactly the kind of analytics writing this system exists in
9700opposition to.
9701
9702## The structural argument: JavaScript can only see what ran JavaScript
9703
9704This is the whole thing, and it is not a matter of degree.
9705
9706A tag-based analytics product observes a page view when a script loads
9707and executes. It therefore cannot see:
9708
9709- **Ad-blocked visitors.** The most technical part of any audience blocks
9710 the most common tags by default. For developer tools, that is precisely
9711 the people evaluating you.
9712- **Clients with no JavaScript** — curl, RSS readers, link previewers,
9713 text browsers, accessibility tooling, anything scripted.
9714- **Bots and scanners.** Invisible to the tag, and they are a large share
9715 of what actually arrives.
9716- **Errors.** A 404, a 500, a page that failed before the tag ran. The
9717 requests most worth knowing about are the ones least likely to report
9718 themselves.
9719- **Anything that is not a page.** API calls, webhooks, feeds, downloads.
9720
9721The server, by definition, sees every request it answered. It cannot be
9722blocked from knowing what it did.
9723
9724## The ownership argument
9725
9726A hosted analytics vendor holds a complete copy of your traffic and
9727returns you a summary, subject to sampling, thresholds, and categories
9728like ""not provided"". You are shown a fraction of your own data, and the
9729fraction is chosen by the party who kept all of it.
9730
9731Here the raw rows are yours, on your disk, in a format `grep` can read.
9732Nothing is sampled. Nothing is withheld. No third party receives a copy
9733of your visitors' requests as a side effect of you wanting to count them,
9734which is also the part your visitors would care about if anybody asked
9735them.
9736
9737## What it captures that a tag cannot
9738
9739Because it sits at the server, the same data answers questions analytics
9740products usually cannot:
9741
9742- **Attack traffic.** Scanners walking a list of URLs that do not exist
9743 here, credential-stuffing attempts, payloads arriving in headers. One
9744 Log4Shell probe with an AWS-credential exfiltration template in its
9745 `Referer` was found this way — inert against a Python server, but worth
9746 seeing.
9747- **Which of your own machines did what.** Deploys and scripted calls are
9748 labelled `operator`, not hidden, so you can tell your traffic from
9749 everyone else's rather than guessing.
9750- **Failure shape.** A flood of 404s from one address is a security
9751 signal that a page-view counter would never show you.
9752- **Multi-site truth.** One process may answer for several hostnames; the
9753 `host` is recorded, so ""did anyone read the pitch"" and ""did anyone try
9754 the product"" are different questions with different answers.
9755
9756## Who counts as a person
9757
9758The hard part of first-party analytics is not collection, it is
9759classification. Traffic is sorted into three kinds and never merged:
9760
9761**operator** — yours. An owner IP, or a request carrying the admin token.
9762Labelled rather than hidden: seeing your own visit is how you confirm the
9763page is recording anything at all, and a counter that silently drops the
9764only traffic you can verify is a counter you cannot trust.
9765
9766**bot** — a client that never behaved like a person. The test is
9767behavioural, not a user-agent blocklist, because the blocklist is always
9768out of date and the behaviour is not. An address must have loaded a real
9769page **and** done one thing a prober does not: opened a second distinct
9770page, arrived from a referrer, or carried a session. A declared crawler
9771stays a bot even when it fetches something real — Googlebot is honest
9772about what it is, and honesty should not promote it. So does anything
9773sending an attack payload in a header.
9774
9775**visitor** — everything else. A person, probably.
9776
9777The rule earned its strictness. A first version counted any address that
9778loaded a real page, and `/` answers 200 to everyone: on a real week that
9779promoted nearly half the ""visitors"" — 187 fell to 92 — from probers who
9780touched the front page and left. A metric that turns a quiet day into a
9781busy one is worse than no metric, because it misleads exactly when
9782somebody is relying on it.
9783
9784## What it costs — read this part
9785
9786**""Unique"" means a distinct IP address, and an IP is not a person.** An
9787office behind one connection counts once. A phone moving between wifi and
9788cellular counts twice. A CGNAT range counts thousands of people as a
9789handful.
9790
9791That is the honest unit available to a server that sets no *cross-site*
9792cookie and fingerprints nobody, and the alternative — a durable
9793cross-site identifier — is a thing this system deliberately does not
9794have. The `/visitors` page says so on the page, in words, rather than
9795presenting a confident number.
9796
9797A first-party visitor cookie (below) improves this for anyone who accepts
9798one, and does not change the ceiling: it makes returning visitors
9799countable, and it still cannot tell a phone from a laptop.
9800
9801Also absent, and not coming:
9802
9803- **No cross-device identity.** The same person on a laptop and a phone
9804 is two visitors.
9805- **No demographics, interests, or inferred attributes.** Those come from
9806 a surveillance network; there isn't one.
9807- **No client-side interaction data** — scroll depth, rage clicks, time
9808 on page — unless an app deliberately posts an event. The server knows
9809 what it served, not what someone did with it afterwards.
9810- **Bounded history.** `page_views` is capped by age *and* row count
9811 (`DBBASIC_ANALYTICS_RETENTION_DAYS`, `DBBASIC_ANALYTICS_MAX_ROWS`),
9812 because a log nobody told how large it may get is a log that fails on
9813 the worst day rather than an ordinary one. Long-horizon trends need
9814 rolling those up before they age out.
9815
9816If you need cross-device attribution and demographic segments, this will
9817not give them to you, and no amount of configuration will.
9818
9819## Operating it
9820
9821- **Off by default.** `DBBASIC_ANALYTICS=on` is an operator choice,
9822 because it adds a write to every request.
9823- **The visitor cookie is off by default and switched on separately**
9824 (`DBBASIC_ANALYTICS_VISITOR_COOKIE=on`), because it is the one thing
9825 here that stores an identifier on a visitor's device. See the cookie
9826 section below for what that commits you to; `/privacy` states it for
9827 your visitors, folded from these same settings.
9828- Asset, health and polling paths are skipped; 4xx and 5xx are
9829 deliberately **kept**, because a 404 flood is the signal.
9830- The write is offloaded off the event loop and is best-effort:
9831 analytics must never break a response.
9832- Retention runs as a scheduled daemon pass with both bounds.
9833- Rollups (`rollup_definitions`) group the raw rows into top paths, top
9834 addresses, status codes and daily counts. Set `min_group_size` on the
9835 path and address rollups — without it, every one-hit scanner probe
9836 becomes its own row and the ""top paths"" report becomes a copy of the
9837 log rather than a summary of it.
9838
9839## Goals and funnels
9840
9841`conversions` is a collection: `event_type`, optional
9842`session_id`/`user_id`, and a JSON `metadata` blob, append storage,
9843written server-side by whichever app owns the goal. A rollup can group it
9844by `event_type`.
9845
9846**It is written now, and the implementation was as small as promised** —
9847because this system is full of transitions and a conversion is simply a
9848transition worth counting. `system_record_conversion` (app-analytics)
9849declares `HANDLES` on the changes that already happen and records four
9850goals:
9851
9852| event_type | the transition |
9853|---|---|
9854| `order_confirmed` | a sale order reached `confirmed` or anywhere past it |
9855| `order_collected` | a pickup order was actually handed over |
9856| `payment_received` | money arrived against an invoice, and did not bounce |
9857| `scan_confirmed` | a human confirmed a scanned document into the books |
9858
9859Nothing new is observed. Four facts the server already knew simply never
9860landed anywhere a report could see them.
9861
9862It is a **reaction**, in the same posture as `system_order_email`:
9863post-commit, best-effort, never a gate, and it cannot fail the write that
9864confirmed the order. On a box with no app-analytics installed it counts
9865nothing and says which — because silence would read as ""already counted"",
9866and the shop would never find out it had no numbers.
9867
9868**Idempotent by provenance.** The change dispatcher is at-least-once, and
9869`status is confirmed` is a state an order *sits in* rather than an edge it
9870crosses, so the handler sees the same order on every later write. Each
9871conversion carries its source in the metadata blob
9872(`{""source"": ""orders/ord-1""}`) and one whose `(event_type, source)` pair
9873already exists is not written again. A double-counted conversion is a
9874permanent overcount in a report nothing downstream will ever correct.
9875
9876**These four rows carry no `session_id`, and that is stated rather than
9877hidden.** A back-office transition has no browser anywhere near it — an
9878order confirmed by staff the next morning was not a page view — and the
9879basket's `carts.session_token` is a different identifier in a different
9880namespace, so stamping it here would merge two populations and produce a
9881funnel that looks stitched and is not. The funnel counts them in
9882`unthreaded_conversions` and says so on the page. The field is there for
9883the day an app records a goal from a request that did carry the cookie.
9884
9885### The folds
9886
9887`object_conversions.py` is the pure half — rows in, numbers out, no I/O,
9888the same posture as `object_visitors.py`. Three folds, each of which
9889returns its own caveat **as data** so a surface cannot render the number
9890without being handed the qualification to render beside it:
9891
9892- **`funnel(page_views, conversions, steps)`** — ordered steps, each a
9893 path prefix (`/shop`) or an `event_type`. Per step: distinct visitors
9894 who reached it and drop-off from the step before. It is a *sequence* —
9895 a visitor counts at step N only if they reached N−1 and then did N
9896 afterwards, because somebody who bought last week and browsed today did
9897 not convert today. Threaded by `session_id`, falling back to the IP
9898 address for rows that carry none, and the result reports
9899 `ip_stitched` / `ip_stitched_pct` per step and overall. A funnel that
9900 hides how much of itself is guesswork is the kind of analytics this
9901 page exists in opposition to.
9902- **`returning_visitors(page_views, window_days)`** — new versus
9903 returning by token, returning `floor: True` and the caveat text itself.
9904 See the ceiling below.
9905- **`time_to_conversion(page_views, conversions)`** — median and spread of
9906 days between a visitor's first recorded page and their conversion, with
9907 the two ways of not knowing reported apart: `unthreaded` (no token at
9908 all) and `no_first_view` (the first visit aged out of a bounded log).
9909 Dropping the second silently would bias the median toward fast
9910 conversions — the number would improve every time retention was
9911 shortened, which is the most dangerous shape a metric can have.
9912
9913### Describing your own funnel
9914
9915A funnel is a settings row, not code. `analytics.funnel_steps` in
9916`app_settings` is a JSON list:
9917
9918```json
9919[""/shop"", ""/checkout"", ""order_confirmed""]
9920```
9921
9922A step beginning with `/` is a path prefix; anything else is a conversion
9923`event_type`. The long form takes labels:
9924`[{""label"": ""Browsed"", ""path"": ""/shop""}, …]`. With nothing configured,
9925`/visitors` shows what to configure — built from the paths and event types
9926this server has actually recorded, so it can be copied rather than
9927invented — instead of an empty table that reads as ""nobody converted"". A
9928malformed setting is reported as a malformed setting, because a
9929misconfigured funnel and a funnel nobody entered look identical and only
9930one of them is your fault.
9931
9932## The visitor cookie — the one addition worth making, and its limits
9933
9934Everything above is what a server can know without asking the browser to
9935remember anything. There is exactly one addition that changes the class
9936of question this can answer, and it deserves stating carefully, because
9937it is also the point at which analytics products usually stop being
9938honest.
9939
9940Before it, `page_views.session_id` read a cookie that nothing ever set,
9941so every visitor looked new on every visit and a journey could only be
9942stitched by address — precisely wrong for the multi-visit paths that
9943matter. Somebody reads the pitch on Monday, comes back Thursday, and
9944tries the product: that was three strangers.
9945
9946**A first-party cookie with a long-but-not-indefinite expiry fixes it.**
9947It is `dbbasic_visitor`, set by `object_server` on a page response when
9948the cookie is switched on, analytics is on, and the request carried none;
9949its value is stamped into `page_views.session_id` on the same request that
9950mints it — so the first page of a visit is threaded rather than the
9951second. It unlocks the questions a site actually has:
9952
9953- **New versus returning** — the most useful ratio a site has, and
9954 invisible without it.
9955- **Return paths** — what somebody read first, and what brought them
9956 back.
9957- **Time to conversion** — days between first visit and purchase, which
9958 is what tells you whether your funnel is a funnel or a queue.
9959- **Real funnels**, stitched across visits rather than within one.
9960
9961### It is OFF by default, and that is the whole consent posture
9962
9963`DBBASIC_ANALYTICS_VISITOR_COOKIE` gates it, and unset — the default —
9964**no visitor cookie is set at all**. The server behaves exactly as it did
9965before the cookie existed: visitors counted by address, nothing written
9966to anybody's device, no ePrivacy consent trigger, no banner needed
9967anywhere in the world.
9968
9969That is deliberate, and the reason is worth stating because it is the
9970whole design. The consent rule people call ""the cookie law"" is the
9971ePrivacy Directive, not GDPR, and it fires on *storing or reading
9972something on a device that is not strictly necessary for the service the
9973user asked for*. Everything else this server puts in a browser —
9974`dbbasic_session`, the shop's `cart` — is strictly necessary for a thing
9975the person actively asked for, and `page_views` is server-side logging
9976rather than device storage. **One cookie is the entire consent problem**,
9977so the default that needs the least law is the one where it does not
9978exist.
9979
9980Turning it on is a deliberate operator act, and *that* is the moment the
9981obligation appears. So the obligation travels with the setting rather
9982than living only here: the daemon's and the server's boot lines say, in
9983words, that it ""stores an identifier on a visitor's device; in the EU/UK
9984that requires consent"" (`object_analytics.OBLIGATION`), and `/privacy`
9985grows a row in its cookie table with the real `Max-Age` the moment it is
9986switched on.
9987
9988It is a separate switch from `DBBASIC_ANALYTICS` on purpose. Recording
9989what the server did and asking a browser to remember who it is are
9990different acts under different instruments, and one flag for both would
9991mean an operator who wanted traffic numbers could not have them without
9992also taking on a consent obligation. With the cookie off, everything on
9993`/visitors` still works except new-versus-returning — which is what
9994Plausible and Fathom sell as a feature.
9995
9996DNT and Sec-GPC suppression is unchanged and applies on top: switching
9997the cookie on does not switch those off.
9998
9999### The five rules, for a box that switches it on
10000
10001The rules that keep it honest are not optional, and none of them is left
10002to judgement: each is enforced in code and pinned by a test in
10003`tests/test_conversions.py`, because a rule that lives only in a document
10004is a rule the next person breaks in one line without noticing.
10005
100061. **First-party and same-site only.** Set by this server, sent only to
10007 this server, `Path=/`, `SameSite=Lax`, `HttpOnly`. Not readable by
10008 JavaScript, not a cross-site identifier — the same posture the shop's
10009 basket cookie already takes, and the reason page objects are never
10010 handed the identity session cookie. It is also withheld from page
10011 objects themselves (`_object_visible_cookies` drops it alongside
10012 `dbbasic_session`): they have no use for it, since the server stamps
10013 it into `page_views` itself, and a token only the capture hook can see
10014 cannot be joined to anything by a package the operator installed.
100152. **An opaque random token.** `secrets.token_urlsafe`. It says *this
10016 browser has been here before*. It carries no name, no email, no
10017 account, and no meaning outside this site's own logs — and nothing
10018 derived from the request, because a token hashed from IP and user
10019 agent would be a stable identifier nobody could ever clear.
100203. **A stated expiry.** `DBBASIC_ANALYTICS_VISITOR_DAYS`, default 180 —
10021 long enough to see a return visit, short enough to lapse rather than
10022 follow somebody for years. Junk and non-positive values fall back to
10023 the default rather than being read as unbounded: ""indefinite"" is how a
10024 session identifier quietly becomes a permanent one.
100254. **It never becomes identity.** A signed-in `user_id` is already
10026 recorded separately, and the visitor token must not be joined to it to
10027 retroactively de-anonymise earlier browsing. That join is the exact
10028 move that turns analytics into surveillance, and it is one line of
10029 code away at all times — so `build_page_view` and `build_conversion`
10030 write **at most one of the two columns**, preferring the anonymous
10031 one. A signed-in member's traffic is still counted; it simply is not
10032 labelled with who they are. Where there is no token (a DNT visitor, an
10033 API call) `user_id` is written as before, because there is nothing for
10034 it to be correlated with.
100355. **Honoured refusal.** `DNT: 1` and `Sec-GPC: 1` mean no cookie is set —
10036 not a shorter one, not an anonymised one. Those visitors are still
10037 **counted** by the IP rule, because refusing to be remembered is not
10038 refusing to be counted; they simply appear new each visit, which is the
10039 correct consequence of having asked not to be remembered. `DNT: 0` is
10040 consent and an absent header is no statement at all; reading either as
10041 a refusal would mean this server never remembered anybody.
10042
10043It is also never set on a path a browser did not choose to visit: assets,
10044health checks and polling endpoints (`should_capture`) and the API and
10045collection surfaces a script talks to (`is_page_path`). An API client is
10046not a browser and cannot be a returning visitor; handing it a cookie only
10047writes an identifier into somebody's cron job. A 404 *is* a page — often
10048the first one a visitor sees — so those are threaded like any other.
10049
10050The ceiling worth knowing: a cleared cookie is a new visitor, a second
10051browser is a second visitor, and a phone is a third. Returning-visitor
10052counts are a **floor, never a census** — they undercount, which is the
10053right direction for a number to be wrong in. `/visitors` says so on the
10054page, next to the number, and reports how many addresses had no cookie at
10055all so the size of the blind spot is on the same screen as the ratio it
10056is missing from.
10057
10058Funnels are then an ordinary fold over `page_views` grouped by that
10059token — `object_conversions.funnel` — and `conversions.session_id` is the
10060field waiting for the goals that are recorded from a real request.
10061
10062## Where to look
10063
10064- `/visitors` — people, bots and your own traffic, by hour and by day,
10065 with referrers, landing pages and a per-host split; then new versus
10066 returning, goals by `event_type`, your configured funnel and time to
10067 conversion. Deliberately one page rather than a second `/funnels`: ""how
10068 many came"", ""how many had been before"" and ""how many bought"" are one
10069 question asked at increasing depth, and a conversion rate whose
10070 denominator lives on another screen is a conversion rate nobody
10071 computes.
10072- `/analytics` — path, status and referrer detail from the rollups.
10073- `/privacy` — what visitors are told, folded from these same settings by
10074 `app-privacy`, so the retention on this page and the retention on that
10075 one are the same number read from the same place. `/cookies` is its
10076 cookie table alone.
10077- `object_analytics.py` — capture, configuration and the visitor-cookie
10078 decision.
10079- `object_visitors.py` — the classifier and the traffic fold, pure and
10080 tested.
10081- `object_conversions.py` — the goal record, the funnel, new-versus-
10082 returning and time-to-conversion, pure and tested.
10083- `packages/app-analytics/objects/system/record_conversion.py` — the
10084 handler that turns transitions into `conversions` rows, idempotently."
10085comparisons Comparisons: What DBBASIC Deletes, and What That Costs Platform & Extras 920 An honest comparison of DBBASIC against Django/Rails, Kubernetes, JS meta-frameworks, SaaS suites, and metered AI platforms — what each one makes you carry, and what deleting it costs. "Rails announced itself by giving developers an enemy: enterprise
10086ceremony. It was right, and it won. Twenty years later the ceremony
10087grew back in new clothes — deploy pipelines for one-line changes,
10088migration rituals, SaaS rent on your own data, and metered AI where
10089every question pays a vendor.
10090
10091This page compares honestly. Every framework here is good software
10092built by serious people; the differences are about what each one makes
10093you carry. Where DBBASIC deletes something, the cost of the deletion is
10094stated too. Claims about speed and effort come with receipts from this
10095project's own history.
10096
10097## The one-line version of each
10098
10099| Stack | What you carry | What DBBASIC deletes | What deleting costs |
10100|---|---|---|---|
10101| Django / Rails | ORM, migrations, deploys, middleware, signals, app server + DB | All of it: records are files, schemas are live data, changes serve on the next request | No relational joins; big-data queries need indexes you don't have yet |
10102| Enterprise / k8s | Clusters, YAML, service meshes, CI/CD, staging ladders | One VM, one process, systemd | No horizontal scaling story; deliberate single-box boundary |
10103| JS meta-frameworks | Build steps, bundlers, hydration, node_modules | Pages are one Python file emitting HTML + small fetch calls | No component ecosystem; you write your own widgets or generate them from schemas |
10104| SaaS suites (Basecamp, Notion, Airtable) | Monthly rent, their servers, their data model, their API limits | Self-hosted; your disk, your TSVs, greppable | You are the operator: backups and TLS are your (automated) chores |
10105| No-code platforms | A ceiling: when the builder can't express it, you're stuck | Schemas generate the easy 90%; real Python objects catch the rest | Objects are code; someone (or some AI) still writes them |
10106| Metered AI platforms | Per-seat AI pricing, vendor lock, their tools only | Bring your own key, pick any model, hand it your tools, audited as you | You pay your provider directly; no bundled ""free"" AI |
10107| Hosted analytics (GA & friends) | A tag in every page, a vendor's complete copy of your traffic, and a sampled summary back | The server counts what it answered; raw rows are a TSV on your disk | No cross-device identity, no demographics, no client-side interaction data; ""unique"" means a distinct IP |
10108
10109## Against the ceremony, specifically
10110
10111**Migrations.** Django asks: edit models.py, generate a migration,
10112review it, run it in each environment, in order, forever. DBBASIC asks:
10113PUT the schema; it versions itself; rollback is one call. The cost:
10114no schema-enforced relational integrity beyond `relation` existence
10115checks — the engine validates pointers, not cascades. That is a
10116stated design position, not an accident.
10117
10118**Deploys.** The receipt: during one working day this project
10119shipped twelve application packages, an AI runtime, file storage, and
10120a sharing model to a live server — the home page changed four times,
10121one page object shipped five times, and the only restarts were for the
10122core server file, each a few seconds. Every change was versioned and
10123reversible. The equivalent Django day is spent watching pipelines.
10124
10125**Middleware and signals.** A large Django application at scale can
10126easily run six or more middleware layers and hundreds of lines of
10127signal handlers on every write. Here a request is a router match, a
10128cached file read, permission string checks, JSON out. That is most of
10129why a small Python file server outruns a ""more powerful"" stack: the
10130fastest layer is the one that is not there.
10131
10132**""But will it scale?""** Rails answered this correctly in 2005 and we
10133repeat it: optimize for the loop first. DBBASIC adds a number — TSV
10134scans in page cache are instant to roughly hundreds of thousands of
10135rows per collection, which is beyond most personal and team software,
10136and per-collection files mean a hot collection can gain an index later
10137without a platform rewrite.
10138
10139## What Rails-era slogans become here
10140
10141- *Convention over configuration* → **Schemas declare semantics,
10142 never widgets.** One declaration drives forms, tables, validation,
10143 search, AI tools, and permissions on every surface.
10144- *Don't repeat yourself* → **One permission engine, every surface.**
10145 A row filter written once binds the web page, the API, search,
10146 files, MCP agents, and AI tool calls identically.
10147- *Less software* → **Apps are data.** Eleven of twelve suite apps
10148 contain zero server code; several contain no code at all.
10149- *Optimize for programmer happiness* → **Optimize for the loop.**
10150 Edit → live → inspect → undo, in seconds, with history. For humans
10151 and for AI agents, which compounds it: the machine that writes the
10152 code is also freed from the ceremony.
10153- *Opinionated software* → unchanged, and still true. If you want
10154 microservices, joins, or a component framework, this is honestly not
10155 your server.
10156
10157## The new argument Rails could not make
10158
10159Rails freed the developer from ceremony. DBBASIC also frees the
10160*software* from rent. A person with a small VM gets the suite, the
10161data in files they can read, an AI operator that uses their own key
10162and obeys their own permissions, and an MCP surface any agent can
10163drive — MIT licensed, small enough to read end to end. The enemy is
10164not another framework. It is the idea that your own tools should bill
10165you monthly to change."
10166logic-decisions Business Logic Decisions Design Decisions 1000 A living log of where business-logic rules go and why, covering doctrine on stamping vs. deriving values, time-driven state, immutable money, and integer-denomination amounts. "The living log of *where rules go and why* — the companion to `business-logic-patterns.md` (the placement guide) the same way `ui-decisions.md` companions the generative renderers. Because the primitives are shared, a placement decision here is not a per-module choice — it applies to every module at once, and an inconsistent placement is a bug, not a style preference.
10167
10168Format per decision: **Decision · Rationale · Applies to · Status.**
10169
10170## Decided
10171
10172### 1. Stamp point-in-time facts; derive live facts.
10173
10174- **Decision:** A value that is a fact about a *moment* (the tax rate an invoice was issued under, the price a line was sold at, the owner who created a record) is **stamped into the record at write time** — by the server or a pre-write hook — and never recomputed from today's reference data. A value that should always reflect *current* stored state (an invoice total, a payment balance, a full name) is **derived** by a formula/rollup and recomputes when inputs change.
10175- **Rationale:** Reference data changes; history must not. Deriving yesterday's invoice tax from today's rate table silently rewrites the past — the classic sales-tax bug. Conversely, stamping a total that should track its lines guarantees drift. The question ""moment-fact or live-fact?"" decides stamp vs derive every time.
10176- **Applies to:** tax rates, sold-at prices, FX rates on transaction dates, ownership; vs. totals, balances, counts, display names.
10177- **Status:** doctrine. First stamping precedent: server-side `owner_id`; first derivation adopters: fin_journals totals, contacts.full_name.
10178
10179### 2. Time-driven state belongs to the daemon; never to read-time checks.
10180
10181- **Decision:** When the passage of time changes a record's meaning (an invoice becomes *overdue*, a task becomes *stale*, an approval expires), a **daemon scheduled pass** makes the transition through the ordinary write/transition machinery — the state is *stored*. No surface ever computes ""is it overdue?"" at read time.
10182- **Rationale:** Nothing writes when time passes, so materialize-on-write can't see it — and read-time checks multiply into disagreeing copies (list says overdue, report says current). A stored transition is permission-checked, transition-guarded, realtime-pushed, notify-able, and audited — one truth for every surface. The daemon's `after_hours` auto-transitions already established the pattern.
10183- **Applies to:** aging/overdue states, expiries, escalation steps, SLA timers.
10184- **Status:** doctrine; aging/dunning are the next adopters (need the payments collection first).
10185
10186### 3. Money never mutates; it moves.
10187
10188- **Decision:** Corrections to money (and inventory) are **new, append-only compensating records** — a refund pointing at its payment, a reversing journal, a stock move — never edits to the original. Invariants on the movement (""refund ≤ payment − prior refunds"", ""journal balances"") are pre-write **hooks**; displayed positions (""refunded_cents"", ""on-hand"") are **rollups**.
10189- **Rationale:** Auditability is structural, not procedural: if the original can't change, the history is the ledger. Every mature money system converges here (double-entry is 700 years of this doctrine); systems that edit payment rows in place cannot answer ""was this refund correctly credited?"" because the question has no record to point at.
10190- **Applies to:** payments/refunds, journals (posted = immutable + reversals), stock_moves; `storage: append` is the substrate.
10191- **Status:** doctrine. Journals + stock_moves live it today; refunds adopt it when payments land.
10192
10193### 4. Extract the primitive after the pattern repeats — never before.
10194
10195- **Decision:** New business logic is first written **concretely** in the cheapest correct primitive for that one module — usually a hook (three lines of real code) or a plain object. Only after substantially the same shape appears in **two or three real modules** is it promoted to a declarative schema key, and the concrete versions are then deleted onto it.
10196- **Rationale:** This is how every good layer here was actually built: four hand-rolled comment tables → `capabilities.comments`; eight bespoke view pages → one detail renderer; one balance rule → the hook primitive; the hook's repeated shapes → formulas and rollups. The graveyard of ""business operating systems"" is full of the opposite order: a workflow/rules engine designed first, with reality forced through it — and the first rule that doesn't fit becomes the wall. An abstraction extracted from three working instances fits by construction. (UI precedent: ui-decisions #10.)
10197- **Applies to:** every ""should this be declarative?"" debate. The hook is the staging area; schema keys are the graduation.
10198- **Status:** doctrine — the house method, now named.
10199
10200### 5. State freezes fields — in hooks now, a schema key when it repeats.
10201
10202- **Decision:** ""A posted invoice's amounts are immutable"" (and every rule of the shape *fields X are locked when status = Y*) is expressed in each module's pre-write hook for now: reject when `existing.status` is in the frozen set and `changes` touches a locked field. When the third module writes that same hook, promote it to a declarative field/schema key (working name `locked_when`) and delete the hooks onto it, per #4.
10203- **Rationale:** It's clearly a repeating shape, but guessing the declarative form before seeing real instances is how bad keys get designed — does it lock per-field or per-schema? Does admin bypass? Does it compose with transitions? The hooks will tell us.
10204- **Applies to:** posted journals/invoices, completed orders, archived records.
10205- **Status:** pattern named; first hook instances pending (journals are the natural first).
10206
10207### 6. Gates before the write; reactions after — never crossed.
10208
10209- **Decision:** A pre-write hook may reject or stamp but never notify, email, or call external systems. Notifications and side effects run **after** the write via notify_rules / trigger objects / connectors, and never block or undo it.
10210- **Rationale:** A gate with side effects leaks that a denied write was attempted, couples save latency to SMTP, and can half-happen. A reaction that gates would mean the write's fate depends on a mail server. The two postures also fail differently by design: gates fail **closed** (a broken gate rejects), reactions and derived values fail **soft** (a broken formula stores empty; a failed notify retries).
10211- **Applies to:** every hook, every notify rule, every connector.
10212- **Status:** doctrine (established in `event-hooks-decisions.md`; restated here as the load-bearing boundary of the whole map).
10213
10214### 7. Generated documents carry provenance and face the same gates.
10215
10216- **Decision:** Every machine-composed record (a journal from a payment, a reversal, a recurring adjusting entry) stamps `generated_from` with its source (""payments/{id}"", ""reversal:{id}"", ""fin_recurring/{id}:{period}""), composers check that stamp before composing (replays are no-ops), and a generated entry passes the same balance verification a human entry would — composers re-verify because storage-layer writes bypass the HTTP-only hook.
10217- **Rationale:** Idempotency by provenance makes event redelivery, daemon restarts, and manual re-runs safe by construction — no dedup tables, no distributed locks; the ledger itself remembers what was generated. And a machine allowed to skip the gates would make generated books less trustworthy than hand-kept ones — the opposite of the point.
10218- **Applies to:** system_books composers, action_reverse_journal, fin_recurring_runner; every future composer.
10219- **Status:** doctrine, shipped with the books spine.
10220
10221### 8. Independent evidence is never edited to agree with our records
10222
10223- **Decision:** Records authored by an outside party — imported bank statement lines today, provider payout feeds and scanned documents next — are stored append-only with the original text preserved verbatim, and their factual fields are immutable after the write. What an operator may add is their *own* work about that evidence: what it matched, how it was resolved, a correction recorded as a new record. Corrections to the evidence itself come from re-importing a corrected file.
10224- **Rationale:** The reason comparing our books to a bank statement is the strongest practical control is that the bank's copy was written by someone with no stake in our numbers. The moment a line can be edited, that independence evaporates and the check becomes self-certification — reconciliation degenerates into making the two sides match rather than explaining why they differ. This is also why the *act* of reconciling is an ordinary attributed update rather than a silent background match: reconciliation performed invisibly by the same hands that move the money is the classic embezzlement pattern, and an attributed change log makes who-decided-what structural instead of procedural.
10225- **Applies to:** `bank_lines` (hook-enforced immutability of posted_on/amount/description/raw/identity fields), imported statement provenance (`bank_statement_imports`); the same posture is required of scanned-document intake and provider payout imports when they land.
10226- **Corollary — a check that cannot run must say so:** an import records which self-checks *ran*, not only which passed, so a statement that carried no balances never reads as trustworthy as one whose arithmetic was verified. Silent skipping is how controls rot.
10227- **Status:** doctrine, shipped with bank import.
10228
10229### 9. Reactions belong to the write path; storage-level writers must say so
10230
10231- **Decision:** Event handlers dispatch on the HTTP write path only. An object that writes through `object_records` directly (a runner, an action) does **not** trigger `HANDLES` reactions, so if such a write needs its reaction it must perform it explicitly — using the *shared* primitive with the *same* provenance marker the handler would have used, never a private copy of the logic.
10232- **Rationale:** Sharing the marker makes the two paths idempotent against each other: whichever runs first wins, the other is a no-op. Copying the logic instead would fork the rule and guarantee eventual divergence.
10233- **Instances so far (counting toward #4):** (1) `action_apply_count` composes its own inventory-variance journal; (2) `action_resolve_bank_line` composes the NSF bounce reversal under `payments/{id}:bounced`, the marker `system_books` also uses.
10234- **At the third instance, extract instead of working around:** the real fix is a daemon pass that dispatches handlers from the record-change log — the same shape `process_notifications` already uses to avoid depending on synchronous dispatch. Do not write a third workaround.
10235- **Status:** observed twice; extraction pending the third.
10236
10237### 10. An amount is an integer in its denomination's smallest unit
10238
10239- **Decision:** Money is stored as a whole number of the smallest unit of its own denomination, and a `denominations` record declares what that unit is (USD scale 2, JPY scale 0, BTC scale 8, ETH scale 18, XAU scale 4). Floats never appear; `Decimal` is used only at the edges to convert human text to and from integers. Amounts in different denominations are never added — combining them requires converting through a rate **stamped at the transaction moment** (#1), which is a deliberate act.
10240- **Rationale:** This is the generalization of the founding ""money is integer cents"" rule rather than a replacement for it — cents were always *USD minor units*. A business holds value in several denominations at once (a bank balance, bitcoin, a stablecoin, a box of cash, gift cards, maybe a weight of metal), and they divide differently. The alternative — a per-denomination `decimal_places` field beside one fixed-scale decimal column — is what a predecessor system did, and it lets a schema *declare* more precision than it can *hold*: the first 18-decimal asset truncates silently, while every ordinary dollar carries six meaningless digits where rounding dust collects. Integer minor units are exact by construction, have no ceiling (Python ints are unbounded), and are what payment processors and chains already use.
10241- **Corollary — rounding is a decision, not a parse.** Converting text that does not fit the scale (0.005 USD) raises; a caller that genuinely wants rounding calls `quantize_minor` and says so. Half a cent silently becoming a cent is how a per-row error turns into a real number at scale.
10242- **Applies to:** `object_money.py`, the `denominations` collection; new value-bearing collections use `*_minor` + a denomination. Existing USD-only collections keep `*_cents`, which is honest for them — a big-bang rename buys nothing.
10243- **Status:** doctrine, shipped with denominations; the value-account layer it founds is specified in `plan/value-accounts-and-denominations-spec.md`.
10244
10245## How to use this
10246
10247- Placing a new rule? Walk `business-logic-patterns.md`'s flowchart; if the placement needed a judgment call, record it here.
10248- If two modules place the same kind of rule differently, that's a bug: pick one, log it, and fix the outlier.
10249- Counting instances for #4: when you write the second concrete copy of a shape, note it here so the third triggers extraction.
10250"
10251ui-decisions UI Interaction Decisions Design Decisions 1010 A living log of the small, repeated interaction decisions the generative UI makes — detail vs edit, page vs in-place, when a list caps, when a row links. "A living log of the small, repeated interaction decisions the generative UI
10252makes — detail vs edit, page vs in-place, when a list caps, when a row links.
10253The companion to the design-system guide (`design-system.md`): that doc is the *architecture* (semantics →
10254tokens → renderers, ""define once, project everywhere""); this doc is the
10255*behavior* those renderers agree on.
10256
10257**Why write these down.** Because the UI is one generative renderer, a decision
10258here is not a per-screen choice — it applies to *every* app at once. The value
10259is not picking the clever option on each screen; it's that the same rule holds
10260everywhere, so nothing surprises you. When a decision is inconsistent across
10261screens, that is a bug (a copy leaked), not a style preference. Most of these
10262were found by screenshotting the running product and noticing the inconsistency.
10263
10264Format per decision: **Decision · Rationale · Applies to · Status.** Add new
10265ones as they're settled; move Open Questions up as they're decided.
10266
10267---
10268
10269## Decided
10270
10271### 1. Detail is a page (its own URL); edit and create are in-place.
10272- **Decision:** Opening a record's *detail* navigates to `/{collection}/{id}`.
10273 *Editing* or *creating* a record happens in-place (an inline form on the
10274 current page), never a separate page.
10275- **Rationale:** Detail is a *destination* — it must be linkable, bookmarkable,
10276 back-button-able (""everything has a url""), and it can hold a **composed view**
10277 (detail block + related records + aggregate) a modal can't. Edit/create is a
10278 transient *action* — you must not lose your list position or scroll to do it.
10279- **Applies to:** every list/board row and card (→ detail page); every `+ New X`
10280 button and every row `✎` (→ in-place form).
10281- **Status:** shipped. Rows/cards link to `/{collection}/{id}`; `+ New`/`✎`
10282 reveal the generated form in place.
10283
10284### 2. Reports and logs have no detail link and no row actions.
10285- **Decision:** A list over a generated/read-only collection (a rollup target,
10286 a log) shows neither edit/delete buttons nor a click-through to detail.
10287- **Rationale:** A rollup row (top IP → hits) has no detail page and can't be
10288 edited — the next recompute overwrites it. Offering either is a dead end.
10289- **Applies to:** any list block with `row_actions: false` (which also implies
10290 `link: false`).
10291- **Status:** shipped (analytics reports).
10292
10293### 3. Lists render-cap at 50 rows, with Show-all.
10294- **Decision:** A list renders at most 50 rows (overridable per block via
10295 `row_limit`) with a **Show all / Show fewer** toggle; the data is already
10296 fetched, so expanding is free.
10297- **Rationale:** Without a cap, a busy collection (a real-traffic rollup, a big
10298 browse) renders a 50,000-px page. Found by screenshotting live `/analytics`.
10299- **Applies to:** every list surface in every app, automatically.
10300- **Status:** shipped.
10301
10302### 4. Every record has a reachable URL; detail routes accept any id.
10303- **Decision:** A detail route is `/{collection}/{id}` with **no `:uuid`
10304 constraint** — any id segment routes to the detail view. Records still get
10305 UUIDs on create; the route simply doesn't *reject* other ids.
10306- **Rationale:** ""Everything has a url"" must hold for *every* record, including
10307 ones with friendly/seed ids. Once rows and cards link (decision 1), a
10308 `:uuid`-only route 404s exactly the records a demo shows first. A missing
10309 record is the view's own not-found state, not a routing 404.
10310- **Applies to:** every detail route in every package. (The `:uuid` matcher
10311 still exists for a route that deliberately wants to enforce shape.)
10312- **Status:** shipped.
10313
10314### 5. Board columns flex to fit; scroll only when genuinely too many.
10315- **Decision:** Board columns are `flex: 1 1 200px` (min 200, max 340), not a
10316 fixed width. A few columns fill the container; the board scrolls **internally**
10317 (never the page) only when columns would go below their min width.
10318- **Rationale:** Fixed-width columns overflowed the container and forced a
10319 *page-level* horizontal scrollbar for a mostly-empty board. The page must
10320 never scroll sideways.
10321- **Applies to:** every board (kanban / lead-pipeline).
10322- **Status:** shipped.
10323
10324### 6. A report renders as a sorted top-N.
10325- **Decision:** A list block can declare `sort_by` (+ `sort_dir`) and
10326 `row_limit` so it renders a true top-N by a real field (top IPs by hits desc),
10327 not insertion order.
10328- **Rationale:** ""Top"" must mean *ranked*, or the report is noise.
10329- **Applies to:** analytics and any report-shaped list.
10330- **Status:** shipped.
10331
10332### 7. A data-dense collection renders as a live table (`list_mode: ""table""`).
10333- **Decision:** A schema (or a view's list block) can declare `list_mode:
10334 ""table""` to render its `list_fields` as a dense HTML table — sortable column
10335 headers, one row per record, cells formatted by field type (money `_cents` →
10336 `$`, boolean → Yes/No, datetime → relative). It reuses the same
10337 `startRowList` engine as the rich-row list, so it **inherits for free**: the
10338 search box, the 50-row cap + Show-all (#3), client-side sort (#6),
10339 row→detail navigation (#1), and — the differentiator — **realtime
10340 re-render** on the change log. A table here is not a static snapshot the way
10341 a conventional server-rendered table library produces — it updates live as records change.
10342- **Rationale:** Rich rows (title + snippet) suit content/feeds; they waste
10343 space and hide structure for inherently tabular data (invoices, contacts,
10344 links, a rollup). One renderer, chosen per collection by a single schema key,
10345 keeps every table consistent instead of each app hand-rolling one.
10346- **Applies to:** any collection whose index goes through `dbbasicList`; opt in
10347 with `views.list_mode: ""table""` + `list_fields`. Header click toggles
10348 sort field/dir; the whole row is the detail link.
10349- **Status:** shipped. Live on `/articles` (Title · Published On · Published).
10350
10351### 8. The generated form matches a hand-built one (spacing + themed selects).
10352- **Decision:** The generative form uses **one** spacing mechanism (grid
10353 `gap`), collapses a field's error line until there's an actual message
10354 (`.err:empty { display: none }`), and themes every `<select>` itself
10355 (`appearance: none` + our own chevron).
10356- **Rationale:** The generated form was looser than a bespoke one for two
10357 invisible reasons — it inherited *both* the grid gap and the global
10358 `.stack > * + *` margin (doubled space), and reserved a blank 1rem error
10359 line under every field (~44px between fields vs ~14px). And an unstyled
10360 `<select>` renders as Safari's beveled native control but Chrome's flat one,
10361 so the same page looked different per browser. A generic renderer has to be
10362 *at least as good* as the hand-built page it replaces, or migrating onto it
10363 (decision below) is a regression. Found by comparing the bespoke Projects
10364 form to the generated Notes form side by side.
10365- **Applies to:** every generated form (`window.dbbasicForm`) and every
10366 `<select>` in the product.
10367- **Status:** shipped. Measured 14px inter-field gap, no reserved error space.
10368
10369### 9. A board collection can toggle to a table (persisted per-collection).
10370- **Decision:** A collection declared `list_mode: ""board""` that also has
10371 `list_fields` shows a small **Board / Table** segmented control; the table is
10372 the same generative table from decision #7. The choice is stored per
10373 collection in `localStorage` and survives navigation.
10374- **Rationale:** A kanban is the right default for a workflow surface, but the
10375 same records are often better *scanned* as a dense sortable table — the
10376 classic kanban⇄list toggle. Both renderers already exist behind one
10377 `resolveListMode`, so it's a switch, not new rendering. Switching **reloads**
10378 rather than swapping in place: `dbbasicSubscribe` has no unsubscribe, so an
10379 in-place swap would leave the previous mode's change-log handler live and
10380 double-render. One page load = one subscription.
10381- **Applies to:** any board-declared collection with `list_fields` (tasks
10382 today). Tree/calendar toggles are a possible follow-on, same mechanism.
10383- **Status:** shipped (tasks Board/Table).
10384
10385### 10. Make the generic layer ≥ the bespoke page before deleting the bespoke.
10386- **Decision:** When a hand-built page does something the generic renderer
10387 doesn't (or does *better*), fix the generic renderer to that bar **first**,
10388 then delete the bespoke page onto it — never the reverse order.
10389- **Rationale:** A bespoke page that looks better than the generic one isn't
10390 just debt to delete; it's a spec for a quality gap in the shared layer. The
10391 Projects form (tight) was showing what the generated form (loose, #8) should
10392 be. Delete-first would have made Projects *worse*; fix-first made every
10393 form better and Projects free.
10394- **Applies to:** every ""migrate a bespoke page onto the generator"" task
10395 (Projects done; analytics/commerce pages next).
10396- **Status:** doctrine. First applied migrating `site/projects.py` (~60 lines
10397 of hand-rolled table/form/fetch deleted) onto `dbbasicList` + `dbbasicForm`.
10398
10399### 11. A collection filters by declaring `filter_fields` (works on every mode).
10400- **Decision:** A schema's `views.filter_fields` renders a **filter bar** above
10401 the list — an enum field → a select of its options, a boolean → Yes/No. A
10402 pick ANDs into the fetch `where` and reloads the active mode. Filters compose
10403 with the caller's own `where` (entity scope, a parent FK) and with the
10404 client-side search box; a **Clear** button appears once anything is set.
10405- **Rationale:** The list renderer had sort + live but not filters — the third
10406 leg of ""a table, but generative and alive."" Filtering is server-side (the
10407 same `field=value` path the board already uses, applied after the permission
10408 row filter), so a filtered list can only ever *narrow* what the viewer may
10409 already see — never widen it. One schema key, filters on list, table, and
10410 board at once.
10411- **Applies to:** any collection that declares `filter_fields` (tasks:
10412 status + urgency today). Text is left to the search box; a date-range control
10413 is the next increment (needs the dotted-operator `gte/lte` query path).
10414- **Status:** shipped. Verified server-side narrowing (6 tasks → 2 on
10415 status=open) and that the bar renders in board and table.
10416
10417### 12. A collection opts into behaviors via `capabilities` (not just display).
10418- **Decision:** Beyond the display keys (`list_mode`/`filter_fields`/…), a
10419 schema declares generic *behaviors* under a root-level `capabilities` key,
10420 wired by the platform rather than hand-built per app. First one:
10421 `capabilities.comments: true` → the detail page grows a comment thread
10422 (`window.dbbasicThread`, backed by the polymorphic `thread_comments`
10423 collection), no per-view block, no per-app comment table.
10424- **Rationale:** Comments existed **four times** (task_comments, thread_comments,
10425 profile_comments, interactions) — the same ""a record has a thread hung off
10426 it"" copied per app. That's a workaround for a layer that should be uniform.
10427 One capability flag collapses them: the widget is polymorphic
10428 (parent_collection + parent_id), so any collection gets comments by declaring
10429 one key. This is the *behavior/connection* layer sitting on top of the
10430 display layer — the same ""define once, project everywhere,"" for what a
10431 collection *does* and *connects to*, not just how it looks.
10432- **Applies to:** any collection (tasks today). `capabilities` is whitelisted in
10433 schema normalization and surfaced in `/api/schema` so the client can wire it.
10434- **Status:** shipped (two capabilities now). `comments`: task_comments
10435 migrated onto thread_comments, related block removed, live widget post
10436 verified. `attachments`: `window.dbbasicAttachments` (served at
10437 `/attachments`) uploads/lists/downloads/deletes files for a record via the
10438 polymorphic `files.(parent_collection, parent_id)` pair — no per-collection
10439 FK; tasks adopts both, the old `related:files` block removed, a live
10440 widget upload verified. `renderDetail` mounts them via one
10441 `maybeMountCapabilities`. `shareable` is the foundational one — it changes
10442 the permission check, not just a widget. `capabilities.shareable`
10443 + a `{id: $shared_records}` read rule let a record's OWNER grant other users
10444 access via the `record_shares` collection. `window.dbbasicShare` (owner-only,
10445 renders nothing for non-owners) writes grants only through the owner-checked
10446 `/api/share` endpoint (authorized against the TARGET record's ownership — a
10447 cross-collection fact a row_filter can't express, and the thing that stops
10448 self-serve escalation). A new collection-aware `$shared_records` row-filter
10449 variable (the collection is threaded into resolution) keeps a share on a task
10450 from leaking a same-id row in another collection. Security boundary verified
10451 live: grantee reads (200), non-grantee denied (403), non-owner can't grant
10452 (403). All three capabilities now mount in one detail page from flags alone.
10453- **Gotchas found building it:** a field that is `required` **and** `read_only`
10454 can never be created through the HTTP write path (only a server-side
10455 `preserve_read_only` bypass) — `thread_comments.parent_*` had to drop
10456 `read_only` to be settable on create. And `owner_id` (a `public:hidden` field)
10457 is server-set from the session; a client value is rejected, and it's redacted
10458 from other readers — so comments stamp a separate `author_name` for
10459 attribution.
10460
10461### 13. Every data collection carries created_at; detail always shows it.
10462- **Decision:** `created_at` (datetime, read-only, server-owned) is a baseline
10463 field on every user-facing data collection, and the detail generator always
10464 surfaces `created_at`/`updated_at` as record metadata — even when
10465 `detail_fields` curates the main fields — the way the list row already shows
10466 a timestamp.
10467- **Rationale:** ""When was this created"" is baseline info every record should
10468 carry and every detail should show; `detail_fields` is for curating the
10469 *content* fields, not for suppressing metadata. Found by spotting records
10470 with no timestamp (a collection missing the field entirely — e.g. projects)
10471 and detail pages hiding it (tasks/contacts/forum_topics curated it out).
10472- **Applies to:** all user-facing collections (config/system ones —
10473 feature_flags, user_prefs, shell_*, *_definitions, project_access,
10474 ai_prices, dbbasic_probe — intentionally omit it). Detail metadata append is
10475 automatic in the shared renderer.
10476- **Status:** shipped. Added the field to projects + organizations/files/
10477 events/interactions/time_logs/tags/templates; detail appends the timestamps.
10478 (Old records predate the field and read it back empty — new ones get a
10479 server-set value.)
10480
10481### 14. A relation field shows its target's name, not the raw FK id.
10482- **Decision:** A field with `relation:{collection, display_field}` renders the
10483 referenced record's display value (""Acme Corp"") in the table cell and board
10484 card — never the raw id (""o-acme""). Each referenced collection is fetched once
10485 per render surface to build an id→label map; fails soft to the id when
10486 unmapped. Board cards additionally drop any card field whose value is already
10487 part of the title (so a full-name title doesn't repeat first/last below).
10488- **Rationale:** The detail page already resolves relations (form.py's
10489 `control`); the list surfaces showed raw ids, which read as broken data.
10490 ""Same field, same display, everywhere"" — a relation is a name to a human,
10491 not a UUID. Found by screenshotting the contacts board.
10492- **Applies to:** every table and board over a collection with a relation field.
10493- **Status:** shipped. Verified contacts board + table show ""Acme Corp"".
10494
10495### 15. Derived values are stored values (formulas & rollups materialize).
10496- **Decision:** A computed field's `formula`/`rollup` value is **written into
10497 the record** whenever its inputs change — never computed on read, never
10498 client-side.
10499- **Rationale:** Stored values mean the entire generative stack (lists,
10500 tables, boards, detail, search, server-side filters, sort, CSV, backups,
10501 grep) works with zero special cases, and realtime is free: a child-line
10502 edit rewrites the parent's stored total → change log fires → every open
10503 browser updates. Staleness is confined to definition changes, handled by an
10504 explicit `recompute_computed_fields` backfill — never a runtime loop.
10505- **Applies to:** every `type: computed` field with a `formula` or `rollup`
10506 (fin_journals debit/credit totals; contacts full_name).
10507- **Status:** shipped.
10508
10509---
10510
10511## Open questions (decide, then move up)
10512
10513### A. Create UX on data-heavy pages.
10514`+ New X` currently reveals a full-width inline form that pushes the list/board
10515down and out of view. Fine when empty; disruptive when the page has data. Options:
10516a collapsible/side form, a modal, or navigating to a dedicated create route.
10517Leaning: keep in-place (decision #1) but render the form as a **right-side drawer
10518or a modal** on data-heavy index pages, so the list stays visible. Undecided.
10519
10520### B. Triage lists → master-detail split-pane.
10521An inbox / review-queue / messaging thread list processes many items fast;
10522navigating away per item (decision #1) is wrong there. Proposed: a
10523`list_mode: ""split""` — click a row, preview in a right pane, never navigate.
10524A per-app opt-in exception to decision #1, not the default. Not built.
10525
10526### C. Detail view de-emphasizes the raw id.
10527The generated detail renders the record's `id` (a UUID) as the first field row —
10528noise. Proposal: hide `id`/computed fields from the default detail field table
10529(they're still reachable), the way `list_fields` already curates the list.
10530
10531### D. ~~Seed data should use UUIDs.~~ RESOLVED (decision 4).
10532Was: friendly seed ids (`c-grace`, `t-demo-1`) 404'd against `{id:uuid}` routes
10533once rows linked. Resolved by relaxing detail routes to accept any id (decision
105344) rather than rewriting seed ids and their FK references — every record is
10535reachable whatever its id shape.
10536
10537---
10538
10539## How to use this
10540
10541- Before adding a per-screen interaction, check here — the decision may already
10542 exist and just needs wiring.
10543- If you find two screens behaving differently for the same interaction, that's
10544 a bug: pick the rule (add it here) and fix it in the shared renderer, not the
10545 screen.
10546- These decisions live in `packages/app-theme/objects/site/` (list/form/detail
10547 generators) and `packages/app-views/objects/site/view_render.py` (block →
10548 generator config), the single places that project them everywhere.
10549"
10550event-hooks-decisions Event Hooks: Decisions and Why Design Decisions 1020 Design rationale for DBBASIC's post-commit event handlers, why they shipped after feature flags, and a real production trap around override-root objects. "This page records the design decisions behind event handler objects (Phase 5) and how they were sequenced against `user_prefs` + `feature_flags` (Phase 6). It's written the way the decisions were made — before implementation — so you understand *why* the system works this way, not just what it does.
10551
10552## Decision 1: Post-commit, best-effort dispatch
10553
10554Handlers run **after** a record write commits, isolated in the normal object sandbox; a handler failure is logged and never breaks or rolls back the write. There is no pre-write mutation or rejection in this phase.
10555
10556**Why.** The write path is the platform's most load-bearing promise: a valid write succeeds fast. Synchronous in-path handlers make every write's latency and reliability hostage to the worst installed handler — the exact footgun that made in-transaction trigger systems (classic SQL triggers, early Salesforce Apex) notorious, and the reason the systems that aged well converged on post-commit dispatch: Stripe/GitHub webhooks, Django's `on_commit` signals, Rails' `after_commit`. Post-commit covers the real 90% of use cases — notify, sync, enrich other records, log, kick off follow-on work. Validation and derivation already have a home (schema rules, computed fields); pre-write hooks could become a later, separately-gated phase if genuinely needed.
10557
10558This mirrors the shape of the platform's realtime push mechanism: `_publish_record_change_event` fires post-commit, and handlers are just one more subscriber to that stream. The hook point already existed, so adding handlers can't destabilize writes.
10559
10560Safety posture:
10561
10562- Gated off by default (`DBBASIC_ENABLE_EVENT_HANDLERS`)
10563- Depth/reentry guard — a handler's own writes fire events too, but dispatch depth is capped, so there are no infinite loops
10564- Standard sandbox timeout per handler
10565- Failures land in the object's log tagged with the triggering event's correlation ID
10566
10567## Decision 2: System/package objects only (this phase)
10568
10569Only operator-installed objects may declare `HANDLES = [...]`; they run in a defined service context. User-authored (`u_*`) handlers are deferred.
10570
10571**Why.** ""User code runs when *other people's* data changes"" is a privilege question with real teeth — whose rows can the handler read? who is the actor on its writes? The operator case needs none of that resolved, because the operator already owns the instance. Deferring user handlers keeps this phase's blast radius limited to code the operator explicitly installed, and lets the identity-threading design mature before it's exposed to untrusted authors. This is the same staging pattern used for overrides generally: opt-in, dormant by default.
10572
10573### The trap this creates, found in production
10574
10575`build_index` scans object sources and skips anything whose `kind` is not `""system""` — that check is how this decision gets enforced. An object living in the override root has `kind=""override""`, so a handler installed there declares `HANDLES`, installs cleanly, reports `""written""`, and then receives no events at all. Nothing errors; the dispatcher simply never learns the object exists.
10576
10577This bit a real deploy: `install_package` sent every new object to `roots[0]`, and `get_object_roots()` returns `[override, base]` in lookup order, so new package objects landed in the override root by default. It's fixed now — new objects go to the base root, and an object that already exists stays wherever it lives, so a customization is never yanked out from under an upgrade — but the underlying sharp edge remains by design: **an override cannot handle events.**
10578
10579If a handler is silent, check its `kind` before checking its code:
10580
10581```python
10582import object_handlers, object_namespace
10583print(object_handlers.build_index().get(""your_collection.record.created""))
10584print([(s.object_id, s.kind) for s in object_namespace.iter_object_sources()
10585 if s.object_id == ""system_your_handler""])
10586```
10587
10588## Decision 3: Phase 6 (prefs + flags) ships first
10589
10590**Why.**
10591
105921. It's data plus permission rules layered on machinery that already exists — collections, owner-scoped row filters, the generators — near-zero new engine code, so it's the fast, clean win.
105932. Feature flags are *how event handlers should ship*: handlers land dark behind a flag, get canaried, and roll back by flipping a flag instead of uninstalling code. Building flags first means the riskier phase ships the safe way.
105943. Flags convert a whole class of would-be code customizations into configuration, shrinking the reconcile surface — the same motive behind the rest of the upgrade system.
10595
10596Phase 6 shape: a `user_prefs` collection (owner-scoped `user_id`/`key`/`value`), a `feature_flags` collection for instance-wide toggles, and package manifests that may declare `features` with defaults stamped at install time. The effective value resolves as user override → instance toggle → package default, through one shared helper (a `/flags` object, exposed client-side as `window.dbbasicFlags`).
10597
10598## Shipping order
10599
106001. Phase 6: `user_prefs` + `feature_flags`.
106012. Phase 5a: `HANDLES` dispatch (post-commit, system objects only, flag-gated).
106023. Phase 5b: generator page slots (`before_list`, `row_actions`) — separable, UI-only, can ship alongside either of the above.
106034. Later, only if pulled by real need: pre-write hooks, and user-authored handlers running as their own author.
10604
10605## Update: pre-write hooks landed
10606
10607The ""later, only if pulled by real need"" condition in step 4 above was eventually met — the need being custom cross-field/cross-collection validation on generatively-formed collections, which is the extensibility question every schema-driven system eventually has to answer.
10608
10609`hooks.before_write` now runs a declared object synchronously in the generic write path: after permission checks, before persist; it can reject or transform the write; it **fails closed**; and it's opt-in per collection, so the latency concern from Decision 1 stays bounded — collections with no hook pay one cached dict lookup and nothing more. After-write work remains the event system's job; nothing about pre-write hooks changes that split.
10610
10611See the validation and logic guide for the full contract on `before_write` hooks.
10612"