Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

dbbasic-object-server/docs/runtime-contract.md
Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
runtime-contract.md · 28.9 KB · 905 lines

Runtime Contract

This document records the public contract that DBBASIC Object Server code should preserve while the implementation is extracted from the working prototype.

The goal is compatibility. New public code should not create a parallel runtime shape that breaks existing DBBASIC objects, DBBASIC Scroll, or the current dbbasic_object_core import path without an explicit migration plan.

Package Boundary

The existing runtime package is:

from dbbasic_object_core.runtime.object_runtime import ObjectRuntime

For now, dbbasic_object_core is the compatibility target for the core runtime. If a future dbbasic_object_server package is added, it should start as a thin server or command-line layer around dbbasic_object_core, not as a replacement runtime namespace.

Object Source Root

New object source should live under:

objects/

The object source root can be overridden with:

DBBASIC_OBJECTS_DIR=/path/to/objects

Public code should not use the old prototype directory name as a default.

Object ID Resolution

The daemon-facing object ID rules are:

  • category_name resolves to objects/category/name.py
  • u_{user_id}_{name} resolves to objects/users/{user_id}/{name}.py
  • trigger objects live under objects/triggers/

Public code should use object_namespace.py for object source lookup rather than reimplementing these rules in each server, daemon, or tool surface.

Current trigger object names:

  • scheduler
  • queue
  • events

Runtime Interface

The daemon expects a runtime object with:

runtime.load_object(path, object_id=None)

path is a Path or string pointing to an object source file. object_id is optional when the ID can be derived from the file name, and required when executing a resolved target object by ID.

The returned object must expose:

obj.logger
obj.state_manager
obj.execute(method, payload)

Loaded object modules receive _logger and _state_manager. The current public runtime injects a minimal state manager with get, set, get_all, and reload.

Storage Boundary

The default runtime storage is file-backed:

  • object source under objects/
  • object state under data/state/
  • object logs under data/logs/
  • object versions under data/versions/
  • object source changes under data/source_changes/
  • object file changes under data/file_changes/
  • object-owned files under data/files/
  • collection records under data/collections/
  • collection record changes under data/record_changes/
  • collection schemas under data/schemas/

SQL databases, SQLite, external HTTP APIs, and AI APIs are optional integrations. They should be reachable from objects or packages when needed, but the base runtime should not require a database service before the first useful app can run.

That boundary is intentional. It keeps the single-VM deployment small, makes backup/restore understandable, and lets Scroll inspect source, state, logs, versions, schemas, and diagrams without turning SQL into the default storage contract.

Runtime backups should use object_backup.py. The portable archive contains object source plus data/state/, data/logs/, data/versions/, data/source_changes/, data/schema_versions/, data/record_changes/, data/package_changes/, data/file_changes/, data/files/, data/schemas/, and data/collections/. It deliberately leaves deployment secrets, service files, virtualenvs, git history, lock files, temp files, and ephemeral rate-limit files outside the archive.

Object Execution Result

Public code should use object_execution.py for the shared execution result shape. The future ASGI server, daemon, Scroll execute button, and AI repair loop should all be able to depend on the same fields:

  • object_id
  • method
  • path
  • ok
  • result
  • error
  • started_at
  • finished_at
  • duration_ms

Execution failures should be captured as data instead of being flattened into plain strings. Error data includes:

  • type
  • message
  • traceback

This is part of the 100x dev loop: execute an object, inspect the structured error and traceback, patch the source, and keep the version trail close to the runtime feedback.

Object HTTP Responses

The runtime returns plain Python values. The ASGI/web layer turns those values into HTTP responses.

Compatibility response shapes:

return {"status": "ok", "count": 1}

Normal dicts are JSON responses and are not wrapped in another envelope.

return {
    "content_type": "text/html; charset=utf-8",
    "body": "<!doctype html><h1>DBBASIC</h1>",
}

Dicts with content_type and body become raw HTTP responses. The body may be text or bytes. This is the object shape used by generated views, HTML pages, and binary/image objects.

return (201, [("Content-Type", "text/plain")], [b"created"])

Tuples in (status, headers, body) form are passed through as low-level HTTP responses. body may be bytes, text, or a list of bytes/text parts.

Plain string returns are served as text/html; charset=utf-8. Plain bytes returns are served as application/octet-stream.

Object Versions

Public code should use object_versions.py for source version storage. The storage format intentionally matches the working prototype:

data/versions/{object_id}/metadata.tsv
data/versions/{object_id}/v1.txt
data/versions/{object_id}/v2.txt

metadata.tsv fields are:

  • version_id
  • timestamp
  • author
  • message
  • hash

History is returned newest first and does not include source content. Fetching a specific version returns metadata plus content.

Rollback is non-destructive. Rolling back to version N creates a new latest version containing the old content, preserving the full history.

The future runtime should keep the prototype behavior:

  • save an initial version when an object is first loaded and no history exists
  • on source update, save the new code as a version, write it to the source file,

reload the object, and log the update

  • on rollback, create the rollback version, write it to the source file, reload

the object, and log the rollback

Public server/runtime code should use object_source.py for the source file read, update, and rollback steps so source writes and version storage do not drift into separate implementations.

Object Source Changes

Public code should use object_source_changes.py for source edit and rollback activity storage. This is separate from data/versions/: versions store recoverable source snapshots, while source changes store the operator timeline that Scroll can show as admin activity.

Source changes are append-only JSON Lines:

data/source_changes/{object_id}/changes.jsonl

The helper exposes:

append_source_change(object_id, action, version_id, from_version_id=None, actor="api", message="", correlation_id=None, details=None, base_dir="data")
list_source_changes(object_id, base_dir="data", limit=100, offset=0)

Entries include change_id, timestamp, object_id, action, version_id, from_version_id, actor, message, correlation_id, and details. History is returned newest first and does not include source content.

Object File Changes

Public code should use object_file_changes.py for object-owned file write activity storage. This is separate from data/files/: files store current content, while file changes store the operator timeline for uploads, overwrites, and deletes.

File changes are append-only JSON Lines:

data/file_changes/{object_id}/changes.jsonl

The helper exposes:

append_file_change(object_id, action, file_name, file_size=None, actor="api", message="", correlation_id=None, details=None, base_dir="data")
list_file_changes(object_id, file_name=None, base_dir="data", limit=100, offset=0)

Entries include change_id, timestamp, object_id, action, file_name, file_size, actor, message, correlation_id, and details. History is returned newest first and does not include file content.

State Manager Interface

The daemon expects trigger objects to expose a state manager with:

state_manager.reload()
state_manager.get_all()
state_manager.get(key)
state_manager.set(key, value)

Values used by scheduler, queue, and events are JSON strings stored by key.

State storage intentionally matches the working prototype:

data/state/{object_id}/state.tsv

Rows are either:

key<TAB>value
key<TAB>value<TAB>timestamp

The public state reader skips an optional key<TAB>value<TAB>timestamp header, ignores malformed rows, and coerces values to int, then float, otherwise keeps strings.

The public runtime-owned state manager writes sorted rows in timestamp format:

key<TAB>value<TAB>timestamp

There is still no public HTTP state-write endpoint. Object code writes state through _state_manager.

Log Storage

Log storage intentionally matches the working prototype:

data/logs/{object_id}/log.tsv
data/logs/{object_id}/log-*.tsv
data/logs/{object_id}/log-*.tsv.gz

Rows are TSV with a header. The default fields are:

  • entry_id
  • timestamp
  • level
  • message
  • method
  • status
  • duration_ms
  • error_type
  • error

Object code may add extra fields such as method, user_id, or request metadata. The public log reader returns the current log.tsv first, then rotated log-*.tsv and log-*.tsv.gz files in sorted order, and supports exact level filtering and limit.

The active log.tsv stays plain TSV so appends, tail, and crash recovery stay simple. When it reaches the configured size limit, the runtime rotates it to a timestamped file and gzip-compresses the rotated file by default.

Log maintenance settings:

  • DBBASIC_LOG_MAX_BYTES controls active log size before rotation. The default

is 10485760 bytes. Values less than or equal to 0 disable rotation.

  • DBBASIC_LOG_COMPRESS_ROTATED controls gzip compression of new rotated logs.

The default is true. Set it to false for plain rotated TSV files.

  • DBBASIC_LOG_KEEP_ROTATED controls garbage collection of rotated logs. The

default is 32. Values less than or equal to 0 keep all rotated logs.

Compressed rotated logs remain ordinary files on disk. Operators can inspect them without expanding a second copy:

gzip -cd data/logs/site_home/log-*.tsv.gz

This is at-rest compression. HTTP response compression, log replication compression, and backup-transfer compression are separate transport concerns. Those should compress larger batches or responses, not tiny realtime events by default.

The public ASGI execution path appends one log entry after each object method run. Successful runs use DEBUG with status=success; failed runs use ERROR with status=error, error_type, and error.

Loaded object modules also receive _logger. The public logger helper writes to the same per-object TSV path and exposes:

_logger.log(level, message, **fields)
_logger.debug(message, **fields)
_logger.info(message, **fields)
_logger.warning(message, **fields)
_logger.error(message, **fields)
_logger.critical(message, **fields)
_logger.get_logs(level=None, limit=None, offset=0, **filters)

This keeps object-owned application logs and runtime execution logs together without creating a second logging format.

Object Files

Object-owned files live under:

data/files/{object_id}/

The public helper module is object_files.py. It exposes read helpers plus write helpers that the HTTP layer must gate behind admin auth, permissions, size limits, and deployment flags:

list_object_files(object_id, base_dir="data")
read_object_file(object_id, filename, base_dir="data")
write_object_file(object_id, filename, content, base_dir="data", overwrite=False)
delete_object_file(object_id, filename, base_dir="data")

list_object_files returns dictionaries with:

  • name
  • size
  • modified

read_object_file returns raw bytes plus the same metadata. Filenames are validated before filesystem access. Empty names, absolute paths, null bytes, and .. traversal are rejected.

Upload and delete are intentionally not public yet. They need explicit request size limits, content policy, audit logs, and server-enforced permissions.

Object Collections

Collections are a read-only derived view. The public helper modules are object_collections.py and object_records.py.

The server infers collection names from object IDs:

  • site_home belongs to site
  • apps_widget_counter belongs to apps
  • u_42_deals belongs to deals

Permission rules can also name a collection before any object exists for it. Record files can name a collection before any object exists for it too. That lets Scroll show planned or data-backed collections while the server keeps /objects as the stable source of executable behavior.

Collection summaries include:

  • object count
  • object owner set
  • object kind counts
  • object-owned file count
  • number of objects with state
  • number of objects with logs
  • whether a TSV record file exists
  • exact permission rule summary for that collection

The log portion is intentionally a cheap file-presence check. Listing collections should not scan every log entry on a busy server.

Collection records live under:

data/collections/{collection}/records.tsv

The TSV file must have a header row and an id column. The current public record API is:

list_collection_records(collection, base_dir="data", limit=100, offset=0)
read_collection_records(collection, base_dir="data")
get_collection_record(collection, record_id, base_dir="data")
create_collection_record(collection, record, base_dir="data")
update_collection_record(collection, record_id, changes, base_dir="data")
delete_collection_record(collection, record_id, base_dir="data")

Values are stored and returned as strings. Record writes preserve existing columns and append new columns from submitted fields. Record IDs are unique and cannot be changed by update.

New DBBASIC-facing record IDs are UUIDv4 strings by default when the server creates them. Imported application data can keep legacy IDs in compatibility columns, but new URLs, generated UI, change logs, and object references should not depend on sequential database integers. This came from the Django/PostgreSQL migration path: existing tables may arrive with integer primary keys and JSONB-style flexible data, while new DBBASIC resources need stable, non-enumerable IDs that can move between object servers and packages.

If a manual schema exists for the collection, create/update validates known fields before writing the TSV:

  • required: true, validation.required, and validation.not_null require a

non-empty value

  • default fills missing or empty fields during create
  • type can validate integer, number, currency, boolean, date,

datetime, and enum

  • enum, validation.choices, or validation.in restrict allowed values
  • validation.min, validation.max, validation.min_length,

validation.max_length, validation.pattern, and validation.regex are enforced

  • type: "computed", computed, or read-only metadata rejects submitted

writes for that field

Unknown fields are still allowed. This preserves the DBBASIC rule that schemas can tighten important collections without turning every object into a migration project.

Schemas are metadata, not a mandatory deployment gate. A collection can start schemaless, accept extra fields, and later add schema metadata for generated forms, validation, relations, field permissions, and admin views. This keeps the fast object/data loop while still allowing stricter app surfaces when needed.

The HTTP record routes are admin-gated by default. When permission audit or enforcement is enabled, read routes use the server permission policy. Record lists apply row filters before pagination; record detail checks evaluate the selected row and can redact fields. Record writes require either the admin token or enforcement mode with an allowed create, update, or delete decision. Audit-only mode records write decisions without granting mutation access. When enforcement is enabled, schema field permissions are also applied: hidden fields are removed from record reads, and submitted fields with read or hidden access are rejected before the TSV write.

Record mutations also append change entries under:

data/record_changes/{collection}/changes.jsonl

The public helper module is object_record_changes.py. It exposes:

append_record_change(collection, record_id, action, before, after, base_dir="data")
list_record_changes(collection, record_id=None, base_dir="data", limit=100, offset=0)

Request/action correlation IDs are UUIDv4 values. The ASGI app accepts X-DBBASIC-Correlation-ID, normalizes it only when it is a valid UUIDv4, and otherwise creates a new one. Source versions, source-change entries, object-owned logs, execution errors, and permission audits should carry the current correlation ID so Scroll and AI tooling can trace one action across code, data, logs, and rollback history.

Entries are JSON Lines with:

  • change_id
  • timestamp
  • collection
  • record_id
  • action
  • actor
  • message
  • changed_fields
  • before
  • after

New change_id values are UUIDv4. Old timestamp-shaped change IDs remain readable so existing package and record history can be migrated in place.

This is the durable record history that Scroll can show in admin and rollback screens. It is intentionally separate from event delivery. Scheduler, queue, and event trigger objects can later publish notifications from the same facts without making callback delivery the only audit trail.

Collection Schemas

Schema metadata lives under:

data/schemas/{collection}.json

The public helper module is object_schemas.py. It exposes read and admin-write operations:

list_schemas(base_dir="data")
get_schema(collection, base_dir="data")
replace_schema(collection, payload, base_dir="data")

A schema file can describe fields, validation rules, computed values, UI hints, field permissions, and relations:

{
  "title": "Invoices",
  "version": 1,
  "fields": [
    {
      "name": "customer_id",
      "type": "relation",
      "required": true,
      "relation": {"collection": "contacts"},
      "validation": {"not_null": true}
    },
    {
      "name": "cost_price",
      "type": "currency",
      "permissions": {"admin": "edit", "sales": "hidden"},
      "ui": {"section": "totals"}
    }
  ]
}

If a collection exists but no manual schema file exists, the server can return an empty derived schema. That lets Scroll render the collection first and add field metadata later without breaking the object API.

Schema metadata is not the permission system by itself. Permission rules still control access modes, ownership, sharing, row filters, and field allow/deny lists. Schema fields tell tools what exists and can refine individual edit/read/hidden field behavior for generated forms. The server enforces those schema field permissions only after the broader policy has allowed the row.

Schema writes are available through the admin-only replacement path. They are atomic file replacements under data/schemas/ and preserve UI/view metadata so Scroll can save generated form, table, dashboard, validation, relation, and field permission definitions.

Every admin schema write also records a schema version under:

data/schema_versions/{collection}/metadata.tsv
data/schema_versions/{collection}/vN.json

Rollbacks are non-destructive. A rollback creates a new latest version from an older schema JSON file and then replaces data/schemas/{collection}.json. This gives admin tools and Scroll a change log for who changed a schema, what message they left, and what can be restored. User-level schema editing still needs real sessions and broader permission checks before it should be exposed outside trusted admin tooling.

Scheduler State

Scheduler state keys begin with:

task_

Each value is a JSON object with fields such as:

  • id
  • object_id
  • method
  • payload
  • type
  • schedule
  • status
  • next_run
  • last_run
  • run_count

The daemon executes active tasks when next_run is due.

Queue State

Queue state keys begin with:

msg_

Each value is a JSON object with fields such as:

  • id
  • queue_name
  • message
  • priority_level
  • status
  • created_at
  • visible_after
  • expires_at
  • attempts
  • max_attempts

message should include:

  • object_id
  • method
  • payload

The daemon marks messages as processing, completed, pending, expired, or failed.

Event State

The public helper module is object_events.py. It writes event and subscription state into:

data/state/events/state.tsv

This keeps the HTTP API, Scroll, and the daemon on one event shape.

Event subscription keys begin with:

sub_

Event keys begin with:

event_

Subscriptions include:

  • id
  • event_type
  • callback_url
  • last_event_id
  • delivery

Events include:

  • id
  • event_type
  • payload
  • source
  • actor
  • timestamp
  • created_at

The daemon delivers matching events to the subscription callback URL and records delivery status on the subscription. delivery.status is idle, ok, or failed, with attempt counts, success/failure counts, the last HTTP status code, and the last error text. last_event_id advances only after successful delivery. A failed callback leaves the cursor in place so the daemon can retry instead of silently skipping the event.

The current HTTP event API is admin-gated:

GET /events
POST /events
DELETE /events?keep_count=1000&keep_seconds=604800
GET /events/deliveries
GET /events/subscriptions
POST /events/subscriptions
DELETE /events/subscriptions?event_type=invoice.created&subscriber_id=scroll
Authorization: Token <token>

Record change history is still the durable audit trail. Events are for notifications, triggers, listeners, webhooks, and future worker/actor-style flows that need to react to those facts. By default, collection record POST, PUT, and DELETE mutations publish metadata-only collection.record.created, collection.record.updated, and collection.record.deleted events from the saved change entry. Set DBBASIC_ENABLE_RECORD_EVENTS=false to disable that publication path.

Record mutation event payloads include change_id, collection, record_id, action, actor, timestamp, and changed_fields. They do not include full before or after snapshots until subscriber permission filtering exists.

Event retention is built into the server and daemon cleanup path. The default is to keep the newest 1000 events and prune events older than seven days. Configure that with:

DBBASIC_EVENT_KEEP_COUNT=1000
DBBASIC_EVENT_KEEP_SECONDS=604800

Set either value to 0 to disable that part of pruning. Pruning deletes only event_ rows. It does not delete sub_ rows, and it protects the event id held by any subscription last_event_id so the daemon does not lose its delivery cursor. If a subscription has a failed delivery attempt, pruning also protects the failed event id so retry state remains visible to Scroll and the daemon.

Daemon posture is exposed through an admin-token-gated read-only endpoint:

GET /daemon/status
Authorization: Token <token>

The response summarizes trigger source presence, scheduler task counts, due tasks, queue message counts, visible and delayed pending messages, event counts, subscription delivery status, pending deliveries, event retention settings, and rate-limit cleanup pressure. It does not execute daemon work and does not expose event payloads. This gives Scroll a Flower-style operator view without adding a separate queue dashboard service to the single-VM deployment.

Trusted operators can also manage scheduler tasks and queue messages through admin-token-gated routes:

GET /daemon/scheduler/tasks
POST /daemon/scheduler/tasks
PATCH /daemon/scheduler/tasks/{task_id}
DELETE /daemon/scheduler/tasks/{task_id}
GET /daemon/queue/messages
POST /daemon/queue/messages
PATCH /daemon/queue/messages/{message_id}
DELETE /daemon/queue/messages/{message_id}

These routes write the same task_ and msg_ rows that object_daemon.py already polls. Task and message payloads are redacted by default so Scroll can show operator posture without leaking secrets into broad status views. Use include_payload=true only for trusted admin screens.

Packages

Packages are the DBBASIC way to ship reusable app pieces without turning every app into a separate deploy process. A package is a directory containing a manifest plus package-owned objects, schemas, permissions, seed data, and migrations:

The intended package catalog starts from reusable app primitives: messages, projects, tasks, notes, files, articles, links, contacts, finance, events, workers, templates, catalog entries, time tracking, AI usage, objects, forums, documentation, and APIs. Those pieces should be installable and inspectable as packages, not buried inside one private monolith.

packages/{package_id}/
  dbbasic-package.json
  objects/
  schemas/
  permissions/
  seed/
  migrations/

Package source can be overridden with:

DBBASIC_PACKAGES_DIR=/path/to/packages

The current public helper module is object_packages.py. It exposes package discovery, non-mutating install planning, and conservative install writes:

list_packages(root="packages")
get_package(package_id, root="packages")
dry_run_package(package_id, root="packages", base_dir="data", object_roots=None)
install_package(package_id, root="packages", base_dir="data", object_roots=None, allow_replace=False)

The manifest format is JSON so Python 3.10 installs do not need another parser. The first required manifest is:

{
  "id": "hello-world",
  "name": "Hello World",
  "version": "0.1.0",
  "description": "Minimal DBBASIC package",
  "compatibility": {"object_server": ">=0.0.1"},
  "dependencies": [],
  "objects": [{"id": "hello_world", "path": "objects/hello/world.py"}],
  "schemas": [],
  "permissions": [],
  "seed": [],
  "migrations": []
}

Package paths are package-relative. Absolute paths, null bytes, and .. traversal are rejected. Dry-runs report what would be created, replaced, merged, applied, or skipped without writing object source or data.

The first install contract is intentionally conservative:

  • objects are written under the configured object root
  • schemas are validated and written under data/schemas/
  • seed TSV is created only when the target collection records file does not

already exist

  • existing objects and schemas require allow_replace=True
  • permission fragments ({"rules": [...]}) are validated, stamped with

package provenance, and merged into the deployment policy, skipping rules that already exist so reinstalls stay idempotent; invalid fragments block the install

  • migrations are rejected until explicit run semantics are added

Package change history lives here:

data/package_changes/{package_id}/changes.jsonl

The public helper module is object_package_changes.py. It exposes:

append_package_change(package_id, action, package_version=None, actor="api", details=None, base_dir="data")
get_package_change(package_id, change_id, base_dir="data")
list_package_changes(package_id, base_dir="data", limit=100, offset=0)
dry_run_change_details(plan)

The current server appends a dry_run change for GET /packages/{package_id}?dry_run=true. It appends install_requested and then either installed or failed for POST /packages/{package_id}/install. The details are intentionally compact: package identity, safe/install flags, action counts by package section, and warnings. Successful installs include restore-point metadata. Package source is not copied into the changelog.

Package install writes require both admin auth and:

DBBASIC_ENABLE_PACKAGE_INSTALLS=true

The HTTP install route creates a runtime restore point before object, schema, or seed writes. Restore points default to data/backups/; set DBBASIC_BACKUPS_DIR to place them on a separate backup volume.

Package restore is a separate admin-gated route:

POST /packages/{package_id}/restore
DBBASIC_ENABLE_PACKAGE_RESTORE=true
{"change_id": "...", "confirm": "restore-runtime"}

The change id must refer to a package change that contains restore-point metadata. The server restores that runtime snapshot with overwrite and prune-extra enabled, then records restore_requested and rolled_back package change rows after the restore completes. This is a whole-runtime restore point, not a selective package uninstall.

Keep this disabled on public staging unless the route is on a private admin surface and backups have been tested.

Public Safety

Tests and docs should use only safe placeholder values:

  • 127.0.0.1 for localhost samples
  • 192.0.2.0/24, 198.51.100.0/24, or 203.0.113.0/24 for documentation IPs
  • example.com, example.net, or example.org for documentation domains

Do not commit real deployment IPs, hostnames, tokens, private URLs, or personal filesystem paths.