askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
object_change_dispatch.py · 13.0 KB · 315 lines
| 1 | """Change-log event dispatch -- the real fix for docs/logic-decisions.md #9. |
| 2 | |
| 3 | The problem this closes: HANDLES-declared event handlers only fire from |
| 4 | object_server's synchronous HTTP write path (see object_handlers.py and |
| 5 | object_server._dispatch_event_handlers). A runner or an action that writes |
| 6 | through object_records directly never passes through that path, so its |
| 7 | write never triggers the reactions a handler would give an equivalent HTTP |
| 8 | write. Doctrine #9 tracked two workarounds that paid for this gap by hand |
| 9 | (action_apply_count composing its own inventory-variance journal; |
| 10 | action_resolve_bank_line composing the NSF bounce reversal) and said the |
| 11 | third instance must buy the real fix instead of a third workaround: a |
| 12 | daemon pass that dispatches handlers from the record-change log, so a |
| 13 | write reaches its reactions no matter which path wrote it -- the same |
| 14 | shape object_daemon.process_notifications already uses to avoid depending |
| 15 | on synchronous dispatch for notify_rules. |
| 16 | |
| 17 | Payload parity with the HTTP path (read this twice, it bit production |
| 18 | once): object_server's dispatcher builds |
| 19 | |
| 20 | {"event": "<collection>.record.<created|updated|deleted>", |
| 21 | "collection": collection, "record_id": record_id, "action": action} |
| 22 | |
| 23 | where ``action`` is the RAW verb from the write ("create"/"update"/ |
| 24 | "delete") and ``event`` uses the past-participle form. A handler that |
| 25 | mixed the two up (checking ``action == "created"`` instead of mapping |
| 26 | "create" -> "created" first) silently matched nothing and skipped every |
| 27 | entry in production -- see the comment in |
| 28 | packages/app-payments/objects/system/books.py's EVENT(). This module |
| 29 | builds the exact same shape from the exact same raw fields a record-change |
| 30 | entry already carries (object_record_changes stores the raw action |
| 31 | untouched), so a handler cannot tell which path dispatched it. |
| 32 | |
| 33 | Double-dispatch, honestly: an HTTP write already dispatched its handlers |
| 34 | synchronously before this pass ever sees the resulting change-log entry. |
| 35 | This module has no way to know that happened -- object_record_changes |
| 36 | does not record whether HANDLES already ran for an entry, and this file |
| 37 | is not allowed to touch object_server.py to add that bookkeeping. So this |
| 38 | pass does NOT prevent a handler from running twice across the two paths. |
| 39 | What it promises instead is the same thing doctrine #7 already requires |
| 40 | of every composer here: idempotency by provenance (a generated_from |
| 41 | marker such as "payments/{id}" or "payments/{id}:bounced"), which is |
| 42 | exactly how action_apply_count and action_resolve_bank_line stay correct |
| 43 | under the two-workaround world already. At-least-once is therefore the |
| 44 | right guarantee to promise here, not exactly-once -- a handler that isn't |
| 45 | idempotent under redelivery is a bug in the handler, not in this |
| 46 | dispatcher. |
| 47 | |
| 48 | What this module DOES guard against is re-firing the SAME (change_id, |
| 49 | handler_object_id) pair across this pass's own polls -- a cursor file |
| 50 | (same posture as process_notifications: first run stamps "now" and never |
| 51 | backfills history) plus a small marker file recording pairs dispatched |
| 52 | since the cursor last advanced. The marker exists because the cursor only |
| 53 | advances once per successful batch; without it, a crash between |
| 54 | dispatching entry 3 of 10 and persisting the new cursor would replay |
| 55 | entries 1-10 (not just 4-10) on the next poll. The marker file is cleared |
| 56 | every time the cursor successfully advances, so it never grows past one |
| 57 | in-flight batch. |
| 58 | |
| 59 | Finally, this whole pass is opt-in: DBBASIC_ENABLE_CHANGE_DISPATCH is |
| 60 | unset (OFF) by default, mirroring object_handlers.HANDLERS_ENABLED_ENV's |
| 61 | posture that a dispatch behavior this new should be a deliberate operator |
| 62 | act, not a silent default. |
| 63 | """ |
| 64 | from __future__ import annotations |
| 65 | |
| 66 | import os |
| 67 | from datetime import datetime, timezone |
| 68 | from pathlib import Path |
| 69 | from typing import Any, Iterable |
| 70 | |
| 71 | import object_collections |
| 72 | import object_handlers |
| 73 | import object_record_changes |
| 74 | from object_execution import ObjectExecutionRequest, execute_object |
| 75 | from python_object_runtime import PythonObjectRuntime |
| 76 | |
| 77 | ENABLE_ENV = "DBBASIC_ENABLE_CHANGE_DISPATCH" |
| 78 | _TRUE_VALUES = {"1", "true", "yes", "on"} |
| 79 | |
| 80 | CURSOR_FILE_NAME = ".change_dispatch_cursor" |
| 81 | MARKER_FILE_NAME = ".change_dispatch_dispatched" |
| 82 | |
| 83 | DEFAULT_LIMIT = 200 |
| 84 | |
| 85 | # One runtime, module-level, mirrors object_server.py's `_runtime = |
| 86 | # PythonObjectRuntime()` and the test fixtures' `RUNTIME = |
| 87 | # python_object_runtime.PythonObjectRuntime()` -- cheap to construct, no |
| 88 | # per-call state worth re-creating on every poll. |
| 89 | _RUNTIME = PythonObjectRuntime() |
| 90 | |
| 91 | |
| 92 | def change_dispatch_enabled(*, base_dir: Path | str) -> bool: |
| 93 | """Return True when the change-log dispatch pass may run. |
| 94 | |
| 95 | Env-only and OFF by default -- unlike object_notify's |
| 96 | notify_pass_enabled (a feature_flags row, default ON: a brownout lever |
| 97 | for a feature already adopted), this pass makes an installed handler |
| 98 | start reacting to writes that never used to reach it. That is a |
| 99 | behavior change for existing objects, not a lever on an existing one, |
| 100 | so it needs an explicit operator opt-in rather than a data default. |
| 101 | ``base_dir`` is accepted for signature symmetry with the other |
| 102 | ``*_enabled(*, base_dir)`` checks in this codebase; this particular |
| 103 | check does not read anything under it. |
| 104 | """ |
| 105 | value = os.environ.get(ENABLE_ENV, "") |
| 106 | return value.strip().lower() in _TRUE_VALUES |
| 107 | |
| 108 | |
| 109 | def dispatch_pending( |
| 110 | *, |
| 111 | base_dir: Path | str, |
| 112 | roots: Iterable[Path] | None = None, |
| 113 | limit: int = DEFAULT_LIMIT, |
| 114 | ) -> dict[str, Any]: |
| 115 | """Dispatch HANDLES handlers for record-change entries newer than the cursor. |
| 116 | |
| 117 | Returns {"processed", "dispatched", "skipped", "errors", "results"}: |
| 118 | |
| 119 | - processed: change-log entries examined this pass (across every |
| 120 | collection any installed handler declares interest in). |
| 121 | - dispatched: handler EVENT calls actually attempted (success or |
| 122 | failure both count -- this is "how much work happened", not "how |
| 123 | much succeeded"). |
| 124 | - skipped: entries whose event had no matching handler, plus |
| 125 | (change_id, handler_id) pairs already recorded as dispatched by an |
| 126 | earlier, not-yet-committed pass over the same batch. |
| 127 | - errors: dispatched calls whose handler raised or returned a |
| 128 | non-ok ObjectExecutionResult. |
| 129 | - results: one dict per attempted dispatch (change_id, collection, |
| 130 | record_id, action, event, handler_id, ok, error). |
| 131 | |
| 132 | Gracefully returns the all-zero shape (never raises) when: the pass is |
| 133 | disabled, no installed object declares a HANDLES entry shaped like |
| 134 | "<collection>.record.<created|updated|deleted>", or a collection has |
| 135 | no change log yet. One raising handler, or one unreadable change-log |
| 136 | file, never stops the rest of the batch -- same posture as |
| 137 | object_daemon.process_notifications. |
| 138 | """ |
| 139 | base_dir = Path(base_dir) |
| 140 | result: dict[str, Any] = { |
| 141 | "processed": 0, |
| 142 | "dispatched": 0, |
| 143 | "skipped": 0, |
| 144 | "errors": 0, |
| 145 | "results": [], |
| 146 | } |
| 147 | |
| 148 | if not change_dispatch_enabled(base_dir=base_dir): |
| 149 | return result |
| 150 | |
| 151 | try: |
| 152 | index = object_handlers.build_index(roots) |
| 153 | except Exception: |
| 154 | # Source discovery is best-effort here -- a broken package source |
| 155 | # elsewhere must not stop this pass either. |
| 156 | return result |
| 157 | |
| 158 | collections = _collections_with_record_handlers(index) |
| 159 | if not collections: |
| 160 | return result |
| 161 | |
| 162 | cursor_path = base_dir / CURSOR_FILE_NAME |
| 163 | try: |
| 164 | cursor = cursor_path.read_text(encoding="utf-8").strip() |
| 165 | except OSError: |
| 166 | cursor = "" |
| 167 | |
| 168 | if not cursor: |
| 169 | # First run: stamp "now" and dispatch only FUTURE changes. A brand |
| 170 | # new install (or a flag flipped on long after go-live) must never |
| 171 | # replay years of history through every handler the moment it |
| 172 | # turns on. |
| 173 | now_iso = datetime.now(timezone.utc).isoformat() |
| 174 | cursor_path.parent.mkdir(parents=True, exist_ok=True) |
| 175 | cursor_path.write_text(now_iso, encoding="utf-8") |
| 176 | return result |
| 177 | |
| 178 | fresh: list[dict[str, Any]] = [] |
| 179 | for collection in collections: |
| 180 | try: |
| 181 | payload = object_record_changes.list_record_changes( |
| 182 | collection, base_dir=base_dir, limit=limit |
| 183 | ) |
| 184 | except (object_collections.InvalidCollectionNameError, OSError, ValueError): |
| 185 | continue |
| 186 | for change in payload.get("changes") or []: |
| 187 | ts = str(change.get("timestamp") or "") |
| 188 | if ts > cursor: |
| 189 | fresh.append(change) |
| 190 | fresh.sort(key=lambda c: str(c.get("timestamp") or "")) |
| 191 | |
| 192 | result["processed"] = len(fresh) |
| 193 | if not fresh: |
| 194 | return result |
| 195 | |
| 196 | marker_path = base_dir / MARKER_FILE_NAME |
| 197 | dispatched_pairs = _load_markers(marker_path) |
| 198 | |
| 199 | max_ts = cursor |
| 200 | for change in fresh: |
| 201 | # Per-entry isolation: a malformed change or a handler exception |
| 202 | # must never stop the rest of the batch or prevent the cursor from |
| 203 | # advancing past everything that DID work. |
| 204 | try: |
| 205 | ts = str(change.get("timestamp") or "") |
| 206 | if ts > max_ts: |
| 207 | max_ts = ts |
| 208 | |
| 209 | collection = str(change.get("collection") or "") |
| 210 | record_id = str(change.get("record_id") or "") |
| 211 | action = str(change.get("action") or "") |
| 212 | change_id = str(change.get("change_id") or "") |
| 213 | |
| 214 | event = object_handlers.event_name(collection, action) |
| 215 | handler_ids = index.get(event) if event else None |
| 216 | if not handler_ids: |
| 217 | result["skipped"] += 1 |
| 218 | continue |
| 219 | |
| 220 | # Same signal-shaped payload object_server._dispatch_event_handlers |
| 221 | # builds: no record body, RAW action, participle event name. |
| 222 | event_payload = { |
| 223 | "event": event, |
| 224 | "collection": collection, |
| 225 | "record_id": record_id, |
| 226 | "action": action, |
| 227 | } |
| 228 | |
| 229 | for handler_id in handler_ids: |
| 230 | pair_key = f"{change_id}\t{handler_id}" |
| 231 | if pair_key in dispatched_pairs: |
| 232 | result["skipped"] += 1 |
| 233 | continue |
| 234 | |
| 235 | entry: dict[str, Any] = { |
| 236 | "change_id": change_id, |
| 237 | "collection": collection, |
| 238 | "record_id": record_id, |
| 239 | "action": action, |
| 240 | "event": event, |
| 241 | "handler_id": handler_id, |
| 242 | "ok": False, |
| 243 | "error": None, |
| 244 | } |
| 245 | try: |
| 246 | request = ObjectExecutionRequest( |
| 247 | object_id=handler_id, |
| 248 | method="EVENT", |
| 249 | payload=event_payload, |
| 250 | correlation_id=change.get("correlation_id") or None, |
| 251 | ) |
| 252 | exec_result = execute_object(_RUNTIME, request, roots) |
| 253 | entry["ok"] = bool(exec_result.ok) |
| 254 | if not exec_result.ok and exec_result.error is not None: |
| 255 | entry["error"] = exec_result.error.message |
| 256 | except Exception as exc: # noqa: BLE001 -- one bad handler must not stop others |
| 257 | entry["ok"] = False |
| 258 | entry["error"] = str(exc)[:200] |
| 259 | |
| 260 | result["dispatched"] += 1 |
| 261 | if not entry["ok"]: |
| 262 | result["errors"] += 1 |
| 263 | result["results"].append(entry) |
| 264 | dispatched_pairs.add(pair_key) |
| 265 | _append_marker(marker_path, pair_key) |
| 266 | except Exception: # noqa: BLE001 -- one bad change must not stop the batch |
| 267 | continue |
| 268 | |
| 269 | if max_ts != cursor: |
| 270 | cursor_path.write_text(max_ts, encoding="utf-8") |
| 271 | # The whole batch between the old and new cursor just committed; |
| 272 | # nothing in it can ever be re-read (list_record_changes only |
| 273 | # returns entries newer than the cursor), so the markers guarding |
| 274 | # against a mid-batch crash are no longer needed. |
| 275 | _clear_markers(marker_path) |
| 276 | |
| 277 | return result |
| 278 | |
| 279 | |
| 280 | def _collections_with_record_handlers(index: dict[str, list[str]]) -> list[str]: |
| 281 | """Return the sorted set of collections any HANDLES entry names. |
| 282 | |
| 283 | Only "<collection>.record.<created|updated|deleted>" shaped events are |
| 284 | record-change events this pass can act on; anything else in HANDLES |
| 285 | (a future, non-record event kind) is silently not our concern here. |
| 286 | """ |
| 287 | participles = {"created", "updated", "deleted"} |
| 288 | collections: set[str] = set() |
| 289 | for event in index: |
| 290 | parts = event.split(".record.", 1) |
| 291 | if len(parts) != 2 or parts[1] not in participles or not parts[0]: |
| 292 | continue |
| 293 | collections.add(parts[0]) |
| 294 | return sorted(collections) |
| 295 | |
| 296 | |
| 297 | def _load_markers(path: Path) -> set[str]: |
| 298 | try: |
| 299 | text = path.read_text(encoding="utf-8") |
| 300 | except OSError: |
| 301 | return set() |
| 302 | return {line for line in text.splitlines() if line.strip()} |
| 303 | |
| 304 | |
| 305 | def _append_marker(path: Path, key: str) -> None: |
| 306 | path.parent.mkdir(parents=True, exist_ok=True) |
| 307 | with path.open("a", encoding="utf-8") as handle: |
| 308 | handle.write(key + "\n") |
| 309 | |
| 310 | |
| 311 | def _clear_markers(path: Path) -> None: |
| 312 | try: |
| 313 | path.write_text("", encoding="utf-8") |
| 314 | except OSError: |
| 315 | pass |