askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
object_materialize.py · 56.5 KB · 1287 lines
| 1 | """Materialize block: declared generation jobs that CREATE new records. |
| 2 | |
| 3 | Implements ``plan/vocabulary/61-materialize-spec.md``. Sibling to |
| 4 | ``object_rollups.py`` (14-rollup-spec.md): both are declarative, |
| 5 | schema/config-parameterized, ride ``object_daemon.py``'s poll loop as one |
| 6 | more pass, and both write to a collection nothing else writes to. The |
| 7 | difference that makes this a distinct block rather than a rollup variant: |
| 8 | a rollup recompute REPLACES the whole answer every time (naturally |
| 9 | idempotent, disposable output); a materialize pass CREATES new records |
| 10 | other records come to depend on (a journal, a depreciation posting, a |
| 11 | seeded task), so a double-run is not "stale," it is a duplicate -- |
| 12 | genuinely destructive. Everything below exists to make "generate this |
| 13 | period's output" idempotent by construction rather than by discipline. |
| 14 | |
| 15 | **The three worked shapes, and how one mechanism spans them (61's Purpose |
| 16 | and Coverage):** |
| 17 | |
| 18 | 1. Recurring journals (``fin_recurring`` -> ``fin_journals`` + |
| 19 | ``fin_journal_lines``, ``trigger.mode: "scheduled"``): a header row per |
| 20 | due period, its child lines parsed VERBATIM from the source row's own |
| 21 | ``child_source_field`` JSON blob (already shaped like the child |
| 22 | schema's rows) -- see ``_build_child_rows_from_template``. |
| 23 | 2. Depreciation (``products`` -> ``fin_journals`` + ``fin_journal_lines``, |
| 24 | ``trigger.mode: "scheduled_fixed"``): 61's own Open Questions flags |
| 25 | this as genuinely underspecified ("this spec assumes a |
| 26 | debit_account_id/credit_account_id pair added to the definition |
| 27 | record... flagged as an implementation-time field, not fully specified |
| 28 | here"). This module's resolution, decided at implementation time: when |
| 29 | a definition declares ``child_collection`` but NO |
| 30 | ``child_source_field``, that absence IS the signal for the |
| 31 | "synthesized two-line depreciation shape" -- the definition MUST then |
| 32 | carry ``debit_account_id``/``credit_account_id`` plus a reserved |
| 33 | ``mapping.amount`` entry whose op is ``depreciation_amount``; exactly |
| 34 | two child lines are synthesized (debit the expense account, credit |
| 35 | accumulated depreciation, same amount both lines) rather than parsed |
| 36 | from source-row JSON. See ``_build_depreciation_child_rows``. Only |
| 37 | ``straight_line`` ships; a definition requesting ``declining`` is |
| 38 | rejected at parse time (``_validate_mapping_entry``), never silently |
| 39 | mis-computed -- 61's Purpose and Coverage names declining-balance as |
| 40 | the one worked case that brushes the escalate-to-custom-object line. |
| 41 | 3. Template execution / CreateWork (``tasks`` self-referential, |
| 42 | ``trigger.mode: "event"``, ``output_collection == source_collection``): |
| 43 | detected structurally (output equals source) rather than by a separate |
| 44 | flag. This is an UPDATE to the SAME row, gated by the fill-only-if-empty |
| 45 | rule (61's Events section: "must never clobber a field the record's |
| 46 | own creator already set explicitly"). 61's own worked example 3 |
| 47 | describes the CreateWork mechanism as a template-relation hop (fetch |
| 48 | the ``templates`` row a ``template_id``-shaped field points at, parse |
| 49 | its ``default_values`` JSON, fill empty fields) -- notably NOT |
| 50 | expressed through this block's ordinary ``mapping`` vocabulary, since |
| 51 | ``mapping`` is explicitly forbidden from cross-collection joins |
| 52 | (Out of Scope). This module supports BOTH: an ordinary |
| 53 | ``mapping``-driven fill (works for any CreateWork-shaped definition, |
| 54 | generic, no relation lookup) applied first, then an opportunistic |
| 55 | template-default-values hop (``_apply_template_defaults``, worked |
| 56 | example 3's exact mechanism) applied second -- both fill-only-if-empty, |
| 57 | so whichever gets there first for a given field wins and the other is |
| 58 | a no-op for that field. This is a deliberate interpretation of an |
| 59 | underspecified corner, documented here rather than by omission. |
| 60 | |
| 61 | **Crash-safety -- the honest, non-transactional sequencing (61's Storage |
| 62 | section, read literally, implemented literally):** |
| 63 | |
| 64 | 1. Compute the ENTIRE output in memory first, including the balance |
| 65 | check (exact integer cents, never float), BEFORE any write touches |
| 66 | disk. Unequal debit/credit sums -> abort, zero rows written, zero |
| 67 | risk of a half-posted journal (``BalanceCheckFailed``, raised |
| 68 | before ``_generate_new_record`` writes anything). |
| 69 | 2. Write child rows FIRST. Each child's own id is ALSO deterministic |
| 70 | (``{header_id}_line_{n}``), so a duplicate-id create from a |
| 71 | resumed partial run is a harmless, caught, skipped no-op |
| 72 | (``object_records.DuplicateRecordIdError``), never fatal. |
| 73 | 3. Write the header LAST, at its deterministic id |
| 74 | (``idempotency_key``). The header's EXISTENCE, not the children's, |
| 75 | is what "this period is generated" means -- the daemon's due-check |
| 76 | (``_record_exists``) queries ``output_collection`` (the header |
| 77 | collection) for that id, never the child collection. A crash |
| 78 | between the last child write and the header write leaves inert, |
| 79 | orphaned, headerless child rows -- an accepted, bounded, ACTUAL |
| 80 | failure mode (not swept under the rug): the next pass sees no |
| 81 | header, re-attempts, re-writes any still-missing children |
| 82 | (idempotent by id), and writes the header this time. This module |
| 83 | does NOT claim cross-collection atomicity, because there isn't |
| 84 | any -- see 61's Storage section for the full honest accounting of |
| 85 | why this ordering is what "detectably safe rather than silently |
| 86 | wrong" actually buys, and what it does not. |
| 87 | |
| 88 | **Scheduled due-set, recomputed from scratch every pass, same stance |
| 89 | 14-rollup-spec.md takes for its own scheduled recompute:** the daemon |
| 90 | pass does NOT trust a source row's own ``next_run``/``last_run`` (or this |
| 91 | definition's own ``last_run_at``) as the mechanism of truth. Every pass |
| 92 | recomputes which periods SHOULD exist between a row's anchor point and |
| 93 | now, and checks the deterministic header id's existence in |
| 94 | ``output_collection`` for each -- missing means generate, present means |
| 95 | skip, never an error. ``last_run_at``/a source row's own |
| 96 | ``next_run``/``last_run`` are stamped afterward purely as a freshness |
| 97 | DISPLAY, never read back as the correctness gate. |
| 98 | |
| 99 | **generated_from -- built, not left open.** 61's Open Questions flags this |
| 100 | as "arguably should just be built." When ``output_collection``'s schema |
| 101 | declares a plain ``generated_from`` field, every top-level generated |
| 102 | record (and CreateWork's in-place fill, fill-only-if-empty like every |
| 103 | other field) is stamped with a small JSON blob (``definition_id``, |
| 104 | ``source_id``, ``period_start``) -- additive, opt-in per output schema |
| 105 | (a schema that hasn't added the field simply never receives it, no |
| 106 | error), first-class provenance rather than only inferable from the |
| 107 | deterministic id's string shape. |
| 108 | """ |
| 109 | from __future__ import annotations |
| 110 | |
| 111 | import calendar |
| 112 | import json |
| 113 | import re |
| 114 | from dataclasses import dataclass |
| 115 | from datetime import date, datetime, timedelta, timezone |
| 116 | from typing import Any, Mapping |
| 117 | |
| 118 | import object_collections |
| 119 | import object_feature_flags |
| 120 | import object_records |
| 121 | import object_schemas |
| 122 | |
| 123 | MATERIALIZE_DEFINITIONS_COLLECTION = "materialize_definitions" |
| 124 | FEATURE_FLAGS_COLLECTION = "feature_flags" |
| 125 | MATERIALIZE_ENABLED_FLAG = "materialize_enabled" |
| 126 | DEFAULT_ACTOR = "daemon:materialize" |
| 127 | |
| 128 | _GRANULARITIES = frozenset({"daily", "weekly", "monthly", "quarterly", "yearly"}) |
| 129 | _TRIGGER_MODES = frozenset({"scheduled", "scheduled_fixed", "event"}) |
| 130 | _MAPPING_OP_KEYS = frozenset({"from", "from_period", "literal", "template", "if", "depreciation_amount"}) |
| 131 | _TEMPLATE_TOKEN_RE = re.compile(r"\{([a-zA-Z0-9_.]+)\}") |
| 132 | |
| 133 | # Safety valve against a malformed/huge span producing an unbounded period |
| 134 | # loop (e.g. an anchor_field decades in the past with a daily granularity). |
| 135 | # Not part of the spec; a defensive bound only. |
| 136 | _MAX_PERIODS_PER_ROW = 100_000 |
| 137 | |
| 138 | |
| 139 | class DefinitionError(ValueError): |
| 140 | """Raised when a materialize_definitions record cannot be generated as-is. |
| 141 | |
| 142 | Covers malformed JSON sub-fields, violations of 61's own doctrine (a |
| 143 | filter value that isn't a flat scalar, an unrecognized mapping op, |
| 144 | ``declining`` depreciation requested), and per-row generation problems |
| 145 | (unparseable child_source_field JSON, a missing anchor field). The |
| 146 | daemon pass (object_daemon.process_materializations) and |
| 147 | materialize_run both catch this per-definition or per-row, matching |
| 148 | object_rollups.DefinitionError's isolation posture. |
| 149 | """ |
| 150 | |
| 151 | |
| 152 | class MissingCollectionError(DefinitionError): |
| 153 | """A definition names a source/output/child collection that doesn't exist. |
| 154 | |
| 155 | Distinguished from a generic DefinitionError so the daemon can log it |
| 156 | exactly once per process (61's Degradation: "warned-once-per-process, |
| 157 | matching process_stale_transitions' _WARNED_UNKNOWN_AUTO_TRANSITION_ |
| 158 | COLLECTIONS pattern exactly") rather than re-logging every poll. |
| 159 | """ |
| 160 | |
| 161 | def __init__(self, collection: str, role: str): |
| 162 | super().__init__(f"{role} collection '{collection}' does not exist") |
| 163 | self.collection = collection |
| 164 | self.role = role |
| 165 | |
| 166 | |
| 167 | class BalanceCheckFailed(DefinitionError): |
| 168 | """A generation's computed child rows don't balance -- zero rows written. |
| 169 | |
| 170 | Raised BEFORE any write touches disk (see module docstring's |
| 171 | crash-safety section); the caller's per-row isolation logs this and |
| 172 | retries next pass, since nothing was written and the deterministic id |
| 173 | therefore never came to exist. |
| 174 | """ |
| 175 | |
| 176 | |
| 177 | @dataclass(frozen=True) |
| 178 | class Period: |
| 179 | """One computed generation period (scheduled/scheduled_fixed only). |
| 180 | |
| 181 | ``index``/``total`` are the 1-based period number and the definition's |
| 182 | total period cap -- present only for ``scheduled_fixed`` (depreciation's |
| 183 | "which month is this, out of how many"), None for ``scheduled``. |
| 184 | """ |
| 185 | |
| 186 | start: date |
| 187 | end: date |
| 188 | label: str |
| 189 | index: int | None |
| 190 | total: int | None |
| 191 | |
| 192 | |
| 193 | @dataclass(frozen=True) |
| 194 | class MaterializeConfig: |
| 195 | definition_id: str |
| 196 | name: str |
| 197 | source_collection: str |
| 198 | source_filter: dict[str, Any] |
| 199 | trigger_mode: str # "scheduled" | "scheduled_fixed" | "event" |
| 200 | trigger_interval_seconds: int | None |
| 201 | # scheduled mode |
| 202 | anchor_field: str | None |
| 203 | frequency_field: str | None # names a FIELD on the source row (e.g. "frequency") |
| 204 | # scheduled_fixed mode |
| 205 | start_field: str | None |
| 206 | granularity: str | None # a literal fixed granularity (e.g. "monthly") |
| 207 | periods_field: str | None # names a FIELD on the source row (e.g. "useful_life_months") |
| 208 | # output |
| 209 | output_collection: str |
| 210 | child_collection: str | None |
| 211 | child_source_field: str | None |
| 212 | child_link_field: str | None |
| 213 | idempotency_key: str |
| 214 | mapping: dict[str, Any] |
| 215 | balance_check: dict[str, str] | None |
| 216 | debit_account_id: str | None |
| 217 | credit_account_id: str | None |
| 218 | synthesized_amount_entry: dict[str, Any] | None |
| 219 | actor: str |
| 220 | enabled: bool |
| 221 | block: bool |
| 222 | stamp_generated_from: bool |
| 223 | |
| 224 | |
| 225 | # --- Feature flag / due gates ------------------------------------------------- |
| 226 | |
| 227 | def materialize_pass_enabled(*, base_dir: Any) -> bool: |
| 228 | """Block-wide kill switch, ``<block>_enabled`` convention, mirroring |
| 229 | object_rollups.rollup_pass_enabled exactly (same defaults, same |
| 230 | reasoning): default ON, a missing/unreadable feature_flags collection |
| 231 | or a blank value both resolve to "on"; only an explicit off/false/0/no |
| 232 | value turns the pass off. Unlike 14's posture (stale-but-valid reads |
| 233 | keep serving), 61's Degradation is explicit that here "off means off, |
| 234 | full stop, for BOTH the scheduled and manual paths" -- callers (the |
| 235 | daemon pass, materialize_run, materialize_seed's EVENT handler) are |
| 236 | each responsible for checking this before doing anything, since there |
| 237 | is no "serve what's on disk" fallback for a write-side block. |
| 238 | """ |
| 239 | return object_feature_flags.kill_switch_on(MATERIALIZE_ENABLED_FLAG, base_dir=base_dir) |
| 240 | |
| 241 | |
| 242 | def is_definition_enabled(record: Mapping[str, str]) -> bool: |
| 243 | """Per-definition pause -- independent of the block-wide flag. Blank |
| 244 | defaults to enabled, matching object_rollups.is_definition_enabled. |
| 245 | """ |
| 246 | return _truthy(record.get("enabled"), default=True) |
| 247 | |
| 248 | |
| 249 | def is_definition_blocked(record: Mapping[str, str]) -> bool: |
| 250 | """The harder stop (61's Degradation): ``block: true`` refuses even |
| 251 | ``materialize_run``'s manual path with a distinct error, unlike |
| 252 | ``enabled: false`` (pause, resume later without thinking). |
| 253 | """ |
| 254 | return _truthy(record.get("block"), default=False) |
| 255 | |
| 256 | |
| 257 | def is_definition_due(record: Mapping[str, str], *, now: datetime | None = None) -> bool: |
| 258 | """SCHEDULED/SCHEDULED_FIXED due gate: ``last_run_at`` + |
| 259 | ``trigger.interval_seconds``, both read straight off the raw record -- |
| 260 | mirrors object_rollups.is_definition_due's structure exactly. A |
| 261 | never-run definition (``last_run_at`` blank) is always due. This gate |
| 262 | only decides whether the daemon PASS attempts a definition this poll; |
| 263 | it is not the correctness mechanism (that's the deterministic header |
| 264 | id, see module docstring) -- a due-but-already-generated period is |
| 265 | simply reported as skipped, not an error. |
| 266 | """ |
| 267 | now = now or datetime.now(timezone.utc) |
| 268 | last_run_raw = (record.get("last_run_at") or "").strip() |
| 269 | if not last_run_raw: |
| 270 | return True |
| 271 | trigger = _json_field(record, "trigger", None, definition_id=record.get("id") or "<unknown>") |
| 272 | interval = _positive_int(trigger.get("interval_seconds")) if isinstance(trigger, dict) else None |
| 273 | if interval is None: |
| 274 | return True |
| 275 | last_run = _parse_iso(last_run_raw) |
| 276 | if last_run is None: |
| 277 | return True |
| 278 | return (now - last_run).total_seconds() >= interval |
| 279 | |
| 280 | |
| 281 | # --- Definition parsing ------------------------------------------------- |
| 282 | |
| 283 | def parse_definition( |
| 284 | record: Mapping[str, str], *, base_dir: Any, roots: Any = None |
| 285 | ) -> MaterializeConfig: |
| 286 | """Validate and normalize one ``materialize_definitions`` row. |
| 287 | |
| 288 | Raises ``DefinitionError``/``MissingCollectionError`` for anything |
| 289 | malformed or referencing a collection that doesn't exist yet -- the |
| 290 | daemon (object_daemon.process_materializations) and materialize_run |
| 291 | both isolate this per-definition, never letting one bad definition |
| 292 | stop any other (61's Events/Degradation). |
| 293 | """ |
| 294 | definition_id = (record.get("id") or "").strip() |
| 295 | if not definition_id: |
| 296 | raise DefinitionError("materialize definition is missing an id") |
| 297 | |
| 298 | name = (record.get("name") or "").strip() or definition_id |
| 299 | |
| 300 | source_collection = (record.get("source_collection") or "").strip() |
| 301 | if not source_collection or not object_collections.validate_collection_name(source_collection): |
| 302 | raise DefinitionError(f"{definition_id}: source_collection is required and must be a valid collection name") |
| 303 | try: |
| 304 | object_schemas.get_schema(source_collection, base_dir=base_dir, roots=roots) |
| 305 | except object_schemas.SchemaNotFoundError: |
| 306 | raise MissingCollectionError(source_collection, "source") from None |
| 307 | |
| 308 | source_filter = _parse_filter(record, definition_id=definition_id) |
| 309 | |
| 310 | trigger = _json_field(record, "trigger", None, definition_id=definition_id) |
| 311 | if not isinstance(trigger, dict) or not trigger: |
| 312 | raise DefinitionError(f"{definition_id}: trigger is required and must be a JSON object") |
| 313 | trigger_mode = str(trigger.get("mode") or "").strip() |
| 314 | if trigger_mode not in _TRIGGER_MODES: |
| 315 | raise DefinitionError(f"{definition_id}: trigger.mode must be one of {sorted(_TRIGGER_MODES)}") |
| 316 | |
| 317 | trigger_interval_seconds: int | None = None |
| 318 | anchor_field = frequency_field = start_field = granularity = periods_field = None |
| 319 | |
| 320 | if trigger_mode == "scheduled": |
| 321 | trigger_interval_seconds = _positive_int(trigger.get("interval_seconds")) |
| 322 | if trigger_interval_seconds is None: |
| 323 | raise DefinitionError(f"{definition_id}: trigger.interval_seconds is required and must be a positive integer") |
| 324 | frequency_field = str(trigger.get("period_field") or "").strip() |
| 325 | if not frequency_field: |
| 326 | raise DefinitionError( |
| 327 | f"{definition_id}: trigger.period_field (the source row field naming its own " |
| 328 | "frequency, e.g. \"frequency\") is required for scheduled mode" |
| 329 | ) |
| 330 | anchor_field = str(trigger.get("anchor_field") or "").strip() |
| 331 | if not anchor_field: |
| 332 | raise DefinitionError(f"{definition_id}: trigger.anchor_field is required for scheduled mode") |
| 333 | elif trigger_mode == "scheduled_fixed": |
| 334 | trigger_interval_seconds = _positive_int(trigger.get("interval_seconds")) |
| 335 | if trigger_interval_seconds is None: |
| 336 | raise DefinitionError(f"{definition_id}: trigger.interval_seconds is required and must be a positive integer") |
| 337 | granularity = str(trigger.get("period_field") or "").strip().lower() |
| 338 | if granularity not in _GRANULARITIES: |
| 339 | raise DefinitionError( |
| 340 | f"{definition_id}: trigger.period_field must be a fixed granularity, one of " |
| 341 | f"{sorted(_GRANULARITIES)}, for scheduled_fixed mode" |
| 342 | ) |
| 343 | start_field = str(trigger.get("start_field") or "").strip() |
| 344 | if not start_field: |
| 345 | raise DefinitionError(f"{definition_id}: trigger.start_field is required for scheduled_fixed mode") |
| 346 | end_condition = trigger.get("end_condition") |
| 347 | if not isinstance(end_condition, dict): |
| 348 | raise DefinitionError(f"{definition_id}: trigger.end_condition is required for scheduled_fixed mode") |
| 349 | periods_field = str(end_condition.get("periods_field") or "").strip() |
| 350 | if not periods_field: |
| 351 | raise DefinitionError(f"{definition_id}: trigger.end_condition.periods_field is required for scheduled_fixed mode") |
| 352 | else: # event |
| 353 | on = str(trigger.get("on") or "").strip() |
| 354 | if on != "record.created": |
| 355 | raise DefinitionError(f"{definition_id}: trigger.on must be 'record.created' for event mode") |
| 356 | |
| 357 | output_collection = (record.get("output_collection") or "").strip() |
| 358 | if not output_collection or not object_collections.validate_collection_name(output_collection): |
| 359 | raise DefinitionError(f"{definition_id}: output_collection is required and must be a valid collection name") |
| 360 | try: |
| 361 | object_schemas.get_schema(output_collection, base_dir=base_dir, roots=roots) |
| 362 | except object_schemas.SchemaNotFoundError: |
| 363 | raise MissingCollectionError(output_collection, "output") from None |
| 364 | |
| 365 | child_collection = (record.get("child_collection") or "").strip() or None |
| 366 | child_source_field = (record.get("child_source_field") or "").strip() or None |
| 367 | child_link_field = (record.get("child_link_field") or "").strip() or None |
| 368 | if child_collection: |
| 369 | if not object_collections.validate_collection_name(child_collection): |
| 370 | raise DefinitionError(f"{definition_id}: child_collection must be a valid collection name") |
| 371 | try: |
| 372 | object_schemas.get_schema(child_collection, base_dir=base_dir, roots=roots) |
| 373 | except object_schemas.SchemaNotFoundError: |
| 374 | raise MissingCollectionError(child_collection, "child") from None |
| 375 | if not child_link_field: |
| 376 | raise DefinitionError(f"{definition_id}: child_link_field is required when child_collection is set") |
| 377 | |
| 378 | idempotency_key = (record.get("idempotency_key") or "").strip() |
| 379 | if not idempotency_key: |
| 380 | raise DefinitionError(f"{definition_id}: idempotency_key is required") |
| 381 | |
| 382 | mapping_raw = _json_field(record, "mapping", {}, definition_id=definition_id) |
| 383 | if mapping_raw is None: |
| 384 | mapping_raw = {} |
| 385 | if not isinstance(mapping_raw, dict): |
| 386 | raise DefinitionError(f"{definition_id}: mapping must be a JSON object") |
| 387 | mapping = dict(mapping_raw) |
| 388 | for field_name, entry in mapping.items(): |
| 389 | _validate_mapping_entry(field_name, entry, definition_id=definition_id) |
| 390 | |
| 391 | balance_check_raw = _json_field(record, "balance_check", None, definition_id=definition_id) |
| 392 | balance_check: dict[str, str] | None = None |
| 393 | if balance_check_raw is not None: |
| 394 | if not isinstance(balance_check_raw, dict): |
| 395 | raise DefinitionError(f"{definition_id}: balance_check must be a JSON object") |
| 396 | debit_field = str(balance_check_raw.get("debit_field") or "").strip() |
| 397 | credit_field = str(balance_check_raw.get("credit_field") or "").strip() |
| 398 | if not debit_field or not credit_field: |
| 399 | raise DefinitionError(f"{definition_id}: balance_check requires debit_field and credit_field") |
| 400 | if not child_collection: |
| 401 | raise DefinitionError(f"{definition_id}: balance_check requires child_collection") |
| 402 | balance_check = {"debit_field": debit_field, "credit_field": credit_field} |
| 403 | |
| 404 | debit_account_id = (record.get("debit_account_id") or "").strip() or None |
| 405 | credit_account_id = (record.get("credit_account_id") or "").strip() or None |
| 406 | |
| 407 | synthesized_amount_entry: dict[str, Any] | None = None |
| 408 | if child_collection and not child_source_field: |
| 409 | # Absence of child_source_field alongside a present child_collection |
| 410 | # is the synthesized depreciation-lines shape's own signal -- see |
| 411 | # module docstring's worked example 2 discussion. |
| 412 | amount_entry = mapping.pop("amount", None) |
| 413 | if not isinstance(amount_entry, dict) or "depreciation_amount" not in amount_entry: |
| 414 | raise DefinitionError( |
| 415 | f"{definition_id}: a child_collection with no child_source_field is the " |
| 416 | "synthesized depreciation-lines shape and requires " |
| 417 | "mapping.amount = {'depreciation_amount': {...}}" |
| 418 | ) |
| 419 | if not debit_account_id or not credit_account_id: |
| 420 | raise DefinitionError( |
| 421 | f"{definition_id}: the synthesized depreciation-lines shape requires " |
| 422 | "debit_account_id and credit_account_id" |
| 423 | ) |
| 424 | if not balance_check: |
| 425 | raise DefinitionError( |
| 426 | f"{definition_id}: the synthesized depreciation-lines shape requires " |
| 427 | "balance_check (its debit_field/credit_field name the two generated " |
| 428 | "lines' amount fields)" |
| 429 | ) |
| 430 | synthesized_amount_entry = amount_entry |
| 431 | |
| 432 | actor = (record.get("actor") or "").strip() or DEFAULT_ACTOR |
| 433 | enabled = _truthy(record.get("enabled"), default=True) |
| 434 | block = _truthy(record.get("block"), default=False) |
| 435 | |
| 436 | stamp_generated_from = _schema_has_field(output_collection, "generated_from", base_dir=base_dir, roots=roots) |
| 437 | |
| 438 | return MaterializeConfig( |
| 439 | definition_id=definition_id, |
| 440 | name=name, |
| 441 | source_collection=source_collection, |
| 442 | source_filter=source_filter, |
| 443 | trigger_mode=trigger_mode, |
| 444 | trigger_interval_seconds=trigger_interval_seconds, |
| 445 | anchor_field=anchor_field, |
| 446 | frequency_field=frequency_field, |
| 447 | start_field=start_field, |
| 448 | granularity=granularity, |
| 449 | periods_field=periods_field, |
| 450 | output_collection=output_collection, |
| 451 | child_collection=child_collection, |
| 452 | child_source_field=child_source_field, |
| 453 | child_link_field=child_link_field, |
| 454 | idempotency_key=idempotency_key, |
| 455 | mapping=mapping, |
| 456 | balance_check=balance_check, |
| 457 | debit_account_id=debit_account_id, |
| 458 | credit_account_id=credit_account_id, |
| 459 | synthesized_amount_entry=synthesized_amount_entry, |
| 460 | actor=actor, |
| 461 | enabled=enabled, |
| 462 | block=block, |
| 463 | stamp_generated_from=stamp_generated_from, |
| 464 | ) |
| 465 | |
| 466 | |
| 467 | def _validate_mapping_entry(field_name: str, entry: Any, *, definition_id: str) -> None: |
| 468 | if not isinstance(entry, dict): |
| 469 | raise DefinitionError(f"{definition_id}: mapping.{field_name} must be a JSON object") |
| 470 | |
| 471 | if "if" in entry: |
| 472 | cond = entry.get("if") |
| 473 | if not isinstance(cond, dict) or not isinstance(cond.get("source_field"), str) or not cond.get("source_field"): |
| 474 | raise DefinitionError(f"{definition_id}: mapping.{field_name}.if requires a source_field") |
| 475 | if "equals" not in cond: |
| 476 | raise DefinitionError(f"{definition_id}: mapping.{field_name}.if requires 'equals'") |
| 477 | if "then" not in entry or "else" not in entry: |
| 478 | raise DefinitionError(f"{definition_id}: mapping.{field_name} with 'if' requires 'then' and 'else'") |
| 479 | return |
| 480 | |
| 481 | if "depreciation_amount" in entry: |
| 482 | op = entry["depreciation_amount"] |
| 483 | if not isinstance(op, dict): |
| 484 | raise DefinitionError(f"{definition_id}: mapping.{field_name}.depreciation_amount must be a JSON object") |
| 485 | method = str(op.get("method") or "").strip() |
| 486 | if method == "declining": |
| 487 | raise DefinitionError( |
| 488 | f"{definition_id}: mapping.{field_name} requests depreciation method 'declining', " |
| 489 | "which is out of scope for v1 (straight_line only) -- see " |
| 490 | "plan/vocabulary/61-materialize-spec.md's Open Questions" |
| 491 | ) |
| 492 | if method != "straight_line": |
| 493 | raise DefinitionError(f"{definition_id}: mapping.{field_name}.depreciation_amount.method must be 'straight_line'") |
| 494 | for key in ("cost_field", "salvage_field", "life_field"): |
| 495 | value = op.get(key) |
| 496 | if not isinstance(value, str) or not value: |
| 497 | raise DefinitionError(f"{definition_id}: mapping.{field_name}.depreciation_amount.{key} is required") |
| 498 | return |
| 499 | |
| 500 | if "from" in entry: |
| 501 | if not isinstance(entry["from"], str) or not entry["from"]: |
| 502 | raise DefinitionError(f"{definition_id}: mapping.{field_name}.from must name a source field") |
| 503 | return |
| 504 | |
| 505 | if "from_period" in entry: |
| 506 | if entry["from_period"] not in ("period_start", "period_end", "period_label"): |
| 507 | raise DefinitionError( |
| 508 | f"{definition_id}: mapping.{field_name}.from_period must be one of " |
| 509 | "period_start/period_end/period_label" |
| 510 | ) |
| 511 | return |
| 512 | |
| 513 | if "literal" in entry: |
| 514 | return |
| 515 | |
| 516 | if "template" in entry: |
| 517 | if not isinstance(entry["template"], str): |
| 518 | raise DefinitionError(f"{definition_id}: mapping.{field_name}.template must be a string") |
| 519 | return |
| 520 | |
| 521 | raise DefinitionError( |
| 522 | f"{definition_id}: mapping.{field_name} must use one of from/from_period/literal/" |
| 523 | "template/if/depreciation_amount" |
| 524 | ) |
| 525 | |
| 526 | |
| 527 | def _parse_filter(record: Mapping[str, str], *, definition_id: str) -> dict[str, Any]: |
| 528 | raw = _json_field(record, "source_filter", {}, definition_id=definition_id) |
| 529 | if raw is None: |
| 530 | return {} |
| 531 | if not isinstance(raw, dict): |
| 532 | raise DefinitionError(f"{definition_id}: source_filter must be a flat JSON object") |
| 533 | normalized: dict[str, Any] = {} |
| 534 | for key, value in raw.items(): |
| 535 | if isinstance(value, (dict, list)): |
| 536 | raise DefinitionError( |
| 537 | f"{definition_id}: source_filter.{key} must be a scalar -- materialize filters are " |
| 538 | "flat equality only, ANDed, same as 14-rollup-spec.md's filter language" |
| 539 | ) |
| 540 | if value is None: |
| 541 | raise DefinitionError(f"{definition_id}: source_filter.{key} must not be null") |
| 542 | if isinstance(value, bool): |
| 543 | value = "true" if value else "false" |
| 544 | normalized[key] = value |
| 545 | return normalized |
| 546 | |
| 547 | |
| 548 | def _schema_has_field(collection: str, field_name: str, *, base_dir: Any, roots: Any = None) -> bool: |
| 549 | try: |
| 550 | schema = object_schemas.get_schema(collection, base_dir=base_dir, roots=roots) |
| 551 | except object_schemas.SchemaNotFoundError: |
| 552 | return False |
| 553 | return any(field.get("name") == field_name for field in schema.get("fields", [])) |
| 554 | |
| 555 | |
| 556 | # --- Period computation ------------------------------------------------- |
| 557 | |
| 558 | def _compute_scheduled_periods( |
| 559 | source_row: Mapping[str, str], config: MaterializeConfig, *, now: date |
| 560 | ) -> list[Period]: |
| 561 | granularity = str(source_row.get(config.frequency_field) or "").strip().lower() |
| 562 | if granularity not in _GRANULARITIES: |
| 563 | raise DefinitionError( |
| 564 | f"source row has an invalid or missing '{config.frequency_field}' " |
| 565 | f"(must be one of {sorted(_GRANULARITIES)}); got {granularity!r}" |
| 566 | ) |
| 567 | anchor_raw = source_row.get(config.anchor_field) or "" |
| 568 | origin = _parse_date(anchor_raw) |
| 569 | if origin is None: |
| 570 | origin = _parse_date(source_row.get("created_at") or "") |
| 571 | if origin is None: |
| 572 | raise DefinitionError( |
| 573 | f"source row has no usable '{config.anchor_field}' (nor created_at) to anchor periods" |
| 574 | ) |
| 575 | return _step_periods(origin, granularity, now=now, cap=None) |
| 576 | |
| 577 | |
| 578 | def _compute_scheduled_fixed_periods( |
| 579 | source_row: Mapping[str, str], config: MaterializeConfig, *, now: date |
| 580 | ) -> list[Period]: |
| 581 | origin = _parse_date(source_row.get(config.start_field) or "") |
| 582 | if origin is None: |
| 583 | raise DefinitionError(f"source row has no usable '{config.start_field}' to anchor periods") |
| 584 | cap = _positive_int(source_row.get(config.periods_field)) |
| 585 | if cap is None: |
| 586 | raise DefinitionError(f"source row's '{config.periods_field}' must be a positive integer") |
| 587 | return _step_periods(origin, config.granularity, now=now, cap=cap) |
| 588 | |
| 589 | |
| 590 | def _step_periods(origin: date, granularity: str, *, now: date, cap: int | None) -> list[Period]: |
| 591 | periods: list[Period] = [] |
| 592 | cursor = origin |
| 593 | index = 0 |
| 594 | guard = 0 |
| 595 | while cursor <= now: |
| 596 | index += 1 |
| 597 | if cap is not None and index > cap: |
| 598 | break |
| 599 | period_end = _advance_date(cursor, granularity, 1) |
| 600 | periods.append( |
| 601 | Period( |
| 602 | start=cursor, |
| 603 | end=period_end, |
| 604 | label=_period_label(cursor, granularity), |
| 605 | index=index if cap is not None else None, |
| 606 | total=cap, |
| 607 | ) |
| 608 | ) |
| 609 | cursor = period_end |
| 610 | guard += 1 |
| 611 | if guard >= _MAX_PERIODS_PER_ROW: |
| 612 | break |
| 613 | return periods |
| 614 | |
| 615 | |
| 616 | def _due_periods_for_row( |
| 617 | config: MaterializeConfig, source_row: Mapping[str, str], *, now: date |
| 618 | ) -> list[Period | None]: |
| 619 | if config.trigger_mode == "event": |
| 620 | return [None] |
| 621 | if config.trigger_mode == "scheduled": |
| 622 | return _compute_scheduled_periods(source_row, config, now=now) |
| 623 | if config.trigger_mode == "scheduled_fixed": |
| 624 | return _compute_scheduled_fixed_periods(source_row, config, now=now) |
| 625 | raise DefinitionError(f"unknown trigger mode: {config.trigger_mode}") # pragma: no cover -- pre-validated |
| 626 | |
| 627 | |
| 628 | def _advance_date(d: date, granularity: str, steps: int = 1) -> date: |
| 629 | if granularity == "daily": |
| 630 | return d + timedelta(days=steps) |
| 631 | if granularity == "weekly": |
| 632 | return d + timedelta(weeks=steps) |
| 633 | if granularity == "monthly": |
| 634 | return _add_months(d, steps) |
| 635 | if granularity == "quarterly": |
| 636 | return _add_months(d, steps * 3) |
| 637 | if granularity == "yearly": |
| 638 | return _add_months(d, steps * 12) |
| 639 | raise DefinitionError(f"unknown granularity: {granularity}") # pragma: no cover -- pre-validated |
| 640 | |
| 641 | |
| 642 | def _add_months(d: date, months: int) -> date: |
| 643 | total = d.month - 1 + months |
| 644 | year = d.year + total // 12 |
| 645 | month = total % 12 + 1 |
| 646 | day = min(d.day, calendar.monthrange(year, month)[1]) |
| 647 | return date(year, month, day) |
| 648 | |
| 649 | |
| 650 | def _period_label(period_start: date, granularity: str) -> str: |
| 651 | if granularity == "daily": |
| 652 | return period_start.isoformat() |
| 653 | if granularity == "weekly": |
| 654 | iso_year, iso_week, _ = period_start.isocalendar() |
| 655 | return f"{iso_year}-W{iso_week:02d}" |
| 656 | if granularity == "monthly": |
| 657 | return f"{period_start.year:04d}-{period_start.month:02d}" |
| 658 | if granularity == "quarterly": |
| 659 | quarter = (period_start.month - 1) // 3 + 1 |
| 660 | return f"{period_start.year}-Q{quarter}" |
| 661 | if granularity == "yearly": |
| 662 | return str(period_start.year) |
| 663 | raise DefinitionError(f"unknown granularity: {granularity}") # pragma: no cover -- pre-validated |
| 664 | |
| 665 | |
| 666 | def _parse_date(value: Any) -> date | None: |
| 667 | text = (value or "").strip() if isinstance(value, str) else "" |
| 668 | if not text: |
| 669 | return None |
| 670 | try: |
| 671 | if "T" in text or " " in text: |
| 672 | return datetime.fromisoformat(text.replace("Z", "+00:00")).date() |
| 673 | return date.fromisoformat(text) |
| 674 | except ValueError: |
| 675 | return None |
| 676 | |
| 677 | |
| 678 | # --- Mapping vocabulary ------------------------------------------------- |
| 679 | |
| 680 | def _eval_entry( |
| 681 | field_name: str, entry: Mapping[str, Any], *, source_row: Mapping[str, str], |
| 682 | period: Period | None, definition_id: str, |
| 683 | ) -> str: |
| 684 | if "from" in entry: |
| 685 | return _stringify(source_row.get(entry["from"], "")) |
| 686 | |
| 687 | if "from_period" in entry: |
| 688 | if period is None: |
| 689 | raise DefinitionError( |
| 690 | f"{definition_id}: mapping.{field_name} uses from_period but this generation " |
| 691 | "has no period (event/manual single-shot/CreateWork)" |
| 692 | ) |
| 693 | key = entry["from_period"] |
| 694 | if key == "period_start": |
| 695 | return period.start.isoformat() |
| 696 | if key == "period_end": |
| 697 | return period.end.isoformat() |
| 698 | if key == "period_label": |
| 699 | return period.label |
| 700 | raise DefinitionError( # pragma: no cover -- pre-validated |
| 701 | f"{definition_id}: mapping.{field_name}.from_period must be period_start/period_end/period_label" |
| 702 | ) |
| 703 | |
| 704 | if "literal" in entry: |
| 705 | return _stringify(entry["literal"]) |
| 706 | |
| 707 | if "template" in entry: |
| 708 | return _render_template( |
| 709 | entry["template"], source_row=source_row, period=period, |
| 710 | definition_id=definition_id, field_name=field_name, |
| 711 | ) |
| 712 | |
| 713 | if "if" in entry: |
| 714 | cond = entry["if"] |
| 715 | actual = source_row.get(cond["source_field"]) |
| 716 | matches = _loose_equals(actual, cond["equals"]) |
| 717 | return _stringify(entry["then"] if matches else entry["else"]) |
| 718 | |
| 719 | if "depreciation_amount" in entry: |
| 720 | return _eval_depreciation( |
| 721 | entry["depreciation_amount"], source_row, period, |
| 722 | definition_id=definition_id, field_name=field_name, |
| 723 | ) |
| 724 | |
| 725 | raise DefinitionError(f"{definition_id}: mapping.{field_name} uses an unrecognized op") # pragma: no cover |
| 726 | |
| 727 | |
| 728 | def _loose_equals(actual: Any, expected: Any) -> bool: |
| 729 | if isinstance(expected, bool): |
| 730 | expected_str = "true" if expected else "false" |
| 731 | else: |
| 732 | expected_str = str(expected) |
| 733 | return str(actual if actual is not None else "") == expected_str |
| 734 | |
| 735 | |
| 736 | def _render_template( |
| 737 | template: str, *, source_row: Mapping[str, str], period: Period | None, |
| 738 | definition_id: str, field_name: str, |
| 739 | ) -> str: |
| 740 | def replace(match: re.Match[str]) -> str: |
| 741 | token = match.group(1) |
| 742 | if token == "period_label": |
| 743 | if period is None: |
| 744 | raise DefinitionError( |
| 745 | f"{definition_id}: mapping.{field_name} template uses {{period_label}} but " |
| 746 | "this generation has no period" |
| 747 | ) |
| 748 | return period.label |
| 749 | if token.startswith("source."): |
| 750 | field = token[len("source."):] |
| 751 | return _stringify(source_row.get(field, "")) |
| 752 | raise DefinitionError( |
| 753 | f"{definition_id}: mapping.{field_name} template references unknown placeholder '{{{token}}}'" |
| 754 | ) |
| 755 | |
| 756 | return _TEMPLATE_TOKEN_RE.sub(replace, template) |
| 757 | |
| 758 | |
| 759 | def _eval_depreciation( |
| 760 | op: Mapping[str, Any], source_row: Mapping[str, str], period: Period | None, |
| 761 | *, definition_id: str, field_name: str, |
| 762 | ) -> str: |
| 763 | if period is None or period.total is None or period.index is None: |
| 764 | raise DefinitionError( |
| 765 | f"{definition_id}: mapping.{field_name} uses depreciation_amount, which requires a " |
| 766 | "scheduled_fixed trigger with an end_condition.periods_field cap" |
| 767 | ) |
| 768 | monthly, remainder = _straight_line_split(op, source_row, definition_id=definition_id, field_name=field_name) |
| 769 | if period.index == period.total: |
| 770 | return str(monthly + remainder) |
| 771 | return str(monthly) |
| 772 | |
| 773 | |
| 774 | def _straight_line_split( |
| 775 | op: Mapping[str, Any], source_row: Mapping[str, str], *, definition_id: str, field_name: str, |
| 776 | ) -> tuple[int, int]: |
| 777 | cost = _to_cents(source_row.get(op["cost_field"])) |
| 778 | salvage = _to_cents(source_row.get(op["salvage_field"])) |
| 779 | life = _positive_int(source_row.get(op["life_field"])) |
| 780 | if life is None: |
| 781 | raise DefinitionError( |
| 782 | f"{definition_id}: mapping.{field_name}.depreciation_amount.life_field must be a " |
| 783 | "positive integer on the source row" |
| 784 | ) |
| 785 | depreciable = cost - salvage |
| 786 | monthly = depreciable // life |
| 787 | remainder = depreciable - (monthly * life) |
| 788 | return monthly, remainder |
| 789 | |
| 790 | |
| 791 | # --- Generation orchestration ------------------------------------------------- |
| 792 | |
| 793 | def generate_definition( |
| 794 | record: Mapping[str, str], *, base_dir: Any, roots: Any = None, now: datetime | None = None, |
| 795 | ) -> dict[str, Any]: |
| 796 | """Parse ``record`` and run its full due-set generation. |
| 797 | |
| 798 | The SAME function the daemon's scheduled pass and ``materialize_run``'s |
| 799 | manual path both call (61's Events: "the exact same idempotent, |
| 800 | per-source-row function") -- for scheduled/scheduled_fixed this |
| 801 | iterates every matching source row and every due period; for event |
| 802 | mode (when called manually, since the daemon's scheduled pass never |
| 803 | calls this on an event-mode definition -- see object_daemon. |
| 804 | process_materializations) this iterates every matching source row |
| 805 | once each, no period. Returns {"definition_id", "checked", "generated", |
| 806 | "skipped_already_generated", "errors": [{"source_id", "period_start", |
| 807 | "error"}, ...]}. Raises DefinitionError/MissingCollectionError for a |
| 808 | malformed definition -- the caller isolates that per-definition; every |
| 809 | per-row/per-period failure INSIDE a valid definition is instead |
| 810 | captured in the returned "errors" list, never raised, so one bad row |
| 811 | never stops any other (61's two-level isolation, mirroring |
| 812 | process_stale_transitions). |
| 813 | """ |
| 814 | config = parse_definition(record, base_dir=base_dir, roots=roots) |
| 815 | return generate_config(config, base_dir=base_dir, roots=roots, now=now) |
| 816 | |
| 817 | |
| 818 | def generate_config( |
| 819 | config: MaterializeConfig, *, base_dir: Any, roots: Any = None, now: datetime | None = None, |
| 820 | ) -> dict[str, Any]: |
| 821 | now_dt = now or datetime.now(timezone.utc) |
| 822 | now_date = now_dt.date() |
| 823 | |
| 824 | source_rows = object_records.read_collection_records(config.source_collection, base_dir=base_dir, roots=roots) |
| 825 | if config.source_filter: |
| 826 | source_rows = object_records.filter_records(source_rows, config.source_filter) |
| 827 | |
| 828 | checked = generated = skipped = 0 |
| 829 | errors: list[dict[str, Any]] = [] |
| 830 | |
| 831 | for row in source_rows: |
| 832 | source_id = row.get("id") or "<unknown>" |
| 833 | try: |
| 834 | periods = _due_periods_for_row(config, row, now=now_date) |
| 835 | except Exception as exc: |
| 836 | errors.append({"source_id": source_id, "period_start": None, "error": str(exc)}) |
| 837 | continue |
| 838 | |
| 839 | for period in periods: |
| 840 | checked += 1 |
| 841 | try: |
| 842 | result = generate_one(config, row, period, base_dir=base_dir, roots=roots) |
| 843 | except Exception as exc: |
| 844 | errors.append({ |
| 845 | "source_id": source_id, |
| 846 | "period_start": period.start.isoformat() if period is not None else None, |
| 847 | "error": str(exc), |
| 848 | }) |
| 849 | continue |
| 850 | |
| 851 | if result["status"] == "generated": |
| 852 | generated += 1 |
| 853 | elif result["status"] == "skipped_already_generated": |
| 854 | skipped += 1 |
| 855 | else: # pragma: no cover -- defensive, generate_one only returns these two |
| 856 | errors.append({ |
| 857 | "source_id": source_id, |
| 858 | "period_start": period.start.isoformat() if period is not None else None, |
| 859 | "error": f"unexpected status: {result['status']}", |
| 860 | }) |
| 861 | |
| 862 | return { |
| 863 | "definition_id": config.definition_id, |
| 864 | "checked": checked, |
| 865 | "generated": generated, |
| 866 | "skipped_already_generated": skipped, |
| 867 | "errors": errors, |
| 868 | } |
| 869 | |
| 870 | |
| 871 | def generate_one( |
| 872 | config: MaterializeConfig, source_row: Mapping[str, str], period: Period | None, |
| 873 | *, base_dir: Any, roots: Any = None, |
| 874 | ) -> dict[str, Any]: |
| 875 | """Generate (or confirm-already-generated) ONE (source row, period) pair. |
| 876 | |
| 877 | Branches structurally on ``output_collection == source_collection`` |
| 878 | (CreateWork: an in-place update, no new record) vs. every other shape |
| 879 | (a new header record, optionally with children) -- see module |
| 880 | docstring. |
| 881 | """ |
| 882 | if config.output_collection == config.source_collection: |
| 883 | return _generate_creatework(config, source_row, base_dir=base_dir, roots=roots) |
| 884 | return _generate_new_record(config, source_row, period, base_dir=base_dir, roots=roots) |
| 885 | |
| 886 | |
| 887 | def _generate_new_record( |
| 888 | config: MaterializeConfig, source_row: Mapping[str, str], period: Period | None, |
| 889 | *, base_dir: Any, roots: Any = None, |
| 890 | ) -> dict[str, Any]: |
| 891 | header_id = _render_idempotency_key(config, source_row=source_row, period=period) |
| 892 | |
| 893 | # The ONLY due-check: does the header already exist? (module docstring) |
| 894 | if _record_exists(config.output_collection, header_id, base_dir=base_dir, roots=roots): |
| 895 | return {"status": "skipped_already_generated", "header_id": header_id} |
| 896 | |
| 897 | header_fields: dict[str, Any] = {"id": header_id} |
| 898 | for field_name, entry in config.mapping.items(): |
| 899 | header_fields[field_name] = _eval_entry( |
| 900 | field_name, entry, source_row=source_row, period=period, definition_id=config.definition_id, |
| 901 | ) |
| 902 | if config.stamp_generated_from: |
| 903 | header_fields["generated_from"] = _generated_from_value(config, source_row=source_row, period=period) |
| 904 | |
| 905 | child_rows = _build_child_rows(config, source_row=source_row, header_id=header_id, period=period) |
| 906 | |
| 907 | if config.balance_check: |
| 908 | debit_total, credit_total = _sum_balance(config, child_rows) |
| 909 | if debit_total != credit_total: |
| 910 | raise BalanceCheckFailed( |
| 911 | f"{config.definition_id}: unbalanced generation for source " |
| 912 | f"{source_row.get('id')} period {period.start.isoformat() if period else '-'}: " |
| 913 | f"debits={debit_total} credits={credit_total} cents -- aborted before any write" |
| 914 | ) |
| 915 | |
| 916 | # Crash-safe ordering: children first (idempotent by id), header last |
| 917 | # (the commit signal) -- see module docstring's Storage section. |
| 918 | for child in child_rows: |
| 919 | try: |
| 920 | object_records.create_collection_record( |
| 921 | config.child_collection, child, base_dir=base_dir, roots=roots, actor=config.actor, |
| 922 | ) |
| 923 | except object_records.DuplicateRecordIdError: |
| 924 | pass # a prior partial run already wrote this line -- harmless |
| 925 | |
| 926 | try: |
| 927 | object_records.create_collection_record( |
| 928 | config.output_collection, header_fields, base_dir=base_dir, roots=roots, actor=config.actor, |
| 929 | ) |
| 930 | except object_records.DuplicateRecordIdError: |
| 931 | # Raced with another pass/process between our existence-check and |
| 932 | # now -- the other writer's header is the real one; ours is a |
| 933 | # no-op, not a failure. |
| 934 | return {"status": "skipped_already_generated", "header_id": header_id} |
| 935 | |
| 936 | return {"status": "generated", "header_id": header_id} |
| 937 | |
| 938 | |
| 939 | def _generate_creatework( |
| 940 | config: MaterializeConfig, source_row: Mapping[str, str], *, base_dir: Any, roots: Any = None, |
| 941 | ) -> dict[str, Any]: |
| 942 | """CreateWork shape: fill empty fields on the SAME row, never create. |
| 943 | |
| 944 | Two independent fill mechanisms, both fill-only-if-empty (61's Events |
| 945 | "correctness rule, not a nicety"), applied in order so a field either |
| 946 | mechanism has already filled is left alone by the other: |
| 947 | 1. this definition's own ``mapping`` (generic, works for any |
| 948 | CreateWork-shaped definition). |
| 949 | 2. an opportunistic hop through a ``templates`` relation field, per |
| 950 | 61's worked example 3 exactly (``_apply_template_defaults``). |
| 951 | The whole ``changes`` dict is built in memory and written in ONE |
| 952 | ``update_collection_record`` call -- a malformed template blob aborts |
| 953 | the row cleanly with nothing written, same "compute fully, then write |
| 954 | once" discipline as the header+children path, at a smaller scale. |
| 955 | """ |
| 956 | source_id = source_row.get("id") or "" |
| 957 | if not source_id: |
| 958 | return {"status": "skipped_already_generated"} |
| 959 | |
| 960 | changes: dict[str, str] = {} |
| 961 | for field_name, entry in config.mapping.items(): |
| 962 | current = source_row.get(field_name) |
| 963 | if current not in (None, ""): |
| 964 | continue # fill-only-if-empty |
| 965 | value = _eval_entry(field_name, entry, source_row=source_row, period=None, definition_id=config.definition_id) |
| 966 | if value in (None, ""): |
| 967 | continue |
| 968 | changes[field_name] = value |
| 969 | |
| 970 | _apply_template_defaults(config, source_row, changes, base_dir=base_dir, roots=roots) |
| 971 | |
| 972 | if config.stamp_generated_from and "generated_from" not in changes and ( |
| 973 | source_row.get("generated_from") in (None, "") |
| 974 | ): |
| 975 | changes["generated_from"] = _generated_from_value(config, source_row=source_row, period=None) |
| 976 | |
| 977 | if not changes: |
| 978 | return {"status": "skipped_already_generated", "header_id": source_id} |
| 979 | |
| 980 | object_records.update_collection_record( |
| 981 | config.output_collection, source_id, changes, base_dir=base_dir, roots=roots, actor=config.actor, |
| 982 | ) |
| 983 | return {"status": "generated", "header_id": source_id} |
| 984 | |
| 985 | |
| 986 | def _apply_template_defaults( |
| 987 | config: MaterializeConfig, source_row: Mapping[str, str], changes: dict[str, str], |
| 988 | *, base_dir: Any, roots: Any = None, |
| 989 | ) -> None: |
| 990 | """61's worked example 3, literally: find a relation field on the |
| 991 | OUTPUT schema pointing at ``templates``, follow it, parse that |
| 992 | template's ``default_values`` JSON, fill-only-if-empty. Best-effort in |
| 993 | the sense that "no such field," "no id set," "row not found" are all |
| 994 | quiet no-ops (this definition simply isn't the templates-relation |
| 995 | shape, or the reference is dangling) -- but a malformed |
| 996 | ``default_values`` JSON blob is allowed to raise, surfacing as a |
| 997 | logged per-row error rather than silently doing nothing, since that |
| 998 | is a real data problem worth an operator's attention. |
| 999 | """ |
| 1000 | try: |
| 1001 | schema = object_schemas.get_schema(config.output_collection, base_dir=base_dir, roots=roots) |
| 1002 | except object_schemas.SchemaNotFoundError: |
| 1003 | return |
| 1004 | |
| 1005 | template_field = None |
| 1006 | for field in schema.get("fields", []): |
| 1007 | relation = field.get("relation") or {} |
| 1008 | if relation.get("collection") == "templates": |
| 1009 | template_field = field.get("name") |
| 1010 | break |
| 1011 | if not template_field: |
| 1012 | return |
| 1013 | |
| 1014 | template_id = source_row.get(template_field) |
| 1015 | if not template_id: |
| 1016 | return |
| 1017 | |
| 1018 | try: |
| 1019 | template_row = object_records.get_collection_record("templates", template_id, base_dir=base_dir, roots=roots) |
| 1020 | except ( |
| 1021 | object_records.RecordNotFoundError, |
| 1022 | object_records.InvalidRecordIdError, |
| 1023 | object_collections.CollectionNotFoundError, |
| 1024 | object_collections.InvalidCollectionNameError, |
| 1025 | ): |
| 1026 | return |
| 1027 | |
| 1028 | raw_defaults = (template_row.get("default_values") or "").strip() |
| 1029 | if not raw_defaults: |
| 1030 | return |
| 1031 | |
| 1032 | defaults = json.loads(raw_defaults) # malformed JSON deliberately propagates -- see docstring |
| 1033 | if not isinstance(defaults, dict): |
| 1034 | return |
| 1035 | |
| 1036 | for field_name, value in defaults.items(): |
| 1037 | if field_name in changes: |
| 1038 | continue |
| 1039 | current = source_row.get(field_name) |
| 1040 | if current not in (None, ""): |
| 1041 | continue |
| 1042 | changes[field_name] = _stringify(value) |
| 1043 | |
| 1044 | |
| 1045 | def _record_exists(collection: str, record_id: str, *, base_dir: Any, roots: Any = None) -> bool: |
| 1046 | try: |
| 1047 | object_records.get_collection_record(collection, record_id, base_dir=base_dir, roots=roots) |
| 1048 | return True |
| 1049 | except object_records.RecordNotFoundError: |
| 1050 | return False |
| 1051 | |
| 1052 | |
| 1053 | def _build_child_rows( |
| 1054 | config: MaterializeConfig, *, source_row: Mapping[str, str], header_id: str, period: Period | None, |
| 1055 | ) -> list[dict[str, str]]: |
| 1056 | if not config.child_collection: |
| 1057 | return [] |
| 1058 | if config.child_source_field: |
| 1059 | return _build_child_rows_from_template(config, source_row=source_row, header_id=header_id) |
| 1060 | if config.synthesized_amount_entry is not None: |
| 1061 | return _build_depreciation_child_rows(config, source_row=source_row, header_id=header_id, period=period) |
| 1062 | return [] # pragma: no cover -- parse_definition already requires one of the above |
| 1063 | |
| 1064 | |
| 1065 | def _build_child_rows_from_template( |
| 1066 | config: MaterializeConfig, *, source_row: Mapping[str, str], header_id: str, |
| 1067 | ) -> list[dict[str, str]]: |
| 1068 | raw = (source_row.get(config.child_source_field) or "").strip() |
| 1069 | if not raw: |
| 1070 | return [] |
| 1071 | try: |
| 1072 | lines = json.loads(raw) |
| 1073 | except json.JSONDecodeError as exc: |
| 1074 | raise DefinitionError(f"{config.definition_id}: {config.child_source_field} is not valid JSON: {exc}") from exc |
| 1075 | if not isinstance(lines, list): |
| 1076 | raise DefinitionError(f"{config.definition_id}: {config.child_source_field} must be a JSON list of line objects") |
| 1077 | |
| 1078 | rows: list[dict[str, str]] = [] |
| 1079 | for index, line in enumerate(lines): |
| 1080 | if not isinstance(line, dict): |
| 1081 | raise DefinitionError(f"{config.definition_id}: {config.child_source_field}[{index}] must be a JSON object") |
| 1082 | child = {str(key): _stringify(value) for key, value in line.items()} |
| 1083 | child["id"] = f"{header_id}_line_{index}" |
| 1084 | child[config.child_link_field] = header_id |
| 1085 | rows.append(child) |
| 1086 | return rows |
| 1087 | |
| 1088 | |
| 1089 | def _build_depreciation_child_rows( |
| 1090 | config: MaterializeConfig, *, source_row: Mapping[str, str], header_id: str, period: Period | None, |
| 1091 | ) -> list[dict[str, str]]: |
| 1092 | amount = _eval_entry( |
| 1093 | "amount", config.synthesized_amount_entry, source_row=source_row, period=period, |
| 1094 | definition_id=config.definition_id, |
| 1095 | ) |
| 1096 | debit_field = config.balance_check["debit_field"] |
| 1097 | credit_field = config.balance_check["credit_field"] |
| 1098 | return [ |
| 1099 | { |
| 1100 | "id": f"{header_id}_line_0", config.child_link_field: header_id, |
| 1101 | "account_id": config.debit_account_id, debit_field: amount, credit_field: "0", |
| 1102 | }, |
| 1103 | { |
| 1104 | "id": f"{header_id}_line_1", config.child_link_field: header_id, |
| 1105 | "account_id": config.credit_account_id, debit_field: "0", credit_field: amount, |
| 1106 | }, |
| 1107 | ] |
| 1108 | |
| 1109 | |
| 1110 | def _sum_balance(config: MaterializeConfig, child_rows: list[dict[str, str]]) -> tuple[int, int]: |
| 1111 | debit_field = config.balance_check["debit_field"] |
| 1112 | credit_field = config.balance_check["credit_field"] |
| 1113 | debit_total = sum(_to_cents(row.get(debit_field)) for row in child_rows) |
| 1114 | credit_total = sum(_to_cents(row.get(credit_field)) for row in child_rows) |
| 1115 | return debit_total, credit_total |
| 1116 | |
| 1117 | |
| 1118 | def _render_idempotency_key(config: MaterializeConfig, *, source_row: Mapping[str, str], period: Period | None) -> str: |
| 1119 | available: dict[str, str] = { |
| 1120 | "definition_id": config.definition_id, |
| 1121 | "source_id": source_row.get("id") or "", |
| 1122 | } |
| 1123 | if period is not None: |
| 1124 | available["period_start"] = period.start.isoformat() |
| 1125 | try: |
| 1126 | return config.idempotency_key.format(**available) |
| 1127 | except KeyError as exc: |
| 1128 | raise DefinitionError( |
| 1129 | f"{config.definition_id}: idempotency_key references {exc} which is not available " |
| 1130 | "for this generation (period_start only applies to scheduled/scheduled_fixed)" |
| 1131 | ) from exc |
| 1132 | |
| 1133 | |
| 1134 | def _generated_from_value(config: MaterializeConfig, *, source_row: Mapping[str, str], period: Period | None) -> str: |
| 1135 | payload = { |
| 1136 | "definition_id": config.definition_id, |
| 1137 | "source_id": source_row.get("id") or "", |
| 1138 | "period_start": period.start.isoformat() if period is not None else None, |
| 1139 | } |
| 1140 | return json.dumps(payload, separators=(",", ":")) |
| 1141 | |
| 1142 | |
| 1143 | # --- Event mode (CreateWork dispatch) ------------------------------------------------- |
| 1144 | |
| 1145 | def event_definitions_for_collection( |
| 1146 | collection: str, *, base_dir: Any, roots: Any = None, |
| 1147 | ) -> list[dict[str, str]]: |
| 1148 | """Return every enabled, non-blocked, event-mode definition whose |
| 1149 | ``source_collection`` is ``collection`` -- the raw records (not |
| 1150 | parsed), cheap enough to call on every dispatch (one collection read, |
| 1151 | one JSON parse per row); the full ``parse_definition`` validation |
| 1152 | happens once ``generate_one_event`` actually attempts a generation. |
| 1153 | """ |
| 1154 | try: |
| 1155 | definitions = object_records.read_collection_records( |
| 1156 | MATERIALIZE_DEFINITIONS_COLLECTION, base_dir=base_dir, roots=roots |
| 1157 | ) |
| 1158 | except (object_collections.CollectionNotFoundError, object_collections.InvalidCollectionNameError): |
| 1159 | return [] |
| 1160 | |
| 1161 | matches = [] |
| 1162 | for definition in definitions: |
| 1163 | if is_definition_blocked(definition) or not is_definition_enabled(definition): |
| 1164 | continue |
| 1165 | if (definition.get("source_collection") or "").strip() != collection: |
| 1166 | continue |
| 1167 | trigger = _json_field(definition, "trigger", None, definition_id=definition.get("id") or "<unknown>") |
| 1168 | if not isinstance(trigger, dict) or trigger.get("mode") != "event": |
| 1169 | continue |
| 1170 | matches.append(definition) |
| 1171 | return matches |
| 1172 | |
| 1173 | |
| 1174 | def generate_one_event( |
| 1175 | definition_record: Mapping[str, str], source_row: Mapping[str, str], *, base_dir: Any, roots: Any = None, |
| 1176 | ) -> dict[str, Any]: |
| 1177 | """Apply ONE event-mode definition to ONE just-created source row -- |
| 1178 | what ``materialize_seed``'s ``EVENT`` handler calls per dispatched |
| 1179 | record.created event, per definition matching that collection. |
| 1180 | """ |
| 1181 | config = parse_definition(definition_record, base_dir=base_dir, roots=roots) |
| 1182 | if config.trigger_mode != "event": |
| 1183 | raise DefinitionError(f"{config.definition_id} is not an event-mode definition") |
| 1184 | return generate_one(config, source_row, None, base_dir=base_dir, roots=roots) |
| 1185 | |
| 1186 | |
| 1187 | def compute_event_handles(*, base_dir: Any, roots: Any = None) -> list[str]: |
| 1188 | """Return the sorted, de-duplicated ``<collection>.record.created`` |
| 1189 | event list every currently enabled, non-blocked, event-mode definition |
| 1190 | implies -- i.e. what ``materialize_seed``'s static ``HANDLES`` literal |
| 1191 | WOULD need to contain for automatic on-create dispatch. |
| 1192 | |
| 1193 | Pure computation, no side effects. In v1 nothing wires this into the |
| 1194 | object's source at runtime (we do NOT rewrite installed objects under |
| 1195 | the poll loop -- see object_daemon.process_materializations); it exists |
| 1196 | for inspection/tests and as the input a future *deliberate, scheduled* |
| 1197 | HANDLES-sync mechanism (with its own tests and a real performance |
| 1198 | rationale) would use. Event-mode definitions run via the manual path |
| 1199 | (materialize_run) until then. |
| 1200 | """ |
| 1201 | try: |
| 1202 | definitions = object_records.read_collection_records( |
| 1203 | MATERIALIZE_DEFINITIONS_COLLECTION, base_dir=base_dir, roots=roots |
| 1204 | ) |
| 1205 | except (object_collections.CollectionNotFoundError, object_collections.InvalidCollectionNameError): |
| 1206 | return [] |
| 1207 | |
| 1208 | events: set[str] = set() |
| 1209 | for definition in definitions: |
| 1210 | if is_definition_blocked(definition) or not is_definition_enabled(definition): |
| 1211 | continue |
| 1212 | trigger = _json_field(definition, "trigger", None, definition_id=definition.get("id") or "<unknown>") |
| 1213 | if not isinstance(trigger, dict) or trigger.get("mode") != "event": |
| 1214 | continue |
| 1215 | source_collection = (definition.get("source_collection") or "").strip() |
| 1216 | if source_collection: |
| 1217 | events.add(f"{source_collection}.record.created") |
| 1218 | return sorted(events) |
| 1219 | |
| 1220 | |
| 1221 | # NOTE: there is deliberately no runtime "sync HANDLES by rewriting the |
| 1222 | # installed materialize_seed.py" function here. Rewriting an existing |
| 1223 | # object's source under the daemon poll loop is the wrong shape (expensive, |
| 1224 | # surprising, self-modifying). materialize_seed ships inert (HANDLES == []) |
| 1225 | # and event-mode runs manually (materialize_run). If automatic on-create |
| 1226 | # dispatch is ever worth wiring, compute_event_handles above is the input a |
| 1227 | # separate, scheduled, tested job would consume -- not this module at import |
| 1228 | # or the poll loop at runtime. |
| 1229 | |
| 1230 | |
| 1231 | # --- Small shared helpers ------------------------------------------------- |
| 1232 | |
| 1233 | def _stringify(value: Any) -> str: |
| 1234 | if value is None: |
| 1235 | return "" |
| 1236 | if isinstance(value, bool): |
| 1237 | return "true" if value else "false" |
| 1238 | return str(value) |
| 1239 | |
| 1240 | |
| 1241 | def _to_cents(value: Any) -> int: |
| 1242 | text = str(value if value is not None else "").strip() |
| 1243 | if not text: |
| 1244 | return 0 |
| 1245 | try: |
| 1246 | return int(text) |
| 1247 | except ValueError as exc: |
| 1248 | raise DefinitionError(f"expected an integer cents value, got {value!r}") from exc |
| 1249 | |
| 1250 | |
| 1251 | def _truthy(value: Any, *, default: bool) -> bool: |
| 1252 | if value is None or value == "": |
| 1253 | return default |
| 1254 | return str(value).strip().lower() in {"true", "1", "yes", "on"} |
| 1255 | |
| 1256 | |
| 1257 | def _positive_int(value: Any) -> int | None: |
| 1258 | try: |
| 1259 | parsed = int(str(value).strip()) |
| 1260 | except (TypeError, ValueError): |
| 1261 | return None |
| 1262 | return parsed if parsed > 0 else None |
| 1263 | |
| 1264 | |
| 1265 | def _parse_iso(value: str) -> datetime | None: |
| 1266 | try: |
| 1267 | parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) |
| 1268 | except ValueError: |
| 1269 | return None |
| 1270 | if parsed.tzinfo is None: |
| 1271 | parsed = parsed.replace(tzinfo=timezone.utc) |
| 1272 | return parsed |
| 1273 | |
| 1274 | |
| 1275 | def _json_field(record: Mapping[str, str], name: str, default: Any, *, definition_id: str) -> Any: |
| 1276 | raw = record.get(name) |
| 1277 | if raw is None: |
| 1278 | return default |
| 1279 | if not isinstance(raw, str): |
| 1280 | return raw |
| 1281 | raw = raw.strip() |
| 1282 | if not raw: |
| 1283 | return default |
| 1284 | try: |
| 1285 | return json.loads(raw) |
| 1286 | except json.JSONDecodeError as exc: |
| 1287 | raise DefinitionError(f"{definition_id}: {name} is not valid JSON: {exc}") from exc |