Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

dbbasic-object-server/object_materialize.py
Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
object_materialize.py · 56.5 KB · 1287 lines
1"""Materialize block: declared generation jobs that CREATE new records.
2
3Implements ``plan/vocabulary/61-materialize-spec.md``. Sibling to
4``object_rollups.py`` (14-rollup-spec.md): both are declarative,
5schema/config-parameterized, ride ``object_daemon.py``'s poll loop as one
6more pass, and both write to a collection nothing else writes to. The
7difference that makes this a distinct block rather than a rollup variant:
8a rollup recompute REPLACES the whole answer every time (naturally
9idempotent, disposable output); a materialize pass CREATES new records
10other records come to depend on (a journal, a depreciation posting, a
11seeded task), so a double-run is not "stale," it is a duplicate --
12genuinely destructive. Everything below exists to make "generate this
13period's output" idempotent by construction rather than by discipline.
14
15**The three worked shapes, and how one mechanism spans them (61's Purpose
16and Coverage):**
17
181. Recurring journals (``fin_recurring`` -> ``fin_journals`` +
19 ``fin_journal_lines``, ``trigger.mode: "scheduled"``): a header row per
20 due period, its child lines parsed VERBATIM from the source row's own
21 ``child_source_field`` JSON blob (already shaped like the child
22 schema's rows) -- see ``_build_child_rows_from_template``.
232. Depreciation (``products`` -> ``fin_journals`` + ``fin_journal_lines``,
24 ``trigger.mode: "scheduled_fixed"``): 61's own Open Questions flags
25 this as genuinely underspecified ("this spec assumes a
26 debit_account_id/credit_account_id pair added to the definition
27 record... flagged as an implementation-time field, not fully specified
28 here"). This module's resolution, decided at implementation time: when
29 a definition declares ``child_collection`` but NO
30 ``child_source_field``, that absence IS the signal for the
31 "synthesized two-line depreciation shape" -- the definition MUST then
32 carry ``debit_account_id``/``credit_account_id`` plus a reserved
33 ``mapping.amount`` entry whose op is ``depreciation_amount``; exactly
34 two child lines are synthesized (debit the expense account, credit
35 accumulated depreciation, same amount both lines) rather than parsed
36 from source-row JSON. See ``_build_depreciation_child_rows``. Only
37 ``straight_line`` ships; a definition requesting ``declining`` is
38 rejected at parse time (``_validate_mapping_entry``), never silently
39 mis-computed -- 61's Purpose and Coverage names declining-balance as
40 the one worked case that brushes the escalate-to-custom-object line.
413. Template execution / CreateWork (``tasks`` self-referential,
42 ``trigger.mode: "event"``, ``output_collection == source_collection``):
43 detected structurally (output equals source) rather than by a separate
44 flag. This is an UPDATE to the SAME row, gated by the fill-only-if-empty
45 rule (61's Events section: "must never clobber a field the record's
46 own creator already set explicitly"). 61's own worked example 3
47 describes the CreateWork mechanism as a template-relation hop (fetch
48 the ``templates`` row a ``template_id``-shaped field points at, parse
49 its ``default_values`` JSON, fill empty fields) -- notably NOT
50 expressed through this block's ordinary ``mapping`` vocabulary, since
51 ``mapping`` is explicitly forbidden from cross-collection joins
52 (Out of Scope). This module supports BOTH: an ordinary
53 ``mapping``-driven fill (works for any CreateWork-shaped definition,
54 generic, no relation lookup) applied first, then an opportunistic
55 template-default-values hop (``_apply_template_defaults``, worked
56 example 3's exact mechanism) applied second -- both fill-only-if-empty,
57 so whichever gets there first for a given field wins and the other is
58 a no-op for that field. This is a deliberate interpretation of an
59 underspecified corner, documented here rather than by omission.
60
61**Crash-safety -- the honest, non-transactional sequencing (61's Storage
62section, read literally, implemented literally):**
63
64 1. Compute the ENTIRE output in memory first, including the balance
65 check (exact integer cents, never float), BEFORE any write touches
66 disk. Unequal debit/credit sums -> abort, zero rows written, zero
67 risk of a half-posted journal (``BalanceCheckFailed``, raised
68 before ``_generate_new_record`` writes anything).
69 2. Write child rows FIRST. Each child's own id is ALSO deterministic
70 (``{header_id}_line_{n}``), so a duplicate-id create from a
71 resumed partial run is a harmless, caught, skipped no-op
72 (``object_records.DuplicateRecordIdError``), never fatal.
73 3. Write the header LAST, at its deterministic id
74 (``idempotency_key``). The header's EXISTENCE, not the children's,
75 is what "this period is generated" means -- the daemon's due-check
76 (``_record_exists``) queries ``output_collection`` (the header
77 collection) for that id, never the child collection. A crash
78 between the last child write and the header write leaves inert,
79 orphaned, headerless child rows -- an accepted, bounded, ACTUAL
80 failure mode (not swept under the rug): the next pass sees no
81 header, re-attempts, re-writes any still-missing children
82 (idempotent by id), and writes the header this time. This module
83 does NOT claim cross-collection atomicity, because there isn't
84 any -- see 61's Storage section for the full honest accounting of
85 why this ordering is what "detectably safe rather than silently
86 wrong" actually buys, and what it does not.
87
88**Scheduled due-set, recomputed from scratch every pass, same stance
8914-rollup-spec.md takes for its own scheduled recompute:** the daemon
90pass does NOT trust a source row's own ``next_run``/``last_run`` (or this
91definition's own ``last_run_at``) as the mechanism of truth. Every pass
92recomputes which periods SHOULD exist between a row's anchor point and
93now, and checks the deterministic header id's existence in
94``output_collection`` for each -- missing means generate, present means
95skip, never an error. ``last_run_at``/a source row's own
96``next_run``/``last_run`` are stamped afterward purely as a freshness
97DISPLAY, never read back as the correctness gate.
98
99**generated_from -- built, not left open.** 61's Open Questions flags this
100as "arguably should just be built." When ``output_collection``'s schema
101declares a plain ``generated_from`` field, every top-level generated
102record (and CreateWork's in-place fill, fill-only-if-empty like every
103other field) is stamped with a small JSON blob (``definition_id``,
104``source_id``, ``period_start``) -- additive, opt-in per output schema
105(a schema that hasn't added the field simply never receives it, no
106error), first-class provenance rather than only inferable from the
107deterministic id's string shape.
108"""
109from __future__ import annotations
110
111import calendar
112import json
113import re
114from dataclasses import dataclass
115from datetime import date, datetime, timedelta, timezone
116from typing import Any, Mapping
117
118import object_collections
119import object_feature_flags
120import object_records
121import object_schemas
122
123MATERIALIZE_DEFINITIONS_COLLECTION = "materialize_definitions"
124FEATURE_FLAGS_COLLECTION = "feature_flags"
125MATERIALIZE_ENABLED_FLAG = "materialize_enabled"
126DEFAULT_ACTOR = "daemon:materialize"
127
128_GRANULARITIES = frozenset({"daily", "weekly", "monthly", "quarterly", "yearly"})
129_TRIGGER_MODES = frozenset({"scheduled", "scheduled_fixed", "event"})
130_MAPPING_OP_KEYS = frozenset({"from", "from_period", "literal", "template", "if", "depreciation_amount"})
131_TEMPLATE_TOKEN_RE = re.compile(r"\{([a-zA-Z0-9_.]+)\}")
132
133# Safety valve against a malformed/huge span producing an unbounded period
134# loop (e.g. an anchor_field decades in the past with a daily granularity).
135# Not part of the spec; a defensive bound only.
136_MAX_PERIODS_PER_ROW = 100_000
137
138
139class DefinitionError(ValueError):
140 """Raised when a materialize_definitions record cannot be generated as-is.
141
142 Covers malformed JSON sub-fields, violations of 61's own doctrine (a
143 filter value that isn't a flat scalar, an unrecognized mapping op,
144 ``declining`` depreciation requested), and per-row generation problems
145 (unparseable child_source_field JSON, a missing anchor field). The
146 daemon pass (object_daemon.process_materializations) and
147 materialize_run both catch this per-definition or per-row, matching
148 object_rollups.DefinitionError's isolation posture.
149 """
150
151
152class MissingCollectionError(DefinitionError):
153 """A definition names a source/output/child collection that doesn't exist.
154
155 Distinguished from a generic DefinitionError so the daemon can log it
156 exactly once per process (61's Degradation: "warned-once-per-process,
157 matching process_stale_transitions' _WARNED_UNKNOWN_AUTO_TRANSITION_
158 COLLECTIONS pattern exactly") rather than re-logging every poll.
159 """
160
161 def __init__(self, collection: str, role: str):
162 super().__init__(f"{role} collection '{collection}' does not exist")
163 self.collection = collection
164 self.role = role
165
166
167class BalanceCheckFailed(DefinitionError):
168 """A generation's computed child rows don't balance -- zero rows written.
169
170 Raised BEFORE any write touches disk (see module docstring's
171 crash-safety section); the caller's per-row isolation logs this and
172 retries next pass, since nothing was written and the deterministic id
173 therefore never came to exist.
174 """
175
176
177@dataclass(frozen=True)
178class Period:
179 """One computed generation period (scheduled/scheduled_fixed only).
180
181 ``index``/``total`` are the 1-based period number and the definition's
182 total period cap -- present only for ``scheduled_fixed`` (depreciation's
183 "which month is this, out of how many"), None for ``scheduled``.
184 """
185
186 start: date
187 end: date
188 label: str
189 index: int | None
190 total: int | None
191
192
193@dataclass(frozen=True)
194class MaterializeConfig:
195 definition_id: str
196 name: str
197 source_collection: str
198 source_filter: dict[str, Any]
199 trigger_mode: str # "scheduled" | "scheduled_fixed" | "event"
200 trigger_interval_seconds: int | None
201 # scheduled mode
202 anchor_field: str | None
203 frequency_field: str | None # names a FIELD on the source row (e.g. "frequency")
204 # scheduled_fixed mode
205 start_field: str | None
206 granularity: str | None # a literal fixed granularity (e.g. "monthly")
207 periods_field: str | None # names a FIELD on the source row (e.g. "useful_life_months")
208 # output
209 output_collection: str
210 child_collection: str | None
211 child_source_field: str | None
212 child_link_field: str | None
213 idempotency_key: str
214 mapping: dict[str, Any]
215 balance_check: dict[str, str] | None
216 debit_account_id: str | None
217 credit_account_id: str | None
218 synthesized_amount_entry: dict[str, Any] | None
219 actor: str
220 enabled: bool
221 block: bool
222 stamp_generated_from: bool
223
224
225# --- Feature flag / due gates -------------------------------------------------
226
227def materialize_pass_enabled(*, base_dir: Any) -> bool:
228 """Block-wide kill switch, ``<block>_enabled`` convention, mirroring
229 object_rollups.rollup_pass_enabled exactly (same defaults, same
230 reasoning): default ON, a missing/unreadable feature_flags collection
231 or a blank value both resolve to "on"; only an explicit off/false/0/no
232 value turns the pass off. Unlike 14's posture (stale-but-valid reads
233 keep serving), 61's Degradation is explicit that here "off means off,
234 full stop, for BOTH the scheduled and manual paths" -- callers (the
235 daemon pass, materialize_run, materialize_seed's EVENT handler) are
236 each responsible for checking this before doing anything, since there
237 is no "serve what's on disk" fallback for a write-side block.
238 """
239 return object_feature_flags.kill_switch_on(MATERIALIZE_ENABLED_FLAG, base_dir=base_dir)
240
241
242def is_definition_enabled(record: Mapping[str, str]) -> bool:
243 """Per-definition pause -- independent of the block-wide flag. Blank
244 defaults to enabled, matching object_rollups.is_definition_enabled.
245 """
246 return _truthy(record.get("enabled"), default=True)
247
248
249def is_definition_blocked(record: Mapping[str, str]) -> bool:
250 """The harder stop (61's Degradation): ``block: true`` refuses even
251 ``materialize_run``'s manual path with a distinct error, unlike
252 ``enabled: false`` (pause, resume later without thinking).
253 """
254 return _truthy(record.get("block"), default=False)
255
256
257def is_definition_due(record: Mapping[str, str], *, now: datetime | None = None) -> bool:
258 """SCHEDULED/SCHEDULED_FIXED due gate: ``last_run_at`` +
259 ``trigger.interval_seconds``, both read straight off the raw record --
260 mirrors object_rollups.is_definition_due's structure exactly. A
261 never-run definition (``last_run_at`` blank) is always due. This gate
262 only decides whether the daemon PASS attempts a definition this poll;
263 it is not the correctness mechanism (that's the deterministic header
264 id, see module docstring) -- a due-but-already-generated period is
265 simply reported as skipped, not an error.
266 """
267 now = now or datetime.now(timezone.utc)
268 last_run_raw = (record.get("last_run_at") or "").strip()
269 if not last_run_raw:
270 return True
271 trigger = _json_field(record, "trigger", None, definition_id=record.get("id") or "<unknown>")
272 interval = _positive_int(trigger.get("interval_seconds")) if isinstance(trigger, dict) else None
273 if interval is None:
274 return True
275 last_run = _parse_iso(last_run_raw)
276 if last_run is None:
277 return True
278 return (now - last_run).total_seconds() >= interval
279
280
281# --- Definition parsing -------------------------------------------------
282
283def parse_definition(
284 record: Mapping[str, str], *, base_dir: Any, roots: Any = None
285) -> MaterializeConfig:
286 """Validate and normalize one ``materialize_definitions`` row.
287
288 Raises ``DefinitionError``/``MissingCollectionError`` for anything
289 malformed or referencing a collection that doesn't exist yet -- the
290 daemon (object_daemon.process_materializations) and materialize_run
291 both isolate this per-definition, never letting one bad definition
292 stop any other (61's Events/Degradation).
293 """
294 definition_id = (record.get("id") or "").strip()
295 if not definition_id:
296 raise DefinitionError("materialize definition is missing an id")
297
298 name = (record.get("name") or "").strip() or definition_id
299
300 source_collection = (record.get("source_collection") or "").strip()
301 if not source_collection or not object_collections.validate_collection_name(source_collection):
302 raise DefinitionError(f"{definition_id}: source_collection is required and must be a valid collection name")
303 try:
304 object_schemas.get_schema(source_collection, base_dir=base_dir, roots=roots)
305 except object_schemas.SchemaNotFoundError:
306 raise MissingCollectionError(source_collection, "source") from None
307
308 source_filter = _parse_filter(record, definition_id=definition_id)
309
310 trigger = _json_field(record, "trigger", None, definition_id=definition_id)
311 if not isinstance(trigger, dict) or not trigger:
312 raise DefinitionError(f"{definition_id}: trigger is required and must be a JSON object")
313 trigger_mode = str(trigger.get("mode") or "").strip()
314 if trigger_mode not in _TRIGGER_MODES:
315 raise DefinitionError(f"{definition_id}: trigger.mode must be one of {sorted(_TRIGGER_MODES)}")
316
317 trigger_interval_seconds: int | None = None
318 anchor_field = frequency_field = start_field = granularity = periods_field = None
319
320 if trigger_mode == "scheduled":
321 trigger_interval_seconds = _positive_int(trigger.get("interval_seconds"))
322 if trigger_interval_seconds is None:
323 raise DefinitionError(f"{definition_id}: trigger.interval_seconds is required and must be a positive integer")
324 frequency_field = str(trigger.get("period_field") or "").strip()
325 if not frequency_field:
326 raise DefinitionError(
327 f"{definition_id}: trigger.period_field (the source row field naming its own "
328 "frequency, e.g. \"frequency\") is required for scheduled mode"
329 )
330 anchor_field = str(trigger.get("anchor_field") or "").strip()
331 if not anchor_field:
332 raise DefinitionError(f"{definition_id}: trigger.anchor_field is required for scheduled mode")
333 elif trigger_mode == "scheduled_fixed":
334 trigger_interval_seconds = _positive_int(trigger.get("interval_seconds"))
335 if trigger_interval_seconds is None:
336 raise DefinitionError(f"{definition_id}: trigger.interval_seconds is required and must be a positive integer")
337 granularity = str(trigger.get("period_field") or "").strip().lower()
338 if granularity not in _GRANULARITIES:
339 raise DefinitionError(
340 f"{definition_id}: trigger.period_field must be a fixed granularity, one of "
341 f"{sorted(_GRANULARITIES)}, for scheduled_fixed mode"
342 )
343 start_field = str(trigger.get("start_field") or "").strip()
344 if not start_field:
345 raise DefinitionError(f"{definition_id}: trigger.start_field is required for scheduled_fixed mode")
346 end_condition = trigger.get("end_condition")
347 if not isinstance(end_condition, dict):
348 raise DefinitionError(f"{definition_id}: trigger.end_condition is required for scheduled_fixed mode")
349 periods_field = str(end_condition.get("periods_field") or "").strip()
350 if not periods_field:
351 raise DefinitionError(f"{definition_id}: trigger.end_condition.periods_field is required for scheduled_fixed mode")
352 else: # event
353 on = str(trigger.get("on") or "").strip()
354 if on != "record.created":
355 raise DefinitionError(f"{definition_id}: trigger.on must be 'record.created' for event mode")
356
357 output_collection = (record.get("output_collection") or "").strip()
358 if not output_collection or not object_collections.validate_collection_name(output_collection):
359 raise DefinitionError(f"{definition_id}: output_collection is required and must be a valid collection name")
360 try:
361 object_schemas.get_schema(output_collection, base_dir=base_dir, roots=roots)
362 except object_schemas.SchemaNotFoundError:
363 raise MissingCollectionError(output_collection, "output") from None
364
365 child_collection = (record.get("child_collection") or "").strip() or None
366 child_source_field = (record.get("child_source_field") or "").strip() or None
367 child_link_field = (record.get("child_link_field") or "").strip() or None
368 if child_collection:
369 if not object_collections.validate_collection_name(child_collection):
370 raise DefinitionError(f"{definition_id}: child_collection must be a valid collection name")
371 try:
372 object_schemas.get_schema(child_collection, base_dir=base_dir, roots=roots)
373 except object_schemas.SchemaNotFoundError:
374 raise MissingCollectionError(child_collection, "child") from None
375 if not child_link_field:
376 raise DefinitionError(f"{definition_id}: child_link_field is required when child_collection is set")
377
378 idempotency_key = (record.get("idempotency_key") or "").strip()
379 if not idempotency_key:
380 raise DefinitionError(f"{definition_id}: idempotency_key is required")
381
382 mapping_raw = _json_field(record, "mapping", {}, definition_id=definition_id)
383 if mapping_raw is None:
384 mapping_raw = {}
385 if not isinstance(mapping_raw, dict):
386 raise DefinitionError(f"{definition_id}: mapping must be a JSON object")
387 mapping = dict(mapping_raw)
388 for field_name, entry in mapping.items():
389 _validate_mapping_entry(field_name, entry, definition_id=definition_id)
390
391 balance_check_raw = _json_field(record, "balance_check", None, definition_id=definition_id)
392 balance_check: dict[str, str] | None = None
393 if balance_check_raw is not None:
394 if not isinstance(balance_check_raw, dict):
395 raise DefinitionError(f"{definition_id}: balance_check must be a JSON object")
396 debit_field = str(balance_check_raw.get("debit_field") or "").strip()
397 credit_field = str(balance_check_raw.get("credit_field") or "").strip()
398 if not debit_field or not credit_field:
399 raise DefinitionError(f"{definition_id}: balance_check requires debit_field and credit_field")
400 if not child_collection:
401 raise DefinitionError(f"{definition_id}: balance_check requires child_collection")
402 balance_check = {"debit_field": debit_field, "credit_field": credit_field}
403
404 debit_account_id = (record.get("debit_account_id") or "").strip() or None
405 credit_account_id = (record.get("credit_account_id") or "").strip() or None
406
407 synthesized_amount_entry: dict[str, Any] | None = None
408 if child_collection and not child_source_field:
409 # Absence of child_source_field alongside a present child_collection
410 # is the synthesized depreciation-lines shape's own signal -- see
411 # module docstring's worked example 2 discussion.
412 amount_entry = mapping.pop("amount", None)
413 if not isinstance(amount_entry, dict) or "depreciation_amount" not in amount_entry:
414 raise DefinitionError(
415 f"{definition_id}: a child_collection with no child_source_field is the "
416 "synthesized depreciation-lines shape and requires "
417 "mapping.amount = {'depreciation_amount': {...}}"
418 )
419 if not debit_account_id or not credit_account_id:
420 raise DefinitionError(
421 f"{definition_id}: the synthesized depreciation-lines shape requires "
422 "debit_account_id and credit_account_id"
423 )
424 if not balance_check:
425 raise DefinitionError(
426 f"{definition_id}: the synthesized depreciation-lines shape requires "
427 "balance_check (its debit_field/credit_field name the two generated "
428 "lines' amount fields)"
429 )
430 synthesized_amount_entry = amount_entry
431
432 actor = (record.get("actor") or "").strip() or DEFAULT_ACTOR
433 enabled = _truthy(record.get("enabled"), default=True)
434 block = _truthy(record.get("block"), default=False)
435
436 stamp_generated_from = _schema_has_field(output_collection, "generated_from", base_dir=base_dir, roots=roots)
437
438 return MaterializeConfig(
439 definition_id=definition_id,
440 name=name,
441 source_collection=source_collection,
442 source_filter=source_filter,
443 trigger_mode=trigger_mode,
444 trigger_interval_seconds=trigger_interval_seconds,
445 anchor_field=anchor_field,
446 frequency_field=frequency_field,
447 start_field=start_field,
448 granularity=granularity,
449 periods_field=periods_field,
450 output_collection=output_collection,
451 child_collection=child_collection,
452 child_source_field=child_source_field,
453 child_link_field=child_link_field,
454 idempotency_key=idempotency_key,
455 mapping=mapping,
456 balance_check=balance_check,
457 debit_account_id=debit_account_id,
458 credit_account_id=credit_account_id,
459 synthesized_amount_entry=synthesized_amount_entry,
460 actor=actor,
461 enabled=enabled,
462 block=block,
463 stamp_generated_from=stamp_generated_from,
464 )
465
466
467def _validate_mapping_entry(field_name: str, entry: Any, *, definition_id: str) -> None:
468 if not isinstance(entry, dict):
469 raise DefinitionError(f"{definition_id}: mapping.{field_name} must be a JSON object")
470
471 if "if" in entry:
472 cond = entry.get("if")
473 if not isinstance(cond, dict) or not isinstance(cond.get("source_field"), str) or not cond.get("source_field"):
474 raise DefinitionError(f"{definition_id}: mapping.{field_name}.if requires a source_field")
475 if "equals" not in cond:
476 raise DefinitionError(f"{definition_id}: mapping.{field_name}.if requires 'equals'")
477 if "then" not in entry or "else" not in entry:
478 raise DefinitionError(f"{definition_id}: mapping.{field_name} with 'if' requires 'then' and 'else'")
479 return
480
481 if "depreciation_amount" in entry:
482 op = entry["depreciation_amount"]
483 if not isinstance(op, dict):
484 raise DefinitionError(f"{definition_id}: mapping.{field_name}.depreciation_amount must be a JSON object")
485 method = str(op.get("method") or "").strip()
486 if method == "declining":
487 raise DefinitionError(
488 f"{definition_id}: mapping.{field_name} requests depreciation method 'declining', "
489 "which is out of scope for v1 (straight_line only) -- see "
490 "plan/vocabulary/61-materialize-spec.md's Open Questions"
491 )
492 if method != "straight_line":
493 raise DefinitionError(f"{definition_id}: mapping.{field_name}.depreciation_amount.method must be 'straight_line'")
494 for key in ("cost_field", "salvage_field", "life_field"):
495 value = op.get(key)
496 if not isinstance(value, str) or not value:
497 raise DefinitionError(f"{definition_id}: mapping.{field_name}.depreciation_amount.{key} is required")
498 return
499
500 if "from" in entry:
501 if not isinstance(entry["from"], str) or not entry["from"]:
502 raise DefinitionError(f"{definition_id}: mapping.{field_name}.from must name a source field")
503 return
504
505 if "from_period" in entry:
506 if entry["from_period"] not in ("period_start", "period_end", "period_label"):
507 raise DefinitionError(
508 f"{definition_id}: mapping.{field_name}.from_period must be one of "
509 "period_start/period_end/period_label"
510 )
511 return
512
513 if "literal" in entry:
514 return
515
516 if "template" in entry:
517 if not isinstance(entry["template"], str):
518 raise DefinitionError(f"{definition_id}: mapping.{field_name}.template must be a string")
519 return
520
521 raise DefinitionError(
522 f"{definition_id}: mapping.{field_name} must use one of from/from_period/literal/"
523 "template/if/depreciation_amount"
524 )
525
526
527def _parse_filter(record: Mapping[str, str], *, definition_id: str) -> dict[str, Any]:
528 raw = _json_field(record, "source_filter", {}, definition_id=definition_id)
529 if raw is None:
530 return {}
531 if not isinstance(raw, dict):
532 raise DefinitionError(f"{definition_id}: source_filter must be a flat JSON object")
533 normalized: dict[str, Any] = {}
534 for key, value in raw.items():
535 if isinstance(value, (dict, list)):
536 raise DefinitionError(
537 f"{definition_id}: source_filter.{key} must be a scalar -- materialize filters are "
538 "flat equality only, ANDed, same as 14-rollup-spec.md's filter language"
539 )
540 if value is None:
541 raise DefinitionError(f"{definition_id}: source_filter.{key} must not be null")
542 if isinstance(value, bool):
543 value = "true" if value else "false"
544 normalized[key] = value
545 return normalized
546
547
548def _schema_has_field(collection: str, field_name: str, *, base_dir: Any, roots: Any = None) -> bool:
549 try:
550 schema = object_schemas.get_schema(collection, base_dir=base_dir, roots=roots)
551 except object_schemas.SchemaNotFoundError:
552 return False
553 return any(field.get("name") == field_name for field in schema.get("fields", []))
554
555
556# --- Period computation -------------------------------------------------
557
558def _compute_scheduled_periods(
559 source_row: Mapping[str, str], config: MaterializeConfig, *, now: date
560) -> list[Period]:
561 granularity = str(source_row.get(config.frequency_field) or "").strip().lower()
562 if granularity not in _GRANULARITIES:
563 raise DefinitionError(
564 f"source row has an invalid or missing '{config.frequency_field}' "
565 f"(must be one of {sorted(_GRANULARITIES)}); got {granularity!r}"
566 )
567 anchor_raw = source_row.get(config.anchor_field) or ""
568 origin = _parse_date(anchor_raw)
569 if origin is None:
570 origin = _parse_date(source_row.get("created_at") or "")
571 if origin is None:
572 raise DefinitionError(
573 f"source row has no usable '{config.anchor_field}' (nor created_at) to anchor periods"
574 )
575 return _step_periods(origin, granularity, now=now, cap=None)
576
577
578def _compute_scheduled_fixed_periods(
579 source_row: Mapping[str, str], config: MaterializeConfig, *, now: date
580) -> list[Period]:
581 origin = _parse_date(source_row.get(config.start_field) or "")
582 if origin is None:
583 raise DefinitionError(f"source row has no usable '{config.start_field}' to anchor periods")
584 cap = _positive_int(source_row.get(config.periods_field))
585 if cap is None:
586 raise DefinitionError(f"source row's '{config.periods_field}' must be a positive integer")
587 return _step_periods(origin, config.granularity, now=now, cap=cap)
588
589
590def _step_periods(origin: date, granularity: str, *, now: date, cap: int | None) -> list[Period]:
591 periods: list[Period] = []
592 cursor = origin
593 index = 0
594 guard = 0
595 while cursor <= now:
596 index += 1
597 if cap is not None and index > cap:
598 break
599 period_end = _advance_date(cursor, granularity, 1)
600 periods.append(
601 Period(
602 start=cursor,
603 end=period_end,
604 label=_period_label(cursor, granularity),
605 index=index if cap is not None else None,
606 total=cap,
607 )
608 )
609 cursor = period_end
610 guard += 1
611 if guard >= _MAX_PERIODS_PER_ROW:
612 break
613 return periods
614
615
616def _due_periods_for_row(
617 config: MaterializeConfig, source_row: Mapping[str, str], *, now: date
618) -> list[Period | None]:
619 if config.trigger_mode == "event":
620 return [None]
621 if config.trigger_mode == "scheduled":
622 return _compute_scheduled_periods(source_row, config, now=now)
623 if config.trigger_mode == "scheduled_fixed":
624 return _compute_scheduled_fixed_periods(source_row, config, now=now)
625 raise DefinitionError(f"unknown trigger mode: {config.trigger_mode}") # pragma: no cover -- pre-validated
626
627
628def _advance_date(d: date, granularity: str, steps: int = 1) -> date:
629 if granularity == "daily":
630 return d + timedelta(days=steps)
631 if granularity == "weekly":
632 return d + timedelta(weeks=steps)
633 if granularity == "monthly":
634 return _add_months(d, steps)
635 if granularity == "quarterly":
636 return _add_months(d, steps * 3)
637 if granularity == "yearly":
638 return _add_months(d, steps * 12)
639 raise DefinitionError(f"unknown granularity: {granularity}") # pragma: no cover -- pre-validated
640
641
642def _add_months(d: date, months: int) -> date:
643 total = d.month - 1 + months
644 year = d.year + total // 12
645 month = total % 12 + 1
646 day = min(d.day, calendar.monthrange(year, month)[1])
647 return date(year, month, day)
648
649
650def _period_label(period_start: date, granularity: str) -> str:
651 if granularity == "daily":
652 return period_start.isoformat()
653 if granularity == "weekly":
654 iso_year, iso_week, _ = period_start.isocalendar()
655 return f"{iso_year}-W{iso_week:02d}"
656 if granularity == "monthly":
657 return f"{period_start.year:04d}-{period_start.month:02d}"
658 if granularity == "quarterly":
659 quarter = (period_start.month - 1) // 3 + 1
660 return f"{period_start.year}-Q{quarter}"
661 if granularity == "yearly":
662 return str(period_start.year)
663 raise DefinitionError(f"unknown granularity: {granularity}") # pragma: no cover -- pre-validated
664
665
666def _parse_date(value: Any) -> date | None:
667 text = (value or "").strip() if isinstance(value, str) else ""
668 if not text:
669 return None
670 try:
671 if "T" in text or " " in text:
672 return datetime.fromisoformat(text.replace("Z", "+00:00")).date()
673 return date.fromisoformat(text)
674 except ValueError:
675 return None
676
677
678# --- Mapping vocabulary -------------------------------------------------
679
680def _eval_entry(
681 field_name: str, entry: Mapping[str, Any], *, source_row: Mapping[str, str],
682 period: Period | None, definition_id: str,
683) -> str:
684 if "from" in entry:
685 return _stringify(source_row.get(entry["from"], ""))
686
687 if "from_period" in entry:
688 if period is None:
689 raise DefinitionError(
690 f"{definition_id}: mapping.{field_name} uses from_period but this generation "
691 "has no period (event/manual single-shot/CreateWork)"
692 )
693 key = entry["from_period"]
694 if key == "period_start":
695 return period.start.isoformat()
696 if key == "period_end":
697 return period.end.isoformat()
698 if key == "period_label":
699 return period.label
700 raise DefinitionError( # pragma: no cover -- pre-validated
701 f"{definition_id}: mapping.{field_name}.from_period must be period_start/period_end/period_label"
702 )
703
704 if "literal" in entry:
705 return _stringify(entry["literal"])
706
707 if "template" in entry:
708 return _render_template(
709 entry["template"], source_row=source_row, period=period,
710 definition_id=definition_id, field_name=field_name,
711 )
712
713 if "if" in entry:
714 cond = entry["if"]
715 actual = source_row.get(cond["source_field"])
716 matches = _loose_equals(actual, cond["equals"])
717 return _stringify(entry["then"] if matches else entry["else"])
718
719 if "depreciation_amount" in entry:
720 return _eval_depreciation(
721 entry["depreciation_amount"], source_row, period,
722 definition_id=definition_id, field_name=field_name,
723 )
724
725 raise DefinitionError(f"{definition_id}: mapping.{field_name} uses an unrecognized op") # pragma: no cover
726
727
728def _loose_equals(actual: Any, expected: Any) -> bool:
729 if isinstance(expected, bool):
730 expected_str = "true" if expected else "false"
731 else:
732 expected_str = str(expected)
733 return str(actual if actual is not None else "") == expected_str
734
735
736def _render_template(
737 template: str, *, source_row: Mapping[str, str], period: Period | None,
738 definition_id: str, field_name: str,
739) -> str:
740 def replace(match: re.Match[str]) -> str:
741 token = match.group(1)
742 if token == "period_label":
743 if period is None:
744 raise DefinitionError(
745 f"{definition_id}: mapping.{field_name} template uses {{period_label}} but "
746 "this generation has no period"
747 )
748 return period.label
749 if token.startswith("source."):
750 field = token[len("source."):]
751 return _stringify(source_row.get(field, ""))
752 raise DefinitionError(
753 f"{definition_id}: mapping.{field_name} template references unknown placeholder '{{{token}}}'"
754 )
755
756 return _TEMPLATE_TOKEN_RE.sub(replace, template)
757
758
759def _eval_depreciation(
760 op: Mapping[str, Any], source_row: Mapping[str, str], period: Period | None,
761 *, definition_id: str, field_name: str,
762) -> str:
763 if period is None or period.total is None or period.index is None:
764 raise DefinitionError(
765 f"{definition_id}: mapping.{field_name} uses depreciation_amount, which requires a "
766 "scheduled_fixed trigger with an end_condition.periods_field cap"
767 )
768 monthly, remainder = _straight_line_split(op, source_row, definition_id=definition_id, field_name=field_name)
769 if period.index == period.total:
770 return str(monthly + remainder)
771 return str(monthly)
772
773
774def _straight_line_split(
775 op: Mapping[str, Any], source_row: Mapping[str, str], *, definition_id: str, field_name: str,
776) -> tuple[int, int]:
777 cost = _to_cents(source_row.get(op["cost_field"]))
778 salvage = _to_cents(source_row.get(op["salvage_field"]))
779 life = _positive_int(source_row.get(op["life_field"]))
780 if life is None:
781 raise DefinitionError(
782 f"{definition_id}: mapping.{field_name}.depreciation_amount.life_field must be a "
783 "positive integer on the source row"
784 )
785 depreciable = cost - salvage
786 monthly = depreciable // life
787 remainder = depreciable - (monthly * life)
788 return monthly, remainder
789
790
791# --- Generation orchestration -------------------------------------------------
792
793def generate_definition(
794 record: Mapping[str, str], *, base_dir: Any, roots: Any = None, now: datetime | None = None,
795) -> dict[str, Any]:
796 """Parse ``record`` and run its full due-set generation.
797
798 The SAME function the daemon's scheduled pass and ``materialize_run``'s
799 manual path both call (61's Events: "the exact same idempotent,
800 per-source-row function") -- for scheduled/scheduled_fixed this
801 iterates every matching source row and every due period; for event
802 mode (when called manually, since the daemon's scheduled pass never
803 calls this on an event-mode definition -- see object_daemon.
804 process_materializations) this iterates every matching source row
805 once each, no period. Returns {"definition_id", "checked", "generated",
806 "skipped_already_generated", "errors": [{"source_id", "period_start",
807 "error"}, ...]}. Raises DefinitionError/MissingCollectionError for a
808 malformed definition -- the caller isolates that per-definition; every
809 per-row/per-period failure INSIDE a valid definition is instead
810 captured in the returned "errors" list, never raised, so one bad row
811 never stops any other (61's two-level isolation, mirroring
812 process_stale_transitions).
813 """
814 config = parse_definition(record, base_dir=base_dir, roots=roots)
815 return generate_config(config, base_dir=base_dir, roots=roots, now=now)
816
817
818def generate_config(
819 config: MaterializeConfig, *, base_dir: Any, roots: Any = None, now: datetime | None = None,
820) -> dict[str, Any]:
821 now_dt = now or datetime.now(timezone.utc)
822 now_date = now_dt.date()
823
824 source_rows = object_records.read_collection_records(config.source_collection, base_dir=base_dir, roots=roots)
825 if config.source_filter:
826 source_rows = object_records.filter_records(source_rows, config.source_filter)
827
828 checked = generated = skipped = 0
829 errors: list[dict[str, Any]] = []
830
831 for row in source_rows:
832 source_id = row.get("id") or "<unknown>"
833 try:
834 periods = _due_periods_for_row(config, row, now=now_date)
835 except Exception as exc:
836 errors.append({"source_id": source_id, "period_start": None, "error": str(exc)})
837 continue
838
839 for period in periods:
840 checked += 1
841 try:
842 result = generate_one(config, row, period, base_dir=base_dir, roots=roots)
843 except Exception as exc:
844 errors.append({
845 "source_id": source_id,
846 "period_start": period.start.isoformat() if period is not None else None,
847 "error": str(exc),
848 })
849 continue
850
851 if result["status"] == "generated":
852 generated += 1
853 elif result["status"] == "skipped_already_generated":
854 skipped += 1
855 else: # pragma: no cover -- defensive, generate_one only returns these two
856 errors.append({
857 "source_id": source_id,
858 "period_start": period.start.isoformat() if period is not None else None,
859 "error": f"unexpected status: {result['status']}",
860 })
861
862 return {
863 "definition_id": config.definition_id,
864 "checked": checked,
865 "generated": generated,
866 "skipped_already_generated": skipped,
867 "errors": errors,
868 }
869
870
871def generate_one(
872 config: MaterializeConfig, source_row: Mapping[str, str], period: Period | None,
873 *, base_dir: Any, roots: Any = None,
874) -> dict[str, Any]:
875 """Generate (or confirm-already-generated) ONE (source row, period) pair.
876
877 Branches structurally on ``output_collection == source_collection``
878 (CreateWork: an in-place update, no new record) vs. every other shape
879 (a new header record, optionally with children) -- see module
880 docstring.
881 """
882 if config.output_collection == config.source_collection:
883 return _generate_creatework(config, source_row, base_dir=base_dir, roots=roots)
884 return _generate_new_record(config, source_row, period, base_dir=base_dir, roots=roots)
885
886
887def _generate_new_record(
888 config: MaterializeConfig, source_row: Mapping[str, str], period: Period | None,
889 *, base_dir: Any, roots: Any = None,
890) -> dict[str, Any]:
891 header_id = _render_idempotency_key(config, source_row=source_row, period=period)
892
893 # The ONLY due-check: does the header already exist? (module docstring)
894 if _record_exists(config.output_collection, header_id, base_dir=base_dir, roots=roots):
895 return {"status": "skipped_already_generated", "header_id": header_id}
896
897 header_fields: dict[str, Any] = {"id": header_id}
898 for field_name, entry in config.mapping.items():
899 header_fields[field_name] = _eval_entry(
900 field_name, entry, source_row=source_row, period=period, definition_id=config.definition_id,
901 )
902 if config.stamp_generated_from:
903 header_fields["generated_from"] = _generated_from_value(config, source_row=source_row, period=period)
904
905 child_rows = _build_child_rows(config, source_row=source_row, header_id=header_id, period=period)
906
907 if config.balance_check:
908 debit_total, credit_total = _sum_balance(config, child_rows)
909 if debit_total != credit_total:
910 raise BalanceCheckFailed(
911 f"{config.definition_id}: unbalanced generation for source "
912 f"{source_row.get('id')} period {period.start.isoformat() if period else '-'}: "
913 f"debits={debit_total} credits={credit_total} cents -- aborted before any write"
914 )
915
916 # Crash-safe ordering: children first (idempotent by id), header last
917 # (the commit signal) -- see module docstring's Storage section.
918 for child in child_rows:
919 try:
920 object_records.create_collection_record(
921 config.child_collection, child, base_dir=base_dir, roots=roots, actor=config.actor,
922 )
923 except object_records.DuplicateRecordIdError:
924 pass # a prior partial run already wrote this line -- harmless
925
926 try:
927 object_records.create_collection_record(
928 config.output_collection, header_fields, base_dir=base_dir, roots=roots, actor=config.actor,
929 )
930 except object_records.DuplicateRecordIdError:
931 # Raced with another pass/process between our existence-check and
932 # now -- the other writer's header is the real one; ours is a
933 # no-op, not a failure.
934 return {"status": "skipped_already_generated", "header_id": header_id}
935
936 return {"status": "generated", "header_id": header_id}
937
938
939def _generate_creatework(
940 config: MaterializeConfig, source_row: Mapping[str, str], *, base_dir: Any, roots: Any = None,
941) -> dict[str, Any]:
942 """CreateWork shape: fill empty fields on the SAME row, never create.
943
944 Two independent fill mechanisms, both fill-only-if-empty (61's Events
945 "correctness rule, not a nicety"), applied in order so a field either
946 mechanism has already filled is left alone by the other:
947 1. this definition's own ``mapping`` (generic, works for any
948 CreateWork-shaped definition).
949 2. an opportunistic hop through a ``templates`` relation field, per
950 61's worked example 3 exactly (``_apply_template_defaults``).
951 The whole ``changes`` dict is built in memory and written in ONE
952 ``update_collection_record`` call -- a malformed template blob aborts
953 the row cleanly with nothing written, same "compute fully, then write
954 once" discipline as the header+children path, at a smaller scale.
955 """
956 source_id = source_row.get("id") or ""
957 if not source_id:
958 return {"status": "skipped_already_generated"}
959
960 changes: dict[str, str] = {}
961 for field_name, entry in config.mapping.items():
962 current = source_row.get(field_name)
963 if current not in (None, ""):
964 continue # fill-only-if-empty
965 value = _eval_entry(field_name, entry, source_row=source_row, period=None, definition_id=config.definition_id)
966 if value in (None, ""):
967 continue
968 changes[field_name] = value
969
970 _apply_template_defaults(config, source_row, changes, base_dir=base_dir, roots=roots)
971
972 if config.stamp_generated_from and "generated_from" not in changes and (
973 source_row.get("generated_from") in (None, "")
974 ):
975 changes["generated_from"] = _generated_from_value(config, source_row=source_row, period=None)
976
977 if not changes:
978 return {"status": "skipped_already_generated", "header_id": source_id}
979
980 object_records.update_collection_record(
981 config.output_collection, source_id, changes, base_dir=base_dir, roots=roots, actor=config.actor,
982 )
983 return {"status": "generated", "header_id": source_id}
984
985
986def _apply_template_defaults(
987 config: MaterializeConfig, source_row: Mapping[str, str], changes: dict[str, str],
988 *, base_dir: Any, roots: Any = None,
989) -> None:
990 """61's worked example 3, literally: find a relation field on the
991 OUTPUT schema pointing at ``templates``, follow it, parse that
992 template's ``default_values`` JSON, fill-only-if-empty. Best-effort in
993 the sense that "no such field," "no id set," "row not found" are all
994 quiet no-ops (this definition simply isn't the templates-relation
995 shape, or the reference is dangling) -- but a malformed
996 ``default_values`` JSON blob is allowed to raise, surfacing as a
997 logged per-row error rather than silently doing nothing, since that
998 is a real data problem worth an operator's attention.
999 """
1000 try:
1001 schema = object_schemas.get_schema(config.output_collection, base_dir=base_dir, roots=roots)
1002 except object_schemas.SchemaNotFoundError:
1003 return
1004
1005 template_field = None
1006 for field in schema.get("fields", []):
1007 relation = field.get("relation") or {}
1008 if relation.get("collection") == "templates":
1009 template_field = field.get("name")
1010 break
1011 if not template_field:
1012 return
1013
1014 template_id = source_row.get(template_field)
1015 if not template_id:
1016 return
1017
1018 try:
1019 template_row = object_records.get_collection_record("templates", template_id, base_dir=base_dir, roots=roots)
1020 except (
1021 object_records.RecordNotFoundError,
1022 object_records.InvalidRecordIdError,
1023 object_collections.CollectionNotFoundError,
1024 object_collections.InvalidCollectionNameError,
1025 ):
1026 return
1027
1028 raw_defaults = (template_row.get("default_values") or "").strip()
1029 if not raw_defaults:
1030 return
1031
1032 defaults = json.loads(raw_defaults) # malformed JSON deliberately propagates -- see docstring
1033 if not isinstance(defaults, dict):
1034 return
1035
1036 for field_name, value in defaults.items():
1037 if field_name in changes:
1038 continue
1039 current = source_row.get(field_name)
1040 if current not in (None, ""):
1041 continue
1042 changes[field_name] = _stringify(value)
1043
1044
1045def _record_exists(collection: str, record_id: str, *, base_dir: Any, roots: Any = None) -> bool:
1046 try:
1047 object_records.get_collection_record(collection, record_id, base_dir=base_dir, roots=roots)
1048 return True
1049 except object_records.RecordNotFoundError:
1050 return False
1051
1052
1053def _build_child_rows(
1054 config: MaterializeConfig, *, source_row: Mapping[str, str], header_id: str, period: Period | None,
1055) -> list[dict[str, str]]:
1056 if not config.child_collection:
1057 return []
1058 if config.child_source_field:
1059 return _build_child_rows_from_template(config, source_row=source_row, header_id=header_id)
1060 if config.synthesized_amount_entry is not None:
1061 return _build_depreciation_child_rows(config, source_row=source_row, header_id=header_id, period=period)
1062 return [] # pragma: no cover -- parse_definition already requires one of the above
1063
1064
1065def _build_child_rows_from_template(
1066 config: MaterializeConfig, *, source_row: Mapping[str, str], header_id: str,
1067) -> list[dict[str, str]]:
1068 raw = (source_row.get(config.child_source_field) or "").strip()
1069 if not raw:
1070 return []
1071 try:
1072 lines = json.loads(raw)
1073 except json.JSONDecodeError as exc:
1074 raise DefinitionError(f"{config.definition_id}: {config.child_source_field} is not valid JSON: {exc}") from exc
1075 if not isinstance(lines, list):
1076 raise DefinitionError(f"{config.definition_id}: {config.child_source_field} must be a JSON list of line objects")
1077
1078 rows: list[dict[str, str]] = []
1079 for index, line in enumerate(lines):
1080 if not isinstance(line, dict):
1081 raise DefinitionError(f"{config.definition_id}: {config.child_source_field}[{index}] must be a JSON object")
1082 child = {str(key): _stringify(value) for key, value in line.items()}
1083 child["id"] = f"{header_id}_line_{index}"
1084 child[config.child_link_field] = header_id
1085 rows.append(child)
1086 return rows
1087
1088
1089def _build_depreciation_child_rows(
1090 config: MaterializeConfig, *, source_row: Mapping[str, str], header_id: str, period: Period | None,
1091) -> list[dict[str, str]]:
1092 amount = _eval_entry(
1093 "amount", config.synthesized_amount_entry, source_row=source_row, period=period,
1094 definition_id=config.definition_id,
1095 )
1096 debit_field = config.balance_check["debit_field"]
1097 credit_field = config.balance_check["credit_field"]
1098 return [
1099 {
1100 "id": f"{header_id}_line_0", config.child_link_field: header_id,
1101 "account_id": config.debit_account_id, debit_field: amount, credit_field: "0",
1102 },
1103 {
1104 "id": f"{header_id}_line_1", config.child_link_field: header_id,
1105 "account_id": config.credit_account_id, debit_field: "0", credit_field: amount,
1106 },
1107 ]
1108
1109
1110def _sum_balance(config: MaterializeConfig, child_rows: list[dict[str, str]]) -> tuple[int, int]:
1111 debit_field = config.balance_check["debit_field"]
1112 credit_field = config.balance_check["credit_field"]
1113 debit_total = sum(_to_cents(row.get(debit_field)) for row in child_rows)
1114 credit_total = sum(_to_cents(row.get(credit_field)) for row in child_rows)
1115 return debit_total, credit_total
1116
1117
1118def _render_idempotency_key(config: MaterializeConfig, *, source_row: Mapping[str, str], period: Period | None) -> str:
1119 available: dict[str, str] = {
1120 "definition_id": config.definition_id,
1121 "source_id": source_row.get("id") or "",
1122 }
1123 if period is not None:
1124 available["period_start"] = period.start.isoformat()
1125 try:
1126 return config.idempotency_key.format(**available)
1127 except KeyError as exc:
1128 raise DefinitionError(
1129 f"{config.definition_id}: idempotency_key references {exc} which is not available "
1130 "for this generation (period_start only applies to scheduled/scheduled_fixed)"
1131 ) from exc
1132
1133
1134def _generated_from_value(config: MaterializeConfig, *, source_row: Mapping[str, str], period: Period | None) -> str:
1135 payload = {
1136 "definition_id": config.definition_id,
1137 "source_id": source_row.get("id") or "",
1138 "period_start": period.start.isoformat() if period is not None else None,
1139 }
1140 return json.dumps(payload, separators=(",", ":"))
1141
1142
1143# --- Event mode (CreateWork dispatch) -------------------------------------------------
1144
1145def event_definitions_for_collection(
1146 collection: str, *, base_dir: Any, roots: Any = None,
1147) -> list[dict[str, str]]:
1148 """Return every enabled, non-blocked, event-mode definition whose
1149 ``source_collection`` is ``collection`` -- the raw records (not
1150 parsed), cheap enough to call on every dispatch (one collection read,
1151 one JSON parse per row); the full ``parse_definition`` validation
1152 happens once ``generate_one_event`` actually attempts a generation.
1153 """
1154 try:
1155 definitions = object_records.read_collection_records(
1156 MATERIALIZE_DEFINITIONS_COLLECTION, base_dir=base_dir, roots=roots
1157 )
1158 except (object_collections.CollectionNotFoundError, object_collections.InvalidCollectionNameError):
1159 return []
1160
1161 matches = []
1162 for definition in definitions:
1163 if is_definition_blocked(definition) or not is_definition_enabled(definition):
1164 continue
1165 if (definition.get("source_collection") or "").strip() != collection:
1166 continue
1167 trigger = _json_field(definition, "trigger", None, definition_id=definition.get("id") or "<unknown>")
1168 if not isinstance(trigger, dict) or trigger.get("mode") != "event":
1169 continue
1170 matches.append(definition)
1171 return matches
1172
1173
1174def generate_one_event(
1175 definition_record: Mapping[str, str], source_row: Mapping[str, str], *, base_dir: Any, roots: Any = None,
1176) -> dict[str, Any]:
1177 """Apply ONE event-mode definition to ONE just-created source row --
1178 what ``materialize_seed``'s ``EVENT`` handler calls per dispatched
1179 record.created event, per definition matching that collection.
1180 """
1181 config = parse_definition(definition_record, base_dir=base_dir, roots=roots)
1182 if config.trigger_mode != "event":
1183 raise DefinitionError(f"{config.definition_id} is not an event-mode definition")
1184 return generate_one(config, source_row, None, base_dir=base_dir, roots=roots)
1185
1186
1187def compute_event_handles(*, base_dir: Any, roots: Any = None) -> list[str]:
1188 """Return the sorted, de-duplicated ``<collection>.record.created``
1189 event list every currently enabled, non-blocked, event-mode definition
1190 implies -- i.e. what ``materialize_seed``'s static ``HANDLES`` literal
1191 WOULD need to contain for automatic on-create dispatch.
1192
1193 Pure computation, no side effects. In v1 nothing wires this into the
1194 object's source at runtime (we do NOT rewrite installed objects under
1195 the poll loop -- see object_daemon.process_materializations); it exists
1196 for inspection/tests and as the input a future *deliberate, scheduled*
1197 HANDLES-sync mechanism (with its own tests and a real performance
1198 rationale) would use. Event-mode definitions run via the manual path
1199 (materialize_run) until then.
1200 """
1201 try:
1202 definitions = object_records.read_collection_records(
1203 MATERIALIZE_DEFINITIONS_COLLECTION, base_dir=base_dir, roots=roots
1204 )
1205 except (object_collections.CollectionNotFoundError, object_collections.InvalidCollectionNameError):
1206 return []
1207
1208 events: set[str] = set()
1209 for definition in definitions:
1210 if is_definition_blocked(definition) or not is_definition_enabled(definition):
1211 continue
1212 trigger = _json_field(definition, "trigger", None, definition_id=definition.get("id") or "<unknown>")
1213 if not isinstance(trigger, dict) or trigger.get("mode") != "event":
1214 continue
1215 source_collection = (definition.get("source_collection") or "").strip()
1216 if source_collection:
1217 events.add(f"{source_collection}.record.created")
1218 return sorted(events)
1219
1220
1221# NOTE: there is deliberately no runtime "sync HANDLES by rewriting the
1222# installed materialize_seed.py" function here. Rewriting an existing
1223# object's source under the daemon poll loop is the wrong shape (expensive,
1224# surprising, self-modifying). materialize_seed ships inert (HANDLES == [])
1225# and event-mode runs manually (materialize_run). If automatic on-create
1226# dispatch is ever worth wiring, compute_event_handles above is the input a
1227# separate, scheduled, tested job would consume -- not this module at import
1228# or the poll loop at runtime.
1229
1230
1231# --- Small shared helpers -------------------------------------------------
1232
1233def _stringify(value: Any) -> str:
1234 if value is None:
1235 return ""
1236 if isinstance(value, bool):
1237 return "true" if value else "false"
1238 return str(value)
1239
1240
1241def _to_cents(value: Any) -> int:
1242 text = str(value if value is not None else "").strip()
1243 if not text:
1244 return 0
1245 try:
1246 return int(text)
1247 except ValueError as exc:
1248 raise DefinitionError(f"expected an integer cents value, got {value!r}") from exc
1249
1250
1251def _truthy(value: Any, *, default: bool) -> bool:
1252 if value is None or value == "":
1253 return default
1254 return str(value).strip().lower() in {"true", "1", "yes", "on"}
1255
1256
1257def _positive_int(value: Any) -> int | None:
1258 try:
1259 parsed = int(str(value).strip())
1260 except (TypeError, ValueError):
1261 return None
1262 return parsed if parsed > 0 else None
1263
1264
1265def _parse_iso(value: str) -> datetime | None:
1266 try:
1267 parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
1268 except ValueError:
1269 return None
1270 if parsed.tzinfo is None:
1271 parsed = parsed.replace(tzinfo=timezone.utc)
1272 return parsed
1273
1274
1275def _json_field(record: Mapping[str, str], name: str, default: Any, *, definition_id: str) -> Any:
1276 raw = record.get(name)
1277 if raw is None:
1278 return default
1279 if not isinstance(raw, str):
1280 return raw
1281 raw = raw.strip()
1282 if not raw:
1283 return default
1284 try:
1285 return json.loads(raw)
1286 except json.JSONDecodeError as exc:
1287 raise DefinitionError(f"{definition_id}: {name} is not valid JSON: {exc}") from exc