Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

dbbasic-object-server/object_template_runs.py
Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
object_template_runs.py · 14.5 KB · 362 lines
1"""object_template_runs -- the arithmetic of running a template.
2
3Pure. No I/O, no clock, no data directory. The same posture as
4object_rates, object_cart and object_promotions: the package objects in
5packages/app-runner do the reading and writing, and everything here is
6testable without either.
7
8## What a run IS
9
10A run is a COMMITMENT the moment it is queued: money is placed on hold
11against it, a provider may be paid real cash to execute it, and the terms
12it runs under are stamped onto its own row so that editing the template
13tomorrow restates nothing that already happened
14(docs/logic-decisions.md #1). A template is edited constantly -- that is
15what makes it a template -- so the run must carry its own copy of the
16handler, the body, the price and the model that were in force when the
17user pressed the button. A user reading last month's run history is
18reading what last month actually cost, not what this month's prices would
19have made it cost.
20
21## The money model: hold, release, charge
22
23Three wallet movements, and the naming is the design:
24
25* **hold** (negative) -- placed at queue time for the stamped price. Goes
26 through the wallet gate, which is the point: the moment the gate is
27 asked is the moment money is committed, so a user with 5 cents cannot
28 queue four hundred runs and present the provider bill later.
29* **release** (positive) -- the hold handed back at settlement, always in
30 full, always paired to its hold by provenance marker.
31* **charge** -- an ordinary `debit` for what the run actually cost the
32 user. Deliberately NOT a new kind: every report that already reads
33 debits keeps working, and a released hold never masquerades as a
34 refund. A hold is not a debit that got undone; it is money that was
35 never spent, and the ledger should read that way.
36
37A successful run settles as release + debit (sum: the price). A failed
38run settles as release alone (sum: zero). Nothing is ever mutated and the
39balance stays derived, so every state -- reserved, spent, given back --
40is reconstructible from the entries alone.
41
42## Provenance markers
43
44Each movement carries a `generated_from` marker naming the run and the
45leg (`template_run/<id>/hold`, `/release`, `/charge`). Settlement checks
46for the release marker before writing, which is what makes it idempotent:
47the runner and the sweeper can both try to settle the same run and the
48second one finds the marker and writes nothing (doctrine #7). The failure
49this prevents is not theoretical -- it is a crash between "settled" and
50"status updated", replayed at the next pass, giving the money back twice.
51"""
52
53import json
54import re
55
56# The one non-negotiable ordering rule, written down where both the
57# runner and the sweeper import it: a stale run is never re-queued.
58# Retrying a paid, non-idempotent provider call automatically is the sort
59# of feature that looks like robustness and is actually an unbounded
60# spend. A retry is a NEW run, with a new hold, started by a person.
61TERMINAL_STATUSES = ("succeeded", "failed", "abandoned")
62
63# `running` sits between claimed and terminal: the provider has the job,
64# we have a provider_job_id, and no pass is executing anything. It is
65# owned but idle, which is why it needs its own status rather than
66# stretching `claimed` to cover it.
67RUNNING_STATUS = "running"
68
69# The one state a machine must not resolve. A submitted job whose outcome
70# we never learned is money that MAY already be spent at the provider:
71# releasing the hold gives away what we paid, and charging bills a
72# customer for something they may never have received. Both are wrong and
73# the box genuinely cannot tell which, so it stops and asks. Deliberately
74# NOT in TERMINAL_STATUSES -- that tuple is what drives the sweeper's
75# automatic hold release, and the whole point is that this one is not
76# released automatically.
77NEEDS_REVIEW_STATUS = "needs_review"
78
79HOLD_MARKER = "template_run/{run_id}/hold"
80RELEASE_MARKER = "template_run/{run_id}/release"
81CHARGE_MARKER = "template_run/{run_id}/charge"
82
83_PLACEHOLDER = re.compile(r"\{([a-zA-Z_][a-zA-Z0-9_]*)\}")
84
85
86def _text(value):
87 return str(value if value is not None else "").strip()
88
89
90def _int(value, default=0):
91 try:
92 return int(_text(value) or default)
93 except (TypeError, ValueError):
94 return default
95
96
97# --- the form ----------------------------------------------------------------
98
99def form_fields(template):
100 """The template's declared form, as a list of {name, required, ...}.
101
102 templates.schema is a JSON Schema string (stored as text because the
103 field-type contract has no json type). Only the parts a runner needs
104 are read: property names and the required list. A template with no
105 schema has no form and accepts anything, which is the right default
106 for a body with no placeholders.
107 """
108 raw = _text(template.get("schema"))
109 if not raw:
110 return []
111 try:
112 parsed = json.loads(raw)
113 except ValueError:
114 return []
115 properties = parsed.get("properties")
116 if not isinstance(properties, dict):
117 return []
118 required = set(parsed.get("required") or [])
119 return [{"name": name, "required": name in required,
120 "title": _text((spec or {}).get("title")) or name}
121 for name, spec in properties.items()]
122
123
124def problems(template, form_data):
125 """Every reason this submission cannot run, at once.
126
127 A list rather than a first failure -- a form that reveals one problem
128 per attempt is a form people abandon, which is the same argument
129 object_promotions.blockers makes about checkouts.
130 """
131 found = []
132 handler = _text(template.get("handler"))
133 if not handler:
134 found.append(
135 f"The template '{_text(template.get('name')) or '?'}' declares no "
136 f"handler, so there is nothing to run it with. Set `handler` on "
137 f"the template (e.g. 'echo' or 'ai_text').")
138
139 form_data = form_data if isinstance(form_data, dict) else {}
140 for field in form_fields(template):
141 if field["required"] and not _text(form_data.get(field["name"])):
142 found.append(f"'{field['title']}' is required and was not provided.")
143
144 price = _int(template.get("run_cost_cents"))
145 if price < 0:
146 found.append("run_cost_cents is negative; a template cannot pay "
147 "people to run it.")
148 return found
149
150
151def render(body, form_data):
152 """The template body with {name} placeholders filled from the form.
153
154 Unknown placeholders stay as written rather than erroring or blanking:
155 a brace in prose ("{TBD}") must not kill a run, and a silently emptied
156 placeholder is a prompt that quietly lost its subject -- the worst
157 outcome, because the provider answers SOMETHING and charges for it.
158 """
159 form_data = form_data if isinstance(form_data, dict) else {}
160
161 def fill(match):
162 key = match.group(1)
163 if key in form_data:
164 return str(form_data[key])
165 return match.group(0)
166
167 return _PLACEHOLDER.sub(fill, str(body or ""))
168
169
170# --- stamping ------------------------------------------------------------------
171
172def stamp(template, form_data, *, model=""):
173 """The terms of this run, copied at queue time.
174
175 Everything execution or a later reader needs, taken from the template
176 NOW, so the run is immune to every future edit of the template. The
177 rendered body is stamped rather than re-rendered later for the same
178 reason: rendering reads the template, and by execution time the
179 template may say something else.
180 """
181 return {
182 "template_id": _text(template.get("id")),
183 "template_name": _text(template.get("name")),
184 "handler": _text(template.get("handler")),
185 "model": _text(model),
186 "form_data": json.dumps(form_data if isinstance(form_data, dict) else {},
187 sort_keys=True),
188 "rendered_body": render(template.get("body"), form_data),
189 "instructions": _text(template.get("instructions")),
190 "price_cents": str(max(0, _int(template.get("run_cost_cents")))),
191 }
192
193
194# --- settlement ----------------------------------------------------------------
195
196def settlement(run, *, succeeded):
197 """The wallet movements that close this run. Pure: returns a list of
198 entries for the caller to write, after checking the release marker.
199
200 Success: release the hold in full, then an ordinary debit for the
201 stamped price. Failure: release alone -- the user pays nothing for a
202 run that did not do the thing, whatever it cost US at the provider
203 (our cost is recorded on the run for margin, not billed to them).
204
205 A free run (price 0) had no hold and settles to no entries at all.
206 """
207 run_id = _text(run.get("id"))
208 price = _int(run.get("price_cents"))
209 wallet_id = _text(run.get("wallet_id"))
210 if price <= 0 or not wallet_id:
211 return []
212
213 entries = [{
214 "wallet_id": wallet_id,
215 "amount_minor": str(price),
216 "kind": "release",
217 "description": f"Hold released: run {run_id}",
218 "reference": f"template_runs/{run_id}",
219 "generated_from": RELEASE_MARKER.format(run_id=run_id),
220 "owner_id": _text(run.get("owner_id")),
221 }]
222 if succeeded:
223 entries.append({
224 "wallet_id": wallet_id,
225 "amount_minor": str(-price),
226 "kind": "debit",
227 "description": (f"Template run: "
228 f"{_text(run.get('template_name')) or run_id}"),
229 "reference": f"template_runs/{run_id}",
230 "generated_from": CHARGE_MARKER.format(run_id=run_id),
231 "owner_id": _text(run.get("owner_id")),
232 })
233 return entries
234
235
236def hold_entry(run_id, wallet_id, price_cents, owner_id, template_name=""):
237 """The hold placed at queue time. Negative, so the wallet gate fires
238 on it -- which is the entire reason a hold exists: the gate is asked
239 at the moment of commitment, not at the moment of spending."""
240 price = max(0, _int(price_cents))
241 if price == 0:
242 return None
243 return {
244 "wallet_id": _text(wallet_id),
245 "amount_minor": str(-price),
246 "kind": "hold",
247 "description": f"Hold: template run {_text(template_name) or run_id}",
248 "reference": f"template_runs/{run_id}",
249 "generated_from": HOLD_MARKER.format(run_id=run_id),
250 "owner_id": _text(owner_id),
251 }
252
253
254def already_settled(run_id, wallet_entries):
255 """Has this run's hold already been released?
256
257 THE idempotency check. The runner settling a run it just executed and
258 the sweeper settling a run whose worker died can both reach here; the
259 marker means at most one of them writes. Checked against the entries
260 themselves, never against run.status, because the crash this guards
261 is precisely the one between writing money and updating status.
262 """
263 marker = RELEASE_MARKER.format(run_id=_text(run_id))
264 return any(_text(row.get("generated_from")) == marker
265 for row in wallet_entries or ())
266
267
268def outstanding_holds(wallet_entries):
269 """Sum of holds not yet released, by run id -- what is reserved right
270 now. A fold over the entries, like every other balance here."""
271 held = {}
272 for row in wallet_entries or ():
273 marker = _text(row.get("generated_from"))
274 if marker.startswith("template_run/"):
275 parts = marker.split("/")
276 if len(parts) == 3 and parts[2] in ("hold", "release"):
277 run_id = parts[1]
278 held[run_id] = held.get(run_id, 0) + _int(row.get("amount_minor"))
279 return {run_id: -total for run_id, total in held.items() if total != 0}
280
281
282def poll_disposition(run, *, now, stale_seconds, max_run_seconds):
283 """What should be done with this run right now? Pure; no clock.
284
285 The money rule for asynchronous jobs lives here, in one place, so the
286 sweeper and the runner cannot disagree about it:
287
288 - **claimed, heartbeat stopped, no provider_job_id** -> "abandon".
289 Nothing was submitted, so nothing was spent, and the hold is
290 released in full. This is the original sweeper behaviour and it
291 stays exactly right for the case it was written for.
292
293 - **running (or claimed WITH a provider_job_id), heartbeat stopped**
294 -> "poll". NOT abandoned. The provider is holding the job and can
295 simply be asked, so a dead worker is not a lost job -- and crucially
296 the hold is NOT released, because a submitted job may already have
297 cost everything. This is the case the original sweeper got actively
298 wrong for media.
299
300 - **running past max_run_seconds** -> "review". Polling has not
301 resolved it for longer than any job should take, so it stops being
302 a machine's decision. The hold stays put.
303
304 - anything else -> None, leave it alone.
305
306 A claimed run that carries a provider_job_id is treated as running
307 even if the status write never landed: the id is the evidence that a
308 call was made, and evidence outranks a status field that a crash may
309 have prevented.
310 """
311 status = _text(run.get("status"))
312 if status in TERMINAL_STATUSES or status == NEEDS_REVIEW_STATUS:
313 return None
314
315 submitted = bool(_text(run.get("provider_job_id")))
316 beat = _text(run.get("heartbeat_at")) or _text(run.get("claimed_at"))
317
318 if status == RUNNING_STATUS or (status == "claimed" and submitted):
319 started = _text(run.get("claimed_at")) or beat
320 if started:
321 deadline = _shift_iso(started, abs(int(max_run_seconds)))
322 if _text(now) > deadline:
323 return "review"
324 return "poll"
325
326 if status == "claimed":
327 if not beat:
328 return "abandon"
329 if beat < _shift_iso(_text(now), -abs(int(stale_seconds))):
330 return "abandon"
331 return None
332
333
334def is_stale(run, *, now, stale_seconds):
335 """Has this claimed run's heartbeat stopped?
336
337 String comparison on ISO-8601 timestamps, which sort lexicographically
338 -- the same convention the rest of the codebase leans on. `now` is a
339 parameter, never read from a clock here, so the sweeper's tests do not
340 need to sleep.
341 """
342 if _text(run.get("status")) not in ("claimed",):
343 return False
344 beat = _text(run.get("heartbeat_at")) or _text(run.get("claimed_at"))
345 if not beat:
346 return True
347 cutoff = _shift_iso(_text(now), -abs(int(stale_seconds)))
348 return beat < cutoff
349
350
351def _shift_iso(stamp, seconds):
352 """An ISO timestamp moved by N seconds, staying pure (no clock)."""
353 from datetime import datetime, timedelta
354
355 text = stamp.rstrip("Z")
356 try:
357 parsed = datetime.fromisoformat(text)
358 except ValueError:
359 return stamp
360 shifted = parsed + timedelta(seconds=seconds)
361 out = shifted.isoformat()
362 return out + "Z" if stamp.endswith("Z") else out