Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
record_conversion.py · 12.4 KB · 312 lines
1"""system_record_conversion -- the goals finally get written down.
2
3`conversions` has existed as a collection for a while and NOTHING wrote to
4it. The shape was right and the wiring was missing, which is a worse
5failure than an absent feature: a page can render an empty table and read
6as "nobody converted" when the truth is "nothing was ever recorded".
7
8The implementation is small, because this system is full of transitions
9and **a conversion is simply a transition worth counting**. Nothing new
10had to be observed; four facts the server already knew just never landed
11anywhere a report could see them:
12
13 order_confirmed an order reached confirmed (or anywhere past it)
14 order_collected a pickup order was actually handed over
15 payment_received money arrived against an invoice
16 scan_confirmed a human confirmed a scanned document into the books
17
18Placement is the same posture as system_order_email, deliberately: a
19REACTION, post-commit, best-effort, never a gate. It cannot fail the write
20that confirmed the order. Every failure lands in the result rather than in
21an exception, and a shop with no app-analytics installed keeps taking
22orders and is told plainly that nothing was counted.
23
24**Idempotent by provenance (#7).** The change dispatcher promises
25at-least-once delivery, and worse: `status is confirmed` is a state an
26order SITS in, not an edge it crosses, so every later write to a confirmed
27order fires this again. Each conversion therefore carries its source in
28the metadata blob --
29
30 {"source": "orders/ord-1", ...}
31
32-- and the handler refuses to write one whose (event_type, source) pair it
33can already find. That is the property the tests care about most: a
34replayed event must record nothing, because a double-counted conversion is
35a report that says the shop did twice the business it did, and there is no
36later signal that would ever correct it.
37
38The marker lives in metadata rather than in a column of its own because
39the schema is fixed and correct, and a private bookkeeping field for one
40handler is a worse trade than a documented key in a blob that already
41exists for exactly this kind of detail.
42
43**No session token is stamped, and that is honest rather than lazy.**
44`conversions.session_id` is the thread a funnel is stitched by, and these
45four goals have no browser anywhere near them: an order confirmed by staff
46the next morning is a back-office write with no cookie in scope, and the
47basket's `carts.session_token` is a DIFFERENT identifier in a different
48namespace -- treating it as the visitor token would silently merge two
49populations and produce a funnel that looks stitched and is not. So these
50rows are unthreaded, object_conversions.funnel counts them in
51`unthreaded_conversions` and says so on the page, and the day an app
52records a goal from a request that DID carry the visitor cookie, the
53field is there waiting.
54
55**`user_id` and `session_id` are never both written** --
56object_conversions.build_conversion enforces it. See docs/analytics.md,
57cookie rule 4.
58"""
59
60import os
61
62import object_conversions
63import object_records
64
65HANDLES = [
66 "orders.record.created",
67 "orders.record.updated",
68 "payments.record.created",
69 "scans.record.updated",
70]
71
72ACTOR = "system_record_conversion"
73
74# Anywhere at or past `confirmed` counts, not the exact value only -- the
75# same wider net system_order_email takes, and for the same reason: the
76# zero-touch shop confirms and ships in one burst of writes, so an order
77# can be `shipped` by the time this handler first sees it. A sale that
78# never counted because it moved too fast is exactly the undercount this
79# exists to end. The marker makes the wider net safe.
80CONFIRMED_ONWARD = {"confirmed", "processing", "partial", "shipped",
81 "delivered", "preparing", "ready", "collected"}
82
83# The counter's own goal, and a genuinely different fact from the sale:
84# the money was taken when the order was confirmed, and the food was
85# handed over when somebody walked in for it. A shop that wants to know
86# how many of its ready orders were actually collected cannot ask that of
87# `order_confirmed`.
88COLLECTED = "collected"
89
90# A bounced payment is money that did not arrive. It stays a payments row
91# (money moves, never mutates) and it is not a conversion.
92BOUNCED = "bounced"
93
94SCAN_CONFIRMED = "confirmed"
95
96EVENT_ORDER_CONFIRMED = "order_confirmed"
97EVENT_ORDER_COLLECTED = "order_collected"
98EVENT_PAYMENT_RECEIVED = "payment_received"
99EVENT_SCAN_CONFIRMED = "scan_confirmed"
100
101# Sale orders only. A purchase order reaching `confirmed` is money going
102# OUT; counting it as a conversion would report a shop's own buying as its
103# business won.
104SALE_DOC_TYPE = "sale"
105
106
107def _base_dir():
108 return os.environ.get("DBBASIC_DATA_DIR", "data")
109
110
111def _text(value):
112 return str(value if value is not None else "").strip()
113
114
115def _int(value):
116 try:
117 return int(str(value).strip() or 0)
118 except (TypeError, ValueError):
119 return 0
120
121
122def _record(base, collection, record_id):
123 if not record_id:
124 return None
125 try:
126 return object_records.get_collection_record(collection, record_id,
127 base_dir=base)
128 except Exception:
129 return None
130
131
132def _recorded(base):
133 """Every (event_type, source) already counted, or None when the
134 conversions collection cannot be read at all.
135
136 None is a REFUSAL, not an empty set -- the same argument
137 system_order_email's `_sent_markers` makes, and the asymmetry is the
138 same. A conversion this pass skips will be recorded by the next event
139 on the same record, and there is always a next event because the
140 dispatcher replays. A conversion recorded twice is a permanent
141 overcount in a report nothing will ever correct. When in doubt, count
142 nothing, and say WHICH doubt -- "app-analytics is not installed" must
143 never read as "already counted".
144 """
145 try:
146 rows = object_records.read_collection_records(
147 object_conversions.CONVERSIONS_COLLECTION, base_dir=base)
148 except Exception:
149 return None
150 return object_conversions.recorded_sources(rows)
151
152
153def _write(base, *, event_type, source, metadata, user_id=""):
154 """One conversions row, or the reason there is not one."""
155 record = object_conversions.build_conversion(
156 event_type=event_type, source=source, metadata=metadata,
157 user_id=user_id)
158 try:
159 object_records.create_collection_record(
160 object_conversions.CONVERSIONS_COLLECTION, record,
161 base_dir=base, actor=ACTOR)
162 except Exception as exc:
163 return str(exc)[:200]
164 return ""
165
166
167def _goals_for_order(order):
168 """(event_type, metadata) pairs this order has earned.
169
170 Money is carried in minor units exactly as the record holds them --
171 no formatting, no currency conversion, no second opinion about what
172 the order is worth. A report that reformats money is a report that can
173 disagree with the document it describes.
174 """
175 status = _text(order.get("status"))
176 number = _text(order.get("number")) or _text(order.get("id"))
177 common = {
178 "order_number": number,
179 "amount_cents": _int(order.get("total_cents")),
180 "currency": _text(order.get("currency")) or "USD",
181 }
182 goals = []
183 if status in CONFIRMED_ONWARD:
184 goals.append((EVENT_ORDER_CONFIRMED, dict(common)))
185 if status == COLLECTED:
186 goals.append((EVENT_ORDER_COLLECTED,
187 dict(common, fulfillment_method=_text(
188 order.get("fulfillment_method")))))
189 return goals
190
191
192def _on_order(base, order_id, recorded):
193 order = _record(base, "orders", order_id)
194 if order is None:
195 return {"ok": True, "skipped": "order gone", "recorded": []}
196
197 # doc_type is read with .get so an orders schema that predates the
198 # sale/purchase split still counts its orders rather than none of them.
199 doc_type = _text(order.get("doc_type"))
200 if doc_type and doc_type != SALE_DOC_TYPE:
201 return {"ok": True, "recorded": [],
202 "skipped": f"{doc_type} order: not a conversion",
203 "order_id": order_id}
204
205 source = object_conversions.source_ref("orders", order_id)
206 return _apply(base, source, _goals_for_order(order), recorded,
207 extra={"order_id": order_id})
208
209
210def _on_payment(base, payment_id, recorded):
211 payment = _record(base, "payments", payment_id)
212 if payment is None:
213 return {"ok": True, "skipped": "payment gone", "recorded": []}
214 if _text(payment.get("status")) == BOUNCED:
215 return {"ok": True, "recorded": [],
216 "skipped": "bounced: the money did not arrive",
217 "payment_id": payment_id}
218
219 source = object_conversions.source_ref("payments", payment_id)
220 metadata = {
221 "amount_cents": _int(payment.get("amount_cents")),
222 "invoice_id": _text(payment.get("invoice_id")),
223 "method": _text(payment.get("method")),
224 }
225 return _apply(base, source, [(EVENT_PAYMENT_RECEIVED, metadata)],
226 recorded, extra={"payment_id": payment_id})
227
228
229def _on_scan(base, scan_id, recorded):
230 scan = _record(base, "scans", scan_id)
231 if scan is None:
232 return {"ok": True, "skipped": "scan gone", "recorded": []}
233 if _text(scan.get("status")) != SCAN_CONFIRMED:
234 return {"ok": True, "recorded": [],
235 "skipped": "not confirmed yet", "scan_id": scan_id}
236
237 source = object_conversions.source_ref("scans", scan_id)
238 # What it BECAME, not what was in it. The extraction is a guess and
239 # the image is evidence; neither belongs in an analytics row.
240 metadata = {
241 "kind": _text(scan.get("category_hint")),
242 "became": _text(scan.get("confirmed_record")),
243 }
244 return _apply(base, source, [(EVENT_SCAN_CONFIRMED, metadata)],
245 recorded, extra={"scan_id": scan_id})
246
247
248def _apply(base, source, goals, recorded, *, extra=None):
249 written, already, errors = [], [], {}
250 for event_type, metadata in goals:
251 if (event_type, source) in recorded:
252 already.append(event_type)
253 continue
254 error = _write(base, event_type=event_type, source=source,
255 metadata=metadata)
256 if error:
257 errors[event_type] = error
258 else:
259 written.append(event_type)
260 # Within one invocation too: an order that is both confirmed
261 # and collected writes two DIFFERENT event types, but a goal
262 # must never be written twice by one pass either.
263 recorded.add((event_type, source))
264 result = {"ok": True, "recorded": written, "skipped_already_counted": already,
265 "source": source}
266 result.update(extra or {})
267 if errors:
268 result["errors"] = errors
269 return result
270
271
272def EVENT(request):
273 # The dispatcher's payload carries the RAW verb ("create"/"update");
274 # the event NAME uses the participle. Accept both.
275 action = _text(request.get("action"))
276 action = {"create": "created", "update": "updated",
277 "delete": "deleted"}.get(action, action)
278 if action not in ("created", "updated"):
279 return {"ok": True, "skipped": "not a create or update", "recorded": []}
280
281 record_id = _text(request.get("record_id") or request.get("id"))
282 if not record_id:
283 return {"ok": True, "skipped": "no record id", "recorded": []}
284
285 collection = _text(request.get("collection"))
286 if collection not in ("orders", "payments", "scans"):
287 return {"ok": True, "recorded": [],
288 "skipped": f"nothing to count about {collection or 'nothing'}"}
289
290 base = _base_dir()
291 recorded = _recorded(base)
292 if recorded is None:
293 # Honest degradation: app-analytics is not installed, or the
294 # collection cannot be read. Nothing is counted and the reason
295 # says which -- silence here would look exactly like "already
296 # counted", and the shop would never find out it had no numbers.
297 return {"ok": True, "recorded": [],
298 "skipped": "conversions collection unreadable "
299 "(app-analytics not installed?); nothing counted"}
300
301 if collection == "orders":
302 return _on_order(base, record_id, recorded)
303 if collection == "payments":
304 return _on_payment(base, record_id, recorded)
305 return _on_scan(base, record_id, recorded)
306
307
308# EVENT is the verb the change dispatcher calls handlers with; POST stays
309# as an alias so an operator can poke the handler by hand over HTTP. A
310# handler shipped with only POST silently matched nothing in production
311# once, and every handler in this house has carried both ever since.
312POST = EVENT