askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
publish_head.py · 12.8 KB · 324 lines
| 1 | """system_publish_head -- take a digest of each ledger and lodge it where |
| 2 | this server cannot reach. |
| 3 | |
| 4 | Runs daily from the package's own schedule, and by hand whenever an |
| 5 | operator wants a fresh anchor before something important. |
| 6 | |
| 7 | ## Why this is a system object rather than an action |
| 8 | |
| 9 | plan/tamper-evidence-spec.md called it `action_publish_head`. It is a |
| 10 | system object because it is TIME-DRIVEN, and time-driven work belongs to |
| 11 | the daemon (docs/logic-decisions.md #2). An anchoring pass that only |
| 12 | happens when somebody remembers to press a button is an anchoring pass |
| 13 | with a gap exactly where an incident is. It can still be executed by hand |
| 14 | -- the daily schedule and the button are the same object, so they cannot |
| 15 | drift apart. |
| 16 | |
| 17 | ## What one pass does |
| 18 | |
| 19 | For each configured ledger: fold `object_ledger_head.head()` over the |
| 20 | rows, compare it with the newest anchor already recorded, and if anything |
| 21 | has changed, write a new anchor and submit the digest to every notary in |
| 22 | `notary.endpoints`. |
| 23 | |
| 24 | **Nothing is submitted anywhere except a digest.** No collection name a |
| 25 | stranger could use, no row count -- wait, the label DOES carry the |
| 26 | collection and the count, because they are useless to an outsider and |
| 27 | essential to the operator reading their own anchor back. What never |
| 28 | travels is a single row of content, which is the entire reason lodging a |
| 29 | digest with a stranger is a reasonable thing to do. |
| 30 | |
| 31 | ## The idempotency, and why it is not the usual one |
| 32 | |
| 33 | A ledger that has not changed since yesterday produces the same digest |
| 34 | over the same row count, and this pass writes nothing. That is not to |
| 35 | avoid a duplicate row for tidiness -- it is because an anchor's whole |
| 36 | content is `(row_count, digest)`, so a second identical anchor carries no |
| 37 | information a reader could act on, while a hundred of them make the |
| 38 | ladder `locate()` walks longer with no extra resolution. |
| 39 | |
| 40 | ## What it will not pretend |
| 41 | |
| 42 | **Anchoring to nowhere is recorded honestly.** With no notaries |
| 43 | configured, the anchor is still written and its status is `recorded` with |
| 44 | `notary_count` 0 -- which site_ledger_integrity renders as "these anchors |
| 45 | prove nothing against anyone with access to this server", in those words. |
| 46 | The alternative, a green tick for a digest nobody outside this box holds, |
| 47 | would be the single most misleading thing this package could do. |
| 48 | |
| 49 | **A notary that refuses does not lose the anchor.** The row is written |
| 50 | either way, with status `failed` and the reason in `note`. A digest only |
| 51 | this server holds is worth less than one a stranger holds and is worth |
| 52 | considerably more than no digest at all -- and tomorrow's pass will lodge |
| 53 | the same history under a longer prefix. |
| 54 | |
| 55 | **It does not verify.** Checking anchors against the ledger is |
| 56 | site_ledger_integrity's job, deliberately kept out of the writer: a pass |
| 57 | that both took anchors and blessed them would be marking its own homework. |
| 58 | """ |
| 59 | |
| 60 | import json |
| 61 | import os |
| 62 | import socket |
| 63 | import urllib.error |
| 64 | import urllib.request |
| 65 | |
| 66 | import object_ledger_head |
| 67 | import object_notary |
| 68 | import object_records |
| 69 | import object_schemas |
| 70 | |
| 71 | ACTOR = "system_publish_head" |
| 72 | ANCHORS = "anchors" |
| 73 | SETTINGS_COLLECTION = "app_settings" |
| 74 | |
| 75 | COLLECTIONS_KEY = "ledger.anchored_collections" |
| 76 | ENDPOINTS_KEY = "notary.endpoints" |
| 77 | BASE_URL_KEY = "portal.base_url" |
| 78 | |
| 79 | # Tests and local development only -- see |
| 80 | # object_notary.is_self_endpoint. An ENV VAR rather than a |
| 81 | # setting so that nobody can switch off the self-anchoring check |
| 82 | # from a settings page, which is exactly where somebody about to |
| 83 | # fool themselves would be looking. |
| 84 | ALLOW_LOOPBACK_ENV = "DBBASIC_NOTARY_ALLOW_LOOPBACK" |
| 85 | |
| 86 | TIMEOUT_SECONDS = 10 |
| 87 | MAX_RESPONSE_BYTES = 64 * 1024 |
| 88 | |
| 89 | # The ledgers worth anchoring when nobody has said otherwise. The rule |
| 90 | # behind the list, from plan/tamper-evidence-spec.md: ANCHOR WHAT SOMEBODY |
| 91 | # WOULD PROFIT FROM CHANGING. page_views is absent and stays absent -- |
| 92 | # nobody profits from forging traffic and it is the highest-volume writer |
| 93 | # on the box, so anchoring it would cost a full read of the largest file |
| 94 | # here every day to protect the least valuable thing in it. |
| 95 | DEFAULT_COLLECTIONS = ( |
| 96 | "wallet_entries", # customer money; the highest-value target |
| 97 | "fin_journal_lines", # the books an auditor arrives to check |
| 98 | "payments", |
| 99 | "refunds", |
| 100 | "stock_moves", # shrinkage is theft, and a backdated |
| 101 | # adjustment is how it gets hidden |
| 102 | "notarizations", # the notary's own log, which is otherwise |
| 103 | # the one thing here nobody is watching |
| 104 | ) |
| 105 | |
| 106 | |
| 107 | def _base_dir(): |
| 108 | return os.environ.get("DBBASIC_DATA_DIR", "data") |
| 109 | |
| 110 | |
| 111 | def _text(value): |
| 112 | return str(value if value is not None else "").strip() |
| 113 | |
| 114 | |
| 115 | def _settings(base): |
| 116 | try: |
| 117 | rows = object_records.read_collection_records(SETTINGS_COLLECTION, |
| 118 | base_dir=base) |
| 119 | except Exception: |
| 120 | return {} |
| 121 | return {_text(row.get("key")): _text(row.get("value")) |
| 122 | for row in rows if _text(row.get("key"))} |
| 123 | |
| 124 | |
| 125 | def _csv_setting(settings, key): |
| 126 | """Comma-separated setting into a list, honouring the "not configured" |
| 127 | sentinel. object_notary owns the parser so this pass and |
| 128 | site_ledger_integrity cannot disagree about what `none` means.""" |
| 129 | return object_notary.endpoints_from_setting(settings.get(key, "")) |
| 130 | |
| 131 | |
| 132 | def _schema_fields(base, collection): |
| 133 | """The field list, in schema order, minus the bookkeeping columns. |
| 134 | |
| 135 | `_op` is the append-storage marker and belongs to the file rather than |
| 136 | to the record, so hashing it would tie the digest to a storage detail |
| 137 | that compaction rewrites -- exactly what hashing logical rows is meant |
| 138 | to avoid. |
| 139 | """ |
| 140 | try: |
| 141 | schema = object_schemas.get_schema(collection, base_dir=base) |
| 142 | except Exception: |
| 143 | return [] |
| 144 | fields = [] |
| 145 | for field in (schema or {}).get("fields") or []: |
| 146 | name = _text(field.get("name")) |
| 147 | if name and name != "_op": |
| 148 | fields.append(name) |
| 149 | return fields |
| 150 | |
| 151 | |
| 152 | def _newest_anchor(anchors, collection): |
| 153 | newest = None |
| 154 | for row in anchors: |
| 155 | if _text(row.get("collection")) != collection: |
| 156 | continue |
| 157 | if newest is None or _text(row.get("created_at")) > _text( |
| 158 | newest.get("created_at")): |
| 159 | newest = row |
| 160 | return newest |
| 161 | |
| 162 | |
| 163 | def _lodge(endpoint, digest, label, *, own_base_url=""): |
| 164 | """Submit one digest to one notary. Returns (ok, detail). |
| 165 | |
| 166 | Refuses an endpoint that is this server before making any request. |
| 167 | Lodging with yourself always succeeds and proves nothing, and the cost |
| 168 | of letting it through is not a wasted request -- it is |
| 169 | site_ledger_integrity swapping "anchored, but only here" for |
| 170 | "verified, and held elsewhere" on the strength of it, which turns an |
| 171 | honest missing claim into a specific false one. |
| 172 | |
| 173 | Deliberately narrow: a POST to a known object with a known body, no |
| 174 | redirects followed, a timeout, and a bounded read. Every failure mode |
| 175 | -- refused, timed out, 4xx, unparseable -- comes back as (False, |
| 176 | reason) rather than raising, because one unreachable notary must not |
| 177 | stop the pass from lodging with the others or from recording the |
| 178 | anchor locally. |
| 179 | """ |
| 180 | allow_loopback = os.environ.get(ALLOW_LOOPBACK_ENV, "").strip().lower() in ( |
| 181 | "1", "true", "yes", "on") |
| 182 | if object_notary.is_self_endpoint(endpoint, own_base_url, |
| 183 | allow_loopback=allow_loopback): |
| 184 | return False, (f"{endpoint} is this server. An anchor a server holds " |
| 185 | f"about itself is not independent, so it is not " |
| 186 | f"counted -- point this at a notary somebody else " |
| 187 | f"runs.") |
| 188 | |
| 189 | url = endpoint.rstrip("/") + "/objects/action_notarize" |
| 190 | body = json.dumps({"digest": digest, "algorithm": object_ledger_head.ALGORITHM, |
| 191 | "label": label}).encode("utf-8") |
| 192 | request = urllib.request.Request( |
| 193 | url, data=body, method="POST", |
| 194 | headers={"Content-Type": "application/json", |
| 195 | "User-Agent": "dbbasic-object-server/publish-head"}) |
| 196 | try: |
| 197 | with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response: |
| 198 | payload = response.read(MAX_RESPONSE_BYTES) |
| 199 | except urllib.error.HTTPError as error: |
| 200 | return False, f"{endpoint} answered {error.code}" |
| 201 | except (urllib.error.URLError, socket.timeout, TimeoutError, OSError) as error: |
| 202 | return False, f"{endpoint} unreachable ({error})" |
| 203 | |
| 204 | try: |
| 205 | answer = json.loads(payload.decode("utf-8")) |
| 206 | except (ValueError, UnicodeDecodeError): |
| 207 | return False, f"{endpoint} answered something that was not JSON" |
| 208 | |
| 209 | if answer.get("found") is True: |
| 210 | # `already_recorded` is a success, and an interesting one: it means |
| 211 | # this exact digest was lodged there before, so the history it |
| 212 | # covers has an even older independent timestamp than this pass. |
| 213 | return True, ("already held" if answer.get("already_recorded") |
| 214 | else "accepted") |
| 215 | return False, f"{endpoint} refused: {_text(answer.get('error'))[:120]}" |
| 216 | |
| 217 | |
| 218 | def _anchor_one(base, collection, endpoints, anchors, *, own_base_url=""): |
| 219 | fields = _schema_fields(base, collection) |
| 220 | if not fields: |
| 221 | return {"collection": collection, "status": "skipped", |
| 222 | "reason": "no schema on this box"} |
| 223 | |
| 224 | try: |
| 225 | rows = object_records.read_collection_records(collection, base_dir=base) |
| 226 | except Exception as error: |
| 227 | return {"collection": collection, "status": "skipped", |
| 228 | "reason": f"unreadable ({error})"} |
| 229 | |
| 230 | folded = object_ledger_head.head(rows, fields, collection=collection) |
| 231 | |
| 232 | previous = _newest_anchor(anchors, collection) |
| 233 | if previous is not None and _text(previous.get("digest")) == folded["digest"]: |
| 234 | return {"collection": collection, "status": "unchanged", |
| 235 | "row_count": folded["row_count"], |
| 236 | "reason": "nothing appended since the last anchor"} |
| 237 | |
| 238 | label = f"{collection} head @ {folded['row_count']} rows" |
| 239 | lodged, notes = [], [] |
| 240 | for endpoint in endpoints: |
| 241 | ok, detail = _lodge(endpoint, folded["digest"], label, |
| 242 | own_base_url=own_base_url) |
| 243 | if ok: |
| 244 | lodged.append(endpoint) |
| 245 | notes.append(f"{endpoint}: {detail}") |
| 246 | |
| 247 | if not endpoints: |
| 248 | status = "recorded" |
| 249 | elif len(lodged) == len(endpoints): |
| 250 | status = "published" |
| 251 | elif lodged: |
| 252 | status = "partial" |
| 253 | else: |
| 254 | status = "failed" |
| 255 | |
| 256 | record = { |
| 257 | "collection": collection, |
| 258 | "row_count": str(folded["row_count"]), |
| 259 | "digest": folded["digest"], |
| 260 | "algorithm": folded["algorithm"], |
| 261 | "scheme": folded["scheme"], |
| 262 | "covered_fields": ",".join(folded["fields"]), |
| 263 | "notaries": ",".join(lodged), |
| 264 | "notary_count": str(len(lodged)), |
| 265 | "status": status, |
| 266 | "note": "; ".join(notes)[:500], |
| 267 | } |
| 268 | try: |
| 269 | stored = object_records.create_collection_record( |
| 270 | ANCHORS, record, base_dir=base, actor=ACTOR) |
| 271 | except Exception as error: |
| 272 | return {"collection": collection, "status": "error", |
| 273 | "reason": f"the anchor could not be written: {error}"} |
| 274 | |
| 275 | return {"collection": collection, "status": status, |
| 276 | "row_count": folded["row_count"], "digest": folded["digest"], |
| 277 | "notary_count": len(lodged), "anchor_id": stored.get("id")} |
| 278 | |
| 279 | |
| 280 | def EVENT(request): |
| 281 | """The daemon entry point. |
| 282 | |
| 283 | Named EVENT with POST aliased below because the change dispatcher and |
| 284 | the scheduler call EVENT -- declaring only POST is a pass that silently |
| 285 | never runs, which has been a real production bug here twice. |
| 286 | """ |
| 287 | request = request or {} |
| 288 | base = _base_dir() |
| 289 | settings = _settings(base) |
| 290 | |
| 291 | requested = _csv_setting(settings, COLLECTIONS_KEY) or list(DEFAULT_COLLECTIONS) |
| 292 | endpoints = _csv_setting(settings, ENDPOINTS_KEY) |
| 293 | |
| 294 | try: |
| 295 | anchors = object_records.read_collection_records(ANCHORS, base_dir=base) |
| 296 | except Exception: |
| 297 | anchors = [] |
| 298 | |
| 299 | own_base_url = settings.get(BASE_URL_KEY, "") |
| 300 | results = [_anchor_one(base, collection, endpoints, anchors, |
| 301 | own_base_url=own_base_url) |
| 302 | for collection in requested] |
| 303 | |
| 304 | written = [r for r in results |
| 305 | if r["status"] in ("recorded", "published", "partial", "failed")] |
| 306 | independent = sum(r.get("notary_count", 0) for r in written) |
| 307 | |
| 308 | return { |
| 309 | "ok": True, |
| 310 | "anchored": len(written), |
| 311 | "considered": len(results), |
| 312 | "endpoints": endpoints, |
| 313 | "independent_lodgements": independent, |
| 314 | "results": results, |
| 315 | "warning": ( |
| 316 | f"No notary endpoints are configured ({ENDPOINTS_KEY}), so these " |
| 317 | f"anchors are held only by the server that took them. Anybody " |
| 318 | f"able to rewrite a ledger here can rewrite its anchor beside it. " |
| 319 | f"They become evidence when the digest is also held somewhere " |
| 320 | f"else." if not endpoints else ""), |
| 321 | } |
| 322 | |
| 323 | |
| 324 | POST = EVENT |