Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
publish_head.py · 12.8 KB · 324 lines
1"""system_publish_head -- take a digest of each ledger and lodge it where
2this server cannot reach.
3
4Runs daily from the package's own schedule, and by hand whenever an
5operator wants a fresh anchor before something important.
6
7## Why this is a system object rather than an action
8
9plan/tamper-evidence-spec.md called it `action_publish_head`. It is a
10system object because it is TIME-DRIVEN, and time-driven work belongs to
11the daemon (docs/logic-decisions.md #2). An anchoring pass that only
12happens when somebody remembers to press a button is an anchoring pass
13with a gap exactly where an incident is. It can still be executed by hand
14-- the daily schedule and the button are the same object, so they cannot
15drift apart.
16
17## What one pass does
18
19For each configured ledger: fold `object_ledger_head.head()` over the
20rows, compare it with the newest anchor already recorded, and if anything
21has changed, write a new anchor and submit the digest to every notary in
22`notary.endpoints`.
23
24**Nothing is submitted anywhere except a digest.** No collection name a
25stranger could use, no row count -- wait, the label DOES carry the
26collection and the count, because they are useless to an outsider and
27essential to the operator reading their own anchor back. What never
28travels is a single row of content, which is the entire reason lodging a
29digest with a stranger is a reasonable thing to do.
30
31## The idempotency, and why it is not the usual one
32
33A ledger that has not changed since yesterday produces the same digest
34over the same row count, and this pass writes nothing. That is not to
35avoid a duplicate row for tidiness -- it is because an anchor's whole
36content is `(row_count, digest)`, so a second identical anchor carries no
37information a reader could act on, while a hundred of them make the
38ladder `locate()` walks longer with no extra resolution.
39
40## What it will not pretend
41
42**Anchoring to nowhere is recorded honestly.** With no notaries
43configured, the anchor is still written and its status is `recorded` with
44`notary_count` 0 -- which site_ledger_integrity renders as "these anchors
45prove nothing against anyone with access to this server", in those words.
46The alternative, a green tick for a digest nobody outside this box holds,
47would be the single most misleading thing this package could do.
48
49**A notary that refuses does not lose the anchor.** The row is written
50either way, with status `failed` and the reason in `note`. A digest only
51this server holds is worth less than one a stranger holds and is worth
52considerably more than no digest at all -- and tomorrow's pass will lodge
53the same history under a longer prefix.
54
55**It does not verify.** Checking anchors against the ledger is
56site_ledger_integrity's job, deliberately kept out of the writer: a pass
57that both took anchors and blessed them would be marking its own homework.
58"""
59
60import json
61import os
62import socket
63import urllib.error
64import urllib.request
65
66import object_ledger_head
67import object_notary
68import object_records
69import object_schemas
70
71ACTOR = "system_publish_head"
72ANCHORS = "anchors"
73SETTINGS_COLLECTION = "app_settings"
74
75COLLECTIONS_KEY = "ledger.anchored_collections"
76ENDPOINTS_KEY = "notary.endpoints"
77BASE_URL_KEY = "portal.base_url"
78
79# Tests and local development only -- see
80# object_notary.is_self_endpoint. An ENV VAR rather than a
81# setting so that nobody can switch off the self-anchoring check
82# from a settings page, which is exactly where somebody about to
83# fool themselves would be looking.
84ALLOW_LOOPBACK_ENV = "DBBASIC_NOTARY_ALLOW_LOOPBACK"
85
86TIMEOUT_SECONDS = 10
87MAX_RESPONSE_BYTES = 64 * 1024
88
89# The ledgers worth anchoring when nobody has said otherwise. The rule
90# behind the list, from plan/tamper-evidence-spec.md: ANCHOR WHAT SOMEBODY
91# WOULD PROFIT FROM CHANGING. page_views is absent and stays absent --
92# nobody profits from forging traffic and it is the highest-volume writer
93# on the box, so anchoring it would cost a full read of the largest file
94# here every day to protect the least valuable thing in it.
95DEFAULT_COLLECTIONS = (
96 "wallet_entries", # customer money; the highest-value target
97 "fin_journal_lines", # the books an auditor arrives to check
98 "payments",
99 "refunds",
100 "stock_moves", # shrinkage is theft, and a backdated
101 # adjustment is how it gets hidden
102 "notarizations", # the notary's own log, which is otherwise
103 # the one thing here nobody is watching
104)
105
106
107def _base_dir():
108 return os.environ.get("DBBASIC_DATA_DIR", "data")
109
110
111def _text(value):
112 return str(value if value is not None else "").strip()
113
114
115def _settings(base):
116 try:
117 rows = object_records.read_collection_records(SETTINGS_COLLECTION,
118 base_dir=base)
119 except Exception:
120 return {}
121 return {_text(row.get("key")): _text(row.get("value"))
122 for row in rows if _text(row.get("key"))}
123
124
125def _csv_setting(settings, key):
126 """Comma-separated setting into a list, honouring the "not configured"
127 sentinel. object_notary owns the parser so this pass and
128 site_ledger_integrity cannot disagree about what `none` means."""
129 return object_notary.endpoints_from_setting(settings.get(key, ""))
130
131
132def _schema_fields(base, collection):
133 """The field list, in schema order, minus the bookkeeping columns.
134
135 `_op` is the append-storage marker and belongs to the file rather than
136 to the record, so hashing it would tie the digest to a storage detail
137 that compaction rewrites -- exactly what hashing logical rows is meant
138 to avoid.
139 """
140 try:
141 schema = object_schemas.get_schema(collection, base_dir=base)
142 except Exception:
143 return []
144 fields = []
145 for field in (schema or {}).get("fields") or []:
146 name = _text(field.get("name"))
147 if name and name != "_op":
148 fields.append(name)
149 return fields
150
151
152def _newest_anchor(anchors, collection):
153 newest = None
154 for row in anchors:
155 if _text(row.get("collection")) != collection:
156 continue
157 if newest is None or _text(row.get("created_at")) > _text(
158 newest.get("created_at")):
159 newest = row
160 return newest
161
162
163def _lodge(endpoint, digest, label, *, own_base_url=""):
164 """Submit one digest to one notary. Returns (ok, detail).
165
166 Refuses an endpoint that is this server before making any request.
167 Lodging with yourself always succeeds and proves nothing, and the cost
168 of letting it through is not a wasted request -- it is
169 site_ledger_integrity swapping "anchored, but only here" for
170 "verified, and held elsewhere" on the strength of it, which turns an
171 honest missing claim into a specific false one.
172
173 Deliberately narrow: a POST to a known object with a known body, no
174 redirects followed, a timeout, and a bounded read. Every failure mode
175 -- refused, timed out, 4xx, unparseable -- comes back as (False,
176 reason) rather than raising, because one unreachable notary must not
177 stop the pass from lodging with the others or from recording the
178 anchor locally.
179 """
180 allow_loopback = os.environ.get(ALLOW_LOOPBACK_ENV, "").strip().lower() in (
181 "1", "true", "yes", "on")
182 if object_notary.is_self_endpoint(endpoint, own_base_url,
183 allow_loopback=allow_loopback):
184 return False, (f"{endpoint} is this server. An anchor a server holds "
185 f"about itself is not independent, so it is not "
186 f"counted -- point this at a notary somebody else "
187 f"runs.")
188
189 url = endpoint.rstrip("/") + "/objects/action_notarize"
190 body = json.dumps({"digest": digest, "algorithm": object_ledger_head.ALGORITHM,
191 "label": label}).encode("utf-8")
192 request = urllib.request.Request(
193 url, data=body, method="POST",
194 headers={"Content-Type": "application/json",
195 "User-Agent": "dbbasic-object-server/publish-head"})
196 try:
197 with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response:
198 payload = response.read(MAX_RESPONSE_BYTES)
199 except urllib.error.HTTPError as error:
200 return False, f"{endpoint} answered {error.code}"
201 except (urllib.error.URLError, socket.timeout, TimeoutError, OSError) as error:
202 return False, f"{endpoint} unreachable ({error})"
203
204 try:
205 answer = json.loads(payload.decode("utf-8"))
206 except (ValueError, UnicodeDecodeError):
207 return False, f"{endpoint} answered something that was not JSON"
208
209 if answer.get("found") is True:
210 # `already_recorded` is a success, and an interesting one: it means
211 # this exact digest was lodged there before, so the history it
212 # covers has an even older independent timestamp than this pass.
213 return True, ("already held" if answer.get("already_recorded")
214 else "accepted")
215 return False, f"{endpoint} refused: {_text(answer.get('error'))[:120]}"
216
217
218def _anchor_one(base, collection, endpoints, anchors, *, own_base_url=""):
219 fields = _schema_fields(base, collection)
220 if not fields:
221 return {"collection": collection, "status": "skipped",
222 "reason": "no schema on this box"}
223
224 try:
225 rows = object_records.read_collection_records(collection, base_dir=base)
226 except Exception as error:
227 return {"collection": collection, "status": "skipped",
228 "reason": f"unreadable ({error})"}
229
230 folded = object_ledger_head.head(rows, fields, collection=collection)
231
232 previous = _newest_anchor(anchors, collection)
233 if previous is not None and _text(previous.get("digest")) == folded["digest"]:
234 return {"collection": collection, "status": "unchanged",
235 "row_count": folded["row_count"],
236 "reason": "nothing appended since the last anchor"}
237
238 label = f"{collection} head @ {folded['row_count']} rows"
239 lodged, notes = [], []
240 for endpoint in endpoints:
241 ok, detail = _lodge(endpoint, folded["digest"], label,
242 own_base_url=own_base_url)
243 if ok:
244 lodged.append(endpoint)
245 notes.append(f"{endpoint}: {detail}")
246
247 if not endpoints:
248 status = "recorded"
249 elif len(lodged) == len(endpoints):
250 status = "published"
251 elif lodged:
252 status = "partial"
253 else:
254 status = "failed"
255
256 record = {
257 "collection": collection,
258 "row_count": str(folded["row_count"]),
259 "digest": folded["digest"],
260 "algorithm": folded["algorithm"],
261 "scheme": folded["scheme"],
262 "covered_fields": ",".join(folded["fields"]),
263 "notaries": ",".join(lodged),
264 "notary_count": str(len(lodged)),
265 "status": status,
266 "note": "; ".join(notes)[:500],
267 }
268 try:
269 stored = object_records.create_collection_record(
270 ANCHORS, record, base_dir=base, actor=ACTOR)
271 except Exception as error:
272 return {"collection": collection, "status": "error",
273 "reason": f"the anchor could not be written: {error}"}
274
275 return {"collection": collection, "status": status,
276 "row_count": folded["row_count"], "digest": folded["digest"],
277 "notary_count": len(lodged), "anchor_id": stored.get("id")}
278
279
280def EVENT(request):
281 """The daemon entry point.
282
283 Named EVENT with POST aliased below because the change dispatcher and
284 the scheduler call EVENT -- declaring only POST is a pass that silently
285 never runs, which has been a real production bug here twice.
286 """
287 request = request or {}
288 base = _base_dir()
289 settings = _settings(base)
290
291 requested = _csv_setting(settings, COLLECTIONS_KEY) or list(DEFAULT_COLLECTIONS)
292 endpoints = _csv_setting(settings, ENDPOINTS_KEY)
293
294 try:
295 anchors = object_records.read_collection_records(ANCHORS, base_dir=base)
296 except Exception:
297 anchors = []
298
299 own_base_url = settings.get(BASE_URL_KEY, "")
300 results = [_anchor_one(base, collection, endpoints, anchors,
301 own_base_url=own_base_url)
302 for collection in requested]
303
304 written = [r for r in results
305 if r["status"] in ("recorded", "published", "partial", "failed")]
306 independent = sum(r.get("notary_count", 0) for r in written)
307
308 return {
309 "ok": True,
310 "anchored": len(written),
311 "considered": len(results),
312 "endpoints": endpoints,
313 "independent_lodgements": independent,
314 "results": results,
315 "warning": (
316 f"No notary endpoints are configured ({ENDPOINTS_KEY}), so these "
317 f"anchors are held only by the server that took them. Anybody "
318 f"able to rewrite a ledger here can rewrite its anchor beside it. "
319 f"They become evidence when the digest is also held somewhere "
320 f"else." if not endpoints else ""),
321 }
322
323
324POST = EVENT