Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

dbbasic-object-server/object_notary.py
Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
object_notary.py · 16.4 KB · 381 lines
1"""object_notary -- store a digest, check it later.
2
3Pure. No I/O, no clock, no data directory. The package in
4packages/app-notary calls this; the tests exercise it without either.
5
6## What this is for
7
8Every tamper-evidence scheme on this box eventually runs into the same
9wall: a hash chain that lives inside the file it protects, and is only
10ever checked by the machine that wrote it, defends against accident and
11casual editing but not against anybody who actually wanted to. Recomputing
12a chain is a couple of seconds' work. What makes it expensive is recording
13the head somewhere the attacker cannot reach.
14
15This is that somewhere. A notarization is an assertion by an independent
16party that a particular digest existed no later than a particular moment.
17It is deliberately the dumbest possible thing -- a list of (digest, when),
18never edited, held by someone other than the person whose records the
19digest describes -- because the security property is INDEPENDENCE, not
20cleverness. A dumb store run by a third party is a stronger guarantee than
21elaborate cryptography run by the party with the motive.
22
23## The three refusals, which are the design
24
25**It never stores content.** Only digests. A notary that holds your
26documents is a breach liability and a subpoena target; one that holds
27thirty-two bytes is neither. It is also what lets a submitter prove
28something existed without revealing what it was.
29
30**Nothing is ever amended or removed.** There is no update and no delete,
31not as an oversight but as the product: a notary with an edit path is a
32notary whose operator can be leaned on. `notarizations` is append storage,
33the permission rules grant neither, and no object here writes one twice.
34
35**It attests EXISTENCE and nothing else.** Not authorship, not ownership,
36not meaning, not that the digest is of what the submitter says it is.
37`attestation()` below is the exact wording, and every surface renders it,
38because the difference between "this digest existed on the 26th of July"
39and "Dan owned this file on the 26th of July" is the difference between a
40useful service and a misleading one.
41
42## Idempotency runs BACKWARDS from the usual rule
43
44Everywhere else in this building a repeated write is suppressed to avoid a
45duplicate (docs/logic-decisions.md #7). Here the same suppression exists
46for a different and stronger reason: the claim being made is "this existed
47BY then", and a later submission cannot make a thing earlier. So
48`first_seen` returns the EARLIEST row for a digest and the surfaces return
49that row for every subsequent submission -- resubmitting is not an error
50and is not a new record, it is a lookup that happens to be spelled as a
51write. Nobody can walk a timestamp forward by submitting again, which is
52the only direction an attacker would want to walk it.
53
54## What is deliberately NOT here
55
56**No signature over the receipt.** A signed receipt would let a submitter
57prove what the notary said even if the notary later denied it, which is a
58real strengthening and needs a key, a rotation story and a published
59public key -- see plan/notary-spec.md. Today's answer is that the record
60is public, so anybody can check it at any time, and a notary that removed
61a row would be visibly missing it.
62
63**No chain over the notary's own log.** It belongs there and it is not
64here yet, because doing it correctly means computing each link under the
65same lock that appends the row -- a storage-layer change, not a package
66one. Writing it at the package layer would read the tail, then append
67without the lock, and two submissions landing together would fork the
68chain. A chain that forks under ordinary load is worse than no chain,
69because it spends its credibility on false alarms.
70
71**No Merkle batching.** One published root covering a million submissions
72is the right answer at volume and pure overhead below it.
73"""
74
75import hashlib
76import urllib.parse
77
78# Hosts that are always this machine, whatever anybody configured.
79LOOPBACK_HOSTS = frozenset({
80 "localhost", "127.0.0.1", "0.0.0.0", "::1", "[::1]", "ip6-localhost",
81})
82
83
84def is_self_endpoint(endpoint, own_base_url="", *, allow_loopback=False):
85 """Is this "independent" notary actually the same server?
86
87 The hole this closes is small to describe and would undo the whole
88 honesty design: an operator can point `notary.endpoints` at their own
89 box, every lodgement succeeds, and the integrity page swaps "anchored,
90 but only here" for "verified, and held elsewhere" -- a reassuring
91 sentence about a digest that is still held by exactly one party, the
92 one with the motive. The banner would then be worse than no banner,
93 because it would be a specific false claim rather than a missing one.
94
95 Detection is BEST EFFORT and cannot be otherwise: a determined
96 operator can put their own server behind another hostname, and no
97 check from inside the process can see through that. What it does catch
98 is the two cases that happen by accident and by well-meaning
99 misunderstanding -- a loopback address, and the server's own
100 configured public URL. Those are the ways somebody self-anchors while
101 believing they have done the right thing, which is the population this
102 guard exists for.
103
104 `allow_loopback` exists for tests and local development, where a
105 notary on 127.0.0.1 is the only notary there is. Callers gate it on an
106 ENVIRONMENT VARIABLE rather than a setting, deliberately: an operator
107 clicking through a settings page must not be able to switch off the
108 check that stops them lying to themselves, while somebody running the
109 process for a test obviously can. Note that a second process on the
110 same machine is not independent in any sense that matters -- same
111 disk, same root, same backup -- so this really is a development
112 affordance and not a supported deployment.
113 """
114 host = _host_of(endpoint)
115 if not host:
116 return False
117 if host in LOOPBACK_HOSTS or host.startswith("127."):
118 return not allow_loopback
119 own = _host_of(own_base_url)
120 return bool(own) and host == own
121
122
123# app_settings requires a non-empty value, so "not configured" needs a
124# word. `none` is already this codebase's sentinel for it --
125# carrier.provider reads "", "none" and "manual" the same way -- and an
126# operator who writes it to mean nothing must not get a daily failed
127# lodgement against a host literally called `none`, which is a false alarm
128# indistinguishable from a real notary being down.
129NOT_CONFIGURED = frozenset({"none", "off", "disabled", "-"})
130
131
132def endpoints_from_setting(value):
133 """Parse `notary.endpoints` into a list. THE one parser.
134
135 Lives here rather than in either object because both the pass that
136 lodges digests and the page that reports on them must agree about what
137 "none" means, and they cannot import each other. When they disagreed
138 -- briefly, on the live box -- the page reported an endpoint that the
139 pass was correctly ignoring, which is a small lie in the one place
140 whose whole job is to be believed.
141 """
142 return [part.strip() for part in str(value or "").split(",")
143 if part.strip() and part.strip().lower() not in NOT_CONFIGURED]
144
145
146def _host_of(url):
147 text = str(url if url is not None else "").strip().lower()
148 if not text:
149 return ""
150 if "//" not in text:
151 text = "//" + text
152 try:
153 return (urllib.parse.urlsplit(text).hostname or "").strip()
154 except ValueError:
155 return ""
156
157
158# Digest algorithms this notary will accept, and the hex length each one
159# produces. Restricted rather than open: accepting an arbitrary algorithm
160# name means storing a claim nobody can check, and accepting a short one
161# (md5, sha1) means storing a claim somebody can forge a collision against
162# -- which is precisely the attack a notary exists to prevent.
163ALGORITHMS = {
164 "sha256": 64,
165 "sha512": 128,
166}
167
168DEFAULT_ALGORITHM = "sha256"
169
170# Labels are the submitter's own note about what a digest was -- "payroll
171# ledger head, 26 July", "contract with Acme, signed copy". Free text,
172# capped, and NEVER required: a submitter who does not want to say what a
173# digest is has an excellent reason not to, and the whole point of storing
174# only a hash is that they do not have to.
175MAX_LABEL = 200
176
177_HEX = set("0123456789abcdef")
178
179
180def normalize_algorithm(value):
181 """The algorithm name, lower-cased and trimmed, or "" if unsupported."""
182 name = str(value if value is not None else "").strip().lower()
183 return name if name in ALGORITHMS else ""
184
185
186def normalize_digest(value, algorithm=DEFAULT_ALGORITHM):
187 """A digest in canonical form, or "" if it is not one.
188
189 Lower-cased, trimmed, and checked against the exact hex length the
190 named algorithm produces. Case is normalised rather than rejected
191 because a hex digest is the same value in either case and losing a
192 lookup to a capital letter is not a policy anybody chose -- the same
193 reasoning object_promotions applies to a code typed off a postcard.
194
195 A leading "0x" or a "sha256:" prefix is stripped, because both are
196 ordinary ways for a tool to hand one over and neither changes the
197 value.
198 """
199 text = str(value if value is not None else "").strip().lower()
200 for prefix in ("0x", "sha256:", "sha512:"):
201 if text.startswith(prefix):
202 text = text[len(prefix):]
203 text = text.strip()
204 expected = ALGORITHMS.get(normalize_algorithm(algorithm) or algorithm)
205 if expected is None:
206 return ""
207 if len(text) != expected:
208 return ""
209 if not text or not set(text) <= _HEX:
210 return ""
211 return text
212
213
214def normalize_label(value):
215 """The submitter's note, trimmed to MAX_LABEL, tabs and newlines gone.
216
217 Tabs would break the TSV row this becomes and newlines would break the
218 line, so they are collapsed to spaces rather than rejected: a label is
219 a courtesy field and refusing a submission over whitespace would lose
220 a notarization to a formatting detail.
221 """
222 text = str(value if value is not None else "")
223 text = text.replace("\t", " ").replace("\r", " ").replace("\n", " ")
224 return " ".join(text.split())[:MAX_LABEL]
225
226
227def digest_of(data, algorithm=DEFAULT_ALGORITHM):
228 """The digest of some bytes, for callers that hold the content.
229
230 Present so a caller anchoring its own ledger head does not reach for
231 hashlib and pick a different algorithm by accident. Note that a
232 submitter with anything confidential should hash it themselves and
233 send only the result -- this function existing does not mean content
234 should travel to the notary, and it never does from any surface here.
235 """
236 name = normalize_algorithm(algorithm)
237 if not name:
238 raise ValueError(f"Unsupported digest algorithm: {algorithm!r}")
239 if isinstance(data, str):
240 data = data.encode("utf-8")
241 return hashlib.new(name, data).hexdigest()
242
243
244def problems(digest, algorithm=DEFAULT_ALGORITHM):
245 """Every reason this submission cannot be accepted, at once.
246
247 A list rather than a first failure, and each entry names the actual
248 constraint -- "a sha256 digest is 64 hex characters; that one is 40"
249 rather than "invalid digest", because the commonest cause of the
250 second sentence is somebody having hashed with sha1 and having no way
251 to find that out from the refusal.
252 """
253 found = []
254 name = normalize_algorithm(algorithm)
255 if not name:
256 supported = ", ".join(sorted(ALGORITHMS))
257 found.append(
258 f"'{str(algorithm).strip()}' is not an algorithm this notary "
259 f"accepts. Supported: {supported}. Shorter digests (md5, sha1) "
260 f"are refused rather than unsupported: a notary whose digests "
261 f"can be collided proves nothing.")
262 return found
263
264 raw = str(digest if digest is not None else "").strip()
265 if not raw:
266 found.append("A digest is required. Hash the thing you want to "
267 "notarize and send the hex digest -- never the thing "
268 "itself.")
269 return found
270
271 if not normalize_digest(raw, name):
272 expected = ALGORITHMS[name]
273 cleaned = raw.lower()
274 for prefix in ("0x", "sha256:", "sha512:"):
275 if cleaned.startswith(prefix):
276 cleaned = cleaned[len(prefix):]
277 if set(cleaned) - _HEX:
278 found.append(f"That is not hex. A {name} digest is {expected} "
279 f"characters of 0-9 and a-f.")
280 else:
281 found.append(f"A {name} digest is {expected} hex characters; "
282 f"that one is {len(cleaned)}.")
283 return found
284
285
286def first_seen(digest, rows, algorithm=DEFAULT_ALGORITHM):
287 """The EARLIEST notarization of this digest, or None.
288
289 Earliest, not latest, and this is the whole idempotency rule: the
290 claim a notarization makes is "this existed by then", so a second
291 submission cannot improve on the first and must not be allowed to
292 replace it. Anybody able to walk their own timestamp forward by
293 resubmitting could quietly discard the only fact the record carries.
294
295 Ties on created_at resolve to the row that appears first in the log,
296 which in an append collection is the row that was written first.
297 """
298 target = normalize_digest(digest, algorithm)
299 if not target:
300 return None
301 name = normalize_algorithm(algorithm) or DEFAULT_ALGORITHM
302 best = None
303 best_stamp = None
304 for row in rows or ():
305 if str(row.get("digest") or "").strip().lower() != target:
306 continue
307 row_algorithm = (str(row.get("algorithm") or "").strip().lower()
308 or DEFAULT_ALGORITHM)
309 if row_algorithm != name:
310 continue
311 stamp = str(row.get("created_at") or "")
312 if best is None or (stamp and best_stamp and stamp < best_stamp):
313 best, best_stamp = row, stamp
314 return best
315
316
317def attestation(row=None):
318 """Exactly what a notarization claims, and exactly what it does not.
319
320 One definition, rendered by every surface -- the submission response,
321 the check page and the JSON -- so they cannot drift into three
322 different strengths of the same promise. Overstating this is the only
323 way a service this simple can do harm: a reader who takes "notarized"
324 to mean "verified", "owned" or "approved" has been misled by the word
325 rather than by anything the record says.
326 """
327 when = str((row or {}).get("created_at") or "")
328 subject = f"was recorded here at {when}" if when else "is recorded here"
329 return {
330 "proves": [
331 f"This exact digest {subject}, and therefore the data it was "
332 f"computed from existed no later than that moment.",
333 "The record is public and append-only: it can be checked by "
334 "anyone, at any time, without an account.",
335 ],
336 "does_not_prove": [
337 "WHO created the data. Anybody can notarize anybody's digest.",
338 "That the submitter owned it, wrote it, or had any right to it.",
339 "That the data did not also exist much earlier. This is an "
340 "upper bound on its age and nothing more.",
341 "Anything about the CONTENT. This notary never saw it and "
342 "cannot say what the digest is of.",
343 ],
344 "rests_on": [
345 "This notary's clock being right and its operator being honest. "
346 "Trusting one notary is trusting one party -- which is a large "
347 "improvement on trusting yourself, and is not the end of the "
348 "argument. Anchor a digest with several independent notaries "
349 "and forging it means compromising all of them.",
350 ],
351 }
352
353
354def receipt(row, *, found=True):
355 """The submission and lookup answer, in one shape.
356
357 Both surfaces return this so a submitter's receipt and a checker's
358 answer are the same object -- if they could differ, the interesting
359 case is the one where they do, and nobody would be watching it.
360 """
361 if not found or not row:
362 return {
363 "found": False,
364 "attestation": {
365 "proves": [],
366 "does_not_prove": [
367 "That the data did not exist. A digest that was never "
368 "submitted is simply unknown here; absence from this "
369 "log is not evidence of anything.",
370 ],
371 "rests_on": [],
372 },
373 }
374 return {
375 "found": True,
376 "digest": str(row.get("digest") or ""),
377 "algorithm": str(row.get("algorithm") or DEFAULT_ALGORITHM),
378 "first_seen_at": str(row.get("created_at") or ""),
379 "label": str(row.get("label") or ""),
380 "attestation": attestation(row),
381 }