askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
object_notary.py · 16.4 KB · 381 lines
| 1 | """object_notary -- store a digest, check it later. |
| 2 | |
| 3 | Pure. No I/O, no clock, no data directory. The package in |
| 4 | packages/app-notary calls this; the tests exercise it without either. |
| 5 | |
| 6 | ## What this is for |
| 7 | |
| 8 | Every tamper-evidence scheme on this box eventually runs into the same |
| 9 | wall: a hash chain that lives inside the file it protects, and is only |
| 10 | ever checked by the machine that wrote it, defends against accident and |
| 11 | casual editing but not against anybody who actually wanted to. Recomputing |
| 12 | a chain is a couple of seconds' work. What makes it expensive is recording |
| 13 | the head somewhere the attacker cannot reach. |
| 14 | |
| 15 | This is that somewhere. A notarization is an assertion by an independent |
| 16 | party that a particular digest existed no later than a particular moment. |
| 17 | It is deliberately the dumbest possible thing -- a list of (digest, when), |
| 18 | never edited, held by someone other than the person whose records the |
| 19 | digest describes -- because the security property is INDEPENDENCE, not |
| 20 | cleverness. A dumb store run by a third party is a stronger guarantee than |
| 21 | elaborate cryptography run by the party with the motive. |
| 22 | |
| 23 | ## The three refusals, which are the design |
| 24 | |
| 25 | **It never stores content.** Only digests. A notary that holds your |
| 26 | documents is a breach liability and a subpoena target; one that holds |
| 27 | thirty-two bytes is neither. It is also what lets a submitter prove |
| 28 | something existed without revealing what it was. |
| 29 | |
| 30 | **Nothing is ever amended or removed.** There is no update and no delete, |
| 31 | not as an oversight but as the product: a notary with an edit path is a |
| 32 | notary whose operator can be leaned on. `notarizations` is append storage, |
| 33 | the permission rules grant neither, and no object here writes one twice. |
| 34 | |
| 35 | **It attests EXISTENCE and nothing else.** Not authorship, not ownership, |
| 36 | not meaning, not that the digest is of what the submitter says it is. |
| 37 | `attestation()` below is the exact wording, and every surface renders it, |
| 38 | because the difference between "this digest existed on the 26th of July" |
| 39 | and "Dan owned this file on the 26th of July" is the difference between a |
| 40 | useful service and a misleading one. |
| 41 | |
| 42 | ## Idempotency runs BACKWARDS from the usual rule |
| 43 | |
| 44 | Everywhere else in this building a repeated write is suppressed to avoid a |
| 45 | duplicate (docs/logic-decisions.md #7). Here the same suppression exists |
| 46 | for a different and stronger reason: the claim being made is "this existed |
| 47 | BY then", and a later submission cannot make a thing earlier. So |
| 48 | `first_seen` returns the EARLIEST row for a digest and the surfaces return |
| 49 | that row for every subsequent submission -- resubmitting is not an error |
| 50 | and is not a new record, it is a lookup that happens to be spelled as a |
| 51 | write. Nobody can walk a timestamp forward by submitting again, which is |
| 52 | the only direction an attacker would want to walk it. |
| 53 | |
| 54 | ## What is deliberately NOT here |
| 55 | |
| 56 | **No signature over the receipt.** A signed receipt would let a submitter |
| 57 | prove what the notary said even if the notary later denied it, which is a |
| 58 | real strengthening and needs a key, a rotation story and a published |
| 59 | public key -- see plan/notary-spec.md. Today's answer is that the record |
| 60 | is public, so anybody can check it at any time, and a notary that removed |
| 61 | a row would be visibly missing it. |
| 62 | |
| 63 | **No chain over the notary's own log.** It belongs there and it is not |
| 64 | here yet, because doing it correctly means computing each link under the |
| 65 | same lock that appends the row -- a storage-layer change, not a package |
| 66 | one. Writing it at the package layer would read the tail, then append |
| 67 | without the lock, and two submissions landing together would fork the |
| 68 | chain. A chain that forks under ordinary load is worse than no chain, |
| 69 | because it spends its credibility on false alarms. |
| 70 | |
| 71 | **No Merkle batching.** One published root covering a million submissions |
| 72 | is the right answer at volume and pure overhead below it. |
| 73 | """ |
| 74 | |
| 75 | import hashlib |
| 76 | import urllib.parse |
| 77 | |
| 78 | # Hosts that are always this machine, whatever anybody configured. |
| 79 | LOOPBACK_HOSTS = frozenset({ |
| 80 | "localhost", "127.0.0.1", "0.0.0.0", "::1", "[::1]", "ip6-localhost", |
| 81 | }) |
| 82 | |
| 83 | |
| 84 | def is_self_endpoint(endpoint, own_base_url="", *, allow_loopback=False): |
| 85 | """Is this "independent" notary actually the same server? |
| 86 | |
| 87 | The hole this closes is small to describe and would undo the whole |
| 88 | honesty design: an operator can point `notary.endpoints` at their own |
| 89 | box, every lodgement succeeds, and the integrity page swaps "anchored, |
| 90 | but only here" for "verified, and held elsewhere" -- a reassuring |
| 91 | sentence about a digest that is still held by exactly one party, the |
| 92 | one with the motive. The banner would then be worse than no banner, |
| 93 | because it would be a specific false claim rather than a missing one. |
| 94 | |
| 95 | Detection is BEST EFFORT and cannot be otherwise: a determined |
| 96 | operator can put their own server behind another hostname, and no |
| 97 | check from inside the process can see through that. What it does catch |
| 98 | is the two cases that happen by accident and by well-meaning |
| 99 | misunderstanding -- a loopback address, and the server's own |
| 100 | configured public URL. Those are the ways somebody self-anchors while |
| 101 | believing they have done the right thing, which is the population this |
| 102 | guard exists for. |
| 103 | |
| 104 | `allow_loopback` exists for tests and local development, where a |
| 105 | notary on 127.0.0.1 is the only notary there is. Callers gate it on an |
| 106 | ENVIRONMENT VARIABLE rather than a setting, deliberately: an operator |
| 107 | clicking through a settings page must not be able to switch off the |
| 108 | check that stops them lying to themselves, while somebody running the |
| 109 | process for a test obviously can. Note that a second process on the |
| 110 | same machine is not independent in any sense that matters -- same |
| 111 | disk, same root, same backup -- so this really is a development |
| 112 | affordance and not a supported deployment. |
| 113 | """ |
| 114 | host = _host_of(endpoint) |
| 115 | if not host: |
| 116 | return False |
| 117 | if host in LOOPBACK_HOSTS or host.startswith("127."): |
| 118 | return not allow_loopback |
| 119 | own = _host_of(own_base_url) |
| 120 | return bool(own) and host == own |
| 121 | |
| 122 | |
| 123 | # app_settings requires a non-empty value, so "not configured" needs a |
| 124 | # word. `none` is already this codebase's sentinel for it -- |
| 125 | # carrier.provider reads "", "none" and "manual" the same way -- and an |
| 126 | # operator who writes it to mean nothing must not get a daily failed |
| 127 | # lodgement against a host literally called `none`, which is a false alarm |
| 128 | # indistinguishable from a real notary being down. |
| 129 | NOT_CONFIGURED = frozenset({"none", "off", "disabled", "-"}) |
| 130 | |
| 131 | |
| 132 | def endpoints_from_setting(value): |
| 133 | """Parse `notary.endpoints` into a list. THE one parser. |
| 134 | |
| 135 | Lives here rather than in either object because both the pass that |
| 136 | lodges digests and the page that reports on them must agree about what |
| 137 | "none" means, and they cannot import each other. When they disagreed |
| 138 | -- briefly, on the live box -- the page reported an endpoint that the |
| 139 | pass was correctly ignoring, which is a small lie in the one place |
| 140 | whose whole job is to be believed. |
| 141 | """ |
| 142 | return [part.strip() for part in str(value or "").split(",") |
| 143 | if part.strip() and part.strip().lower() not in NOT_CONFIGURED] |
| 144 | |
| 145 | |
| 146 | def _host_of(url): |
| 147 | text = str(url if url is not None else "").strip().lower() |
| 148 | if not text: |
| 149 | return "" |
| 150 | if "//" not in text: |
| 151 | text = "//" + text |
| 152 | try: |
| 153 | return (urllib.parse.urlsplit(text).hostname or "").strip() |
| 154 | except ValueError: |
| 155 | return "" |
| 156 | |
| 157 | |
| 158 | # Digest algorithms this notary will accept, and the hex length each one |
| 159 | # produces. Restricted rather than open: accepting an arbitrary algorithm |
| 160 | # name means storing a claim nobody can check, and accepting a short one |
| 161 | # (md5, sha1) means storing a claim somebody can forge a collision against |
| 162 | # -- which is precisely the attack a notary exists to prevent. |
| 163 | ALGORITHMS = { |
| 164 | "sha256": 64, |
| 165 | "sha512": 128, |
| 166 | } |
| 167 | |
| 168 | DEFAULT_ALGORITHM = "sha256" |
| 169 | |
| 170 | # Labels are the submitter's own note about what a digest was -- "payroll |
| 171 | # ledger head, 26 July", "contract with Acme, signed copy". Free text, |
| 172 | # capped, and NEVER required: a submitter who does not want to say what a |
| 173 | # digest is has an excellent reason not to, and the whole point of storing |
| 174 | # only a hash is that they do not have to. |
| 175 | MAX_LABEL = 200 |
| 176 | |
| 177 | _HEX = set("0123456789abcdef") |
| 178 | |
| 179 | |
| 180 | def normalize_algorithm(value): |
| 181 | """The algorithm name, lower-cased and trimmed, or "" if unsupported.""" |
| 182 | name = str(value if value is not None else "").strip().lower() |
| 183 | return name if name in ALGORITHMS else "" |
| 184 | |
| 185 | |
| 186 | def normalize_digest(value, algorithm=DEFAULT_ALGORITHM): |
| 187 | """A digest in canonical form, or "" if it is not one. |
| 188 | |
| 189 | Lower-cased, trimmed, and checked against the exact hex length the |
| 190 | named algorithm produces. Case is normalised rather than rejected |
| 191 | because a hex digest is the same value in either case and losing a |
| 192 | lookup to a capital letter is not a policy anybody chose -- the same |
| 193 | reasoning object_promotions applies to a code typed off a postcard. |
| 194 | |
| 195 | A leading "0x" or a "sha256:" prefix is stripped, because both are |
| 196 | ordinary ways for a tool to hand one over and neither changes the |
| 197 | value. |
| 198 | """ |
| 199 | text = str(value if value is not None else "").strip().lower() |
| 200 | for prefix in ("0x", "sha256:", "sha512:"): |
| 201 | if text.startswith(prefix): |
| 202 | text = text[len(prefix):] |
| 203 | text = text.strip() |
| 204 | expected = ALGORITHMS.get(normalize_algorithm(algorithm) or algorithm) |
| 205 | if expected is None: |
| 206 | return "" |
| 207 | if len(text) != expected: |
| 208 | return "" |
| 209 | if not text or not set(text) <= _HEX: |
| 210 | return "" |
| 211 | return text |
| 212 | |
| 213 | |
| 214 | def normalize_label(value): |
| 215 | """The submitter's note, trimmed to MAX_LABEL, tabs and newlines gone. |
| 216 | |
| 217 | Tabs would break the TSV row this becomes and newlines would break the |
| 218 | line, so they are collapsed to spaces rather than rejected: a label is |
| 219 | a courtesy field and refusing a submission over whitespace would lose |
| 220 | a notarization to a formatting detail. |
| 221 | """ |
| 222 | text = str(value if value is not None else "") |
| 223 | text = text.replace("\t", " ").replace("\r", " ").replace("\n", " ") |
| 224 | return " ".join(text.split())[:MAX_LABEL] |
| 225 | |
| 226 | |
| 227 | def digest_of(data, algorithm=DEFAULT_ALGORITHM): |
| 228 | """The digest of some bytes, for callers that hold the content. |
| 229 | |
| 230 | Present so a caller anchoring its own ledger head does not reach for |
| 231 | hashlib and pick a different algorithm by accident. Note that a |
| 232 | submitter with anything confidential should hash it themselves and |
| 233 | send only the result -- this function existing does not mean content |
| 234 | should travel to the notary, and it never does from any surface here. |
| 235 | """ |
| 236 | name = normalize_algorithm(algorithm) |
| 237 | if not name: |
| 238 | raise ValueError(f"Unsupported digest algorithm: {algorithm!r}") |
| 239 | if isinstance(data, str): |
| 240 | data = data.encode("utf-8") |
| 241 | return hashlib.new(name, data).hexdigest() |
| 242 | |
| 243 | |
| 244 | def problems(digest, algorithm=DEFAULT_ALGORITHM): |
| 245 | """Every reason this submission cannot be accepted, at once. |
| 246 | |
| 247 | A list rather than a first failure, and each entry names the actual |
| 248 | constraint -- "a sha256 digest is 64 hex characters; that one is 40" |
| 249 | rather than "invalid digest", because the commonest cause of the |
| 250 | second sentence is somebody having hashed with sha1 and having no way |
| 251 | to find that out from the refusal. |
| 252 | """ |
| 253 | found = [] |
| 254 | name = normalize_algorithm(algorithm) |
| 255 | if not name: |
| 256 | supported = ", ".join(sorted(ALGORITHMS)) |
| 257 | found.append( |
| 258 | f"'{str(algorithm).strip()}' is not an algorithm this notary " |
| 259 | f"accepts. Supported: {supported}. Shorter digests (md5, sha1) " |
| 260 | f"are refused rather than unsupported: a notary whose digests " |
| 261 | f"can be collided proves nothing.") |
| 262 | return found |
| 263 | |
| 264 | raw = str(digest if digest is not None else "").strip() |
| 265 | if not raw: |
| 266 | found.append("A digest is required. Hash the thing you want to " |
| 267 | "notarize and send the hex digest -- never the thing " |
| 268 | "itself.") |
| 269 | return found |
| 270 | |
| 271 | if not normalize_digest(raw, name): |
| 272 | expected = ALGORITHMS[name] |
| 273 | cleaned = raw.lower() |
| 274 | for prefix in ("0x", "sha256:", "sha512:"): |
| 275 | if cleaned.startswith(prefix): |
| 276 | cleaned = cleaned[len(prefix):] |
| 277 | if set(cleaned) - _HEX: |
| 278 | found.append(f"That is not hex. A {name} digest is {expected} " |
| 279 | f"characters of 0-9 and a-f.") |
| 280 | else: |
| 281 | found.append(f"A {name} digest is {expected} hex characters; " |
| 282 | f"that one is {len(cleaned)}.") |
| 283 | return found |
| 284 | |
| 285 | |
| 286 | def first_seen(digest, rows, algorithm=DEFAULT_ALGORITHM): |
| 287 | """The EARLIEST notarization of this digest, or None. |
| 288 | |
| 289 | Earliest, not latest, and this is the whole idempotency rule: the |
| 290 | claim a notarization makes is "this existed by then", so a second |
| 291 | submission cannot improve on the first and must not be allowed to |
| 292 | replace it. Anybody able to walk their own timestamp forward by |
| 293 | resubmitting could quietly discard the only fact the record carries. |
| 294 | |
| 295 | Ties on created_at resolve to the row that appears first in the log, |
| 296 | which in an append collection is the row that was written first. |
| 297 | """ |
| 298 | target = normalize_digest(digest, algorithm) |
| 299 | if not target: |
| 300 | return None |
| 301 | name = normalize_algorithm(algorithm) or DEFAULT_ALGORITHM |
| 302 | best = None |
| 303 | best_stamp = None |
| 304 | for row in rows or (): |
| 305 | if str(row.get("digest") or "").strip().lower() != target: |
| 306 | continue |
| 307 | row_algorithm = (str(row.get("algorithm") or "").strip().lower() |
| 308 | or DEFAULT_ALGORITHM) |
| 309 | if row_algorithm != name: |
| 310 | continue |
| 311 | stamp = str(row.get("created_at") or "") |
| 312 | if best is None or (stamp and best_stamp and stamp < best_stamp): |
| 313 | best, best_stamp = row, stamp |
| 314 | return best |
| 315 | |
| 316 | |
| 317 | def attestation(row=None): |
| 318 | """Exactly what a notarization claims, and exactly what it does not. |
| 319 | |
| 320 | One definition, rendered by every surface -- the submission response, |
| 321 | the check page and the JSON -- so they cannot drift into three |
| 322 | different strengths of the same promise. Overstating this is the only |
| 323 | way a service this simple can do harm: a reader who takes "notarized" |
| 324 | to mean "verified", "owned" or "approved" has been misled by the word |
| 325 | rather than by anything the record says. |
| 326 | """ |
| 327 | when = str((row or {}).get("created_at") or "") |
| 328 | subject = f"was recorded here at {when}" if when else "is recorded here" |
| 329 | return { |
| 330 | "proves": [ |
| 331 | f"This exact digest {subject}, and therefore the data it was " |
| 332 | f"computed from existed no later than that moment.", |
| 333 | "The record is public and append-only: it can be checked by " |
| 334 | "anyone, at any time, without an account.", |
| 335 | ], |
| 336 | "does_not_prove": [ |
| 337 | "WHO created the data. Anybody can notarize anybody's digest.", |
| 338 | "That the submitter owned it, wrote it, or had any right to it.", |
| 339 | "That the data did not also exist much earlier. This is an " |
| 340 | "upper bound on its age and nothing more.", |
| 341 | "Anything about the CONTENT. This notary never saw it and " |
| 342 | "cannot say what the digest is of.", |
| 343 | ], |
| 344 | "rests_on": [ |
| 345 | "This notary's clock being right and its operator being honest. " |
| 346 | "Trusting one notary is trusting one party -- which is a large " |
| 347 | "improvement on trusting yourself, and is not the end of the " |
| 348 | "argument. Anchor a digest with several independent notaries " |
| 349 | "and forging it means compromising all of them.", |
| 350 | ], |
| 351 | } |
| 352 | |
| 353 | |
| 354 | def receipt(row, *, found=True): |
| 355 | """The submission and lookup answer, in one shape. |
| 356 | |
| 357 | Both surfaces return this so a submitter's receipt and a checker's |
| 358 | answer are the same object -- if they could differ, the interesting |
| 359 | case is the one where they do, and nobody would be watching it. |
| 360 | """ |
| 361 | if not found or not row: |
| 362 | return { |
| 363 | "found": False, |
| 364 | "attestation": { |
| 365 | "proves": [], |
| 366 | "does_not_prove": [ |
| 367 | "That the data did not exist. A digest that was never " |
| 368 | "submitted is simply unknown here; absence from this " |
| 369 | "log is not evidence of anything.", |
| 370 | ], |
| 371 | "rests_on": [], |
| 372 | }, |
| 373 | } |
| 374 | return { |
| 375 | "found": True, |
| 376 | "digest": str(row.get("digest") or ""), |
| 377 | "algorithm": str(row.get("algorithm") or DEFAULT_ALGORITHM), |
| 378 | "first_seen_at": str(row.get("created_at") or ""), |
| 379 | "label": str(row.get("label") or ""), |
| 380 | "attestation": attestation(row), |
| 381 | } |