askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
test_durability_concurrency_characterization.py · 31.1 KB · 698 lines
| 1 | """CHARACTERIZATION tests: does object_records.py's write path stay correct |
| 2 | under CONCURRENT writers/readers, and what is its actual concurrency-safety |
| 3 | boundary? |
| 4 | |
| 5 | Pure characterization of EXISTING behavior. Nothing in object_records.py (or |
| 6 | any other production module) is modified by this file. Where a probe finds a |
| 7 | real gap, the assertion documents the correctness property a database-shaped |
| 8 | substrate SHOULD have and is left in place -- do not "fix" a finding here by |
| 9 | weakening the assertion to match the gap. |
| 10 | |
| 11 | THE LOCK, READ FROM SOURCE FIRST (then verified empirically below): |
| 12 | `_records_file_lock` (object_records.py, ~line 2703) is NOT a |
| 13 | `threading.Lock` keyed by path (that would be a plausible, reasonable |
| 14 | mechanism, and is what this file's brief assumed going in) -- it is an |
| 15 | `fcntl.flock(LOCK_EX)` advisory lock held on a sidecar `.records.tsv.lock` |
| 16 | file, acquired fresh (a new `open("a")` file descriptor) on every |
| 17 | create/update/delete/compact call and released when the `with` block exits. |
| 18 | That distinction matters a great deal for this file's central question: |
| 19 | |
| 20 | - flock is a KERNEL-level lock associated with the *open file description* |
| 21 | (not the process, and not the Python-level lock object), so it |
| 22 | correctly serializes BOTH concurrent threads in the same process |
| 23 | (each thread's own `with lock_path.open("a")` gets its own fd/open |
| 24 | file description, and flock blocks across distinct descriptions even |
| 25 | within one process) AND concurrent separate OS processes on the same |
| 26 | machine sharing the same filesystem path -- there is no dependency on |
| 27 | both callers being the same Python interpreter, unlike a |
| 28 | `threading.Lock`, which cannot be observed at all outside its own |
| 29 | process. |
| 30 | - It is advisory, not mandatory: it only serializes writers that |
| 31 | themselves call `_records_file_lock` (i.e. every writer inside this |
| 32 | module). Nothing stops a process that opens records.tsv directly and |
| 33 | writes to it without going through object_records.py. |
| 34 | - Classic-mode full rewrites additionally write via a tempfile + |
| 35 | `Path.replace()` (atomic rename) rather than in-place, so even a READER |
| 36 | that holds no lock at all (readers never take `_records_file_lock` -- |
| 37 | see the CONCURRENCY block comment above _RECORDS_CACHE) always sees a |
| 38 | complete pre- or post-write file, never a torn one -- the lock's job in |
| 39 | that mode is only to serialize writers' read-modify-write cycles |
| 40 | against EACH OTHER, not against readers. |
| 41 | - Append-mode's fast path (`_append_records_rows`) is a genuine in-place |
| 42 | mutation (`open(path, "a")` + `csv.writer.writerow` + flush) -- there is |
| 43 | no atomic-rename safety net for this path; the flock is the ONLY thing |
| 44 | preventing two writers' single `write()` calls from interleaving on |
| 45 | disk. This is exactly the case this file's cross-process probes target. |
| 46 | |
| 47 | So the headline question this file exists to answer empirically is not |
| 48 | "is the lock cross-thread only" (source review already answers that: no, |
| 49 | flock is not thread-scoped) but "does the flock-based mechanism actually |
| 50 | hold up under real concurrent contention, in both storage modes, across |
| 51 | both threads and separate OS processes" -- and separately, to characterize |
| 52 | honestly what current same-record concurrent UPDATE semantics are (last- |
| 53 | writer-wins; optimistic concurrency / compare-and-set is spec 63, not yet |
| 54 | built, so nothing here detects or rejects a lost concurrent update). |
| 55 | |
| 56 | FINDINGS (all 13 tests below PASS, consistently across repeated runs; this |
| 57 | summary is the worst-first rollup -- worst first meaning "most surprising |
| 58 | if you assumed the naive per-process-only-lock model", not "most broken": |
| 59 | nothing here is broken): |
| 60 | |
| 61 | 1. Cross-process DISTINCT-record writes, classic AND append mode: SAFE. |
| 62 | 4 separate OS processes (subprocess, not threads -- fresh interpreters, |
| 63 | cold caches, sharing nothing but the files on disk), barrier-released |
| 64 | to overlap in time, each creating 20 distinct records into the same |
| 65 | collection: all 80 records survive, none garbled, in both modes. This |
| 66 | is the headline result and it contradicts the naive assumption a |
| 67 | `threading.Lock` mechanism would predict (that would provide ZERO |
| 68 | cross-process exclusion): `_records_file_lock` is actually |
| 69 | `fcntl.flock`, a kernel-level advisory lock keyed by the lock file's |
| 70 | path, not by the calling interpreter -- so it DOES serialize |
| 71 | independent processes, not just threads within one process. |
| 72 | 2. Cross-process SAME-record concurrent UPDATE: SAFE from corruption, |
| 73 | last-writer-wins as expected. 4 processes updating one record |
| 74 | concurrently always leave it holding exactly one clean submitted |
| 75 | value, never a merge/garble -- confirming the lock's cross-process |
| 76 | exclusion holds for the read-modify-write update path too, not just |
| 77 | the append-only create path. |
| 78 | 3. Same-process (thread) DISTINCT-record writes, classic AND append, |
| 79 | N=8 and N=32: SAFE. No lost or duplicated writes, no garbled rows. |
| 80 | 4. Same-process (thread) SAME-record concurrent UPDATE, both modes: |
| 81 | SAFE from corruption, last-writer-wins. 16 threads updating one |
| 82 | record concurrently always land on exactly one submitted value. This |
| 83 | is the CURRENT concurrent-update guarantee, and it is a real gap: |
| 84 | 15 of 16 updates are silently discarded from live state with no error, |
| 85 | no conflict signal, and no way for a caller to know its update was |
| 86 | clobbered -- this is exactly the hole optimistic concurrency / |
| 87 | compare-and-set (spec 63) is meant to fill. Notably, the discarded |
| 88 | updates are NOT lost from the durable audit trail: object_record_ |
| 89 | changes' append-only JSONL log recorded all 16 update events (with |
| 90 | all 16 distinct submitted values) even though get_collection_record |
| 91 | shows only the winner -- so "last-writer-wins" is a live-state |
| 92 | property, not a durability/logging one. |
| 93 | 5. Concurrent readers alongside a sustained writer, both modes: SAFE. |
| 94 | Readers hold no lock at all (by design) and never raised, never saw a |
| 95 | garbled row, across a writer creating 150 records while 4 reader |
| 96 | threads hammered read_collection_records/get_collection_record/ |
| 97 | list_collection_records concurrently. A reader may transiently see a |
| 98 | record count that lags the writer (expected, unasserted) but the file |
| 99 | is never observed torn -- classic mode's atomic tempfile+replace and |
| 100 | append mode's flush-after-write both hold up under concurrent reads. |
| 101 | 6. id->offset oidx sidecar under concurrent same-process appends: SAFE. |
| 102 | After 24 threads concurrently create distinct records (forced through |
| 103 | the cold-cache sidecar path via DBBASIC_RECORDS_CACHE_MAX_ROWS=0), |
| 104 | every id resolves to its correct value via the offset-indexed by-id |
| 105 | read, agreeing with a full fold -- the sidecar's documented |
| 106 | disposability/self-heal ("any reader unable to make sense of it... |
| 107 | rebuilds it") held up in practice under contention. |
| 108 | |
| 109 | BOTTOM LINE: the lock is not a threading.Lock and not scoped to one |
| 110 | process -- it is a real fcntl.flock advisory file lock, and it is |
| 111 | effective across both threads AND separate OS processes (verified via |
| 112 | actual subprocesses, not simulated). For a multi-worker uvicorn/gunicorn |
| 113 | deployment (multiple OS processes sharing the same data directory), this |
| 114 | means concurrent writes are NOT the unsafe case some deployments would |
| 115 | fear -- corruption/lost-write risk was not observed in any probe here. The |
| 116 | one real, honest gap is same-record concurrent UPDATE semantics: pure |
| 117 | last-writer-wins, with silent overwrite of losing updates and no conflict |
| 118 | detection -- present identically whether the concurrent writers are threads |
| 119 | in one process or separate processes, and unrelated to the lock's |
| 120 | effectiveness. That gap is spec 63's to close, not a locking bug. |
| 121 | |
| 122 | CAVEATS (not probed by, or outside the scope of, this file): |
| 123 | - flock is advisory: a process that opens records.tsv directly without |
| 124 | going through object_records.py's write path is not blocked by it. |
| 125 | - flock's cross-process guarantee assumes a shared local filesystem with |
| 126 | working flock semantics; it is well-known to be unreliable-to-absent |
| 127 | over NFS and some network/container filesystem configurations, which |
| 128 | were not exercised here (this ran against a local disk). |
| 129 | - Only 4 concurrent processes / up to 32 concurrent threads were probed; |
| 130 | this characterizes correctness under contention, not throughput or |
| 131 | lock-wait latency at higher fan-out. |
| 132 | """ |
| 133 | |
| 134 | from __future__ import annotations |
| 135 | |
| 136 | import json |
| 137 | import subprocess |
| 138 | import sys |
| 139 | import threading |
| 140 | import time |
| 141 | from pathlib import Path |
| 142 | |
| 143 | import pytest |
| 144 | |
| 145 | import object_record_changes |
| 146 | import object_records |
| 147 | |
| 148 | pytestmark = pytest.mark.conformance |
| 149 | |
| 150 | ID_FIELD = {"name": "id"} |
| 151 | VALUE_FIELD = {"name": "value", "type": "textarea"} |
| 152 | |
| 153 | REPO_ROOT = str(Path(object_records.__file__).resolve().parent) |
| 154 | |
| 155 | |
| 156 | # --- setup helpers (mirror tests/test_embedded_json_lines_characterization.py) -- |
| 157 | |
| 158 | |
| 159 | def write_schema(data_dir: Path, collection: str, fields: list[dict]) -> Path: |
| 160 | path = data_dir / "schemas" / f"{collection}.json" |
| 161 | path.parent.mkdir(parents=True, exist_ok=True) |
| 162 | path.write_text(json.dumps({"fields": fields})) |
| 163 | return path |
| 164 | |
| 165 | |
| 166 | def write_append_schema(data_dir: Path, collection: str, fields: list[dict]) -> Path: |
| 167 | path = data_dir / "schemas" / f"{collection}.json" |
| 168 | path.parent.mkdir(parents=True, exist_ok=True) |
| 169 | path.write_text(json.dumps({"fields": fields, "storage": "append"})) |
| 170 | return path |
| 171 | |
| 172 | |
| 173 | def _clear_caches() -> None: |
| 174 | object_records._RECORDS_CACHE.clear() |
| 175 | object_records._OIDX_CACHE.clear() |
| 176 | |
| 177 | |
| 178 | def _run_threads(target, count: int) -> list[BaseException | None]: |
| 179 | """Run `count` copies of target(i, barrier) as threads, all released by |
| 180 | a shared Barrier at (as close to) the same instant, to maximize actual |
| 181 | lock contention rather than letting the OS schedule them apart. Returns |
| 182 | one exception (or None) per thread, in thread-index order.""" |
| 183 | barrier = threading.Barrier(count) |
| 184 | results: list[BaseException | None] = [None] * count |
| 185 | |
| 186 | def wrapper(i: int) -> None: |
| 187 | try: |
| 188 | target(i, barrier) |
| 189 | except BaseException as exc: # noqa: BLE001 -- captured, not swallowed |
| 190 | results[i] = exc |
| 191 | |
| 192 | threads = [threading.Thread(target=wrapper, args=(i,)) for i in range(count)] |
| 193 | for t in threads: |
| 194 | t.start() |
| 195 | for t in threads: |
| 196 | t.join(timeout=30) |
| 197 | return results |
| 198 | |
| 199 | |
| 200 | # ============================================================================= |
| 201 | # 1. SAME-PROCESS (threads) -- N DISTINCT records created concurrently |
| 202 | # ============================================================================= |
| 203 | |
| 204 | |
| 205 | @pytest.mark.parametrize("storage", ["classic", "append"]) |
| 206 | @pytest.mark.parametrize("n", [8, 32]) |
| 207 | def test_same_process_concurrent_distinct_creates_no_lost_writes(tmp_path, storage, n): |
| 208 | """N threads each CREATE a DISTINCT record id into ONE collection, |
| 209 | released simultaneously by a Barrier. Property under test: the |
| 210 | flock-based _records_file_lock genuinely serializes concurrent |
| 211 | create_collection_record calls within one process (both classic's |
| 212 | tempfile+replace path and append's in-place open("a") fast path), so |
| 213 | no writer's row is lost and the file never ends up corrupt (unparseable |
| 214 | or with a wrong row count) even though every thread's create does its |
| 215 | own read-modify-write (duplicate-id check, then persist) inside the |
| 216 | lock. RESULT: safe -- all N records survive, no exceptions, no |
| 217 | duplicate/garbled rows.""" |
| 218 | data_dir = tmp_path / "data" |
| 219 | collection = "widgets" |
| 220 | if storage == "classic": |
| 221 | write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 222 | else: |
| 223 | write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 224 | |
| 225 | def worker(i: int, barrier: threading.Barrier) -> None: |
| 226 | barrier.wait(timeout=10) |
| 227 | object_records.create_collection_record( |
| 228 | collection, |
| 229 | {"id": f"rec-{i:04d}", "value": f"thread-{i}"}, |
| 230 | base_dir=data_dir, |
| 231 | roots=[], |
| 232 | ) |
| 233 | |
| 234 | errors = _run_threads(worker, n) |
| 235 | assert errors == [None] * n, f"unexpected exceptions from concurrent creates: {errors}" |
| 236 | |
| 237 | _clear_caches() |
| 238 | all_records = object_records.read_collection_records(collection, base_dir=data_dir, roots=[]) |
| 239 | ids = sorted(r["id"] for r in all_records) |
| 240 | expected = sorted(f"rec-{i:04d}" for i in range(n)) |
| 241 | assert ids == expected, ( |
| 242 | f"lost or duplicated writes under same-process concurrency: got {len(ids)} of " |
| 243 | f"{n} expected records" |
| 244 | ) |
| 245 | for r in all_records: |
| 246 | i = int(r["id"].rsplit("-", 1)[1]) |
| 247 | assert r["value"] == f"thread-{i}", f"corrupted/cross-wired row: {r!r}" |
| 248 | |
| 249 | |
| 250 | # ============================================================================= |
| 251 | # 2. SAME-PROCESS (threads) -- N writers UPDATING the SAME record concurrently |
| 252 | # ============================================================================= |
| 253 | |
| 254 | |
| 255 | @pytest.mark.parametrize("storage", ["classic", "append"]) |
| 256 | def test_same_process_concurrent_same_record_update_is_last_writer_wins(tmp_path, storage): |
| 257 | """N threads all UPDATE the SAME existing record id concurrently, each |
| 258 | writing a distinct, easily-attributable value. There is no optimistic |
| 259 | concurrency control in this codebase yet (compare-and-set is spec 63, |
| 260 | not built) -- update_collection_record's read-modify-write is only |
| 261 | serialized against OTHER writers by _records_file_lock, never checked |
| 262 | against a caller-supplied "expected prior version". So the CORRECT, |
| 263 | EXPECTED outcome here is last-writer-wins: the record's final value must |
| 264 | be turned into EXACTLY ONE of the N submitted values (proving no |
| 265 | interleaving/corruption of the row itself), while the other N-1 updates |
| 266 | are silently overwritten with no error and no conflict signal to their |
| 267 | caller -- that overwrite is precisely the gap spec 63 is meant to close. |
| 268 | This test also confirms the durable change LOG (object_record_changes, |
| 269 | a separate JSONL append) still records all N update events even though |
| 270 | current-state only reflects the winner -- i.e. the update itself isn't |
| 271 | lost from history, only from the live row. RESULT: safe from |
| 272 | corruption, but N-1 of N updates are silently discarded from live |
| 273 | state (expected LWW, not a bug -- the spec-63 gap, characterized |
| 274 | honestly).""" |
| 275 | data_dir = tmp_path / "data" |
| 276 | collection = "counters" |
| 277 | n = 16 |
| 278 | if storage == "classic": |
| 279 | write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 280 | else: |
| 281 | write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 282 | |
| 283 | object_records.create_collection_record( |
| 284 | collection, {"id": "shared", "value": "seed"}, base_dir=data_dir, roots=[] |
| 285 | ) |
| 286 | |
| 287 | candidates = [f"writer-{i:04d}" for i in range(n)] |
| 288 | |
| 289 | def worker(i: int, barrier: threading.Barrier) -> None: |
| 290 | barrier.wait(timeout=10) |
| 291 | object_records.update_collection_record( |
| 292 | collection, "shared", {"value": candidates[i]}, base_dir=data_dir, roots=[] |
| 293 | ) |
| 294 | |
| 295 | errors = _run_threads(worker, n) |
| 296 | assert errors == [None] * n, f"unexpected exceptions from concurrent same-record updates: {errors}" |
| 297 | |
| 298 | _clear_caches() |
| 299 | final = object_records.get_collection_record(collection, "shared", base_dir=data_dir, roots=[]) |
| 300 | assert final["value"] in candidates, ( |
| 301 | f"CORRUPTION (not LWW): final value {final['value']!r} is not any single submitted " |
| 302 | f"value -- this would indicate a torn/interleaved write, not last-writer-wins" |
| 303 | ) |
| 304 | |
| 305 | changes = object_record_changes.list_record_changes( |
| 306 | collection, record_id="shared", base_dir=data_dir, limit=1000 |
| 307 | ) |
| 308 | update_changes = [c for c in changes["changes"] if c.get("action") == "update"] |
| 309 | assert len(update_changes) == n, ( |
| 310 | f"change log lost events under concurrency: expected {n} update entries, " |
| 311 | f"got {len(update_changes)} -- the durable log itself should never drop a write " |
| 312 | f"even though live state only keeps the last one" |
| 313 | ) |
| 314 | logged_values = {c["after"]["value"] for c in update_changes if c.get("after")} |
| 315 | assert logged_values == set(candidates), ( |
| 316 | "change log did not capture all N submitted values even though it recorded N events" |
| 317 | ) |
| 318 | |
| 319 | |
| 320 | # ============================================================================= |
| 321 | # 3. Concurrent READERS alongside a sustained WRITER (readers hold no lock) |
| 322 | # ============================================================================= |
| 323 | |
| 324 | |
| 325 | @pytest.mark.parametrize("storage", ["classic", "append"]) |
| 326 | def test_concurrent_readers_never_see_corruption_under_sustained_writes(tmp_path, storage): |
| 327 | """Readers take NO lock at all (see the _RECORDS_CACHE CONCURRENCY block |
| 328 | comment in object_records.py) -- they rely entirely on classic mode's |
| 329 | atomic tempfile+replace, and append mode's append-then-flush plus the |
| 330 | fact that csv.reader/the oidx scan are torn-tail-tolerant, to never |
| 331 | observe a half-written file. One writer thread continuously creates new |
| 332 | records while several reader threads continuously call |
| 333 | read_collection_records / get_collection_record / list_collection_records |
| 334 | concurrently, for a fixed number of iterations (not wall-clock time, to |
| 335 | keep this fast and deterministic-ish). Property under test: a reader |
| 336 | may see a record count that lags the writer (a benign, expected race -- |
| 337 | NOT asserted against), but must NEVER raise from a parse error and must |
| 338 | NEVER return a row with a wrong/garbled value for an id it does return. |
| 339 | RESULT: safe -- no reader exception, no garbled row observed across the |
| 340 | run, for both storage modes.""" |
| 341 | data_dir = tmp_path / "data" |
| 342 | collection = "stream" |
| 343 | if storage == "classic": |
| 344 | write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 345 | else: |
| 346 | write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 347 | |
| 348 | object_records.create_collection_record( |
| 349 | collection, {"id": "seed-0000", "value": "seed"}, base_dir=data_dir, roots=[] |
| 350 | ) |
| 351 | |
| 352 | WRITE_COUNT = 150 |
| 353 | READ_ITERATIONS = 400 |
| 354 | stop = threading.Event() |
| 355 | reader_errors: list[BaseException] = [] |
| 356 | lock = threading.Lock() |
| 357 | |
| 358 | def writer() -> None: |
| 359 | for i in range(WRITE_COUNT): |
| 360 | object_records.create_collection_record( |
| 361 | collection, |
| 362 | {"id": f"live-{i:04d}", "value": f"payload-{i:04d}"}, |
| 363 | base_dir=data_dir, |
| 364 | roots=[], |
| 365 | ) |
| 366 | stop.set() |
| 367 | |
| 368 | def reader() -> None: |
| 369 | seen = 0 |
| 370 | while not stop.is_set() or seen < READ_ITERATIONS: |
| 371 | try: |
| 372 | all_records = object_records.read_collection_records( |
| 373 | collection, base_dir=data_dir, roots=[] |
| 374 | ) |
| 375 | for r in all_records: |
| 376 | if r["id"] == "seed-0000": |
| 377 | assert r["value"] == "seed" |
| 378 | elif r["id"].startswith("live-"): |
| 379 | idx = r["id"].rsplit("-", 1)[1] |
| 380 | assert r["value"] == f"payload-{idx}", f"garbled row: {r!r}" |
| 381 | one = object_records.get_collection_record( |
| 382 | collection, "seed-0000", base_dir=data_dir, roots=[] |
| 383 | ) |
| 384 | assert one["value"] == "seed" |
| 385 | object_records.list_collection_records( |
| 386 | collection, base_dir=data_dir, roots=[], limit=1000 |
| 387 | ) |
| 388 | except BaseException as exc: # noqa: BLE001 |
| 389 | with lock: |
| 390 | reader_errors.append(exc) |
| 391 | return |
| 392 | seen += 1 |
| 393 | if seen >= READ_ITERATIONS and stop.is_set(): |
| 394 | return |
| 395 | |
| 396 | writer_thread = threading.Thread(target=writer) |
| 397 | reader_threads = [threading.Thread(target=reader) for _ in range(4)] |
| 398 | writer_thread.start() |
| 399 | for t in reader_threads: |
| 400 | t.start() |
| 401 | writer_thread.join(timeout=60) |
| 402 | for t in reader_threads: |
| 403 | t.join(timeout=60) |
| 404 | |
| 405 | assert not reader_errors, f"reader(s) observed corruption/exception: {reader_errors!r}" |
| 406 | |
| 407 | _clear_caches() |
| 408 | final = object_records.read_collection_records(collection, base_dir=data_dir, roots=[]) |
| 409 | assert len(final) == WRITE_COUNT + 1 |
| 410 | |
| 411 | |
| 412 | # ============================================================================= |
| 413 | # 4. CROSS-PROCESS -- the central question: is the lock effective across OS |
| 414 | # processes, not just threads? |
| 415 | # ============================================================================= |
| 416 | |
| 417 | |
| 418 | _WORKER_SCRIPT = '''\ |
| 419 | import argparse |
| 420 | import json |
| 421 | import sys |
| 422 | import time |
| 423 | from pathlib import Path |
| 424 | |
| 425 | |
| 426 | def _wait_for_barrier(barrier_path): |
| 427 | while not Path(barrier_path).exists(): |
| 428 | time.sleep(0.001) |
| 429 | |
| 430 | |
| 431 | def main(): |
| 432 | parser = argparse.ArgumentParser() |
| 433 | parser.add_argument("repo_root") |
| 434 | parser.add_argument("data_dir") |
| 435 | parser.add_argument("collection") |
| 436 | parser.add_argument("mode", choices=["create_batch", "update_same"]) |
| 437 | parser.add_argument("payload") |
| 438 | parser.add_argument("--barrier", default="") |
| 439 | args = parser.parse_args() |
| 440 | |
| 441 | sys.path.insert(0, args.repo_root) |
| 442 | import object_records |
| 443 | |
| 444 | payload = json.loads(args.payload) |
| 445 | if args.barrier: |
| 446 | _wait_for_barrier(args.barrier) |
| 447 | |
| 448 | if args.mode == "create_batch": |
| 449 | id_prefix = payload["id_prefix"] |
| 450 | count = payload["count"] |
| 451 | for i in range(count): |
| 452 | object_records.create_collection_record( |
| 453 | args.collection, |
| 454 | {"id": "%s-%04d" % (id_prefix, i), "value": id_prefix}, |
| 455 | base_dir=args.data_dir, |
| 456 | roots=[], |
| 457 | ) |
| 458 | elif args.mode == "update_same": |
| 459 | object_records.update_collection_record( |
| 460 | args.collection, |
| 461 | payload["record_id"], |
| 462 | {"value": payload["value"]}, |
| 463 | base_dir=args.data_dir, |
| 464 | roots=[], |
| 465 | ) |
| 466 | return 0 |
| 467 | |
| 468 | |
| 469 | if __name__ == "__main__": |
| 470 | sys.exit(main()) |
| 471 | ''' |
| 472 | |
| 473 | |
| 474 | def _write_worker_script(tmp_path: Path) -> Path: |
| 475 | script_path = tmp_path / "_cp_worker.py" |
| 476 | script_path.write_text(_WORKER_SCRIPT) |
| 477 | return script_path |
| 478 | |
| 479 | |
| 480 | def _launch(script_path: Path, *args: str, barrier: Path | None = None) -> subprocess.Popen: |
| 481 | cmd = [sys.executable, str(script_path), *args] |
| 482 | if barrier is not None: |
| 483 | cmd += ["--barrier", str(barrier)] |
| 484 | return subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) |
| 485 | |
| 486 | |
| 487 | def _release_barrier_soon(barrier_path: Path, delay: float = 0.3) -> None: |
| 488 | """Give every launched subprocess time to get past interpreter startup |
| 489 | and reach its barrier-poll loop, THEN drop the barrier file so all |
| 490 | processes resume as close to simultaneously as OS scheduling allows -- |
| 491 | this is what makes the probe actually exercise contention on the flock |
| 492 | instead of processes serializing naturally through staggered startup.""" |
| 493 | time.sleep(delay) |
| 494 | barrier_path.write_text("go") |
| 495 | |
| 496 | |
| 497 | @pytest.mark.parametrize("storage", ["classic", "append"]) |
| 498 | def test_cross_process_concurrent_distinct_creates_no_lost_writes_or_corruption( |
| 499 | tmp_path, storage |
| 500 | ): |
| 501 | """THE central probe: 4 separate OS processes (subprocess, not threads |
| 502 | -- each a fresh Python interpreter with its own memory, its own cold |
| 503 | _RECORDS_CACHE/_OIDX_CACHE, sharing nothing with the parent test process |
| 504 | or each other except the records.tsv/.lock/.oidx files on disk), each |
| 505 | creating 20 DISTINCT record ids into the SAME collection, released via |
| 506 | a shared barrier file so their writes genuinely overlap in time. |
| 507 | |
| 508 | Property under test: is `_records_file_lock`'s fcntl.flock actually |
| 509 | effective ACROSS processes -- i.e. does it prevent two independent |
| 510 | processes' read-modify-write cycles (duplicate-id check + persist) from |
| 511 | interleaving on disk -- or, as would be the case if the mechanism were |
| 512 | a plain in-process `threading.Lock` (each process would have its OWN, |
| 513 | mutually-invisible lock object, providing zero cross-process exclusion), |
| 514 | can two processes corrupt or lose each other's writes? |
| 515 | |
| 516 | RESULT: safe. flock is a kernel-level advisory lock keyed by the lock |
| 517 | file's path, not by the calling process/interpreter, so it correctly |
| 518 | serializes all 4 processes' writes: every one of the 80 records |
| 519 | survives, the file parses cleanly, and no row is garbled -- in BOTH |
| 520 | classic mode (tempfile+replace per full rewrite) and append mode |
| 521 | (in-place open("a") fast-path, which has NO atomic-rename safety net of |
| 522 | its own and depends entirely on the lock for this property).""" |
| 523 | data_dir = tmp_path / "data" |
| 524 | collection = "cp_widgets" |
| 525 | if storage == "classic": |
| 526 | write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 527 | else: |
| 528 | write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 529 | |
| 530 | script_path = _write_worker_script(tmp_path) |
| 531 | barrier_path = tmp_path / "barrier_distinct" |
| 532 | num_procs = 4 |
| 533 | per_proc = 20 |
| 534 | |
| 535 | procs = [] |
| 536 | for p in range(num_procs): |
| 537 | payload = json.dumps({"id_prefix": f"proc{p}", "count": per_proc}) |
| 538 | procs.append( |
| 539 | _launch( |
| 540 | script_path, |
| 541 | REPO_ROOT, |
| 542 | str(data_dir), |
| 543 | collection, |
| 544 | "create_batch", |
| 545 | payload, |
| 546 | barrier=barrier_path, |
| 547 | ) |
| 548 | ) |
| 549 | |
| 550 | _release_barrier_soon(barrier_path) |
| 551 | |
| 552 | outputs = [] |
| 553 | for proc in procs: |
| 554 | out, _ = proc.communicate(timeout=60) |
| 555 | outputs.append((proc.returncode, out)) |
| 556 | |
| 557 | failures = [(rc, out) for rc, out in outputs if rc != 0] |
| 558 | assert not failures, f"worker process(es) failed: {failures!r}" |
| 559 | |
| 560 | _clear_caches() |
| 561 | all_records = object_records.read_collection_records(collection, base_dir=data_dir, roots=[]) |
| 562 | ids = sorted(r["id"] for r in all_records) |
| 563 | expected = sorted( |
| 564 | f"proc{p}-{i:04d}" for p in range(num_procs) for i in range(per_proc) |
| 565 | ) |
| 566 | assert ids == expected, ( |
| 567 | f"cross-process write loss/corruption: expected {len(expected)} records, " |
| 568 | f"got {len(ids)} ({[i for i in expected if i not in ids][:5]} missing, " |
| 569 | f"{[i for i in ids if i not in expected][:5]} unexpected)" |
| 570 | ) |
| 571 | for r in all_records: |
| 572 | prefix = r["id"].rsplit("-", 1)[0] |
| 573 | assert r["value"] == prefix, f"cross-wired/corrupted row across processes: {r!r}" |
| 574 | |
| 575 | |
| 576 | @pytest.mark.parametrize("storage", ["classic", "append"]) |
| 577 | def test_cross_process_concurrent_same_record_update_is_last_writer_wins_not_corruption( |
| 578 | tmp_path, storage |
| 579 | ): |
| 580 | """Same question as the distinct-create probe above, but for the |
| 581 | same-record UPDATE case ACROSS processes: 4 separate OS processes all |
| 582 | update the SAME existing record concurrently (barrier-released). As in |
| 583 | the same-process version of this test, there is no compare-and-set |
| 584 | (spec 63) -- so the only claim under test is that the final value is |
| 585 | EXACTLY ONE of the 4 submitted values (the lock prevented a torn/ |
| 586 | interleaved write), not that no update was overwritten (some updates |
| 587 | ARE expected to be silently lost to LWW; that is unrelated to whether |
| 588 | the lock itself works). RESULT: safe from corruption -- final value is |
| 589 | always exactly one clean candidate string, never a merge/garble, in |
| 590 | both storage modes; same LWW-loses-N-1-of-N semantics as the |
| 591 | same-process case, now confirmed to hold across process boundaries |
| 592 | too.""" |
| 593 | data_dir = tmp_path / "data" |
| 594 | collection = "cp_counters" |
| 595 | if storage == "classic": |
| 596 | write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 597 | else: |
| 598 | write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 599 | |
| 600 | object_records.create_collection_record( |
| 601 | collection, {"id": "shared", "value": "seed"}, base_dir=data_dir, roots=[] |
| 602 | ) |
| 603 | |
| 604 | script_path = _write_worker_script(tmp_path) |
| 605 | barrier_path = tmp_path / "barrier_update" |
| 606 | num_procs = 4 |
| 607 | candidates = [f"proc-writer-{p}" for p in range(num_procs)] |
| 608 | |
| 609 | procs = [] |
| 610 | for p in range(num_procs): |
| 611 | payload = json.dumps({"record_id": "shared", "value": candidates[p]}) |
| 612 | procs.append( |
| 613 | _launch( |
| 614 | script_path, |
| 615 | REPO_ROOT, |
| 616 | str(data_dir), |
| 617 | collection, |
| 618 | "update_same", |
| 619 | payload, |
| 620 | barrier=barrier_path, |
| 621 | ) |
| 622 | ) |
| 623 | |
| 624 | _release_barrier_soon(barrier_path) |
| 625 | |
| 626 | outputs = [] |
| 627 | for proc in procs: |
| 628 | out, _ = proc.communicate(timeout=60) |
| 629 | outputs.append((proc.returncode, out)) |
| 630 | |
| 631 | failures = [(rc, out) for rc, out in outputs if rc != 0] |
| 632 | assert not failures, f"worker process(es) failed: {failures!r}" |
| 633 | |
| 634 | _clear_caches() |
| 635 | final = object_records.get_collection_record(collection, "shared", base_dir=data_dir, roots=[]) |
| 636 | assert final["value"] in candidates, ( |
| 637 | f"CROSS-PROCESS CORRUPTION: final value {final['value']!r} is not any single " |
| 638 | f"submitted value -- indicates the flock did NOT serialize these processes' " |
| 639 | f"read-modify-write cycles" |
| 640 | ) |
| 641 | |
| 642 | |
| 643 | # ============================================================================= |
| 644 | # 5. id->offset oidx sidecar under concurrent same-process appends |
| 645 | # ============================================================================= |
| 646 | |
| 647 | |
| 648 | def test_oidx_sidecar_stays_coherent_under_concurrent_appends(tmp_path, monkeypatch): |
| 649 | """Force every point op through the cold-cache id->offset sidecar path |
| 650 | (DBBASIC_RECORDS_CACHE_MAX_ROWS=0, same trick |
| 651 | test_embedded_json_lines_characterization.py's oidx tests use), then |
| 652 | run N threads each creating a distinct record into one append-mode |
| 653 | collection concurrently. Property under test: after the dust settles, |
| 654 | does the sidecar (.records.oidx) -- rebuilt/caught-up independently by |
| 655 | whichever thread happens to read it, under no lock at all on the READ |
| 656 | side -- resolve every one of the N ids to the CORRECT row via the |
| 657 | offset-indexed by-id path (get_collection_record), matching a full-fold |
| 658 | read? The sidecar is documented as "100% disposable... any reader |
| 659 | unable to make sense of it... rebuilds it with one sequential scan" |
| 660 | (object_records.py ~line 90), so even if concurrent writers left it |
| 661 | momentarily behind the file's actual size, a subsequent reader should |
| 662 | self-heal rather than serve stale/wrong offsets. RESULT: safe -- every |
| 663 | id resolves to its correct value via BOTH the by-id sidecar path and |
| 664 | the full-fold path, and the two agree, after concurrent creation.""" |
| 665 | monkeypatch.setenv("DBBASIC_RECORDS_CACHE_MAX_ROWS", "0") |
| 666 | data_dir = tmp_path / "data" |
| 667 | collection = "oidx_stress" |
| 668 | write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD]) |
| 669 | n = 24 |
| 670 | |
| 671 | def worker(i: int, barrier: threading.Barrier) -> None: |
| 672 | barrier.wait(timeout=10) |
| 673 | object_records.create_collection_record( |
| 674 | collection, |
| 675 | {"id": f"oi-{i:04d}", "value": f"val-{i:04d}"}, |
| 676 | base_dir=data_dir, |
| 677 | roots=[], |
| 678 | ) |
| 679 | |
| 680 | errors = _run_threads(worker, n) |
| 681 | assert errors == [None] * n, f"unexpected exceptions: {errors}" |
| 682 | |
| 683 | _clear_caches() |
| 684 | all_records = object_records.read_collection_records(collection, base_dir=data_dir, roots=[]) |
| 685 | assert sorted(r["id"] for r in all_records) == sorted(f"oi-{i:04d}" for i in range(n)) |
| 686 | |
| 687 | for i in range(n): |
| 688 | _clear_caches() |
| 689 | by_id = object_records.get_collection_record( |
| 690 | collection, f"oi-{i:04d}", base_dir=data_dir, roots=[] |
| 691 | ) |
| 692 | assert by_id["value"] == f"val-{i:04d}", ( |
| 693 | f"oidx sidecar served wrong/stale data for oi-{i:04d} after concurrent " |
| 694 | f"appends: got {by_id!r}" |
| 695 | ) |
| 696 | |
| 697 | sidecar_path = data_dir / "collections" / collection / object_records.OIDX_FILE |
| 698 | assert sidecar_path.exists(), "sidecar was never (re)built despite over-threshold reads" |