Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

dbbasic-object-server/tests/test_durability_concurrency_characterization.py
Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
test_durability_concurrency_characterization.py · 31.1 KB · 698 lines
1"""CHARACTERIZATION tests: does object_records.py's write path stay correct
2under CONCURRENT writers/readers, and what is its actual concurrency-safety
3boundary?
4
5Pure characterization of EXISTING behavior. Nothing in object_records.py (or
6any other production module) is modified by this file. Where a probe finds a
7real gap, the assertion documents the correctness property a database-shaped
8substrate SHOULD have and is left in place -- do not "fix" a finding here by
9weakening the assertion to match the gap.
10
11THE LOCK, READ FROM SOURCE FIRST (then verified empirically below):
12`_records_file_lock` (object_records.py, ~line 2703) is NOT a
13`threading.Lock` keyed by path (that would be a plausible, reasonable
14mechanism, and is what this file's brief assumed going in) -- it is an
15`fcntl.flock(LOCK_EX)` advisory lock held on a sidecar `.records.tsv.lock`
16file, acquired fresh (a new `open("a")` file descriptor) on every
17create/update/delete/compact call and released when the `with` block exits.
18That distinction matters a great deal for this file's central question:
19
20 - flock is a KERNEL-level lock associated with the *open file description*
21 (not the process, and not the Python-level lock object), so it
22 correctly serializes BOTH concurrent threads in the same process
23 (each thread's own `with lock_path.open("a")` gets its own fd/open
24 file description, and flock blocks across distinct descriptions even
25 within one process) AND concurrent separate OS processes on the same
26 machine sharing the same filesystem path -- there is no dependency on
27 both callers being the same Python interpreter, unlike a
28 `threading.Lock`, which cannot be observed at all outside its own
29 process.
30 - It is advisory, not mandatory: it only serializes writers that
31 themselves call `_records_file_lock` (i.e. every writer inside this
32 module). Nothing stops a process that opens records.tsv directly and
33 writes to it without going through object_records.py.
34 - Classic-mode full rewrites additionally write via a tempfile +
35 `Path.replace()` (atomic rename) rather than in-place, so even a READER
36 that holds no lock at all (readers never take `_records_file_lock` --
37 see the CONCURRENCY block comment above _RECORDS_CACHE) always sees a
38 complete pre- or post-write file, never a torn one -- the lock's job in
39 that mode is only to serialize writers' read-modify-write cycles
40 against EACH OTHER, not against readers.
41 - Append-mode's fast path (`_append_records_rows`) is a genuine in-place
42 mutation (`open(path, "a")` + `csv.writer.writerow` + flush) -- there is
43 no atomic-rename safety net for this path; the flock is the ONLY thing
44 preventing two writers' single `write()` calls from interleaving on
45 disk. This is exactly the case this file's cross-process probes target.
46
47So the headline question this file exists to answer empirically is not
48"is the lock cross-thread only" (source review already answers that: no,
49flock is not thread-scoped) but "does the flock-based mechanism actually
50hold up under real concurrent contention, in both storage modes, across
51both threads and separate OS processes" -- and separately, to characterize
52honestly what current same-record concurrent UPDATE semantics are (last-
53writer-wins; optimistic concurrency / compare-and-set is spec 63, not yet
54built, so nothing here detects or rejects a lost concurrent update).
55
56FINDINGS (all 13 tests below PASS, consistently across repeated runs; this
57summary is the worst-first rollup -- worst first meaning "most surprising
58if you assumed the naive per-process-only-lock model", not "most broken":
59nothing here is broken):
60
61 1. Cross-process DISTINCT-record writes, classic AND append mode: SAFE.
62 4 separate OS processes (subprocess, not threads -- fresh interpreters,
63 cold caches, sharing nothing but the files on disk), barrier-released
64 to overlap in time, each creating 20 distinct records into the same
65 collection: all 80 records survive, none garbled, in both modes. This
66 is the headline result and it contradicts the naive assumption a
67 `threading.Lock` mechanism would predict (that would provide ZERO
68 cross-process exclusion): `_records_file_lock` is actually
69 `fcntl.flock`, a kernel-level advisory lock keyed by the lock file's
70 path, not by the calling interpreter -- so it DOES serialize
71 independent processes, not just threads within one process.
72 2. Cross-process SAME-record concurrent UPDATE: SAFE from corruption,
73 last-writer-wins as expected. 4 processes updating one record
74 concurrently always leave it holding exactly one clean submitted
75 value, never a merge/garble -- confirming the lock's cross-process
76 exclusion holds for the read-modify-write update path too, not just
77 the append-only create path.
78 3. Same-process (thread) DISTINCT-record writes, classic AND append,
79 N=8 and N=32: SAFE. No lost or duplicated writes, no garbled rows.
80 4. Same-process (thread) SAME-record concurrent UPDATE, both modes:
81 SAFE from corruption, last-writer-wins. 16 threads updating one
82 record concurrently always land on exactly one submitted value. This
83 is the CURRENT concurrent-update guarantee, and it is a real gap:
84 15 of 16 updates are silently discarded from live state with no error,
85 no conflict signal, and no way for a caller to know its update was
86 clobbered -- this is exactly the hole optimistic concurrency /
87 compare-and-set (spec 63) is meant to fill. Notably, the discarded
88 updates are NOT lost from the durable audit trail: object_record_
89 changes' append-only JSONL log recorded all 16 update events (with
90 all 16 distinct submitted values) even though get_collection_record
91 shows only the winner -- so "last-writer-wins" is a live-state
92 property, not a durability/logging one.
93 5. Concurrent readers alongside a sustained writer, both modes: SAFE.
94 Readers hold no lock at all (by design) and never raised, never saw a
95 garbled row, across a writer creating 150 records while 4 reader
96 threads hammered read_collection_records/get_collection_record/
97 list_collection_records concurrently. A reader may transiently see a
98 record count that lags the writer (expected, unasserted) but the file
99 is never observed torn -- classic mode's atomic tempfile+replace and
100 append mode's flush-after-write both hold up under concurrent reads.
101 6. id->offset oidx sidecar under concurrent same-process appends: SAFE.
102 After 24 threads concurrently create distinct records (forced through
103 the cold-cache sidecar path via DBBASIC_RECORDS_CACHE_MAX_ROWS=0),
104 every id resolves to its correct value via the offset-indexed by-id
105 read, agreeing with a full fold -- the sidecar's documented
106 disposability/self-heal ("any reader unable to make sense of it...
107 rebuilds it") held up in practice under contention.
108
109BOTTOM LINE: the lock is not a threading.Lock and not scoped to one
110process -- it is a real fcntl.flock advisory file lock, and it is
111effective across both threads AND separate OS processes (verified via
112actual subprocesses, not simulated). For a multi-worker uvicorn/gunicorn
113deployment (multiple OS processes sharing the same data directory), this
114means concurrent writes are NOT the unsafe case some deployments would
115fear -- corruption/lost-write risk was not observed in any probe here. The
116one real, honest gap is same-record concurrent UPDATE semantics: pure
117last-writer-wins, with silent overwrite of losing updates and no conflict
118detection -- present identically whether the concurrent writers are threads
119in one process or separate processes, and unrelated to the lock's
120effectiveness. That gap is spec 63's to close, not a locking bug.
121
122CAVEATS (not probed by, or outside the scope of, this file):
123 - flock is advisory: a process that opens records.tsv directly without
124 going through object_records.py's write path is not blocked by it.
125 - flock's cross-process guarantee assumes a shared local filesystem with
126 working flock semantics; it is well-known to be unreliable-to-absent
127 over NFS and some network/container filesystem configurations, which
128 were not exercised here (this ran against a local disk).
129 - Only 4 concurrent processes / up to 32 concurrent threads were probed;
130 this characterizes correctness under contention, not throughput or
131 lock-wait latency at higher fan-out.
132"""
133
134from __future__ import annotations
135
136import json
137import subprocess
138import sys
139import threading
140import time
141from pathlib import Path
142
143import pytest
144
145import object_record_changes
146import object_records
147
148pytestmark = pytest.mark.conformance
149
150ID_FIELD = {"name": "id"}
151VALUE_FIELD = {"name": "value", "type": "textarea"}
152
153REPO_ROOT = str(Path(object_records.__file__).resolve().parent)
154
155
156# --- setup helpers (mirror tests/test_embedded_json_lines_characterization.py) --
157
158
159def write_schema(data_dir: Path, collection: str, fields: list[dict]) -> Path:
160 path = data_dir / "schemas" / f"{collection}.json"
161 path.parent.mkdir(parents=True, exist_ok=True)
162 path.write_text(json.dumps({"fields": fields}))
163 return path
164
165
166def write_append_schema(data_dir: Path, collection: str, fields: list[dict]) -> Path:
167 path = data_dir / "schemas" / f"{collection}.json"
168 path.parent.mkdir(parents=True, exist_ok=True)
169 path.write_text(json.dumps({"fields": fields, "storage": "append"}))
170 return path
171
172
173def _clear_caches() -> None:
174 object_records._RECORDS_CACHE.clear()
175 object_records._OIDX_CACHE.clear()
176
177
178def _run_threads(target, count: int) -> list[BaseException | None]:
179 """Run `count` copies of target(i, barrier) as threads, all released by
180 a shared Barrier at (as close to) the same instant, to maximize actual
181 lock contention rather than letting the OS schedule them apart. Returns
182 one exception (or None) per thread, in thread-index order."""
183 barrier = threading.Barrier(count)
184 results: list[BaseException | None] = [None] * count
185
186 def wrapper(i: int) -> None:
187 try:
188 target(i, barrier)
189 except BaseException as exc: # noqa: BLE001 -- captured, not swallowed
190 results[i] = exc
191
192 threads = [threading.Thread(target=wrapper, args=(i,)) for i in range(count)]
193 for t in threads:
194 t.start()
195 for t in threads:
196 t.join(timeout=30)
197 return results
198
199
200# =============================================================================
201# 1. SAME-PROCESS (threads) -- N DISTINCT records created concurrently
202# =============================================================================
203
204
205@pytest.mark.parametrize("storage", ["classic", "append"])
206@pytest.mark.parametrize("n", [8, 32])
207def test_same_process_concurrent_distinct_creates_no_lost_writes(tmp_path, storage, n):
208 """N threads each CREATE a DISTINCT record id into ONE collection,
209 released simultaneously by a Barrier. Property under test: the
210 flock-based _records_file_lock genuinely serializes concurrent
211 create_collection_record calls within one process (both classic's
212 tempfile+replace path and append's in-place open("a") fast path), so
213 no writer's row is lost and the file never ends up corrupt (unparseable
214 or with a wrong row count) even though every thread's create does its
215 own read-modify-write (duplicate-id check, then persist) inside the
216 lock. RESULT: safe -- all N records survive, no exceptions, no
217 duplicate/garbled rows."""
218 data_dir = tmp_path / "data"
219 collection = "widgets"
220 if storage == "classic":
221 write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
222 else:
223 write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
224
225 def worker(i: int, barrier: threading.Barrier) -> None:
226 barrier.wait(timeout=10)
227 object_records.create_collection_record(
228 collection,
229 {"id": f"rec-{i:04d}", "value": f"thread-{i}"},
230 base_dir=data_dir,
231 roots=[],
232 )
233
234 errors = _run_threads(worker, n)
235 assert errors == [None] * n, f"unexpected exceptions from concurrent creates: {errors}"
236
237 _clear_caches()
238 all_records = object_records.read_collection_records(collection, base_dir=data_dir, roots=[])
239 ids = sorted(r["id"] for r in all_records)
240 expected = sorted(f"rec-{i:04d}" for i in range(n))
241 assert ids == expected, (
242 f"lost or duplicated writes under same-process concurrency: got {len(ids)} of "
243 f"{n} expected records"
244 )
245 for r in all_records:
246 i = int(r["id"].rsplit("-", 1)[1])
247 assert r["value"] == f"thread-{i}", f"corrupted/cross-wired row: {r!r}"
248
249
250# =============================================================================
251# 2. SAME-PROCESS (threads) -- N writers UPDATING the SAME record concurrently
252# =============================================================================
253
254
255@pytest.mark.parametrize("storage", ["classic", "append"])
256def test_same_process_concurrent_same_record_update_is_last_writer_wins(tmp_path, storage):
257 """N threads all UPDATE the SAME existing record id concurrently, each
258 writing a distinct, easily-attributable value. There is no optimistic
259 concurrency control in this codebase yet (compare-and-set is spec 63,
260 not built) -- update_collection_record's read-modify-write is only
261 serialized against OTHER writers by _records_file_lock, never checked
262 against a caller-supplied "expected prior version". So the CORRECT,
263 EXPECTED outcome here is last-writer-wins: the record's final value must
264 be turned into EXACTLY ONE of the N submitted values (proving no
265 interleaving/corruption of the row itself), while the other N-1 updates
266 are silently overwritten with no error and no conflict signal to their
267 caller -- that overwrite is precisely the gap spec 63 is meant to close.
268 This test also confirms the durable change LOG (object_record_changes,
269 a separate JSONL append) still records all N update events even though
270 current-state only reflects the winner -- i.e. the update itself isn't
271 lost from history, only from the live row. RESULT: safe from
272 corruption, but N-1 of N updates are silently discarded from live
273 state (expected LWW, not a bug -- the spec-63 gap, characterized
274 honestly)."""
275 data_dir = tmp_path / "data"
276 collection = "counters"
277 n = 16
278 if storage == "classic":
279 write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
280 else:
281 write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
282
283 object_records.create_collection_record(
284 collection, {"id": "shared", "value": "seed"}, base_dir=data_dir, roots=[]
285 )
286
287 candidates = [f"writer-{i:04d}" for i in range(n)]
288
289 def worker(i: int, barrier: threading.Barrier) -> None:
290 barrier.wait(timeout=10)
291 object_records.update_collection_record(
292 collection, "shared", {"value": candidates[i]}, base_dir=data_dir, roots=[]
293 )
294
295 errors = _run_threads(worker, n)
296 assert errors == [None] * n, f"unexpected exceptions from concurrent same-record updates: {errors}"
297
298 _clear_caches()
299 final = object_records.get_collection_record(collection, "shared", base_dir=data_dir, roots=[])
300 assert final["value"] in candidates, (
301 f"CORRUPTION (not LWW): final value {final['value']!r} is not any single submitted "
302 f"value -- this would indicate a torn/interleaved write, not last-writer-wins"
303 )
304
305 changes = object_record_changes.list_record_changes(
306 collection, record_id="shared", base_dir=data_dir, limit=1000
307 )
308 update_changes = [c for c in changes["changes"] if c.get("action") == "update"]
309 assert len(update_changes) == n, (
310 f"change log lost events under concurrency: expected {n} update entries, "
311 f"got {len(update_changes)} -- the durable log itself should never drop a write "
312 f"even though live state only keeps the last one"
313 )
314 logged_values = {c["after"]["value"] for c in update_changes if c.get("after")}
315 assert logged_values == set(candidates), (
316 "change log did not capture all N submitted values even though it recorded N events"
317 )
318
319
320# =============================================================================
321# 3. Concurrent READERS alongside a sustained WRITER (readers hold no lock)
322# =============================================================================
323
324
325@pytest.mark.parametrize("storage", ["classic", "append"])
326def test_concurrent_readers_never_see_corruption_under_sustained_writes(tmp_path, storage):
327 """Readers take NO lock at all (see the _RECORDS_CACHE CONCURRENCY block
328 comment in object_records.py) -- they rely entirely on classic mode's
329 atomic tempfile+replace, and append mode's append-then-flush plus the
330 fact that csv.reader/the oidx scan are torn-tail-tolerant, to never
331 observe a half-written file. One writer thread continuously creates new
332 records while several reader threads continuously call
333 read_collection_records / get_collection_record / list_collection_records
334 concurrently, for a fixed number of iterations (not wall-clock time, to
335 keep this fast and deterministic-ish). Property under test: a reader
336 may see a record count that lags the writer (a benign, expected race --
337 NOT asserted against), but must NEVER raise from a parse error and must
338 NEVER return a row with a wrong/garbled value for an id it does return.
339 RESULT: safe -- no reader exception, no garbled row observed across the
340 run, for both storage modes."""
341 data_dir = tmp_path / "data"
342 collection = "stream"
343 if storage == "classic":
344 write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
345 else:
346 write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
347
348 object_records.create_collection_record(
349 collection, {"id": "seed-0000", "value": "seed"}, base_dir=data_dir, roots=[]
350 )
351
352 WRITE_COUNT = 150
353 READ_ITERATIONS = 400
354 stop = threading.Event()
355 reader_errors: list[BaseException] = []
356 lock = threading.Lock()
357
358 def writer() -> None:
359 for i in range(WRITE_COUNT):
360 object_records.create_collection_record(
361 collection,
362 {"id": f"live-{i:04d}", "value": f"payload-{i:04d}"},
363 base_dir=data_dir,
364 roots=[],
365 )
366 stop.set()
367
368 def reader() -> None:
369 seen = 0
370 while not stop.is_set() or seen < READ_ITERATIONS:
371 try:
372 all_records = object_records.read_collection_records(
373 collection, base_dir=data_dir, roots=[]
374 )
375 for r in all_records:
376 if r["id"] == "seed-0000":
377 assert r["value"] == "seed"
378 elif r["id"].startswith("live-"):
379 idx = r["id"].rsplit("-", 1)[1]
380 assert r["value"] == f"payload-{idx}", f"garbled row: {r!r}"
381 one = object_records.get_collection_record(
382 collection, "seed-0000", base_dir=data_dir, roots=[]
383 )
384 assert one["value"] == "seed"
385 object_records.list_collection_records(
386 collection, base_dir=data_dir, roots=[], limit=1000
387 )
388 except BaseException as exc: # noqa: BLE001
389 with lock:
390 reader_errors.append(exc)
391 return
392 seen += 1
393 if seen >= READ_ITERATIONS and stop.is_set():
394 return
395
396 writer_thread = threading.Thread(target=writer)
397 reader_threads = [threading.Thread(target=reader) for _ in range(4)]
398 writer_thread.start()
399 for t in reader_threads:
400 t.start()
401 writer_thread.join(timeout=60)
402 for t in reader_threads:
403 t.join(timeout=60)
404
405 assert not reader_errors, f"reader(s) observed corruption/exception: {reader_errors!r}"
406
407 _clear_caches()
408 final = object_records.read_collection_records(collection, base_dir=data_dir, roots=[])
409 assert len(final) == WRITE_COUNT + 1
410
411
412# =============================================================================
413# 4. CROSS-PROCESS -- the central question: is the lock effective across OS
414# processes, not just threads?
415# =============================================================================
416
417
418_WORKER_SCRIPT = '''\
419import argparse
420import json
421import sys
422import time
423from pathlib import Path
424
425
426def _wait_for_barrier(barrier_path):
427 while not Path(barrier_path).exists():
428 time.sleep(0.001)
429
430
431def main():
432 parser = argparse.ArgumentParser()
433 parser.add_argument("repo_root")
434 parser.add_argument("data_dir")
435 parser.add_argument("collection")
436 parser.add_argument("mode", choices=["create_batch", "update_same"])
437 parser.add_argument("payload")
438 parser.add_argument("--barrier", default="")
439 args = parser.parse_args()
440
441 sys.path.insert(0, args.repo_root)
442 import object_records
443
444 payload = json.loads(args.payload)
445 if args.barrier:
446 _wait_for_barrier(args.barrier)
447
448 if args.mode == "create_batch":
449 id_prefix = payload["id_prefix"]
450 count = payload["count"]
451 for i in range(count):
452 object_records.create_collection_record(
453 args.collection,
454 {"id": "%s-%04d" % (id_prefix, i), "value": id_prefix},
455 base_dir=args.data_dir,
456 roots=[],
457 )
458 elif args.mode == "update_same":
459 object_records.update_collection_record(
460 args.collection,
461 payload["record_id"],
462 {"value": payload["value"]},
463 base_dir=args.data_dir,
464 roots=[],
465 )
466 return 0
467
468
469if __name__ == "__main__":
470 sys.exit(main())
471'''
472
473
474def _write_worker_script(tmp_path: Path) -> Path:
475 script_path = tmp_path / "_cp_worker.py"
476 script_path.write_text(_WORKER_SCRIPT)
477 return script_path
478
479
480def _launch(script_path: Path, *args: str, barrier: Path | None = None) -> subprocess.Popen:
481 cmd = [sys.executable, str(script_path), *args]
482 if barrier is not None:
483 cmd += ["--barrier", str(barrier)]
484 return subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
485
486
487def _release_barrier_soon(barrier_path: Path, delay: float = 0.3) -> None:
488 """Give every launched subprocess time to get past interpreter startup
489 and reach its barrier-poll loop, THEN drop the barrier file so all
490 processes resume as close to simultaneously as OS scheduling allows --
491 this is what makes the probe actually exercise contention on the flock
492 instead of processes serializing naturally through staggered startup."""
493 time.sleep(delay)
494 barrier_path.write_text("go")
495
496
497@pytest.mark.parametrize("storage", ["classic", "append"])
498def test_cross_process_concurrent_distinct_creates_no_lost_writes_or_corruption(
499 tmp_path, storage
500):
501 """THE central probe: 4 separate OS processes (subprocess, not threads
502 -- each a fresh Python interpreter with its own memory, its own cold
503 _RECORDS_CACHE/_OIDX_CACHE, sharing nothing with the parent test process
504 or each other except the records.tsv/.lock/.oidx files on disk), each
505 creating 20 DISTINCT record ids into the SAME collection, released via
506 a shared barrier file so their writes genuinely overlap in time.
507
508 Property under test: is `_records_file_lock`'s fcntl.flock actually
509 effective ACROSS processes -- i.e. does it prevent two independent
510 processes' read-modify-write cycles (duplicate-id check + persist) from
511 interleaving on disk -- or, as would be the case if the mechanism were
512 a plain in-process `threading.Lock` (each process would have its OWN,
513 mutually-invisible lock object, providing zero cross-process exclusion),
514 can two processes corrupt or lose each other's writes?
515
516 RESULT: safe. flock is a kernel-level advisory lock keyed by the lock
517 file's path, not by the calling process/interpreter, so it correctly
518 serializes all 4 processes' writes: every one of the 80 records
519 survives, the file parses cleanly, and no row is garbled -- in BOTH
520 classic mode (tempfile+replace per full rewrite) and append mode
521 (in-place open("a") fast-path, which has NO atomic-rename safety net of
522 its own and depends entirely on the lock for this property)."""
523 data_dir = tmp_path / "data"
524 collection = "cp_widgets"
525 if storage == "classic":
526 write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
527 else:
528 write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
529
530 script_path = _write_worker_script(tmp_path)
531 barrier_path = tmp_path / "barrier_distinct"
532 num_procs = 4
533 per_proc = 20
534
535 procs = []
536 for p in range(num_procs):
537 payload = json.dumps({"id_prefix": f"proc{p}", "count": per_proc})
538 procs.append(
539 _launch(
540 script_path,
541 REPO_ROOT,
542 str(data_dir),
543 collection,
544 "create_batch",
545 payload,
546 barrier=barrier_path,
547 )
548 )
549
550 _release_barrier_soon(barrier_path)
551
552 outputs = []
553 for proc in procs:
554 out, _ = proc.communicate(timeout=60)
555 outputs.append((proc.returncode, out))
556
557 failures = [(rc, out) for rc, out in outputs if rc != 0]
558 assert not failures, f"worker process(es) failed: {failures!r}"
559
560 _clear_caches()
561 all_records = object_records.read_collection_records(collection, base_dir=data_dir, roots=[])
562 ids = sorted(r["id"] for r in all_records)
563 expected = sorted(
564 f"proc{p}-{i:04d}" for p in range(num_procs) for i in range(per_proc)
565 )
566 assert ids == expected, (
567 f"cross-process write loss/corruption: expected {len(expected)} records, "
568 f"got {len(ids)} ({[i for i in expected if i not in ids][:5]} missing, "
569 f"{[i for i in ids if i not in expected][:5]} unexpected)"
570 )
571 for r in all_records:
572 prefix = r["id"].rsplit("-", 1)[0]
573 assert r["value"] == prefix, f"cross-wired/corrupted row across processes: {r!r}"
574
575
576@pytest.mark.parametrize("storage", ["classic", "append"])
577def test_cross_process_concurrent_same_record_update_is_last_writer_wins_not_corruption(
578 tmp_path, storage
579):
580 """Same question as the distinct-create probe above, but for the
581 same-record UPDATE case ACROSS processes: 4 separate OS processes all
582 update the SAME existing record concurrently (barrier-released). As in
583 the same-process version of this test, there is no compare-and-set
584 (spec 63) -- so the only claim under test is that the final value is
585 EXACTLY ONE of the 4 submitted values (the lock prevented a torn/
586 interleaved write), not that no update was overwritten (some updates
587 ARE expected to be silently lost to LWW; that is unrelated to whether
588 the lock itself works). RESULT: safe from corruption -- final value is
589 always exactly one clean candidate string, never a merge/garble, in
590 both storage modes; same LWW-loses-N-1-of-N semantics as the
591 same-process case, now confirmed to hold across process boundaries
592 too."""
593 data_dir = tmp_path / "data"
594 collection = "cp_counters"
595 if storage == "classic":
596 write_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
597 else:
598 write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
599
600 object_records.create_collection_record(
601 collection, {"id": "shared", "value": "seed"}, base_dir=data_dir, roots=[]
602 )
603
604 script_path = _write_worker_script(tmp_path)
605 barrier_path = tmp_path / "barrier_update"
606 num_procs = 4
607 candidates = [f"proc-writer-{p}" for p in range(num_procs)]
608
609 procs = []
610 for p in range(num_procs):
611 payload = json.dumps({"record_id": "shared", "value": candidates[p]})
612 procs.append(
613 _launch(
614 script_path,
615 REPO_ROOT,
616 str(data_dir),
617 collection,
618 "update_same",
619 payload,
620 barrier=barrier_path,
621 )
622 )
623
624 _release_barrier_soon(barrier_path)
625
626 outputs = []
627 for proc in procs:
628 out, _ = proc.communicate(timeout=60)
629 outputs.append((proc.returncode, out))
630
631 failures = [(rc, out) for rc, out in outputs if rc != 0]
632 assert not failures, f"worker process(es) failed: {failures!r}"
633
634 _clear_caches()
635 final = object_records.get_collection_record(collection, "shared", base_dir=data_dir, roots=[])
636 assert final["value"] in candidates, (
637 f"CROSS-PROCESS CORRUPTION: final value {final['value']!r} is not any single "
638 f"submitted value -- indicates the flock did NOT serialize these processes' "
639 f"read-modify-write cycles"
640 )
641
642
643# =============================================================================
644# 5. id->offset oidx sidecar under concurrent same-process appends
645# =============================================================================
646
647
648def test_oidx_sidecar_stays_coherent_under_concurrent_appends(tmp_path, monkeypatch):
649 """Force every point op through the cold-cache id->offset sidecar path
650 (DBBASIC_RECORDS_CACHE_MAX_ROWS=0, same trick
651 test_embedded_json_lines_characterization.py's oidx tests use), then
652 run N threads each creating a distinct record into one append-mode
653 collection concurrently. Property under test: after the dust settles,
654 does the sidecar (.records.oidx) -- rebuilt/caught-up independently by
655 whichever thread happens to read it, under no lock at all on the READ
656 side -- resolve every one of the N ids to the CORRECT row via the
657 offset-indexed by-id path (get_collection_record), matching a full-fold
658 read? The sidecar is documented as "100% disposable... any reader
659 unable to make sense of it... rebuilds it with one sequential scan"
660 (object_records.py ~line 90), so even if concurrent writers left it
661 momentarily behind the file's actual size, a subsequent reader should
662 self-heal rather than serve stale/wrong offsets. RESULT: safe -- every
663 id resolves to its correct value via BOTH the by-id sidecar path and
664 the full-fold path, and the two agree, after concurrent creation."""
665 monkeypatch.setenv("DBBASIC_RECORDS_CACHE_MAX_ROWS", "0")
666 data_dir = tmp_path / "data"
667 collection = "oidx_stress"
668 write_append_schema(data_dir, collection, [ID_FIELD, VALUE_FIELD])
669 n = 24
670
671 def worker(i: int, barrier: threading.Barrier) -> None:
672 barrier.wait(timeout=10)
673 object_records.create_collection_record(
674 collection,
675 {"id": f"oi-{i:04d}", "value": f"val-{i:04d}"},
676 base_dir=data_dir,
677 roots=[],
678 )
679
680 errors = _run_threads(worker, n)
681 assert errors == [None] * n, f"unexpected exceptions: {errors}"
682
683 _clear_caches()
684 all_records = object_records.read_collection_records(collection, base_dir=data_dir, roots=[])
685 assert sorted(r["id"] for r in all_records) == sorted(f"oi-{i:04d}" for i in range(n))
686
687 for i in range(n):
688 _clear_caches()
689 by_id = object_records.get_collection_record(
690 collection, f"oi-{i:04d}", base_dir=data_dir, roots=[]
691 )
692 assert by_id["value"] == f"val-{i:04d}", (
693 f"oidx sidecar served wrong/stale data for oi-{i:04d} after concurrent "
694 f"appends: got {by_id!r}"
695 )
696
697 sidecar_path = data_dir / "collections" / collection / object_records.OIDX_FILE
698 assert sidecar_path.exists(), "sidecar was never (re)built despite over-threshold reads"