askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
scheduler_attention.py · 3.3 KB · 89 lines
| 1 | """system_scheduler_attention -- scheduled work that failed overnight. |
| 2 | |
| 3 | COUNT {} -> {count, detail} |
| 4 | |
| 5 | Severity `urgent` in the manifest, and it is the only provider on this |
| 6 | server that gets it, because every other queue on the home page is work |
| 7 | waiting for a person, and this one is the machine reporting that it |
| 8 | stopped doing its own. A failed nightly pass is invisible by |
| 9 | construction: the invoice aging that did not run raises nothing, emails |
| 10 | nobody, and looks exactly like a night with no overdue invoices. That is |
| 11 | the failure `scheduler_runs` was created to end -- runs became a RECORD |
| 12 | instead of a stdout line only ssh could reach -- and this object is the |
| 13 | half that makes somebody read them. |
| 14 | |
| 15 | A 24-hour window rather than "since you last looked", because the counts |
| 16 | have no per-user state and never should: this is a fact about the server, |
| 17 | identical for everyone, and a window that depends on who is asking is a |
| 18 | number two people can argue about. Yesterday's failure that has since |
| 19 | succeeded still counts today; a pass that is broken will keep counting |
| 20 | tomorrow, and one that was transient ages out on its own. |
| 21 | |
| 22 | The detail names how many distinct TASKS are affected, because one task |
| 23 | failing forty times is one broken pass and forty tasks failing once each |
| 24 | is an incident. |
| 25 | |
| 26 | COUNT rather than GET or POST: the verb is the contract with the daemon's |
| 27 | attention pass. A missing collection reads as zero; a genuine failure |
| 28 | raises, so the rollup records it and keeps the last count rather than |
| 29 | reporting a healthy scheduler nobody checked. |
| 30 | """ |
| 31 | |
| 32 | import os |
| 33 | from datetime import datetime, timedelta, timezone |
| 34 | |
| 35 | import object_records |
| 36 | |
| 37 | ACTOR = "system_scheduler_attention" |
| 38 | |
| 39 | WINDOW_HOURS = 24 |
| 40 | # An `ok` that is blank is unknown, not failed. Counting it would turn a |
| 41 | # hand-edited or half-written row into a fake incident, and the point of |
| 42 | # an urgent badge is that it is never crying wolf. |
| 43 | FALSE_TEXT = ("false", "0", "no", "off") |
| 44 | |
| 45 | |
| 46 | def _base_dir(): |
| 47 | return os.environ.get("DBBASIC_DATA_DIR", "data") |
| 48 | |
| 49 | |
| 50 | def _text(value): |
| 51 | return str(value if value is not None else "").strip() |
| 52 | |
| 53 | |
| 54 | def _stamp(value): |
| 55 | text = _text(value).replace("Z", "+00:00") |
| 56 | if not text: |
| 57 | return None |
| 58 | try: |
| 59 | parsed = datetime.fromisoformat(text) |
| 60 | except ValueError: |
| 61 | return None |
| 62 | return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc) |
| 63 | |
| 64 | |
| 65 | def COUNT(request): |
| 66 | try: |
| 67 | rows = object_records.read_collection_records("scheduler_runs", |
| 68 | base_dir=_base_dir()) |
| 69 | except Exception: |
| 70 | return {"count": 0} |
| 71 | |
| 72 | cutoff = datetime.now(timezone.utc) - timedelta(hours=WINDOW_HOURS) |
| 73 | failures = [] |
| 74 | for row in rows: |
| 75 | if _text(row.get("ok")).lower() not in FALSE_TEXT: |
| 76 | continue |
| 77 | # A run with no readable start time is still a run, but it cannot |
| 78 | # be placed in the window, so it is left out rather than counted |
| 79 | # forever. |
| 80 | started = _stamp(row.get("started_at")) or _stamp(row.get("created_at")) |
| 81 | if started is None or started < cutoff: |
| 82 | continue |
| 83 | failures.append(_text(row.get("task_id")) or _text(row.get("object_id"))) |
| 84 | |
| 85 | if not failures: |
| 86 | return {"count": 0} |
| 87 | tasks = len({name for name in failures if name}) |
| 88 | detail = f"across {tasks} task(s) in the last {WINDOW_HOURS}h" if tasks else "" |
| 89 | return {"count": len(failures), "detail": detail} |