askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
object_analytics.py · 18.0 KB · 405 lines
| 1 | """Analytics -- first-party traffic capture (port of q9's analytics app). |
| 2 | |
| 3 | Most platforms forgot about analytics; this one treats it as a native, |
| 4 | inspectable collection. Every non-asset HTTP request appends one row to the |
| 5 | append-mode `page_views` collection -- path, method, status, ip, user-agent, |
| 6 | referrer, session, is_owner. That gives live traffic visibility (including bot |
| 7 | attacks -- unlike q9 we deliberately DO capture 4xx, because a flood of 404s |
| 8 | from one IP is exactly what you want to see) and a rollup source for dashboards. |
| 9 | |
| 10 | Two deliberate design notes: |
| 11 | |
| 12 | * **append storage.** page_views is write-hot and log-shaped -- the textbook |
| 13 | append-mode case (docs/storage-modes.md). Every request is one O(1) append. |
| 14 | This is also the platform's heaviest write path, so it doubles as the stress |
| 15 | test for massive-file handling and for the retention/rotation pass |
| 16 | (object_daemon.process_analytics_retention) -- id-fold compaction can't shrink |
| 17 | a pure event log (nothing is superseded), so retention is a time-windowed |
| 18 | rewrite, and page_views is what exercises it. |
| 19 | |
| 20 | * **off by default.** Capturing a row per request is an operator choice, gated |
| 21 | by `DBBASIC_ANALYTICS` (env), so a deploy never silently starts writing. |
| 22 | |
| 23 | * **the visitor cookie is off by default too, and separately.** |
| 24 | `DBBASIC_ANALYTICS_VISITOR_COOKIE` gates the one thing here that touches a |
| 25 | visitor's device. Unset -- the default -- this module never asks a browser |
| 26 | to remember anything, which is the posture that needs no consent banner |
| 27 | anywhere in the world. Two switches rather than one because a log and a |
| 28 | device identifier are different acts: an operator who wants traffic numbers |
| 29 | should not have to take on a consent obligation to get them. |
| 30 | |
| 31 | This module is the pure/testable half: config, the skip-path rule, the visitor |
| 32 | cookie decision, and building the record. The daemon owns retention; |
| 33 | object_server owns the capture hook and the one line that puts the cookie on a |
| 34 | response. |
| 35 | """ |
| 36 | |
| 37 | from __future__ import annotations |
| 38 | |
| 39 | import os |
| 40 | import secrets |
| 41 | from typing import Any, Mapping |
| 42 | |
| 43 | import object_visitors |
| 44 | |
| 45 | PAGE_VIEWS_COLLECTION = "page_views" |
| 46 | |
| 47 | ANALYTICS_ENABLED_ENV = "DBBASIC_ANALYTICS" |
| 48 | OWNER_IPS_ENV = "DBBASIC_ANALYTICS_OWNER_IPS" |
| 49 | RETENTION_DAYS_ENV = "DBBASIC_ANALYTICS_RETENTION_DAYS" |
| 50 | DEFAULT_RETENTION_DAYS = 30 |
| 51 | # A second bound, because time alone does not bound a log. The retention |
| 52 | # window says how far BACK to keep; it says nothing about how much can |
| 53 | # arrive inside it, so a bot storm or a busy deploy day can fill a small |
| 54 | # machine's disk and memory without a single row aging out. This is the |
| 55 | # cap that was missing when a demo box hit 675MB resident and started |
| 56 | # swapping: 30 days of retention, faithfully applied, on a log nobody had |
| 57 | # told how big it was allowed to get. |
| 58 | MAX_ROWS_ENV = "DBBASIC_ANALYTICS_MAX_ROWS" |
| 59 | DEFAULT_MAX_ROWS = 200_000 |
| 60 | |
| 61 | # Prefixes that are asset/infra/polling noise, not real page hits. NOTE we do |
| 62 | # NOT skip `/api/` (bots hammer APIs -- that traffic is the point) nor 4xx/5xx |
| 63 | # (a 404 flood is the signal). Only genuinely uninteresting paths are dropped. |
| 64 | SKIP_PREFIXES = ( |
| 65 | "/static/", "/assets/", "/favicon", "/apple-touch-icon", |
| 66 | "/.well-known/", "/robots.txt", "/sitemap", |
| 67 | "/metrics", "/healthz", "/health", "/ping", |
| 68 | "/realtime", "/ws", "/__", |
| 69 | ) |
| 70 | |
| 71 | # --- the visitor cookie ------------------------------------------------------ |
| 72 | # |
| 73 | # The one thing this server asks a browser to remember. Named like the |
| 74 | # identity cookie's sibling (`dbbasic_session`) because that is what it is: |
| 75 | # same first-party, same-site, HttpOnly posture, and deliberately NOT the |
| 76 | # same string. See docs/analytics.md "The visitor cookie" for the five |
| 77 | # rules; every one of them is enforced below rather than merely written |
| 78 | # down, because rule 4 in particular is one line of code away from being |
| 79 | # broken at all times. |
| 80 | VISITOR_COOKIE_NAME = "dbbasic_visitor" |
| 81 | |
| 82 | # --- and it is OFF unless an operator turns it on ---------------------------- |
| 83 | # |
| 84 | # The whole consent problem on this server is one cookie. Everything else |
| 85 | # it stores on a device is strictly necessary for something the visitor |
| 86 | # asked for -- the session they signed into, the basket they filled -- and |
| 87 | # server-side logging is not device storage at all. So the default that |
| 88 | # needs the least law is the one where this cookie does not exist: |
| 89 | # `page_views` still counts visitors by address, nothing is written to |
| 90 | # anybody's browser, and the ePrivacy consent trigger (storing a |
| 91 | # non-essential identifier on a device) never fires anywhere in the world. |
| 92 | # |
| 93 | # Turning it ON is therefore a deliberate operator act, and it is the |
| 94 | # moment the obligation appears -- so the obligation travels WITH the |
| 95 | # setting rather than living in a document: OBLIGATION below is printed on |
| 96 | # the daemon's and the server's boot lines, in words, by whoever is |
| 97 | # starting the process. An operator who flips this should not have to |
| 98 | # already know what they have taken on. |
| 99 | VISITOR_COOKIE_ENV = "DBBASIC_ANALYTICS_VISITOR_COOKIE" |
| 100 | |
| 101 | OBLIGATION = ( |
| 102 | "stores an identifier on a visitor's device; " |
| 103 | "in the EU/UK that requires consent" |
| 104 | ) |
| 105 | NO_OBLIGATION = ( |
| 106 | "no identifier is stored on any device; visitors are counted by address" |
| 107 | ) |
| 108 | |
| 109 | # What `build_page_view` read before this cookie existed: a cookie literally |
| 110 | # named `session_id`, which nothing in this repo has ever set. Kept as a |
| 111 | # READ-ONLY fallback so a box that acquired one from somewhere still folds, |
| 112 | # and never written -- a second name being set would be two populations of |
| 113 | # visitors that cannot be compared. |
| 114 | LEGACY_VISITOR_COOKIE_NAME = "session_id" |
| 115 | |
| 116 | VISITOR_DAYS_ENV = "DBBASIC_ANALYTICS_VISITOR_DAYS" |
| 117 | # Six months: long enough to see a return visit, short enough to lapse |
| 118 | # rather than follow somebody for years. "Indefinite" is how a session |
| 119 | # identifier quietly becomes a permanent one. |
| 120 | DEFAULT_VISITOR_DAYS = 180 |
| 121 | |
| 122 | # Refusals this server honours. Both mean the same thing in different |
| 123 | # vocabularies, and both mean NO COOKIE -- not a shorter one, not an |
| 124 | # anonymised one. Those visitors are still counted by the IP rule; they |
| 125 | # simply appear new on every visit, which is the correct consequence of |
| 126 | # having asked not to be remembered. |
| 127 | DO_NOT_TRACK_HEADERS = ("dnt", "sec-gpc") |
| 128 | |
| 129 | _TRUE = {"1", "true", "yes", "on"} |
| 130 | _MAX_UA = 500 |
| 131 | _MAX_REFERRER = 255 |
| 132 | _MAX_PATH = 500 |
| 133 | _MAX_HOST = 253 # the longest a DNS name can legally be |
| 134 | |
| 135 | |
| 136 | def analytics_enabled(env: Mapping[str, str] | None = None) -> bool: |
| 137 | """Off unless DBBASIC_ANALYTICS is truthy -- a deliberate operator opt-in, |
| 138 | since it adds a write to every request.""" |
| 139 | env = os.environ if env is None else env |
| 140 | return (env.get(ANALYTICS_ENABLED_ENV) or "").strip().lower() in _TRUE |
| 141 | |
| 142 | |
| 143 | def owner_ips(env: Mapping[str, str] | None = None) -> frozenset[str]: |
| 144 | """IPs whose traffic is flagged `is_owner` so reports can exclude it |
| 145 | (DBBASIC_ANALYTICS_OWNER_IPS, comma-separated).""" |
| 146 | env = os.environ if env is None else env |
| 147 | raw = env.get(OWNER_IPS_ENV) or "" |
| 148 | return frozenset(part.strip() for part in raw.split(",") if part.strip()) |
| 149 | |
| 150 | |
| 151 | def retention_days(env: Mapping[str, str] | None = None) -> int: |
| 152 | env = os.environ if env is None else env |
| 153 | try: |
| 154 | value = int(str(env.get(RETENTION_DAYS_ENV, "")).strip()) |
| 155 | except (TypeError, ValueError): |
| 156 | return DEFAULT_RETENTION_DAYS |
| 157 | return value if value > 0 else DEFAULT_RETENTION_DAYS |
| 158 | |
| 159 | |
| 160 | def max_rows(env: Mapping[str, str] | None = None) -> int: |
| 161 | """Hard cap on stored page_views rows; 0 disables the cap.""" |
| 162 | env = os.environ if env is None else env |
| 163 | try: |
| 164 | value = int(str(env.get(MAX_ROWS_ENV, "")).strip()) |
| 165 | except (TypeError, ValueError): |
| 166 | return DEFAULT_MAX_ROWS |
| 167 | return value if value >= 0 else DEFAULT_MAX_ROWS |
| 168 | |
| 169 | |
| 170 | def should_capture(path: str) -> bool: |
| 171 | """True for a real page hit -- not an asset/infra/polling path.""" |
| 172 | p = path or "/" |
| 173 | return not p.startswith(SKIP_PREFIXES) |
| 174 | |
| 175 | |
| 176 | def visitor_days(env: Mapping[str, str] | None = None) -> int: |
| 177 | """How long the visitor cookie lives, in days (DBBASIC_ANALYTICS_VISITOR_DAYS). |
| 178 | |
| 179 | A stated expiry is rule 3, so this cannot return "forever": a junk or |
| 180 | non-positive value falls back to the default rather than being treated |
| 181 | as unbounded. |
| 182 | """ |
| 183 | env = os.environ if env is None else env |
| 184 | try: |
| 185 | value = int(str(env.get(VISITOR_DAYS_ENV, "")).strip()) |
| 186 | except (TypeError, ValueError): |
| 187 | return DEFAULT_VISITOR_DAYS |
| 188 | return value if value > 0 else DEFAULT_VISITOR_DAYS |
| 189 | |
| 190 | |
| 191 | def visitor_cookie_enabled(env: Mapping[str, str] | None = None) -> bool: |
| 192 | """Whether this server may set the visitor cookie at all. |
| 193 | |
| 194 | Off unless DBBASIC_ANALYTICS_VISITOR_COOKIE is truthy. Unset, the |
| 195 | server behaves exactly as it did before the cookie existed: IP-based |
| 196 | visitor counting, nothing stored on anybody's device, and no consent |
| 197 | trigger. New-versus-returning is the one thing that becomes |
| 198 | unavailable, and that is the trade -- the same trade Plausible and |
| 199 | Fathom sell as a feature. |
| 200 | |
| 201 | Deliberately a separate switch from DBBASIC_ANALYTICS: recording what |
| 202 | the server did (a log) and asking a browser to remember who it is |
| 203 | (device storage) are different acts under different instruments, and |
| 204 | one flag for both would mean an operator who wanted traffic numbers |
| 205 | could not have them without also taking on a consent obligation. |
| 206 | """ |
| 207 | env = os.environ if env is None else env |
| 208 | return (env.get(VISITOR_COOKIE_ENV) or "").strip().lower() in _TRUE |
| 209 | |
| 210 | |
| 211 | def visitor_cookie_posture(env: Mapping[str, str] | None = None) -> str: |
| 212 | """One line for a boot banner, stating the obligation rather than the |
| 213 | setting. `DBBASIC_ANALYTICS_VISITOR_COOKIE=on` is not information; what |
| 214 | it commits the operator to is.""" |
| 215 | if not visitor_cookie_enabled(env): |
| 216 | return f"visitor cookie off (default) -- {NO_OBLIGATION}" |
| 217 | return (f"visitor cookie ON ({visitor_days(env)}d, {VISITOR_COOKIE_NAME}) " |
| 218 | f"-- {OBLIGATION}") |
| 219 | |
| 220 | |
| 221 | def _cookie_value(cookie_header: str, name: str) -> str: |
| 222 | for part in (cookie_header or "").split(";"): |
| 223 | key, _, value = part.strip().partition("=") |
| 224 | if key == name: |
| 225 | return value |
| 226 | return "" |
| 227 | |
| 228 | |
| 229 | def refuses_tracking(headers: Mapping[str, str]) -> bool: |
| 230 | """True when this request asked not to be remembered. |
| 231 | |
| 232 | `DNT: 1` and `Sec-GPC: 1` are the two ways a browser says it, and the |
| 233 | answer here is the same for both: no cookie is set, ever. Only the |
| 234 | exact value "1" counts -- `DNT: 0` is a positive statement of consent |
| 235 | and an absent header is no statement at all, and reading either as a |
| 236 | refusal would mean this server never remembered anybody. |
| 237 | """ |
| 238 | for name in DO_NOT_TRACK_HEADERS: |
| 239 | if str(headers.get(name) or "").strip() == "1": |
| 240 | return True |
| 241 | return False |
| 242 | |
| 243 | |
| 244 | def visitor_token(headers: Mapping[str, str]) -> str: |
| 245 | """The visitor token this request already carries, or "". |
| 246 | |
| 247 | Reads the current name first and the legacy `session_id` cookie second, |
| 248 | so a box that has one of the old ones keeps its thread instead of being |
| 249 | counted as a stranger on the day this shipped. |
| 250 | """ |
| 251 | cookie_header = headers.get("cookie") or "" |
| 252 | return (_cookie_value(cookie_header, VISITOR_COOKIE_NAME) |
| 253 | or _cookie_value(cookie_header, LEGACY_VISITOR_COOKIE_NAME)) |
| 254 | |
| 255 | |
| 256 | def new_visitor_token() -> str: |
| 257 | """An opaque token. It says *this browser has been here before* and |
| 258 | nothing else: no name, no email, no account, no meaning outside this |
| 259 | site's own logs, and nothing derived from the request that could be |
| 260 | reversed into one (rule 2). A fingerprint hashed from IP + user agent |
| 261 | would be a stable cross-visit identifier nobody could clear, which is |
| 262 | the thing this deliberately is not.""" |
| 263 | return secrets.token_urlsafe(16) |
| 264 | |
| 265 | |
| 266 | def should_set_visitor_cookie( |
| 267 | path: str, headers: Mapping[str, str], *, |
| 268 | env: Mapping[str, str] | None = None, |
| 269 | ) -> bool: |
| 270 | """Whether THIS response should mint a visitor cookie. |
| 271 | |
| 272 | Every clause is one of the rules, in the order they can refuse: |
| 273 | |
| 274 | * the cookie is not switched on. This is the FIRST refusal and the |
| 275 | default one: with DBBASIC_ANALYTICS_VISITOR_COOKIE unset nothing is |
| 276 | ever stored on a visitor's device, so there is no consent question |
| 277 | to answer. Every other clause below is about a visitor who could |
| 278 | have been given a cookie; this one is about a server that does not |
| 279 | hand them out. |
| 280 | * analytics is off -- nothing is being recorded, so a cookie would be |
| 281 | an identifier collected for no purpose at all, which is the worst |
| 282 | possible trade. |
| 283 | * the request refused (DNT / Sec-GPC). |
| 284 | * it already has one; a second would reset the thread on every page. |
| 285 | * it is not a page. `should_capture` drops assets and health checks, |
| 286 | `object_visitors.is_page_path` drops the API and collection surfaces |
| 287 | a script talks to. An API client is not a browser and cannot be a |
| 288 | returning visitor; handing it a cookie only writes an identifier |
| 289 | into somebody's automation. |
| 290 | |
| 291 | No status gate on purpose: a 404 is a real page to the person who |
| 292 | typed the URL, and it is quite often the FIRST thing a visitor sees. |
| 293 | Refusing to thread that visit would lose exactly the journey worth |
| 294 | knowing about. |
| 295 | """ |
| 296 | if not visitor_cookie_enabled(env): |
| 297 | return False |
| 298 | if not analytics_enabled(env): |
| 299 | return False |
| 300 | if refuses_tracking(headers): |
| 301 | return False |
| 302 | if visitor_token(headers): |
| 303 | return False |
| 304 | return should_capture(path) and object_visitors.is_page_path(path) |
| 305 | |
| 306 | |
| 307 | def visitor_cookie_header( |
| 308 | token: str, *, days: int | None = None, secure: bool = True, |
| 309 | ) -> str: |
| 310 | """The `set-cookie` value for a minted visitor token. |
| 311 | |
| 312 | First-party, same-site, HttpOnly, path-wide, with a stated Max-Age -- |
| 313 | rules 1 and 3 written as attributes. HttpOnly matters more here than |
| 314 | it looks: nothing on the page needs to read this, so anything that |
| 315 | CAN read it is either an XSS payload or a third-party script, and both |
| 316 | would turn a first-party counter into somebody else's identifier. |
| 317 | """ |
| 318 | days = visitor_days() if days is None else days |
| 319 | attributes = [ |
| 320 | f"{VISITOR_COOKIE_NAME}={token}", |
| 321 | "Path=/", |
| 322 | "HttpOnly", |
| 323 | "SameSite=Lax", |
| 324 | f"Max-Age={int(days) * 86400}", |
| 325 | ] |
| 326 | if secure: |
| 327 | attributes.append("Secure") |
| 328 | return "; ".join(attributes) |
| 329 | |
| 330 | |
| 331 | def build_page_view( |
| 332 | *, path: str, method: str, status: int, ip: str, |
| 333 | headers: Mapping[str, str], owners: frozenset[str], |
| 334 | user_id: str = "", is_operator: bool = False, |
| 335 | minted_visitor_token: str = "", |
| 336 | ) -> dict[str, str]: |
| 337 | """The page_views record one request implies. Pure -- created_at is stamped |
| 338 | by the record layer on write. |
| 339 | |
| 340 | ## The visitor token and `user_id` are NEVER both written |
| 341 | |
| 342 | Read this before adding the join, because the join is one line and it |
| 343 | will look reasonable. `session_id` here is the anonymous visitor |
| 344 | cookie: an opaque token that means "this browser has been here |
| 345 | before". `user_id` is an account. A row carrying BOTH is the join -- |
| 346 | it de-anonymises every earlier row with that token, retroactively, |
| 347 | for a person who was never asked. That is the exact move that turns |
| 348 | analytics into surveillance (docs/analytics.md, cookie rule 4), and |
| 349 | the reason it is enforced here rather than left to judgement is that |
| 350 | the enforcement has to survive somebody who has not read the doc. |
| 351 | |
| 352 | So: when a row carries a visitor token, `user_id` is dropped. A |
| 353 | signed-in member's traffic is still counted -- it is still a row, with |
| 354 | a path, a status and a token -- it simply is not labelled with who |
| 355 | they are. Where there is no token (a visitor who sent DNT, an API |
| 356 | call, a browser that refuses cookies) `user_id` is written as before, |
| 357 | because there is nothing for it to be correlated WITH. |
| 358 | |
| 359 | `minted_visitor_token` is the cookie this very response is setting: |
| 360 | the browser has not sent it back yet, so without it the first page of |
| 361 | every visit would be threadless and every funnel would start one step |
| 362 | late. |
| 363 | |
| 364 | `is_owner` marks traffic that is the SITE'S OWN rather than a visitor's, |
| 365 | so reports can exclude it (every analytics rollup filters on it). Two |
| 366 | things set it, and the second was missing for a long time: |
| 367 | |
| 368 | * the request came from an owner IP (cheap, no auth lookup on the hot |
| 369 | path), and |
| 370 | * `is_operator` -- the caller authenticated as the operator, which the |
| 371 | server decides from the admin token. |
| 372 | |
| 373 | Without the second, every deploy call, every package install and every |
| 374 | scripted `POST /objects/...` counted as a visitor page view. On one real |
| 375 | box that produced a "Top paths" report where 45% of the rows had been hit |
| 376 | exactly once and the contents were vulnerability scanners and the |
| 377 | operator's own automation -- a report of the observer, not the observed. |
| 378 | |
| 379 | Note what this deliberately does NOT flag: a signed-in member browsing |
| 380 | the site. That is real traffic and worth measuring; which pages members |
| 381 | actually use is among the most useful things this collection knows. The |
| 382 | line is between somebody USING the site and somebody OPERATING it. |
| 383 | """ |
| 384 | ua = (headers.get("user-agent") or "")[:_MAX_UA] |
| 385 | referrer = (headers.get("referer") or headers.get("referrer") or "")[:_MAX_REFERRER] |
| 386 | session_id = minted_visitor_token or visitor_token(headers) |
| 387 | return { |
| 388 | # Which site was asked. One process here serves both a marketing |
| 389 | # domain and the app, and without this their visitors are one |
| 390 | # number that answers neither question. Port stripped so |
| 391 | # example.com and example.com:443 are one host rather than two. |
| 392 | "host": (headers.get("host") or "").split(":")[0].lower()[:_MAX_HOST], |
| 393 | "path": (path or "/")[:_MAX_PATH], |
| 394 | "method": (method or "GET").upper(), |
| 395 | "status": str(int(status)), |
| 396 | "ip": ip or "", |
| 397 | "user_agent": ua, |
| 398 | "referrer": referrer, |
| 399 | "session_id": session_id, |
| 400 | # Rule 4, enforced rather than documented: never both. See the |
| 401 | # docstring above -- the two columns exist, and a row holding both |
| 402 | # is the join that de-anonymises everything the token ever did. |
| 403 | "user_id": "" if session_id else (user_id or ""), |
| 404 | "is_owner": "true" if (is_operator or ip in owners) else "false", |
| 405 | } |