Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

dbbasic-object-server/object_analytics.py
Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
object_analytics.py · 18.0 KB · 405 lines
1"""Analytics -- first-party traffic capture (port of q9's analytics app).
2
3Most platforms forgot about analytics; this one treats it as a native,
4inspectable collection. Every non-asset HTTP request appends one row to the
5append-mode `page_views` collection -- path, method, status, ip, user-agent,
6referrer, session, is_owner. That gives live traffic visibility (including bot
7attacks -- unlike q9 we deliberately DO capture 4xx, because a flood of 404s
8from one IP is exactly what you want to see) and a rollup source for dashboards.
9
10Two deliberate design notes:
11
12* **append storage.** page_views is write-hot and log-shaped -- the textbook
13 append-mode case (docs/storage-modes.md). Every request is one O(1) append.
14 This is also the platform's heaviest write path, so it doubles as the stress
15 test for massive-file handling and for the retention/rotation pass
16 (object_daemon.process_analytics_retention) -- id-fold compaction can't shrink
17 a pure event log (nothing is superseded), so retention is a time-windowed
18 rewrite, and page_views is what exercises it.
19
20* **off by default.** Capturing a row per request is an operator choice, gated
21 by `DBBASIC_ANALYTICS` (env), so a deploy never silently starts writing.
22
23* **the visitor cookie is off by default too, and separately.**
24 `DBBASIC_ANALYTICS_VISITOR_COOKIE` gates the one thing here that touches a
25 visitor's device. Unset -- the default -- this module never asks a browser
26 to remember anything, which is the posture that needs no consent banner
27 anywhere in the world. Two switches rather than one because a log and a
28 device identifier are different acts: an operator who wants traffic numbers
29 should not have to take on a consent obligation to get them.
30
31This module is the pure/testable half: config, the skip-path rule, the visitor
32cookie decision, and building the record. The daemon owns retention;
33object_server owns the capture hook and the one line that puts the cookie on a
34response.
35"""
36
37from __future__ import annotations
38
39import os
40import secrets
41from typing import Any, Mapping
42
43import object_visitors
44
45PAGE_VIEWS_COLLECTION = "page_views"
46
47ANALYTICS_ENABLED_ENV = "DBBASIC_ANALYTICS"
48OWNER_IPS_ENV = "DBBASIC_ANALYTICS_OWNER_IPS"
49RETENTION_DAYS_ENV = "DBBASIC_ANALYTICS_RETENTION_DAYS"
50DEFAULT_RETENTION_DAYS = 30
51# A second bound, because time alone does not bound a log. The retention
52# window says how far BACK to keep; it says nothing about how much can
53# arrive inside it, so a bot storm or a busy deploy day can fill a small
54# machine's disk and memory without a single row aging out. This is the
55# cap that was missing when a demo box hit 675MB resident and started
56# swapping: 30 days of retention, faithfully applied, on a log nobody had
57# told how big it was allowed to get.
58MAX_ROWS_ENV = "DBBASIC_ANALYTICS_MAX_ROWS"
59DEFAULT_MAX_ROWS = 200_000
60
61# Prefixes that are asset/infra/polling noise, not real page hits. NOTE we do
62# NOT skip `/api/` (bots hammer APIs -- that traffic is the point) nor 4xx/5xx
63# (a 404 flood is the signal). Only genuinely uninteresting paths are dropped.
64SKIP_PREFIXES = (
65 "/static/", "/assets/", "/favicon", "/apple-touch-icon",
66 "/.well-known/", "/robots.txt", "/sitemap",
67 "/metrics", "/healthz", "/health", "/ping",
68 "/realtime", "/ws", "/__",
69)
70
71# --- the visitor cookie ------------------------------------------------------
72#
73# The one thing this server asks a browser to remember. Named like the
74# identity cookie's sibling (`dbbasic_session`) because that is what it is:
75# same first-party, same-site, HttpOnly posture, and deliberately NOT the
76# same string. See docs/analytics.md "The visitor cookie" for the five
77# rules; every one of them is enforced below rather than merely written
78# down, because rule 4 in particular is one line of code away from being
79# broken at all times.
80VISITOR_COOKIE_NAME = "dbbasic_visitor"
81
82# --- and it is OFF unless an operator turns it on ----------------------------
83#
84# The whole consent problem on this server is one cookie. Everything else
85# it stores on a device is strictly necessary for something the visitor
86# asked for -- the session they signed into, the basket they filled -- and
87# server-side logging is not device storage at all. So the default that
88# needs the least law is the one where this cookie does not exist:
89# `page_views` still counts visitors by address, nothing is written to
90# anybody's browser, and the ePrivacy consent trigger (storing a
91# non-essential identifier on a device) never fires anywhere in the world.
92#
93# Turning it ON is therefore a deliberate operator act, and it is the
94# moment the obligation appears -- so the obligation travels WITH the
95# setting rather than living in a document: OBLIGATION below is printed on
96# the daemon's and the server's boot lines, in words, by whoever is
97# starting the process. An operator who flips this should not have to
98# already know what they have taken on.
99VISITOR_COOKIE_ENV = "DBBASIC_ANALYTICS_VISITOR_COOKIE"
100
101OBLIGATION = (
102 "stores an identifier on a visitor's device; "
103 "in the EU/UK that requires consent"
104)
105NO_OBLIGATION = (
106 "no identifier is stored on any device; visitors are counted by address"
107)
108
109# What `build_page_view` read before this cookie existed: a cookie literally
110# named `session_id`, which nothing in this repo has ever set. Kept as a
111# READ-ONLY fallback so a box that acquired one from somewhere still folds,
112# and never written -- a second name being set would be two populations of
113# visitors that cannot be compared.
114LEGACY_VISITOR_COOKIE_NAME = "session_id"
115
116VISITOR_DAYS_ENV = "DBBASIC_ANALYTICS_VISITOR_DAYS"
117# Six months: long enough to see a return visit, short enough to lapse
118# rather than follow somebody for years. "Indefinite" is how a session
119# identifier quietly becomes a permanent one.
120DEFAULT_VISITOR_DAYS = 180
121
122# Refusals this server honours. Both mean the same thing in different
123# vocabularies, and both mean NO COOKIE -- not a shorter one, not an
124# anonymised one. Those visitors are still counted by the IP rule; they
125# simply appear new on every visit, which is the correct consequence of
126# having asked not to be remembered.
127DO_NOT_TRACK_HEADERS = ("dnt", "sec-gpc")
128
129_TRUE = {"1", "true", "yes", "on"}
130_MAX_UA = 500
131_MAX_REFERRER = 255
132_MAX_PATH = 500
133_MAX_HOST = 253 # the longest a DNS name can legally be
134
135
136def analytics_enabled(env: Mapping[str, str] | None = None) -> bool:
137 """Off unless DBBASIC_ANALYTICS is truthy -- a deliberate operator opt-in,
138 since it adds a write to every request."""
139 env = os.environ if env is None else env
140 return (env.get(ANALYTICS_ENABLED_ENV) or "").strip().lower() in _TRUE
141
142
143def owner_ips(env: Mapping[str, str] | None = None) -> frozenset[str]:
144 """IPs whose traffic is flagged `is_owner` so reports can exclude it
145 (DBBASIC_ANALYTICS_OWNER_IPS, comma-separated)."""
146 env = os.environ if env is None else env
147 raw = env.get(OWNER_IPS_ENV) or ""
148 return frozenset(part.strip() for part in raw.split(",") if part.strip())
149
150
151def retention_days(env: Mapping[str, str] | None = None) -> int:
152 env = os.environ if env is None else env
153 try:
154 value = int(str(env.get(RETENTION_DAYS_ENV, "")).strip())
155 except (TypeError, ValueError):
156 return DEFAULT_RETENTION_DAYS
157 return value if value > 0 else DEFAULT_RETENTION_DAYS
158
159
160def max_rows(env: Mapping[str, str] | None = None) -> int:
161 """Hard cap on stored page_views rows; 0 disables the cap."""
162 env = os.environ if env is None else env
163 try:
164 value = int(str(env.get(MAX_ROWS_ENV, "")).strip())
165 except (TypeError, ValueError):
166 return DEFAULT_MAX_ROWS
167 return value if value >= 0 else DEFAULT_MAX_ROWS
168
169
170def should_capture(path: str) -> bool:
171 """True for a real page hit -- not an asset/infra/polling path."""
172 p = path or "/"
173 return not p.startswith(SKIP_PREFIXES)
174
175
176def visitor_days(env: Mapping[str, str] | None = None) -> int:
177 """How long the visitor cookie lives, in days (DBBASIC_ANALYTICS_VISITOR_DAYS).
178
179 A stated expiry is rule 3, so this cannot return "forever": a junk or
180 non-positive value falls back to the default rather than being treated
181 as unbounded.
182 """
183 env = os.environ if env is None else env
184 try:
185 value = int(str(env.get(VISITOR_DAYS_ENV, "")).strip())
186 except (TypeError, ValueError):
187 return DEFAULT_VISITOR_DAYS
188 return value if value > 0 else DEFAULT_VISITOR_DAYS
189
190
191def visitor_cookie_enabled(env: Mapping[str, str] | None = None) -> bool:
192 """Whether this server may set the visitor cookie at all.
193
194 Off unless DBBASIC_ANALYTICS_VISITOR_COOKIE is truthy. Unset, the
195 server behaves exactly as it did before the cookie existed: IP-based
196 visitor counting, nothing stored on anybody's device, and no consent
197 trigger. New-versus-returning is the one thing that becomes
198 unavailable, and that is the trade -- the same trade Plausible and
199 Fathom sell as a feature.
200
201 Deliberately a separate switch from DBBASIC_ANALYTICS: recording what
202 the server did (a log) and asking a browser to remember who it is
203 (device storage) are different acts under different instruments, and
204 one flag for both would mean an operator who wanted traffic numbers
205 could not have them without also taking on a consent obligation.
206 """
207 env = os.environ if env is None else env
208 return (env.get(VISITOR_COOKIE_ENV) or "").strip().lower() in _TRUE
209
210
211def visitor_cookie_posture(env: Mapping[str, str] | None = None) -> str:
212 """One line for a boot banner, stating the obligation rather than the
213 setting. `DBBASIC_ANALYTICS_VISITOR_COOKIE=on` is not information; what
214 it commits the operator to is."""
215 if not visitor_cookie_enabled(env):
216 return f"visitor cookie off (default) -- {NO_OBLIGATION}"
217 return (f"visitor cookie ON ({visitor_days(env)}d, {VISITOR_COOKIE_NAME}) "
218 f"-- {OBLIGATION}")
219
220
221def _cookie_value(cookie_header: str, name: str) -> str:
222 for part in (cookie_header or "").split(";"):
223 key, _, value = part.strip().partition("=")
224 if key == name:
225 return value
226 return ""
227
228
229def refuses_tracking(headers: Mapping[str, str]) -> bool:
230 """True when this request asked not to be remembered.
231
232 `DNT: 1` and `Sec-GPC: 1` are the two ways a browser says it, and the
233 answer here is the same for both: no cookie is set, ever. Only the
234 exact value "1" counts -- `DNT: 0` is a positive statement of consent
235 and an absent header is no statement at all, and reading either as a
236 refusal would mean this server never remembered anybody.
237 """
238 for name in DO_NOT_TRACK_HEADERS:
239 if str(headers.get(name) or "").strip() == "1":
240 return True
241 return False
242
243
244def visitor_token(headers: Mapping[str, str]) -> str:
245 """The visitor token this request already carries, or "".
246
247 Reads the current name first and the legacy `session_id` cookie second,
248 so a box that has one of the old ones keeps its thread instead of being
249 counted as a stranger on the day this shipped.
250 """
251 cookie_header = headers.get("cookie") or ""
252 return (_cookie_value(cookie_header, VISITOR_COOKIE_NAME)
253 or _cookie_value(cookie_header, LEGACY_VISITOR_COOKIE_NAME))
254
255
256def new_visitor_token() -> str:
257 """An opaque token. It says *this browser has been here before* and
258 nothing else: no name, no email, no account, no meaning outside this
259 site's own logs, and nothing derived from the request that could be
260 reversed into one (rule 2). A fingerprint hashed from IP + user agent
261 would be a stable cross-visit identifier nobody could clear, which is
262 the thing this deliberately is not."""
263 return secrets.token_urlsafe(16)
264
265
266def should_set_visitor_cookie(
267 path: str, headers: Mapping[str, str], *,
268 env: Mapping[str, str] | None = None,
269) -> bool:
270 """Whether THIS response should mint a visitor cookie.
271
272 Every clause is one of the rules, in the order they can refuse:
273
274 * the cookie is not switched on. This is the FIRST refusal and the
275 default one: with DBBASIC_ANALYTICS_VISITOR_COOKIE unset nothing is
276 ever stored on a visitor's device, so there is no consent question
277 to answer. Every other clause below is about a visitor who could
278 have been given a cookie; this one is about a server that does not
279 hand them out.
280 * analytics is off -- nothing is being recorded, so a cookie would be
281 an identifier collected for no purpose at all, which is the worst
282 possible trade.
283 * the request refused (DNT / Sec-GPC).
284 * it already has one; a second would reset the thread on every page.
285 * it is not a page. `should_capture` drops assets and health checks,
286 `object_visitors.is_page_path` drops the API and collection surfaces
287 a script talks to. An API client is not a browser and cannot be a
288 returning visitor; handing it a cookie only writes an identifier
289 into somebody's automation.
290
291 No status gate on purpose: a 404 is a real page to the person who
292 typed the URL, and it is quite often the FIRST thing a visitor sees.
293 Refusing to thread that visit would lose exactly the journey worth
294 knowing about.
295 """
296 if not visitor_cookie_enabled(env):
297 return False
298 if not analytics_enabled(env):
299 return False
300 if refuses_tracking(headers):
301 return False
302 if visitor_token(headers):
303 return False
304 return should_capture(path) and object_visitors.is_page_path(path)
305
306
307def visitor_cookie_header(
308 token: str, *, days: int | None = None, secure: bool = True,
309) -> str:
310 """The `set-cookie` value for a minted visitor token.
311
312 First-party, same-site, HttpOnly, path-wide, with a stated Max-Age --
313 rules 1 and 3 written as attributes. HttpOnly matters more here than
314 it looks: nothing on the page needs to read this, so anything that
315 CAN read it is either an XSS payload or a third-party script, and both
316 would turn a first-party counter into somebody else's identifier.
317 """
318 days = visitor_days() if days is None else days
319 attributes = [
320 f"{VISITOR_COOKIE_NAME}={token}",
321 "Path=/",
322 "HttpOnly",
323 "SameSite=Lax",
324 f"Max-Age={int(days) * 86400}",
325 ]
326 if secure:
327 attributes.append("Secure")
328 return "; ".join(attributes)
329
330
331def build_page_view(
332 *, path: str, method: str, status: int, ip: str,
333 headers: Mapping[str, str], owners: frozenset[str],
334 user_id: str = "", is_operator: bool = False,
335 minted_visitor_token: str = "",
336) -> dict[str, str]:
337 """The page_views record one request implies. Pure -- created_at is stamped
338 by the record layer on write.
339
340 ## The visitor token and `user_id` are NEVER both written
341
342 Read this before adding the join, because the join is one line and it
343 will look reasonable. `session_id` here is the anonymous visitor
344 cookie: an opaque token that means "this browser has been here
345 before". `user_id` is an account. A row carrying BOTH is the join --
346 it de-anonymises every earlier row with that token, retroactively,
347 for a person who was never asked. That is the exact move that turns
348 analytics into surveillance (docs/analytics.md, cookie rule 4), and
349 the reason it is enforced here rather than left to judgement is that
350 the enforcement has to survive somebody who has not read the doc.
351
352 So: when a row carries a visitor token, `user_id` is dropped. A
353 signed-in member's traffic is still counted -- it is still a row, with
354 a path, a status and a token -- it simply is not labelled with who
355 they are. Where there is no token (a visitor who sent DNT, an API
356 call, a browser that refuses cookies) `user_id` is written as before,
357 because there is nothing for it to be correlated WITH.
358
359 `minted_visitor_token` is the cookie this very response is setting:
360 the browser has not sent it back yet, so without it the first page of
361 every visit would be threadless and every funnel would start one step
362 late.
363
364 `is_owner` marks traffic that is the SITE'S OWN rather than a visitor's,
365 so reports can exclude it (every analytics rollup filters on it). Two
366 things set it, and the second was missing for a long time:
367
368 * the request came from an owner IP (cheap, no auth lookup on the hot
369 path), and
370 * `is_operator` -- the caller authenticated as the operator, which the
371 server decides from the admin token.
372
373 Without the second, every deploy call, every package install and every
374 scripted `POST /objects/...` counted as a visitor page view. On one real
375 box that produced a "Top paths" report where 45% of the rows had been hit
376 exactly once and the contents were vulnerability scanners and the
377 operator's own automation -- a report of the observer, not the observed.
378
379 Note what this deliberately does NOT flag: a signed-in member browsing
380 the site. That is real traffic and worth measuring; which pages members
381 actually use is among the most useful things this collection knows. The
382 line is between somebody USING the site and somebody OPERATING it.
383 """
384 ua = (headers.get("user-agent") or "")[:_MAX_UA]
385 referrer = (headers.get("referer") or headers.get("referrer") or "")[:_MAX_REFERRER]
386 session_id = minted_visitor_token or visitor_token(headers)
387 return {
388 # Which site was asked. One process here serves both a marketing
389 # domain and the app, and without this their visitors are one
390 # number that answers neither question. Port stripped so
391 # example.com and example.com:443 are one host rather than two.
392 "host": (headers.get("host") or "").split(":")[0].lower()[:_MAX_HOST],
393 "path": (path or "/")[:_MAX_PATH],
394 "method": (method or "GET").upper(),
395 "status": str(int(status)),
396 "ip": ip or "",
397 "user_agent": ua,
398 "referrer": referrer,
399 "session_id": session_id,
400 # Rule 4, enforced rather than documented: never both. See the
401 # docstring above -- the two columns exist, and a row holding both
402 # is the join that de-anonymises everything the token ever did.
403 "user_id": "" if session_id else (user_id or ""),
404 "is_owner": "true" if (is_operator or ip in owners) else "false",
405 }