Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
visitors.py · 22.8 KB · 514 lines
1"""/visitors -- did anyone actually turn up?
2
3The existing /analytics page answers "what was requested" from the
4rollups: top paths, top IPs, status codes. This one answers a different
5question, and it is the one an operator asks after telling somebody about
6their server: how many PEOPLE came, when, and from where.
7
8Three things it refuses to do, each because the obvious version misleads:
9
10**It does not report one number.** Unique addresses, undifferentiated,
11are dominated by scanners -- on this server 45% of distinct paths were
12hit exactly once by things probing for /wp-login.php. Visitors, bots and
13our own traffic are counted apart and shown apart, so a quiet day reads
14as a quiet day.
15
16**It does not hide our own hits.** They are labelled, not excluded.
17Seeing your own visit is how you confirm the page is recording anything;
18a counter that silently drops the only traffic you can verify is a
19counter you cannot trust.
20
21**It does not pretend an IP is a person.** The page says so, in words, on
22the page. An office behind one NAT is one visitor and a phone switching
23networks is two. The first-party visitor cookie improves the RETURNING
24question without moving that ceiling at all, and the new-versus-returning
25block says on the page that it is a floor rather than a census: a cleared
26cookie is a new visitor, and this server fingerprints nobody.
27
28Computed live rather than rolled up, deliberately: page_views is capped
29(20k rows on the demo box) so a single pass is cheap, and an operator
30checking whether their announcement worked wants NOW, not "as of the last
31five-minute pass". The one-pass fold lives in object_visitors, which is
32pure and tested; this object renders it.
33
34## Why goals and funnels landed HERE and not on a /funnels of their own
35
36A second page was the obvious move and it is the wrong one. "How many
37came", "how many of them had been before" and "how many of them bought"
38are one question asked at increasing depth, and splitting them puts the
39denominator on one screen and the numerator on another -- a conversion
40rate you have to hold in your head across a navigation is a conversion
41rate nobody computes. The mechanical argument agrees: a /funnels page
42would need its own sign-in refusal, its own capture-is-off branch, its own
43host filter and its own copy of every honesty caveat on this page, and a
44duplicated caveat is a caveat that goes stale on one of the two copies.
45
46What a funnel adds is one settings row (`analytics.funnel_steps`), so a
47shop describes its own without code. When nothing is configured this page
48shows what to configure -- built from the event types actually recorded on
49THIS server, so it can be copied rather than invented -- instead of an
50empty table that reads as "nobody converted".
51"""
52
53import html
54import json
55import os
56
57import object_analytics
58import object_conversions
59import object_records
60import object_visitors
61
62DATA_DIR_ENV = "DBBASIC_DATA_DIR"
63
64_STYLE = """
65.vis-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: .75rem; margin: 1rem 0 1.5rem; }
66.vis-card { border: 1px solid var(--line, #38384a); border-radius: 8px; padding: .8rem; }
67.vis-card .n { font-size: 1.9rem; font-weight: 600; line-height: 1.1; }
68.vis-card .k { font-size: .8rem; opacity: .7; }
69.vis-table { width: 100%; border-collapse: collapse; margin: .4rem 0 1.5rem; }
70.vis-table th, .vis-table td { text-align: left; padding: .3rem .5rem; border-bottom: 1px solid var(--line, #38384a); font-variant-numeric: tabular-nums; }
71.vis-table td.num, .vis-table th.num { text-align: right; }
72.bar { display: inline-block; height: .7rem; background: var(--accent, #b5713a); border-radius: 2px; vertical-align: middle; }
73.bar.bot { background: var(--line, #55556a); }
74.quiet td { opacity: .45; }
75.vis-note { border-left: 3px solid var(--accent, #b5713a); padding: .1rem 0 .1rem .7rem; margin: .4rem 0 1.2rem; font-size: .85rem; opacity: .85; }
76.vis-setup { border: 1px dashed var(--line, #38384a); border-radius: 8px; padding: .8rem 1rem; margin: .4rem 0 1.5rem; }
77.vis-setup pre { overflow-x: auto; font-size: .8rem; }
78"""
79
80
81def _esc(value):
82 return html.escape(str(value if value is not None else ""), quote=True)
83
84
85def _data_dir():
86 return os.environ.get(DATA_DIR_ENV, object_records.DEFAULT_DATA_DIR)
87
88
89def _bar(count, peak, *, bot=False):
90 if not count or not peak:
91 return ""
92 width = max(2, round(120 * count / peak))
93 return f'<span class="bar{" bot" if bot else ""}" style="width:{width}px"></span>'
94
95
96def _series_rows(series, *, label_slice=None, hide_empty=False):
97 peak = max([row["visitors"] for row in series] or [0]) or 1
98 out = []
99 for row in series:
100 if hide_empty and not (row["visitors"] or row["bots"] or row["operator_views"]):
101 continue
102 quiet = "" if row["visitors"] else ' class="quiet"'
103 label = row["bucket"][label_slice] if label_slice else row["bucket"]
104 out.append(
105 f"<tr{quiet}><td>{_esc(label)}</td>"
106 f'<td class="num">{row["visitors"]}</td>'
107 f'<td>{_bar(row["visitors"], peak)}</td>'
108 f'<td class="num">{row["views"]}</td>'
109 f'<td class="num">{row["bots"]}</td>'
110 f'<td class="num">{row["operator_views"]}</td></tr>')
111 return "".join(out) or '<tr><td colspan="6" class="hint">Nothing recorded.</td></tr>'
112
113
114# --- goals, funnels and the returning visitor ---------------------------------
115
116def _rows(collection, base):
117 """A collection's rows, or None when it cannot be read at all.
118
119 None is not an empty list here. A missing `conversions` collection
120 means app-analytics' schema was never installed, and rendering that as
121 "0 conversions" would be the exact failure this whole page is written
122 against: a confident zero where the truth is "nothing was ever
123 recorded".
124 """
125 try:
126 return object_records.read_collection_records(collection, base_dir=base)
127 except Exception:
128 return None
129
130
131def _setting(base, key):
132 """Duplicated on purpose, like every other package that reads
133 app_settings -- see docs/logic-decisions.md #4."""
134 try:
135 for row in object_records.read_collection_records("app_settings",
136 base_dir=base):
137 if row.get("key") == key and str(row.get("value") or "").strip():
138 return str(row["value"]).strip()
139 except Exception:
140 pass
141 return ""
142
143
144def _cookie_note():
145 """What this box asks a browser to remember, folded from the setting
146 rather than asserted. The old wording said "the one thing this server
147 does ask a browser to remember is a first-party cookie" as a flat
148 fact; with the cookie defaulted off that sentence was false on every
149 default install, and an analytics page that misdescribes its own
150 collection is the failure this whole package is written against."""
151 if not object_analytics.visitor_cookie_enabled():
152 return ("This server asks the browser to remember nothing at all for "
153 "analytics: the first-party "
154 f"<code>{_esc(object_analytics.VISITOR_COOKIE_NAME)}</code> "
155 "cookie is off, which is the default, so every visit looks new "
156 "and no identifier is stored on anybody's device.")
157 return ("The one thing this server does ask a browser to remember is a "
158 "first-party "
159 f"<code>{_esc(object_analytics.VISITOR_COOKIE_NAME)}</code> cookie — "
160 "an opaque token, HttpOnly, SameSite, "
161 f"{object_analytics.visitor_days()} days, never set for anyone "
162 "sending Do Not Track or Global Privacy Control, and never joined "
163 "to a signed-in account. It makes returning visitors countable and "
164 "it does not raise the ceiling: it still cannot tell a phone from a "
165 "laptop, and nobody is fingerprinted.")
166
167
168def _caveat_tail(caveat):
169 """The caveat minus its own headline, so the page can bold the
170 headline and keep the sentence. Read from the fold rather than
171 retyped: two copies of an honesty note is one copy that goes stale."""
172 tail = caveat.split(":", 1)[-1].strip()
173 return tail[:1].upper() + tail[1:]
174
175
176def _returning_block(rows, days):
177 # New-versus-returning is the one number the visitor cookie buys, so
178 # with the cookie off it is not a zero -- it is a question this box
179 # deliberately does not ask. Rendering "0% returning" here would be
180 # exactly the confident-zero failure the rest of this page is written
181 # against, and it would also be the page quietly disagreeing with
182 # /privacy, which says no identifier is stored.
183 if not object_analytics.visitor_cookie_enabled():
184 return f"""
185<h2 style="font-size:1rem">New versus returning</h2>
186<p class="vis-note"><strong>Not measured on this server.</strong> The
187first-party <code>{_esc(object_analytics.VISITOR_COOKIE_NAME)}</code> cookie is
188off (the default), so nothing is stored on a visitor's device and a returning
189visitor is indistinguishable from a new one. Everything above still works —
190this is the one question that goes away. Setting
191<code>{_esc(object_analytics.VISITOR_COOKIE_ENV)}=on</code> answers it, and
192{_esc(object_analytics.OBLIGATION)}. See <a href="/privacy">/privacy</a>, which
193states whichever of the two you chose.</p>
194"""
195 fold = object_conversions.returning_visitors(rows, window_days=days)
196 cards = "".join(
197 f'<div class="vis-card"><div class="n">{value}</div>'
198 f'<div class="k">{_esc(label)}</div></div>'
199 for label, value in (
200 ("new (cookie seen once)", fold["new"]),
201 ("returning", fold["returning"]),
202 ("% returning", f'{fold["returning_pct"]}%'),
203 ("addresses with no cookie", fold["no_token_addresses"]),
204 ))
205 return f"""
206<h2 style="font-size:1rem">New versus returning</h2>
207<div class="vis-grid">{cards}</div>
208<p class="vis-note"><strong>A floor, never a census.</strong>
209{_esc(_caveat_tail(object_conversions.FLOOR_CAVEAT))}
210The {fold["no_token_addresses"]} address(es) with no cookie are visitors who
211sent Do Not Track or Global Privacy Control, clients that keep no cookies,
212and traffic recorded before the cookie existed — they are counted as
213visitors above and cannot appear on either side of this split.</p>
214"""
215
216
217def _goals_block(conversions, days):
218 if conversions is None:
219 return """
220<h2 style="font-size:1rem">Goals</h2>
221<p class="hint">No <code>conversions</code> collection on this server — install
222app-analytics' schema. Nothing before that moment can be recovered, because
223nothing was written.</p>
224"""
225 summary = object_conversions.by_event_type(conversions, window_days=days)
226 body = "".join(
227 f'<tr><td><code>{_esc(row["event_type"])}</code></td>'
228 f'<td class="num">{row["count"]}</td>'
229 f'<td class="num">{row["threaded"]}</td>'
230 f'<td><time datetime="{_esc(row["last"])}">'
231 f'{_esc(row["last"][:16].replace("T", " "))}</time></td></tr>'
232 for row in summary
233 ) or ('<tr><td colspan="4" class="hint">No conversions recorded in this '
234 'window. Goals are written by a transition — an order confirmed, a '
235 'payment received, a scan confirmed — so this stays empty until one '
236 'happens with the change dispatcher running.</td></tr>')
237 return f"""
238<h2 style="font-size:1rem">Goals</h2>
239<table class="vis-table">
240<thead><tr><th>Event</th><th class="num">Count</th>
241<th class="num">Threaded</th><th>Last</th></tr></thead>
242<tbody>{body}</tbody></table>
243<p class="hint"><strong>Threaded</strong> is how many carry a visitor token and
244can therefore appear in a funnel. A goal recorded by a back-office transition
245has no browser anywhere near it, so it counts here and cannot be attributed to
246a visitor — said out loud rather than left to explain a funnel that looks
247broken.</p>
248"""
249
250
251def _funnel_setup(conversions, summary):
252 """What to configure, built from what this server has actually seen.
253
254 An empty funnel table would read as "nobody converted". An example
255 made of paths and event types this box has really recorded can be
256 copied straight into the setting, which is the difference between
257 documentation and a next step.
258 """
259 landing = [row["path"] for row in summary["landing"][:2]] or ["/"]
260 events = [row["event_type"]
261 for row in object_conversions.by_event_type(conversions or [])[:1]]
262 example = json.dumps(landing + (events or ["order_confirmed"]))
263 return f"""
264<h2 style="font-size:1rem">Funnel</h2>
265<div class="vis-setup">
266<p style="margin-top:0">No funnel is configured, so there is nothing to show —
267which is not the same as nobody converting. A funnel is a settings row, not
268code: add <code>{_esc(object_conversions.FUNNEL_STEPS_SETTING)}</code> to
269<a href="/collections/app_settings/records">app settings</a> as a JSON list of
270ordered steps. A step beginning with <code>/</code> is a path prefix; anything
271else is a conversion <code>event_type</code>.</p>
272<pre><code>{_esc(example)}</code></pre>
273<p class="hint" style="margin-bottom:0">Steps built from what this server has
274actually recorded, so it can be copied rather than invented. The long form
275takes labels: <code>[{{"label": "Browsed", "path": "/shop"}}, …]</code>.</p>
276</div>
277"""
278
279
280def _timing_block(page_views, conversions):
281 fold = object_conversions.time_to_conversion(page_views, conversions or [])
282 if not fold["count"]:
283 return ""
284 cards = "".join(
285 f'<div class="vis-card"><div class="n">{value}</div>'
286 f'<div class="k">{_esc(label)}</div></div>'
287 for label, value in (
288 ("median days to convert", fold["median_days"]),
289 ("quarter convert within", fold["p25_days"]),
290 ("three quarters within", fold["p75_days"]),
291 ("same-day conversions", fold["same_day"]),
292 ))
293 caveats = "".join(f'<p class="vis-note">{_esc(text)}</p>'
294 for text in fold["caveats"])
295 return f"""
296<h2 style="font-size:1rem">Time to conversion</h2>
297<div class="vis-grid">{cards}</div>
298<p class="hint">Days between a visitor's first recorded page and their
299conversion, over the {fold["count"]} conversion(s) that carry a visitor token
300AND still have a first page view in the log. A median of zero is a shop people
301buy from on arrival; a median of nine is a decision somebody goes away and
302thinks about.</p>
303{caveats}
304"""
305
306
307def _funnel_row(row):
308 """One step. The first step has no drop-off — it is the denominator,
309 and rendering "0% lost" there invites reading it as a step everyone
310 passed rather than as the population everything else is measured
311 against."""
312 lost = "" if row["drop_off_pct"] is None else str(row["drop_off"])
313 pct = "" if row["drop_off_pct"] is None else f'{row["drop_off_pct"]}%'
314 return (f'<tr><td>{_esc(row["label"])}</td>'
315 f'<td><code>{_esc(row["match"])}</code></td>'
316 f'<td class="num">{row["visitors"]}</td>'
317 f'<td class="num">{row["of_entered_pct"]}%</td>'
318 f'<td class="num">{lost}</td>'
319 f'<td class="num">{pct}</td>'
320 f'<td class="num">{row["ip_stitched"]}</td></tr>')
321
322
323def _funnel_block(page_views, conversions, summary, base):
324 raw = _setting(base, object_conversions.FUNNEL_STEPS_SETTING)
325 steps, error = object_conversions.parse_funnel_steps(raw)
326 if error:
327 return f"""
328<h2 style="font-size:1rem">Funnel</h2>
329<p class="vis-note"><code>{_esc(object_conversions.FUNNEL_STEPS_SETTING)}</code>
330could not be read: {_esc(error)}. Said out loud rather than shown as an empty
331funnel — a misconfigured funnel and a funnel nobody entered look identical, and
332only one of them is your fault.</p>
333"""
334 if not steps:
335 return _funnel_setup(conversions, summary)
336
337 fold = object_conversions.funnel(page_views, conversions or [], steps)
338 body = "".join(_funnel_row(row) for row in fold["steps"])
339 caveats = "".join(f'<p class="vis-note">{_esc(text)}</p>'
340 for text in fold["caveats"])
341 return f"""
342<h2 style="font-size:1rem">Funnel</h2>
343<table class="vis-table">
344<thead><tr><th>Step</th><th>Matches</th><th class="num">Visitors</th>
345<th class="num">Of first</th><th class="num">Lost</th>
346<th class="num">Drop-off</th><th class="num">IP-stitched</th></tr></thead>
347<tbody>{body}</tbody></table>
348<p class="hint">Threaded by the visitor cookie, falling back to the IP address
349for rows that carry none — the <strong>IP-stitched</strong> column is how much
350of each step is that fallback. An IP is not a person: an office behind one
351connection is one very decisive shopper. Steps are a sequence, so a visitor
352counts at a step only if they reached the one before it and did this one
353afterwards.</p>
354{caveats}
355"""
356
357
358def GET(request):
359 identity = request.get("_identity", {}) or {}
360 user_id = identity.get("user_id")
361 if not user_id:
362 body = ('<p class="hint"><a href="/login?next=/visitors">Sign in</a> '
363 'to see visitor numbers.</p>')
364 return _page(body)
365
366 if not object_analytics.analytics_enabled():
367 return _page('<p class="hint">Analytics capture is off. Set '
368 '<code>DBBASIC_ANALYTICS=on</code> to start recording '
369 'page views; nothing before that moment can be recovered, '
370 'because nothing was written.</p>')
371
372 try:
373 rows = object_records.read_collection_records(
374 object_analytics.PAGE_VIEWS_COLLECTION, base_dir=_data_dir())
375 except Exception:
376 return _page('<p class="hint">No page_views collection yet — install '
377 'app-analytics and let one request through.</p>')
378
379 try:
380 days = max(1, min(31, int(str(request.get("days") or 7))))
381 except (TypeError, ValueError):
382 days = 7
383
384 host = str(request.get("host") or "").strip().lower()
385 summary = object_visitors.summarize(rows, days=days, host=host)
386 totals = summary["totals"]
387 retention = object_analytics.retention_days()
388
389 # Everything below the traffic tables is scoped to the same host, so a
390 # per-site view cannot show one site's visitors above another site's
391 # funnel. Conversions carry no host of their own -- they are recorded
392 # by a transition, not a request -- so they are not filtered, and the
393 # funnel says how much of itself it could attribute.
394 base = _data_dir()
395 scoped = ([row for row in rows if str(row.get("host") or "") == host]
396 if host else rows)
397 conversions = _rows(object_conversions.CONVERSIONS_COLLECTION, base)
398
399 cards = "".join(
400 f'<div class="vis-card"><div class="n">{value}</div>'
401 f'<div class="k">{_esc(label)}</div></div>'
402 for label, value in (
403 (f"unique visitors, {days}d", totals["visitor"]["unique"]),
404 (f"visitor page views, {days}d", totals["visitor"]["views"]),
405 ("bot addresses", totals["bot"]["unique"]),
406 ("your own requests", totals["operator"]["views"]),
407 ))
408
409 referrers = "".join(
410 f'<tr><td>{_esc(row["referrer"])}</td>'
411 f'<td class="num">{row["visitors"]}</td></tr>'
412 for row in summary["referrers"]
413 ) or '<tr><td colspan="2" class="hint">No visitors yet in this window.</td></tr>'
414
415 landing = "".join(
416 f'<tr><td><code>{_esc(row["path"])}</code></td>'
417 f'<td class="num">{row["visitors"]}</td></tr>'
418 for row in summary["landing"]
419 ) or '<tr><td colspan="2" class="hint">No pages loaded by a visitor yet.</td></tr>'
420
421 hosts = "".join(
422 f'<tr><td><a href="/visitors?days={days}&host={_esc(row["host"])}">'
423 f'{_esc(row["host"])}</a></td>'
424 f'<td class="num">{row["visitors"]}</td></tr>'
425 for row in summary["hosts"]
426 ) or '<tr><td colspan="2" class="hint">No visitors yet in this window.</td></tr>'
427
428 head = ('<thead><tr><th>When</th><th class="num">Visitors</th><th></th>'
429 '<th class="num">Views</th><th class="num">Bots</th>'
430 '<th class="num">Yours</th></tr></thead>')
431
432 body = f"""
433<div class="breadcrumb"><a href="/">Home</a> / Visitors</div>
434<div class="pagehead"><h1>Visitors{" &middot; " + _esc(host) if host else ""}</h1>
435 <span class="muted">last {days} days &middot;
436 <a href="/visitors?days=1">1</a> &middot; <a href="/visitors?days=7">7</a> &middot;
437 <a href="/visitors?days=30">30</a></span></div>
438
439<div class="vis-grid">{cards}</div>
440
441{'<p class="hint"><a href="/visitors?days=' + str(days) + '">&larr; all sites</a></p>' if host else ''}
442<h2 style="font-size:1rem">Which site</h2>
443<p class="hint" style="margin-top:0">One process answers for more than one
444hostname here, so a combined figure answers neither "did anyone read the
445pitch" nor "did anyone try the product".</p>
446<table class="vis-table">
447<thead><tr><th>Host</th><th class="num">Visitors</th></tr></thead>
448<tbody>{hosts}</tbody></table>
449
450<h2 style="font-size:1rem">Today, by hour (UTC)</h2>
451<table class="vis-table">{head}<tbody>
452{_series_rows(summary["hours"], label_slice=slice(11, 13), hide_empty=True)}
453</tbody></table>
454
455<h2 style="font-size:1rem">By day</h2>
456<table class="vis-table">{head}<tbody>
457{_series_rows(summary["days"])}
458</tbody></table>
459
460<h2 style="font-size:1rem">Where visitors came from</h2>
461<table class="vis-table">
462<thead><tr><th>Referrer</th><th class="num">Visitors</th></tr></thead>
463<tbody>{referrers}</tbody></table>
464
465<h2 style="font-size:1rem">What they landed on</h2>
466<table class="vis-table">
467<thead><tr><th>Page</th><th class="num">Visitors</th></tr></thead>
468<tbody>{landing}</tbody></table>
469
470{_returning_block(scoped, days)}
471{_goals_block(conversions, days)}
472{_funnel_block(scoped, conversions, summary, base)}
473{_timing_block(scoped, conversions)}
474
475<p class="hint">A <strong>visitor</strong> is an address that loaded a real
476page <em>and</em> did one thing a prober does not — opened a second page,
477arrived from a referrer, or carried a session. A lone hit on
478<code>/</code> is not counted, because <code>/</code> answers 200 to
479everyone. A <strong>bot</strong> is everything else: scanners walking a
480list of URLs that do not exist here, declared crawlers, and anything
481sending an attack payload in a header. <strong>Yours</strong> is traffic from an
482owner address or carrying the admin token, shown rather than hidden so you
483can check the page is recording at all.</p>
484<p class="hint">Unique means distinct IP address, which is an
485approximation and not a person: an office behind one connection counts
486once, a phone moving between wifi and cellular counts twice. {_cookie_note()}
487Page views are kept for {retention} days
488&middot; <a href="/analytics">path and status detail</a>
489&middot; <a href="/privacy">what visitors are told</a>.</p>
490"""
491 return _page(body)
492
493
494def _page(body):
495 return {
496 "content_type": "text/html; charset=utf-8",
497 "body": f"""<!doctype html>
498<html lang="en">
499<head>
500<meta charset="utf-8">
501<meta name="viewport" content="width=device-width, initial-scale=1">
502<title>Visitors</title>
503<link rel="stylesheet" href="/style">
504<style>{_STYLE}</style>
505</head>
506<body>
507<div class="wrap">
508<header class="app"><h1><a href="/">DBBASIC</a></h1></header>
509{body}
510</div>
511<script src="/nav"></script>
512</body>
513</html>""",
514 }