Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.

0 downloads

Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-object-server.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-object-server.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-object-server.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 signup: the person chooses the password and name on the confirm page 83c7aa6 2 weeks, 5 days ago History
export_subject_data.py · 10.0 KB · 253 lines
1"""action_export_subject_data -- everything this server holds against one
2email address, folded into one JSON document.
3
4POST {email} -> {ok, email, collections: {...}, counts: {...},
5 unavailable: [...], not_searchable: [...]}
6
7The access/portability half of a subject request, and the half this system
8can do honestly. Almost everything a customer gives this server is keyed
9by `customer_email` -- orders, invoices, carts, disputes -- and the rest
10hangs off those by id: payments belong to an invoice, shipments to an
11order. So the export is a fold, not a search, and it can be complete in a
12way a keyword search over a database never is.
13
14## The property it is judged on: nothing keyed to anybody else
15
16An export that leaks a second customer's order into the first customer's
17subject request is not a partial success, it is a data breach performed by
18the compliance feature. So the joins run only in the safe direction: the
19subject's invoice ids select payments, the subject's order ids select
20shipments, and a payment whose invoice is not the subject's is never
21looked at again. tests/test_privacy.py stages two customers and asserts
22one export contains nothing of the other's -- that is the test that
23matters here, and it is worth more than every other assertion in the file.
24
25## Admin-executed, always
26
27There is no self-service door. A page that returned somebody's entire
28order history to whoever typed their email address would be a lookup
29oracle for anyone who knows a customer's address, and the identity check a
30real subject request needs -- "prove you are this person" -- is a human
31judgement rather than an HTTP parameter. So this refuses anyone who is not
32an operator, and the operator runs it after satisfying themselves who they
33are talking to.
34
35## What it deliberately does NOT return, and says so
36
37`page_views` and untethered `conversions` rows are keyed by IP address and
38by an opaque visitor token with no link to a person. They cannot be
39searched for a subject in either direction, and guessing -- "these hits
40came from an address that once loaded her invoice" -- would invent a link
41this system spent real effort not creating. They are reported in
42`not_searchable` with that reason rather than silently omitted, because a
43silent omission reads as "we hold nothing".
44
45Conversions ARE returned where they are genuinely identifiable: a row
46whose `user_id` is this subject's account, or whose provenance names one
47of their own orders or invoices.
48
49## There is no erasure here, and that is a decision rather than a gap
50
51Do not add one to this object. Erasure and Doctrine #8 (evidence is never
52edited) genuinely conflict: orders, invoices and payments are business
53records with statutory retention, and deleting a row an invoice
54references does not anonymise a customer, it breaks the ledger. The
55resolution the plan argues for is redaction-with-a-tombstone -- the LEDGER
56keeps the fact, the CONTACT record loses the identifier, and the tombstone
57records that a redaction happened so a later reader knows the gap is
58deliberate rather than corruption. That needs a doctrine decision about
59which fields are contact and which are evidence, per collection, before
60any code is written. A `delete_subject_data` written before that decision
61would make the choice silently, one collection at a time, in the one place
62it can never be undone.
63"""
64
65import json
66import os
67
68import object_identity
69import object_records
70
71ACTOR = "action_export_subject_data"
72DATA_DIR_ENV = "DBBASIC_DATA_DIR"
73
74# Collections that carry the subject's email directly, and the field that
75# carries it. This is the whole primary index: everything else is reached
76# through an id on one of these.
77EMAIL_KEYED = (
78 ("orders", "customer_email"),
79 ("invoices", "customer_email"),
80 ("carts", "customer_email"),
81 ("disputes", "customer_email"),
82 ("contacts", "email"),
83)
84
85# Collections reached only through a record already proven to be the
86# subject's. (collection, field, which primary collection's ids fill it)
87LINKED = (
88 ("payments", "invoice_id", "invoices"),
89 ("shipments", "order_id", "orders"),
90)
91
92CONVERSIONS = "conversions"
93
94NOT_SEARCHABLE = (
95 {
96 "collection": "page_views",
97 "reason": "Traffic rows are keyed by network address and by an opaque "
98 "visitor token that is never joined to an account, so they "
99 "cannot be matched to a person in either direction. They are "
100 "deleted on the retention period stated at /privacy.",
101 },
102 {
103 "collection": "conversions",
104 "reason": "Conversion rows recorded by a back-office transition carry "
105 "no identity at all. Rows that DO name this subject -- by "
106 "account id, or by naming one of their own orders or "
107 "invoices -- are included in the export above.",
108 },
109)
110
111
112def _base_dir():
113 return os.environ.get(DATA_DIR_ENV, "data")
114
115
116def _rows(collection, base):
117 """A collection's rows, or None when this box does not have it.
118
119 None and [] are different answers and the export reports them
120 differently: "this server does not record shipments" is not "this
121 person has no shipments"."""
122 try:
123 return object_records.read_collection_records(collection, base_dir=base)
124 except Exception:
125 return None
126
127
128def _matches(row, field, email):
129 return str(row.get(field) or "").strip().lower() == email
130
131
132def _subject_user_id(base, email):
133 """The account id for this email, if this server has one.
134
135 Read from identity rather than guessed, and used only to select
136 conversion rows that already name it.
137 """
138 try:
139 users = object_identity.list_users(base_dir=base)
140 except Exception:
141 return ""
142 for user in users:
143 if str(user.get("email") or "").strip().lower() == email:
144 return str(user.get("user_id") or "")
145 return ""
146
147
148def _identifiable_conversions(base, email, order_ids, invoice_ids):
149 """Conversion rows that genuinely name this subject.
150
151 Two ways a row can: it carries their account id, or its provenance
152 blob names a record already proven to be theirs
153 (`{"source": "orders/ord-1"}`). Nothing is matched by visitor token --
154 that token is anonymous by construction and joining it to a person
155 here would perform exactly the de-anonymisation the analytics rules
156 forbid, inside the feature meant to protect them.
157 """
158 rows = _rows(CONVERSIONS, base)
159 if rows is None:
160 return None
161
162 user_id = _subject_user_id(base, email)
163 sources = {f"orders/{oid}" for oid in order_ids}
164 sources |= {f"invoices/{iid}" for iid in invoice_ids}
165
166 matched = []
167 for row in rows:
168 row_user = str(row.get("user_id") or "").strip()
169 if user_id and row_user == user_id:
170 matched.append(row)
171 continue
172 try:
173 metadata = json.loads(str(row.get("metadata") or "") or "{}")
174 except (TypeError, ValueError):
175 continue
176 source = str((metadata or {}).get("source") or "").strip()
177 if source and source in sources:
178 matched.append(row)
179 return matched
180
181
182def POST(request):
183 identity = (request or {}).get("_identity") or {}
184 if "admin" not in (identity.get("roles") or []):
185 # No self-service door on purpose -- see the module docstring.
186 return {"status": 403,
187 "error": "Only an operator may export a subject's data. "
188 "Identifying the person asking is a human judgement, "
189 "not an HTTP parameter."}
190
191 email = str((request or {}).get("email") or "").strip().lower()
192 if not email:
193 return {"status": 400, "error": "email is required"}
194
195 base = _base_dir()
196 collections = {}
197 counts = {}
198 unavailable = []
199
200 for collection, field in EMAIL_KEYED:
201 rows = _rows(collection, base)
202 if rows is None:
203 unavailable.append(collection)
204 continue
205 matched = [row for row in rows if _matches(row, field, email)]
206 collections[collection] = matched
207 counts[collection] = len(matched)
208
209 # Ids proven to belong to this subject. Everything below selects from
210 # these and never from the email again, so a linked row can only be
211 # reached through a record already established as theirs.
212 owned = {name: {str(row.get("id") or "") for row in collections.get(name, [])}
213 for name in ("orders", "invoices")}
214
215 for collection, field, via in LINKED:
216 rows = _rows(collection, base)
217 if rows is None:
218 unavailable.append(collection)
219 continue
220 ids = {value for value in owned.get(via, set()) if value}
221 matched = [row for row in rows
222 if str(row.get(field) or "").strip() in ids]
223 collections[collection] = matched
224 counts[collection] = len(matched)
225
226 conversions = _identifiable_conversions(
227 base, email, owned.get("orders", set()), owned.get("invoices", set()))
228 if conversions is None:
229 unavailable.append(CONVERSIONS)
230 else:
231 collections[CONVERSIONS] = conversions
232 counts[CONVERSIONS] = len(conversions)
233
234 return {
235 "status": 200,
236 "ok": True,
237 "email": email,
238 "collections": collections,
239 "counts": counts,
240 "total": sum(counts.values()),
241 # Which collections this box does not have at all, kept apart from
242 # the ones it has and found nothing in: "we do not ship things" and
243 # "you have no shipments" are different answers to a subject
244 # request and only one of them is about them.
245 "unavailable": sorted(set(unavailable)),
246 "not_searchable": [dict(entry) for entry in NOT_SEARCHABLE],
247 # Erasure is deliberately not offered here. See the docstring.
248 "erasure": "Not available through this action. Business records have "
249 "a statutory retention period and the redaction design "
250 "that would honour an erasure request without breaking the "
251 "ledger has not been decided yet; write to the contact "
252 "address on /privacy.",
253 }