askrobots / dbbasic-object-server
Public
Runs live, versioned Python application objects on one VM, with a suite of apps: projects, notes, tasks, files and more.
danq9
signup: the person chooses the password and name on the confirm page
83c7aa6
2 weeks, 5 days ago
History
export_subject_data.py · 10.0 KB · 253 lines
| 1 | """action_export_subject_data -- everything this server holds against one |
| 2 | email address, folded into one JSON document. |
| 3 | |
| 4 | POST {email} -> {ok, email, collections: {...}, counts: {...}, |
| 5 | unavailable: [...], not_searchable: [...]} |
| 6 | |
| 7 | The access/portability half of a subject request, and the half this system |
| 8 | can do honestly. Almost everything a customer gives this server is keyed |
| 9 | by `customer_email` -- orders, invoices, carts, disputes -- and the rest |
| 10 | hangs off those by id: payments belong to an invoice, shipments to an |
| 11 | order. So the export is a fold, not a search, and it can be complete in a |
| 12 | way a keyword search over a database never is. |
| 13 | |
| 14 | ## The property it is judged on: nothing keyed to anybody else |
| 15 | |
| 16 | An export that leaks a second customer's order into the first customer's |
| 17 | subject request is not a partial success, it is a data breach performed by |
| 18 | the compliance feature. So the joins run only in the safe direction: the |
| 19 | subject's invoice ids select payments, the subject's order ids select |
| 20 | shipments, and a payment whose invoice is not the subject's is never |
| 21 | looked at again. tests/test_privacy.py stages two customers and asserts |
| 22 | one export contains nothing of the other's -- that is the test that |
| 23 | matters here, and it is worth more than every other assertion in the file. |
| 24 | |
| 25 | ## Admin-executed, always |
| 26 | |
| 27 | There is no self-service door. A page that returned somebody's entire |
| 28 | order history to whoever typed their email address would be a lookup |
| 29 | oracle for anyone who knows a customer's address, and the identity check a |
| 30 | real subject request needs -- "prove you are this person" -- is a human |
| 31 | judgement rather than an HTTP parameter. So this refuses anyone who is not |
| 32 | an operator, and the operator runs it after satisfying themselves who they |
| 33 | are talking to. |
| 34 | |
| 35 | ## What it deliberately does NOT return, and says so |
| 36 | |
| 37 | `page_views` and untethered `conversions` rows are keyed by IP address and |
| 38 | by an opaque visitor token with no link to a person. They cannot be |
| 39 | searched for a subject in either direction, and guessing -- "these hits |
| 40 | came from an address that once loaded her invoice" -- would invent a link |
| 41 | this system spent real effort not creating. They are reported in |
| 42 | `not_searchable` with that reason rather than silently omitted, because a |
| 43 | silent omission reads as "we hold nothing". |
| 44 | |
| 45 | Conversions ARE returned where they are genuinely identifiable: a row |
| 46 | whose `user_id` is this subject's account, or whose provenance names one |
| 47 | of their own orders or invoices. |
| 48 | |
| 49 | ## There is no erasure here, and that is a decision rather than a gap |
| 50 | |
| 51 | Do not add one to this object. Erasure and Doctrine #8 (evidence is never |
| 52 | edited) genuinely conflict: orders, invoices and payments are business |
| 53 | records with statutory retention, and deleting a row an invoice |
| 54 | references does not anonymise a customer, it breaks the ledger. The |
| 55 | resolution the plan argues for is redaction-with-a-tombstone -- the LEDGER |
| 56 | keeps the fact, the CONTACT record loses the identifier, and the tombstone |
| 57 | records that a redaction happened so a later reader knows the gap is |
| 58 | deliberate rather than corruption. That needs a doctrine decision about |
| 59 | which fields are contact and which are evidence, per collection, before |
| 60 | any code is written. A `delete_subject_data` written before that decision |
| 61 | would make the choice silently, one collection at a time, in the one place |
| 62 | it can never be undone. |
| 63 | """ |
| 64 | |
| 65 | import json |
| 66 | import os |
| 67 | |
| 68 | import object_identity |
| 69 | import object_records |
| 70 | |
| 71 | ACTOR = "action_export_subject_data" |
| 72 | DATA_DIR_ENV = "DBBASIC_DATA_DIR" |
| 73 | |
| 74 | # Collections that carry the subject's email directly, and the field that |
| 75 | # carries it. This is the whole primary index: everything else is reached |
| 76 | # through an id on one of these. |
| 77 | EMAIL_KEYED = ( |
| 78 | ("orders", "customer_email"), |
| 79 | ("invoices", "customer_email"), |
| 80 | ("carts", "customer_email"), |
| 81 | ("disputes", "customer_email"), |
| 82 | ("contacts", "email"), |
| 83 | ) |
| 84 | |
| 85 | # Collections reached only through a record already proven to be the |
| 86 | # subject's. (collection, field, which primary collection's ids fill it) |
| 87 | LINKED = ( |
| 88 | ("payments", "invoice_id", "invoices"), |
| 89 | ("shipments", "order_id", "orders"), |
| 90 | ) |
| 91 | |
| 92 | CONVERSIONS = "conversions" |
| 93 | |
| 94 | NOT_SEARCHABLE = ( |
| 95 | { |
| 96 | "collection": "page_views", |
| 97 | "reason": "Traffic rows are keyed by network address and by an opaque " |
| 98 | "visitor token that is never joined to an account, so they " |
| 99 | "cannot be matched to a person in either direction. They are " |
| 100 | "deleted on the retention period stated at /privacy.", |
| 101 | }, |
| 102 | { |
| 103 | "collection": "conversions", |
| 104 | "reason": "Conversion rows recorded by a back-office transition carry " |
| 105 | "no identity at all. Rows that DO name this subject -- by " |
| 106 | "account id, or by naming one of their own orders or " |
| 107 | "invoices -- are included in the export above.", |
| 108 | }, |
| 109 | ) |
| 110 | |
| 111 | |
| 112 | def _base_dir(): |
| 113 | return os.environ.get(DATA_DIR_ENV, "data") |
| 114 | |
| 115 | |
| 116 | def _rows(collection, base): |
| 117 | """A collection's rows, or None when this box does not have it. |
| 118 | |
| 119 | None and [] are different answers and the export reports them |
| 120 | differently: "this server does not record shipments" is not "this |
| 121 | person has no shipments".""" |
| 122 | try: |
| 123 | return object_records.read_collection_records(collection, base_dir=base) |
| 124 | except Exception: |
| 125 | return None |
| 126 | |
| 127 | |
| 128 | def _matches(row, field, email): |
| 129 | return str(row.get(field) or "").strip().lower() == email |
| 130 | |
| 131 | |
| 132 | def _subject_user_id(base, email): |
| 133 | """The account id for this email, if this server has one. |
| 134 | |
| 135 | Read from identity rather than guessed, and used only to select |
| 136 | conversion rows that already name it. |
| 137 | """ |
| 138 | try: |
| 139 | users = object_identity.list_users(base_dir=base) |
| 140 | except Exception: |
| 141 | return "" |
| 142 | for user in users: |
| 143 | if str(user.get("email") or "").strip().lower() == email: |
| 144 | return str(user.get("user_id") or "") |
| 145 | return "" |
| 146 | |
| 147 | |
| 148 | def _identifiable_conversions(base, email, order_ids, invoice_ids): |
| 149 | """Conversion rows that genuinely name this subject. |
| 150 | |
| 151 | Two ways a row can: it carries their account id, or its provenance |
| 152 | blob names a record already proven to be theirs |
| 153 | (`{"source": "orders/ord-1"}`). Nothing is matched by visitor token -- |
| 154 | that token is anonymous by construction and joining it to a person |
| 155 | here would perform exactly the de-anonymisation the analytics rules |
| 156 | forbid, inside the feature meant to protect them. |
| 157 | """ |
| 158 | rows = _rows(CONVERSIONS, base) |
| 159 | if rows is None: |
| 160 | return None |
| 161 | |
| 162 | user_id = _subject_user_id(base, email) |
| 163 | sources = {f"orders/{oid}" for oid in order_ids} |
| 164 | sources |= {f"invoices/{iid}" for iid in invoice_ids} |
| 165 | |
| 166 | matched = [] |
| 167 | for row in rows: |
| 168 | row_user = str(row.get("user_id") or "").strip() |
| 169 | if user_id and row_user == user_id: |
| 170 | matched.append(row) |
| 171 | continue |
| 172 | try: |
| 173 | metadata = json.loads(str(row.get("metadata") or "") or "{}") |
| 174 | except (TypeError, ValueError): |
| 175 | continue |
| 176 | source = str((metadata or {}).get("source") or "").strip() |
| 177 | if source and source in sources: |
| 178 | matched.append(row) |
| 179 | return matched |
| 180 | |
| 181 | |
| 182 | def POST(request): |
| 183 | identity = (request or {}).get("_identity") or {} |
| 184 | if "admin" not in (identity.get("roles") or []): |
| 185 | # No self-service door on purpose -- see the module docstring. |
| 186 | return {"status": 403, |
| 187 | "error": "Only an operator may export a subject's data. " |
| 188 | "Identifying the person asking is a human judgement, " |
| 189 | "not an HTTP parameter."} |
| 190 | |
| 191 | email = str((request or {}).get("email") or "").strip().lower() |
| 192 | if not email: |
| 193 | return {"status": 400, "error": "email is required"} |
| 194 | |
| 195 | base = _base_dir() |
| 196 | collections = {} |
| 197 | counts = {} |
| 198 | unavailable = [] |
| 199 | |
| 200 | for collection, field in EMAIL_KEYED: |
| 201 | rows = _rows(collection, base) |
| 202 | if rows is None: |
| 203 | unavailable.append(collection) |
| 204 | continue |
| 205 | matched = [row for row in rows if _matches(row, field, email)] |
| 206 | collections[collection] = matched |
| 207 | counts[collection] = len(matched) |
| 208 | |
| 209 | # Ids proven to belong to this subject. Everything below selects from |
| 210 | # these and never from the email again, so a linked row can only be |
| 211 | # reached through a record already established as theirs. |
| 212 | owned = {name: {str(row.get("id") or "") for row in collections.get(name, [])} |
| 213 | for name in ("orders", "invoices")} |
| 214 | |
| 215 | for collection, field, via in LINKED: |
| 216 | rows = _rows(collection, base) |
| 217 | if rows is None: |
| 218 | unavailable.append(collection) |
| 219 | continue |
| 220 | ids = {value for value in owned.get(via, set()) if value} |
| 221 | matched = [row for row in rows |
| 222 | if str(row.get(field) or "").strip() in ids] |
| 223 | collections[collection] = matched |
| 224 | counts[collection] = len(matched) |
| 225 | |
| 226 | conversions = _identifiable_conversions( |
| 227 | base, email, owned.get("orders", set()), owned.get("invoices", set())) |
| 228 | if conversions is None: |
| 229 | unavailable.append(CONVERSIONS) |
| 230 | else: |
| 231 | collections[CONVERSIONS] = conversions |
| 232 | counts[CONVERSIONS] = len(conversions) |
| 233 | |
| 234 | return { |
| 235 | "status": 200, |
| 236 | "ok": True, |
| 237 | "email": email, |
| 238 | "collections": collections, |
| 239 | "counts": counts, |
| 240 | "total": sum(counts.values()), |
| 241 | # Which collections this box does not have at all, kept apart from |
| 242 | # the ones it has and found nothing in: "we do not ship things" and |
| 243 | # "you have no shipments" are different answers to a subject |
| 244 | # request and only one of them is about them. |
| 245 | "unavailable": sorted(set(unavailable)), |
| 246 | "not_searchable": [dict(entry) for entry in NOT_SEARCHABLE], |
| 247 | # Erasure is deliberately not offered here. See the docstring. |
| 248 | "erasure": "Not available through this action. Business records have " |
| 249 | "a statutory retention period and the redaction design " |
| 250 | "that would honour an erasure request without breaking the " |
| 251 | "ledger has not been decided yet; write to the contact " |
| 252 | "address on /privacy.", |
| 253 | } |