nda

README.md

@@ -9,6 +9,8 @@ Module 4: the buyer side — buyers, contacts, NDA rounds, deals, the guided
 Module 5: notes, todos and the "Today" view with the follow-up workflow.
 Module 6a: Dropbox Sign as the source of incoming NDAs — inbox, one-click
 import and PDF filing on the NAS.
+Module 6b: form-field extraction from the signed requests — no AI, no review
+step, the answers come straight out of `response_data`.
 
 The UI and all domain constants are English.
 
@@ -155,7 +157,7 @@ directory aborts the scan with an error naming the path.
 3. Copy the project folder to the AI machine, run `docker compose up -d --build`
 4. Restore the dump: `docker compose exec -T db psql -U bizmatch bizmatch < backup.sql`
 
-## API (as of module 6a)
+## API (as of module 6b)
 
 | Method | Path                        | Purpose                                          | Session |
 | ------ | --------------------------- | ------------------------------------------------ | ------- |
@@ -198,6 +200,7 @@ directory aborts the scan with an error naming the path.
 | POST   | /api/nda-inbox/refresh      | start the background walk (202 / 409 if running) | yes     |
 | POST   | /api/nda-inbox/import       | import one request into buyer/contact/nda(/deal) | yes     |
 | POST   | /api/nda-inbox/sync         | refresh the status of every pending imported NDA | yes     |
+| POST   | /api/nda-inbox/backfill-fields | retrofit form fields onto signed rounds       | yes     |
 | GET    | /api/ndas/:id/file          | stream the filed NDA PDF (Range + ETag)          | yes     |
 
 Everything except health, staff (GET+POST) and login requires the session
@@ -335,6 +338,100 @@ time. Nothing re-fetches a request per id unnecessarily: sync's candidate set
 is `SENT AND NOT declined`, so signed and declined rounds are never fetched
 again.
 
+The inbox is ordered and dated by **`coalesce(signed_at, created_at)`** — the
+moment that matters is when the request was signed, falling back to when it
+arrived while it still is not. 158 of the 566 mirrored requests were signed on
+a different day than they were created, so sorting by `created_at` put them in
+the wrong place.
+
+### Form fields (module 6b)
+
+A signed request carries every answer of the NDA form in `response_data`, so
+importing one needs **no extraction model and no review step** — the values are
+read off directly. Verified against the live account, an entry looks like:
+
+```json
+{ "name": "Textbox1", "type": "text", "required": true,
+  "api_id": "9fb8331d-…", "value": "kkm Foods", "signature_id": "…" }
+```
+
+`name` is always present, so the mapping is by name; the `api_id`s are stable
+per template slot and are indexed as a fallback key. Two traps the real payload
+contains: **checkbox values are the strings `"true"`/`"false"`**, not booleans,
+and signature/initials slots carry the literal string `"null"`. Both are
+handled in `readValues()`, which also drops empty answers.
+
+| Field | Goes to |
+| --- | --- |
+| `Textbox1` | `buyer.company_name` |
+| `Textbox2` / `Textbox4` | `contact.phone` / `contact.cell` |
+| `Textbox5` | `buyer.address` + `buyer.state`, falling back to `Textbox18` — see below |
+| `Textbox6` | compared with the signer e-mail — see below |
+| `Textbox7` | `buyer.how_heard` |
+| `Checkbox1` / `Checkbox2` | `buyer.interested_in_updates` = true / false, neither = null |
+| `Textbox8` | `nda.preferred_businesses_text` |
+| `Textbox9` | `buyer.background_experience` |
+| `Textbox10` / `Textbox11` | `nda.total_purchase_price` / `nda.down_payment` |
+| `Textbox12`…`Textbox15` | `nda.income_requirements` / `accountant` / `attorney` / `bank` |
+| `Textbox16` | compared with the signer name — see below |
+| `DateSigned1` | `nda.intro_date`, parsed from `"07 / 27 / 2026"` |
+
+The template also sends `Textbox3`, `Textbox17`, `Textbox19`, `Textbox20` (a
+signature block repeating company, phone and e-mail) and a second
+`DateSigned2`. Those are **not mapped** — `Textbox18` from the same block *is*,
+as the address fallback described below. The whole array is stored verbatim in
+`nda.raw_form_data`, so a mapping mistake can be corrected later without going
+back to Dropbox for every round.
+
+Two values are never written, only reported, because they would corrupt the
+identity the deduplication relies on:
+
+* `Textbox6` differing from the signer's e-mail → a note on the buyer,
+  `NDA form lists different email: <value>`. The signer address stays the
+  contact's e-mail.
+* `Textbox16` differing from the signer's name → `Form names prospective
+  buyer: <value>`. `nda.signer_name` stays the Dropbox signer.
+
+Everything is written with `coalesce(nullif(btrim(col), ''), <new>)`, i.e.
+**fill only what is empty**. On a freshly created buyer every column is NULL so
+that fills all of them; on a reused buyer it can never overwrite curated data.
+The same guard covers the round, which is what makes the retrofit safe to run
+over NDAs people have already edited by hand.
+
+`POST /api/nda-inbox/backfill-fields` is that retrofit: it walks every round
+with a `dropbox_sign_id`, status `SIGNED` and `raw_form_data IS NULL`, fetches
+it with a 500 ms pause between calls, and answers
+`{candidates, filled, empty, failed, warnings}`. It doubles as the initial load
+for the first three months and is safe to re-run — a filled round is no longer
+a candidate.
+
+Two normalisations run over the extracted values before they are stored. Both
+only affect the mapped columns — `raw_form_data` always keeps the verbatim
+answers, so nothing is lost.
+
+**Null markers.** A value that is nothing but a "does not apply" marker becomes
+NULL: `/^(n|na|n\/a|none|nil|x+|-+|\.+)$/i` after trimming — so `n`, `na`,
+`N/A`, `none`, `nil`, `x`/`xx`/`xxx`, any run of dashes and any run of dots.
+It is deliberately anchored, which is what keeps `"NASA"`, `"Nancy"`,
+`"x-ray"`, `"N. Smith"` and `"none of the above"`. Junk that is not a marker
+(`"open"`, `"enough"`) is kept too: that is what the signer wrote, and only an
+exact marker is safe to discard.
+
+> ⚠️ **Two-track normalisation — keep both in sync.** The legacy vision
+> pipeline in the QC repo carries its own `NULL_MARKERS` for the scanned-PDF
+> route. The `x` → `x+` widening above has **not** been applied there. Whoever
+> next touches that pipeline must make the same change, otherwise the same NDA
+> yields `"xx"` from the scan route and `NULL` from the Dropbox Sign route, and
+> the two sources silently disagree about what "no answer" means.
+
+**Address fallback.** The form asks for the address twice — `Textbox5` in the
+body and `Textbox18` in the signature block — and signers routinely type only a
+street in the first and the complete address in the second. When `Textbox5`
+carries no state, `Textbox18` is tried; if *it* has one, its address **and**
+state are taken together, since combining a street from one field with a state
+from the other would invent an address. If neither has a state, `Textbox5`'s
+street is kept as-is and the state stays NULL.
+
 `POST /api/nda-inbox/import` is one transaction: buyer + contact (the same
 reuse rules as the guided inquiry — see `ensureBuyerAndContact`), the NDA round
 with `dropbox_sign_id` (UNIQUE, so a second import is a `409`), and optionally
@@ -468,6 +565,7 @@ migrations/   numbered SQL migrations
   004_reset_….sql       one-time reset of that column to NULL
   005_nda_dropbox_….sql dropbox_sign_id, signer_name, declined on nda
   006_ds_request_….sql  ds_request mirror + the app_meta key/value table
+  007_nda_form_….sql    income/accountant/attorney/bank + raw_form_data on nda
 src/
   config.ts           env configuration
   db.ts               pg pool, query helpers, withTransaction
@@ -478,6 +576,7 @@ src/
   dropbox-sign.ts     thin Dropbox Sign REST client (list, get, download)
   nda-files.ts        naming, filing and active/inactive moves of NDA PDFs
   nda-refresh.ts      the background walk that mirrors requests into ds_request
+  nda-fields.ts       response_data -> buyer/contact/nda, fill-only-what-is-empty
   server.ts           Fastify app (health, staff, login, businesses, file, static)
   buyer-routes.ts     buyers, contacts, NDA rounds, deals, the inquiry flow
   workflow-routes.ts  notes, todos, documents, the Today view, follow-ups