bizmatch-app/README.md

# BizMatch Phase 2 — Broker Workflow App

Module 1: foundation (Docker Compose, PostgreSQL, schema, migrations, login).
Module 2: business scan (NAS -> DB) and the first UI.
Module 3: recursive file listing, PDF streaming from the NAS and the ported
pdf.js viewer.
Module 4: the buyer side — buyers, contacts, NDA rounds, deals, the guided
"New inquiry" flow with duplicate detection and the deal status transitions.
Module 5: notes, todos and the "Today" view with the follow-up workflow.
Module 6a: Dropbox Sign as the source of incoming NDAs — inbox, one-click
import and PDF filing on the NAS.
Module 6b: form-field extraction from the signed requests — no AI, no review
step, the answers come straight out of `response_data`.

The UI and all domain constants are English.

## Setup on 192.168.100.99 (Ubuntu 24.04)

Prerequisite: Docker + Compose plugin (`sudo apt install docker.io docker-compose-v2`).

```bash
# Unpack the project, then:
cd bizmatch-app
cp .env.example .env          # optionally adjust DB_PASSWORD
docker compose up -d --build
```

The app applies all migrations on start and then listens on
`http://192.168.100.99:8090`.

> **Upgrading from Module 1:** `001_init.sql` was rewritten in place (German
> constants -> English). Applying it needs a fresh database:
> `docker compose down -v && docker compose up -d --build`. The DB held no
> production data yet, so there is nothing to migrate.

### Schema notes

`001_init.sql` holds the full base schema (Buyer / Contact / NDA / Deal /
Business / Note / Todo / Document / ExtractionJob / Staff) and is never edited
again. `002_buyer_fields.sql` adds the fields the buyer side actually collects
and is purely additive, so it applies to an existing database:

| Table     | Added                                                                                |
| --------- | ------------------------------------------------------------------------------------ |
| `contact` | `cell`                                                                               |
| `buyer`   | `address`, `state`, `background_experience`, `how_heard`, `interested_in_updates`    |
| `nda`     | `total_purchase_price`, `down_payment`, `intro_date`                                 |

`003_interested_in_updates_nullable.sql` then drops the `NOT NULL` and the
default from `buyer.interested_in_updates`: the value comes off scanned intake
sheets where the field is frequently blank, so `NULL` means "not answered" and
has to stay distinct from `false` ("explicitly no"). The buyer detail shows it
as a Yes / No / not answered control, and `PATCH /api/buyers/:id` accepts all
three. The two price fields stay `text` on purpose: the paper forms contain entries like "1.2M + inventory" that no
numeric type survives. The migration also adds the three index expressions the
duplicate check needs (`lower(btrim(name))` and the digits-only phone/cell).

First smoke test:

```bash
curl http://localhost:8090/api/health
# -> {"ok":true}

# Create the three staff members (adjust the names):
curl -X POST localhost:8090/api/staff -H 'content-type: application/json' -d '{"name":"Chris"}'
curl -X POST localhost:8090/api/staff -H 'content-type: application/json' -d '{"name":"..."}'
```

## Dev mode (without the app container)

```bash
docker compose up -d db       # database only
npm install
set -a; source .env; set +a
npm run dev                   # tsx watch, migrations run on start

# second terminal — frontend with hot reload, /api is proxied to :8090
cd web && npm install && npm run dev
```

## Test data

The acceptance scripts run against the **real** development database, so rows
they create are visible in the UI like any other. Two rules keep that from
turning into litter:

1. **Tag it.** Every contact an acceptance run creates uses an e-mail under
   `@stubtest.invalid` (a reserved TLD that can never be a real address), and
   every stubbed Dropbox Sign request uses an id starting with `req-`. Those
   two markers are what makes test rows identifiable later, when nobody
   remembers which "Priya Raman" came from where.
2. **Remove it.** Every script ends with a cleanup step wired to `trap … EXIT`,
   so it runs even when the script fails halfway:

```bash
cleanup_stub_data() {
  docker compose exec -T db psql -U bizmatch bizmatch -q \
    -c "DELETE FROM buyer WHERE id IN (
          SELECT buyer_id FROM contact
           WHERE lower(btrim(email)) LIKE '%@stubtest.invalid');" \
    -c "DELETE FROM nda WHERE dropbox_sign_id LIKE 'req-%';" \
    -c "DELETE FROM ds_request WHERE signature_request_id LIKE 'req-%';"
}
trap cleanup_stub_data EXIT
```

Deleting the buyer cascades to its contacts, NDA rounds, deals, notes and
todos, so the one statement is enough.

**Do not truncate `ds_request`.** It was a throwaway cache only while no API
key was configured; now it mirrors the real signature requests, and a stub run
adds its rows *alongside* them. Delete the `req-*` ones and leave
`ds_last_refresh_at` alone — a truncate would throw away the real mirror and
silently misreport when it was last refreshed.

Anything that writes files (NDA filing) must point `NDA_ROOT` at a scratch
directory for the run; never let a test write into the real NAS tree.

### The Dropbox Sign stub

`scripts/ds-stub.mjs` serves the two endpoints the inbox uses and reproduces
the property that broke the refresh: **its search index lags its list.** A
fixture entry is `{request, indexed}` — the truth the plain list and the per-id
endpoint hand out, and the possibly stale (or entirely missing) copy that
`query=` searches. It parses the range syntax for real, so `[a TO b]` and
`{a TO b}` genuinely differ. Point `DROPBOX_SIGN_BASE_URL` at it.

`node scripts/acceptance-nda-refresh.mjs` is the acceptance run: it starts the
real server against the stub and the real development database and asserts that
an incremental refresh picks up a request the index does not have yet and an
old pending one signed since, that a full reload includes the boundary day,
that a sync leaves `ds_last_refresh_at` untouched, and that `covers_from` never
narrows. It snapshots every `ds_*` key in `app_meta` and writes it back
afterwards, deletes its `req-*` rows, and runs with
`SYNC_PENDING_MAX_AGE_DAYS=0` so the sync cannot start re-fetching the real
mirror row by row.

## NAS mount

Mount it on the host via NFS, e.g. in `/etc/fstab`:

```
<truenas-ip>:/mnt/<pool>/bizmatch  /mnt/bizmatch-nas  nfs  ro,soft,timeo=100  0  0
```

The compose file already passes `NAS_ROOT` (default `/mnt/bizmatch-nas`) into the
app container.

**Module 6a needs write access**: filing signed NDAs writes below `NDA_ROOT`
(default `<NAS_ROOT>/AA Buyers NDA's/Buyers NDA's A-Z`), so the mount has to be
`rw` rather than `ro`. Everything else the app does with the NAS is read-only;
if the mount stays read-only, imports still succeed and only report the failed
filing as a warning.

### Business directories

Directly below `NAS_ROOT` there are three status directories; every immediate
subdirectory of those is one business. The directory names are configurable
(they contain spaces and are treated as opaque strings):

| Env var            | Default        | Business status |
| ------------------ | -------------- | --------------- |
| `NAS_DIR_ACTIVE`   | `AAA = ACTIVE`   | ACTIVE        |
| `NAS_DIR_SOLD`     | `AAA = SOLD`     | SOLD          |
| `NAS_DIR_INACTIVE` | `AAA = INACTIVE` | INACTIVE      |

The scan is idempotent: existing rows are matched by name and only updated when
`nas_path` or `status` changed. Businesses that exist in the DB but no longer on
disk are kept and only reported as a warning in the log. A missing configured
directory aborts the scan with an error naming the path.

## Moving to the AI machine (later)

1. `docker compose down` on .99
2. Take a dump: `docker compose exec db pg_dump -U bizmatch bizmatch > backup.sql`
3. Copy the project folder to the AI machine, run `docker compose up -d --build`
4. Restore the dump: `docker compose exec -T db psql -U bizmatch bizmatch < backup.sql`

## API (as of module 6b)

| Method | Path                        | Purpose                                          | Session |
| ------ | --------------------------- | ------------------------------------------------ | ------- |
| GET    | /api/health                 | liveness incl. DB check                          | no      |
| GET    | /api/staff                  | staff list                                       | no      |
| POST   | /api/staff                  | create staff member `{name}`                     | no      |
| POST   | /api/login                  | login via `{staff_id}` → session cookie          | no      |
| GET    | /api/me                     | signed-in staff member                           | yes     |
| POST   | /api/logout                 | sign out                                         | yes     |
| POST   | /api/businesses/scan        | scan the NAS → `{scanned, inserted, updated, missing}` | yes |
| GET    | /api/businesses             | list `?status=&search=` + counts per status      | yes     |
| GET    | /api/businesses/:id         | single business incl. `nas_path`                 | yes     |
| GET    | /api/businesses/:id/files   | recursive listing, max depth 3 (PDFs first)      | yes     |
| GET    | /api/businesses/:id/file    | stream one file, `?path=<relative>`              | yes     |
| GET    | /api/businesses/:id/deals   | buyer activity on one business, newest first     | yes     |
| GET    | /api/buyers                 | list `?search=&status=` + counts per buyer status | yes    |
| GET    | /api/buyers/duplicates      | candidates for `?email=&name=&phone=`            | yes     |
| POST   | /api/inquiries              | guided new-inquiry flow (one transaction)        | yes     |
| GET    | /api/buyers/:id             | buyer incl. `contacts[]` and `ndas[].deals[]`    | yes     |
| PATCH  | /api/buyers/:id             | identity fields + status (+ `end_open_deals`)    | yes     |
| POST   | /api/buyers/:id/contacts    | add a contact                                    | yes     |
| PATCH  | /api/contacts/:id           | edit a contact (incl. `is_primary`)              | yes     |
| PATCH  | /api/ndas/:id               | edit one NDA round                               | yes     |
| POST   | /api/ndas/:id/deals         | add a business to an existing round              | yes     |
| POST   | /api/deals/:id/status       | `{status, comment?}` — transition + note         | yes     |
| GET    | /api/deals/:id/notes        | notes of one deal, newest first                  | yes     |
| POST   | /api/deals/:id/follow-up-sent | `{comment?, rearm}` — note + re-arm or clear   | yes     |
| POST   | /api/notes                  | create a note on exactly one reference object    | yes     |
| GET    | /api/notes                  | notes of one object, `?…_id=` (+`include_related`) | yes   |
| PATCH  | /api/notes/:id              | edit `text` / `highlight`                        | yes     |
| DELETE | /api/notes/:id              | delete a note                                    | yes     |
| POST   | /api/todos                  | create a todo                                    | yes     |
| GET    | /api/todos                  | `?assigned_to=&status=` + one ref id as scope    | yes     |
| PATCH  | /api/todos/:id              | text, due_at, assigned_to, kind, document_id     | yes     |
| POST   | /api/todos/:id/done         | close it (`done_by`/`done_at` = session, now)    | yes     |
| POST   | /api/todos/:id/reopen       | reopen it and clear both                         | yes     |
| POST   | /api/documents              | pin a business file so a REVIEW todo can link it | yes     |
| GET    | /api/today                  | `?staff_id=` — the day's work + nav counts       | yes     |
| GET    | /api/nda-inbox              | mirrored signature requests, `?since=<iso date>` | yes     |
| POST   | /api/nda-inbox/refresh      | start the background walk (202 / 409 if running) | yes     |
| POST   | /api/nda-inbox/import       | import one request into buyer/contact/nda(/deal) | yes     |
| POST   | /api/nda-inbox/sync         | background: re-check pending NDAs + stale mirror rows | yes |
| POST   | /api/nda-inbox/backfill-fields | retrofit form fields onto signed rounds       | yes     |
| DELETE | /api/deals/:id              | delete a deal; notes/todos cascade               | yes     |
| GET    | /api/ndas/:id/file          | stream the filed NDA PDF (Range + ETag)          | yes     |

Everything except health, staff (GET+POST) and login requires the session
cookie; without it the API answers `401`.

### Buyer side (module 4)

Domain rules, all enforced in the API:

* **buyer** is the buying party, **contact** are its 1..n people, **nda** is one
  inquiry round (a returning buyer signs a *new* NDA), **deal** is
  buyer↔business inside one round. There is deliberately no uniqueness on
  `(buyer_id, business_id)` — a returning buyer gets a new round with new deals
  and the history stays visible.
* Deal flow `NEW → INFO_SENT → DUE_DILIGENCE → LOI → CLOSING`, `ENDED` from
  anywhere. `POST /api/deals/:id/status` rejects only a no-op (`409`);
  everything else is allowed on purpose, because corrections have to be
  possible. Entering `INFO_SENT` sets `follow_up_at = today + 14`, entering
  `ENDED` clears it.
* When an NDA becomes `SIGNED` it gets a `signed_at` (default today) and its
  buyer is set back to `ACTIVE`.
* Deactivating a buyer with `{"status":"DEACTIVATED","end_open_deals":true}`
  ends all their non-`ENDED` deals; the response always carries
  `open_deal_count` so the UI can warn first.

`GET /api/buyers/duplicates` matches exactly, never fuzzily: normalised e-mail
(`lower(btrim(…))`), case-insensitive contact name, and phone **or** cell
compared digits-only, so `(361) 555-0101` and `3615550101` are the same number.
Numbers with fewer than 7 digits are ignored. A candidate reports every reason
it matched in `matched_on`.

`POST /api/inquiries` is the guided flow and runs in one transaction. Without
`buyer_id` it creates buyer + primary contact; with `buyer_id` it reuses the
buyer and only adds the contact when no existing contact of that buyer has the
same normalised e-mail or the same name. It then creates the NDA round and one
deal, and returns
`{buyer_id, nda_id, deal_id, created:{buyer, contact}}`. The optional
`backfill` block (`deal_status`, `nda_status`, `signed_at`, `nda_nas_path`)
files a paper record in its real state — a backfilled `INFO_SENT` still arms
the 14-day follow-up, later statuses do not.

### File listing and streaming

`/files` walks the business directory recursively (max depth 3), skipping
dotfiles, dot-directories and symlinks, and returns
`{ path, size, mtime }` with `path` relative to the business directory and
always posix-separated. PDFs come first, then everything else, each group
alphabetical.

`/file?path=…` streams one of those files straight from disk
(`createReadStream`, never buffered):

* the path is resolved against `nas_path` and then `realpath`-validated to be
  inside `realpath(business dir)`. Absolute paths, `..`, leading dots, empty
  paths and symlinks pointing out of the tree get `400`; a missing file `404`.
* single-range HTTP `Range` requests answer `206` with `Content-Range`,
  unsatisfiable ones `416`.
* `ETag` is derived from mtime + size, `If-None-Match` answers `304`.
* `.pdf` is served as `application/pdf` (inline), anything else as
  `application/octet-stream` with `Content-Disposition: attachment`.

### Notes, todos and Today (module 5)

A **note** hangs off exactly one object (buyer, deal, business or NDA round), a
**todo** off at most one — the DB enforces both, and the API mirrors it so a
wrong body gets a `400` instead of a constraint violation. Every note and todo
row carries a `context` object `{type, id, label, buyer_id, business_id}`: the
label is the business name, the buyer's company/contact name or
`Round <date>`, and the two ids let the UI link straight to the right page.
`GET /api/notes?buyer_id=…&include_related=true` additionally folds in the
notes of that buyer's rounds and deals, which is what the buyer page shows.

`GET /api/today` is the daily workqueue, narrowed to one person with
`?staff_id=` and covering the whole team without it:

* **todos** — OPEN, `due_at <= today`, each flagged `overdue` when
  `due_at < today`
* **follow_ups** — deals with `follow_up_at <= today` that are not `ENDED`;
  "mine" means `created_by`
* **pending_ndas** — rounds still `SENT` and not declined after
  `NDA_REMINDER_DAYS` (env, default 14); "mine" means `created_by`
* **counts** — the numbers behind the nav badge

Follow-ups and pending NDAs are **virtual**: they are derived from the deal and
nda rows on every request and never materialise as todo rows, so there is
nothing to keep in sync or clean up. Answering one is
`POST /api/deals/:id/follow-up-sent`, which always writes a note and then
either re-arms the reminder for another 14 days (`rearm: true`) or clears
`follow_up_at` — `409` on a deal that is already ENDED. Giving up on the deal
instead is the normal `POST /api/deals/:id/status` with `ENDED`.

`POST /api/documents` is a stopgap for the REVIEW todo's file picker: it pins
one business file (validated through the same resolver the file streaming uses)
as a `document` row so `todo.document_id` can point at it. Real document
management follows in module 6.

### NDA inbox — Dropbox Sign (module 6a)

Buyers sign their NDA in Dropbox Sign, so that account is the source of new
rounds. `DROPBOX_SIGN_API_KEY` (Basic auth, key as the username, empty
password) enables the inbox routes; without it they answer `503` with a clear
message and the rest of the app is unaffected. The key lives in `.env` only —
it is never committed and never logged.

Only requests whose title starts with **`Buyer Forms - NDA`** are considered.

**The template was renamed around 2026-06-29.** Requests created before that
are titled `Buyer Forms -<Name> - <Business>`, without the ` - NDA`. The
refresh therefore runs **two legs**:

| leg | query | when |
| --- | --- | --- |
| current | `title:"Buyer Forms - NDA" AND created:{<cutoff> TO *}` | always |
| pre-rename | `title:"Buyer Forms" AND created:{<cutoff> TO 2026-06-30}` | only when the window starts before `RENAME_DATE` |

Both feed the same upsert, so the inbox needs no notion of the two formats —
rows are rows. An incremental refresh never pays for the second leg: its
cutoff is days old, well past the rename.

The legacy query is deliberately **`Buyer Forms`, not `Buyer Forms -`**. The
API does not treat the trailing punctuation as part of the phrase: with the
hyphen it returns 70 requests for May/June, without it 535, and ~87% of the
wider set are genuine pre-rename NDAs. Precision comes from the code-side
safety net instead — `/^Buyer Forms -(?!.*NDA)/` plus a signer — which is why
the loose phrase is safe here. Unbounded it would not be: `title:"Buyer Forms"`
matches 11,877 of the account's 13,264 requests, so the date bound is what
makes it selective.

**The inbox is DB-backed.** Proxying the list endpoint on every view mount did
not survive contact with the real account: 700+ requests in a 90-day window
means 7–8 paged calls, ~74s of latency, throttling (Dropbox answers `409` as
well as `429`), and the whole list thrown away on a tab switch. So the requests
are mirrored into `ds_request` by a background task, and

* `GET /api/nda-inbox` reads **only** that table — one indexed query, instant,
  unaffected by tab switches. It returns the same per-request shape as before
  (status, signer, `imported`, `known_buyer` by exact normalised e-mail, up to
  five `business_suggestions` by word overlap with the title remainder) plus
  `last_refresh_at`, `refresh_state` and `covers_from`.
* Filtering is **DB-side**: `?status=pending|signed|declined` and `?q=` over
  signer name and e-mail. The per-status counts describe the whole matching
  set (search applied, status not), so the chips stay meaningful while one is
  active — the same rule the business and buyer lists follow.
* `covers_from` is how far back the mirror actually reaches. A walk stopped by
  the `MAX_PAGES` cap covers less than it was asked for, so it records the
  oldest date it got to and the inbox says "showing data from …" instead of
  presenting a short list as complete. **Coverage only ever widens**: the value
  is written as `LEAST(stored, reached)` in one conditional upsert, so no walk
  and no race can make the mirror claim *less* than it holds. It had drifted to
  a date two months later than the mirror's own oldest row;
  `008_covers_from_widen.sql` corrects the stored value once from
  `min(created_at)` of `ds_request`. Only a full reload writes it at all — an
  incremental pass reads the newest pages and establishes no window.
* `POST /api/nda-inbox/refresh` starts the walk and returns `202` immediately,
  or `409` when one is already running — the slot is claimed with a conditional
  upsert on `app_meta`, so two clicks cannot start two walks. A `running` state
  left behind by a killed process is reset at startup.
* **`?since=` means "re-read that whole window"**, so the UI only sends it when
  the user actually moved the date picker, once; a plain Refresh posts without
  it and catches up incrementally. Sending it on every click was what made the
  refresh feel slow in production — the button then said Refresh but did a full
  reload every time. The button reflects this: *Refresh* vs *Reload window*.

#### Two walks, because the endpoint has two behaviours

| mode | when | what it reads |
| --- | --- | --- |
| `incremental` | every plain Refresh | pages 1-2 of the **unfiltered** list, no query, no cutoff |
| `full` | first run, or an explicit `?since=` | the two title queries over the whole window |

**The search index lags the list.** A request created two minutes ago is on
page 1 of the plain list and in *no* filtered result at all. An incremental
walk that searched therefore mirrored nothing while reporting a clean run —
which is how the mirror sat four days behind while every refresh looked
successful. So the incremental pass does not search. It reads the first two
pages as they come (200 requests, two calls) and keeps whatever passes the
title checks — **both** formats, since recent activity reaches back over the
rename.

That works because **the list is ordered by last activity, not by creation**:
verified live, a request created 30 July and signed 3 August sits above ones
created 3 August. So the same two pages carry the brand-new requests *and* the
old pending ones that were signed since — one pass, no per-id fetching. That is
what makes the sync's per-id re-check a backstop rather than the only way an
old row ever updates.

A **full reload searches server-side** rather than paging through everything
and discarding most of it, which turns a 13k-request account into the ~360 that
are ours:

```
query=title:"Buyer Forms - NDA" AND created:[<cutoff-date> TO *]
```

The bounds are **inclusive** (`[…]`, not `{…}`): the exclusive form dropped
every request created on the cutoff day itself. Index lag is irrelevant here —
the window reaches months past it. The two client-side filters are kept as
safety nets: a title that should not have come back is filtered out *and logged
as a warning*, and the timestamp check drops anything older than midnight of
the cutoff day, since the API's date clause is only day-granular.

Both walks pause 500 ms between calls. On `429`/`409` they honour `Retry-After`
but wait at least 10 s, retry a page up to three times, and log the response
body once per run at warn level — we still do not know what Dropbox means by
the `409` it sometimes sends.

Measured against the live account:

| | pages | seen | stored | duration |
| --- | --- | --- | --- | --- |
| full 90-day reload | 4 | 359 | 359 | 33 s |
| incremental, six days behind | 2 | 200 | 187 | 3 s |
| incremental, up to date | 2 | 200 | ~15 new | 3 s |

`seen` is now the unfiltered page count, so it is always `100 × pages` — an
incremental pass re-storing rows it already has is free (the upsert is
idempotent) and cheaper than any attempt to be clever about it.

**Every walk logs one line**, which is what makes the next anomaly readable
without a debugger:

```
[nda-refresh] walk mode=incremental query=unfiltered window=none (newest activity first, no cutoff) pages=2 seen=200 stored=187
[nda-refresh] walk mode=full leg=current format query="title:\"Buyer Forms - NDA\" AND created:[2026-05-01 TO *]" window=[2026-05-01 TO *] pages=1 seen=2 stored=2
```

#### The two jobs' meta keys are disjoint, and nothing shares a write

`app_meta` holds `ds_refresh_state` (`idle` | `running` | `error:<msg>`),
`ds_last_refresh_at`, `ds_last_refresh_result` and `ds_mirror_covers_from` for
the refresh, and `ds_sync_state` / `ds_last_sync_at` / `ds_last_sync_result`
for the sync. They never cross: the sync *reads* `ds_last_refresh_at` to bound
its candidate set and must not write it.

`ds_last_refresh_at` is not a "the job ran" stamp — it is the boundary the sync
uses to decide which rows the walk can no longer reach, so it may only be
written by a **completed walk that stored what it found**. The shared
`startTask()` therefore no longer stamps `lastAt` for its caller: a helper that
writes "this ran at" on every resolved job is exactly how the marker moved
forward over data nobody had mirrored. Each job writes its own key at the point
where its work is done.

Import and sync are unchanged and still address one `signature_request_id` at a
time. Nothing re-fetches a request per id unnecessarily: sync's candidate set
is `SENT AND NOT declined`, so signed and declined rounds are never fetched
again.

The inbox is ordered and dated by **`coalesce(signed_at, created_at)`** — the
moment that matters is when the request was signed, falling back to when it
arrived while it still is not. 158 of the 566 mirrored requests were signed on
a different day than they were created, so sorting by `created_at` put them in
the wrong place.

### Form fields (module 6b)

A signed request carries every answer of the NDA form in `response_data`, so
importing one needs **no extraction model and no review step** — the values are
read off directly. Verified against the live account, an entry looks like:

```json
{ "name": "Textbox1", "type": "text", "required": true,
  "api_id": "9fb8331d-…", "value": "kkm Foods", "signature_id": "…" }
```

`name` is always present, so the mapping is by name; the `api_id`s are stable
per template slot and are indexed as a fallback key. Two traps the real payload
contains: **checkbox values are the strings `"true"`/`"false"`**, not booleans,
and signature/initials slots carry the literal string `"null"`. Both are
handled in `readValues()`, which also drops empty answers.

| Field | Goes to |
| --- | --- |
| `Textbox1` | `buyer.company_name` |
| `Textbox2` / `Textbox4` | `contact.phone` / `contact.cell` |
| `Textbox5` | `buyer.address` + `buyer.state`, falling back to `Textbox18` — see below |
| `Textbox6` | compared with the signer e-mail — see below |
| `Textbox7` | `buyer.how_heard` |
| `Checkbox1` / `Checkbox2` | `buyer.interested_in_updates` = true / false, neither = null |
| `Textbox8` | `nda.preferred_businesses_text` |
| `Textbox9` | `buyer.background_experience` |
| `Textbox10` / `Textbox11` | `nda.total_purchase_price` / `nda.down_payment` |
| `Textbox12`…`Textbox15` | `nda.income_requirements` / `accountant` / `attorney` / `bank` |
| `Textbox16` | compared with the signer name — see below |
| `DateSigned1` | `nda.intro_date`, parsed from `"07 / 27 / 2026"` |

The template also sends `Textbox3`, `Textbox17`, `Textbox19`, `Textbox20` (a
signature block repeating company, phone and e-mail) and a second
`DateSigned2`. Those are **not mapped** — `Textbox18` from the same block *is*,
as the address fallback described below. The whole array is stored verbatim in
`nda.raw_form_data`, so a mapping mistake can be corrected later without going
back to Dropbox for every round.

Two values are never written, only reported, because they would corrupt the
identity the deduplication relies on:

* `Textbox6` differing from the signer's e-mail → a note on the buyer,
  `NDA form lists different email: <value>`. The signer address stays the
  contact's e-mail.
* `Textbox16` differing from the signer's name → `Form names prospective
  buyer: <value>`. `nda.signer_name` stays the Dropbox signer.

Everything is written with `coalesce(nullif(btrim(col), ''), <new>)`, i.e.
**fill only what is empty**. On a freshly created buyer every column is NULL so
that fills all of them; on a reused buyer it can never overwrite curated data.
The same guard covers the round, which is what makes the retrofit safe to run
over NDAs people have already edited by hand.

`POST /api/nda-inbox/backfill-fields` is that retrofit: it walks every round
with a `dropbox_sign_id`, status `SIGNED` and `raw_form_data IS NULL`, fetches
it with a 500 ms pause between calls, and answers
`{candidates, filled, empty, failed, warnings}`. It doubles as the initial load
for the first three months and is safe to re-run — a filled round is no longer
a candidate.

Two normalisations run over the extracted values before they are stored. Both
only affect the mapped columns — `raw_form_data` always keeps the verbatim
answers, so nothing is lost.

**Null markers.** A value that is nothing but a "does not apply" marker becomes
NULL: `/^(n|na|n\/a|none|nil|x+|-+|\.+)$/i` after trimming — so `n`, `na`,
`N/A`, `none`, `nil`, `x`/`xx`/`xxx`, any run of dashes and any run of dots.
It is deliberately anchored, which is what keeps `"NASA"`, `"Nancy"`,
`"x-ray"`, `"N. Smith"` and `"none of the above"`. Junk that is not a marker
(`"open"`, `"enough"`) is kept too: that is what the signer wrote, and only an
exact marker is safe to discard.

> ⚠️ **Two-track normalisation — keep both in sync.** The legacy vision
> pipeline in the QC repo carries its own `NULL_MARKERS` for the scanned-PDF
> route. The `x` → `x+` widening above has **not** been applied there. Whoever
> next touches that pipeline must make the same change, otherwise the same NDA
> yields `"xx"` from the scan route and `NULL` from the Dropbox Sign route, and
> the two sources silently disagree about what "no answer" means.

**Address fallback.** The form asks for the address twice — `Textbox5` in the
body and `Textbox18` in the signature block — and signers routinely type only a
street in the first and the complete address in the second. When `Textbox5`
carries no state, `Textbox18` is tried; if *it* has one, its address **and**
state are taken together, since combining a street from one field with a state
from the other would invent an address. If neither has a state, `Textbox5`'s
street is kept as-is and the state stays NULL.

`POST /api/nda-inbox/import` is one transaction: buyer + contact (the same
reuse rules as the guided inquiry — see `ensureBuyerAndContact`), the NDA round
with `dropbox_sign_id` (UNIQUE, so a second import is a `409`), and optionally
a deal. A signed request becomes `SIGNED` and runs the usual signed rule.
`custom_fields` are ignored — field extraction is a later module.

**Declined rounds** stay `SENT` with `declined = true`: they were never signed,
but they must not look merely outstanding either. They are therefore excluded
from the sync candidate set (a decline is final on the Dropbox side) *and* from
the `pending_ndas` section of the Today view (nobody is waiting for that
signature). So that the event does not vanish silently, flipping to declined —
whether on import or during a sync — writes a highlighted note on the buyer:
`NDA declined via Dropbox Sign, <date of the round>`.

Signed PDFs are filed under `NDA_ROOT`:

```
NDA_ROOT/<letter>/active/<Last>, <First> <YYYY-MM-DD> NDA.pdf   buyer is active
NDA_ROOT/<letter>/<Last>, <First> <YYYY-MM-DD> NDA.pdf          buyer is not
```

`<letter>` is the first letter of the last name, uppercased; anything outside
A–Z is filed under `_`. Collisions get ` (2)`, ` (3)` … before the extension.
Setting a buyer to `DEACTIVATED` moves their filed NDAs out of `active/` and
`ACTIVE` moves them back; each move is wrapped individually, and a failure only
adds `{nda_id, error}` to the `warnings` array of the PATCH response — the
status change itself always stands. The same holds for the import: a PDF that
cannot be downloaded or written returns a `warning` instead of rolling back an
import that already succeeded.

`POST /api/nda-inbox/sync` runs in the background exactly like the refresh —
`202` when started, `409` when one is already running, state and result in
`app_meta` under `ds_sync_state` / `ds_last_sync_at` / `ds_last_sync_result`,
and the UI polls until it reports the counts. It does two things:

1. Re-checks every NDA that still has status `SENT`, a `dropbox_sign_id` and
   `declined = false`, applying the signed rule and filing the PDF for the ones
   that came in, and flagging the ones that were declined.
2. Re-fetches mirror rows the incremental walk can no longer reach — but only
   those worth asking about, which is what keeps the run finite:

| Bound | Why |
| --- | --- |
| `created_at` older than the refresh reaches | anything newer was just re-read by the walk |
| `fetched_at` older than 12 h | without it every run re-fetches the same few hundred rows |
| `created_at` within `SYNC_PENDING_MAX_AGE_DAYS` (env, default 60) | a request pending that long is realistically dead |

A request past the age cutoff is *not* deleted or hidden: it stays in the inbox
as pending and can still be imported by hand. We simply stop asking Dropbox
about it. The answer carries both halves apart —
`{checked, signed, declined, failed, mirror_candidates, mirror_rechecked,
mirror_changed, mirror_failed, warnings, warnings_omitted}` — because a mirror
row that cannot be re-read is a stale cache entry, not an NDA that failed.

Both background jobs share `src/background-task.ts`: the same atomic slot
claim, the same `idle` / `running` / `error:<msg>` state, the same
stale-state reset at startup.

`DROPBOX_SIGN_BASE_URL` exists so the whole flow can be exercised against a
local stub; leave it unset in production.

## Frontend

`web/` is a Vite + React + TypeScript app with Tailwind v4 (no router, no state
library). The header carries the nav entries **Today**, **NDA Inbox**,
**Businesses** and **Buyers**; routing is a hand-rolled `{view, id}` state in
`App.tsx`. Today is the landing view and its nav entry shows a red badge with the signed-in user's
overdue + due todos. There is no polling, so the badge is refreshed on mount,
on every view change and after any action that can move an item off the list.
Views:

* login ("Who is working?")
* today — "My day" / "Team" tabs (Team groups todos by assignee and follow-ups
  by their creator) over three sections: todos (overdue in red, checkbox to
  complete, context chip opens the buyer or business), follow-ups due (with
  "Follow-up sent…" → comment + "wait another 14 days" / "stop waiting", and
  "End deal…" → the status dialog preset to ENDED) and NDA signatures pending.
  Empty state: "Nothing due. Enjoy your coffee."
* nda inbox — renders straight from the mirror on mount, never from Dropbox.
  The header shows "Last refreshed: &lt;relative time&gt;"; Refresh starts the
  background walk and polls every 3s until it reports idle, then reloads the
  table, while the table stays usable throughout. Plus "Sync signatures", and
  one row per request with a status chip, a known-buyer chip linking to the
  buyer (or "new"), a business select preselected to the best suggestion
  ("— no deal —" to import the round on its own) and an Import button.
  Imported rows show a checkmark that opens the buyer instead.
* business list (tabs with counts, search, "Scan NAS now")
* business detail — a master-detail split filling the viewport: file table
  left, PDF viewer right, plus collapsed "Buyer activity", "Notes" and "Todos"
  panels above it
* buyer list (status chips with counts, search over company/contact/e-mail,
  "New inquiry")
* new inquiry — contact + business picker; while typing a known name, e-mail or
  phone a warning panel lists the duplicate candidates with "Use this buyer"
  (locks the buyer, shown as a chip with an undo) or "Create new buyer anyway".
  The collapsible "Backfill existing deal (paper records)" section files
  historic deals in their real state.
* buyer detail — status header with Deactivate/Reactivate (warns about the open
  deals it would end), inline-editable identity panel, contacts with a primary
  star, a notes panel covering the buyer *and* their rounds and deals, a todos
  panel, and the NDA rounds newest first: editable round fields, the deals of
  the round with an action menu (next step, "End deal", plus a "Correct to…"
  section) that opens a comment dialog, and a collapsed "Notes & todos" section
  per deal. A round with a filed PDF gets a "View PDF" button opening the
  pdf.js viewer on `/api/ndas/:id/file`; a declined round carries a red
  "Declined" badge.

Notes are written in a composer at the top of every notes panel; the red flag
button marks a note as important, and flagged notes get a red left border and a
light red background wherever they appear.

In dev, Vite proxies `/api` to `http://localhost:8090`. In production the
Fastify app serves `web/dist` via `@fastify/static` with an SPA fallback to
`index.html` for all non-`/api` routes; the Dockerfile builds the frontend in
its own stage and copies `web/dist` into the runtime image.

### PDF viewer

The viewer is the proven one from the phase-1 Deno desktop app (see
`viewer-phase1/`), ported nearly byte-identical. It lives in
`web/public/viewer/` as plain, unbundled ES modules — Vite serves `public/`
as-is, so the same files work in dev and prod. The React app embeds it in an
`<iframe>`:

```
/viewer/index.html?file=<urlencoded /api/businesses/:id/file?path=...>
```

The page refuses any `file` value that is not a root-relative `/api/` path, and
the iframe is same-origin, so the normal session cookie authenticates it.

`web/public/pdfjs/` holds the pdf.js runtime, copied out of
`node_modules/pdfjs-dist` (pinned to exactly 6.1.200) by
`web/scripts/copy-pdfjs.mjs`, which runs on `predev` and `prebuild` — also
inside the Docker web stage. The directory is generated and git-ignored:

```
web/public/pdfjs/legacy/         pdf.min.mjs + pdf.worker.min.mjs
web/public/pdfjs/wasm/           CCITT-G4/JBIG2, JPEG2000 and ICC decoders
web/public/pdfjs/standard_fonts/ standardFontDataUrl
web/public/pdfjs/iccs/           iccUrl
```

The `wasm/` directory is what makes scanned B/W pages render at all; without it
pdf.js fails the decoders silently and shows blank white canvases.

## Structure

```
migrations/   numbered SQL migrations
  001_init.sql          full schema
  002_buyer_fields.sql  buyer-side fields from the NDA form + intake sheet
  003_…_nullable.sql    interested_in_updates becomes tri-state
  004_reset_….sql       one-time reset of that column to NULL
  005_nda_dropbox_….sql dropbox_sign_id, signer_name, declined on nda
  006_ds_request_….sql  ds_request mirror + the app_meta key/value table
  007_nda_form_….sql    income/accountant/attorney/bank + raw_form_data on nda
src/
  config.ts           env configuration
  db.ts               pg pool, query helpers, withTransaction
  session.ts          the staff-id cookie
  http.ts             input coercion, PATCH/reference helpers, file responses
  migrate.ts          migration runner (transactional, advisory lock)
  business-scan.ts    NAS scan, recursive listing, safe file path resolution
  dropbox-sign.ts     thin Dropbox Sign REST client (list, get, download)
  nda-files.ts        naming, filing and active/inactive moves of NDA PDFs
  background-task.ts  slot claim + idle/running/error state for the two jobs
  nda-refresh.ts      the background walk that mirrors requests into ds_request
  nda-fields.ts       response_data -> buyer/contact/nda, fill-only-what-is-empty
  server.ts           Fastify app (health, staff, login, businesses, file, static)
  buyer-routes.ts     buyers, contacts, NDA rounds, deals, the inquiry flow
  workflow-routes.ts  notes, todos, documents, the Today view, follow-ups
  nda-inbox-routes.ts the NDA inbox, import, signature sync, NDA PDF streaming
web/
  scripts/copy-pdfjs.mjs  pdfjs-dist -> public/pdfjs/ (predev + prebuild)
  public/viewer/      standalone, unbundled pdf.js viewer page
  public/pdfjs/       generated, git-ignored pdf.js runtime
  src/api.ts          typed API client
  src/App.tsx         session gate + nav (with the Today badge) + view switch
  src/components.tsx  shared bits (badges, inline fields, business picker, dialog)
  src/workflow.tsx    notes panel, todo list and the add-todo dialog
  src/views/          Login, Today, NdaInbox, Businesses, BusinessDetail,
                      Buyers, BuyerDetail, NewInquiry
viewer-phase1/        reference copy of the phase-1 desktop viewer
```