nda improvements

README.md

@@ -199,7 +199,7 @@ directory aborts the scan with an error naming the path.
 | GET    | /api/nda-inbox              | mirrored signature requests, `?since=<iso date>` | yes     |
 | POST   | /api/nda-inbox/refresh      | start the background walk (202 / 409 if running) | yes     |
 | POST   | /api/nda-inbox/import       | import one request into buyer/contact/nda(/deal) | yes     |
-| POST   | /api/nda-inbox/sync         | refresh the status of every pending imported NDA | yes     |
+| POST   | /api/nda-inbox/sync         | background: re-check pending NDAs + stale mirror rows | yes |
 | POST   | /api/nda-inbox/backfill-fields | retrofit form fields onto signed rounds       | yes     |
 | GET    | /api/ndas/:id/file          | stream the filed NDA PDF (Range + ETag)          | yes     |
 
@@ -318,15 +318,42 @@ are mirrored into `ds_request` by a background task, and
   (status, signer, `imported`, `known_buyer` by exact normalised e-mail, up to
   five `business_suggestions` by word overlap with the title remainder) plus
   `last_refresh_at` and `refresh_state`.
-* `POST /api/nda-inbox/refresh?since=` starts the walk and returns `202`
-  immediately, or `409` when one is already running — the slot is claimed with
-  a conditional upsert on `app_meta`, so two clicks cannot start two walks. A
-  `running` state left behind by a killed process is reset at startup.
+* `POST /api/nda-inbox/refresh` starts the walk and returns `202` immediately,
+  or `409` when one is already running — the slot is claimed with a conditional
+  upsert on `app_meta`, so two clicks cannot start two walks. A `running` state
+  left behind by a killed process is reset at startup.
+* **`?since=` means "re-read that whole window"**, so the UI only sends it when
+  the user actually moved the date picker, once; a plain Refresh posts without
+  it and catches up incrementally. Sending it on every click was what made the
+  refresh feel slow in production — the button then said Refresh but did a full
+  reload every time. The button reflects this: *Refresh* vs *Reload window*.
 
-The task walks the pages newest-first with a 500 ms pause between calls and
-upserts every NDA request into `ds_request`; re-walking is how a row that was
-pending last time is picked up as signed or declined. It stops at the first
-page whose oldest entry predates the window. On `429`/`409` it honours
+The task **searches server-side** rather than paging through everything and
+discarding most of it. It sends
+
+```
+query=title:"Buyer Forms - NDA" AND created:{<cutoff-date> TO *}
+```
+
+which turns a 13k-request account into the ~360 that are ours. The two
+client-side filters are kept as safety nets: a title that should not have come
+back is filtered out *and logged as a warning*, and the exact-timestamp check
+catches rows from the cutoff day itself, since the API's date clause is only
+day-granular. If the query were ever ignored, the mirror would still be correct
+— just slow again, and the log would say so.
+
+It pages through that filtered set with a 500 ms pause between calls; re-walking
+is how a row that was pending last time is picked up as signed or declined.
+
+Measured against the live account:
+
+| | pages | seen | stored | duration |
+| --- | --- | --- | --- | --- |
+| full 90-day reload | 4 | 359 | 359 | 33 s |
+| incremental, seconds later | 1 | 43 | 43 | 5 s |
+
+`seen` and `stored` now match. The old gap (≈1000 seen, ≈360 stored, 10 pages,
+a minute) *was* the problem: 97% of what it fetched was thrown away. On `429`/`409` it honours
 `Retry-After` but waits at least 10s, retries a page up to three times, and
 logs the response body once per run at warn level — we still do not know what
 Dropbox means by the `409` it sometimes sends. `app_meta` holds
@@ -462,10 +489,33 @@ status change itself always stands. The same holds for the import: a PDF that
 cannot be downloaded or written returns a `warning` instead of rolling back an
 import that already succeeded.
 
-`POST /api/nda-inbox/sync` re-checks every NDA that still has status `SENT`, a
-`dropbox_sign_id` and `declined = false`; it applies the signed rule and files
-the PDF for the ones that came in, flags the ones that were declined, and
-answers with `{checked, signed, declined, failed, warnings}`.
+`POST /api/nda-inbox/sync` runs in the background exactly like the refresh —
+`202` when started, `409` when one is already running, state and result in
+`app_meta` under `ds_sync_state` / `ds_last_sync_at` / `ds_last_sync_result`,
+and the UI polls until it reports the counts. It does two things:
+
+1. Re-checks every NDA that still has status `SENT`, a `dropbox_sign_id` and
+   `declined = false`, applying the signed rule and filing the PDF for the ones
+   that came in, and flagging the ones that were declined.
+2. Re-fetches mirror rows the incremental walk can no longer reach — but only
+   those worth asking about, which is what keeps the run finite:
+
+| Bound | Why |
+| --- | --- |
+| `created_at` older than the refresh reaches | anything newer was just re-read by the walk |
+| `fetched_at` older than 12 h | without it every run re-fetches the same few hundred rows |
+| `created_at` within `SYNC_PENDING_MAX_AGE_DAYS` (env, default 60) | a request pending that long is realistically dead |
+
+A request past the age cutoff is *not* deleted or hidden: it stays in the inbox
+as pending and can still be imported by hand. We simply stop asking Dropbox
+about it. The answer carries both halves apart —
+`{checked, signed, declined, failed, mirror_candidates, mirror_rechecked,
+mirror_changed, mirror_failed, warnings, warnings_omitted}` — because a mirror
+row that cannot be re-read is a stale cache entry, not an NDA that failed.
+
+Both background jobs share `src/background-task.ts`: the same atomic slot
+claim, the same `idle` / `running` / `error:<msg>` state, the same
+stale-state reset at startup.
 
 `DROPBOX_SIGN_BASE_URL` exists so the whole flow can be exercised against a
 local stub; leave it unset in production.
@@ -575,6 +625,7 @@ src/
   business-scan.ts    NAS scan, recursive listing, safe file path resolution
   dropbox-sign.ts     thin Dropbox Sign REST client (list, get, download)
   nda-files.ts        naming, filing and active/inactive moves of NDA PDFs
+  background-task.ts  slot claim + idle/running/error state for the two jobs
   nda-refresh.ts      the background walk that mirrors requests into ds_request
   nda-fields.ts       response_data -> buyer/contact/nda, fill-only-what-is-empty
   server.ts           Fastify app (health, staff, login, businesses, file, static)