fix nda

.claude/settings.local.json

@@ -66,7 +66,10 @@
       "Bash(bash /tmp/claude-1000/-home-aknuth-git-bizmatch-app/0d38fdce-4b18-4159-86e7-135dadf9386d/scratchpad/acceptance6c.sh)",
       "Bash(bash /tmp/claude-1000/-home-aknuth-git-bizmatch-app/0d38fdce-4b18-4159-86e7-135dadf9386d/scratchpad/acceptance6d.sh)",
       "Bash(bash /tmp/claude-1000/-home-aknuth-git-bizmatch-app/0d38fdce-4b18-4159-86e7-135dadf9386d/scratchpad/acceptance6e.sh)",
-      "Bash(awk '{print \"  size:\", $5, \"bytes  modified:\", $6, $7, $8}')"
+      "Bash(awk '{print \"  size:\", $5, \"bytes  modified:\", $6, $7, $8}')",
+      "Bash(curl -s -m 5 http://localhost:8090/api/health)",
+      "Bash(curl -s -i -X POST -H 'cookie: bizmatch_staff=__CMDSUB_OUTPUT__' http://localhost:8090/api/nda-inbox/refresh)",
+      "Bash(grep -n \"The task \\\\*\\\\*searches server-side\\\\*\\\\*\\\\|^It pages through that filtered set\\\\|^Measured against the live account\\\\|^\\\\`ds_last_refresh_at\\\\` and\\\\|^Import and sync are unchanged\" README.md)"
     ]
   }
 }

README.md

@@ -116,6 +116,25 @@ silently misreport when it was last refreshed.
 Anything that writes files (NDA filing) must point `NDA_ROOT` at a scratch
 directory for the run; never let a test write into the real NAS tree.
 
+### The Dropbox Sign stub
+
+`scripts/ds-stub.mjs` serves the two endpoints the inbox uses and reproduces
+the property that broke the refresh: **its search index lags its list.** A
+fixture entry is `{request, indexed}` — the truth the plain list and the per-id
+endpoint hand out, and the possibly stale (or entirely missing) copy that
+`query=` searches. It parses the range syntax for real, so `[a TO b]` and
+`{a TO b}` genuinely differ. Point `DROPBOX_SIGN_BASE_URL` at it.
+
+`node scripts/acceptance-nda-refresh.mjs` is the acceptance run: it starts the
+real server against the stub and the real development database and asserts that
+an incremental refresh picks up a request the index does not have yet and an
+old pending one signed since, that a full reload includes the boundary day,
+that a sync leaves `ds_last_refresh_at` untouched, and that `covers_from` never
+narrows. It snapshots every `ds_*` key in `app_meta` and writes it back
+afterwards, deletes its `req-*` rows, and runs with
+`SYNC_PENDING_MAX_AGE_DAYS=0` so the sync cannot start re-fetching the real
+mirror row by row.
+
 ## NAS mount
 
 Mount it on the host via NFS, e.g. in `/etc/fstab`:
@@ -348,9 +367,13 @@ are mirrored into `ds_request` by a background task, and
 * `covers_from` is how far back the mirror actually reaches. A walk stopped by
   the `MAX_PAGES` cap covers less than it was asked for, so it records the
   oldest date it got to and the inbox says "showing data from …" instead of
-  presenting a short list as complete. Coverage only ever widens: an
+  presenting a short list as complete. **Coverage only ever widens**: the value
-  incremental walk reaching back two days does not un-mirror what a full reload
+  is written as `LEAST(stored, reached)` in one conditional upsert, so no walk
-  fetched last week.
+  and no race can make the mirror claim *less* than it holds. It had drifted to
+  a date two months later than the mirror's own oldest row;
+  `008_covers_from_widen.sql` corrects the stored value once from
+  `min(created_at)` of `ds_request`. Only a full reload writes it at all — an
+  incremental pass reads the newest pages and establishes no window.
 * `POST /api/nda-inbox/refresh` starts the walk and returns `202` immediately,
   or `409` when one is already running — the slot is claimed with a conditional
   upsert on `app_meta`, so two clicks cannot start two walks. A `running` state
@@ -361,37 +384,84 @@ are mirrored into `ds_request` by a background task, and
   refresh feel slow in production — the button then said Refresh but did a full
   reload every time. The button reflects this: *Refresh* vs *Reload window*.
 
-The task **searches server-side** rather than paging through everything and
+#### Two walks, because the endpoint has two behaviours
-discarding most of it. It sends
+
+| mode | when | what it reads |
+| --- | --- | --- |
+| `incremental` | every plain Refresh | pages 1-2 of the **unfiltered** list, no query, no cutoff |
+| `full` | first run, or an explicit `?since=` | the two title queries over the whole window |
+
+**The search index lags the list.** A request created two minutes ago is on
+page 1 of the plain list and in *no* filtered result at all. An incremental
+walk that searched therefore mirrored nothing while reporting a clean run —
+which is how the mirror sat four days behind while every refresh looked
+successful. So the incremental pass does not search. It reads the first two
+pages as they come (200 requests, two calls) and keeps whatever passes the
+title checks — **both** formats, since recent activity reaches back over the
+rename.
+
+That works because **the list is ordered by last activity, not by creation**:
+verified live, a request created 30 July and signed 3 August sits above ones
+created 3 August. So the same two pages carry the brand-new requests *and* the
+old pending ones that were signed since — one pass, no per-id fetching. That is
+what makes the sync's per-id re-check a backstop rather than the only way an
+old row ever updates.
+
+A **full reload searches server-side** rather than paging through everything
+and discarding most of it, which turns a 13k-request account into the ~360 that
+are ours:
 
 ```
-query=title:"Buyer Forms - NDA" AND created:{<cutoff-date> TO *}
+query=title:"Buyer Forms - NDA" AND created:[<cutoff-date> TO *]
 ```
 
-which turns a 13k-request account into the ~360 that are ours. The two
+The bounds are **inclusive** (`[…]`, not `{…}`): the exclusive form dropped
-client-side filters are kept as safety nets: a title that should not have come
+every request created on the cutoff day itself. Index lag is irrelevant here —
-back is filtered out *and logged as a warning*, and the exact-timestamp check
+the window reaches months past it. The two client-side filters are kept as
-catches rows from the cutoff day itself, since the API's date clause is only
+safety nets: a title that should not have come back is filtered out *and logged
-day-granular. If the query were ever ignored, the mirror would still be correct
+as a warning*, and the timestamp check drops anything older than midnight of
-— just slow again, and the log would say so.
+the cutoff day, since the API's date clause is only day-granular.
 
-It pages through that filtered set with a 500 ms pause between calls; re-walking
+Both walks pause 500 ms between calls. On `429`/`409` they honour `Retry-After`
-is how a row that was pending last time is picked up as signed or declined.
+but wait at least 10 s, retry a page up to three times, and log the response
+body once per run at warn level — we still do not know what Dropbox means by
+the `409` it sometimes sends.
 
 Measured against the live account:
 
 | | pages | seen | stored | duration |
 | --- | --- | --- | --- | --- |
 | full 90-day reload | 4 | 359 | 359 | 33 s |
-| incremental, seconds later | 1 | 43 | 43 | 5 s |
+| incremental, six days behind | 2 | 200 | 187 | 3 s |
+| incremental, up to date | 2 | 200 | ~15 new | 3 s |
 
-`seen` and `stored` now match. The old gap (≈1000 seen, ≈360 stored, 10 pages,
+`seen` is now the unfiltered page count, so it is always `100 × pages` — an
-a minute) *was* the problem: 97% of what it fetched was thrown away. On `429`/`409` it honours
+incremental pass re-storing rows it already has is free (the upsert is
-`Retry-After` but waits at least 10s, retries a page up to three times, and
+idempotent) and cheaper than any attempt to be clever about it.
-logs the response body once per run at warn level — we still do not know what
+
-Dropbox means by the `409` it sometimes sends. `app_meta` holds
+**Every walk logs one line**, which is what makes the next anomaly readable
-`ds_last_refresh_at` and `ds_refresh_state` (`idle` | `running` |
+without a debugger:
-`error:<msg>`).
+
+```
+[nda-refresh] walk mode=incremental query=unfiltered window=none (newest activity first, no cutoff) pages=2 seen=200 stored=187
+[nda-refresh] walk mode=full leg=current format query="title:\"Buyer Forms - NDA\" AND created:[2026-05-01 TO *]" window=[2026-05-01 TO *] pages=1 seen=2 stored=2
+```
+
+#### The two jobs' meta keys are disjoint, and nothing shares a write
+
+`app_meta` holds `ds_refresh_state` (`idle` | `running` | `error:<msg>`),
+`ds_last_refresh_at`, `ds_last_refresh_result` and `ds_mirror_covers_from` for
+the refresh, and `ds_sync_state` / `ds_last_sync_at` / `ds_last_sync_result`
+for the sync. They never cross: the sync *reads* `ds_last_refresh_at` to bound
+its candidate set and must not write it.
+
+`ds_last_refresh_at` is not a "the job ran" stamp — it is the boundary the sync
+uses to decide which rows the walk can no longer reach, so it may only be
+written by a **completed walk that stored what it found**. The shared
+`startTask()` therefore no longer stamps `lastAt` for its caller: a helper that
+writes "this ran at" on every resolved job is exactly how the marker moved
+forward over data nobody had mirrored. Each job writes its own key at the point
+where its work is done.
 
 Import and sync are unchanged and still address one `signature_request_id` at a
 time. Nothing re-fetches a request per id unnecessarily: sync's candidate set

migrations/008_covers_from_widen.sql

@@ -0,0 +1,24 @@
+-- BizMatch Phase 2 — module 6a fix: ds_mirror_covers_from may only widen
+--
+-- The value had drifted to a date *later* than the mirror's own oldest row,
+-- so the inbox announced "showing data from <late date>" over a table that in
+-- fact reached months further back. Coverage is now written as
+-- LEAST(existing, reached) (see widenCoverage() in src/nda-refresh.ts); this
+-- corrects the value that is already stored, once, from the only source that
+-- cannot be wrong about it — the mirror itself.
+--
+-- Day-truncated: the mirror holds every request of that day, so midnight is
+-- the honest bound and it is never narrower than the timestamp.
+
+WITH mirror AS (
+  SELECT date_trunc('day', min(created_at) AT TIME ZONE 'UTC') AS from_at FROM ds_request
+)
+INSERT INTO app_meta (key, value)
+SELECT 'ds_mirror_covers_from',
+       to_char(from_at, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')
+  FROM mirror
+ WHERE from_at IS NOT NULL
+ON CONFLICT (key) DO UPDATE
+  SET value = least(app_meta.value, EXCLUDED.value), updated_at = now()
+  -- Only ever widens: a stored value that already reaches further back stands.
+  WHERE app_meta.value IS NULL OR app_meta.value > EXCLUDED.value;

scripts/acceptance-nda-refresh.mjs

@@ -0,0 +1,414 @@
+#!/usr/bin/env node
+/**
+ * Acceptance for the NDA refresh: incremental mode, the meta-key hygiene of
+ * the two background jobs, and the coverage marker.
+ *
+ * It runs the real server against `scripts/ds-stub.mjs` (a Dropbox Sign whose
+ * search index deliberately lags its list) and the real development database,
+ * per the README convention:
+ *
+ *  * every stubbed request id starts with `req-`, and they are deleted again;
+ *  * the `ds_*` keys in `app_meta` are snapshotted before the run and written
+ *    back afterwards, so a test run cannot leave the mirror looking refreshed
+ *    when it is not.
+ *
+ * Nothing else in the database is touched: the stub only ever hands out `req-`
+ * requests, and the ids of the real mirror are unknown to it.
+ *
+ *   node scripts/acceptance-nda-refresh.mjs
+ */
+import { spawn } from 'node:child_process';
+import { mkdtemp, readFile, rm } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import pg from 'pg';
+import { createStub } from './ds-stub.mjs';
+
+const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
+const APP_PORT = 8099;
+const NDA_PREFIX = 'Buyer Forms - NDA';
+/** Before RENAME_DATE, so the full reload runs its pre-rename leg too. */
+const BOUNDARY_DAY = '2026-05-01';
+const LEGACY_DAY = '2026-05-02';
+const META_KEYS = [
+  'ds_refresh_state',
+  'ds_last_refresh_at',
+  'ds_last_refresh_result',
+  'ds_mirror_covers_from',
+  'ds_sync_state',
+  'ds_last_sync_at',
+  'ds_last_sync_result',
+];
+
+// --------------------------------------------------------------- helpers
+let failures = 0;
+function check(name, ok, detail = '') {
+  console.log(`${ok ? '  ok  ' : ' FAIL '} ${name}${detail ? ` — ${detail}` : ''}`);
+  if (!ok) failures += 1;
+}
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+
+/** `npm run dev` sources .env by hand, so the script does the same. */
+async function loadEnv() {
+  const text = await readFile(path.join(ROOT, '.env'), 'utf8').catch(() => '');
+  for (const line of text.split('\n')) {
+    const match = /^\s*([A-Z0-9_]+)\s*=\s*(.*)$/.exec(line);
+    if (!match) continue;
+    const value = match[2].trim().replace(/^["'](.*)["']$/, '$1');
+    if (process.env[match[1]] === undefined) process.env[match[1]] = value;
+  }
+}
+
+const seconds = (date) => Math.floor(date.getTime() / 1000);
+const daysAgo = (n) => seconds(new Date(Date.now() - n * 86_400_000));
+
+function request(id, title, createdAt, { signedAt = null, name, email } = {}) {
+  return {
+    signature_request_id: id,
+    title,
+    created_at: createdAt,
+    is_complete: signedAt !== null,
+    is_declined: false,
+    files_url: '',
+    signatures: [
+      {
+        signer_name: name,
+        signer_email_address: email,
+        status_code: signedAt ? 'signed' : 'awaiting_signature',
+        signed_at: signedAt,
+      },
+    ],
+    response_data: [],
+  };
+}
+
+/**
+ * The fixture, in the order the assertions reason about:
+ *
+ * | # | request | why |
+ * | 0 | created now, **not in the index** | the lag: only the plain list has it |
+ * | 1 | created 40 days ago, signed yesterday, **stale in the index** | activity order puts it on page 1 |
+ * | 2.. | 200 unrelated requests | fill pages 1-2, so anything below is out of the incremental pass' reach |
+ * | 202 | created on the boundary day | only an inclusive `[day TO *]` returns it |
+ * | 203 | pre-rename title | exercises the second leg of the full reload |
+ */
+function buildFixture(stamp) {
+  const fresh = request(`req-${stamp}-fresh`, `${NDA_PREFIX} Nora Fresh - Acme Diner`, daysAgo(0), {
+    name: 'Nora Fresh',
+    email: `nora.fresh@stubtest.invalid`,
+  });
+  const oldSigned = request(
+    `req-${stamp}-old-signed`,
+    `${NDA_PREFIX} Otto Older - Old Deli`,
+    daysAgo(40),
+    { signedAt: daysAgo(1), name: 'Otto Older', email: 'otto.older@stubtest.invalid' },
+  );
+  const boundary = request(
+    `req-${stamp}-boundary`,
+    `${NDA_PREFIX} Bea Boundary - Border Cafe`,
+    seconds(new Date(`${BOUNDARY_DAY}T09:00:00Z`)),
+    { name: 'Bea Boundary', email: 'bea.boundary@stubtest.invalid' },
+  );
+  const legacy = request(
+    `req-${stamp}-legacy`,
+    'Buyer Forms Regina Legacy - Old Bakery',
+    seconds(new Date(`${LEGACY_DAY}T11:00:00Z`)),
+    { name: 'Regina Legacy', email: 'regina.legacy@stubtest.invalid' },
+  );
+
+  // Everything the account is full of and the refresh is not interested in.
+  const noise = Array.from({ length: 200 }, (_, i) =>
+    request(`req-${stamp}-noise-${i}`, `Other Form ${i} - Someone`, daysAgo(2 + i * 0.05), {
+      name: `Noise ${i}`,
+      email: `noise${i}@stubtest.invalid`,
+    }),
+  );
+
+  /** The index copy of the freshly signed one: still pending, as it was. */
+  const staleIndexCopy = request(
+    oldSigned.signature_request_id,
+    oldSigned.title,
+    oldSigned.created_at,
+    { name: 'Otto Older', email: 'otto.older@stubtest.invalid' },
+  );
+
+  return {
+    ids: {
+      fresh: fresh.signature_request_id,
+      oldSigned: oldSigned.signature_request_id,
+      boundary: boundary.signature_request_id,
+      legacy: legacy.signature_request_id,
+    },
+    fixture: [
+      { request: fresh, indexed: null }, // the lag: not searchable yet at all
+      { request: oldSigned, indexed: staleIndexCopy }, // the lag: stale copy
+      ...noise.map((r) => ({ request: r, indexed: r })),
+      { request: boundary, indexed: boundary },
+      { request: legacy, indexed: legacy },
+    ],
+  };
+}
+
+// ------------------------------------------------------------------ main
+await loadEnv();
+if (!process.env.DATABASE_URL) {
+  console.error('DATABASE_URL is missing — run this from the project root with .env in place');
+  process.exit(2);
+}
+
+const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });
+const stamp = String(Date.now());
+const { ids, fixture } = buildFixture(stamp);
+const stub = createStub(fixture);
+let app = null;
+let scratch = null;
+let metaSnapshot = [];
+const appLog = [];
+
+const cleanup = async () => {
+  if (app && !app.killed) {
+    app.kill('SIGTERM');
+    await sleep(300);
+  }
+  await stub.close().catch(() => {});
+  await db
+    .query(`DELETE FROM ds_request WHERE signature_request_id LIKE 'req-%'`)
+    .catch((err) => console.error(`cleanup of ds_request failed: ${err.message}`));
+  // The meta keys go back exactly as they were, updated_at included: a test
+  // run must not leave "last refreshed" pointing at itself.
+  await db
+    .query('DELETE FROM app_meta WHERE key = ANY($1)', [META_KEYS])
+    .catch((err) => console.error(`cleanup of app_meta failed: ${err.message}`));
+  for (const row of metaSnapshot) {
+    await db
+      .query('INSERT INTO app_meta (key, value, updated_at) VALUES ($1, $2, $3)', [
+        row.key,
+        row.value,
+        row.updated_at,
+      ])
+      .catch((err) => console.error(`restoring ${row.key} failed: ${err.message}`));
+  }
+  if (scratch) await rm(scratch, { recursive: true, force: true }).catch(() => {});
+  await db.end().catch(() => {});
+};
+process.on('SIGINT', async () => {
+  await cleanup();
+  process.exit(130);
+});
+
+const meta = async (key) => {
+  const { rows } = await db.query('SELECT value, updated_at FROM app_meta WHERE key = $1', [key]);
+  return rows[0] ?? null;
+};
+const setMeta = (key, value) =>
+  db.query(
+    `INSERT INTO app_meta (key, value) VALUES ($1, $2)
+     ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value, updated_at = now()`,
+    [key, value],
+  );
+const mirrored = async (id) => {
+  const { rows } = await db.query(
+    'SELECT status, signed_at, created_at FROM ds_request WHERE signature_request_id = $1',
+    [id],
+  );
+  return rows[0] ?? null;
+};
+
+try {
+  metaSnapshot = (
+    await db.query('SELECT key, value, updated_at FROM app_meta WHERE key = ANY($1)', [META_KEYS])
+  ).rows;
+  await db.query(`DELETE FROM ds_request WHERE signature_request_id LIKE 'req-%'`);
+
+  const stubUrl = await stub.listen(0);
+  scratch = await mkdtemp(path.join(tmpdir(), 'bizmatch-nda-'));
+
+  // ------------------------------------------------------------ the app
+  app = spawn('npx', ['tsx', 'src/server.ts'], {
+    cwd: ROOT,
+    env: {
+      ...process.env,
+      PORT: String(APP_PORT),
+      DROPBOX_SIGN_API_KEY: 'stub-key',
+      DROPBOX_SIGN_BASE_URL: stubUrl,
+      NDA_ROOT: scratch,
+      // Keeps the sync's per-id re-check off the real mirror: this run is
+      // about which meta keys it writes, not about re-reading 800 rows.
+      SYNC_PENDING_MAX_AGE_DAYS: '0',
+    },
+    stdio: ['ignore', 'pipe', 'pipe'],
+  });
+  for (const stream of [app.stdout, app.stderr]) {
+    stream.setEncoding('utf8');
+    stream.on('data', (chunk) => {
+      for (const line of chunk.split('\n').filter(Boolean)) appLog.push(line);
+    });
+  }
+
+  const base = `http://127.0.0.1:${APP_PORT}`;
+  const { rows: staff } = await db.query('SELECT id FROM staff ORDER BY created_at LIMIT 1');
+  if (!staff[0]) throw new Error('no staff row to sign in as');
+  const headers = { cookie: `bizmatch_staff=${staff[0].id}` };
+
+  for (let attempt = 0; ; attempt += 1) {
+    if (attempt > 100) throw new Error('the app did not come up');
+    const ok = await fetch(`${base}/api/health`)
+      .then((r) => r.ok)
+      .catch(() => false);
+    if (ok) break;
+    await sleep(300);
+  }
+
+  const inbox = async () =>
+    (await fetch(`${base}/api/nda-inbox?since=2026-01-01`, { headers })).json();
+  const settle = async (field) => {
+    for (let attempt = 0; attempt < 300; attempt += 1) {
+      const body = await inbox();
+      if (body[field] !== 'running') return body;
+      await sleep(500);
+    }
+    throw new Error(`${field} never settled`);
+  };
+
+  // ---------------------------------------- 1 · the incremental pass
+  console.log('\n1 · incremental refresh over a lagging search index');
+  await setMeta('ds_last_refresh_at', new Date(Date.now() - 3_600_000).toISOString());
+  await setMeta('ds_refresh_state', 'idle');
+  // Deliberately narrow, so the coverage assertions have something to widen.
+  await setMeta('ds_mirror_covers_from', '2026-12-31T00:00:00.000Z');
+
+  const searchable = stub.search(`title:"${NDA_PREFIX}" AND created:[${BOUNDARY_DAY} TO *]`);
+  check(
+    'the stub search index really lags the list',
+    !searchable.some((r) => r.signature_request_id === ids.fresh) &&
+      searchable.some((r) => r.signature_request_id === ids.boundary),
+    'the fresh request is in no search result, the older ones are',
+  );
+
+  const started = await fetch(`${base}/api/nda-inbox/refresh`, { method: 'POST', headers });
+  check('POST /refresh answers 202', started.status === 202, `got ${started.status}`);
+  const afterIncremental = await settle('refresh_state');
+  const incremental = afterIncremental.last_refresh_result;
+
+  check('it ran in incremental mode', incremental?.mode === 'incremental', JSON.stringify(incremental));
+  check(
+    'it read the unfiltered list, not the index',
+    stub.calls.filter((c) => c.endpoint === 'list' && c.query).length === 0,
+    `queries sent: ${stub.calls.filter((c) => c.query).length}`,
+  );
+  check(
+    `it walked ${2} pages of 100 unfiltered rows`,
+    incremental?.pages === 2 && incremental?.seen === 200,
+    `pages=${incremental?.pages} seen=${incremental?.seen}`,
+  );
+
+  const fresh = await mirrored(ids.fresh);
+  check(
+    'a request created now lands although the index does not have it',
+    fresh?.status === 'pending',
+    `mirror row: ${JSON.stringify(fresh)}`,
+  );
+  const oldSigned = await mirrored(ids.oldSigned);
+  check(
+    'an old pending signed yesterday is picked up without a sync',
+    oldSigned?.status === 'signed' && oldSigned?.signed_at !== null,
+    `mirror row: ${JSON.stringify(oldSigned)}`,
+  );
+  check(
+    'and it stored nothing beyond those two',
+    incremental?.stored === 2,
+    `stored=${incremental?.stored}`,
+  );
+  check(
+    'rows below the first two pages are out of its reach',
+    (await mirrored(ids.boundary)) === null && (await mirrored(ids.legacy)) === null,
+    'boundary and pre-rename rows are not mirrored yet',
+  );
+
+  // ---------------------------------------- 2 · the full reload
+  console.log('\n2 · full reload with inclusive bounds');
+  const callsBefore = stub.calls.length;
+  const reload = await fetch(`${base}/api/nda-inbox/refresh?since=${BOUNDARY_DAY}`, {
+    method: 'POST',
+    headers,
+  });
+  check('POST /refresh?since= answers 202', reload.status === 202, `got ${reload.status}`);
+  const afterFull = await settle('refresh_state');
+  const full = afterFull.last_refresh_result;
+  const queries = stub.calls.slice(callsBefore).filter((c) => c.query).map((c) => c.query);
+
+  check('it ran in full mode', full?.mode === 'full', JSON.stringify(full));
+  check(
+    'every query uses the inclusive range form',
+    queries.length > 0 && queries.every((q) => q.includes('created:[')) &&
+      !queries.some((q) => q.includes('created:{')),
+    queries[0] ?? 'no query sent',
+  );
+  check(
+    'a request created on the boundary day is included',
+    (await mirrored(ids.boundary)) !== null,
+    `${BOUNDARY_DAY}T09:00Z with since=${BOUNDARY_DAY}`,
+  );
+  check(
+    'the pre-rename leg ran and stored its request',
+    (await mirrored(ids.legacy)) !== null && full?.legacy_stored >= 1,
+    `legacy_stored=${full?.legacy_stored}`,
+  );
+  check(
+    'covers_from widened to the reloaded window',
+    (await meta('ds_mirror_covers_from'))?.value === `${BOUNDARY_DAY}T00:00:00.000Z`,
+    `covers_from=${(await meta('ds_mirror_covers_from'))?.value}`,
+  );
+
+  // ---------------------------------------- 3 · coverage never narrows
+  console.log('\n3 · covers_from only ever widens');
+  await setMeta('ds_mirror_covers_from', '2020-01-01T00:00:00.000Z');
+  await fetch(`${base}/api/nda-inbox/refresh?since=${BOUNDARY_DAY}`, { method: 'POST', headers });
+  await settle('refresh_state');
+  check(
+    'a narrower window does not overwrite a wider coverage',
+    (await meta('ds_mirror_covers_from'))?.value === '2020-01-01T00:00:00.000Z',
+    `covers_from=${(await meta('ds_mirror_covers_from'))?.value}`,
+  );
+
+  // ---------------------------------------- 4 · meta-key hygiene
+  console.log('\n4 · the sync never touches the refresh cursor');
+  const refreshBefore = await meta('ds_last_refresh_at');
+  const synced = await fetch(`${base}/api/nda-inbox/sync`, { method: 'POST', headers });
+  check('POST /sync answers 202', synced.status === 202, `got ${synced.status}`);
+  await settle('sync_state');
+  const refreshAfter = await meta('ds_last_refresh_at');
+  check(
+    'ds_last_refresh_at is untouched by a sync run',
+    refreshBefore?.value === refreshAfter?.value &&
+      String(refreshBefore?.updated_at) === String(refreshAfter?.updated_at),
+    `${refreshBefore?.value} @ ${refreshBefore?.updated_at} -> ${refreshAfter?.value} @ ${refreshAfter?.updated_at}`,
+  );
+  check(
+    'the sync stamps its own key instead',
+    (await meta('ds_last_sync_at'))?.value != null,
+    `ds_last_sync_at=${(await meta('ds_last_sync_at'))?.value}`,
+  );
+
+  // ---------------------------------------- the log line
+  console.log('\nwalk log lines the run produced:');
+  for (const line of appLog) {
+    let msg = line;
+    try {
+      msg = JSON.parse(line).msg ?? line;
+    } catch {
+      /* not JSON — a tsx or npm line */
+    }
+    if (typeof msg === 'string' && msg.includes('[nda-refresh] walk')) console.log(`  ${msg}`);
+  }
+} catch (err) {
+  failures += 1;
+  console.error(`\nrun failed: ${err.stack ?? err.message}`);
+} finally {
+  await cleanup();
+}
+
+console.log(failures === 0 ? '\nall assertions passed' : `\n${failures} assertion(s) failed`);
+process.exit(failures === 0 ? 0 : 1);

scripts/ds-stub.mjs

@@ -0,0 +1,139 @@
+/**
+ * A stand-in for the two Dropbox Sign endpoints the NDA inbox uses, with the
+ * one property that broke the refresh in production: **the search index lags
+ * behind the list**.
+ *
+ * `GET /v3/signature_request/list` answers from two different sources:
+ *
+ * * without `query=` — the live list, ordered by last activity (signed_at,
+ *   falling back to created_at), exactly like the real endpoint;
+ * * with `query=` — the *index*, which only holds the snapshot a fixture entry
+ *   declares in `indexed`. A brand-new request is missing from it entirely,
+ *   and a request signed a minute ago is still in it as pending. That is what
+ *   an incremental refresh must not depend on.
+ *
+ * The range syntax is parsed for real, brackets and all, so `[a TO b]`
+ * (inclusive) and `{a TO b}` (exclusive) actually behave differently — which
+ * is what makes the boundary-day assertion meaningful.
+ *
+ * Run standalone for manual pokes:
+ *   node scripts/ds-stub.mjs --port 8099 --fixture some.json
+ */
+import { createServer } from 'node:http';
+import { readFile } from 'node:fs/promises';
+
+const dayOf = (unixSeconds) => new Date(unixSeconds * 1000).toISOString().slice(0, 10);
+
+/** signed_at when there is one, created_at otherwise — the list's sort key. */
+const activityOf = (request) => request.signatures?.[0]?.signed_at ?? request.created_at;
+
+/**
+ * The subset of the query language the refresh sends:
+ * `title:"<phrase>" AND created:[<day> TO <day|*>]`, either bracket style.
+ */
+export function matchesQuery(request, queryString) {
+  const phrase = /title:"([^"]*)"/.exec(queryString)?.[1];
+  if (phrase && !(request.title ?? '').includes(phrase)) return false;
+
+  const range = /created:([[{])\s*(\S+)\s+TO\s+(\S+?)\s*([\]}])/.exec(queryString);
+  if (range) {
+    const [, open, from, to, close] = range;
+    const day = dayOf(request.created_at);
+    // '[' includes the boundary day, '{' excludes it — the whole point.
+    if (from !== '*' && (open === '[' ? day < from : day <= from)) return false;
+    if (to !== '*' && (close === ']' ? day > to : day >= to)) return false;
+  }
+  return true;
+}
+
+/**
+ * `fixture` is a list of `{request, indexed}`: the truth the list and the
+ * per-id endpoint serve, and the possibly stale (or missing) copy the search
+ * index has of it.
+ */
+export function createStub(fixture) {
+  /** Every call the app made — the test asserts on what it asked for. */
+  const calls = [];
+
+  const byActivity = (a, b) => activityOf(b) - activityOf(a);
+
+  const server = createServer((req, res) => {
+    const url = new URL(req.url, 'http://stub');
+    const send = (status, body) => {
+      const payload = JSON.stringify(body);
+      res.writeHead(status, { 'content-type': 'application/json' });
+      res.end(payload);
+    };
+
+    if (url.pathname === '/v3/signature_request/list') {
+      const page = Number(url.searchParams.get('page') ?? '1');
+      const pageSize = Number(url.searchParams.get('page_size') ?? '20');
+      const queryString = url.searchParams.get('query');
+      calls.push({ endpoint: 'list', page, query: queryString });
+
+      const source = queryString
+        ? fixture
+            .map((entry) => entry.indexed)
+            .filter(Boolean)
+            .filter((request) => matchesQuery(request, queryString))
+        : fixture.map((entry) => entry.request);
+      const ordered = [...source].sort(byActivity);
+      const numPages = Math.max(1, Math.ceil(ordered.length / pageSize));
+      const slice = ordered.slice((page - 1) * pageSize, page * pageSize);
+      return send(200, {
+        signature_requests: slice,
+        list_info: { page, num_pages: numPages, num_results: ordered.length },
+      });
+    }
+
+    const single = /^\/v3\/signature_request\/([^/]+)$/.exec(url.pathname);
+    if (single) {
+      const id = decodeURIComponent(single[1]);
+      calls.push({ endpoint: 'get', id });
+      const entry = fixture.find((e) => e.request.signature_request_id === id);
+      // Unknown ids 404 like the real API — the sync walks over real rows the
+      // stub knows nothing about and must simply report them as failures.
+      if (!entry) return send(404, { error: { error_msg: `no such request: ${id}` } });
+      return send(200, { signature_request: entry.request });
+    }
+
+    return send(404, { error: { error_msg: `unhandled path: ${url.pathname}` } });
+  });
+
+  return {
+    calls,
+    /** Resolves to the base URL once it is listening. */
+    listen(port = 0) {
+      return new Promise((resolve) => {
+        server.listen(port, '127.0.0.1', () =>
+          resolve(`http://127.0.0.1:${server.address().port}`),
+        );
+      });
+    },
+    close: () => new Promise((resolve) => server.close(resolve)),
+    /** What a searching walk would have found — the lag, made visible. */
+    search: (queryString) =>
+      fixture
+        .map((entry) => entry.indexed)
+        .filter(Boolean)
+        .filter((request) => matchesQuery(request, queryString)),
+  };
+}
+
+// -------------------------------------------------------- standalone mode
+if (process.argv[1] && import.meta.url.endsWith(process.argv[1].replace(/^.*\//, ''))) {
+  const args = process.argv.slice(2);
+  const arg = (name, fallback) => {
+    const at = args.indexOf(`--${name}`);
+    return at === -1 ? fallback : args[at + 1];
+  };
+  const file = arg('fixture');
+  if (file) {
+    const fixture = JSON.parse(await readFile(file, 'utf8'));
+    const stub = createStub(fixture);
+    console.log(`[ds-stub] listening on ${await stub.listen(Number(arg('port', '8099')))}`);
+  } else {
+    console.error('usage: node scripts/ds-stub.mjs --fixture <file.json> [--port 8099]');
+    process.exit(2);
+  }
+}

src/background-task.ts

@@ -14,7 +14,13 @@ import { query, queryOne } from './db.js';
 export interface TaskKeys {
   /** idle | running | error:<message> */
   state: string;
-  /** ISO timestamp of the last completed run. */
+  /**
+   * ISO timestamp of the last completed run — written by the **job**, never
+   * by this module. Each job owns its own key and the two sets are disjoint
+   * (`ds_*_refresh_*` vs `ds_*_sync_*`), because `ds_last_refresh_at` is not
+   * merely a display value: it is the boundary the sync uses to decide what
+   * the refresh can no longer reach. See startTask().
+   */
   lastAt: string;
   /** JSON result of the last completed run. */
   lastResult: string;
@@ -71,6 +77,13 @@ export async function clearStaleTask(
 /**
  * Claims the slot and runs `job` detached. `false` means somebody else is
  * already running it — the caller answers 409.
+ *
+ * It deliberately does **not** stamp `keys.lastAt`. A shared helper writing
+ * "this ran at" on every resolved job is how `ds_last_refresh_at` came to move
+ * forward on runs that had walked nothing: the refresh cursor then pointed
+ * past requests that were never mirrored, and nothing downstream could tell
+ * the difference. Each job writes its own `lastAt` at the point where it knows
+ * its work is done and stored.
  */
 export async function startTask<T>(
   keys: TaskKeys,
@@ -84,7 +97,6 @@ export async function startTask<T>(
     try {
       const result = await job();
       await setMeta(keys.lastResult, JSON.stringify(result));
-      await setMeta(keys.lastAt, new Date().toISOString());
       await setMeta(keys.state, 'idle');
     } catch (err) {
       const message = (err as Error).message;

src/dropbox-sign.ts

@@ -105,17 +105,23 @@ export interface SignatureRequestPage {
 }
 
 /**
- * The search the refresh runs against the list endpoint. Filtering server-side
+ * The search a *full* reload runs against the list endpoint. Filtering
- * is the difference between reading 13k requests and reading ~330: without it
+ * server-side is the difference between reading 13k requests and reading ~330:
- * the walk pages through every signature request the account ever had and
+ * without it the walk pages through every signature request the account ever
- * throws away the 97% that are not ours.
+ * had and throws away the 97% that are not ours.
  *
- * `created:{<date> TO *}` is the API's range syntax; the date is a plain
+ * `created:[<date> TO *]` is the API's range syntax. The date is a plain
- * calendar day, which is as precise as the filter goes.
+ * calendar day — as precise as the filter goes — and the brackets are the
+ * **inclusive** form: `{…}` excluded the cutoff day itself, which silently
+ * dropped every request created on the boundary. The refresh window is a
+ * whole day either way, so inclusive is the honest bound.
+ *
+ * Only full reloads search: the index behind `query=` lags minutes behind the
+ * plain list, so an incremental pass cannot rely on it (see nda-refresh.ts).
  */
 export function ndaSearchQuery(since: Date): string {
   const day = since.toISOString().slice(0, 10);
-  return `title:"${NDA_TITLE_PREFIX}" AND created:{${day} TO *}`;
+  return `title:"${NDA_TITLE_PREFIX}" AND created:[${day} TO *]`;
 }
 
 /**
@@ -151,9 +157,16 @@ export const LEGACY_TITLE_QUERY = 'Buyer Forms';
 export function legacyNdaSearchQuery(since: Date): string {
   const from = since.toISOString().slice(0, 10);
   const to = RENAME_DATE.toISOString().slice(0, 10);
-  return `title:"${LEGACY_TITLE_QUERY}" AND created:{${from} TO ${to}}`;
+  // Inclusive on both ends, like the current-format query: the upper bound is
+  // already a day past the rename, so including it only widens the harmless
+  // overlap, and including the lower one is what keeps boundary-day requests.
+  return `title:"${LEGACY_TITLE_QUERY}" AND created:[${from} TO ${to}]`;
 }
 
+/** Current-format NDA: the title prefix is the whole test. */
+export const isNdaRequest = (request: SignatureRequest): boolean =>
+  Boolean(request.title?.startsWith(NDA_TITLE_PREFIX));
+
 /**
  * Safety net for the legacy leg: the old title, no "NDA" anywhere in it (that
  * belongs to the other leg), and an actual signer. Anything else the loose

src/nda-inbox-routes.ts

@@ -26,6 +26,7 @@ import {
   clearStaleTask,
   getMeta,
   readTaskState,
+  setMeta,
   startTask,
 } from './background-task.js';
 import { config } from './config.js';
@@ -45,6 +46,11 @@ const FETCH_PAUSE_MS = 500;
  */
 const MIRROR_RECHECK_AFTER_HOURS = 12;
 
+/**
+ * The sync's own keys — no overlap with the refresh's. It reads
+ * `ds_last_refresh_at` to bound its candidate set and must never write it: the
+ * refresh cursor belongs to the walk that stored the data.
+ */
 export const SYNC_KEYS: TaskKeys = {
   state: 'ds_sync_state',
   lastAt: 'ds_last_sync_at',
@@ -316,6 +322,9 @@ async function syncSignatures(
       `${mirrored}/${stale.length} mirror row(s) re-read, ${mirrorChanged} moved on`,
   );
 
+  // The job stamps its own "last completed" key — and only its own.
+  await setMeta(SYNC_KEYS.lastAt, new Date().toISOString());
+
   // A run over a few hundred stale rows would otherwise answer with a few
   // hundred warning strings; the count says how many were left out.
   const MAX_WARNINGS = 10;

src/nda-refresh.ts

@@ -1,10 +1,10 @@
 import type { FastifyBaseLogger } from 'fastify';
-import { withTransaction } from './db.js';
+import { query, withTransaction } from './db.js';
 import {
-  NDA_TITLE_PREFIX,
   RENAME_DATE,
   type SignatureRequest,
   isLegacyNdaRequest,
+  isNdaRequest,
   isRateLimited,
   legacyNdaSearchQuery,
   listPage,
@@ -21,6 +21,15 @@ import { type TaskKeys, clearStaleTask, getMeta, setMeta, startTask } from './ba
  * them a window walk is 7-8 calls and well over a minute — far too slow to sit
  * in front of a view mount, and enough to get throttled. So it runs detached,
  * paced, and at most once at a time.
+ *
+ * Two modes, because the endpoint behaves differently with and without a
+ * `query=`:
+ *
+ * * **incremental** — the first pages of the *unfiltered* list. Cheap, and
+ *   immune to the lag of the search index.
+ * * **full** — the two title queries over a whole window. Only for the first
+ *   run and an explicit `?since=`, where the lag does not matter because the
+ *   window reaches far past it.
  */
 
 /** Between two list calls, so a full walk does not look like a hammer. */
@@ -31,6 +40,13 @@ const MAX_PAGE_ATTEMPTS = 3;
 /** Safety net against a window that never reaches its cutoff. */
 const MAX_PAGES = 50;
 
+/**
+ * How many pages of the unfiltered list an incremental pass reads. Two pages
+ * are 200 requests of recent activity across the whole account — comfortably
+ * more than a day's worth, and two calls instead of a paged window walk.
+ */
+const INCREMENTAL_PAGES = 2;
+
 export const REFRESH_KEYS: TaskKeys = {
   state: 'ds_refresh_state',
   lastAt: 'ds_last_refresh_at',
@@ -55,8 +71,20 @@ export const OVERLAP_MS = 48 * 60 * 60 * 1000;
 
 export type RefreshState = 'idle' | 'running' | `error:${string}`;
 
+/** See the module comment: one reads the plain list, the other searches. */
+export type RefreshMode = 'incremental' | 'full';
+
 const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
 
+const day = (at: Date): string => at.toISOString().slice(0, 10);
+
+/**
+ * The search bounds are calendar days, so the window really starts at midnight
+ * of the cutoff day — the exact-timestamp check has to agree with the query,
+ * or it throws away the boundary-day rows the inclusive bound just fetched.
+ */
+const startOfDay = (at: Date): Date => new Date(`${day(at)}T00:00:00.000Z`);
+
 /**
  * Mirrors a batch of requests. Exported because the sync route re-checks old
  * pending rows one by one, which an incremental walk no longer reaches.
@@ -97,13 +125,19 @@ export async function upsertRequests(requests: SignatureRequest[]): Promise<void
 }
 
 export interface RefreshResult {
+  /** Which of the two walks ran — the log line says the same. */
+  mode: RefreshMode;
   pages: number;
   stored: number;
   seen: number;
   /** Of `stored`, how many came from the pre-rename title format. */
   legacy_stored: number;
-  /** Earliest creation date this walk actually reached, ISO date. */
+  /**
-  covers_from: string;
+   * Earliest creation date this walk covers, ISO date. Null for an
+   * incremental pass: it reads the newest pages and therefore establishes no
+   * window at all — coverage is whatever the last full reload left behind.
+   */
+  covers_from: string | null;
   /** True when MAX_PAGES ended the walk before it reached the cutoff. */
   truncated: boolean;
 }
@@ -120,6 +154,34 @@ interface LegResult {
   complete: boolean;
 }
 
+/**
+ * One line per walk, in one place, so the next anomaly can be read off the log
+ * without guessing what the run actually asked Dropbox for.
+ */
+function logWalk(
+  log: FastifyBaseLogger,
+  fields: {
+    mode: RefreshMode;
+    leg?: string;
+    /** The `query=` sent, or "unfiltered" when the pass sends none. */
+    query: string;
+    window: string;
+    pages: number;
+    seen: number;
+    stored: number;
+    skipped?: number;
+  },
+): void {
+  log.info(
+    `[nda-refresh] walk mode=${fields.mode}` +
+      (fields.leg ? ` leg=${fields.leg}` : '') +
+      ` query=${fields.query === 'unfiltered' ? 'unfiltered' : JSON.stringify(fields.query)}` +
+      ` window=${fields.window} pages=${fields.pages} seen=${fields.seen}` +
+      ` stored=${fields.stored}` +
+      (fields.skipped ? ` skipped=${fields.skipped}` : ''),
+  );
+}
+
 /** Fetches one page, backing off and retrying while Dropbox throttles us. */
 async function fetchPageWithBackoff(
   page: number,
@@ -151,21 +213,21 @@ async function fetchPageWithBackoff(
 }
 
 /**
- * How far back this walk has to reach.
+ * Which walk the next run does, and over which window.
  *
- * A full window is only needed the first time. Afterwards everything older
+ * A full window is only needed the first time, and whenever the caller asks
- * than the previous refresh (minus the overlap) is already mirrored, so the
+ * for one by moving the date picker — that is what an explicit `since` means.
- * walk stops after a page or two instead of re-reading a thousand requests
+ * Everything else is the incremental pass, which needs no window at all.
- * every run. An explicitly requested `since` always wins — that is how the
- * caller asks for a deliberate re-read of a wider period.
  */
-export async function refreshCutoff(since: Date, explicitSince: boolean): Promise<Date> {
+export async function refreshPlan(
-  if (explicitSince) return since;
+  since: Date,
+  explicitSince: boolean,
+): Promise<{ mode: RefreshMode; window: Date }> {
+  if (explicitSince) return { mode: 'full', window: since };
+  // Nothing mirrored yet: there is no "new since" to read.
   const last = await getMeta(META_LAST_REFRESH);
-  if (!last) return since; // first run: the whole window
+  if (!last || Number.isNaN(new Date(last).getTime())) return { mode: 'full', window: since };
-  const incremental = new Date(new Date(last).getTime() - OVERLAP_MS);
+  return { mode: 'incremental', window: since };
-  if (Number.isNaN(incremental.getTime())) return since;
-  return incremental > since ? incremental : since;
 }
 
 /**
@@ -177,7 +239,8 @@ export async function refreshCutoff(since: Date, explicitSince: boolean): Promis
  */
 async function walkLeg(
   label: string,
-  query: string,
+  search: string,
+  window: string,
   cutoff: number,
   accept: (request: SignatureRequest) => boolean,
   log: FastifyBaseLogger,
@@ -192,7 +255,7 @@ async function walkLeg(
 
   for (; page <= MAX_PAGES; page += 1) {
     if (page > 1) await sleep(PAGE_PAUSE_MS);
-    const result = await fetchPageWithBackoff(page, query, log, loggedBody);
+    const result = await fetchPageWithBackoff(page, search, log, loggedBody);
     if (result.requests.length === 0) {
       complete = true;
       break;
@@ -212,8 +275,9 @@ async function walkLeg(
       skipped += rejected;
     }
 
-    // The date clause is day-granular, so a row from the cutoff day itself can
+    // The date clause is day-granular and inclusive, so the window starts at
-    // come back slightly too old.
+    // midnight of the cutoff day; anything older than that the query should
+    // not have returned at all.
     const inWindow = wanted.filter((request) => request.created_at >= cutoff);
     await upsertRequests(inWindow);
     stored += inWindow.length;
@@ -228,26 +292,101 @@ async function walkLeg(
     }
   }
 
-  return { pages: Math.min(page, MAX_PAGES), seen, stored, skipped, oldest, complete };
+  const pages = Math.min(page, MAX_PAGES);
+  logWalk(log, { mode: 'full', leg: label, query: search, window, pages, seen, stored, skipped });
+  return { pages, seen, stored, skipped, oldest, complete };
 }
 
 /**
- * Mirrors every NDA request created since `cutoffAt`.
+ * The incremental pass: the first pages of the **unfiltered** list.
+ *
+ * Two properties of the endpoint make this the right shape:
+ *
+ * 1. The index behind `query=` lags the list itself. A request created two
+ *    minutes ago is on page 1 of the plain list and in *no* filtered result,
+ *    so a searching incremental walk silently mirrors nothing while reporting
+ *    a clean run — which is exactly how the mirror fell four days behind.
+ * 2. The list is ordered by **last activity**, not by creation. A request from
+ *    last month that was signed yesterday sits above one created today, so the
+ *    same two pages that carry the brand-new requests also carry the old
+ *    pending ones that moved on. That is what makes per-id sync a backstop
+ *    rather than the only way those rows ever update.
+ *
+ * There is deliberately no cutoff: a page is 100 rows either way, the upsert
+ * is idempotent, and every date test in here has been a way to lose rows.
+ */
+async function walkRecent(log: FastifyBaseLogger): Promise<RefreshResult> {
+  const loggedBody = { done: false };
+  let pages = 0;
+  let seen = 0;
+  let stored = 0;
+  let legacyStored = 0;
+
+  for (let page = 1; page <= INCREMENTAL_PAGES; page += 1) {
+    if (page > 1) await sleep(PAGE_PAUSE_MS);
+    // No query at all — this is the whole point of the incremental pass.
+    const result = await fetchPageWithBackoff(page, '', log, loggedBody);
+    pages = page;
+    if (result.requests.length === 0) break;
+    seen += result.requests.length;
+
+    // Both title formats, because "recent activity" reaches back over the
+    // rename: a pre-rename request signed today belongs in the mirror too.
+    const wanted = result.requests.filter(
+      (request) => isNdaRequest(request) || isLegacyNdaRequest(request),
+    );
+    await upsertRequests(wanted);
+    stored += wanted.length;
+    legacyStored += wanted.filter((request) => !isNdaRequest(request)).length;
+
+    if (result.page >= result.numPages) break;
+  }
+
+  logWalk(log, {
+    mode: 'incremental',
+    query: 'unfiltered',
+    window: 'none (newest activity first, no cutoff)',
+    pages,
+    seen,
+    stored,
+  });
+  return {
+    mode: 'incremental',
+    pages,
+    stored,
+    seen,
+    legacy_stored: legacyStored,
+    // It read the top of the list, not a window: coverage is untouched.
+    covers_from: null,
+    truncated: false,
+  };
+}
+
+/**
+ * Mirrors every NDA request created since `windowAt` — the full reload.
  *
  * Two legs, because the Dropbox template was renamed on ~2026-06-29: the
  * current `Buyer Forms - NDA …` title, and — only when the window reaches
  * before the rename — the older `Buyer Forms -…` one, bounded to the period
  * where that looser prefix is still selective.
+ *
+ * This is the only mode that searches, and the only one that can widen the
+ * mirror's coverage: it is what "re-read that whole window" means.
  */
-async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResult> {
+async function walkWindow(windowAt: Date, log: FastifyBaseLogger): Promise<RefreshResult> {
+  // The bounds are inclusive calendar days, so the window is the whole cutoff
+  // day, from midnight.
+  const cutoffAt = startOfDay(windowAt);
   const cutoff = Math.floor(cutoffAt.getTime() / 1000);
+  const window = `[${day(cutoffAt)} TO *]`;
   const loggedBody = { done: false };
 
   const current = await walkLeg(
     'current format',
     ndaSearchQuery(cutoffAt),
+    window,
     cutoff,
-    (request) => Boolean(request.title?.startsWith(NDA_TITLE_PREFIX)),
+    isNdaRequest,
     log,
     loggedBody,
   );
@@ -257,6 +396,7 @@ async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResu
     ? await walkLeg(
         'pre-rename format',
         legacyNdaSearchQuery(cutoffAt),
+        `[${day(cutoffAt)} TO ${day(RENAME_DATE)}]`,
         cutoff,
         isLegacyNdaRequest,
         log,
@@ -282,15 +422,18 @@ async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResu
   }
   await widenCoverage(reached);
 
-  log.info(
+  logWalk(log, {
-    `[nda-refresh] walked ${pages} page(s) since ${cutoffAt.toISOString().slice(0, 10)}, ` +
+    mode: 'full',
-      `saw ${seen}, stored ${stored}` +
+    leg: 'total',
-      (legacy ? ` (${legacy.stored} pre-rename)` : '') +
+    query: `${legacy ? 2 : 1} leg(s)`,
-      (current.skipped + (legacy?.skipped ?? 0) > 0
+    window: `[${day(reached)} TO *]${complete ? '' : ' (truncated)'}`,
-        ? `, skipped ${current.skipped + (legacy?.skipped ?? 0)} non-matching`
+    pages,
-        : ''),
+    seen,
-  );
+    stored,
+    skipped: current.skipped + (legacy?.skipped ?? 0),
+  });
   return {
+    mode: 'full',
     pages,
     stored,
     seen,
@@ -301,14 +444,22 @@ async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResu
 }
 
 /**
- * Coverage only ever improves: an incremental walk reaching back two days does
+ * Coverage only ever **widens**: a walk that reached back two days does not
- * not un-mirror what a full reload fetched last week.
+ * un-mirror what a full reload fetched last week. So the stored value is the
+ * LEAST of what is there and what this walk reached — never the newer one.
+ *
+ * Done as one conditional upsert rather than read-then-write, so two runs
+ * cannot interleave and land on the narrower value. Both sides are ISO-8601
+ * UTC strings, which sort chronologically as text.
  */
 async function widenCoverage(reached: Date): Promise<void> {
-  const current = await getMeta(META_COVERS_FROM);
+  await query(
-  const currentAt = current ? new Date(current) : null;
+    `INSERT INTO app_meta (key, value) VALUES ($1, $2)
-  if (currentAt && !Number.isNaN(currentAt.getTime()) && currentAt <= reached) return;
+     ON CONFLICT (key) DO UPDATE
-  await setMeta(META_COVERS_FROM, reached.toISOString());
+       SET value = least(app_meta.value, EXCLUDED.value), updated_at = now()
+       WHERE app_meta.value IS NULL OR app_meta.value > EXCLUDED.value`,
+    [META_COVERS_FROM, reached.toISOString()],
+  );
 }
 
 /**
@@ -321,10 +472,17 @@ export async function startRefresh(
   explicitSince = false,
 ): Promise<boolean> {
   return startTask(REFRESH_KEYS, 'nda-refresh', log, async () => {
-    // Resolved inside the task: the cutoff depends on the previous refresh,
+    // Resolved inside the task: the mode depends on the previous refresh, and
-    // and the slot claim is what serialises two runs against each other.
+    // the slot claim is what serialises two runs against each other.
-    const cutoff = await refreshCutoff(since, explicitSince);
+    const plan = await refreshPlan(since, explicitSince);
-    return walk(cutoff, log);
+    const result =
+      plan.mode === 'incremental' ? await walkRecent(log) : await walkWindow(plan.window, log);
+    // Here and nowhere else. ds_last_refresh_at is not a "the job ran" stamp:
+    // it says a walk completed and stored what it found, and the sync reads it
+    // to decide which rows the walk can no longer reach. Anything that moves
+    // it without a completed walk moves that boundary over unmirrored data.
+    await setMeta(META_LAST_REFRESH, new Date().toISOString());
+    return result;
   });
 }
 

web/src/api.ts

@@ -285,6 +285,12 @@ export interface InboxRow {
 export type RefreshState = 'idle' | 'running' | string;
 
 export interface RefreshResult {
+  /**
+   * `incremental` reads the newest pages of the plain list, `full` re-reads a
+   * whole window through the search index — only after a moved date picker or
+   * on the very first run.
+   */
+  mode: 'incremental' | 'full';
   pages: number;
   seen: number;
   stored: number;

web/src/views/NdaInbox.tsx

@@ -105,9 +105,10 @@ export default function NdaInbox({
   useEffect(() => {
     if (!refreshing && watchingRefresh.current && inbox?.last_refresh_result) {
       watchingRefresh.current = false;
-      const { pages, seen, stored, legacy_stored: legacy } = inbox.last_refresh_result;
+      const { mode, pages, seen, stored, legacy_stored: legacy } = inbox.last_refresh_result;
       setRefreshNotice(
-        `Refreshed: ${stored} request${stored === 1 ? '' : 's'} stored ` +
+        `Refreshed (${mode === 'full' ? 'full window' : 'newest first'}): ` +
+          `${stored} request${stored === 1 ? '' : 's'} stored ` +
           `from ${seen} seen over ${pages} page${pages === 1 ? '' : 's'}` +
           (legacy > 0 ? ` · ${legacy} in the pre-rename title format` : '') +
           '.',