aknuth/bizmatch-app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix nda

Browse Source
This commit is contained in:
Andreas Knuth
2026-08-04 10:26:54 -05:00
parent 96a37a495e
commit da3f8434a1
11 changed files with 926 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.claude/settings.local.json
View File

@@ -66,7 +66,10 @@
"Bash(bash /tmp/claude-1000/-home-aknuth-git-bizmatch-app/0d38fdce-4b18-4159-86e7-135dadf9386d/scratchpad/acceptance6c.sh)",
"Bash(bash /tmp/claude-1000/-home-aknuth-git-bizmatch-app/0d38fdce-4b18-4159-86e7-135dadf9386d/scratchpad/acceptance6d.sh)",
"Bash(bash /tmp/claude-1000/-home-aknuth-git-bizmatch-app/0d38fdce-4b18-4159-86e7-135dadf9386d/scratchpad/acceptance6e.sh)",
"Bash(awk '{print \" size:\", $5, \"bytes modified:\", $6, $7, $8}')"
"Bash(awk '{print \" size:\", $5, \"bytes modified:\", $6, $7, $8}')",
"Bash(curl -s -m 5 http://localhost:8090/api/health)",
"Bash(curl -s -i -X POST -H 'cookie: bizmatch_staff=__CMDSUB_OUTPUT__' http://localhost:8090/api/nda-inbox/refresh)",
"Bash(grep -n \"The task \\\\*\\\\*searches server-side\\\\*\\\\*\\\\|^It pages through that filtered set\\\\|^Measured against the live account\\\\|^\\\\`ds_last_refresh_at\\\\` and\\\\|^Import and sync are unchanged\" README.md)"
]
}
}

114
README.md
View File

@@ -116,6 +116,25 @@ silently misreport when it was last refreshed.
Anything that writes files (NDA filing) must point `NDA_ROOT` at a scratch
directory for the run; never let a test write into the real NAS tree.
### The Dropbox Sign stub
`scripts/ds-stub.mjs` serves the two endpoints the inbox uses and reproduces
the property that broke the refresh: **its search index lags its list.** A
fixture entry is `{request, indexed}` — the truth the plain list and the per-id
endpoint hand out, and the possibly stale (or entirely missing) copy that
`query=` searches. It parses the range syntax for real, so `[a TO b]` and
`{a TO b}` genuinely differ. Point `DROPBOX_SIGN_BASE_URL` at it.
`node scripts/acceptance-nda-refresh.mjs` is the acceptance run: it starts the
real server against the stub and the real development database and asserts that
an incremental refresh picks up a request the index does not have yet and an
old pending one signed since, that a full reload includes the boundary day,
that a sync leaves `ds_last_refresh_at` untouched, and that `covers_from` never
narrows. It snapshots every `ds_*` key in `app_meta` and writes it back
afterwards, deletes its `req-*` rows, and runs with
`SYNC_PENDING_MAX_AGE_DAYS=0` so the sync cannot start re-fetching the real
mirror row by row.
## NAS mount
Mount it on the host via NFS, e.g. in `/etc/fstab`:
@@ -348,9 +367,13 @@ are mirrored into `ds_request` by a background task, and
* `covers_from` is how far back the mirror actually reaches. A walk stopped by
the `MAX_PAGES` cap covers less than it was asked for, so it records the
oldest date it got to and the inbox says "showing data from …" instead of
presenting a short list as complete. Coverage only ever widens: an
incremental walk reaching back two days does not un-mirror what a full reload
fetched last week.
presenting a short list as complete. **Coverage only ever widens**: the value
is written as `LEAST(stored, reached)` in one conditional upsert, so no walk
and no race can make the mirror claim *less* than it holds. It had drifted to
a date two months later than the mirror's own oldest row;
`008_covers_from_widen.sql` corrects the stored value once from
`min(created_at)` of `ds_request`. Only a full reload writes it at all — an
incremental pass reads the newest pages and establishes no window.
* `POST /api/nda-inbox/refresh` starts the walk and returns `202` immediately,
or `409` when one is already running — the slot is claimed with a conditional
upsert on `app_meta`, so two clicks cannot start two walks. A `running` state
@@ -361,37 +384,84 @@ are mirrored into `ds_request` by a background task, and
refresh feel slow in production — the button then said Refresh but did a full
reload every time. The button reflects this: *Refresh* vs *Reload window*.
The task **searches server-side** rather than paging through everything and
discarding most of it. It sends
#### Two walks, because the endpoint has two behaviours
| mode | when | what it reads |
| --- | --- | --- |
| `incremental` | every plain Refresh | pages 1-2 of the **unfiltered** list, no query, no cutoff |
| `full` | first run, or an explicit `?since=` | the two title queries over the whole window |
**The search index lags the list.** A request created two minutes ago is on
page 1 of the plain list and in *no* filtered result at all. An incremental
walk that searched therefore mirrored nothing while reporting a clean run —
which is how the mirror sat four days behind while every refresh looked
successful. So the incremental pass does not search. It reads the first two
pages as they come (200 requests, two calls) and keeps whatever passes the
title checks — **both** formats, since recent activity reaches back over the
rename.
That works because **the list is ordered by last activity, not by creation**:
verified live, a request created 30 July and signed 3 August sits above ones
created 3 August. So the same two pages carry the brand-new requests *and* the
old pending ones that were signed since — one pass, no per-id fetching. That is
what makes the sync's per-id re-check a backstop rather than the only way an
old row ever updates.
A **full reload searches server-side** rather than paging through everything
and discarding most of it, which turns a 13k-request account into the ~360 that
are ours:
```
query=title:"Buyer Forms - NDA" AND created:{<cutoff-date> TO *}
query=title:"Buyer Forms - NDA" AND created:[<cutoff-date> TO *]
```
which turns a 13k-request account into the ~360 that are ours. The two
client-side filters are kept as safety nets: a title that should not have come
back is filtered out *and logged as a warning*, and the exact-timestamp check
catches rows from the cutoff day itself, since the API's date clause is only
day-granular. If the query were ever ignored, the mirror would still be correct
— just slow again, and the log would say so.
The bounds are **inclusive** (`[…]`, not `{…}`): the exclusive form dropped
every request created on the cutoff day itself. Index lag is irrelevant here —
the window reaches months past it. The two client-side filters are kept as
safety nets: a title that should not have come back is filtered out *and logged
as a warning*, and the timestamp check drops anything older than midnight of
the cutoff day, since the API's date clause is only day-granular.
It pages through that filtered set with a 500 ms pause between calls; re-walking
is how a row that was pending last time is picked up as signed or declined.
Both walks pause 500 ms between calls. On `429`/`409` they honour `Retry-After`
but wait at least 10 s, retry a page up to three times, and log the response
body once per run at warn level — we still do not know what Dropbox means by
the `409` it sometimes sends.
Measured against the live account:
| | pages | seen | stored | duration |
| --- | --- | --- | --- | --- |
| full 90-day reload | 4 | 359 | 359 | 33 s |
| incremental, seconds later | 1 | 43 | 43 | 5 s |
| incremental, six days behind | 2 | 200 | 187 | 3 s |
| incremental, up to date | 2 | 200 | ~15 new | 3 s |
`seen` and `stored` now match. The old gap (≈1000 seen, ≈360 stored, 10 pages,
a minute) *was* the problem: 97% of what it fetched was thrown away. On `429`/`409` it honours
`Retry-After` but waits at least 10s, retries a page up to three times, and
logs the response body once per run at warn level — we still do not know what
Dropbox means by the `409` it sometimes sends. `app_meta` holds
`ds_last_refresh_at` and `ds_refresh_state` (`idle` | `running` |
`error:<msg>`).
`seen` is now the unfiltered page count, so it is always `100 × pages` — an
incremental pass re-storing rows it already has is free (the upsert is
idempotent) and cheaper than any attempt to be clever about it.
**Every walk logs one line**, which is what makes the next anomaly readable
without a debugger:
```
[nda-refresh] walk mode=incremental query=unfiltered window=none (newest activity first, no cutoff) pages=2 seen=200 stored=187
[nda-refresh] walk mode=full leg=current format query="title:\"Buyer Forms - NDA\" AND created:[2026-05-01 TO *]" window=[2026-05-01 TO *] pages=1 seen=2 stored=2
```
#### The two jobs' meta keys are disjoint, and nothing shares a write
`app_meta` holds `ds_refresh_state` (`idle` | `running` | `error:<msg>`),
`ds_last_refresh_at`, `ds_last_refresh_result` and `ds_mirror_covers_from` for
the refresh, and `ds_sync_state` / `ds_last_sync_at` / `ds_last_sync_result`
for the sync. They never cross: the sync *reads* `ds_last_refresh_at` to bound
its candidate set and must not write it.
`ds_last_refresh_at` is not a "the job ran" stamp — it is the boundary the sync
uses to decide which rows the walk can no longer reach, so it may only be
written by a **completed walk that stored what it found**. The shared
`startTask()` therefore no longer stamps `lastAt` for its caller: a helper that
writes "this ran at" on every resolved job is exactly how the marker moved
forward over data nobody had mirrored. Each job writes its own key at the point
where its work is done.
Import and sync are unchanged and still address one `signature_request_id` at a
time. Nothing re-fetches a request per id unnecessarily: sync's candidate set

24
migrations/008_covers_from_widen.sql Normal file
View File

@@ -0,0 +1,24 @@
-- BizMatch Phase 2 — module 6a fix: ds_mirror_covers_from may only widen
--
-- The value had drifted to a date *later* than the mirror's own oldest row,
-- so the inbox announced "showing data from <late date>" over a table that in
-- fact reached months further back. Coverage is now written as
-- LEAST(existing, reached) (see widenCoverage() in src/nda-refresh.ts); this
-- corrects the value that is already stored, once, from the only source that
-- cannot be wrong about it — the mirror itself.
--
-- Day-truncated: the mirror holds every request of that day, so midnight is
-- the honest bound and it is never narrower than the timestamp.
WITH mirror AS (
SELECT date_trunc('day', min(created_at) AT TIME ZONE 'UTC') AS from_at FROM ds_request
)
INSERT INTO app_meta (key, value)
SELECT 'ds_mirror_covers_from',
to_char(from_at, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')
FROM mirror
WHERE from_at IS NOT NULL
ON CONFLICT (key) DO UPDATE
SET value = least(app_meta.value, EXCLUDED.value), updated_at = now()
-- Only ever widens: a stored value that already reaches further back stands.
WHERE app_meta.value IS NULL OR app_meta.value > EXCLUDED.value;

414
scripts/acceptance-nda-refresh.mjs Normal file
View File

@@ -0,0 +1,414 @@
#!/usr/bin/env node
/**
* Acceptance for the NDA refresh: incremental mode, the meta-key hygiene of
* the two background jobs, and the coverage marker.
*
* It runs the real server against `scripts/ds-stub.mjs` (a Dropbox Sign whose
* search index deliberately lags its list) and the real development database,
* per the README convention:
*
* * every stubbed request id starts with `req-`, and they are deleted again;
* * the `ds_*` keys in `app_meta` are snapshotted before the run and written
* back afterwards, so a test run cannot leave the mirror looking refreshed
* when it is not.
*
* Nothing else in the database is touched: the stub only ever hands out `req-`
* requests, and the ids of the real mirror are unknown to it.
*
* node scripts/acceptance-nda-refresh.mjs
*/
import { spawn } from 'node:child_process';
import { mkdtemp, readFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import pg from 'pg';
import { createStub } from './ds-stub.mjs';
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
const APP_PORT = 8099;
const NDA_PREFIX = 'Buyer Forms - NDA';
/** Before RENAME_DATE, so the full reload runs its pre-rename leg too. */
const BOUNDARY_DAY = '2026-05-01';
const LEGACY_DAY = '2026-05-02';
const META_KEYS = [
'ds_refresh_state',
'ds_last_refresh_at',
'ds_last_refresh_result',
'ds_mirror_covers_from',
'ds_sync_state',
'ds_last_sync_at',
'ds_last_sync_result',
];
// --------------------------------------------------------------- helpers
let failures = 0;
function check(name, ok, detail = '') {
console.log(`${ok ? ' ok ' : ' FAIL '} ${name}${detail ? ` — ${detail}` : ''}`);
if (!ok) failures += 1;
}
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
/** `npm run dev` sources .env by hand, so the script does the same. */
async function loadEnv() {
const text = await readFile(path.join(ROOT, '.env'), 'utf8').catch(() => '');
for (const line of text.split('\n')) {
const match = /^\s*([A-Z0-9_]+)\s*=\s*(.*)$/.exec(line);
if (!match) continue;
const value = match[2].trim().replace(/^["'](.*)["']$/, '$1');
if (process.env[match[1]] === undefined) process.env[match[1]] = value;
}
}
const seconds = (date) => Math.floor(date.getTime() / 1000);
const daysAgo = (n) => seconds(new Date(Date.now() - n * 86_400_000));
function request(id, title, createdAt, { signedAt = null, name, email } = {}) {
return {
signature_request_id: id,
title,
created_at: createdAt,
is_complete: signedAt !== null,
is_declined: false,
files_url: '',
signatures: [
{
signer_name: name,
signer_email_address: email,
status_code: signedAt ? 'signed' : 'awaiting_signature',
signed_at: signedAt,
},
],
response_data: [],
};
}
/**
* The fixture, in the order the assertions reason about:
*
* | # | request | why |
* | 0 | created now, **not in the index** | the lag: only the plain list has it |
* | 1 | created 40 days ago, signed yesterday, **stale in the index** | activity order puts it on page 1 |
* | 2.. | 200 unrelated requests | fill pages 1-2, so anything below is out of the incremental pass' reach |
* | 202 | created on the boundary day | only an inclusive `[day TO *]` returns it |
* | 203 | pre-rename title | exercises the second leg of the full reload |
*/
function buildFixture(stamp) {
const fresh = request(`req-${stamp}-fresh`, `${NDA_PREFIX} Nora Fresh - Acme Diner`, daysAgo(0), {
name: 'Nora Fresh',
email: `nora.fresh@stubtest.invalid`,
});
const oldSigned = request(
`req-${stamp}-old-signed`,
`${NDA_PREFIX} Otto Older - Old Deli`,
daysAgo(40),
{ signedAt: daysAgo(1), name: 'Otto Older', email: 'otto.older@stubtest.invalid' },
);
const boundary = request(
`req-${stamp}-boundary`,
`${NDA_PREFIX} Bea Boundary - Border Cafe`,
seconds(new Date(`${BOUNDARY_DAY}T09:00:00Z`)),
{ name: 'Bea Boundary', email: 'bea.boundary@stubtest.invalid' },
);
const legacy = request(
`req-${stamp}-legacy`,
'Buyer Forms Regina Legacy - Old Bakery',
seconds(new Date(`${LEGACY_DAY}T11:00:00Z`)),
{ name: 'Regina Legacy', email: 'regina.legacy@stubtest.invalid' },
);
// Everything the account is full of and the refresh is not interested in.
const noise = Array.from({ length: 200 }, (_, i) =>
request(`req-${stamp}-noise-${i}`, `Other Form ${i} - Someone`, daysAgo(2 + i * 0.05), {
name: `Noise ${i}`,
email: `noise${i}@stubtest.invalid`,
}),
);
/** The index copy of the freshly signed one: still pending, as it was. */
const staleIndexCopy = request(
oldSigned.signature_request_id,
oldSigned.title,
oldSigned.created_at,
{ name: 'Otto Older', email: 'otto.older@stubtest.invalid' },
);
return {
ids: {
fresh: fresh.signature_request_id,
oldSigned: oldSigned.signature_request_id,
boundary: boundary.signature_request_id,
legacy: legacy.signature_request_id,
},
fixture: [
{ request: fresh, indexed: null }, // the lag: not searchable yet at all
{ request: oldSigned, indexed: staleIndexCopy }, // the lag: stale copy
...noise.map((r) => ({ request: r, indexed: r })),
{ request: boundary, indexed: boundary },
{ request: legacy, indexed: legacy },
],
};
}
// ------------------------------------------------------------------ main
await loadEnv();
if (!process.env.DATABASE_URL) {
console.error('DATABASE_URL is missing — run this from the project root with .env in place');
process.exit(2);
}
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const stamp = String(Date.now());
const { ids, fixture } = buildFixture(stamp);
const stub = createStub(fixture);
let app = null;
let scratch = null;
let metaSnapshot = [];
const appLog = [];
const cleanup = async () => {
if (app && !app.killed) {
app.kill('SIGTERM');
await sleep(300);
}
await stub.close().catch(() => {});
await db
.query(`DELETE FROM ds_request WHERE signature_request_id LIKE 'req-%'`)
.catch((err) => console.error(`cleanup of ds_request failed: ${err.message}`));
// The meta keys go back exactly as they were, updated_at included: a test
// run must not leave "last refreshed" pointing at itself.
await db
.query('DELETE FROM app_meta WHERE key = ANY($1)', [META_KEYS])
.catch((err) => console.error(`cleanup of app_meta failed: ${err.message}`));
for (const row of metaSnapshot) {
await db
.query('INSERT INTO app_meta (key, value, updated_at) VALUES ($1, $2, $3)', [
row.key,
row.value,
row.updated_at,
])
.catch((err) => console.error(`restoring ${row.key} failed: ${err.message}`));
}
if (scratch) await rm(scratch, { recursive: true, force: true }).catch(() => {});
await db.end().catch(() => {});
};
process.on('SIGINT', async () => {
await cleanup();
process.exit(130);
});
const meta = async (key) => {
const { rows } = await db.query('SELECT value, updated_at FROM app_meta WHERE key = $1', [key]);
return rows[0] ?? null;
};
const setMeta = (key, value) =>
db.query(
`INSERT INTO app_meta (key, value) VALUES ($1, $2)
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value, updated_at = now()`,
[key, value],
);
const mirrored = async (id) => {
const { rows } = await db.query(
'SELECT status, signed_at, created_at FROM ds_request WHERE signature_request_id = $1',
[id],
);
return rows[0] ?? null;
};
try {
metaSnapshot = (
await db.query('SELECT key, value, updated_at FROM app_meta WHERE key = ANY($1)', [META_KEYS])
).rows;
await db.query(`DELETE FROM ds_request WHERE signature_request_id LIKE 'req-%'`);
const stubUrl = await stub.listen(0);
scratch = await mkdtemp(path.join(tmpdir(), 'bizmatch-nda-'));
// ------------------------------------------------------------ the app
app = spawn('npx', ['tsx', 'src/server.ts'], {
cwd: ROOT,
env: {
...process.env,
PORT: String(APP_PORT),
DROPBOX_SIGN_API_KEY: 'stub-key',
DROPBOX_SIGN_BASE_URL: stubUrl,
NDA_ROOT: scratch,
// Keeps the sync's per-id re-check off the real mirror: this run is
// about which meta keys it writes, not about re-reading 800 rows.
SYNC_PENDING_MAX_AGE_DAYS: '0',
},
stdio: ['ignore', 'pipe', 'pipe'],
});
for (const stream of [app.stdout, app.stderr]) {
stream.setEncoding('utf8');
stream.on('data', (chunk) => {
for (const line of chunk.split('\n').filter(Boolean)) appLog.push(line);
});
}
const base = `http://127.0.0.1:${APP_PORT}`;
const { rows: staff } = await db.query('SELECT id FROM staff ORDER BY created_at LIMIT 1');
if (!staff[0]) throw new Error('no staff row to sign in as');
const headers = { cookie: `bizmatch_staff=${staff[0].id}` };
for (let attempt = 0; ; attempt += 1) {
if (attempt > 100) throw new Error('the app did not come up');
const ok = await fetch(`${base}/api/health`)
.then((r) => r.ok)
.catch(() => false);
if (ok) break;
await sleep(300);
}
const inbox = async () =>
(await fetch(`${base}/api/nda-inbox?since=2026-01-01`, { headers })).json();
const settle = async (field) => {
for (let attempt = 0; attempt < 300; attempt += 1) {
const body = await inbox();
if (body[field] !== 'running') return body;
await sleep(500);
}
throw new Error(`${field} never settled`);
};
// ---------------------------------------- 1 · the incremental pass
console.log('\n1 · incremental refresh over a lagging search index');
await setMeta('ds_last_refresh_at', new Date(Date.now() - 3_600_000).toISOString());
await setMeta('ds_refresh_state', 'idle');
// Deliberately narrow, so the coverage assertions have something to widen.
await setMeta('ds_mirror_covers_from', '2026-12-31T00:00:00.000Z');
const searchable = stub.search(`title:"${NDA_PREFIX}" AND created:[${BOUNDARY_DAY} TO *]`);
check(
'the stub search index really lags the list',
!searchable.some((r) => r.signature_request_id === ids.fresh) &&
searchable.some((r) => r.signature_request_id === ids.boundary),
'the fresh request is in no search result, the older ones are',
);
const started = await fetch(`${base}/api/nda-inbox/refresh`, { method: 'POST', headers });
check('POST /refresh answers 202', started.status === 202, `got ${started.status}`);
const afterIncremental = await settle('refresh_state');
const incremental = afterIncremental.last_refresh_result;
check('it ran in incremental mode', incremental?.mode === 'incremental', JSON.stringify(incremental));
check(
'it read the unfiltered list, not the index',
stub.calls.filter((c) => c.endpoint === 'list' && c.query).length === 0,
`queries sent: ${stub.calls.filter((c) => c.query).length}`,
);
check(
`it walked ${2} pages of 100 unfiltered rows`,
incremental?.pages === 2 && incremental?.seen === 200,
`pages=${incremental?.pages} seen=${incremental?.seen}`,
);
const fresh = await mirrored(ids.fresh);
check(
'a request created now lands although the index does not have it',
fresh?.status === 'pending',
`mirror row: ${JSON.stringify(fresh)}`,
);
const oldSigned = await mirrored(ids.oldSigned);
check(
'an old pending signed yesterday is picked up without a sync',
oldSigned?.status === 'signed' && oldSigned?.signed_at !== null,
`mirror row: ${JSON.stringify(oldSigned)}`,
);
check(
'and it stored nothing beyond those two',
incremental?.stored === 2,
`stored=${incremental?.stored}`,
);
check(
'rows below the first two pages are out of its reach',
(await mirrored(ids.boundary)) === null && (await mirrored(ids.legacy)) === null,
'boundary and pre-rename rows are not mirrored yet',
);
// ---------------------------------------- 2 · the full reload
console.log('\n2 · full reload with inclusive bounds');
const callsBefore = stub.calls.length;
const reload = await fetch(`${base}/api/nda-inbox/refresh?since=${BOUNDARY_DAY}`, {
method: 'POST',
headers,
});
check('POST /refresh?since= answers 202', reload.status === 202, `got ${reload.status}`);
const afterFull = await settle('refresh_state');
const full = afterFull.last_refresh_result;
const queries = stub.calls.slice(callsBefore).filter((c) => c.query).map((c) => c.query);
check('it ran in full mode', full?.mode === 'full', JSON.stringify(full));
check(
'every query uses the inclusive range form',
queries.length > 0 && queries.every((q) => q.includes('created:[')) &&
!queries.some((q) => q.includes('created:{')),
queries[0] ?? 'no query sent',
);
check(
'a request created on the boundary day is included',
(await mirrored(ids.boundary)) !== null,
`${BOUNDARY_DAY}T09:00Z with since=${BOUNDARY_DAY}`,
);
check(
'the pre-rename leg ran and stored its request',
(await mirrored(ids.legacy)) !== null && full?.legacy_stored >= 1,
`legacy_stored=${full?.legacy_stored}`,
);
check(
'covers_from widened to the reloaded window',
(await meta('ds_mirror_covers_from'))?.value === `${BOUNDARY_DAY}T00:00:00.000Z`,
`covers_from=${(await meta('ds_mirror_covers_from'))?.value}`,
);
// ---------------------------------------- 3 · coverage never narrows
console.log('\n3 · covers_from only ever widens');
await setMeta('ds_mirror_covers_from', '2020-01-01T00:00:00.000Z');
await fetch(`${base}/api/nda-inbox/refresh?since=${BOUNDARY_DAY}`, { method: 'POST', headers });
await settle('refresh_state');
check(
'a narrower window does not overwrite a wider coverage',
(await meta('ds_mirror_covers_from'))?.value === '2020-01-01T00:00:00.000Z',
`covers_from=${(await meta('ds_mirror_covers_from'))?.value}`,
);
// ---------------------------------------- 4 · meta-key hygiene
console.log('\n4 · the sync never touches the refresh cursor');
const refreshBefore = await meta('ds_last_refresh_at');
const synced = await fetch(`${base}/api/nda-inbox/sync`, { method: 'POST', headers });
check('POST /sync answers 202', synced.status === 202, `got ${synced.status}`);
await settle('sync_state');
const refreshAfter = await meta('ds_last_refresh_at');
check(
'ds_last_refresh_at is untouched by a sync run',
refreshBefore?.value === refreshAfter?.value &&
String(refreshBefore?.updated_at) === String(refreshAfter?.updated_at),
`${refreshBefore?.value} @ ${refreshBefore?.updated_at} -> ${refreshAfter?.value} @ ${refreshAfter?.updated_at}`,
);
check(
'the sync stamps its own key instead',
(await meta('ds_last_sync_at'))?.value != null,
`ds_last_sync_at=${(await meta('ds_last_sync_at'))?.value}`,
);
// ---------------------------------------- the log line
console.log('\nwalk log lines the run produced:');
for (const line of appLog) {
let msg = line;
try {
msg = JSON.parse(line).msg ?? line;
} catch {
/* not JSON — a tsx or npm line */
}
if (typeof msg === 'string' && msg.includes('[nda-refresh] walk')) console.log(` ${msg}`);
}
} catch (err) {
failures += 1;
console.error(`\nrun failed: ${err.stack ?? err.message}`);
} finally {
await cleanup();
}
console.log(failures === 0 ? '\nall assertions passed' : `\n${failures} assertion(s) failed`);
process.exit(failures === 0 ? 0 : 1);

139
scripts/ds-stub.mjs Normal file
View File

@@ -0,0 +1,139 @@
/**
* A stand-in for the two Dropbox Sign endpoints the NDA inbox uses, with the
* one property that broke the refresh in production: **the search index lags
* behind the list**.
*
* `GET /v3/signature_request/list` answers from two different sources:
*
* * without `query=` — the live list, ordered by last activity (signed_at,
* falling back to created_at), exactly like the real endpoint;
* * with `query=` — the *index*, which only holds the snapshot a fixture entry
* declares in `indexed`. A brand-new request is missing from it entirely,
* and a request signed a minute ago is still in it as pending. That is what
* an incremental refresh must not depend on.
*
* The range syntax is parsed for real, brackets and all, so `[a TO b]`
* (inclusive) and `{a TO b}` (exclusive) actually behave differently — which
* is what makes the boundary-day assertion meaningful.
*
* Run standalone for manual pokes:
* node scripts/ds-stub.mjs --port 8099 --fixture some.json
*/
import { createServer } from 'node:http';
import { readFile } from 'node:fs/promises';
const dayOf = (unixSeconds) => new Date(unixSeconds * 1000).toISOString().slice(0, 10);
/** signed_at when there is one, created_at otherwise — the list's sort key. */
const activityOf = (request) => request.signatures?.[0]?.signed_at ?? request.created_at;
/**
* The subset of the query language the refresh sends:
* `title:"<phrase>" AND created:[<day> TO <day|*>]`, either bracket style.
*/
export function matchesQuery(request, queryString) {
const phrase = /title:"([^"]*)"/.exec(queryString)?.[1];
if (phrase && !(request.title ?? '').includes(phrase)) return false;
const range = /created:([[{])\s*(\S+)\s+TO\s+(\S+?)\s*([\]}])/.exec(queryString);
if (range) {
const [, open, from, to, close] = range;
const day = dayOf(request.created_at);
// '[' includes the boundary day, '{' excludes it — the whole point.
if (from !== '*' && (open === '[' ? day < from : day <= from)) return false;
if (to !== '*' && (close === ']' ? day > to : day >= to)) return false;
}
return true;
}
/**
* `fixture` is a list of `{request, indexed}`: the truth the list and the
* per-id endpoint serve, and the possibly stale (or missing) copy the search
* index has of it.
*/
export function createStub(fixture) {
/** Every call the app made — the test asserts on what it asked for. */
const calls = [];
const byActivity = (a, b) => activityOf(b) - activityOf(a);
const server = createServer((req, res) => {
const url = new URL(req.url, 'http://stub');
const send = (status, body) => {
const payload = JSON.stringify(body);
res.writeHead(status, { 'content-type': 'application/json' });
res.end(payload);
};
if (url.pathname === '/v3/signature_request/list') {
const page = Number(url.searchParams.get('page') ?? '1');
const pageSize = Number(url.searchParams.get('page_size') ?? '20');
const queryString = url.searchParams.get('query');
calls.push({ endpoint: 'list', page, query: queryString });
const source = queryString
? fixture
.map((entry) => entry.indexed)
.filter(Boolean)
.filter((request) => matchesQuery(request, queryString))
: fixture.map((entry) => entry.request);
const ordered = [...source].sort(byActivity);
const numPages = Math.max(1, Math.ceil(ordered.length / pageSize));
const slice = ordered.slice((page - 1) * pageSize, page * pageSize);
return send(200, {
signature_requests: slice,
list_info: { page, num_pages: numPages, num_results: ordered.length },
});
}
const single = /^\/v3\/signature_request\/([^/]+)$/.exec(url.pathname);
if (single) {
const id = decodeURIComponent(single[1]);
calls.push({ endpoint: 'get', id });
const entry = fixture.find((e) => e.request.signature_request_id === id);
// Unknown ids 404 like the real API — the sync walks over real rows the
// stub knows nothing about and must simply report them as failures.
if (!entry) return send(404, { error: { error_msg: `no such request: ${id}` } });
return send(200, { signature_request: entry.request });
}
return send(404, { error: { error_msg: `unhandled path: ${url.pathname}` } });
});
return {
calls,
/** Resolves to the base URL once it is listening. */
listen(port = 0) {
return new Promise((resolve) => {
server.listen(port, '127.0.0.1', () =>
resolve(`http://127.0.0.1:${server.address().port}`),
);
});
},
close: () => new Promise((resolve) => server.close(resolve)),
/** What a searching walk would have found — the lag, made visible. */
search: (queryString) =>
fixture
.map((entry) => entry.indexed)
.filter(Boolean)
.filter((request) => matchesQuery(request, queryString)),
};
}
// -------------------------------------------------------- standalone mode
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].replace(/^.*\//, ''))) {
const args = process.argv.slice(2);
const arg = (name, fallback) => {
const at = args.indexOf(`--${name}`);
return at === -1 ? fallback : args[at + 1];
};
const file = arg('fixture');
if (file) {
const fixture = JSON.parse(await readFile(file, 'utf8'));
const stub = createStub(fixture);
console.log(`[ds-stub] listening on ${await stub.listen(Number(arg('port', '8099')))}`);
} else {
console.error('usage: node scripts/ds-stub.mjs --fixture <file.json> [--port 8099]');
process.exit(2);
}
}

16
src/background-task.ts
View File

@@ -14,7 +14,13 @@ import { query, queryOne } from './db.js';
export interface TaskKeys {
/** idle | running | error:<message> */
state: string;
/** ISO timestamp of the last completed run. */
/**
* ISO timestamp of the last completed run — written by the **job**, never
* by this module. Each job owns its own key and the two sets are disjoint
* (`ds_*_refresh_*` vs `ds_*_sync_*`), because `ds_last_refresh_at` is not
* merely a display value: it is the boundary the sync uses to decide what
* the refresh can no longer reach. See startTask().
*/
lastAt: string;
/** JSON result of the last completed run. */
lastResult: string;
@@ -71,6 +77,13 @@ export async function clearStaleTask(
/**
* Claims the slot and runs `job` detached. `false` means somebody else is
* already running it — the caller answers 409.
*
* It deliberately does **not** stamp `keys.lastAt`. A shared helper writing
* "this ran at" on every resolved job is how `ds_last_refresh_at` came to move
* forward on runs that had walked nothing: the refresh cursor then pointed
* past requests that were never mirrored, and nothing downstream could tell
* the difference. Each job writes its own `lastAt` at the point where it knows
* its work is done and stored.
*/
export async function startTask<T>(
keys: TaskKeys,
@@ -84,7 +97,6 @@ export async function startTask<T>(
try {
const result = await job();
await setMeta(keys.lastResult, JSON.stringify(result));
await setMeta(keys.lastAt, new Date().toISOString());
await setMeta(keys.state, 'idle');
} catch (err) {
const message = (err as Error).message;

29
src/dropbox-sign.ts
View File

@@ -105,17 +105,23 @@ export interface SignatureRequestPage {
}
/**
* The search the refresh runs against the list endpoint. Filtering server-side
* is the difference between reading 13k requests and reading ~330: without it
* the walk pages through every signature request the account ever had and
* throws away the 97% that are not ours.
* The search a *full* reload runs against the list endpoint. Filtering
* server-side is the difference between reading 13k requests and reading ~330:
* without it the walk pages through every signature request the account ever
* had and throws away the 97% that are not ours.
*
* `created:{<date> TO *}` is the API's range syntax; the date is a plain
* calendar day, which is as precise as the filter goes.
* `created:[<date> TO *]` is the API's range syntax. The date is a plain
* calendar day — as precise as the filter goes — and the brackets are the
* **inclusive** form: `{…}` excluded the cutoff day itself, which silently
* dropped every request created on the boundary. The refresh window is a
* whole day either way, so inclusive is the honest bound.
*
* Only full reloads search: the index behind `query=` lags minutes behind the
* plain list, so an incremental pass cannot rely on it (see nda-refresh.ts).
*/
export function ndaSearchQuery(since: Date): string {
const day = since.toISOString().slice(0, 10);
return `title:"${NDA_TITLE_PREFIX}" AND created:{${day} TO *}`;
return `title:"${NDA_TITLE_PREFIX}" AND created:[${day} TO *]`;
}
/**
@@ -151,9 +157,16 @@ export const LEGACY_TITLE_QUERY = 'Buyer Forms';
export function legacyNdaSearchQuery(since: Date): string {
const from = since.toISOString().slice(0, 10);
const to = RENAME_DATE.toISOString().slice(0, 10);
return `title:"${LEGACY_TITLE_QUERY}" AND created:{${from} TO ${to}}`;
// Inclusive on both ends, like the current-format query: the upper bound is
// already a day past the rename, so including it only widens the harmless
// overlap, and including the lower one is what keeps boundary-day requests.
return `title:"${LEGACY_TITLE_QUERY}" AND created:[${from} TO ${to}]`;
}
/** Current-format NDA: the title prefix is the whole test. */
export const isNdaRequest = (request: SignatureRequest): boolean =>
Boolean(request.title?.startsWith(NDA_TITLE_PREFIX));
/**
* Safety net for the legacy leg: the old title, no "NDA" anywhere in it (that
* belongs to the other leg), and an actual signer. Anything else the loose

9
src/nda-inbox-routes.ts
View File

@@ -26,6 +26,7 @@ import {
clearStaleTask,
getMeta,
readTaskState,
setMeta,
startTask,
} from './background-task.js';
import { config } from './config.js';
@@ -45,6 +46,11 @@ const FETCH_PAUSE_MS = 500;
*/
const MIRROR_RECHECK_AFTER_HOURS = 12;
/**
* The sync's own keys — no overlap with the refresh's. It reads
* `ds_last_refresh_at` to bound its candidate set and must never write it: the
* refresh cursor belongs to the walk that stored the data.
*/
export const SYNC_KEYS: TaskKeys = {
state: 'ds_sync_state',
lastAt: 'ds_last_sync_at',
@@ -316,6 +322,9 @@ async function syncSignatures(
`${mirrored}/${stale.length} mirror row(s) re-read, ${mirrorChanged} moved on`,
);
// The job stamps its own "last completed" key — and only its own.
await setMeta(SYNC_KEYS.lastAt, new Date().toISOString());
// A run over a few hundred stale rows would otherwise answer with a few
// hundred warning strings; the count says how many were left out.
const MAX_WARNINGS = 10;

242
src/nda-refresh.ts
View File

@@ -1,10 +1,10 @@
import type { FastifyBaseLogger } from 'fastify';
import { withTransaction } from './db.js';
import { query, withTransaction } from './db.js';
import {
NDA_TITLE_PREFIX,
RENAME_DATE,
type SignatureRequest,
isLegacyNdaRequest,
isNdaRequest,
isRateLimited,
legacyNdaSearchQuery,
listPage,
@@ -21,6 +21,15 @@ import { type TaskKeys, clearStaleTask, getMeta, setMeta, startTask } from './ba
* them a window walk is 7-8 calls and well over a minute — far too slow to sit
* in front of a view mount, and enough to get throttled. So it runs detached,
* paced, and at most once at a time.
*
* Two modes, because the endpoint behaves differently with and without a
* `query=`:
*
* * **incremental** — the first pages of the *unfiltered* list. Cheap, and
* immune to the lag of the search index.
* * **full** — the two title queries over a whole window. Only for the first
* run and an explicit `?since=`, where the lag does not matter because the
* window reaches far past it.
*/
/** Between two list calls, so a full walk does not look like a hammer. */
@@ -31,6 +40,13 @@ const MAX_PAGE_ATTEMPTS = 3;
/** Safety net against a window that never reaches its cutoff. */
const MAX_PAGES = 50;
/**
* How many pages of the unfiltered list an incremental pass reads. Two pages
* are 200 requests of recent activity across the whole account — comfortably
* more than a day's worth, and two calls instead of a paged window walk.
*/
const INCREMENTAL_PAGES = 2;
export const REFRESH_KEYS: TaskKeys = {
state: 'ds_refresh_state',
lastAt: 'ds_last_refresh_at',
@@ -55,8 +71,20 @@ export const OVERLAP_MS = 48 * 60 * 60 * 1000;
export type RefreshState = 'idle' | 'running' | `error:${string}`;
/** See the module comment: one reads the plain list, the other searches. */
export type RefreshMode = 'incremental' | 'full';
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const day = (at: Date): string => at.toISOString().slice(0, 10);
/**
* The search bounds are calendar days, so the window really starts at midnight
* of the cutoff day — the exact-timestamp check has to agree with the query,
* or it throws away the boundary-day rows the inclusive bound just fetched.
*/
const startOfDay = (at: Date): Date => new Date(`${day(at)}T00:00:00.000Z`);
/**
* Mirrors a batch of requests. Exported because the sync route re-checks old
* pending rows one by one, which an incremental walk no longer reaches.
@@ -97,13 +125,19 @@ export async function upsertRequests(requests: SignatureRequest[]): Promise<void
}
export interface RefreshResult {
/** Which of the two walks ran — the log line says the same. */
mode: RefreshMode;
pages: number;
stored: number;
seen: number;
/** Of `stored`, how many came from the pre-rename title format. */
legacy_stored: number;
/** Earliest creation date this walk actually reached, ISO date. */
covers_from: string;
/**
* Earliest creation date this walk covers, ISO date. Null for an
* incremental pass: it reads the newest pages and therefore establishes no
* window at all — coverage is whatever the last full reload left behind.
*/
covers_from: string | null;
/** True when MAX_PAGES ended the walk before it reached the cutoff. */
truncated: boolean;
}
@@ -120,6 +154,34 @@ interface LegResult {
complete: boolean;
}
/**
* One line per walk, in one place, so the next anomaly can be read off the log
* without guessing what the run actually asked Dropbox for.
*/
function logWalk(
log: FastifyBaseLogger,
fields: {
mode: RefreshMode;
leg?: string;
/** The `query=` sent, or "unfiltered" when the pass sends none. */
query: string;
window: string;
pages: number;
seen: number;
stored: number;
skipped?: number;
},
): void {
log.info(
`[nda-refresh] walk mode=${fields.mode}` +
(fields.leg ? ` leg=${fields.leg}` : '') +
` query=${fields.query === 'unfiltered' ? 'unfiltered' : JSON.stringify(fields.query)}` +
` window=${fields.window} pages=${fields.pages} seen=${fields.seen}` +
` stored=${fields.stored}` +
(fields.skipped ? ` skipped=${fields.skipped}` : ''),
);
}
/** Fetches one page, backing off and retrying while Dropbox throttles us. */
async function fetchPageWithBackoff(
page: number,
@@ -151,21 +213,21 @@ async function fetchPageWithBackoff(
}
/**
* How far back this walk has to reach.
* Which walk the next run does, and over which window.
*
* A full window is only needed the first time. Afterwards everything older
* than the previous refresh (minus the overlap) is already mirrored, so the
* walk stops after a page or two instead of re-reading a thousand requests
* every run. An explicitly requested `since` always wins — that is how the
* caller asks for a deliberate re-read of a wider period.
* A full window is only needed the first time, and whenever the caller asks
* for one by moving the date picker — that is what an explicit `since` means.
* Everything else is the incremental pass, which needs no window at all.
*/
export async function refreshCutoff(since: Date, explicitSince: boolean): Promise<Date> {
if (explicitSince) return since;
export async function refreshPlan(
since: Date,
explicitSince: boolean,
): Promise<{ mode: RefreshMode; window: Date }> {
if (explicitSince) return { mode: 'full', window: since };
// Nothing mirrored yet: there is no "new since" to read.
const last = await getMeta(META_LAST_REFRESH);
if (!last) return since; // first run: the whole window
const incremental = new Date(new Date(last).getTime() - OVERLAP_MS);
if (Number.isNaN(incremental.getTime())) return since;
return incremental > since ? incremental : since;
if (!last || Number.isNaN(new Date(last).getTime())) return { mode: 'full', window: since };
return { mode: 'incremental', window: since };
}
/**
@@ -177,7 +239,8 @@ export async function refreshCutoff(since: Date, explicitSince: boolean): Promis
*/
async function walkLeg(
label: string,
query: string,
search: string,
window: string,
cutoff: number,
accept: (request: SignatureRequest) => boolean,
log: FastifyBaseLogger,
@@ -192,7 +255,7 @@ async function walkLeg(
for (; page <= MAX_PAGES; page += 1) {
if (page > 1) await sleep(PAGE_PAUSE_MS);
const result = await fetchPageWithBackoff(page, query, log, loggedBody);
const result = await fetchPageWithBackoff(page, search, log, loggedBody);
if (result.requests.length === 0) {
complete = true;
break;
@@ -212,8 +275,9 @@ async function walkLeg(
skipped += rejected;
}
// The date clause is day-granular, so a row from the cutoff day itself can
// come back slightly too old.
// The date clause is day-granular and inclusive, so the window starts at
// midnight of the cutoff day; anything older than that the query should
// not have returned at all.
const inWindow = wanted.filter((request) => request.created_at >= cutoff);
await upsertRequests(inWindow);
stored += inWindow.length;
@@ -228,26 +292,101 @@ async function walkLeg(
}
}
return { pages: Math.min(page, MAX_PAGES), seen, stored, skipped, oldest, complete };
const pages = Math.min(page, MAX_PAGES);
logWalk(log, { mode: 'full', leg: label, query: search, window, pages, seen, stored, skipped });
return { pages, seen, stored, skipped, oldest, complete };
}
/**
* Mirrors every NDA request created since `cutoffAt`.
* The incremental pass: the first pages of the **unfiltered** list.
*
* Two properties of the endpoint make this the right shape:
*
* 1. The index behind `query=` lags the list itself. A request created two
* minutes ago is on page 1 of the plain list and in *no* filtered result,
* so a searching incremental walk silently mirrors nothing while reporting
* a clean run — which is exactly how the mirror fell four days behind.
* 2. The list is ordered by **last activity**, not by creation. A request from
* last month that was signed yesterday sits above one created today, so the
* same two pages that carry the brand-new requests also carry the old
* pending ones that moved on. That is what makes per-id sync a backstop
* rather than the only way those rows ever update.
*
* There is deliberately no cutoff: a page is 100 rows either way, the upsert
* is idempotent, and every date test in here has been a way to lose rows.
*/
async function walkRecent(log: FastifyBaseLogger): Promise<RefreshResult> {
const loggedBody = { done: false };
let pages = 0;
let seen = 0;
let stored = 0;
let legacyStored = 0;
for (let page = 1; page <= INCREMENTAL_PAGES; page += 1) {
if (page > 1) await sleep(PAGE_PAUSE_MS);
// No query at all — this is the whole point of the incremental pass.
const result = await fetchPageWithBackoff(page, '', log, loggedBody);
pages = page;
if (result.requests.length === 0) break;
seen += result.requests.length;
// Both title formats, because "recent activity" reaches back over the
// rename: a pre-rename request signed today belongs in the mirror too.
const wanted = result.requests.filter(
(request) => isNdaRequest(request) || isLegacyNdaRequest(request),
);
await upsertRequests(wanted);
stored += wanted.length;
legacyStored += wanted.filter((request) => !isNdaRequest(request)).length;
if (result.page >= result.numPages) break;
}
logWalk(log, {
mode: 'incremental',
query: 'unfiltered',
window: 'none (newest activity first, no cutoff)',
pages,
seen,
stored,
});
return {
mode: 'incremental',
pages,
stored,
seen,
legacy_stored: legacyStored,
// It read the top of the list, not a window: coverage is untouched.
covers_from: null,
truncated: false,
};
}
/**
* Mirrors every NDA request created since `windowAt` — the full reload.
*
* Two legs, because the Dropbox template was renamed on ~2026-06-29: the
* current `Buyer Forms - NDA …` title, and — only when the window reaches
* before the rename — the older `Buyer Forms -…` one, bounded to the period
* where that looser prefix is still selective.
*
* This is the only mode that searches, and the only one that can widen the
* mirror's coverage: it is what "re-read that whole window" means.
*/
async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResult> {
async function walkWindow(windowAt: Date, log: FastifyBaseLogger): Promise<RefreshResult> {
// The bounds are inclusive calendar days, so the window is the whole cutoff
// day, from midnight.
const cutoffAt = startOfDay(windowAt);
const cutoff = Math.floor(cutoffAt.getTime() / 1000);
const window = `[${day(cutoffAt)} TO *]`;
const loggedBody = { done: false };
const current = await walkLeg(
'current format',
ndaSearchQuery(cutoffAt),
window,
cutoff,
(request) => Boolean(request.title?.startsWith(NDA_TITLE_PREFIX)),
isNdaRequest,
log,
loggedBody,
);
@@ -257,6 +396,7 @@ async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResu
? await walkLeg(
'pre-rename format',
legacyNdaSearchQuery(cutoffAt),
`[${day(cutoffAt)} TO ${day(RENAME_DATE)}]`,
cutoff,
isLegacyNdaRequest,
log,
@@ -282,15 +422,18 @@ async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResu
}
await widenCoverage(reached);
log.info(
`[nda-refresh] walked ${pages} page(s) since ${cutoffAt.toISOString().slice(0, 10)}, ` +
`saw ${seen}, stored ${stored}` +
(legacy ? ` (${legacy.stored} pre-rename)` : '') +
(current.skipped + (legacy?.skipped ?? 0) > 0
? `, skipped ${current.skipped + (legacy?.skipped ?? 0)} non-matching`
: ''),
);
logWalk(log, {
mode: 'full',
leg: 'total',
query: `${legacy ? 2 : 1} leg(s)`,
window: `[${day(reached)} TO *]${complete ? '' : ' (truncated)'}`,
pages,
seen,
stored,
skipped: current.skipped + (legacy?.skipped ?? 0),
});
return {
mode: 'full',
pages,
stored,
seen,
@@ -301,14 +444,22 @@ async function walk(cutoffAt: Date, log: FastifyBaseLogger): Promise<RefreshResu
}
/**
* Coverage only ever improves: an incremental walk reaching back two days does
* not un-mirror what a full reload fetched last week.
* Coverage only ever **widens**: a walk that reached back two days does not
* un-mirror what a full reload fetched last week. So the stored value is the
* LEAST of what is there and what this walk reached — never the newer one.
*
* Done as one conditional upsert rather than read-then-write, so two runs
* cannot interleave and land on the narrower value. Both sides are ISO-8601
* UTC strings, which sort chronologically as text.
*/
async function widenCoverage(reached: Date): Promise<void> {
const current = await getMeta(META_COVERS_FROM);
const currentAt = current ? new Date(current) : null;
if (currentAt && !Number.isNaN(currentAt.getTime()) && currentAt <= reached) return;
await setMeta(META_COVERS_FROM, reached.toISOString());
await query(
`INSERT INTO app_meta (key, value) VALUES ($1, $2)
ON CONFLICT (key) DO UPDATE
SET value = least(app_meta.value, EXCLUDED.value), updated_at = now()
WHERE app_meta.value IS NULL OR app_meta.value > EXCLUDED.value`,
[META_COVERS_FROM, reached.toISOString()],
);
}
/**
@@ -321,10 +472,17 @@ export async function startRefresh(
explicitSince = false,
): Promise<boolean> {
return startTask(REFRESH_KEYS, 'nda-refresh', log, async () => {
// Resolved inside the task: the cutoff depends on the previous refresh,
// and the slot claim is what serialises two runs against each other.
const cutoff = await refreshCutoff(since, explicitSince);
return walk(cutoff, log);
// Resolved inside the task: the mode depends on the previous refresh, and
// the slot claim is what serialises two runs against each other.
const plan = await refreshPlan(since, explicitSince);
const result =
plan.mode === 'incremental' ? await walkRecent(log) : await walkWindow(plan.window, log);
// Here and nowhere else. ds_last_refresh_at is not a "the job ran" stamp:
// it says a walk completed and stored what it found, and the sync reads it
// to decide which rows the walk can no longer reach. Anything that moves
// it without a completed walk moves that boundary over unmirrored data.
await setMeta(META_LAST_REFRESH, new Date().toISOString());
return result;
});
}

6
web/src/api.ts
View File

@@ -285,6 +285,12 @@ export interface InboxRow {
export type RefreshState = 'idle' | 'running' | string;
export interface RefreshResult {
/**
* `incremental` reads the newest pages of the plain list, `full` re-reads a
* whole window through the search index — only after a moved date picker or
* on the very first run.
*/
mode: 'incremental' | 'full';
pages: number;
seen: number;
stored: number;

5
web/src/views/NdaInbox.tsx
View File

@@ -105,9 +105,10 @@ export default function NdaInbox({
useEffect(() => {
if (!refreshing && watchingRefresh.current && inbox?.last_refresh_result) {
watchingRefresh.current = false;
const { pages, seen, stored, legacy_stored: legacy } = inbox.last_refresh_result;
const { mode, pages, seen, stored, legacy_stored: legacy } = inbox.last_refresh_result;
setRefreshNotice(
`Refreshed: ${stored} request${stored === 1 ? '' : 's'} stored ` +
`Refreshed (${mode === 'full' ? 'full window' : 'newest first'}): ` +
`${stored} request${stored === 1 ? '' : 's'} stored ` +
`from ${seen} seen over ${pages} page${pages === 1 ? '' : 's'}` +
(legacy > 0 ? ` · ${legacy} in the pre-rename title format` : '') +
'.',