A quality dashboard we maintain started showing totals that were slightly lower than the spreadsheets they came from. Nothing failed. No error appeared in the logs, the sync reported success, and every chart loaded. The numbers were simply short.
What was happening
The data lives in a managed PostgreSQL database on Supabase, and the dashboard reads it through Supabase's REST API. That API caps every response at a maximum number of rows, and the default cap is 1,000. Ask for 1,400 rows and you receive the first 1,000 with a normal success response.
For months the tables were small enough that the cap never mattered. Once a table passed 1,000 rows, every query against it began returning partial data, and the totals drifted without anyone noticing.
The three changes that fixed it
1. Page through results until there are none left
Instead of one request, the sync now asks for 1,000 rows at a time and keeps going until a page comes back short.
async function fetchAll(table, pageSize = 1000) {
const rows = [];
for (let offset = 0; ; offset += pageSize) {
const page = await getRows(table, { limit: pageSize, offset });
rows.push(...page);
if (page.length < pageSize) return rows;
}
}Raising the cap in the project settings also works, but it only moves the cliff further away. Paging works at any size.
2. Retry the requests that fail for passing reasons
More requests means more chances to hit a brief network error. A small retry wrapper with a growing pause between attempts turned those into non-events instead of half-finished syncs.
3. Check the count, and say so when it is wrong
The API can return the true total alongside the rows when you ask for an exact count. The sync now compares the rows it received with that total, and if a sync fails, a named person gets an email with the reason.
The lesson
The dangerous failures are the ones that look like success. Any time a system reads data in bulk, check that it received everything it asked for, and make silence mean that it worked.
This is the kind of plumbing we build into every Legentrix product and custom build. See how the platform handles data.