Public data APIs pipelines deserve a four-point failure checklist

Registry pipelines rarely fail loudly. They keep running and turn out answers that are subtly wrong. Someone who trusted a screen finds the error months later.

This is a list of how that happens, in rough order of how often each case shows up in production, with the check that catches it early.

What to take away

The silent failures

An error stored as an absence. A source that returns an empty result set when you are throttled, rather than a status code, will fill the table with "no record found" for entities that exist. Every downstream reader treats that as a finding. Catch it by exceeding the limit on purpose in a test and asserting that the pipeline refuses to write a nil result it cannot distinguish from an error.

A cache read as current. The registration status field can change with no filing at all, when a deadline passes and the office acts. Any decision path that reads status from storage without a freshness check will eventually approve exactly the case it exists to catch.

A partial backfill treated as complete. A job that stops halfway leaves a table that looks populated. Record the intended universe and the count actually retrieved, per run, and alert on the difference rather than on the absence of errors.

A schema built from one jurisdiction. Fields that are always present in the first source are sometimes absent in the fourth. The pipeline then either rejects valid records or writes defaults that read as data. Build for absence from the first day.

An identifier parsed as a number. Leading zeros vanish, prefixes are dropped, and long strings lose precision. Nothing errors. The joins simply stop matching, quietly, for a subset of records, which is why entity number lookup insists that identifiers are text.

A name used as a join key. Names change, repeat across states, and are punctuated four ways in your own systems. A name join splits one company after a rebrand and merges two after a coincidence, and both look like ordinary data until somebody checks.

The reconciliation failures

The API and the web interface disagree. Different field sets, different matching, occasionally different values. Users compare the two and report a bug that is not yours. Record which channel each value came from so the difference can be explained rather than argued about.

Two registers disagree about one company. A home registration and a foreign qualification describe different registrations, and neither is wrong. Merging them into one row produces a record with contradictory fields and no way to say which office asserted what.

Status vocabularies mapped and then forgotten. A local status string flattened into your own vocabulary loses the difference between never filed, administratively dissolved, and voluntarily dissolved. Keep the original alongside the mapping.

Type labels changing under you. Offices relabel document types on migration. A pipeline that keys on the label rather than on the office's document reference sees a new kind of event that never happened, which is one of the traps set out in filing history. A document reference survives relabeling, which is why one row per document is the rule in filing history data.

The access and permission failures

Terms discovered after launch. Free to view and licensed to use are separate grants, and the second is what decides whether you have a product or an internal tool. This is the ground the public data APIs pillar covers in full, and it belongs before design rather than after.

Rate limits treated as throttling rather than as terms. Limits are frequently stated in the terms of use, which makes exceeding them a breach and not merely a slow night.

No path for a correction or a deletion. When a source corrects, seals, or withdraws a record, a pipeline that cannot propagate the change will keep serving a record the office no longer stands behind. Build the path before you need it.

Personal fields collected because they were in the payload. The decision about individuals' names and addresses has to sit at ingestion. A system that filters on display has already stored everything, and which of your uses are regulated is a question for the relevant rules, with the Federal Trade Commission's guidance for businesses as a sensible first stop.

Access ending. Some terms require deletion of the copy you hold when access ends. A pipeline that cannot delete its own history has a problem that only becomes visible on the worst day.

The failures of shape

Snapshots where a log was needed. A filing history is an append-only log. Modeling it as attributes on the entity destroys the dates that give every row its meaning.

One retrieval date for a whole record. Fields refresh at different rates and sometimes from different channels. A single timestamp on the row is a claim you cannot support about the fields you did not touch.

No record of what was not retrieved. A gap in coverage that is not written down is indistinguishable from a gap in the source. Log the sources you chose not to query and why.

The catch-all check

Almost everything above is caught by one discipline: store the raw response, the channel, and the retrieval date beside every parsed value, and never let a value exist in the system without them.

Registers that document their own interfaces make this easy by naming what they return and what they promise, as the SEC's description of its filing search does for a federal system. Where the register documents nothing, the raw payload is the documentation, and you will need it the first time a field means something other than what you assumed.

Common questions

Which of these should I build for first?

The error-stored-as-absence one, because it corrupts data rather than stopping work, and because it is trivially reproducible in a test. Everything else is easier to repair after the fact.

How do I test a source's failure modes without breaking my access?

Ask the office what the limits are and whether a sandbox exists, then approach the limit deliberately in a small window rather than hammering the service. The point is to see the shape of the failure, which takes very few requests.

Is a nightly full reload safer than an incremental refresh?

Safer against drift, worse against rate limits and terms, and it still needs the same absence-versus-error distinction. Choose by what the source permits rather than by what is easier to write.

Our data looks fine. How would I know if any of this is happening?

Sample by hand. Take twenty records, open the office's own page for each, and compare field by field. That morning's work finds these failures more reliably than any monitoring you can build.