Public data APIs: questions before you build

Public means visible. It does not mean unrestricted, free to redistribute, or safe to build a product on. Every year somebody ships a service backed by registry data, gets a letter, and discovers the difference between a record anyone may look at and a dataset they may ingest, store, and resell.

Public data APIs for registry records raise a few knowable questions, many answered in a terms-of-use page nobody reads. This is what to read it for, before you write the first line of code.

What to take away

Access channels and their terms

Registry data may be available through different channels. A documented API is one programmatic option:

Not every office offers a documented API, and the terms attached to each access channel are usually different. This is the trap: the fact that the search page is free to use by hand tells you nothing about whether the same data may be harvested, stored, or shown to your users. Free to view and licensed to use are separate grants.

The licensing questions to settle first

Answer these before design, not after. Each one can change your architecture.

Redistribution. May you display the data to your own users, or is use limited to your organization's internal purposes? This is the question that decides whether you have a product or an internal tool.

Storage and derived works. May you cache, index, or combine the data with other sources? Some terms permit query-time use but not the construction of a competing database.

Attribution and currency. Many sources require you to name them and to show when the data was retrieved, and to avoid presenting a stale copy as current. If your interface has nowhere to put a source line and a timestamp, you have a design problem, not a legal one.

Commercial use and resale. Whether you may charge for a service built on the data, and whether "bulk" and "per-query" access carry different prices and different permissions.

Personal information. Registry records contain individuals' names and sometimes home addresses. Republishing those at scale is a materially different proposition from an individual looking up one company, and some sources restrict it explicitly. Decide before ingestion whether you will store personal fields at all. The defensible default for a business-verification product is not to: you can answer almost every question your users have from entity-level fields alone.

Corrections and deletions. When the source corrects, seals, or removes a record, what are you obliged to do, and can your pipeline actually do it? A system that cannot propagate a deletion is a system that will eventually be serving a record the source has withdrawn.

Termination. If access ends, what happens to the copy you hold? Some terms require deletion. Build knowing that.

Rate limits as license terms. Limits are often stated in the terms rather than only enforced technically. Exceeding them can be a breach, not just a throttle.

The technical questions

That last row deserves its own paragraph. An API that returns an empty result set when rate-limited, not an error code, will quietly poison your database with "no record found" for entities that exist.

Test the failure modes deliberately: exceed the limit on purpose, send a malformed query, and see what comes back. Then make your pipeline distinguish no match from no answer, and never let the second be stored as the first.

If there is no API

Scraping the search interface is governed by the site's terms and often prohibited outright. Check before assuming, and check the robots directives as well. If the office has no programmatic channel, that is frequently a deliberate decision rather than an oversight.

The better move is to ask, because many offices have a records, data, or public information unit. The state entity directories page can help you identify which state office handles the records.

Describe what you intend to do, ask what access exists and on what terms, and get the answer in writing. This short email occasionally produces access nobody knew was available, and it always produces a record of having asked.

Design consequences

If you take the above seriously, a few things follow about how the system should be built.

Store the raw response and the retrieval timestamp, not just your parsed fields. When a field's meaning turns out to differ from your assumption, the raw payload is the only way to fix history.

Treat every value as evidence with a date. A registry value is a statement about what an office's record said at a moment. Model it that way, value plus source plus retrieved-at, rather than as a current attribute of the company.

Re-verify before decisions. Cached registration status is the field that ages worst and the field most likely to be wired into an automated gate. Never let a stored status drive a credit or onboarding decision without a fresh check, and keep the human sequence in company verification as the thing the automation is serving rather than replacing.

Show provenance in the interface. Source, record date, and retrieval date, next to the value. This satisfies most attribution terms, and it is also just honest.

Reconcile across states, do not merge. The same company holds a different identifier in every state it is registered in. Model those as separate registrations of one entity rather than collapsing them, or you will lose the jurisdiction that gives each record meaning.

Federal sources are a different shape

Federal filing systems are often more API-friendly than state offices, and EDGAR full-text search is the obvious example, genuinely rich, genuinely queryable. The catch is coverage. It holds companies that file with the securities regulator, which is a small fraction of registered businesses, and it says nothing about the private company in the next town.

Building a verification product on federal data alone produces something that works beautifully in demos featuring listed companies and fails on the users' actual counterparties.

Before you build

Write down, for each source you intend to use: what the license permits, whether personal fields are in scope, the rate limit and its enforcement, the refresh lag, the failure behavior, and who at the office you asked. Keep it with the code.

That document is what lets someone eighteen months from now change the pipeline without re-litigating decisions nobody recorded, and what you will be very glad to have if anyone ever asks on what basis you were using the data.

Common questions

Is scraping a public search page allowed?

That depends entirely on the site's terms and its robots directives, and it is often prohibited outright. Read the terms first, and treat the absence of a programmatic channel as a decision the office made rather than a gap to route around.

Can I store the data I retrieve?

Sometimes for internal use only, sometimes not at all, and sometimes freely. Storage, redistribution, and derived works are usually three separate permissions, and the answer decides whether you have a product or an internal tool.

What happens when the source corrects or withdraws a record?

You need a path that propagates the change, and you need it before the situation arises. A pipeline that cannot delete will eventually serve a record the office no longer stands behind.

Should I hold the names of individuals that appear in the records?

Default to not holding them. Almost every question a business-verification product answers can be answered from entity-level fields, and the decision has to sit at ingestion rather than at display.

The API returns fewer fields than the web page. Is that a bug?

Usually not. Channels drift apart, and the difference is rarely documented as a difference. Check field by field before promising anything, and record which channel each stored value came from.