Scraping City Permit Portals Yourself vs Igni

Permit data is public and free — so why not scrape it yourself? See the hidden cost of DIY permit scraping versus a maintained, normalized feed.

Building-permit records are public, and most cities publish them free on open-data portals — so the obvious question is why pay for a feed when you could scrape the portals yourself. For one city, that's a fair instinct. The honest answer for many cities is that the raw data is the cheap part; turning it into a clean, current, comparable stream is where the real cost hides.

This page lays out that hidden cost dimension by dimension and contrasts the do-it-yourself path with a maintained, normalized feed. It's not a knock on building in-house — sometimes that's the right call. It's a realistic look at what per-city schema drift, portal quirks, ZIP and geocoding gaps, ADU classification, and ongoing maintenance actually take. For the underlying mechanics, start with our building-permit data guide.

How they compare

Scraping City Permit Portals Yourself vs Igni: Igni versus Scraping it yourself
Dimension	Igni	Scraping it yourself
Where the data comes from	Official municipal open-data portals, ingested and normalized into one schema.	The same official portals — but you integrate and maintain each one yourself.
Per-city schema	Dozens of different portal systems mapped to a single consistent record shape.	Every city has its own fields, endpoints, statuses and date conventions to learn.
Portal technology	ArcGIS, Socrata and bulk-CSV feeds handled behind one interface.	You write and maintain a separate adapter per platform (ArcGIS / Socrata / CSV).
ZIP & geocoding gaps	Missing ZIPs recovered — reverse-geocoded from geometry or forward-geocoded from the address.	Many feeds ship no usable ZIP or coordinates — you build geocoding to fill the gaps.
ADU classification	Typed ADU classification (detached / attached / junior / conversion / unknown).	Raw work-description text you parse and classify yourself.
California SB-9 / SB-10	Carried as flags on California records.	You would model the eligibility and corridor logic yourself.
Freshness & monitoring	Continuous ingestion, sub-24h where the source portal supports it.	You schedule, run and monitor jobs — and notice when a feed silently changes.
Maintenance burden	Feed and field changes are absorbed for you.	Portals change fields, limits and formats without notice — ongoing upkeep.
Coverage to start	65 cities across 37 US states on day one, expanding by jurisdiction.	One city at a time, as you build and test each integration.
Cost model	Contact-driven during the pilot (pre-revenue, no public self-serve checkout).	The data is free; your engineering and maintenance time is the cost.

The data is free — the integration isn't

Pulling permits from a single city's portal is genuinely easy: hit the endpoint, read the rows, done. The trouble starts the moment you need a second city, then a fifth, then a region. Each jurisdiction is its own small integration project, and the cost isn't the records — it's the plumbing that makes records from twenty different systems usable side by side.

That's the gap a maintained feed fills. Igni ingests from the same official open-data portals you would, then does the normalization, typing and freshness work on top, so you get the authority of the public source without owning the per-city plumbing. If you want to see what a single city's raw record looks like first, try the permit-lookup tool.

Per-city schema drift and portal quirks

Permit portals are not standardized. One city exposes an ArcGIS FeatureServer with epoch-millisecond dates and a building-use column; the next runs Socrata with floating ISO strings and a work-type code; another only offers a bulk CSV you have to stream and de-duplicate. Field names, status vocabularies, export caps and date formats differ everywhere, and a row that looks complete in one city is missing its address or ZIP in another.

Handling that by hand means a custom adapter and a custom mapping per platform, plus the judgment calls — which date means "issued" versus "applied", how to read a status, where the site address actually lives. Igni does that mapping once per jurisdiction and keeps it current, so the output is one comparable schema across every market. See how broadly that reaches on the state coverage hub.

ZIP gaps, geocoding and ADU classification

Two of the most time-consuming problems are invisible until you hit them. First, location: a large share of feeds publish no usable ZIP, and some publish no coordinates either, so you end up building reverse-geocoding from point geometry and forward-geocoding from raw address strings just to make records filterable by area. Second, meaning: a permit's work description is free text, so deciding whether a row is a detached ADU, a garage conversion, or an unrelated remodel is a classification problem, not a lookup.

Igni solves both as part of the pipeline — recovering missing ZIPs and applying a typed ADU classification (detached, attached, junior, conversion, or unknown) — so the feed is filterable and meaningful out of the box. For how that typing works, see our ADU permit guide.

Maintenance is the recurring cost

Even a finished scraper isn't done. Portals get migrated to new permitting systems, rename fields, tighten rate limits, or start returning corrupt future-dated rows — usually without warning. A scraper that worked last quarter can quietly return empty or wrong data, and you only find out when a downstream report looks off. That monitoring-and-repair loop is the part teams consistently underestimate.

With a maintained feed, that upkeep is someone else's job: when a source changes, the mapping is fixed centrally and your output keeps flowing. The trade is the usual build-versus-buy one — control and zero licensing cost on one side, removed maintenance and instant breadth on the other.

When DIY makes sense — and where Igni fits

If you only ever need one or two cities, have engineering time to spare, and want full control of the pipeline, scraping the portals yourself is a perfectly reasonable choice — the data is free and authoritative. The math changes when you need many markets, fresh data, recovered ZIPs, and ADU-level typing without standing up and babysitting an integration for each city.

That breadth-plus-specialization is exactly what Igni is built to remove from your plate, sourced verifiably from official open data. It's contact-driven during the pilot — no public self-serve checkout yet — so to see coverage in your markets, request access. For an audience-specific view, see our page for general contractors.

Frequently asked questions

Can I just scrape city permit portals myself?

Yes — building permits are public and most cities publish them free on open-data portals, so a single-city scraper is straightforward. The cost grows with scale: each additional city is its own integration, and you take on geocoding, ADU classification, and ongoing maintenance as portals change.

Why is permit data so hard to normalize across cities?

There's no standard. Cities publish through different platforms (ArcGIS, Socrata, bulk CSV) with their own fields, status vocabularies, date formats and export limits, and many omit a usable ZIP or coordinates. Making records from many cities comparable means mapping each one and filling the gaps.

Do permit portals offer an API?

Many do — ArcGIS and Socrata portals expose queryable endpoints, and some cities only publish bulk CSV files. The catch is that each behaves differently, so consuming several of them still means writing and maintaining a separate adapter and field mapping per platform.

How is Igni different from scraping it myself?

Igni ingests from the same official open-data portals, then normalizes dozens of systems into one schema, recovers missing ZIPs, applies typed ADU classification, and keeps the mappings current as sources change. You get breadth and freshness without owning the per-city integration and maintenance.

See how Igni fits your markets

Igni is contact-driven during its pilot. Tell us which cities and project types you focus on, and we'll show you the coverage, freshness and ADU typing for your markets.

Request access Browse coverage