A market-intelligence product built from sources that were never meant to be read

The problem

Development-bank and investor colleagues kept asking us the same thing: what's new on the registries, who has done what, and is there anything we should know? Answering it by hand meant hours of clicking through four registry websites that each describe projects differently and change their platforms without warning.

Commercial tools exist, but they're priced for global research desks. Our buyers wanted one region or one theme, kept current, at a fraction of the price.

What we built

• Four registry adapters that read what each registry's own website serves, plus a public research database for classification.
• Immutable snapshots. Every fetch is stored exactly as served. Normalisation runs on read, so a fix applies to all history without re-fetching.
• Change detection. Each run is diffed against the last, so the database holds the full history of every project's status, not just today's state.
• Failure isolation. One registry breaking is logged and flagged, and it doesn't stop the run. When a major registry moved platforms, it broke the old pipeline silently. This one is built so that can't happen quietly again.
• Editions as filters. Global data with regional and thematic editions on top. Click tracking on each edition measures demand before an edition is built.
• Dashboard and briefing generated from the same data, mirrored to Postgres for querying.

Where the AI is

AI wrote and maintains the adapters, which are exactly the kind of brittle, specific code that used to need a dedicated engineer. Commentary is AI-drafted and human-approved: nothing editorial is sent automatically. The weekly question that used to take a day now takes a coffee.

How fast

The pipeline runs every morning on a schedule with nobody watching it. When a registry changes its format, a fix is usually a single adapter change that takes hours, not a rebuild.

How this applies to your business