From scrapers to reliable background-check reports

The challenge

Before working with a person or a vehicle, the operation needs to know their background — records in official and restrictive lists, and the status of vehicles.

Until then this relied on scrapers that read 3 public websites directly. Whenever a site was down or changed its layout, the scraper failed and the check was incomplete — and only those 3 sources were covered.

What changed

• From scraping to an API. Moved to the TusDatos.co API, which aggregates official sources behind a stable interface — no more broken scrapers when a website changes.
• From 3 to 10+ sources. Coverage more than tripled by using the sources the provider already offers.
• Internal data, for the first time. Combined the external results with the company’s own internal data to improve searches and give a more complete picture — something the old process didn’t do.

How it works

1. Request. A user asks for a report by ID document (people) or plate (vehicles), one at a time or as a batch.
2. Report API. A Flask service receives the request and hands it to a per-country processor (starting with Colombia), so each country’s rules and sources stay isolated and new ones can be added.
3. Launch. The processor submits the query to the TusDatos.co API, which aggregates hundreds of official sources. Lookups take time, so the query runs asynchronously instead of making the user wait.
4. Status checker. Airflow background and batch jobs poll the status of each query until its results are ready — large batches never block the platform.
5. Transformer. A transformation step normalizes the raw results into a single report structure and enriches them with our own internal data, giving a more complete picture than the external source alone. Results are stored in MongoDB.
6. Report. A React view presents the final background report for the person or vehicle.

My role

I led, managed and developed the project: planned the migration from scrapers to the API and coordinated the team and timelines, designed the integration and the processing flow, and built the service, the pipelines and the report view.

Result

• 10+ sources instead of 3, with a stable API instead of scrapers that broke.
• Reliable, enriched reports that combine official sources with internal data.
• On demand and in batches, generated in the background without slowing down the platform.