OpenSanctions Refines PEP Data Pipelines and Crawler Reliability

OpenSanctions recently released a large update to its data processing engine and PEP dataset collection. This week saw over 130 commits focusing on structural refactors to the occupancy data model and significant hardening of crawler reliability against source web application firewalls.

The OpenSanctions project maintains a high precision database of persons of interest such as sanctioned entities and Politically Exposed Persons (PEPs). As an open data project, it relies on the Zavod ETL framework to ingest data from hundreds of disparate sources. The latest activity shows a move toward more explicit data modeling and better validation for these large scale pipelines.

A major structural change landed this week with the removal of implicit country propagation in the occupancy helper functions. Previously, the make_occupancy function would automatically copy the country of a political position onto the person holding it. While this was a useful heuristic for national parliaments, it created data quality issues for diplomatic roles and other cross border positions where the position country does not match the person’s citizenship.

In commit 2bf787ba, the project removed the propagate_country parameter from the core occupancy helper. This change forces crawlers to explicitly set citizenship or nationality on Person entities. For existing datasets that relied on this behavior, such as the Wikidata PEP collection, the logic was moved into the specific crawler to ensure data continuity.

This refactor improves the precision of the FollowTheMoney data model used by OpenSanctions. By requiring explicit evidence for a person’s country affiliation, the project avoids the risk of incorrectly labeling a diplomat’s citizenship based on their host country. It also simplifies the core library by removing a complex inference loop that had become a source of technical debt.

Web scraping at scale requires constant adaptation to source changes and defensive measures. Several updates this week focused on making crawlers more resilient to network errors and anti bot protections. For example, the crawler for Croatian public officials was updated in commit 39241e8b to handle intermittent 403 and 415 errors from a source Web Application Firewall (WAF).

The project also addressed schema drift in external sources. The US state terrorist organizations crawler was updated to follow source column renames for date fields. In the Ukraine war sanctions dataset, the team improved the modeling of liquidated companies by treating assignees as successions. These changes ensure that the data remains accurate even as the underlying sources evolve.

Other notable crawler updates include:

• Adding name date party lookups for the Netherlands Senate.
• Fixing production warnings in the Ukraine war sanctions HTML parser.
• Switching the Tunisian CNLCT crawler to a consolidated French source list for better coverage.
• Improving the ESMA SARIS crawler with better fetch retry logic and CSV validation.

Validating large datasets is a performance challenge. OpenSanctions processes millions of statements, and ensuring referential integrity across them requires specialized tools. This week, the project introduced new validation recipes using the qsv command line tool. These recipes allow engineers to perform fast integrity checks on the statements.pack files generated during a crawl.

As documented in the crawler pep skill, these checks verify that every Occupancy entity correctly references a valid Person and Position. Crucially, a new check was added in commit 22bf90cd to identify PEPs who lack any country or citizenship data. This directly validates that the recent removal of country propagation has been correctly handled across all active crawlers.

Using qsv for these checks is a significant improvement over traditional unit tests or custom Python scripts. It allows for high throughput analysis of the raw statement stream, making it easier to catch data quality regressions before they reach the production database.

The project continues to expand its coverage of political figures globally. Over 11 new PEP datasets were released this week, covering various national assemblies and parliaments. This expansion was supported by updates to core dependencies like litellm and rigour, which provide the foundation for LLM assisted data cleaning and entity resolution.

The documentation for these datasets has also been updated to reflect new property priorities. The zavod documentation now explicitly lists which person properties are most important to capture, helping contributors build higher quality crawlers. This focus on documentation ensures that the project can scale its data acquisition efforts while maintaining strict standards for data integrity.

Data consumers and crawler authors should review their pipelines to ensure they are not relying on the now removed country propagation. If you are building new crawlers for OpenSanctions, make sure to:

• Explicitly capture citizenship or nationality from the source data whenever possible.
• Incorporate the new qsv validation recipes into your local development workflow.
• Monitor for 403 or 415 errors that might indicate new WAF restrictions on source sites.
• Check the latest zavod documentation for updates on how to model complex entity relationships like successions for liquidated companies.

These refinements to the OpenSanctions pipeline represent a commitment to data quality and system reliability that is essential for a project of this scale.