dlt Tightens BigQuery Replace and Parquet Writer Control

This week in dlt, a Python data loading framework for ETL and ELT pipelines, the recent activity is mostly about safer destination behavior rather than new connector sprawl. The 13 commits touch BigQuery replace semantics, Parquet writer knobs, Airflow parallel runs, and metric aggregation. For teams running scheduled batch loads, those details matter more than a flashy API.

The clearest data warehouse change is the BigQuery atomic replace commit. It adds an opt in enable_atomic_replace setting on the destination path surfaced through dlt/destinations/impl/bigquery/factory.py. When paired with replace_strategy set to truncate-and-insert and GCS staging, dlt can use a single BigQuery load job with WRITE_TRUNCATE_DATA.

Why care? Standard full refresh paths often trade data visibility for table metadata. The new path is meant to keep the BigQuery table object in place, so policies, labels, descriptions, constraints, and dependent materialized views have a better chance of surviving the refresh. It is still per table, not a transaction across an entire nested chain.

This is deliberately not the default. The code warns and falls back when staging is not GCS, or when the replace strategy is not compatible. That is the right shape for a warehouse feature that changes how refresh windows behave. Operators can test one pipeline and one destination before making it a broad policy.

Two nearby commits suggest the team is still tightening replacement semantics around staging and empty loads. The replace strategy switch fix changes dlt/load/utils.py so the staging dataset stores a schema trimmed to the tables that can actually materialize there. That matters when an existing dataset moves from ordinary truncate-and-insert to insert-from-staging or staging-optimized; the schema hash must notice that staging tables now need to exist even when the logical table schema did not change.

The empty file truncate change pushes a different part of the same theme. When a replace resource yields no data, dlt now uses a zero row job for top level tables instead of relying on package state to register truncation. The visible effect is less special case state in load package info, while the destination still ends up empty where a full refresh promised an empty table.

These are not glamorous changes. They are the sort of details that decide whether a backfill behaves like the previous run, especially with nested tables, table variants, and staging optimized strategies.

The Parquet compression commit is small but useful. It adds a compression option to the Parquet writer path in dlt/common/data_writers/writers.py, with snappy as the default and values such as gzip, brotli, lz4, zstd, and none available through config. The normalized writer can now be driven with NORMALIZE__DATA_WRITER__COMPRESSION, so a pipeline can change internal Parquet codec without changing the resource itself.

For data platforms, this is a practical knob. zstd may reduce object store size. snappy may be the safer default for broad destination support. none can help isolate whether a downstream import problem is a codec issue or a schema issue. The key point is that the choice moves out of source code and into deployment config.

The metrics aggregation fix adds a shared aggregation helper in dlt/common/metrics.py. It routes extract metrics in dlt/extract/extract.py and normalize counts through the same grouping path. The old pattern used grouped iteration, which is easy to get wrong when the input order is not already sorted by the grouping key.

That is painful in parallel normalize runs, where file rotation and multiple workers can reorder job metrics. The change also tests resource metrics with custom counters, table metrics after file rotation, and normalize counts with four and eight workers. This is basic observability hygiene. If row count dashboards are wrong, users cannot tell whether a load lost rows or only the reporting layer lied.

The empty Arrow table fix is narrower. It ensures _dlt_id is added with an explicit Arrow string type when normalizing empty Arrow tables, avoiding the ArrowInvalid path. Empty inputs are common in incremental jobs. They should produce zero rows, not an exception that looks like schema corruption.

• Treat enable_atomic_replace as a destination specific setting. It needs GCS staging and truncate-and-insert, and it is atomic per table rather than per pipeline.
• If storage cost or import compatibility matters, test Parquet compression with representative data before changing production config.
• After upgrading, compare extract, normalize, and load row counts on one noisy pipeline. The metrics fix should make those numbers more trustworthy, but dashboards may expose prior counting mistakes.