Decaf Blog In-depth updates on Decaf software, tooling, and open-source experiments

Redpanda Connect v4.96.0 - BigQuery CDC Write Modes and Doris Stream Load

4 min read

Redpanda Connect v4.96.0 adds BigQuery CDC upsert write modes and a Doris stream load output, plus a DynamoDB CDC shard discovery fix for high shard count tables.

Redpanda Connect v4.96.0 shipped on 11 June 2026. The release is a stable build, not a prerelease. The headline for pipeline operators is native Change Data Capture write support on the BigQuery connector, with automatic metadata column injection for upsert and delete events.

The full release notes and downloads are on the GitHub release page.

BigQuery CDC upsert and delete write modes

The BigQuery output connector gains two write modes aimed at CDC pipelines: upsert and upsert_delete. Both modes inject _CHANGE_TYPE and _CHANGE_SEQUENCE_NUMBER into each row before load. That removes a common hand rolled step where teams map source change events into BigQuery merge compatible metadata outside the connector.

upsert handles insert and update events. upsert_delete extends that path to carry delete semantics through the same metadata contract. Pipelines that already standardize on BigQuery CDC column names can wire Redpanda Connect directly to a warehouse table without a separate transform stage for those fields.

The work landed in pull request #4453. If you run mixed insert, update, and delete streams into BigQuery, review your existing output config against the new mode names before cutover. The connector now owns column injection, so duplicate mapping in a Bloblang mapping or similar processor will produce conflicting values.

For operators comparing this release to prior tags, the CHANGELOG on the main branch lists the full delta. Installation steps are unchanged; the getting started guide still covers binary and container setup.

Doris stream load output

Apache Doris gets a dedicated stream load output in this release. Stream load is the standard high throughput ingest path for Doris: HTTP based bulk load with commit semantics suited to batch and micro batch ELT jobs. Adding it as a first class output means Redpanda Connect can sink transformed records into Doris without an intermediate object store hop or a custom HTTP processor.

The connector was contributed in pull request #4218. Teams running Doris as an analytical serving layer off Kafka or Redpanda topics can now treat Connect as the load agent at the warehouse boundary. Expect configuration around stream load endpoints, table routing, and format selection to mirror other SQL warehouse outputs in the project.

This is a net new sink type. Existing pipelines that today land Parquet or JSON in object storage and trigger a separate Doris load job can evaluate whether a direct stream load path reduces latency and operational surface area. The tradeoff is tighter coupling between the streaming layer and Doris availability during ingest.

DynamoDB CDC shard discovery fix

The dynamodb_cdc input had a correctness gap on tables with many stream shards. DescribeStream responses are paginated. Earlier versions did not walk every page, so the input could miss shards and silently under consume change records on large or long lived tables.

Pull request #4489 adds full pagination over DescribeStream so all shards are registered before consumption starts. For platform teams replicating DynamoDB changes into Kafka, Redpanda, or downstream warehouses, this is a reliability fix rather than a feature addition. If your table has grown shard count over time and CDC lag looked unexplained, upgrading is worth scheduling.

No config key changes are called out in the release notes for this fix. Behavior should improve on upgrade without pipeline edits, though you may see briefly higher startup latency while additional shards are enumerated.

Scope of v4.96.0

This tag is narrow. Three user facing items: BigQuery CDC modes, Doris stream load, and DynamoDB CDC pagination. No breaking changes or migration steps appear in the published notes. Internal refactors, if any, are not highlighted in the release text.

That focus suits teams that need one connector fix or one new sink without absorbing a broad dependency bump. It also means most existing v4.95.x deployments can treat this as a low risk connector patch release if their pipeline touches none of the three areas above. Binaries and release assets are on the v4.96.0 release page.

Where to get it