Apache NiFi Expands Connector APIs and Fixes Kafka Header Encoding

Apache NiFi is a flow based data integration platform used for ETL, streaming ingestion, and edge collection. The last week on main brought 14 commits and a large dependency refresh ahead of 2.11.0-SNAPSHOT. The work that matters for operators is not the UI package bumps. It is connector observability APIs, a Kafka header encoding fix, and several pipeline processor corrections that affect how binary metadata survives a flow.

NiFi’s connector model lets external systems manage flows without driving the full canvas UI. Two commits extend that surface in ways automation teams will notice.

Added endpoints for obtaining Status History landed in ConnectorResource with matching methods on ConnectorClient and JerseyConnectorClient. The change adds 716 lines across the web API, facade layer, and a new ConnectorStatusHistoryIT system test suite. Processors, connections, process groups, and remote process groups inside a connector now expose status history through the same endpoint merger pattern the core UI already uses. That gives external controllers a time series view of backpressure and throughput without scraping provenance logs.

The companion fix in findRemoteProcessGroupIncludingConnectorManaged closes a lookup gap. FlowController previously walked only the root process group when resolving remote process groups. Connector managed components lived outside that tree, so API calls could miss RPGs that the canvas showed correctly. The new method follows the same pattern as findConnectionIncludingConnectorManaged and findRemoteGroupPortIncludingConnectorManaged, and ControllerFacade.java now routes both affected facade methods through it.

If you run NiFi behind a custom control plane or a GitOps style flow publisher, these two commits are the headline. They reduce the gap between what the UI reports and what the connector REST client can query.

Kafka record headers often carry opaque binary tokens: trace IDs, checksums, or vendor specific flags. When ConsumeKafka decoded every header as a charset string, those bytes could corrupt silently.

Add Hex header encoding option to ConsumeKafka introduces a Header Format property with String and Hex modes. In Hex mode, header values become lowercase hexadecimal strings instead of passing through a character set decoder. The existing Header Encoding property keeps its meaning and appears only when the format is String, so current flows need no migration.

The implementation resolves a HeaderValueConverter once in onScheduled and applies it uniformly to FlowFile attributes and to wrapper or inject metadata record header fields. Internal batch headers such as kafka.max.offset and kafka.count still decode as fixed UTF 8, matching how they are written. Integration tests cover all three consumption modes, and the commit also corrects the KeyEncoding HEX description, which claimed uppercase output while HexFormat.of() emits lowercase.

public enum HeaderFormat implements DescribedValue {
    STRING("string", "String",
        "The header value is decoded as a string using the configured Header Encoding character set."),
    HEX("hex", "Hex Encoded",
        "The header value is interpreted as arbitrary binary data and is encoded as a lowercase hexadecimal string");

For pipelines that route Kafka metadata into downstream SQL or JSON transforms, this is a data integrity fix. Operators who saw replacement characters or truncated header attributes on binary payloads should flip the property on new processor versions rather than patching around charset damage in expression language.

The JSLTTransformJSON processor is a common middle step when NiFi normalizes semi structured payloads before loading a warehouse. Fixed single line comment bug, multiple let statements bug addresses three footguns that showed up in production style transforms.

Reading a JSLT file from a resource reference previously joined lines with no separator. Two let bindings on consecutive lines could merge into invalid syntax. The fix uses System.lineSeparator() when collecting file lines, which preserves statement boundaries. Validation errors now surface the underlying parser message instead of a generic wrapper string. File read failures include the IOException reason in the thrown UncheckedIOException.

The new tests cover multiline let blocks and single line comments inside transform files. Teams that store JSLT in external assets rather than inline properties should retest any transform that mixes comments with multiple bindings. The behavior change is small in code but large for anyone who hit silent merge failures on deploy.

A single maintenance commit, Bump Snowflake JDBC from 4.3.0 to 4.3.1, HikariCP from 7.0.2 to 7.1.0, and others, touched pom.xml files across AMQP, email, graph, Snowflake, standard processors, and the Hikari DBCP service bundle. The Snowflake JDBC bump to 4.3.1 and HikariCP 7.1.0 are the ones warehouse operators should track. The same commit also moves Flyway to 12.9.0, the AWS SDK BOM to 2.46.14, and the Neo4j driver to 6.2.0.

Separately, Use HOSTNAME_PORT_VALIDATOR for Token Request Endpoint tightens AwsRdsIamDatabasePasswordProvider. The token request endpoint property now uses HOSTNAME_PORT_VALIDATOR instead of HOSTNAME_PORT_LIST_VALIDATOR. That matches the documented hostname:port format for IAM token signing and rejects comma separated lists that the UI should never have accepted for a single endpoint override.

None of these are feature launches. They are the kind of dependency and validation hygiene that keeps long running NiFi clusters stable when upstream drivers fix connection edge cases.

The tree already carries the version bump to the 2.11.0 snapshot line from June 18. Expect the connector status history endpoints and the ConsumeKafka header format to ship in that line. Test connector clients against the new history routes before upgrading production flows that depend on external orchestration.

AGENTS.md and SECURITY.md discoverability is documentation only, but it signals how the project wants automated contributors and security reviewers to navigate the repo. Worth a read if you maintain a private NiFi fork.

Frontend dependency bumps (dompurify, hono, shell-quote) and provenance test stabilization are minor. The operational story this week is connector observability, binary safe Kafka headers, and JSLT transforms that survive real world file layouts.