Apache Spark Enhances Parquet Reading and Nanosecond Precision

Apache Spark continues to refine its core execution engine with a focus on high precision temporal data and optimized columnar file reading. Recent activity in the master branch shows a significant push toward nanosecond precision across the SQL stack and substantial performance gains for Parquet decoding.

A major performance improvement landed this week targeting the decoding of Parquet files using the DELTA_BINARY_PACKED encoding. The optimization introduced by Ismael Mejia in commit 10711100 focuses on bulk read operations. This change affects how the vectorized reader handles integer and long types when they are stored using delta encoding, which is common in time series data or sorted identifiers.

The impact of this refactor is visible in the project internal benchmarks. For INT64 data using this encoding, the processing rate jumped from approximately 195 million rows per second to over 518 million rows per second on standard hardware. This represents a significant reduction in CPU cycles spent during the scan phase of a query. The changes primarily live in VectorizedDeltaBinaryPackedReader.java, which now handles these batches with fewer branching instructions.

This optimization is particularly relevant for users dealing with massive datasets where I/O and decoding overhead often bottleneck the entire pipeline. By improving the efficiency of the Parquet decoder, the project allows for better utilization of modern CPU architectures during data ingestion tasks.

The project is undergoing a comprehensive rollout of nanosecond precision for timestamp types. While Spark has traditionally operated with microsecond precision, the need for higher fidelity in financial and scientific applications has driven this expansion. This week, several commits have expanded the surface area for these new types.

Maxim Gekk has been leading this effort with updates such as commit ad886be8, which enables casting between nanosecond timestamp types of different precision. This is critical for maintaining compatibility when merging datasets from various sources that might not share the exact same temporal resolution. Further work in commit bfe75dbd reuses a shared internal type for nanosecond precision checks, ensuring consistency across the catalyst optimizer.

The integration extends beyond the internal engine. Support for rendering these high precision types was added to both the Thrift server and Hive results. This means that users querying Spark via external BI tools or the CLI will now see accurate nanosecond representations. The effort involved updating the Types Framework to handle the increased scale of these temporal values.

Security and stability updates have also been prominent in the recent commit history. One notable change for Kubernetes users is commit f7c26bc6, which sets the allowPrivilegeEscalation flag to false by default for both driver and executor containers. This aligns with modern security best practices for containerized workloads, reducing the potential blast radius of a compromised Spark task.

The Spark Thrift Server also received several hardening measures. A change by Dongjoon Hyun in commit 065b62e2 strengthens the cookie signing secret used in the HTTP transport layer. Additionally, the project now disallows setting JVM system properties within individual Thrift Server sessions to prevent malicious users from altering the global state of the shared server process.

To further restrict the execution environment, a new configuration option called spark.sql.reflect.allowList was introduced. This allows administrators to define a strict set of methods that can be invoked via the reflect or java_method SQL functions. This is a vital control for environments where users are allowed to write custom SQL but should not have unrestricted access to the underlying Java runtime.

The Python API and the Spark Connect architecture continue to see refinement as the project moves toward more modular execution. Commit 0f62d3df refactors the SQL_SCALAR_PANDAS_ITER_UDF implementation. This refactor cleans up how Spark manages the lifecycle of pandas iterators during UDF execution, which should improve memory predictability for large scale Python transformations.

Spark Connect, the thin client protocol for Spark, received an update adding a SQLContext wrapper in commit 7071ac1a. This provides better backward compatibility for older Spark scripts being migrated to the Connect architecture. By providing familiar entry points, the project lowers the barrier for adopting the decoupled client server model.

The Python worker also saw several updates in python/pyspark/worker.py, mostly focused on removing deprecated serializers and streamlining the communication between the JVM and Python processes. These changes are part of a broader effort to reduce the overhead of cross language data exchange.

The most significant trend to monitor is the move toward nanosecond precision as a first class citizen in the Spark ecosystem. While microsecond precision was sufficient for most batch processing in the past, the rise of high frequency data ingestion requires this level of detail. Operators should prepare for potential schema changes or storage overhead as these types become more prevalent.

The tightening of security defaults in Kubernetes and the Thrift Server indicates a maturing platform that is increasingly deployed in multi tenant or zero trust environments. If you rely on custom system property settings via the Thrift Server or require privilege escalation in your containers, you will need to explicitly adjust your configurations in upcoming releases.

Finally, the continued focus on vectorized Parquet reads shows that apache/spark is still finding significant performance headroom in its most common storage formats. Upgrading to the latest versions will likely provide immediate benefits for Parquet heavy workloads without any code changes on the user side.