NVIDIA says AI agents can help rewrite Apache Spark UDFs for GPU acceleration, with reported 22x speedups in a 600-billion-row workload.

NVIDIA said AI agents can help rewrite Apache Spark user-defined functions into GPU-accelerated code, a workflow the company says can remove fallback overhead in Spark jobs and, in one reported production case, deliver a 22x speedup.

In a blog post published on 2026-10-13, NVIDIA described how its cuDF plugin for Apache Spark can accelerate analytics workloads by keeping execution on GPUs instead of falling back to CPU processing when Spark encounters UDFs. The company said UDFs are a persistent bottleneck because they introduce opaque custom logic that Spark’s declarative engine cannot parallelize in the same way as built-in SQL and DataFrame expressions.

NVIDIA said that many UDFs can be rewritten in one of three ways: as Spark SQL expressions for simple logic; as cuDF-based implementations through the RapidsUDF interface for more complex columnar processing; or, for the most complex cases, with custom CUDA kernels invoked through JNI. The company framed these as a complexity ladder, with SQL offering the lowest overhead and CUDA the most control.

The post used a Celsius-to-Fahrenheit transformation as an example, first showing a Scala UDF and then a direct SQL rewrite that the cuDF plugin could accelerate transparently. It also showed a RapidsUDF implementation using cuDF ColumnVector and Scalar operations to process data in parallel on the GPU.

NVIDIA said the rewrite workflow is designed to be assisted by coding agents through four skills: scaffolding and design, testing, conversion, benchmarking and optimization.

Source link