Java 8 → 17 under Spark: the list of surprises

On paper, moving a Spark job from Java 8 to Java 17 means changing a version in the build file. In practice, JVM 17 enforces restrictions that 8 did not, and Spark relied heavily on what has just been closed. Here is what you meet, in order.

The module system closes the door

Since Java 9, strong encapsulation prevents reflective access to JDK internals. Spark needs it: its memory manager touches sun.misc.Unsafe, and its serialisers reach into private fields of java.nio and java.util. Under Java 11 you got a warning; under Java 17 it is an error.

The typical symptom is not explicit:

java.lang.reflect.InaccessibleObjectException: Unable to make field
private final byte[] java.lang.String.value accessible:
module java.base does not "opens java.lang" to unnamed module

The fix is a set of options passed to the driver and the executors. Forgetting the second is the most common mistake: the job starts, then fails on the first distributed task.

--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED

In spark-defaults.conf:

spark.driver.extraJavaOptions    --add-opens=java.base/java.lang=ALL-UNNAMED ...
spark.executor.extraJavaOptions  --add-opens=java.base/java.lang=ALL-UNNAMED ...

These options are not a temporary patch: they are part of the normal runtime configuration of Spark on a modern JVM. Version them and document them now, rather than rediscovering them at the next incident.

The garbage collector changes profile

Java 8 defaulted to the Parallel GC, tuned for throughput. Java 9 and later switch to G1, oriented towards pause latency. For a batch job this is not neutral: G1 handles large heaps better but can use more off-heap memory, and behaves differently on long-lived executors.

The symptom is not an exception but a drift: executors killed for memory overrun by the resource manager, when the same configuration worked under Java 8. The cause is usually that off-heap memory was never re-evaluated.

spark.executor.memory              8g
spark.executor.memoryOverhead      2g      # raise this, not only the heap
spark.executor.extraJavaOptions   -XX:+UseG1GC -XX:MaxGCPauseMillis=200

Good practice is to measure before concluding: enable GC logs on one executor during a reference run, and compare time spent in collection before and after.

-Xlog:gc*:file=/tmp/gc-%p.log:time,uptime:filecount=5,filesize=20M

Scala compatibility

The Scala version / Java version pair is constrained, and getting it wrong produces obscure messages. Scala 2.12 supports Java 17 from its recent releases; older ones emit bytecode the JVM 17 refuses. The symptom is an UnsupportedClassVersionError or, more treacherously, a NoSuchMethodError at runtime on a method that does exist.

The check that saves hours of searching:

scala.util.Properties.versionString      // Scala version actually loaded
System.getProperty("java.version")      // driver JVM
sc.range(0, 1).map(_ => System.getProperty("java.version")).collect()  // executor JVM

The last line is the one that matters. A driver on Java 17 with executors still on Java 8 — because the cluster image was not updated — produces deserialisation errors that look nothing like a version problem.

What breaks in tests before production

Three families, in order of frequency.

• Bytecode manipulation libraries. Mockito, ByteBuddy, cglib: all must be upgraded. A test suite that compiles but fails on the first mock() is almost always this.
• Number and date formatting. Java 9 adopted CLDR locale data. A narrow no-break space replaces the ordinary space as the thousands separator in French. Any test comparing a formatted string fails — and what should worry you is when an output file changes format and nobody tests it.
• Pinned transitive dependencies. Jackson, Netty and the compression codecs must align with what Spark ships, otherwise you get class conflicts the classpath hides until distributed execution.

A useful regression test on formatting, precisely because the symptom is invisible:

test("the thousands separator stays the one the filing format expects") {
  val f = java.text.NumberFormat.getInstance(java.util.Locale.FRANCE)
  assert(f.format(1234567) == "1 234 567")   // checks the character actually produced
}

The migration order that limits the damage

The sequence I apply, and the reason for each step:

1. Compile on Java 8, run on JVM 17. This separates compilation problems from runtime problems, and surfaces the missing --add-opens without changing a line of code.
2. Align the test dependencies. They block everything else until they pass.
3. Compile on Java 17, without using new language features. The target is compatibility, not modernisation; mixing the two makes regressions unanalysable.
4. Compare outputs on a reference day. This is the step that catches formatting changes and date shifts — the two defects compilation cannot see.
5. Revisit memory sizing. Only at the end, once behaviour is stable.

The general principle: a JVM upgrade on a critical platform is handled like a data migration, not like a tooling update. What decides success is not compilation, it is comparing the outputs.