Spark 3 and old dates: the proleptic calendar trap

This is the kind of defect that does not look like a bug. The job runs, the file comes out, the schema validates. Only a few dates have shifted by ten days, and only on the oldest records. On a filing covering contracts opened decades ago, those records exist.

What changed between Spark 2.4 and Spark 3

Spark 2.4 used the hybrid Julian-Gregorian calendar, the one behind java.sql.Timestamp and the old java.util.Calendar API. Spark 3 moved to the proleptic Gregorian calendar, the one behind java.time, which applies Gregorian rules uniformly — including before their adoption in 1582.

For any date after 1582 the two calendars agree: no effect. Before that they diverge — by ten days at the reform, more as you go back. A sentinel value such as 0001-01-01, very common as a « not provided » marker in older systems, shifts by several days.

The problem is not the conversion. It is that it is silent: no exception, no warning in the application logs, no rejected row.

Two very different symptoms

Two mechanisms need separating, because they are not fixed in the same place.

String parsing. Spark 3 uses DateTimeFormatter instead of SimpleDateFormat. The latter was lenient; the new one is strict. Patterns that used to pass now throw — yyyy and YYYY do not mean the same thing, and an impossible date like 2021-02-30 was previously rolled silently to 2 March.

Reading files back. This is the vicious case. Parquet and Avro written by Spark 2.4 contain days and microseconds encoded against the old calendar. Read by Spark 3 without rebasing, they yield a different date. Here the source code has not changed at all: only the reader's interpretation has.

Detect before you migrate

The test is one query, and it should run before any decision about settings. You are looking for date columns whose minimum value predates the Gregorian reform.

val colonnesDate = df.schema.fields
  .filter(f => f.dataType == DateType || f.dataType == TimestampType)
  .map(_.name)

val bornes = df.agg(
  colonnesDate.map(c => min(col(c)).as(c)): _*
).collect().head

colonnesDate.zipWithIndex.foreach { case (c, i) =>
  val v = bornes.get(i)
  if (v != null && v.toString < "1583-01-01")
    println(s"[risk] $c contains old dates: $v")
}

On the datasets I have migrated, the same columns come up every time: dates of birth, contract effective dates, and above all technical sentinels (0001-01-01, 1000-01-01) used to mean « missing value ». That last case is the most frequent and the easiest to fix: the real answer is to replace the sentinel with null, not to rebase.

The three policies, and which to choose

Spark exposes distinct switches. Confusing them is the next source of error.

# parsing and formatting of character strings
spark.sql.legacy.timeParserPolicy = EXCEPTION | CORRECTED | LEGACY

# reading and writing files already produced
spark.sql.parquet.datetimeRebaseModeInRead   = EXCEPTION | CORRECTED | LEGACY
spark.sql.parquet.datetimeRebaseModeInWrite  = EXCEPTION | CORRECTED | LEGACY
spark.sql.avro.datetimeRebaseModeInRead      = EXCEPTION | CORRECTED | LEGACY

The three values have very different consequences:

• EXCEPTION — Spark fails when it meets an ambiguous date. It is the default, and it is the right setting during the migration: it turns a silent difference into a loud error, which is exactly what you want at that moment.
• LEGACY — Spark rebases to reproduce 2.4 behaviour. This is what you need to read existing history without rewriting it. The cost is a rebase on every read.
• CORRECTED — Spark reads the values as they are, no rebasing. This is the target once history has been rewritten.

The trajectory that works: EXCEPTION to discover the scale, LEGACY on read so production is not blocked, progressive rewriting of history in CORRECTED, then removal of the settings. What you must not do is set LEGACY everywhere on day one: that resolves the incident and freezes the debt.

The rule that avoids the whole subject

The regression test that protects you durably is not about the settings, it is about the values. On a reference day, compare the output of the old and new chains, date column by date column:

val ecartsDate = ancien.as("a")
  .join(nouveau.as("n"), Seq("id_contrat"))
  .filter(col("a.date_effet") =!= col("n.date_effet"))
  .select("id_contrat", "a.date_effet", "n.date_effet")

require(ecartsDate.isEmpty,
  s"${ecartsDate.count()} contracts whose effective date changed")

This test has a property the settings do not: it keeps protecting after the migration, at the next version upgrade. And it documents a decision that would otherwise be lost — because in two years, nobody will remember why datetimeRebaseModeInRead is set to LEGACY in the cluster configuration.

The general rule is simpler: a sentinel date is not a date. If your model uses 0001-01-01 to mean « unknown », the real fix is not a rebasing parameter. It is a null and a status column.