Introducing TDD in a team that has never done it

A team that does not write tests is not a careless team. It is almost always a team whose code is hard to test, and which rationally concluded the cost exceeded the benefit. Asking for tests without addressing that cause produces accompanying tests: written afterwards, on whatever was easy, and protecting nothing.

Why « write tests » does not work

On an existing computation engine, the first obstacle is structural. The code reads its configuration from a file, opens a session, writes to a database: there is no pure function to call. Writing a test means refactoring first, and refactoring without tests is exactly what nobody wants to do.

This is a bootstrapping problem, not a discipline problem. As long as you treat it as discipline, you get guilt rather than coverage.

The second obstacle is timing. A test pays off months later, when something is modified. The cost is immediate. A team under deadline pressure rationally picks the short term — until it sees the benefit for itself.

Start with characterisation tests

The unlock is to invert the usual order. You do not write tests of what the code should do — nobody knows that with certainty on an old engine. You write tests of what it does, as it is, right or wrong.

Concretely, on a reference day of production: capture the inputs, capture the outputs, turn them into golden files, and write a test that replays and compares.

class EngineCharacterisation extends AnyFunSuite with SparkSessionTest {

  test("the computation reproduces the reference output of 12 March") {
    val input     = readSet("golden/2026-03-12/input.parquet")
    val reference = readSet("golden/2026-03-12/output.parquet")

    val actual = Engine.compute(input, DefaultParameters)

    // order-insensitive comparison, tolerant on floating point
    assertDataFrameEquals(actual, reference, tolerance = 1e-9)
  }
}

This test has three properties no « properly written » test has at the start. It requires no refactoring beforehand. It documents real behaviour, oddities included. And it immediately authorises refactoring, since you will know if you changed anything.

The caveat: a characterisation test also pins the bugs. That is intended. When you fix a bug, you update the golden file in the same commit, with an explanation. The question « why did this value change » then has a written answer somewhere.

The specific case of Spark jobs

Testing Spark has a reputation for being slow, and it is — when you test at the wrong level. The rule that changes everything: separate the transformation from the session. A function taking DataFrames and returning a DataFrame tests locally in a few hundred milliseconds; a function that opens its own session and reads its own paths does not test at all.

// hard to test: the function fetches its inputs and configuration
def compute(): Unit = {
  val spark = SparkSession.builder().getOrCreate()
  val df = spark.read.parquet(Config.path)
  df.filter(...).write.mode("overwrite").parquet(Config.output)
}

// testable: inputs and parameters arrive through the signature
def compute(input: DataFrame, params: Parameters): DataFrame =
  input.filter(col("status") === params.status)
       .withColumn("net_amount", col("amount") - col("fees"))

A local Spark session shared across a suite reduces the cost further:

trait SparkSessionTest extends BeforeAndAfterAll { self: Suite =>
  lazy val spark: SparkSession = SparkSession.builder()
    .master("local[2]")
    .config("spark.sql.shuffle.partitions", "2")   // decisive: 200 by default = slow
    .config("spark.ui.enabled", "false")
    .getOrCreate()
}

Setting shuffle.partitions to 2 is the change that takes a suite from several minutes to a few seconds. It is often that alone which reconciles a team with running the tests before pushing.

Make the rule mechanical, not moral

Once the net is in place, coverage grows if it is backed by a mechanism, not by a reminder in a meeting. Three levers, by effectiveness.

• A threshold on new code, not on the whole codebase. Demanding 80% across an old engine is discouraging and will be gamed. Demanding 80% on the lines a pull request modifies is achievable, and raises global coverage mechanically.
• A mandatory test with every bug fix. The easiest rule to accept, because it is obviously rational: nobody wants to see that bug again.
• Quality as a blocking failure, not a warning. A report you can ignore is ignored. A red pipeline gets handled.

# quality gate on new code only
sonar.qualitygate.wait=true
# minimum new coverage: 80 % of added or modified lines

The cultural shift, though, comes from none of these rules. It comes the day the net catches something: a characterisation test failing on a change everyone believed harmless. That day, the team stops seeing tests as a tax. Just make sure it happens early — and tell the story when it does.

What coverage does not tell you

A high coverage figure is an indicator, not a guarantee. It says which lines were executed, not whether the result was checked. You can reach 95% with tests that assert nothing.

The three questions I ask instead, which are more revealing:

1. How long does the full suite take? Beyond ten minutes it will not be run locally, so it no longer protects you while you write.
2. When a test fails, how long does it take to understand why? A test whose failure cannot be diagnosed will be disabled at the first emergency.
3. Is there a test that would fail if you inverted one condition in the core business computation? That is the only real question — and you can answer it in a minute, by inverting the condition.

Coverage is a way to get a budget and to track a trajectory. What protects production is the content of the assertions.