Oozie → Airflow: what does not translate

Migrating from Oozie to Airflow is usually presented as a port: translate XML into Python, reconnect the jobs, switch off the old one. In a setting where the jobs carry a regulatory deadline, that reading is expensive. The two tools do not share a triggering model — and it is the model, not the syntax, that decides whether a chain lands on time.

Two different mental models

An Oozie coordinator is fundamentally data-oriented. You declare datasets with their location and periodicity, and it triggers the workflow when the input instances exist:

<coordinator-app name="declaratif" frequency="${coord:days(1)}" ...>
  <datasets>
    <dataset name="contrats" frequency="${coord:days(1)}">
      <uri-template>/data/contrats/${YEAR}${MONTH}${DAY}</uri-template>
      <done-flag>_SUCCESS</done-flag>
    </dataset>
  </datasets>
  <input-events>
    <data-in name="in" dataset="contrats">
      <instance>${coord:current(0)}</instance>
    </data-in>
  </input-events>
  ...
</coordinator-app>

The _SUCCESS file is the contract. Until it appears, nothing starts, and the coordinator waits as long as it takes.

An Airflow DAG is time-oriented. It has a schedule, and at the appointed hour it starts. Data availability is not a trigger condition: it is a condition you must reintroduce explicitly, with a sensor.

with DAG("declaratif", schedule="0 1 * * *", catchup=False,
         default_args={"retries": 2}) as dag:

    attend_contrats = FileSensor(
        task_id="attend_contrats",
        filepath="/data/contrats/{{ ds_nodash }}/_SUCCESS",
        poke_interval=120,
        timeout=60 * 60 * 3,      # beyond this you want an alert, not a wait
        mode="reschedule",        # frees the worker between checks
    )

Three parameters carry all the meaning. timeout turns an infinite wait into a visible incident — the thing most often missing from a first migration. mode set to reschedule stops a sensor from occupying a worker slot for three hours, which is the number one cause of a freshly migrated scheduler seizing up. And poke_interval should match the real granularity of the delivery, not the nervousness of whoever wrote the DAG.

The catchup trap

Airflow thinks in intervals. A daily DAG running on 12 August processes the interval of the 11th. Deploy a DAG with a start_date in January and catchup=True — the historical default — and Airflow immediately schedules every missing run since that date.

On a computation chain, that saturates the cluster. On a filing chain it is worse: you regenerate and re-submit files for periods that are already closed.

On a regulatory perimeter, catchup=False is not a preference, it is a safety measure. Backfilling must be a deliberate action, not a side effect of deployment.

The corollary: since backfilling becomes manual, it has to be tooled. A documented, tested recovery command bounded to a date range — not an airflow dags backfill typed under pressure at two in the morning.

What has to be rewritten by hand

Here is what does not translate mechanically, in the order it bites.

• Dependencies between coordinators. Oozie expresses them through shared datasets. Airflow makes you choose: a sensor on the output table, an ExternalTaskSensor on the other DAG, or merging the two DAGs. Each has different recovery properties — this is an architecture decision, not a translation.
• Date semantics. ${coord:current(-1)} becomes {{ data_interval_start }} — but the one-interval shift is a systematic source of error. Check every written partition, not just every translated expression.
• Retries. A restarted Oozie workflow resumes at the first unfinished action. Airflow retries at task level. A task that writes without being idempotent will produce duplicates on its first retry — and the first retry always comes.
• SLAs. Oozie does not carry them natively, Airflow does. Take the opportunity to make them explicit: a legal deadline belongs in the DAG, not in a spreadsheet.

Cutting over: two chains in parallel

The only method I recommend on a critical perimeter is the compared double run. Both orchestrations produce their output for the same period, and a comparison job blocks the cutover as long as a single row differs.

val ancien  = spark.read.parquet(s"/prod/oozie/declaratif/$jour")
val nouveau = spark.read.parquet(s"/prod/airflow/declaratif/$jour")

val cles = Seq("id_contrat", "id_personne")

val ecarts = ancien.as("a")
  .join(nouveau.as("n"), cles, "full_outer")
  .filter(cles.map(k => col(s"a.$k").isNull || col(s"n.$k").isNull)
              .reduce(_ || _) || col("a.montant") =!= col("n.montant"))

require(ecarts.isEmpty, s"${ecarts.count()} differences between the two chains")

The full_outer join is essential: an inner join would see neither the rows that appeared nor the rows that vanished — precisely the class of regression you are looking for.

The cutover criterion must be a number of consecutive clean days, not an impression. On a monthly filing, that means simulating past days rather than waiting months.

What you actually gain

The gain usually advertised is moving from XML to Python. It is real but secondary. The three that matter over time lie elsewhere.

Dependencies become readable. A graph view showing where a chain is stuck replaces reading several coordination files side by side.

Tasks become testable. A DAG is Python code: you can instantiate the graph in a test, check it has no cycle, and check that every critical task carries a timeout and an SLA. That test catches scheduling regressions before production.

def test_critical_tasks_have_a_timeout():
    dag = DagBag().get_dag("declaratif")
    for name in ("attend_contrats", "depot"):
        assert dag.get_task(name).execution_timeout is not None

Observability becomes native. Duration per task, history, alerting on overrun: these are the data that let you say « this job gained an hour » with a chart rather than a memory.

If you keep one rule from this migration: do not port the coordinators, re-express the data contracts. The XML is the symptom; the triggering model is the subject.