Batched background migrations

Status: Implemented Issue: none Parent: GitLabCore reconciler

Goal

A zero-downtime upgrade steps a GitLabCore through every intermediate minor and gates each step on its migrations. GitLab runs batched background migrations asynchronously, enqueued from the post-deployment migrations, so the migrations Job completing does not mean they have finished. Advancing to the next minor while they run risks data inconsistency and failed migrations. An unattended or multi-minor upgrade has no administrator confirming they are done, so the Operator waits on them itself.

Requirements

  • An upgrade step waits, after its workloads roll out and before it records the version and advances, until the batched background migrations of that version are done. Each minor hop of a multi-minor upgrade gates on its own.
  • A migration counts as pending while it is queued (active or paused) or finalizing. The step advances when none are pending.
  • An upgrade waits on the wait Job the chart renders, or, for a chart too old to render one, on a Job the Operator provides. Either way the instance waits.
  • The wait Job never runs on the generic apply path and is never counted as the migrations Job.
  • The wait tolerates a transient database error, retrying a failed check rather than failing the upgrade, but ends after a run of consecutive failures, so a real misconfiguration surfaces rather than polling forever.
  • A failed wait Job is terminal: the instance reports Failed and does not advance. Recovery is an administrator fixing the cause and retrying.
  • An instance opts out with spec.upgrade.skipBatchedMigrationCheck. The step then does not gate.
  • The web UI shows an upgrade waiting on the batched background migrations, and a failed check.

Out of scope

  • The zero-downtime upgrade choreography, which this gate is one step of.
  • Running the batched background migrations, or changing their batch size or schedule. GitLab runs them.
  • Finalizing a stuck migration, or any remediation of a failed backfill.
  • Waiting on batched background migrations outside an upgrade. A fresh install enqueues none to wait on, so the gate stays off.

FAQ

  • Why wait rather than fail when the migrations are slow? A large backfill takes hours or days. Failing would strand an upgrade that is progressing. The step holds, and the instance keeps serving on the version it reached, until the migrations finish.
  • Why is zero pending the point it is safe to advance? It matches what the GitLab upgrade documentation has an administrator confirm before moving to the next version.
  • When is it safe to skip the check? On a development or test instance, where a brief data inconsistency across a multi-minor hop is acceptable. A production instance leaves it on.
  • What does a multi-minor upgrade do? Each minor hop gates on its own batched background migrations before the next begins, which is the sequence the GitLab upgrade path requires.
  • Where is the design recorded? In ADR 40.