Unattended upgrades
Status: Implemented Issue: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/work_items/2237 Parent: GitLabCore reconciler
Goal
An administrator keeps a GitLab instance on a supported version without editing it for every release. The Operator follows the version the administrator chose and upgrades without downtime.
Requirements
- A
GitLabCoreselects its version with one type:Static(a named version),Patch(a minor), orLatest,Previous,OldestSupported(a major). A type with the wrong field is rejected. - A
GitLabCorewithout a version followsLatest.Latest,Previous, andOldestSupportedwithout a major install within the major their count lands on across every released major, and then follow the major the instance runs. - An instance on any type but
Staticupgrades when a matching version is released, with no change to its spec. A major upgrade happens only when the administrator sets or raises the major. - The status reports the version the instance runs, the version it upgrades to, and the version its type selects now. The Operator never writes a resolved version into the spec.
PreviousandOldestSupportedcount minors down from the newest release across majors, and stay within their major. The first release of the next major counts too.- An upgrade in progress keeps its target, unless it is on a
Staticversion. A version released meanwhile is reported, and the instance upgrades to it once the running upgrade completes. - A version below the running one is refused and reported, and the instance keeps running.
- An unattended upgrade starts only after its version renders and the cluster accepts its objects. A version that fails is reported, and the instance keeps running and repaired on its current one.
- A failed unattended upgrade moves on to a newer patch of the same minor, and no further. A failed
Staticupgrade waits for a change of its version. - Each chart version an upgrade renders runs its own chart hooks.
- An instance whose status lost the version it runs, while the workloads of its release still run,
is not reconciled on any type but
Static, and the status says why. AStaticversion takes the instance back. - A chart repository that cannot be reached for a while neither sets
UpgradeabletoFalsenor fails an instance that is installed. - The web UI shows the version type, the version waiting to roll out, and a failed check with the version that failed it.
Out of scope
- Checking that the images of a version exist.
- Unattended major upgrades, downgrades, and rollbacks.
- Maintenance windows, or any schedule for when an upgrade starts.
- The zero-downtime upgrade itself, which unattended upgrades reuse.
FAQ
- Why does an instance without a version follow
Latest? It keeps the instance on a supported version with nothing to fill in, and the version an administrator would pick without a reason to pick another. It stays in the major it installed, because a major upgrade is never unattended. - Which major does a type without one install? The major of the minor it counts down to. With
11.0 the newest release and major 10 ending at 10.11,
Latestinstalls 11.0,Previous10.11, andOldestSupported10.10. Each then stays within its major, as it would with that major set. - What happens when the status of an instance is lost, for example after a Velero restore, which
skips the status by default? Without
status.version, a type would count as if nothing were installed. Without a major,Latestcould render a new major over the running one as an install, with no stepped upgrade. So while the workloads of its release run and the status names neither a deployed nor a target version, the Operator stops and reportsDeployedVersionUnknown. AStaticchartVersionof the running chart version re-applies it and records it, and the type can then be set back. The Operator does not guess the version from the labels of the workloads. - Does that cover a resource that a GitOps tool deletes and creates again? No. Deleting the
resource removes the workloads of its release, so the new resource installs anew, against the
database the old one left. A major, or a
Staticversion, keeps that install to the major the database holds. - Does the Operator write the resolved version into the spec, so that a change of it starts a reconcile? No. It would fight GitOps tools that own the spec. The resolved version lives in the status only.
- Which checks gate an unattended upgrade? Everything that needs no access to the images.
ADR 28 lists them. A version that fails them
is checked again when a newer version appears, when the spec changes, when the ConfigMaps and
Secrets
spec.chart.valuesFromreferences change, when the Operator carries other GitLab charts, or after an hour. - What happens when an upgrade fails after it started, in its migrations or its new pods?
The target is held. A migrations Job that retries a failed attempt has not failed: the upgrade
waits for it and reports the retry. After the Job reports
Failed, the instance moves on to a newer patch of the same minor when one is released, because a fix ships as the next patch. It moves no further. For the details, see Migrations Job failures. - Is a spec change made while an upgrade is in progress dropped? No. A change to a
Staticversion applies at once. Any other change is reported, and takes effect once the instance settles. - What does
OldestSupported(N-2) of major 10 run once major 11 ships, if major 10 ends at 10.11? 10.10 while 11.0 is the newest release, and 10.11 from 11.1 on.Previous(N-1) runs 10.11 from 11.0 on. These are the minors GitLab patches: 18.0.1 shipped with 17.11.3 and 17.10.7. The count follows the GitLab maintenance policy. A major with too few minors resolves to its first, soOldestSupportedof 11 is 11.0 while only 11.0 exists. - Does an instance of major 10 leave 10.11 once GitLab stops patching it? No. A major upgrade is never unattended: raise the major to start one.
- Do the chart hooks run on an unattended upgrade? Yes, for each chart version the upgrade renders. A newer chart can generate a secret the earlier one did not.
- Should a temporary failure to reach the chart repository set
UpgradeabletoFalse? No. Whoever alerts on the condition would be paged for a network blip. The Operator tries again a few minutes later. It still reports a version it could not check yet, and a chart it could not pull, because those describe a pending upgrade rather than the health of the instance.