GitLab Helm chart 10.0 upgrade notes
With the 19.0 release of GitLab, we have released version 10.0 of the GitLab Helm chart.
This document highlights the changes with the highest impact. For a comprehensive overview, see:
Summary of major changes
- NGINX Ingress is now disabled by default and replaced by Gateway API with Envoy Gateway.
- The bundled PostgreSQL, Redis, and MinIO have been removed from the GitLab chart.
- Support for PostgreSQL 16 has been removed. Make sure you are running PostgreSQL 17 before upgrading.
- Resource Owner Password Credentials (ROPC) OAuth grant removed.
For mitigation steps and full details, follow the linked deprecation notices.
Upgrade to 10.6.0
Custom certificates images must provide their scripts
The chart now runs these scripts as the container command, instead of the image’s default command:
- The
certificatesinit container runs/scripts/bundle-certificatesfromglobal.certificates.image. - The
cfssl-self-signcontainer of the self-signed certificates Job runs/scripts/generate-certificatesfromshared-secrets.selfsign.image.
The Cloud Native GitLab images already provide these scripts. If you replace either image with your own, make sure the image provides the script at that path. Otherwise, the Pods that use the image fail to start. For the requirements of the certificates image, see the certificates image contract.
Praefect Service no longer has Prometheus scrape annotations
The Praefect Service no longer has the gitlab.com/prometheus_* and prometheus.io/* scrape
annotations. With the default values, their port annotations were empty. A Prometheus that read
them scraped every port of the Service, including the Praefect gRPC port. The Praefect Pods keep
their scrape annotations, like Gitaly and the other GitLab components, and the Praefect
ServiceMonitor is unchanged.
If your Prometheus discovers scrape targets only from Service annotations, it no longer scrapes
Praefect. Configure it to discover Pods by their annotations or, if you use the Prometheus
Operator, set gitlab.praefect.metrics.serviceMonitor.enabled to true. With the bundled
Prometheus, the kubernetes-pods job still scrapes Praefect, but the kubernetes-service-endpoints
job no longer does. Update any query that selects Praefect metrics with
job="kubernetes-service-endpoints". For the metrics settings, see
Praefect installation command line options.
Praefect Service port renamed to tcp-praefect
The Praefect Service now names its gRPC port tcp-praefect instead of grpc-praefect.
The TLS port keeps its tls- prefix.
If you run Istio sidecars, you can remove the workarounds for the old name,
such as excluding the Praefect port from the sidecar
or patching the Service port name to tcp-praefect.
For the port naming, see Gitaly and Praefect.
Registry configuration rendered without a shell
The registry configure init container now renders config.yml with gomplate instead of
a shell script, so passwords and other secrets can contain any character. Unless you set
registry.init.script:
- The init container image must provide
gomplateon itsPATH. This affects you only if you setregistry.init.imageorglobal.gitlabBase.imageto an image without it. {% %}in registry chart values is no longer evaluated, and is written to the registry configuration unchanged.- The
registry-server-configvolume holds onlyconfig.yml. Extra volume mounts or containers that read other files from it must mount the Secrets instead. - A
delete.enabledvalue in thestorageSecret is now always honored. The script addeddelete.enabled: trueunless the line afterdelete:was an indentedenabled:, sodelete: {enabled: false}, or a comment or blank line beforeenabled: false, left deletes enabled. If you rely on deletes, setdelete.enabled: trueor remove thedeletekey.
For details, see
Configuring the configure init container.
preStop hooks run without a shell
The preStop hooks no longer run /bin/bash:
- The
gitlab-workhorsecontainer and theregistrycontainer use the Kubernetessleepaction to wait before they stop. Thegitlab-workhorsehook is rendered only whengitlab.webservice.workhorse.healthcheckListener.enabledisfalse, the default. - The
webservicecontainer sendsSIGINTto its PID 1 withruby. In the Cloud Native GitLab images, PID 1 istini, which forwards the signal to Puma, or Puma itself whenUSE_TINIis0. If you replace the Webservice image, PID 1 must be Puma or must forwardSIGINTto it. Otherwise, Puma skips its shutdown blackout period. - The
sidekiq,mailroom,gitlab-exporter, andgeo-logcursorcontainers no longer have apreStophook. Kubernetes sends themSIGTERMdirectly.
The sleep action requires Kubernetes 1.30 or later, where the PodLifecycleSleepAction
feature gate is enabled by default. The feature is generally available in Kubernetes 1.34.
On older clusters, the Webservice and registry Deployments fail validation, and
helm upgrade fails.
This change impacts only clusters older than the supported Kubernetes releases. Upgrade your cluster to a supported release before you upgrade the chart.
configure init containers removed
Except for the registry, the GitLab Pods no longer run a configure init container. Their
containers read the Secrets directly.
- Except for the registry chart, the charts no longer have an
init.imagesetting. Helm ignores the value if you still set it. Without theirconfigureinit containers, these Pods no longer pull theglobal.gitlabBase.imageimage. - The Pods still need an
fsGroup, as the defaultsecurityContextsets. Without it, the main container now fails to read the secret files, instead of theconfigureinit container. - Restart the Pods after you change the Secrets.
- With
gitlab.gitaly.cgroups.enabled, the defaultgitlab.gitaly.cgroups.mountpointnow reads/etc/gitaly-cgroup/gitaly-pod-cgroup. If you setmountpointto read the previous path,/etc/gitlab-secrets/gitaly-pod-cgroup, the chart rewrites it to the new path. - GitLab Shell reads the SSH host keys from
/etc/gitlab-ssh-host-keys. With OpenSSH, the chart sets aHostKeydirective for the RSA, ECDSA, and Ed25519 keys. At startup,sshdlogs an error for each of these keys that your host keys Secret doesn’t contain.HostKeydirectives inopensshd.supplemental_configreplace the directives of the chart. The chart points the directives for the keys of the host keys Secret at/etc/gitlab-ssh-host-keys.HostKeydirectives in asshd_config.dsnippet are added to the directives of the chart. Point them at/etc/gitlab-ssh-host-keys, because OpenSSH rejects the copies of the keys in/etc/ssh. To limit the key types thatsshdoffers, setHostKeyAlgorithms.
Upgrade to 10.4.0
KAS advertises native gRPC to agents by default
The address in gitlab_kas.external_url, shown in the agent installation instructions, now
defaults to grpcs://kas.example.com instead of wss://kas.example.com when the chart routes
gRPC to KAS: with Gateway API and the chart-managed Envoy Gateway policies (the default), or
with the NGINX Ingress provider, for which the KAS gRPC Ingress is now rendered by default. Other Ingress controllers and installations with a
relative URL root keep wss://. Agents already connected over wss:// are not affected.
To keep advertising WebSocket, set global.appConfig.gitlab_kas.externalUrl. To skip the gRPC
Ingress, set global.kas.ingress.grpc.enabled: false. For details, see
agent connection protocol.
Upgrade to 10.3
Envoy Gateway upgrade requires new CRDs
The bundled Envoy Gateway chart is upgraded from 1.8.3 to 1.9.1. Helm does not upgrade CRDs, so
apply the Envoy Gateway CRDs for 1.9.1 and the required version 1.6 Gateway API CRDs
before you upgrade GitLab. Otherwise the upgrade fails on the GitLab Shell TCPRoute,
and the API server prunes the new Host header setting on the Webservice listeners. For the upgrade steps, see
Upgrade the Envoy Gateway CRDs.