GitLab Helm chart 10.0 upgrade notes

With the 19.0 release of GitLab, we have released version 10.0 of the GitLab Helm chart.

This document highlights the changes with the highest impact. For a comprehensive overview, see:

Summary of major changes

For mitigation steps and full details, follow the linked deprecation notices.

Upgrade to 10.6.0

Custom certificates images must provide their scripts

The chart now runs these scripts as the container command, instead of the image’s default command:

  • The certificates init container runs /scripts/bundle-certificates from global.certificates.image.
  • The cfssl-self-sign container of the self-signed certificates Job runs /scripts/generate-certificates from shared-secrets.selfsign.image.

The Cloud Native GitLab images already provide these scripts. If you replace either image with your own, make sure the image provides the script at that path. Otherwise, the Pods that use the image fail to start. For the requirements of the certificates image, see the certificates image contract.

Praefect Service no longer has Prometheus scrape annotations

The Praefect Service no longer has the gitlab.com/prometheus_* and prometheus.io/* scrape annotations. With the default values, their port annotations were empty. A Prometheus that read them scraped every port of the Service, including the Praefect gRPC port. The Praefect Pods keep their scrape annotations, like Gitaly and the other GitLab components, and the Praefect ServiceMonitor is unchanged.

If your Prometheus discovers scrape targets only from Service annotations, it no longer scrapes Praefect. Configure it to discover Pods by their annotations or, if you use the Prometheus Operator, set gitlab.praefect.metrics.serviceMonitor.enabled to true. With the bundled Prometheus, the kubernetes-pods job still scrapes Praefect, but the kubernetes-service-endpoints job no longer does. Update any query that selects Praefect metrics with job="kubernetes-service-endpoints". For the metrics settings, see Praefect installation command line options.

Praefect Service port renamed to tcp-praefect

The Praefect Service now names its gRPC port tcp-praefect instead of grpc-praefect. The TLS port keeps its tls- prefix. If you run Istio sidecars, you can remove the workarounds for the old name, such as excluding the Praefect port from the sidecar or patching the Service port name to tcp-praefect. For the port naming, see Gitaly and Praefect.

Registry configuration rendered without a shell

The registry configure init container now renders config.yml with gomplate instead of a shell script, so passwords and other secrets can contain any character. Unless you set registry.init.script:

  • The init container image must provide gomplate on its PATH. This affects you only if you set registry.init.image or global.gitlabBase.image to an image without it.
  • {% %} in registry chart values is no longer evaluated, and is written to the registry configuration unchanged.
  • The registry-server-config volume holds only config.yml. Extra volume mounts or containers that read other files from it must mount the Secrets instead.
  • A delete.enabled value in the storage Secret is now always honored. The script added delete.enabled: true unless the line after delete: was an indented enabled:, so delete: {enabled: false}, or a comment or blank line before enabled: false, left deletes enabled. If you rely on deletes, set delete.enabled: true or remove the delete key.

For details, see Configuring the configure init container.

preStop hooks run without a shell

The preStop hooks no longer run /bin/bash:

  • The gitlab-workhorse container and the registry container use the Kubernetes sleep action to wait before they stop. The gitlab-workhorse hook is rendered only when gitlab.webservice.workhorse.healthcheckListener.enabled is false, the default.
  • The webservice container sends SIGINT to its PID 1 with ruby. In the Cloud Native GitLab images, PID 1 is tini, which forwards the signal to Puma, or Puma itself when USE_TINI is 0. If you replace the Webservice image, PID 1 must be Puma or must forward SIGINT to it. Otherwise, Puma skips its shutdown blackout period.
  • The sidekiq, mailroom, gitlab-exporter, and geo-logcursor containers no longer have a preStop hook. Kubernetes sends them SIGTERM directly.

The sleep action requires Kubernetes 1.30 or later, where the PodLifecycleSleepAction feature gate is enabled by default. The feature is generally available in Kubernetes 1.34. On older clusters, the Webservice and registry Deployments fail validation, and helm upgrade fails.

This change impacts only clusters older than the supported Kubernetes releases. Upgrade your cluster to a supported release before you upgrade the chart.

configure init containers removed

Except for the registry, the GitLab Pods no longer run a configure init container. Their containers read the Secrets directly.

  • Except for the registry chart, the charts no longer have an init.image setting. Helm ignores the value if you still set it. Without their configure init containers, these Pods no longer pull the global.gitlabBase.image image.
  • The Pods still need an fsGroup, as the default securityContext sets. Without it, the main container now fails to read the secret files, instead of the configure init container.
  • Restart the Pods after you change the Secrets.
  • With gitlab.gitaly.cgroups.enabled, the default gitlab.gitaly.cgroups.mountpoint now reads /etc/gitaly-cgroup/gitaly-pod-cgroup. If you set mountpoint to read the previous path, /etc/gitlab-secrets/gitaly-pod-cgroup, the chart rewrites it to the new path.
  • GitLab Shell reads the SSH host keys from /etc/gitlab-ssh-host-keys. With OpenSSH, the chart sets a HostKey directive for the RSA, ECDSA, and Ed25519 keys. At startup, sshd logs an error for each of these keys that your host keys Secret doesn’t contain.
    • HostKey directives in opensshd.supplemental_config replace the directives of the chart. The chart points the directives for the keys of the host keys Secret at /etc/gitlab-ssh-host-keys.
    • HostKey directives in a sshd_config.d snippet are added to the directives of the chart. Point them at /etc/gitlab-ssh-host-keys, because OpenSSH rejects the copies of the keys in /etc/ssh. To limit the key types that sshd offers, set HostKeyAlgorithms.

Upgrade to 10.4.0

KAS advertises native gRPC to agents by default

The address in gitlab_kas.external_url, shown in the agent installation instructions, now defaults to grpcs://kas.example.com instead of wss://kas.example.com when the chart routes gRPC to KAS: with Gateway API and the chart-managed Envoy Gateway policies (the default), or with the NGINX Ingress provider, for which the KAS gRPC Ingress is now rendered by default. Other Ingress controllers and installations with a relative URL root keep wss://. Agents already connected over wss:// are not affected. To keep advertising WebSocket, set global.appConfig.gitlab_kas.externalUrl. To skip the gRPC Ingress, set global.kas.ingress.grpc.enabled: false. For details, see agent connection protocol.

Upgrade to 10.3

Envoy Gateway upgrade requires new CRDs

The bundled Envoy Gateway chart is upgraded from 1.8.3 to 1.9.1. Helm does not upgrade CRDs, so apply the Envoy Gateway CRDs for 1.9.1 and the required version 1.6 Gateway API CRDs before you upgrade GitLab. Otherwise the upgrade fails on the GitLab Shell TCPRoute, and the API server prunes the new Host header setting on the Webservice listeners. For the upgrade steps, see Upgrade the Envoy Gateway CRDs.