The GitLabSecrets reconciler

internal/controller/gitlabsecrets reconciles the apps.gitlab.com/v2alpha1 GitLabSecrets resource into the Secrets the GitLab chart would otherwise generate with its shared-secrets hook Job. For the requirements, see the GitLab secrets spec. For the design, see ADR 35.

The reconciler creates the Secrets of spec.secrets and the CA, TLS, and chain Secrets of spec.certificates. This page also records the behavior of the Job it matches.

The definition

api/v2alpha1/gitlabsecrets_types.go declares the resource. task manifests regenerates config/crd/bases/apps.gitlab.com_gitlabsecrets.yaml, and task generate the deepcopy code. task install_v2alpha1_crds and the end-to-end harness apply the definition with the other v2alpha1 definitions.

Server-side apply rejects a field the definition does not declare, as does a create or update with fieldValidation=Strict. Examples are spec.policy and authority.expiry, which earlier chart drafts rendered. A request without fieldValidation stores the resource without the field and returns a warning.

internal/controller/gitlabsecrets/definition_test.go creates each chart render in internal/controller/gitlabsecrets/testdata/chart/ on envtest. It also checks the rejected values and the bounds. task slow-unit-tests runs it. To run it on its own:

shell 
KUBEBUILDER_ASSETS=$(./setup-envtest use -p path) go test ./internal/controller/gitlabsecrets/ -run TestDefinition

To regenerate the fixtures from a chart checkout, run the script, then the test:

shell 
internal/controller/gitlabsecrets/testdata/chart/render.sh <chart-dir>

The script renders each values/<case>.yaml with values/base.yaml. To add a case, add its values file, and add what it covers to fixtureCovers in the test. The README.md next to the fixtures records the chart commit they come from.

The reconciler

The reconciler runs only in a binary built with the bridge tag, with ENABLE_BRIDGE=true, like the other v2alpha1 reconcilers. task manifests generates its RBAC from the RBAC markers into config/rbac/v2alpha1/, which task install_v2alpha1_crds applies.

A pass groups spec.secrets by name, reads each Secret, and hands both to secretgen.Plan with the readyUnits of its status entry. It then writes what Plan returns:

  • A missing Secret is created with its declared type, the labels, and the generated keys. A missing Secret with a report is not created.
  • An existing Secret gets a JSON merge patch with the added keys, the labels that differ, and the resourceVersion the pass read. The API server refuses the patch with a conflict when the Secret changed after the pass read it.
  • No Secret is updated, deleted, or applied server-side, and none gets an owner reference. The GitLabSecrets gets no finalizer, so deleting it leaves every Secret. A GitLabSecrets that is being deleted gets no writes.

After spec.secrets, a pass reads the three Secrets of each certificate set and hands them to secretgen.PlanCertificates with the readyUnits of the TLS Secret. It writes them the same way, in the order CA, TLS, chain, and stops at the first write that fails. A Secret of the set it did not write then reports that an earlier write failed. A set that was not written whole is not ready, and its TLS Secret keeps the readyUnits it had.

The labels are those of the GitLabSecrets without the operator.gitlab.com/release-* labels. A label the Secret lacks or holds with another value is set, and no label is removed. After the cache holds two GitLabSecrets of the namespace that declare a Secret, neither labels it, not even when it creates the Secret. A Secret created or labeled while only one declared it keeps those labels.

Reads come from the cache. Before a pass writes a Secret or reports it, it reads the Secret again past the cache and plans again if it changed. A certificate set is read again whole, and planned again when any of its Secrets changed. Before its first write, and before it skips a status update that would change nothing, it checks that the cached GitLabSecrets is current, and otherwise requeues after a second. A new certificate set is issued only after both checks, because a key can take seconds to generate. A status update that conflicts requeues the same way.

status.secrets[] holds name, ready, and readyUnits for each Secret of spec.secrets, then for the CA, TLS, and chain Secrets of each certificate set. A Secret of a set is ready when the set is, and only the TLS Secret holds the unit certificate-set. A Secret whose read, plan, or write failed is not ready. It keeps the readyUnits it had, minus the units no longer declared, as on any pass that is not ready. The Ready condition carries the metadata.generation of the pass in observedGeneration:

Status Reason When
True SecretsReady Every Secret holds what it declares.
False SecretsReported Plan or PlanCertificates reported a unit or a Secret. The message names each, up to 4096 bytes.
False SecretsNotWritten A Secret could not be read, planned, or written. The message names it and the error, then the reports.

A write the API server refuses as invalid (422) or too large (413) is not retried. A change to the Secret or the GitLabSecrets starts the next pass. Any other failure is retried with the backoff of the controller. A pass that succeeds does not requeue.

The reconciler emits these events on the GitLabSecrets:

Type Reason When
Normal SecretTypeKept An existing Secret has another type than the declared one. It keeps it and stays ready.
Warning SecretDeclaredTwice Another GitLabSecrets of the namespace declares the Secret. It keeps its labels.

Both events repeat on every pass that finds their cause and writes the Secret or needs no write. The events library aggregates the repeats into a series.

It watches the GitLabSecrets for a change of its generation or its labels, and every Secret. A field index on the declared Secret names, the Secrets of spec.certificates included, maps a changed or deleted Secret to the GitLabSecrets that declare it.

Test the reconciler

  • The unit tests run with go test ./internal/controller/gitlabsecrets/, on a fake client that fails on any update, delete, or server-side apply of a Secret.

  • TestController runs the reconciler through a manager on envtest, with a client that refuses the same writes. task slow-unit-tests runs it. To run it on its own:

    shell 
    KUBEBUILDER_ASSETS=$(./setup-envtest use -p path) go test ./internal/controller/gitlabsecrets/ -run TestController

  • The black-box suite gitlabsecrets deploys the bridge image. It applies the defaults.yaml render to a namespace with no ServiceAccount or RBAC of its own. It checks the Secrets, the certificate set, a deleted key, a deleted chain, and the deletion of the GitLabSecrets. To run it on a disposable cluster:

    shell 
    CONTAINER_CLI=docker task docker-build-bridge
E2E_CLUSTER_PROVIDER=k3s task e2e-suite SUITE=gitlabsecrets

Generation

internal/secretgen decides what to add to one Secret, with no Kubernetes client. Plan takes the entries of spec.secrets that name the Secret, the Secret if it exists, and its readyUnits. It returns the keys to add, the reports, the declared data-bearing units, the next readyUnits, and a note when the existing type differs from the declared one. A Secret that declares more data-bearing units than status holds is reported, and nothing is added to it. So is a Secret whose values would exceed the 1 MiB the API server stores. The package comment shows how a controller uses it.

PlanCertificates plans the CA, TLS, and chain Secrets of an entry of spec.certificates together, as the one unit certificate-set, and returns a result per Secret in the order CA, TLS, chain:

  • With no key of the set present, in missing or existing Secrets, and the unit never ready, it issues a CA and a wildcard certificate: the CA certificate under caKey, tls.crt and tls.key, and the chain under chainKey. The CA key is not stored. It generates the keys only with Generate. Without it, it returns Issue and adds nothing.
  • With the CA certificate, tls.crt, and tls.key present and the chain missing, it derives the chain: the CA certificate, then tls.crt, byte for byte, also after the set was ready. An empty CA certificate or tls.crt is kept, and the chain is reported instead.
  • Any other partial set is reported on the TLS Secret, and so is a ready set that is missing. Nothing of a reported set is added, so none of its missing Secrets is created.
  • An existing set is never reissued, renewed, or checked, an expired or Ed25519 one included.
  • A Secret of the set is ready when the whole set is. An existing Secret of another type keeps it. An immutable Secret that lacks a key, and a Secret that would exceed 1 MiB, are reported.

The tests compare every generated value with the shape of the Job’s output. The fixture is in internal/secretgen/testdata/job/:

  • values.yaml: chart values that render every generator type and option.
  • gitlabsecrets.yaml: the GitLabSecrets that the chart renders for them with shared-secrets.provider: controller.
  • shapes.json: the shapes of the Secrets the Job created for them, recorded by hack/secretshapes, with the chart commits, the Job image, and the date. It holds no secret material.

The certificate sets are compared with cfssl-self-sign, the image of the chart’s selfsign Job. The fixture is in internal/secretgen/testdata/cfssl/shapes.json: the shapes of the three Secrets the Job creates from the files of one run per algorithm and size, ECDSA 256, 384, and 521, and RSA 2048, 3072, 4096, and 8192, and of three ECDSA 256 runs for an IP address, an e-mail address, and a URI as the domain. It records the image digest, the chart commits, the environment, and the date. Besides each certificate, a set records facts that differ on every issue or tie the Secrets together: the serial size, a notBefore on the minute, the issuer and signature of tls.crt, its key, and the chain. The test issues each set for the certificate set of testdata/job/gitlabsecrets.yaml with no accepted difference.

The shape test of spec.secrets accepts only these differences from the recorded shapes:

  • An SSH host key carries no comment, in its .pub or in its private key.
  • secrets.yml ends with a newline.

The generated values also differ from the Job’s in ways a single recording does not show:

  • A random value always has length characters.
  • A generated secrets.yml value that YAML reads as another type, such as all digits, is quoted. The Job writes it bare, so Rails reads a number. An existing bare value stays bare.
  • A rewritten secrets.yml is indented by 2, drops --- and ..., joins the lines of folded scalars, and tags merge keys as !!merge <<:. Every value is kept, and a document of comments only keeps its comments.

Behavior of the Job

The values in the field came from the Job, so the Job is the reference. These results are from 2026-09-30, in the Job’s image cng/kubectl:v19.4.1 and in the arm64 cng/cfssl-self-sign:v19.2.1, whose script matches the CNG default branch. The SAN, serial, key ID, and time of issue rows of the certificates are from 2026-10-02, in the arm64 cng/cfssl-self-sign:master that the chart renders by default, with cfssl dev built with Go 1.26.8.

Subject Behavior
OpenSSL 3.5.7. openssl genrsa and openssl req -keyout write PRIVATE KEY (PKCS#8).
x509 openssl req -x509 writes X.509 v3, sha256WithRSAEncryption, subject and issuer CN=<commonName>, CA:TRUE (critical), and subject and authority key IDs.
sshHostKeys OpenSSH 10.0p2. ssh-keygen -A makes RSA 3072, ECDSA P-256, and Ed25519, as ssh_host_<type>_key and .pub. Private keys are OPENSSH PRIVATE KEY, with no cipher and no KDF, in lines of 70 characters. Public and private keys carry a <user>@<pod> comment. The Job mounts an emptyDir on /etc/ssh, because the image user cannot write there.
random gen_random filters 4096 random bytes with tr, so a narrow character set can return fewer than length characters.
encoding: base64 In chart 10.4.1, stores standard base64 with no wrapping and no trailing newline up to a length of 57, the same as base64-nowrap. Every base64 entry is 32. From 58, the create path fails and the patch path truncates.
wrap: jsonArray Encodes first, then wraps.
Existing Secret Adds only a missing --from-literal key. Never adds a key of rsa, x509, sshHostKeys, or raw bytes, and never changes an existing host-key Secret.
secrets.yml Rewritten on every run, with the declared fields of one environment only. Regenerates an absent, null, or empty-string field, and keeps an empty list. Indents by 2, with a PEM in a | block and list items indented by 4. Has no trailing newline. A rerun writes the same bytes.
Labels Overwrites gitlab.standardLabels and gitlab.commonLabels. Removes app.kubernetes.io/name when global.application.create is false.
Certificate set Every run issues a new CA and wildcard certificate, then creates each of the three Secrets with || true. A run with one Secret missing fills it from the new CA, which the other two do not match.
Certificate structure CA: subject and issuer O=<namespace>, OU=<release>, CN=<caSubject>, CA:TRUE (critical), key usage certificate sign and CRL sign (critical), no authority key ID. Wildcard: subject CN=<domain>, SANs <domain> and *.<domain>, CA:FALSE (critical), key usage digital signature, key encipherment, and certificate sign (critical), also for ECDSA, extended key usage server authentication, and the subject key ID of the CA as its authority key ID. Each of <domain> and *.<domain> is an IP SAN when it is an IP address, else an e-mail SAN when it is an e-mail address, else a URI SAN when url.ParseRequestURI takes it, such as urn:example, /path, or host:8443, else a DNS SAN. Both: X.509 v3, PrintableString names, a serial of 20 random bytes with the top bit clear, and a SHA-1 subject key ID. The chain is ca.pem, then wildcard.pem.
Certificate keys RSA PRIVATE KEY (PKCS#1) or EC PRIVATE KEY (SEC 1).
Certificate sizes RSA from 2048 to 8192, odd sizes included. ECDSA 256, 384, and 521. RSA 1024 fails with RSA key is too weak, 8194 with RSA key size too large, and ECDSA 224 with invalid curve. ed25519 depends on the CNG version, as described below.
Certificate signature RSA from 4096 SHA-512, from 3072 SHA-384, below that SHA-256. ECDSA P-256 SHA-256, P-384 SHA-384, P-521 SHA-512.
Certificate validity CA 43800 hours, wildcard 8760 hours, from the time of issue rounded to the minute, minus 5 minutes: a run at 04:11:06 starts at 04:06:00, one at 04:13:40 at 04:09:00. The script reads EXPIRE while the Job sets EXPIRY, so shared-secrets.selfsign.expiry never applies.

ed25519 follows the cfssl version, not the architecture. cng/cfssl-self-sign before v19.4.0 ships cfssl 1.6.1, which fails ed25519 with invalid algorithm, so charts 10.2 and 10.3 fail it. From v19.4.0, both architectures ship cfssl 1.6.5, which the arm64 image builds from source and reports as dev, so chart 10.4 issues Ed25519 certificates.

Those keys are RFC 8410 PKCS#8, byte-identical to Go’s x509.MarshalPKCS8PrivateKey, under the nonstandard label Ed25519 PRIVATE KEY instead of the PRIVATE KEY of RFC 7468 (csr/csr.go:263-272 in cfssl 1.6.5, unchanged in 1.7.0). NGINX with OpenSSL 3.5.7, Envoy with BoringSSL, and the Envoy Gateway validation reject the label, and load the key once it is relabeled. Go accepts both. Chrome and Firefox do not support an Ed25519 TLS server certificate, whatever the label, which is why the Operator refuses ed25519 certificates.

Reproduce the checks

Print headers and sizes only. The keys are throwaway, but do not paste them anywhere.

  1. Check OpenSSL and the SSH host keys in the Job’s image:

    shell 
    docker run --rm --user 0 --entrypoint /bin/bash \
  registry.gitlab.com/gitlab-org/build/cng/kubectl:v19.4.1 -c '
    openssl version
    openssl genrsa 2048 2>/dev/null | head -1
    ssh-keygen -A
    for pub in /etc/ssh/ssh_host_*.pub; do ssh-keygen -lf "$pub" | cut -d" " -f1,4; done
    head -1 /etc/ssh/ssh_host_rsa_key'

  2. Issue a certificate set for each algorithm and size, then print the signature algorithm, the CA row, and the validity:

    shell 
    for spec in rsa-1024 rsa-2048 rsa-3072 rsa-4096 rsa-8192 ecdsa-256 ecdsa-384 ecdsa-521 ed25519-0; do
  mkdir -m 777 -p "out/$spec"
  docker run --rm -v "$PWD/out/$spec:/output" -e ALGORITHM="${spec%-*}" -e KEY_SIZE="${spec#*-}" \
    -e CA_SUBJECT="GitLab Helm Chart" -e CERT_DOMAIN=example.com -e EXPIRY=3650d \
    registry.gitlab.com/gitlab-org/build/cng/cfssl-self-sign:v19.2.1 >/dev/null 2>&1
  if [ -f "out/$spec/ca.pem" ]; then
    echo "$spec: $(openssl x509 -in "out/$spec/wildcard.pem" -noout -text | grep -m1 'Signature Algorithm')"
  else
    echo "$spec: issues nothing"
  fi
done
openssl x509 -in out/rsa-4096/ca.pem -noout -subject -issuer -dates
openssl x509 -in out/rsa-4096/ca.pem -noout -text | grep -A1 -E 'Constraints|Key Usage'
openssl x509 -in out/rsa-4096/wildcard.pem -noout -dates
rm -rf out

  3. Record the shape fixture of internal/secretgen. Copy the chart checkout that the fixture names, and run these commands from its root. Set OPERATOR to the root of this repository, and make a throwaway cluster the current context:

    shell 
    fixture="$OPERATOR/internal/secretgen/testdata/job"
work=$(mktemp -d)
helm dependency build
helm template gl . -n sg-fixture -f "$fixture/values.yaml" > "$work/render.yaml"
helm template gl . -n sg-fixture -f "$fixture/values.yaml" --set shared-secrets.provider=controller |
  yq 'select(.kind == "GitLabSecrets")' > "$fixture/gitlabsecrets.yaml"
yq 'select(.metadata.name == "gl-shared-secrets" or
  (.kind == "Job" and (.metadata.name | test("^gl-shared-secrets-[0-9a-f]+$"))))' \
  "$work/render.yaml" > "$work/job.yaml"
kubectl create namespace sg-fixture
kubectl -n sg-fixture apply -f "$work/job.yaml"
kubectl -n sg-fixture wait --for=condition=complete job --all --timeout=300s
kubectl -n sg-fixture get secrets -o json > "$work/secrets.json"
kubectl delete namespace sg-fixture
(cd "$OPERATOR" && go run ./hack/secretshapes -secrets "$work/secrets.json" \
  -manifest "$fixture/gitlabsecrets.yaml" -out "$fixture/shapes.json" \
  -source "chart=<chart version and commits>" \
  -source "render=helm <version>, release gl, namespace sg-fixture, values.yaml" \
  -source "job=<Job image>" -source "cluster=<cluster and platform>" -source "recorded=<date>" \
  -source "reproduce=doc/developer/gitlabsecrets.md, Reproduce the checks")
rm -rf "$work"

    Then run go test ./internal/secretgen/.... A shape difference that is not listed in Generation fails the test.

  4. Record the certificate fixture of internal/secretgen. Run the image the chart renders by default, by digest, with the environment of the selfsign Job for values.yaml, once per algorithm and size, and once on ECDSA 256 for a domain of each other kind of SAN. To repeat the recording, take the image and its platform from source.image in testdata/cfssl/shapes.json. Set OPERATOR to the root of this repository:

    shell 
    fixture="$OPERATOR/internal/secretgen/testdata"
image=$(jq -r '.source.image | split(",")[0]' "$fixture/cfssl/shapes.json")
work=$(mktemp -d)
issue() { # <directory> <domain> <algorithm> <size>
  mkdir -m 777 "$work/$1"
  docker run --rm --platform linux/arm64 -v "$work/$1:/output" -e CA_SUBJECT="GitLab Helm Chart" \
    -e CA_ORG=sg-fixture -e CA_ORG_UNIT=gl -e CERT_SUBJECT="$2" -e CERT_DOMAIN="$2" -e EXPIRY=3650d \
    -e ALGORITHM="$3" -e KEY_SIZE="$4" "$image" >/dev/null
}
for spec in rsa-2048 rsa-3072 rsa-4096 rsa-8192 ecdsa-256 ecdsa-384 ecdsa-521; do
  issue "$spec" example.com "${spec%-*}" "${spec#*-}"
done
issue ecdsa-256-ip 192.0.2.10 ecdsa 256
issue ecdsa-256-email user@example.com ecdsa 256
issue ecdsa-256-uri urn:example:gitlab ecdsa 256
(cd "$OPERATOR" && go run ./hack/secretshapes -certificates "$work" \
  -manifest "$fixture/job/gitlabsecrets.yaml" -out "$fixture/cfssl/shapes.json" \
  -source "chart=<chart version and commits>" -source "image=<image, cfssl version, platform>" \
  -source "env=<the environment>" -source "manifest=../job/gitlabsecrets.yaml" \
  -source "recorded=<date>" -source "reproduce=doc/developer/gitlabsecrets.md, Reproduce the checks")
rm -rf "$work"

    Then run go test ./internal/secretgen/.... Any difference fails the test.