Upgrading the Geo sites

  • Tier: Premium, Ultimate
  • Offering: GitLab Self-Managed

Read these sections carefully before updating your Geo sites. Not following version-specific upgrade steps may result in unexpected downtime. If you have any specific questions, contact Support. A database major version upgrade requires re-initializing the PostgreSQL replication to Geo secondaries. This applies to both Linux-packaged and externally-managed databases. This may result in a larger than expected downtime.

Upgrading Geo sites involves performing:

  1. Version-specific upgrade steps, depending on the version being upgraded to or from:
  2. General upgrade steps, for all upgrades.

You cannot upgrade the operating system of a secondary site and then promote it as a way to migrate between operating systems. The secondary site might miss replication of some Git repositories or files. For more information, see Streaming replication.

General upgrade steps

These general upgrade steps require downtime in a multi-node setup. If you want to avoid downtime, consider using zero-downtime upgrades.

To upgrade the Geo sites when a new GitLab version is released, upgrade primary and all secondary sites:

  1. Optional. Pause replication on each secondary site to protect the disaster recovery (DR) capability of the secondary sites. Pause replication when your priority is preserving a clean DR checkpoint during a higher-risk upgrade window. Do not pause replication if your priority is keeping the secondary current and serving read traffic normally during the upgrade, especially in a zero-downtime approach.

  2. SSH into each node of the primary site.

  3. Upgrade GitLab on the primary site.

  4. Perform testing on the primary site, particularly if you paused replication in step 1 to protect DR. For more information about post-upgrade testing, see run upgrade health checks.

  5. Ensure that the secrets in the /etc/gitlab/gitlab-secrets.json file of both the primary site and the secondary site are the same. The file must be the same on all of a site’s nodes.

  6. SSH into each node of secondary sites.

  7. Upgrade GitLab on each secondary site.

  8. If you paused replication in step 1, resume replication on each secondary. Then, restart Puma and Sidekiq on each secondary site. This is to ensure they are initialized against the newer database schema that is now replicated from the previously upgraded primary site.

    shell
    sudo gitlab-ctl restart sidekiq
    sudo gitlab-ctl restart puma
  9. Test primary and secondary sites, and check version in each.

Check status after upgrading

Now that the upgrade process is complete, you may want to check whether everything is working correctly:

  1. Run the Geo Rake task on an application node for the primary and secondary sites. Everything should be green:

    shell
    sudo gitlab-rake gitlab:geo:check
  2. Check the primary site’s Geo dashboard for any errors.

  3. Test the data replication by pushing code to the primary site and see if it is received by secondary sites.

If you encounter any issues, see the Geo troubleshooting guide.

Operating system upgrades

Upgrading the operating system of a Geo site is a separate procedure from upgrading GitLab, and carries a different risk. A change to glibc locale data can invalidate PostgreSQL indexes, and a secondary site can report full replication while some data is not safely replicated. Pausing and resuming replication during an OS upgrade does not mitigate this risk.

Before you upgrade the operating system on any Geo site:

  1. Check whether locale data changes at all on your current and target operating systems. If PostgreSQL on both systems runs on glibc 2.28 or higher, your indexes should continue to work. To check this before you upgrade, verify the collation versions. If the versions don’t match, follow one of the upgrade procedures instead.
  2. If your database is externally managed, see Geo sites with externally managed databases. Otherwise follow the Geo procedure for your chosen approach in Upgrading operating systems for PostgreSQL.