Three-step import

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab Self-Managed

Import your existing container registry metadata. The following procedure is recommended for larger registries (200 GiB or more), or if you want to minimize downtime while completing the import.

Pre-import repositories (step one)

The duration of step one depends mostly on the number of tagged images in your registry, not on the total size of the registry data. In user reports, step one ran at rates from under 1 TB per hour to over 4 TB per hour. Plan with the lower end of this range, and treat any estimate as a lower bound. For example durations, see Plan an import for a large registry.

You may continue to use the registry as normal while step one is being completed.

In GitLab 18.5 and later, if step one is interrupted, you can run it again and skip repositories that were already pre-imported. By default, the import skips repositories pre-imported in the last 72 hours. For more information, see restore interrupted imports.

  1. Ensure the database is disabled in the database section of your /etc/gitlab/gitlab.rb file:

    ruby
    registry['database'] = {
      'enabled' => false, # Must be false!
    }
  2. Save the file and reconfigure GitLab.

  3. Apply database migrations.

  4. Run the first step to begin the import:

    shell
    sudo gitlab-ctl registry-database import --step-one --log-to-stdout
  1. Ensure the database is disabled in the database section of your /etc/gitlab/gitlab.rb file:

    ruby
    registry['database'] = {
      'enabled' => false, # Must be false!
    }
  2. Save the file and reconfigure GitLab.

  3. Apply database migrations.

  4. Run the first step to begin the import:

    shell
    sudo -u registry gitlab-ctl registry-database import --step-one --log-to-stdout

Prerequisites:

  1. Add the database section to your /etc/gitlab/gitlab.rb file, but start with the metadata database disabled:

    ruby
    registry['database'] = {
      'enabled' => false, # Must be false!
      'host' => '<registry_database_host_placeholder_change_me>',
      'port' => 5432, # Default, but set to the port of your database instance if it differs.
      'user' => '<registry_database_username_placeholder_change_me>',
      'password' => '<registry_database_placeholder_change_me>',
      'dbname' => '<registry_database_name_placeholder_change_me>',
      'sslmode' => 'require', # See the PostgreSQL documentation for additional information https://www.postgresql.org/docs/16/libpq-ssl.html.
      'sslcert' => '</path/to/cert.pem>',
      'sslkey' => '</path/to/private.key>',
      'sslrootcert' => '</path/to/ca.pem>'
    }
  2. Save the file and reconfigure GitLab.

  3. Apply database migrations if you have not done so.

  4. Run the first step to begin the import:

    shell
    sudo gitlab-ctl registry-database import --step-one

You should try to schedule the following step as soon as possible to reduce the amount of downtime required. Ideally, less than one week after step one completes. Any new data written to the registry between steps one and two causes step two to take more time.

Import all repository data (step two)

This step requires the registry to be set in read-only mode. Step two is usually much shorter than step one. For example, in one of the user reports, step two took 12.5 hours after a step one of 344 hours. Allow enough time for downtime while step two is being executed.

  1. Ensure the registry is set to read-only mode.

    Edit your /etc/gitlab/gitlab.rb and add the maintenance section to the registry['storage'] configuration. For example, for a gcs-backed registry using a gs://my-company-container-registry bucket, the configuration could be:

    ruby
    ## Object Storage - Container Registry
    registry['storage'] = {
      'gcs' => {
        'bucket' => '<my-company-container-registry>',
        'chunksize' => 5242880
      },
      'maintenance' => {
        'readonly' => {
          'enabled' => true # Must be set to true.
        }
      }
    }
  2. Save the file and reconfigure GitLab.

  3. Run step two of the import:

    shell
    sudo gitlab-ctl registry-database import --step-two --log-to-stdout
  4. If the command completed successfully, all images are now fully imported. You can now enable the database, turn off read-only mode in the configuration, and start the registry service:

    ruby
    registry['database'] = {
      'enabled' => true, # Must be set to true!
    }
    
    ## Object Storage - Container Registry
    registry['storage'] = {
      'gcs' => {
        'bucket' => '<my-company-container-registry>',
        'chunksize' => 5242880
      },
      'maintenance' => { # This section can be removed.
        'readonly' => {
          'enabled' => false
        }
      }
    }
  5. Save the file and reconfigure GitLab.

  1. Ensure the registry is set to read-only mode.

    Edit your /etc/gitlab/gitlab.rb and add the maintenance section to the registry['storage'] configuration. For example, for a gcs-backed registry using a gs://my-company-container-registry bucket, the configuration could be:

    ruby
    ## Object Storage - Container Registry
    registry['storage'] = {
      'gcs' => {
        'bucket' => '<my-company-container-registry>',
        'chunksize' => 5242880
      },
      'maintenance' => {
        'readonly' => {
          'enabled' => true # Must be set to true.
        }
      }
    }
  2. Save the file and reconfigure GitLab.

  3. Run step two of the import:

    shell
    sudo -u registry gitlab-ctl registry-database import --step-two --log-to-stdout
  4. If the command completed successfully, all images are now fully imported. You can now enable the database, turn off read-only mode in the configuration, and start the registry service:

    ruby
    registry['database'] = {
      'enabled' => true, # Must be set to true!
    }
    
    ## Object Storage - Container Registry
    registry['storage'] = {
      'gcs' => {
        'bucket' => '<my-company-container-registry>',
        'chunksize' => 5242880
      },
      'maintenance' => { # This section can be removed.
        'readonly' => {
          'enabled' => false
        }
      }
    }
  5. Save the file and reconfigure GitLab.

  1. Ensure the registry is set to read-only mode.

    Edit your /etc/gitlab/gitlab.rb and add the maintenance section to the registry['storage'] configuration. For example, for a gcs-backed registry using a gs://my-company-container-registry bucket, the configuration could be:

    ruby
    ## Object Storage - Container Registry
    registry['storage'] = {
      'gcs' => {
        'bucket' => '<my-company-container-registry>',
        'chunksize' => 5242880
      },
      'maintenance' => {
        'readonly' => {
          'enabled' => true # Must be set to true.
        }
      }
    }
  2. Save the file and reconfigure GitLab.

  3. Run step two of the import:

    shell
    sudo gitlab-ctl registry-database import --step-two
  4. If the command completed successfully, all images are now fully imported. You can now enable the database, turn off read-only mode in the configuration, and start the registry service:

    ruby
    registry['database'] = {
      'enabled' => true, # Must be set to true!
      'host' => '<registry_database_host_placeholder_change_me>',
      'port' => 5432, # Default, but set to the port of your database instance if it differs.
      'user' => '<registry_database_username_placeholder_change_me>',
      'password' => '<registry_database_placeholder_change_me>',
      'dbname' => '<registry_database_name_placeholder_change_me>',
      'sslmode' => 'require', # See the PostgreSQL documentation for additional information https://www.postgresql.org/docs/16/libpq-ssl.html.
      'sslcert' => '</path/to/cert.pem>',
      'sslkey' => '</path/to/private.key>',
      'sslrootcert' => '</path/to/ca.pem>'
    }
    
    ## Object Storage - Container Registry
    registry['storage'] = {
      'gcs' => {
        'bucket' => '<my-company-container-registry>',
        'chunksize' => 5242880
      },
      'maintenance' => { # This section can be removed.
        'readonly' => {
          'enabled' => false
        }
      }
    }
  5. Save the file and reconfigure GitLab.

You can now use the metadata database for all operations!

Import remaining data (step three)

Even though the registry is now fully using the database for its metadata, it does not yet have access to any potentially unused layer blobs, preventing these blobs from being removed by the online garbage collector.

You may continue to use the registry as normal while step three is being completed. Step three can take longer than step one, but it does not require downtime. If step three is interrupted, you can run it again. Step three imports only the blobs that are not already in the database.

To complete the process, run the final step of the migration:

shell
sudo gitlab-ctl registry-database import --step-three --log-to-stdout
shell
sudo -u registry gitlab-ctl registry-database import --step-three --log-to-stdout
shell
sudo gitlab-ctl registry-database import --step-three

After that command exits successfully, registry metadata is now fully imported to the database.

After import

Large registries can have hundreds of thousands or even millions of blobs queued for garbage collection review after an import. This is expected, and at default worker intervals it takes time to process.

For guidance on what to expect and how to speed up processing, see: