Test Geo replication of upload partitions
Create test data for every upload partition and exercise the partition replicators end to end on the GitLab Development Kit (GDK) or on a reference environment.
This page is a manual test procedure, not development guidance.
In GitLab 19.5 and earlier, the legacy Geo::UploadReplicator replicates and verifies the whole
uploads table by default, because the geo_upload_replication feature flag is on.
In GitLab 19.6 and later, this flag is off by default and the partition replicators, which are
enabled by default, replicate and verify each partition of the uploads table separately.
For more information, see issue 628308.
To change which replicator runs, see switch between the legacy and partition replicators.
Seed one upload per partition
The gitlab:seed:geo_uploads Rake task creates one real Upload in each upload partition.
It builds the parent records off an existing project and attaches a small file through each
model’s uploader, so the rows land in the correct partition and store a real, verifiable blob.
Run the task on the primary site:
# Uses the first project it finds
bundle exec rake gitlab:seed:geo_uploads
# Or target a specific project (recommended: use a project in a group namespace so
# the namespace_uploads partition is covered)
bundle exec rake "gitlab:seed:geo_uploads[group/project]"The task prints a result for each partition and a summary of upload counts by model_type.
The count is the partition’s total row count, including any pre-existing data, not only
the rows this run created.
A partition reports FAILED when its recipe raised an error instead of creating a record.
Check the corresponding FAIL line above the summary for the reason.
Switch between the legacy and partition replicators
The legacy and partition replicators are mutually exclusive.
While the geo_upload_replication flag is turned on, the partition replicators do not emit
events, and the partition registry consistency backfill is suppressed.
With the default flag states, the partition replicators are active. If your environment has stored values for these flags, turn on every partition flag first, then turn off the legacy flag. If a flag is disabled when you disable the legacy flag, the partition stops replicating.
On the primary site, run the following command in a Rails console:
Gitlab::Geo.blob_replicator_classes.select { |replicator| replicator.model < ::Upload }.each do |replicator|
Feature.enable(replicator.replication_enabled_feature_key)
end
Feature.disable(:geo_upload_replication)To test the legacy replicator, turn on geo_upload_replication.
Check verification on the primary site
The primary site calculates a checksum for each blob and stores it in the partition
*_upload_states table.
The Geo::VerificationCronWorker runs every minute, or you can run it directly:
Geo::VerificationCronWorker.new.performTo confirm a single record checksums correctly, verify one replicator directly:
upload = Upload.where(model_type: 'Project').last
record = Geo::ProjectUpload.find(upload.id)
record.replicator.verify
record = Geo::ProjectUpload.find(upload.id)
record.verification_state # 2 means succeeded
record.verification_checksum # populated when verification succeedsCheck replication and verification on a secondary site
On a secondary site, registries are created from replication events and from the
Geo::Secondary::RegistryConsistencyWorker backfill.
Both cron workers run every minute, or you can run them directly:
Geo::Secondary::RegistryConsistencyWorker.new.perform
Geo::VerificationBatchWorker.new.perform_work('project_upload')Check the overall status with the Rake task:
bundle exec rake gitlab:geo:check_replication_verification_statusInspect a single partition registry directly:
Geo::ProjectUploadRegistry.synced.count
Geo::ProjectUploadRegistry.failed.countFor a broader view, the Geo administration area shows replication and verification counts per data type.
Test on a reference or staging environment
Reference environments such as staging behave like production, so the Rake task is dependency free and does not use test factories.
- Deploy the branch that contains the seed task, or run it from an existing checkout.
- Run
gitlab:seed:geo_uploadson the primary site with a Rails runner or console. Confirm the uploads store to object storage when object storage is enabled. - If the environment has stored values for the feature flags, set them through ChatOps, for example
/chatops run feature set geo_project_upload_replication truefor each partition, then/chatops run feature set geo_upload_replication false. - Wait for the cron workers to run, then check
gitlab:geo:check_replication_verification_statusand the Geo administration area.
Cleaning up seeded data
The task tags the leaf records it creates with a geo-seed prefix in their name or message,
so you can find and remove them after testing.
Avatars set on reused records such as the project, group, and user replace the previous avatar
rather than adding rows.
Automated test coverage
The per-replicator contract is covered by shared examples, so most correctness checks do not need a running secondary site. For details, see the blob replicator strategy of the Geo self-service framework.