Geo background jobs

  • Tier: Premium, Ultimate
  • Offering: GitLab Self-Managed, GitLab Dedicated

Geo persists all sync and verification intent in registry tables in the Geo tracking database, not in Sidekiq job arguments. If a job is killed, the registry record retains its state and cron-based schedulers re-enqueue the work. This design makes Geo fundamentally crash-safe.

Registry sync states

Every replicable data type has a registry record on the secondary site that tracks sync state:

State Description
pending Needs sync. Picked up by the sync scheduler cron.
started Sync in progress. If the worker is killed, the record stays in this state.
synced Successfully replicated.
failed Sync failed. Has a retry_at timestamp for exponential backoff retry.

If a sync job is killed, the registry stays in started. Geo::SyncTimeoutCronWorker, which runs every 10 minutes, detects the sync job state and marks the registry as failed with retry backoff. The sync scheduler cron worker then re-enqueues the registry for sync.

Recovery mechanisms

The following cron workers provide automatic recovery. Their schedules are defined in ee/config/schedule.yml and are configurable.

The cron job name matches the name shown in the Cron tab of the Sidekiq dashboard in the Admin area.

Mechanism Worker Cron job name Default schedule Purpose
Sync timeout recovery Geo::SyncTimeoutCronWorker geo_sync_timeout_cron_worker Every 10 min (secondary) Marks registries stuck in started state as failed with retry backoff.
Blob sync scheduler Geo::RegistrySyncWorker geo_registry_sync_worker Every 1 min (secondary) Polls for pending and failed blob registries and enqueues Geo::SyncWorker.
Repository sync scheduler Geo::RepositoryRegistrySyncWorker geo_repository_registry_sync_worker Every 1 min (secondary) Polls for pending and failed repository registries and enqueues Geo::SyncWorker.
Registry consistency Geo::Secondary::RegistryConsistencyWorker geo_secondary_registry_consistency_worker Every 1 min (secondary) Creates missing registry records for untracked replicables. Detects orphaned registries.
Verification timeout Geo::VerificationTimeoutWorker Triggered by geo_verification_cron_worker Every 1 min (primary and secondary) Marks verification stuck in verification_started as verification_failed.
Verification scheduler Geo::VerificationCronWorker geo_verification_cron_worker Every 1 min (primary and secondary) Triggers verification batch, timeout, re-verification, and state backfill workers.

Queue safety reference

The following sections provide per-worker safety information for Geo Sidekiq queues.

Cron workers

Cron workers are scheduled automatically and re-run on the next cron tick if their queue is cleared. All cron worker queues are safe to clear.

Worker What it does Safe to clear queue Negative consequences of clearing Recovery mechanism
Geo::RegistrySyncWorker Polls for pending and failed blob registries. Enqueues Geo::SyncWorker. Yes Sync is delayed until the next cron tick. Re-runs every 1 min.
Geo::RepositoryRegistrySyncWorker Polls for pending and failed repository registries. Enqueues Geo::SyncWorker. Yes Sync is delayed until the next cron tick. Re-runs every 1 min.
Geo::SyncTimeoutCronWorker Finds registries stuck in started state. Marks them failed with retry backoff. Yes Registries stuck in started are not transitioned to failed until the next tick. Sync resumes after the next tick. Re-runs every 10 min.
Geo::Secondary::RegistryConsistencyWorker Scans all registry types. Creates missing registries. Detects orphaned registries and enqueues Geo::DestroyWorker. Yes Missing registries are not created and orphaned registries are not cleaned up until the next tick. Re-runs every 1 min.
Geo::VerificationCronWorker Triggers all verification sub-workers. Yes Verification is delayed until the next cron tick. Re-runs every 1 min.
Geo::VerificationTimeoutWorker Recovers records stuck in verification_started. Yes Records stuck in verification_started are not transitioned until the next tick. Re-runs every 1 min.
Geo::PruneEventLogWorker Deletes old event log entries that all secondaries have consumed (primary only). Yes The event log grows until the worker runs again. No data loss. Re-runs every 5 min.
Geo::MetricsUpdateWorker Computes node status, updates Prometheus gauges, sends status to primary. Yes Metrics become stale until the worker runs again. Re-runs every 1 min.
Geo::SidekiqCronConfigWorker Enables and disables cron jobs based on node type (primary or secondary). Yes Cron job configuration may be incorrect until the worker runs again. Re-runs every 1 min.

Sync workers (secondary)

Worker What it does Safe to clear queue Negative consequences of clearing Recovery mechanism
Geo::SyncWorker Downloads a single blob or fetches a single repository from the primary site. Yes Registries for in-flight jobs stay in started. Sync is delayed until recovery. SyncTimeoutCronWorker transitions stuck registries to failed. Sync scheduler re-enqueues.
Geo::ContainerRepositorySyncWorker Syncs a single container repository from the primary site. Yes Registry retains its state. Sync is delayed until recovery. Sync scheduler re-enqueues.
Geo::BulkRegistryResyncWorker Triggers bulk re-sync of a registry class. Yes Bulk re-sync does not start. Individual registries retain their state. Caller re-enqueues.

Event workers (primary and secondary)

When you clear event worker queues, events in the queue might be lost. Lost events can cause data to be temporarily out of date on the secondary site.

Worker What it does Safe to clear queue Negative consequences of clearing Recovery mechanism
Geo::EventWorker Processes Geo replication events (created, updated, deleted) on the secondary site. Use caution. Losing updated events can leave a resource out of date until re-verification or the next update event. Losing deleted events can leave orphaned files on the secondary site (wasted disk, no data loss). Losing created events has no lasting effect. Has 3 Sidekiq retries. Even if all retries fail, the registry record exists and the sync scheduler re-enqueues. RegistryConsistencyWorker also detects orphaned registries.
Geo::BatchEventCreateWorker Bulk-inserts Geo events on the primary site. Use caution. Events in the queue are lost if cleared. Secondary sites may not learn about changes until re-verification. RegistryConsistencyWorker on the secondary site eventually detects missing registries (every 1 min).
Geo::CreateRepositoryUpdatedEventWorker Creates a Geo event when a repository is updated on the primary site. Use caution. Same as Geo::BatchEventCreateWorker. Same as Geo::BatchEventCreateWorker.

Verification workers (primary and secondary)

Worker What it does Safe to clear queue Negative consequences of clearing Recovery mechanism
Geo::VerificationBatchWorker Checksums batches of records. Yes Verification is delayed. Cron re-enqueues. VerificationTimeoutWorker catches stuck records.
Geo::ReverificationBatchWorker Marks already-verified records for periodic re-verification. Yes Re-verification is delayed. Cron re-enqueues.
Geo::VerificationStateBackfillWorker Backfills verification state table for replicable types. Yes Backfill is delayed. Exclusive lease expires in 30 min. Re-enqueues itself.
Geo::BulkPrimaryVerificationWorker Triggers bulk verification of a model class on the primary site. Yes Bulk verification does not start. Caller re-enqueues.
Geo::BulkRegistryReverificationWorker Triggers bulk re-verification of a registry class on the secondary site. Yes Bulk re-verification does not start. Caller re-enqueues.

Destroy workers (secondary)

When you clear the destroy worker queue, lost jobs can leave orphaned files or repositories on the secondary site.

Worker What it does Safe to clear queue Negative consequences of clearing Recovery mechanism
Geo::DestroyWorker Deletes a replicated file or repository on the secondary site after deletion on the primary site. Use caution. Orphaned files or repositories remain on the secondary site, wasting disk space. No data loss. Has 3 Sidekiq retries. RegistryConsistencyWorker detects orphaned registries and re-enqueues DestroyWorker.