Background operations for security inventory
This document describes the background operations system for tracking long-running bulk operations in the security inventory.
Purpose
Background operations allow users to perform bulk actions across multiple projects asynchronously. The system provides error tracking and email notifications on failure.
Architecture
Flow
flowchart LR
A[BulkUpdateService] --> B[BulkUpdateSchedulerWorker]
B --> C[BackgroundOperationBulkUpdateWorker]
C --> D{Failures?}
D -->|Yes| E[Email Notification]
D -->|No| F[Cleanup]
B --> G[Create Redis Operation]
C --> H[Update Counters]
- Initiate: User triggers bulk operation through
BulkUpdateService - Schedule:
BulkUpdateSchedulerWorkercollects projects and creates Redis operation - Process:
BackgroundOperationBulkUpdateWorkerprocesses projects in batches - Finalize: Last batch detects completion and claims the operation. Batches that complete at the same time race for the claim, and only the batch that wins it sends the email
- Cleanup: Claiming the operation reads the failed items and deletes both Redis entities, so the email is sent from what the winning batch already holds in memory
Components
| Component | Purpose |
|---|---|
Security::Attributes::BulkUpdateSchedulerWorker |
Orchestrates batch scheduling, creates Redis operation |
Security::Attributes::BackgroundOperationBulkUpdateWorker |
Processes projects with tracking |
Gitlab::BackgroundOperations::RedisStore |
Redis state management |
Security::BackgroundOperationMailer |
Failure notification emails |
Redis data structure
Operation entity
Key: background_operation:{operation_id}
Type: Hash
Fields:
- id: operation ID
- operation_type: type of operation (for example, 'attribute_update')
- user_id: user who initiated
- parameters: JSON with operation-specific params
- status: 'pending' | 'running'
- total_items: total projects to process
- successful_items: count of successful projects
- failed_items: count of failed projects
- created_at: timestamp
TTL: 72 hoursFailed items list
Key: background_operation:{operation_id}:failed_items
Type: List
Items: JSON objects with:
- entity_id: ID of failed entity (project or group)
- entity_type: 'Project' or 'Group'
- entity_name: name of the entity
- entity_full_path: full path for linking
- error_message: human-readable error, or an array of them
- error_code: machine-readable error code
- omitted_error_messages: count of messages dropped from error_message, when any
- created_at: timestamp
TTL: 72 hoursThe list holds at most RedisStore::MAX_FAILED_ITEMS entries, each with at most
RedisStore::MAX_ERROR_MESSAGES error messages of at most RedisStore::MAX_ERROR_MESSAGE_LENGTH
characters. That bounds one operation’s error text at roughly 500 KB, rather than growing with the
number of failed items. The failed_items counter on the operation entity stays the true total.
Error codes
| Code | Meaning |
|---|---|
service_error |
Service returned error response |
unexpected_error |
Unexpected exception occurred |
Email notifications
Emails are sent only when failures occur after the operation completes.
Email includes:
- Summary: total, successful, failed counts
- List of failed items (projects or groups) with error details and links. When the failure count exceeds the stored list, the email states how many of the failures it shows
Usage
Creating an operation
operation_id = Gitlab::BackgroundOperations::RedisStore.create_operation(
operation_type: 'attribute_update',
user_id: user.id,
total_items: projects_count,
parameters: { attribute_ids: [1, 2], mode: 'ADD' }
)Recording success or failure
# Record success
Gitlab::BackgroundOperations::RedisStore.increment_successful(operation_id)
# Record failure for a project
Gitlab::BackgroundOperations::RedisStore.add_failed_item(
operation_id,
entity_id: project.id,
entity_type: 'Project',
entity_name: project.name,
entity_full_path: project.full_path,
error_message: 'Permission denied',
error_code: 'service_error'
)
# Record failure for a group
Gitlab::BackgroundOperations::RedisStore.add_failed_item(
operation_id,
entity_id: group.id,
entity_type: 'Group',
entity_name: group.name,
entity_full_path: group.full_path,
error_message: 'Permission denied',
error_code: 'service_error'
)Checking operation status
operation = Gitlab::BackgroundOperations::RedisStore.get_operation(operation_id)
# Returns Operation struct with: id, operation_type, user_id, parameters,
# total_items, successful_items, failed_items
failed_items = Gitlab::BackgroundOperations::RedisStore.get_failed_items(operation_id)
# Returns array of failed item hashesCleanup
Gitlab::BackgroundOperations::RedisStore.delete_operation(operation_id)
# Deletes both the operation hash and failed items list