Gitaly timeouts and retries

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab Self-Managed

Gitaly provides two types of configurable timeouts:

  • Call timeouts, configured by using the GitLab UI.
  • Negotiation timeouts, configured by using Gitaly configuration files.

Configure the call timeouts

Configure the following call timeouts to make sure that long-running Gitaly calls don’t needlessly take up resources.

Prerequisites:

  • Administrator access.

To configure the call timeouts:

  1. In the upper-right corner, select Admin.
  2. Select Settings > Preferences.
  3. Expand the Gitaly timeouts section.
  4. Set each timeout as required.

Available call timeouts

Different call timeouts are available for different Gitaly operations.

Timeout Default Description
Default 55 seconds Timeout for most Gitaly calls (not enforced for git fetch and push operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made in a web request cannot exceed the entire request timeout. It should be shorter than the worker timeout that can be configured for Puma. If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker.
Fast 10 seconds Timeout for fast Gitaly operations used in requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance.
Medium 30 seconds Timeout for Gitaly operations that should be fast (possibly in requests) but preferably not used multiple times in a request. For example, loading blobs. Timeout that should be set between Default and Fast.

Configure the negotiation timeouts

You might need to increase the negotiation timeout:

  • For particularly large repositories.
  • When performing these commands in parallel.

You can configure negotiation timeouts for:

  • git-upload-pack(1), which is invoked by a Gitaly node when you execute git fetch.
  • git-upload-archive(1), which is invoked by a Gitaly node when you execute git archive --remote.

To configure these timeouts:

Edit /etc/gitlab/gitlab.rb:

gitaly['configuration'] = {
    timeout: {
        upload_pack_negotiation: '10m',      # 10 minutes
        upload_archive_negotiation: '20m',   # 20 minutes
    }
}

Edit /home/git/gitaly/config.toml:

[timeout]
upload_pack_negotiation = "10m"
upload_archive_negotiation = "20m"

For the values, use the format of ParseDuration in Go.

These timeouts affect only the negotiation phase of remote Git operations, not the entire transfer.

Gitaly client retries

Gitaly can sometimes be briefly unavailable. For example, during GitLab upgrades. Especially with Gitaly on Kubernetes, where a Pod starts and restarts take a couple of seconds.

To prevent GitLab from returning errors to clients when briefly unavailable, configure Gitaly client retries. When Gitaly client retries are configured and Gitaly is unavailable, Gitaly client such as Rails (GitLab application), Workhorse, and GitLab Shell retry request in an exponential backoff fashion.

Two parameters can be configured:

  • max_attempts: Maximum number of retry attempts between 1 and 5.
  • max_backoff: Maximum amount of time before the client stops retrying. Value must be a duration string, such as 1.4s or 10s.

The backoff multiplier is set to 2 and the initial backoff is derived from the two parameters.

Configuration guidelines

The right configuration depends on your GitLab instance setup and how long Gitaly remains unavailable when such an event occurs:

  • On Kubernetes, a Gitaly Pod can take approximately 10 to 12 seconds to start, depending on the Cloud provider. The time includes how long it takes for the volume to be attached and mounted on the Pod.
  • For Linux package instances, Gitaly might restart much faster because restarting Gitaly is a process restart.

Also keep in mind is that Gitaly can be configured with a graceful shutdown timeout. When Gitaly is shutting down, new requests are rejected but the gRPC server keeps processing in-flight requests until either:

  • They are all served.
  • The shutdown timeout lapses.

This graceful shutdown timeout can play a role in how long Gitaly remains unavailable for new requests.

You should configure client retry with a max_backoff that is equal to or greater than sum of the graceful shutdown + the (re)start time.

Configure client retries

The following configuration applies to Rails (GitLab application), Workhorse, and GitLab Shell and the same configuration applies to all clients.

Provided values are examples and should not be treated as guidelines.

Update your gitlab.rb file with these configurations:

gitlab_rails['gitaly_client_max_attempts'] = 5
gitlab_rails['gitaly_client_max_backoff'] = '1.4s'

Update your values.yml file with these configurations:

global:
  gitaly:
    client:
      maxAttempts: 5
      maxBackoff: '1.4s'