Troubleshooting Artifact Registry
- Tier: Premium, Ultimate
- Offering: GitLab.com
- Status: Beta
When working with Artifact Registry, you might encounter the following issues. Every response
carries an X-Request-ID header. Include that value when you report a problem.
401 Unauthorized when you publish or pull
Four different causes return this same response: a missing credential, an empty credential, a malformed credential, and a credential that does not match. The response is deliberately identical in each case, so it reveals nothing about which cause applies.
The most common cause is an expired token. Artifact Registry tokens are short-lived.
To resolve this issue, get a fresh token and update your client configuration. To confirm the configuration matches, select More actions ( ) > View setup instructions on the repository.
For npm, check that the registry line and the authentication line carry the same URL, including
the trailing slash. npm matches the authentication line by URL prefix, so a mismatch returns
401 without naming the URL.
For Maven, check that the server id in your settings file matches the id on the repository
you deploy to. If the two id values differ, Maven sends no credentials at all.
409 Conflict when you request a token
The exchange returns 409 Conflict when your account belongs to more than one organization that
has Artifact Registry. The response is Your account belongs to more than one organization with Artifact Registry. This is not supported yet.
The exchange works out which organization to use from your memberships. It cannot pick between two organizations, so it refuses instead of guessing.
There is no workaround in beta. If you hit this issue, contact your GitLab account team.
404 Not Found when you request a token
Token exchange runs behind a feature flag named gate_token_exchange_endpoint. The flag is
disabled by default. When the flag is off, POST /api/v4/token_exchange returns 404. The
response says nothing about the flag.
The endpoint needs GitLab 19.1 or later, Enterprise Edition.
To resolve this issue, ask an administrator to enable the flag for your instance. On GitLab.com, contact your GitLab account team instead, because Artifact Registry is invitation-only in beta. For more information, see feature flags.
404 Not Found for a repository you know exists
You might not have a role on the repository. Artifact Registry returns 404 rather than 403
when you cannot read a resource, so a repository you cannot access looks the same as one that
does not exist. Repositories you cannot read are also left out of lists silently.
To resolve this issue, ask an Artifact Admin to assign you a role. A role
change is not immediate. It can take a minute or more to take effect. During that window, the
same request can succeed and then fail again, so one 200 response does not mean the change is
done. Retry for a few minutes before you conclude the role was not assigned.
403 Forbidden when you publish or delete
You can read the repository, but your role does not hold the permission for that action.
Publishing needs the Artifact Contributor role or higher. Deleting artifacts needs the Artifact Manager role or higher.
An empty repository list
A user with no assigned role sees an empty list rather than an error.
To resolve this issue, ask an Artifact Admin for a role.
409 Conflict when you delete a repository
A delete request with destructive=false refuses to touch contents, so a repository that still
holds artifacts returns 409.
To resolve this issue, resend the request with destructive=true, which deletes the repository
and its contents.
For a hosted container repository, emptying it through its /v2/ delete commands does not clear
the 409. Those commands leave behind the row the delete conflicts with. Use destructive=true
instead. This behavior is tracked in issue 371.
For a hosted npm repository, unpublish commands mark packages for removal rather than removing
them at once, so the 409 stands until the removal finishes. Retry the delete after the removal
completes.
A second cause returns the same 409 and destructive=true does not clear it. When a virtual
repository lists the repository as an upstream, the response reads repository is an upstream of a virtual repository; remove the association before deleting it. Remove that association first,
then delete the repository.
409 Conflict when you delete a container manifest
Another manifest references the manifest you are deleting. The response lists the manifests that reference it.
To resolve this issue, delete the referencing manifests first.
503 Service Unavailable from a remote repository
The upstream is marked unhealthy after three consecutive failures. A Retry-After header tells
you when to try again. A read that the cache can serve still succeeds even while the upstream is
unhealthy.
To resolve this issue, test the upstream connection and check the upstream URL and credentials.
A 401, 403, or 404 response from an upstream counts as healthy, because it proves the
upstream is reachable.
A second cause returns 503 with Retry-After: 1 from a container remote repository, and the
upstream health check does not clear it. The registry fetches each uncached layer from the upstream
once, and concurrent pulls of that layer wait for the one fetch. It also limits how many layers one
remote repository fetches at a time. A pull that waits longer than the configured limit returns
503. The client that started the shared fetch can also disconnect before the fetch completes. A
pull that has not received any of the layer yet then returns 503. A pull that is already receiving
the layer gets a truncated response instead, and Docker retries the layer.
To resolve this issue, retry the pull. If large layers return 503 repeatedly, ask your
administrator to raise the fill wait and fill concurrency limits.
An unhealthy upstream is not the only reason a remote repository answers 503.
If the upstream checks out, the registry might be refusing the cache fill
because it is busy instead. See
503 Service Unavailable when you publish, push, install, or pull.
503 Service Unavailable when you publish, push, install, or pull
The registry limits how many uploads it stages at once, because each one holds a
memory buffer. When that limit is reached, the next request waits a few seconds
for a free slot and is then refused with 503 and a Retry-After header.
Uploads are the obvious case. Reads can be refused for the same reason, because some reads stage an upload of their own:
- Installing or pulling something the cache does not hold yet. A remote repository fills its cache from the upstream, and that fill stages the bytes it downloads.
- Reading npm package metadata the registry has to rebuild. A rebuild needs two free slots at the same time, so it can be refused while a single upload still gets a slot.
So a 503 on an install or a pull can mean the registry is busy with uploads,
rather than that anything is wrong with what you asked for.
This is load, not a quota: nothing is wrong with your package, your permissions,
or your repository, and no setting of yours caused it. Retry after the interval
the Retry-After header gives. Most package clients retry a 503 for you,
though some abandon the request instead, so a large push or install may need to
be re-run.
Retrying immediately and in parallel makes it worse, because every retry competes for the same slots. If it keeps happening, your instance administrator can raise the limit or lower the upload chunk size, which buys more concurrent uploads within the same memory budget.
400 MANIFEST_INVALID when you push to a Docker repository
A Docker repository accepts image manifests and manifests that carry a subject. It rejects an artifact that is neither, such as a Helm chart or a WebAssembly module with an empty configuration. Docker manifest schema 1 is also rejected.
To resolve this issue, push to an Open Container Initiative (OCI) format repository instead. The format is fixed when you create the repository, so you must create a new repository.
422 with name, format, and kind are immutable
An update request included name, format, or kind.
These properties are fixed when the repository is created, so the request fails even when the value
matches the stored one.
To resolve this issue, remove those fields from the update request. To change one of these properties, create a new repository.
A path length error when you push a container image
The full path, including the registry handle, the repository name, and the image name, cannot be longer than 255 characters. Separator characters count toward this limit.
To resolve this issue, use shorter names.
An empty response when you unpublish an npm version
npm unpublish <package>@<version> sends the package document to the server. If that upload
goes quiet for longer than the inactivity window, the server stops reading and answers with
an empty body and status 499. The version stays published. A slow but steady upload is not
affected, because the window is measured between two reads rather than over the whole
request.
To resolve this issue, run the command again on a more stable connection. If it keeps
happening, ask an administrator to check npm.unpublish_inactivity_timeout, which sizes the
gap between two reads, and npm.unpublish_read_timeout, the ceiling on the whole request.
Usage figures that do not match
Some figures are stored counters that update in the background, and others are counted at the moment you ask, so two figures over the same content can disagree for a short time.
For what each figure counts and why they differ, see View storage usage.