Failed to remove tag when removing a image from the registry

I have a bunch of images in my registry that I can not remove from the interface. Clicking the remove icon gives a message "Failed to remove tag".

In /var/log/gitlab/registry/current, I see the following messages:

2016-07-25_09:32:40.10174 time="2016-07-25T11:32:40.101599771+02:00" level=error msg="response completed with error" auth.user.name= environment=production err.code="manifest unknown" err.detail="unknown manifest name=development/boss revision=sha256:7821b1eb677a46a0cbe7321de06537c694f8f580b5ee7b9ab6e8c0146fb10e2a" err.message="manifest unknown" go.version=go1.5.4 http.request.host="localhost:5000" http.request.id=640d5b9d-1922-48b9-b9ad-057fc3774b32 http.request.method=HEAD http.request.remoteaddr="127.0.0.1:43148" http.request.uri="/v2/development/boss/manifests/papi-95230115ea6f159b36338e0aa05f10a4fbf646e1" http.request.useragent="Faraday v0.9.2" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.03603ms http.response.status=404 http.response.written=192 instance.id=df73c0b7-cf0a-44b2-a25a-e01db6be4fb8 service=registry vars.name="development/boss" vars.reference=papi-95230115ea6f159b36338e0aa05f10a4fbf646e1 version=v2.4.1 
2016-07-25_09:32:40.10182 127.0.0.1 - - [25/Jul/2016:11:32:40 +0200] "HEAD /v2/development/boss/manifests/papi-95230115ea6f159b36338e0aa05f10a4fbf646e1 HTTP/1.1" 404 192 "" "Faraday v0.9.2"
2016-07-25_09:32:42.88647 time="2016-07-25T11:32:42.886354924+02:00" level=info msg="response completed" environment=production go.version=go1.5.4 http.request.host="localhost:5000" http.request.id=cf08aca5-9cdf-4074-838f-f8f0e33ce8e0 http.request.method=GET http.request.remoteaddr="127.0.0.1:43658" http.request.uri="/v2/development/boss/manifests/pfront-95230115ea6f159b36338e0aa05f10a4fbf646e1" http.request.useragent="Faraday v0.9.2" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=5.506057ms http.response.status=200 http.response.written=3659 instance.id=df73c0b7-cf0a-44b2-a25a-e01db6be4fb8 service=registry version=v2.4.1 
2016-07-25_09:32:42.88651 127.0.0.1 - - [25/Jul/2016:11:32:42 +0200] "GET /v2/development/boss/manifests/pfront-95230115ea6f159b36338e0aa05f10a4fbf646e1 HTTP/1.1" 200 3659 "" "Faraday v0.9.2"
2016-07-25_09:32:49.51210 time="2016-07-25T11:32:49.512030769+02:00" level=info msg="response completed" environment=production go.version=go1.5.4 http.request.host="localhost:5000" http.request.id=6216b19a-b6e0-40a5-9984-fbf674e99644 http.request.method=GET http.request.remoteaddr="127.0.0.1:44870" http.request.uri="/v2/development/boss/manifests/pcore-95230115ea6f159b36338e0aa05f10a4fbf646e1" http.request.useragent="Faraday v0.9.2" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=5.604862ms http.response.status=200 http.response.written=3865 instance.id=df73c0b7-cf0a-44b2-a25a-e01db6be4fb8 service=registry version=v2.4.1 
2016-07-25_09:32:49.51228 127.0.0.1 - - [25/Jul/2016:11:32:49 +0200] "GET /v2/development/boss/manifests/pcore-95230115ea6f159b36338e0aa05f10a4fbf646e1 HTTP/1.1" 200 3865 "" "Faraday v0.9.2"
2016-07-25_09:32:50.98922 time="2016-07-25T11:32:50.989015587+02:00" level=error msg="response completed with error" auth.user.name= environment=production err.code="manifest unknown" err.detail="unknown manifest name=development/boss revision=sha256:7821b1eb677a46a0cbe7321de06537c694f8f580b5ee7b9ab6e8c0146fb10e2a" err.message="manifest unknown" go.version=go1.5.4 http.request.host="localhost:5000" http.request.id=bae2563d-85d7-40b4-9bc5-53b1c942c321 http.request.method=GET http.request.remoteaddr="127.0.0.1:45140" http.request.uri="/v2/development/boss/manifests/papi-95230115ea6f159b36338e0aa05f10a4fbf646e1" http.request.useragent="Faraday v0.9.2" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.692108ms http.response.status=404 http.response.written=192 instance.id=df73c0b7-cf0a-44b2-a25a-e01db6be4fb8 service=registry vars.name="development/boss" vars.reference=papi-95230115ea6f159b36338e0aa05f10a4fbf646e1 version=v2.4.1 
2016-07-25_09:32:50.98924 127.0.0.1 - - [25/Jul/2016:11:32:50 +0200] "GET /v2/development/boss/manifests/papi-95230115ea6f159b36338e0aa05f10a4fbf646e1 HTTP/1.1" 404 192 "" "Faraday v0.9.2"

The interface displays the following:

Changed title: Failed to remove tag → Failed to remove tag when removing a image from the registry

Added 12980 gitlab-foss343072 labels

Still occurs in 8.11.2-ee.

My university is running GitLab CE 8.10.6 and I am facing exactly the same issue. Any update on this?

Is there a way to manually clear the registry from the cli? I just want to remove all images, but I can't due to this issue.

Run in to this issue today running Gitlab CE 8.12.4. Anyone find any workarounds? A manual method to remove the images would be fine as a stop gap.

On Gitlab CE 8.13.1 is same error :(

Same issue on Gitlab 8.14.0 - running in Docker.

Added ~19173 label

mentioned in issue gitlab-foss#20984 (closed)

Fixed by properly adding REGISTRY_STORAGE_DELETE_ENABLED: true to the registry configuration. I had a slightly different error message: err.code=unsupported err.message="The operation is unsupported.".

@michael.arnauts Could you confirm the fix @fbretel proposed? If this works for you too, we should fix this in omnibus and the docker images. Thus move this issue to another project

I had cleaned all images due to this issue, so I can't test anymore... Maybe someone else can?

On Fri, 6 Jan 2017, 09:33 Zeger-Jan van de Weg, gitlab@mg.gitlab.com wrote:

@michael.arnauts https://gitlab.com/michael.arnauts Could you confirm the fix @fbretel https://gitlab.com/fbretel proposed? If this works for you too, we should fix this in omnibus and the docker images. Thus move this issue to another project

— Reply to this email directly or view it on GitLab https://gitlab.com/gitlab-org/gitlab-ce/issues/20215#note_20981520. You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can unsubscribe https://gitlab.com/sent_notifications/REDACTED/unsubscribe from this thread or adjust your notification settings.

-- Michaël Arnauts

mentioned in issue gitlab-com/support-forum#1509

I'm getting the same message. Currently on version: 8.16.2 Most of the time when I delete an image it is successful but periodically the operation fails.

At this point I find the easiest way to simulate the problem is trying to delete several images before the page refreshes.

Is there a way to remove the tags from the UI when they are no longer connected to an image?

Had the same issue where some tags would show incomplete information on Gitlab's "Container Registry" page, and were unable to be deleted:

When trying to delete one of these tags, an error message was shown:

Failed to remove registry tag!

I was able to use the python script found at https://github.com/burnettk/delete-docker-registry-image to delete these tags.

Do this at your own risk!

We are using the Gitlab Omnibus installation. If you installed from source, your paths may vary!

These were the steps I did:

• download the script and CHMOD it to be executable, as described on https://github.com/burnettk/delete-docker-registry-image
• stop the Gitlab container registry: $ gitlab-ctl stop registry
• set the registry data directory: $ export REGISTRY_DATA_DIR=/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/
• start the script in "dry-run" mode to check for problems: $ delete_docker_registry_image --image <repository>:<tag> --verbose --dry-run
• the output contains many errors due to missing layers, but the script is able to delete the tag anyway:

  DEBUG    [2017-05-28 11:37:32,985]  Deleting repository '<repository>' with tag '<tag>'
  DEBUG    [2017-05-28 11:37:39,234]  Looking up filesystem layers for manifest digest <some sha digest>
[...]
  CRITICAL [2017-05-28 11:37:39,305]  Failed to read layers from blob:[Errno 2] No such file or directory: '/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/blobs/sha256/<some other sha digest>/data'
[... many more errors like this]
  INFO     [2017-05-28 11:37:40,446]  DRY_RUN: would have deleted /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/<repository>/_manifests/tags/<tag>

• start the script again to actually delete the tag: $ delete_docker_registry_image --image <repository>:<tag>
• repeat the last two steps for each defective tag
• start the Gitlab container registry: $ gitlab-ctl start registry
• visit your Gitlab's "Container Registry" page and make sure the defective tags are gone

@martingregoire We store images on Amazon S3 & met the same issue, which the repo you referred doesn't support. Is there another solution can fix it?

@jostyee: sorry, I do not know a solution for S3.

on 9.3.7 this is still an issue. if I attempt to pull an image that I can't delete from the registry it fails with: Error response from daemon: manifest for registry.windsorcircle.com/docker/proc:proc-17.01.11.08-rc02 not found

I solved by going into the S3 bucket and finding the relevant tags directory and removing the problem tags from there.

For workaround when you don't have access to s3 (gitlab.com for example) you can recreate (overwrite) the container tags and then remove them. This script helped us, feel free to give it a try for last resort.

#!/bin/bash


## declare an array variable (collect these manually)
arr=(
	1.2.3
	cc462a62ea34abef55e343f0e190c85c25661bfb
	cfac5ba64a1fac65cb5e0f74ba7fe1438e3bad12
	f2867dda406c5c599aada637df682b02a1f89b50
	f78133aaab33e12ba9afe75b4d390c46ab9e3eef
	fa03778a9aedfc3b7dc52b4ed90b147c497e0182
	fef1a85705a52664ae71e9f4bd4aa627695de134
	ffc0c848300af6a66b345aff08db04d1bfe4dcaf
)

repo="registry.gitlab.com/your-group/your-project"

## now loop through the above array
for i in "${arr[@]}"
do
   echo "$i"
   docker build -t $repo:$i .
   docker push $repo:$i
done

# You can access them using echo "${arr[0]}", "${arr[1]}" also

Where the Dockerfile in the actual directory:

FROM alpine
RUN mkdir -p /test
WORKDIR /test

You can optimize in many ways of course. This helped us moving projects where registry couldn't be deleted.

Clever solution! But wouldn't it be better to build 1 image with that docker file then make a bunch of tags with the loop? If you do, then deleting 1 of them will remove all in that name space

moved from gitlab-foss#20215 (moved)