Docker 500 error: Service endpoint with name already exists

We are experiencing same issue with similar configuration and docker v1.12.1. It seems to be a docker problem.
You can manually docker network diconnect -f stale container from network. But this bug will appear again in a few runs.

We are also running docker v1.12.1 on ubuntu1404. Does it make sense to somehow upgrade docker to eventually solve this?

CentOS 7(Kernel Version: 4.4.16-1.el7.elrepo.x86_64) + Docker Version: 1.12.1 + Gitlab runner - 1.5.2 - Gitlab 8.11.4

You can test this commit. It should solve this problem.

Mentioned in merge request !337 (closed)

Added ~58730 ~167310 labels

@grzesiek would you mind taking a look into this issue? Also reported at https://gitlab.zendesk.com/agent/tickets/48842

@tmaczukin Can you take a look too? It looks like a Runner issue.

Did anyone try the version from commit proposed by @Bevaz in https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1642#note_15064802? It seems to be the right way to resolve the problem. Author of the commit have opened a merge request for this (!337 (closed)) so if it will work, we can work on including this into Runner.

@tmaczukin There are merge conflicts there at the moment; can we take ownership of that patch if necessary?

Never mind, the patch applies cleanly. The MR just needs to be rebased.

@stanhu I saw the merge conflicts and I've asked MR author for a rebase. If he will be not able to work on this, then I'll finish this myself :). But first let's test if this patch works - I've never saw this error and it's hard to reproduce.

@tmaczukin Yes, I've offered to send an experimental binary to the customer.

@tmaczukin Customer has gitlab-runner 1.7.1 installed. Can you send them a binary with the changes (https://gitlab.zendesk.com/agent/tickets/48842) and instructions on how to replace the existing one (and back up the old one)? I assume we can just apply the patch on top of 1.7.1 and ship them a binary, but maybe there's a systematic way to do this with a branch and build artifact.

I sent the customer a copy of a build of the runner with a 1.7.1 tag with a modified version of the patch in https://gitlab.com/stanclai/gitlab-ci-multi-runner/commit/011b42a11cd81784d9097225fadef88d04793588. I had to comment out the Force parameter for the docker-client to build, however.

Here are the instructions for anyone who wants to try at home. The patch is included in the tarball:

wget https://s3.amazonaws.com/gitlab-support/runner/gitlab-runner-1-7-1-unplugged-stalled-endpoints.tar.gz
tar xzvf gitlab-runner-1-7-1-unplugged-stalled-endpoints.tar.gz
mv /usr/bin/gitlab-ci-multi-runner /usr/bin/gitlab-ci-multi-runner.orig
cp runner/gitlab-ci-multi-runner /usr/bin/gitlab-ci-multi-runner

@stanhu Also can be found here - https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/pipelines/5016706 - artifacts for binaries and packages builds or download from S3 from URL-s in development job. I've just checked that build was finished with success. If the binary will work then I will open MR and get this into v1.8.0 :)

@stanhu Do you have any feedback?

@tmaczukin Not yet. We are still waiting to hear back from the customer.

Latest update:

Anyhow, from the few jobs that have ran, there has not been a single one of them that has failed with the error that was tied to this issue.

I will report back more later.

@tmaczukin Is there any reason we should not add this patch now?

I don't think so. Will include this patch into 1.8

Mentioned in merge request !390 (merged)

Status changed to closed by merge request !390 (merged)

Status changed to reopened

@tmaczukin customer sent back some updates reporting the same issue is re appearing and some more random errors as well:

Reported Sunday 23

Running with gitlab-ci-multi-runner 1.7.1 (f896af7)

Using Docker executor with image containers.xxxxx.net/containers/composer:php7.0 ...

Pulling docker image containers.xxxxx.net/containers/composer:php7.0 ...

ERROR: Build failed (system failure): API error (500): service endpoint with name runner-43f97f81-project-573-concurrent-0-predefined already exists

Reported Friday 21:

We are still getting failures…

Running with gitlab-ci-multi-runner 1.7.1 (f896af7)

Using Docker executor with image docker:1 ...

Pulling docker image docker:1-dind ...

Starting service docker:1-dind ...

Waiting for services to be up and running...

*** WARNING: Service runner-ed25fbf0-project-563-concurrent-1-docker probably didn't start properly.

Also this one which seems like a network timeout but adding it for context.

7.0-develop-cli: Pulling from containers/php

386a066cd84a: Pulling fs layer

269e95c6053a: Pulling fs layer

6243d5c57a34: Pulling fs layer

3226f5f2196c: Pulling fs layer

6951687c1d21: Pulling fs layer

2753c7ff467d: Pulling fs layer

a319b49a8416: Pulling fs layer

6eb85f3954ea: Pulling fs layer

95c474c59786: Pulling fs layer

2753c7ff467d: Waiting

a319b49a8416: Waiting

6eb85f3954ea: Waiting

95c474c59786: Waiting

6951687c1d21: Waiting

3226f5f2196c: Waiting

error pulling image configuration: Get https://containers.xxxxx.net/v2/containers/php/blobs/sha256:xxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout

ERROR: Build failed: exit code 1

@balameb It looks like binary provided by @stanhu in https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1642#note_18531228 doesn't contain the patch.

In logs you've send I can see:

Running with gitlab-ci-multi-runner 1.7.1 (f896af7)

I've checked the binary from https://s3.amazonaws.com/gitlab-support/runner/gitlab-runner-1-7-1-unplugged-stalled-endpoints.tar.gz:

tmaczukin@apollo:/tmp/runner$ ./gitlab-ci-multi-runner --version
Version:      1.7.1
Git revision: f896af7
Git branch:   1-7-1-unplugged-stalled-endpoints
GO version:   go1.7.3
Built:        Wed, 16 Nov 2016 18:17:38 -0500
OS/Arch:      linux/amd64

Look at Git revision: f896af7. This is exactly the same SHA as for the v1.7.1 tag:

tmaczukin@apollo:~/go/src/gitlab.com/gitlab-org/gitlab-ci-multi-runner$ git describe f896af7
v1.7.1

The patch is already merged into master. The Bleeding Edge version can be installed with:

wget https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/master/binaries/gitlab-ci-multi-runner-linux-amd64
chmod +x gitlab-ci-multi-runner-linux-amd64
mv /usr/bin/gitlab-ci-multi-runner /usr/bin/gitlab-ci-multi-runner.orig
cp ./gitlab-ci-multi-runner-linux-amd64 /usr/bin/gitlab-ci-multi-runner

Revert is done like with previous approach.

Thanks @tmaczukin. It could be the patch was applied to the binary but not reflected in the git revision? Let's try the bleeding edge version in any case.

Revision is set with compilation parameters if you use make or add them yourself. If you added the patch and compiled it by go build ./... then yes, it could be not reflected. I just assumed that you've used make as in our build process.

Thanks @tmaczukin. I'll suggest bleeding edge.

So is bleeding edge 1.8? What is it exactly that we're trying out? That the patch is included?

@balameb Bleeding Edge release is build from master after each push/merge into it. Tomorrow it will be tagged as v1.8 and released as a Stable release :)

And the patch for unplugging containers from docker networks is merged into master.

@tmaczukin results sent from customer today:

Running with gitlab-ci-multi-runner 1.9.0~beta.4.g81a23ec (81a23ec)

Using Docker executor with image containers.spinen.net/containers/shellcheck:master ...

WARNING: Can't disconnect possibly zombie container runner-43f97f81-project-573-concurrent-3-predefined from network bridge -> No such network (2315430c102c90dee4a2adf89aedd614cf9ab6aa5f4aa0a0330c1e2b2a9b056a) or container (runner-43f97f81-project-573-concurrent-3-predefined)

Pulling docker image containers.spinen.net/containers/shellcheck:master ...

ERROR: Build failed (system failure): API error (500): service endpoint with name runner-43f97f81-project-573-concurrent-3-predefined already exists

We are getting tons of these still on several of the jobs.

@balameb In my case issue was related to: docker #20398 issue. I think this happened after my disk storage become full.

I fixed problem via: docker network disconnect -f bridge runner-43f97f81-project-573-concurrent-3-predefined

I wrote some simple script that can help you follow steps to fixup those problems

Thanks @jacek-marchwicki we'll give that a try

We suspect the problem had to do with the root filesystem running out of disk space as described here: https://github.com/docker/docker/issues/20398#issuecomment-263587963

The single runner had 5 GB free and had 50 concurrent builds with only 2 cores.

I had the same problem and after disconnecting the runners as described above I started getting this kind of errors:

ERROR: Build failed (system failure): API error (500): failed to update store for object type *libnetwork.endpoint: timeout

I didn't noticed this kind of API errors before that. It's really rare, but may be somehow connected so I decided to post it here.

mentioned in merge request !432 (closed)

Patch from !390 (merged) is probably not working because it doesn't contain the --force flag. And in the mentioned docker issue it was commented few times that docker network disconnect is working for such case only with the --force flag.

I've prepared !432 (closed) which is adding missing flag. However for now this MR should be treat as proof of concept. It changes a vendored library which means it will be not merged to master in such form. But we can use the binary to test if this resolves the problem.

!432 (closed) will be rebased on top of !301 (merged) (replacing used go docker library with the "official" one from github.com/docker/docker/client) after Docker 1.13 will be released and !301 (merged) will be made ready to merge. Then we can replace calls updated in !432 (closed) and insert the patch into stable version.

Builds for the MR are going on. I'll post here links to download a built version after they will finish.

Binaries/packages for changes in !432 (closed) can be downloaded from:

mentioned in merge request !301 (merged)

mentioned in merge request !496 (closed)

Is the real problem a service container name colision problem?
On my project I have six builds and all using a mysql service and this problem happens much times but aleatory.

I do some tests...

All my six builds follow this skeleton:

test_XXX:
  stage: test
  image: php:5.6
  services:
    - tetraweb/mysql
  script:
    - ....

After change this to use variables:

.build_job_template: &build_definition
  stage: test
  image: php:5.6
  services: [tetraweb/mysql]

# Each six builds
test_XXX:
  <<: *build_definition
  - script:
    - ...

After doing this, apparently the problem was gone. 😅

Is there any timeline for fixing this bug? We experienced this issue in our production environment and have to restart all runner instances occasionally.

@mazong1123 as I said I think this is a Docker issue

For %v9.0 we're switching to official Docker client library and adding force to network disconnect call (https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/commit/f26c698fc2f4860c0a8f9f0d8fcc9bf10e23c1b6#7a056729f6a9c4b323621c1ee3ee2894be489989_871_891) which should resolve this problem finally.

You can test this already with v9.0.0-rc.2. At upcoming Wednesday it will be released as stable release and we will port this back to v1.10.x and v1.11.x after this.

I'm closing this issue since !301 (merged) should resolve it. If the problem will still be present after upgrading to v9.0.0 (or v1.10.x/v1.11.x including the patch) then feel free to reopen it with new reports :).

closed

mentioned in issue #2400

Docker 500 error: Service endpoint with name already exists

Designs

Child items 0

Activity

Admin message

Admin message

Docker 500 error: Service endpoint with name already exists

Activity