Gitlab-ci-multi-runner use Docker API 1.18 istead of API 1.24 or newer

@MMathiaut We're using quite old library for Docker. We're working on switching to the official one in !301 (merged). When it will be done, it should resolve your issue :).

Ok, Thank you very much @tmaczukin

Do you have any idea when the branch !301 (merged) will be integrated ?

@MMathiaut I hope that this will happen in this week. This is a big and important change that we need, and it needs to be processed early in the release process for the month (so we can test it properly).

@MMathiaut !301 (merged) was merged. If you want to test this, you can use the 9.0.0-rc.1 version which was released today (basically this is 1.11.0 with only this one change).

@tmaczukin it's still using 1.18. Looks like it's hardcoded in executors/docker/consts.go:3

Hi @tmaczukin We get the last version : https://docs.gitlab.com/runner/install/bleeding-edge.html and we have the same problem : Gitlab-ci-multi-runner use Docker API 1.18 istead of API 1.24 or newer We have to use an other version ? Other think very strange : we don't have this error when we use a linux docker image (on windows 10 pro system)

added executordocker label

Hmmm, this becomes a bigger problem.

We're setting the 1.18 API version to be compatible with most Docker versions. Because of this problem - https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2271#note_26086366 - we're thinking if we should set this up to 1.19 and deprecate older Docker versions.

Your problem is different. You have Docker servers that don't support versions older than - in this case - 1.24.

As far as I understand how Docker handles API, the minimal API version supported by a server is only a support for another endpoints. Returned data are always the same. So, for example, on my personal laptop I have such version of Docker:

$ docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb2265
 Built:        Thu Feb 23 10:58:26 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb2265
 Built:        Thu Feb 23 10:58:26 2017
 OS/Arch:      linux/amd64
 Experimental: true

Minimum version 1.12 means that I can use Docker clients that are using API versions between 1.12 and 1.26. Each time another endpoint will be used (so client using 1.12 will call /v1.12/info and client using 1.24 will call /v1.24/info). But each time returned data is the same. That means that Runner acting as a Docker client compatible with API 1.18 will be able to work properly with versions supporting 1.18 and above.

Problem described in #2271 is that 1.19 changed types of some fields so without fixing used Docker library the easiest solution is just to set used version to 1.19.

But if we will set this version to 1.24 (to support your case) then Runner will be unable to communicate with any Docker version between 1.7 (API 1.19) and 1.11 (API 1.23)! And this is not good.

I think the best solution from user perspective would be to discover minimal API version supported by used Docker server. In that case, if the version would be not less than minimal version set internally by Runner (currently 1.18), it could be set as used version. However I'm not sure how much complexity will introduce this solution and if this is good to set such version automatically without user's control.

Another option, less complex, would be to add a setting for Docker executor. Currently we can set host, tls_cert_path and tls_verify, and we could add version. This version would be validated if it's not less than minimal version set internally by Runner (currently 1.18; and we could set this version as a default for this setting). This version requires some work to be done by a user but instead of the previous one user is able to chose version which should be used.

@ayufan Any opinions? I would rather go with the second option, as a more boring solution and we can always improve this in the future to set version automatically.

@tmaczukin Thank for your response, I begin to understand the problem ^^ I'm here if you whan to test a pre version. Regards

I hate adding version as we cannot really be sure that user provided version will not break it for any reasons. I would preferably look into making the runner work with at least 1.24 and drop support for 1.18. 1.18 is already very old and limited.

@ayufan I can agree with dropping 1.18. But limiting support to 1.24 will make many versions that are compatible (at lest now) not working with Runner. And here the problem is not that Runner is not supporting 1.24 but that Runner is forcing 1.18 which is not supported by this Docker server.

@tmaczukin It's possible to have a version with an ugly fix ?

@MMathiaut I've created deprecate-docker-api-before-1-24 branch which replaces 1.18 with 1.24. You can use its artifacts until we decide what is the best way to go.

Pipeline is here: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/commits/deprecate-docker-api-before-1-24. After it will finish you will be able to find binaries and packages on S3 via environments section: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/environments/127908.

Ok @tmaczukin Thank you very much

Hi @tmaczukin , we have test the binary of your branch. We don't have the API version error anymore but we have an other problem :

Running with gitlab-ci-multi-runner 9.1.0~beta.29.g85e9d98 (85e9d98) on BOD638 (0feb1ab2) Using Docker executor with image prepack_vs2012 ... ERROR: Failed to create container volume for /builds/vdupuis Failed to import image: Error response from daemon: re-exec error: exit status 1: output: link \?\C:\ProgramData\Docker\windowsfilter\27c28daf948d6ea7ffe33f23c603955207b5851cc61f8f353c50c892f6e2db97\usr\share\terminfo\2\2621 \?\C:\ProgramData\Docker\windowsfilter\27c28daf948d6ea7ffe33f23c603955207b5851cc61f8f353c50c892f6e2db97\usr\share\terminfo\2\2621a: Cannot create a file when that file already exists. ERROR: Preparation failed: Failed to import image: Error response from daemon: re-exec error: exit status 1: output: link \?\C:\ProgramData\Docker\windowsfilter\27c28daf948d6ea7ffe33f23c603955207b5851cc61f8f353c50c892f6e2db97\usr\share\terminfo\2\2621 \?\C:\ProgramData\Docker\windowsfilter\27c28daf948d6ea7ffe33f23c603955207b5851cc61f8f353c50c892f6e2db97\usr\share\terminfo\2\2621a: Cannot create a file when that file already exists. Will be retried in 3s ...

Does that tell you something ?

I create a new issue for the previous case : https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2331

Regards

Hi,

Is it possible to regenerate the artifact please ?

I would like to make some testing, if it works on my Windows 2016 srv.

Here the output :

Running with gitlab-ci-multi-runner 9.2.0~beta.7.g0e9b7e3 (0e9b7e3)
  on WIN-QP3PPJNOITO (918ec17e)
Using Docker executor with image microsoft/nanoserver ...
ERROR: Preparation failed: Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version
Will be retried in 3s ...
Using Docker executor with image microsoft/nanoserver ...
ERROR: Preparation failed: Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version
Will be retried in 3s ...
Using Docker executor with image microsoft/nanoserver ...
ERROR: Preparation failed: Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

Thanks a lot

DR

If nobody will regenerate the artifacts .. you can always fork the repo and run the pipeline yourself (and maybe look into the code and experiment yourself).

ok no problem, thanks a lot, I'm waiting until tuesday & if nobody regenerate it, I fork & test that.

Thanks a lot.

I've tried this branch and it seems to solve the API issue properly. Unfortunately Windows Containers still don't work, see below:

Running with gitlab-ci-multi-runner 9.2.0~beta.2054.g939b340 (939b340)
  on Test Windows Containers (ff59cffe)
Using Docker executor with image microsoft/dotnet-framework:4.6.2-windowsservercore ...
ERROR: Failed to create container volume for /builds/daniel.widerin Failed to import image: Error response from daemon: re-exec error: exit status 1: output: Failed to SetFileBasicInfo failed in Win32: SetFileInformationByHandle \\?\F:\docker\windowsfilter\0f018c6ac38212442c689b29e75bf4d57531a763709b90555934ca9dd29fa612\.dockerenv: Wrong parameter. (0x1f) \\?\F:\docker\windowsfilter\0f018c6ac38212442c689b29e75bf4d57531a763709b90555934ca9dd29fa612\.dockerenv
ERROR: Preparation failed: Failed to import image: Error response from daemon: re-exec error: exit status 1: output: Failed to SetFileBasicInfo failed in Win32: SetFileInformationByHandle \\?\F:\docker\windowsfilter\0f018c6ac38212442c689b29e75bf4d57531a763709b90555934ca9dd29fa612\.dockerenv: Wrong parameter. (0x1f) \\?\F:\docker\windowsfilter\0f018c6ac38212442c689b29e75bf4d57531a763709b90555934ca9dd29fa612\.dockerenv
Will be retried in 3s ...

Any ideas?

Ahh... we've focused the discussion on Docker API version and I didn't realize that this all can be also related to Windows Containers. Not the Windows Docker binary that is accessing Linux-based Docker server, but the Windows container system supported by Docker.

Unfortunately Runner will not work with Windows Containers. Or maybe - which will be more true - we don't know how GitLab Runner with docker executor will behave on Windows Docker using Windows' native containers. However the API and docker command line tool are similar, the internals are quite different: different type of images, different type containers, different architecture. Predefined images that are bundled in GitLab Runner and are loaded in the prepare stage will mostly not work on Windows (these are Linux images which should be executed inside of Linux kernel). Also I don't know if Windows' Docker is supporting things like volumes (which is heavily used by Runner). In the discussion in #2331 I see you have another problem with shell discovering which was also designed to be running on Linux containers.

I think that with some changes (changing the way that shell is discovered, adding a possibility to configure helper image from config.toml as it is done for Kubernetes executor) it should be possible to make Runner more friendly for Windows Container and maybe even run jobs with success. But no one have tested this (Windows Docker was in beta when we've introduced support for Docker in GitLab Runner) and as @ayufan mentioned - to make this working we would need someone who have a lot of Windows experience and know how Windows Docker is behaving.

Hi @tmaczukin,

Do you think there is a chance to have a "gitlab-runner/windows/docker" working in a medium-term version ? I'm available if you want to perform some tests.

Thanks a lot and regards,

@MMathiaut We will likely seek for Accepting Merge Requests on this one, as this is not something that we can do easily.

@tmaczukin

Thanks for a good explanation. Our company builds cross-platform, so we have to support Windows too. And I think we are not alone.

It's not all that doom and gloom - at least I'm able to run GitLab runner inside Windows container, and spawn another Windows container using "docker run ..." in my build script.

For this I couldn't use Docker socket binding as on Linux: https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-socket-binding as Windows named pipe can't be shared as Docker volume. Instead I've configured Docker daemon to listen on TCP and share DOCKER_HOST env var with the container. I haven't tried DinD way, as Docker images on Windows are quite huge, and pulling them inside nested containers would be too heavy.

So, any chance we can get that branch with 1.18 > 1.24 upgrade released by 22nd of June? :)

I'd be happy to do heavy testing for you.

Still the problem with runner 9.3 :(

Running with gitlab-ci-multi-runner 9.3.0 (*****)
  on ****** (*****)
Using Docker executor with image ********* ...
ERROR: Preparation failed: Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version
Will be retried in 3s ...

mentioned in issue #2550

I'm having this issue as well, in a different context. It appears that a fix was merged into the 9.3 runner, but I'm still seeing the issue.

Any workarounds? Downgrading the docker server api in my case will not be possible :(

@roosri, what is the context? There is a fork that changes the API version, but there are still a number of other issues that prevent the Docker Executor from working on Windows. Check out #1775.

The docker api version mismatch is my current issue. I'll try and install that fork with an updated version number.

Unfortunately client libraries still not updated in v10.0.0.

Hi All

Same issue with Gitlab-runner 10.0.2 on Windows 2016 Is there any change of resolving this issue ?

Using Docker executor with image docker:git ...
ERROR: Preparation failed: Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

> PS C:\gitlab-runner> .\gitlab-runner.exe -version
> Version:      10.0.2
> Git revision: a9a76a50
> Git branch:   10-0-stable
> GO version:   go1.8.3
> Built:        Wed, 04 Oct 2017 12:39:07 +0000
> OS/Arch:      windows/amd64
> 
> PS C:\gitlab-runner> docker version
> Client:
>  Version:      17.06.1-ee-2
>  API version:  1.30
>  Go version:   go1.8.3
>  Git commit:   8e43158
>  Built:        Wed Aug 23 21:16:53 2017
>  OS/Arch:      windows/amd64
> 
> Server:
>  Version:      17.06.1-ee-2
>  API version:  1.30 (minimum version 1.24)
>  Go version:   go1.8.3
>  Git commit:   8e43158
>  Built:        Wed Aug 23 21:25:53 2017
>  OS/Arch:      windows/amd64
>  Experimental: false

Gitlab-ci-multi-runner use Docker API 1.18 istead of API 1.24 or newer

Summary

Steps to reproduce

Actual behavior

Used GitLab Runner version

Designs

Child items 0

Activity

Admin message

Admin message

Gitlab-ci-multi-runner use Docker API 1.18 istead of API 1.24 or newer

Summary

Steps to reproduce

Actual behavior

Used GitLab Runner version

Activity