WIP: Provide initial support for windows in docker executor
Overview
The docker executor has been updated to provide initial support for docker containers on windows. The executor works, but has some limitations:
- We have included a Dockerfile to create the windows helper image. The helper image is not compiled into the runner using bindata due to the size of Windows container. At the moment, when using the default helper, it must be present locally.
- The behavior of
getPrebuiltImage()
on windows is different from linux. If the image does not exist locally, it will try to pull from remote. Since the current name does not point to a public repository, this will always fail. - We'd like to propose changing the default helper name/tag to match a public repository. We could then alter the behavior of
getPrebuiltImage()
so that it does: 1) Looks for image locally, 2) Tries to extract from bindata (linux only), 3) Pulls from public repository (linux and windows) - The
helper_image
option has been added to the docker executor config options. This allows someone configuring the runner to point to a different helper image (it looks locally then tries to pull the specified image). This works on both linux and windows. The intention of this is to provide a publically available helper image for windows, so that users do not need to build the image from source or pull and retag into the standard helper image name. The behavior of this command is to look for the specified image locally and then try to pull from remote repository.
- The behavior of
- Services have been disabled on windows (it shows a warning if you attempt to use them). The linking functionality used to enable services does not work in windows.
- Since linking has been deprecated, we'd like to propose moving the Services functionality to not use linking. This would enable windows to support services as well.
- Windows requires Docker API version 1.24. A windows-only
consts_windows.go
file has been created to set the required API to 1.24 on windows, while leaving it at 1.18 for other platforms. - Windows containers currently have an issue where some applications (in our case, git) cannot write to mapped volumes. Our windows helper was unable to do
git clone
-type commands due to this. As a workaround, when thedocker
executor is used on windows, then internally it callsgit clone
-type commands and clones into a temporary directory. The files are then moved into the mapped volume. There is no user impact to this, it just required us to implement workarounds inshells/abstract.go
. These workarounds are windows-only. -
The build output is currently very verbose. Due to the way Windows output to stdout is handled for PowerShell, you see both the input commands and the output - this is very long, since it included setting all of the environement variables. There's also a potential pitfall if someone supplies a command that requires interaction (i.e. callingThis has been resolved. See https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/706#note_42208614.Get-Credential
), where it will not be handled properly. The ideal scenario would be for the-NonInteractive
switch invoking powershell.exe to work when sending commands (right now, it appears to be ignored if not also using-File
or-Command
) - this would solve both the stdout problem and the interaction problem.
Various fixes related to handling docker for windows have also been implemented. Where they may be impactful, they've been wrapped in a conditional check (runtime.GOOS == "windows"
).
We have performed manual testing of the executor on both windows and linux. Unit tests for the docker executor continue to pass. We've had some initial discussions about implementing windows-specific tests into the gitlab-ci.yml
file, but have not yet done so.
Build prebuilt image
We have added a script (make-windows-container.ps1
in the root of the repo) that will build the prebuilt container. Right now, due to the limitations above, this needs to be done on the runner machine (which requires the gitlab runner source code), which is not ideal. To avoid this, in the runner TOML in the docker section, you can specify helper_image
and point to our publically available image: cdhunt/gitlab-runner-helper:winx86_64-latest
.
Does this MR meet the acceptance criteria?
- Documentation has not been updated yet - we wanted to get feedback prior to completing that, in case the design needs to change.
- The build is not currently passing to due to complexity checks - I do not see obvious ways to make this simpler for the affected methods, other than refactoring portions of the code into different methods. We want to solicit feedback on this.
What are the relevant issue numbers?
Merge request reports
Activity
After some digging in to the PowerShell code base, I've found a (hacky!) way to suppress the super-verbose output. Launching
powershell.exe
with a parameter of-noninteractive
and just-
forces the expected behavior (script output is written to stdout, prompt + stdin is not written to stdout).Code has been committed for this MR with this fix.
Edited by username-removed-1463092- Resolved by username-removed-1463092
There's too much Windows-specific code in Docker executor for Windows, preventing to run Docker Containers (on Linux) for Windows. The container runtime should not be detected with
runtime.GOOS
but with asking Docker Engine what runtime it provides: Linux Containers or Windows Containers. Generally, I think that we could reduce the number of changes by generalizing them more.@ayufan Thanks for the feedback, I'll try to incorporate fixes and update later today. Your idea for not using
runtime.GOOS
and instead detecting via docker makes way more sense - we will add that as well.Latest commit (https://gitlab.com/gitlab-org/gitlab-runner/commit/16c3083a7ce117e6a4ac40110a363c3e44b13dad) should resolve
runtime.GOOS
issues as well as the docker+ssh issue. Details on resolvingruntime.GOOS
:The
Prepare()
function for the docker executor now retrieves and stores theVersion
info fromclient.ServerVersion()
. This contains anOs
property, which indicates whether the server is running windows or linux containers.The following changes have been made to replace
runtime.GOOS
:- After
Prepare()
has been called, and we know what the docker engine supports, we use theOs
property supplied by docker engine - In the case of setting executor options, they are set by default to linux-friendly defaults. After
Prepare()
has been called,setDefaultsForWindowsContainers()
is called, which checks if the docker engine is set for windows and overrides the defaults. - In agent registration, there is a switch to correctly set the default cache path. I've left this as
runtime.GOOS
, since even if you're running linux containers, you'd still want the host cache path to be windows-based (i.e.c:\gitlab-runner\cache
rather than/cache
).
@ayufan, please let me know if this is not a suitable solution - in particular, relating to leaving
runtime.GOOS
here: https://gitlab.com/cdhunt/gitlab-ci-multi-runner/blob/feature/docker-executor-windows/commands/register.go#L109.Edit: Sorry - some issues with this commit, I will amend it shortly.
Edit2: Issues have been resolved.
Edited by username-removed-1463092- After
I've reviewed the two methods that are failing cyclo checks. Commit https://gitlab.com/gitlab-org/gitlab-runner/commit/835c341afe3c8decb6d0bc2a92f0bf4b3858e804 cleans up abstract.go and reduced the complexity to 11, but is still too high to pass the check. I don't see any obvious way to reduce the complexity of either
writeCloneFetchCmds()
inabstract.go
further or to reducecreateContainer()
inexecutor_docker.go
.I could refactor into separate methods, but at least to me, there's not an obvious point where I should break apart the methods, and I didn't want to break the methods apart in non-obvious ways just to pass the cyclo checks.
If we must pass the cyclo checks, I can look further at possible points to refactor.
Edited by username-removed-1463092- dockerfiles/build/Dockerfile.windows.x86_64 0 → 100644
1 # escape=` 2 FROM microsoft/nanoserver:10.0.14393.1358 3 SHELL ["powershell.exe", "-C"] 4 5 RUN mkdir C:\temp 6 7 ADD https://github.com/git-for-windows/git/releases/download/v2.13.3.windows.1/MinGit-2.13.3-64-bit.zip C:\temp\MinGit-2.13.3-64-bit.zip 8 RUN Expand-Archive -Path C:\temp\MinGit-2.13.3-64-bit.zip -DestinationPath C:\Git 9 RUN setx PATH '%PATH%;C:\Git\cmd' 10 COPY .\gitlab-runner-helper.exe C:\gitlab-runner\gitlab-runner-helper.exe A multi-stage Dockerfile (eg. https://stefanscherer.github.io/use-multi-stage-builds-for-smaller-windows-images/) could help to build gitlab-runner.exe inside a Windows container, so no Go is needed on host to build the Docker image.