Runner fails to download artifacts. Fails with "... was unexpected at this time"
With gitlab-ci-multi-runner-windows-amd64.exe version 1.4.0 we are seeing a problem that was not present with 1.1.4.
When jobs of one stage complete, the artifacts from jobs of the previous stage may not be downloaded. In the attached .gitlab-ci.yml file, the job "winBuild" creates artifacts needed by the job winSikuliNvidia. The winBuild job runs to successful completion, and the artifacts are persisted to the Gitlab server. However, when the job winSikuliNvidia is launched the following occurs:
Running with gitlab-ci-multi-runner 1.4.0 (5dd9b2f) Using Shell executor... Running on VISSIMLAB-TEST...
Fetching changes...
HEAD is now at 611a4fb Testing CI Checking out 611a4fb0 as srs_osgEarth...
... was unexpected at this time.
ERROR: Build failed: exit status 255
This results in a failure of the job and stage. Examination of the file system on the runner reveals that the repo has been checked out, but none of the artifacts that should be there from the previous build stage have been fetched. It does not seem to have gotten to the point of running the yml yet, or is failing at first attempt.
Replacing the runner with version 1.1.4 corrects the problem. All runners are configured as "shell" runners. Host OS for the runners is Windows 7. Gitlab server is version 8.9.6.
Activity
@tmaczukin, this is the log I got with
gitlab-ci-multi-runner-windows-amd64.exe --debug -l "debug" run:Checking for builds... received build=332 repo_url=https://10.1.9.13:8000/vs/projects.git runner=61d6cd8c Failed to requeue the runner: builds=1 runner=61d6cd8c Running with gitlab-ci-multi-runner 1.4.0 (5dd9b2f) build=332 project=1 runner=61d6cd8c Shell configuration: environment: [] dockercommand: [] command: cmd arguments: - /Q - /C passfile: true extension: cmd build=332 project=1 runner=61d6cd8c Using Shell executor... build=332 project=1 runner=61d6cd8c Waiting for signals... build=332 project=1 runner=61d6cd8c Feeding runners to channel builds=1 Submitting build to coordinator... ok build=332 runner=61d6cd8c ... Appending trace to coordinator... ok build=332 build-log=0-11639 build-status=running code=202 runner=61d6cd8c sent-log=10565-11639 status=202 Accepted WARNING: Build failed: exit status 255 build=332 project=1 runner=61d6cd8c Submitting build to coordinator... ok build=332 runner=61d6cd8c Checking for builds... nothing runner=61d6cd8cThe "was unexpected at this time" error seems to come from invalid cmd script generated but I could not figure out how to see what gets executed.
To find a workaround I tested "powershell" and "bash" shells which are both available on the runner computer and both had problems. One of them might work in some cases, but I have a self signed certificate provided by tls-ca-file config option.
- bash fails because of gitlab-org/gitlab-ci-multi-runner#1515
- powershell gets little further, but fails to handle self signed certificate:
Downloading artifacts for compile (327)... ERROR: Failed to parse PEM in C:\gitlab-multi-runner\builds\61d6cd8c\0\vs\projects.tmp\CI_SERVER_TLS_CA_FILE ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://10.1.9.13:8000/ci/api/v1/builds/327/artifacts: Get https://10.1.9.13:8000/ci/api/v1/builds/327/artifacts: x509: certificate signed by unknown authority id=327 token=f6eb4927 WARNING: Retrying... ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://10.1.9.13:8000/ci/api/v1/builds/327/artifacts: Get https://10.1.9.13:8000/ci/api/v1/builds/327/artifacts: x509: certificate signed by unknown authority id=327 token=f6eb4927 WARNING: Retrying... ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://10.1.9.13:8000/ci/api/v1/builds/327/artifacts: Get https://10.1.9.13:8000/ci/api/v1/builds/327/artifacts: x509: certificate signed by unknown authority id=327 token=f6eb4927 FATAL: invalid argument ERROR: Build failed: exit status 1where i can download earlier version of multi runner for windows ?
Join to the question. Where can I get older versions in case of repetition of regression? @ayufan ?
Edited by username-removed-119702Copy the link to the latest Windows build and change the word "latest" to "v1.3.4" for example to get an older version. I assume the same applies to the other builds.
Latest build: https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-windows-amd64.exe
Example link to get v1.3.4: https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/v1.3.4/binaries/gitlab-ci-multi-runner-windows-amd64.exe
cc/ @ayufan @tmaczukin Can you please look at this? I have a customer in https://gitlab.zendesk.com/agent/tickets/33555 having this same issue.
I have this problem also in 1.4.x and up. It's only certain gitlab-ci-yml files that have the problem. .gitlab-ci.ymlAny ideas how we can enable logging or a more verbose output, output like "... was unexpected at this time" is all we get, and I think that's just "cmd.exe" barfing at the content of an autogenerated .cmd.
I reverted all the way back to gitlab runner 1.0.4 in this case, and this problem goes away there as well.
Running with gitlab-ci-multi-runner 1.4.1 (fae8f18) Using Shell executor... Running on DEV-HTML5... Fetching changes... HEAD is now at 08ffc51 Also change patient result data test to check the patient ID for patient 200 instead of 311. Checking out 2a582afc as master... ... was unexpected at this time. ERROR: Build failed: exit status 255Edited by username-removed-510609I think that having a script section like this might be enough to reproduce... The important bit seems to be the quote marks around the echo. If I was in a dos command prompt and I type echo "hello" I expect this output: "hello"
build: stage: build artifacts: name: "%CI_BUILD_NAME%_%CI_BUILD_REF_NAME%" paths: - src/mycorp.Authorization.Api/bin/Release/netcoreapp1.0/publish - src/mycorp.Worklist.Api/bin/Release/netcoreapp1.0/publish - src/mycorp.Fhir.Api/bin/Release/netcoreapp1.0/publish script: - call fetch.cmd - echo "build script 1 begin build.cmd" - call build.cmd - echo script 1 done - echo "build script 2 begin publish.cmd" - call publish.cmd - echo "script 2 done build task complete"
Hi everybody,
could you please check if #1590 (closed) is a "duplicate" of this issue? I think we made some progress then.
/cc @nick.thomas
could you please check if #1590 (closed) is a "duplicate" of this issue?
yes, i'm sure it's a same bug.
Added osWindows label
Reassigned to @nick.thomas
@dblessing I'll take a look at a fix. As soon as I've got a working cmd.exe ...
mentioned in issue #1590 (closed)
mentioned in merge request !284 (merged)
hi, Nick Thomas. thanks for your work, but the build relating to that merge request is failed. so what actually blocking the fix now? @nick.thomas
@svolatch the build failures are unrelated; https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/276 is working on a fix for them.
@nick.thomas !276 (merged) is green already. what are we waiting for?
@svolatch it needs final review and merge, and https://gitlab.com/gitlab-com/infrastructure/issues/396 needs doing as well. We'll get there :)
@nick.thomas where we are now? pivot feature does not work for a long time
@svolatch I'd love to get it merged for v1.6 - !284 (merged) is passing but we're short on testing of it at the moment.
Is there any chance you'd be willing to test the generated binaries for that MR? We're hoping to get integration tests working on Windows, but that might not happen in time for 1.6
https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/284/builds
-
Passing artifacts to depended stages is working as expected with binaries from !284 (merged)
Running with gitlab-ci-multi-runner 1.6.0~beta.170.g6146004 (6146004) Using Shell executor... Running on TEST_AGENT_003... Cloning repository... Cloning into 'D:/WORK/b13f96d1/0/webapps/nsr'... Checking out 769df2f8 as devUITests... Downloading artifacts for build (16288)... Downloading artifacts from coordinator... okLGTM!
Edited by username-removed-179720 @nick.thomas it works for me too. thank a lot!
Thanks for testing @svolatch @part-timeDev!
Milestone changed to %v1.6
Status changed to closed by merge request !284 (merged)
mentioned in issue #1680 (closed)
Mentioned in issue #1687
I have the latest runner, and I'm running 8.12.4-ee, but cannot get a download of a 2.2GB artifact to complete the download process after the 3 retries.
Is there a better way to debug what the problem is? I have attempted gitlab-runner --debug run, but only shows as much as the UI shows.
Running with gitlab-ci-multi-runner 1.6.1 (c52ad4f) Using Shell executor... Running on ngici-runners... Fetching changes... HEAD is now at 79b97c5 Change tag in CI yaml Checking out 79b97c55 as master... Downloading artifacts for ******** (6901)... ERROR: Downloading artifacts from coordinator... error error=unexpected EOF id=6901 responseStatus=200 OK token=zbj8aex1 WARNING: Retrying...
ERROR: Downloading artifacts from coordinator... error error=unexpected EOF id=6901 responseStatus=200 OK token=zbj8aex1 WARNING: Retrying...
ERROR: Downloading artifacts from coordinator... error error=unexpected EOF id=6901 responseStatus=200 OK token=zbj8aex1 FATAL: invalid argument
ERROR: Build failed: exit status 1Edited by username-removed-785623mentioned in issue #1941
