Verifying runner failed status=404 Not Found when using runner 9.0.0 with CE 8.16.7
Summary
If your central instance runs Gitlab CE 8.16.7 and your runners use gitlab-ci-multi-runner 9.0.0 (because you upgrade them nightly with apt-get) they will stop picking up builds. gitlab-runner verify reports failure to verify
Downgrading to an older runner version fixes the problem.
Background: Central server is maintained by someone else, so I can't upgrade it myself, but I maintain my own runners. To ensure I get any urgent security updates, I install software updates with apt-get upgrade every night.
Steps to reproduce
- Server running Gitlab CE 8.16.7
- Runner running previous version of gitlab-ci-multi-runner, registered with the server and running builds.
- Runner upgrades to gitlab-ci-multi-runner 9.0.0
Actual behavior
- Runner stops picking up builds.
- Server displays green traffic light for runner, does not warn about stuck builds.
- Investigating on runner reveals:
mtandy@gitlab-builder-mtandy:~$ sudo gitlab-runner status
gitlab-runner: Service is running!
mtandy@gitlab-builder-mtandy:~$ sudo gitlab-runner verify
Running in system-mode.
ERROR: Verifying runner... failed runner=4e77b075 status=404 Not Found
ERROR: Verifying runner... failed runner=707a0bc6 status=404 Not Found- Can fix problem by downgrading like so:
sudo apt-get install gitlab-ci-multi-runner=1.9.5
sudo apt-mark hold gitlab-ci-multi-runnerExpected behavior
Either (a) new runner versions should maintain backwards compatibility or (b) something in the wiki or documentation should provide a recommended config to automatically install non-breaking updates without installing breaking updates.
Environment description
Runners on Ubuntu 16.04 LTS. Runners configured in config.toml like:
[[runners]]
name = <snip>
url = <snip>
token = <snip>
executor = "docker"
limit = 1
[runners.docker]
tls_verify = false
image = "ubuntu:16.04"
privileged = false
disable_cache = false
volumes = ["/cache"]
dns = [<snip>]
[runners.cache]
[runners.kubernetes]
host = ""
cert_file = ""
key_file = ""
ca_file = ""
image = ""
namespace = ""
privileged = false
cpus = ""
memory = ""
service_cpus = ""
service_memory = ""Used GitLab Runner version
$ sudo gitlab-runner --version
Version: 9.0.0
Git revision: 08a9e6f
Git branch: 9-0-stable
GO version: go1.7.5
Built: Wed, 22 Mar 2017 16:29:52 +0000
OS/Arch: linux/amd64Activity
I'm seeing the same issue with GitLab CE
8.12.6after an automatic upgrade today! This is unfortunately not the first time that an auto upgrade (gitlab-runner) has broken the whole CI!🤦🏻 Edited by username-removed-416192Same here. It also gives a 401 when trying to set up a new runner. I'm running gitlab ce 9.0.0-ce.0 on Debian 8.7:
$ apt show gitlab-ce Package: gitlab-ce Priority: extra Section: misc Installed-Size: 891 MB Maintainer: GitLab Inc. <support@gitlab.com> Version: 9.0.0-ce.0 Depends: openssh-server Conflicts: gitlab-ee, gitlab Replaces: gitlab-ee, gitlab Download-Size: 289 MB SHA512: 5bb2c87409c7b1485b32aa9136e27bb13c6f92fbf3c15311c991199f69f1b91374310b0d679251d8877b13a84e140220f4d4702925b5d12ebd8b2908826d72b3 Homepage: https://about.gitlab.com/ APT-Sources: https://packages.gitlab.com/gitlab/gitlab-ce/debian/ jessie/main amd64 Packages Description: GitLab Community Edition (including NGINX, Postgres, Redis)And also running gitlab-ci-multi-runner 9.0.0. Downgrading to 1.11.0 "solved" the issue for me
@tmaczukin Could you take a look at this?
As it was mentioned in the release post - https://about.gitlab.com/2017/03/22/gitlab-9-0-released/#gitlab-runner-deprecation - Runner 9.0.0 requires at least GitLab 9.0.0 to work. This is a breaking change that was introduced with major version number upgrade.
We're currently working on improving logs to clearly say that the problem occurs because used GitLab version is not compatible with Runner's version (gitlab-org/gitlab-ci-multi-runner!528).
If you want or need to use older GitLab versions you need to stay with Runner 1.11.x. It will be supported until August 2017 (the same as CI APIv1 and APIv3 will be supported in GitLab 9.X until then).
added wontfix and removed ~58730 labels
@tmaczukin I'm sorry but that's unacceptable. Why would you provide this as an auto-update when it's locked to a certain Gitlab version. Also why would I check Gitlab release notes when I'm updating the runner, when runner has a separate release cycle and was never Gitlab version specific? Very unprofessional IMHO
Why would you provide this as an auto-update when it's locked to a certain Gitlab version
We're not providing any
auto-update. We're providing packages and it's users choice if he wants to update version manually or automatically. The question is if you are updating Runner automatically, then why not GitLab? From the other side - if you have reasons to update GitLab manually, then why you are updating Runner automatically?Please don't get this questions as personal attacs. I'm just pointing that this is not our responsibility to dictate you how to manage packages. We're providing packages, sources, binaries - multiple ways to install software. We're not forcing any management workflow because most times users have they own and don't want to be forced to anything.
Also why would I check Gitlab release notes when I'm updating the runner, when runner has a separate release cycle and was never Gitlab version specific?
This is not true:
-
Many times we were introducing features in Runner that required "GitLab at least X.Y". Also many times we were introducing features in GitLab that required "Runner at least Y.X". Requirement were always mentioned in release notes or in documentation.
-
Since May 2016 (version 1.2.0) we're releasing Runner at the same day as GitLab - each month at 22th. Since then, if there is a feature that requires compatibility between Runner and GitLab, the feature is released at the same month and described in documentation or release. It's now 11 months as we are synchronizing GitLab and GitLab Runners releases. With 9.0 we're going even further and we're synchronizing
major.minorversion number of GitLab and Runner.
As for the
why should I read release notes when I'm updating the Runner- release notes contain information about bug fixes, breaking changes, new requirements, new features. I would ask you - why not to check release notes of new version of software that one is using? I can't force you to do it, it's your decision. But for us this is the only way to notice users that something is changing, something new is added etc.I'm sorry but that's unacceptable.
I disagree. Software is growing in time and it's normal thing that at some moment breaking changes are introduced and something is no longer supported. The thing is to add such changes in predictable moments. We're following the
major.minor.patchversioning convention and for all breaking changes we're waiting for amajorversion change. For last year we were adding a lot of new features to Runner and each time we were doing this with a backward compatibility. 9.0.0 is a moment to cleanup support for things, that we don't plan support anymore.We understand that not everyone wants or can update software to newest version immediately when it is released. That's why:
- In GitLab CE/EE we're still supporting old API used by Runner. It will be supported until August 2017 and until then we will also have support for Runner 1.11. So if anyone wants to upgrade GitLab, but not Runner, he will have a supported version for next six months.
- Since Runner 1.11.x will be supported until August 2017, there will be also a working and supported version to use with older versions of GitLab CE/EE if someone doesn't want or can't update GitLab at this moment.
-
@mrtndwrd I see that you have a problem with registering Runner 9.0 with GitLab 9.0, right? In that case please create an issue. This should work and you're the first person who reports that something is wrong. Here we would need to check what's happening.
@tmaczukin Sorry, I checked it the wrong way, we're not using Gitlab 9.0.0 yet, so you don't have a problem there!
We're not providing any
auto-update. We're providing packages and it's users choice if he wants to update version manually or automatically. The question is if you are updating Runner automatically, then why not GitLab? From the other side - if you have reasons to update GitLab manually, then why you are updating Runner automatically?Okay, not auto-update, but it's part of the normal package update system, which in many cases Gitlab is not. I don't want to auto-update Gitlab as it's always buggy after release, I think I never had issues with the Gitlab runner.
I still think it's very unprofessional to break compatibility like this. FYI, nothing was said about this in gitlab-ci-runner changelog, update script or logs when it stopped working. I always read Gitlab release notes when I'm updating Gitlab, not the runner. If the gitlab-ci-multi-runner is now Gitlab version specific, why not make Gitlab provide the binary? If this means I have to always double check before running apt-get upgrade on my CI workers, I'll just remove the package source from my CI workers.
I know software is a living thing, it changes. But IMHO, in this case, the backwards compatibility should have been created at both sides.
Okay, not auto-update, but it's part of the normal package update system, which in many cases Gitlab is not. I don't want to auto-update Gitlab as it's always buggy after release, I think I never had issues with the Gitlab runner.
Like I said - we can't force users to enable or disable auto-updates.
However because of this problem we will prepare a blog post describing why GitLab Runner 9.0 is not working with GitLab versions older than 9.0 and how to pin packages so they will be not auto-updated in the future (until user will decide so). This is something that we can do.
If the gitlab-ci-multi-runner is now Gitlab version specific, why not make Gitlab provide the binary?
It is not "now". It was many times in the past, when we were introducing features that
require GitLab X.Y to work.Why GitLab is not providing the binary? Because in most cases and for many reasons, Runner is installed on a separated host than GitLab. Runner is also designed to be installed on multiple hosts that can be treat as "insecure" and that can work with one "central" GitLab installation. Also what sense would have force users to install GitLab on, for example, Windows 2012 server (on which GitLab will even not work) to have only Runner that would be used with - for example - GitLab.com?
Runner is part of GitLab's overall architecture and it may be dependent on GitLab's version (like in this case). But by design it is supposed to be distributed independently.
If this means I have to always double check before running apt-get upgrade on my CI workers, I'll just remove the package source from my CI workers.
Like I said before - this is your decision if you want to check what changed in used software or not. I can only encourage you to do so.
FYI, nothing was said about this in gitlab-ci-runner changelog, update script or logs when it stopped working. I always read Gitlab release notes when I'm updating Gitlab, not the runner.
With this I need to agree. We've mentioned this in release post, but there is nothing in changelog. And I think it should be.
I'll update the release process to include in changelog not only the list of changes (which we have currently) but to also explicitly notice that something important was changed (so for example Runner drops support for some software, for some GitLab versions etc.). With such design, 9.0.0's changelog entry could look like this:
v 9.0.0
With GitLab Runner 9.0 we're switching from GitLab's CI API v1 to GitLab's API v4 which is present in GitLab 9.0 and newer. Starting with this version Runner WILL NOT support GitLab versions older than 9.0
- Change dependency from
github.com/fsouza/go-dockerclienttogithub.com/docker/docker/client" !301 (merged) - Update docker-machine version to fix coreos provision !500 (merged)
- Cleanup windows install docs !497 (merged)
- Replace io.Copy with stdcopy.StdCopy for docker output handling !503 (merged)
- Fixes typo: current to concurrent. !508 (merged)
- Modifies autoscale algorithm example !340 (closed)
(...)
I think such change would at least give you a fast information about why Runner stopped to work.
Also please notice that we're working on a fix that will add explicit log notice saying that Runner tries to talk with not compatible GitLab version - !528 (merged). Currently this will only discover if 9.0 is used against 8.x, but I would like to prepare a general check of GitLab's version that could be used in the future. Thanks to this, next time when we introduce any changes that break compatibility with GitLab, we will be able to automatically inform user about this on any action taken with Runner.
@jerbob92 I can only say sorry to you and any other people that were affected by this problem. We've chosen to require GitLab CE/EE 9.0 with Runner 9.0 and we published this information in a standard way that was used in the past. After this discussion I can see that we could do more to inform users about such changes and I hope that with current knowledge (and planned improvements) we will minimize such problems in the future.
And since this discussion slowly becomes a polemic between our opinions I propose to cut it here. Please, don't feel anger or that your voice is omitted. I was thinking for a last few days how we could improve Runner's release process to prevent us all from such problems in the future. Mostly because of your comments.
- Change dependency from
mentioned in issue #2305 (closed)
FWIW: Our gitlab instance did not notice that the runner were dysfunctional, still showing a green state although last contact has been more then 8 days ago, while I'd expect that it shouldn't be so happy about a runner being away for that long ;)
Is this a problem that maybe already has been fixed or should I file an issue for it?
@jerbob92 Just my 50 cents as another Gitlab user: I do think your shouting about the breaking release being unprofessional is inappropriate.
I get the point that a breaking release like that is always annoying, especially if it goes without noticing. Personally, I was annoyed, too, when I ran into a problem out of this, but I also figured that there should certainly be a monitoring check if I rely on the multirunner.
But of course the administrator has to take care when upgrading a software to a new major version. Because thats basically what it is, when a software jumps from 1.x.x to 9.x.x: a major update, most certainly breaking expectations.
One could surely argue that Gitlab should maintain a stable version or alike in favor of distributions, but that would certainly be a fully different statement than calling out people unprofessional.
@aptituz I'm saying the action has been handled unprofessionally, I'm not calling out anyone personally. I'm glad Gitlab is changing stuff to prevent people from running into the same issue. Honestly, maybe it's just me, but I have never had any software break on my after an
apt upgrade. I could have checked all the from > to versions indeed, and it's something I'm going to do from now one.@tmaczukin Maybe the gitlab-runner can work as a sort of wrapper that contains config, where the Gitlab instance provides the actual executable for running the jobs?
@jerbob92 I think that in longer term we aim in making GitLab to push jobs to outside, instead of pull mode as it is now. So Runner would be effectively implemented in GitLab.
