Docker executor fails when docker daemon log-opts is not json-file

I tried to install gitlab-ci-multi-runner with the docker executor on a computer where the docker daemon is configured with the --log-driver=fluentd. The computer is running ubuntu 14.04 (64 bits) with docker 1.9.1 (from official debian package).

Any runner started on that computer fails with the following trace visible in the 'build' page:

gitlab-ci-multi-runner 0.7.2 (998cf5d)
Using Docker executor with image ruby:2.2 ...
Pulling docker image gitlab/gitlab-runner:cache ...
Pulling docker image gitlab/gitlab-runner:build ...
Pulling docker image ruby:2.2 ...


ERROR: Build failed with: Unrecognized input header

If I change the docker daemon options to use the default log-driver of json-file then it works as expected.

I believe the reason is that the docker API call used to attach to the container to stream the logs only works with the json-file or journald log-drivers (as explained here: https://docs.docker.com/engine/reference/api/docker_remote_api_v1.20/#get-container-logs).

I then tried to configure my runner in /etc/gitlab-runner/config.toml to explicitly set log-driver=json-file line, but that option does not seem to be supported (the gitlab-runner daemon starts but the containers still have fluentd in their .HostConfig.LogConfig attribute).

concurrent = 4

[[runners]]
  url = "https://example.com/ci"
  token = "xxxxxxxxxxxxxxxxxxxxxxxxx"
  tls-skip-verify = false
  tls-ca-file = ""
  name = "myhost-ci1"
  executor = "docker"
  [runners.docker]
    image = "ruby:2.2"
    privileged = false
    volumes = ["/cache"]
    log-driver = "json-file"

Would it be possible to allow running the docker executor with the json-file driver when the docker daemon has a different default? I believe that this should be hardcoded in the executor because most custom logging drivers do not support tailing the logs.

I'm getting this error sometimes. I'm using json-file log driver. So i think it's not (only) the log driver. A retry of the affected build "fixes" this, but i can't figure out whats the problem here.

This appears to be some docker issue: https://github.com/docker/docker/issues/5327.

Hm ok, but it's closed an very old. I'm getting it randomly at the end of a build, not the start. in my case it seems to be all good, but on the end of the build I get this message. Maybe also a docker issue.

I'm seeing exactly the same behaviour. Happens on about 30% of builds. The build usually (but not always) succeeds when retrying.

I got this error after upgrading Docker to 1.10.1, fix for me was downgrade it to 1.9.1

I get the error on docker version 1.6.2. I will try upgrading to 1.9.1 and see if the issue goes away.

Milestone changed to v1.1

Added ~58730 label

Added ~159938 label

We are getting this on Docker 1.10.2. It's gotten worse too - it now happens roughly on a third of our builds. I can help debugging this if need be, since this is really hurting our usage of CI in Gitlab.

@mvdan Could you provide some example repository that would reproduce the issue?

This tends to happen often in our fdroid/fdroidserver repo. The tests generate quite a few java signing keys, which is when it fails IIRC.

See https://gitlab.com/fdroid/fdroidserver/builds - nearly all the failures are this very error.

This seems like a pretty common issue: https://github.com/docker/docker/issues/19950.

Could you check the docker debug logs and see if there's anything there?

Reassigned to @ayufan

Status changed to closed by commit aa96ed52c7a8bea6b6679d0bca682404bfed14e4

Hi, I just had a build fail two times with the message:

"ERROR: Build failed: Unrecognized input header"

after the third retry the build passed. This time it was not when pulling docker, but more close to the end of the execution of my tests.

@tmaczukin It appears as something for retries :)