gitlab-runner stuck in pending if after_script used

Summary

gitlab ci jobs do not start if after_script is used in .gitlab-ci.yml

Steps to reproduce

launch a ci job with after_script in .gitlab-ci.yml --> observe the failure launch the same ci job with after_script commented out --> observe no more failure

Actual behavior

When job is started, gets stuck in "Pending..."

Expected behavior

When job is started, job does not get stuck in "Pending"

Relevant logs and/or screenshots

from /var/log/syslog on the machine running gitlab-runner:

Mar 24 14:51:46 ip-172-31-6-70 gitlab-runner[11839]: time="2017-03-24T14:51:46Z" level=warning msg="Checking for jobs... failed" runner=508eb1fd status="Error decoding json payload json: cannot unmarshal array into Go value of type string" #012<nil>
Mar 24 14:51:46 ip-172-31-6-70 gitlab-runner[11839]: time="2017-03-24T14:51:46Z" level=warning msg="Checking for jobs... failed" runner=508eb1fd status="Error decoding json payload json: cannot unmarshal array into Go value of type string"

Environment description

Using Specific Runners. gitlab-runner is running on Ubuntu 16.04. Using GitLab Community Edition 9.0.0 cf020e6

Used GitLab Runner version

Version:      9.0.0
Git revision: 08a9e6f
Git branch:   9-0-stable
GO version:   go1.7.5
Built:        Wed, 22 Mar 2017 16:29:52 +0000
OS/Arch:      linux/386

changed title from Checking for jobs... failed; Error decoding json payload json: cannot unmashal array into Go value of type string to gitlab-runner stuck in pending if after_script used

mentioned in issue #2289 (closed)

Having the same issue using gitlab-ci-multi-runner version 9.0.0 against a 9.0.0 CE instance. Can also confirm that jobs that are part of a pipeline that does not use an after_script can run. I have a pipeline that does not have an after_script that can run jobs using this runner, my other pipelines use after_script and do not run.

Same here. Gitlab 9.0.0 CE, updated gitlab-ci-multi-runner to version 9.0.0. Added after_script to my test job -> hangs indefinitely. Downgraded the runner to 1.11.1 -> works again.

Same here too.

Originally my CI script looked like this:

image: docker:latest

services:
    - docker:dind

build:
  stage: build
  before_script:
    - docker info
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - docker pull $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG || docker pull $CI_REGISTRY_IMAGE:master || true
    - docker build --tag=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --cache-from=$CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG --cache-from=$CI_REGISTRY_IMAGE:master --file=docker/Dockerfile .
  after_script:
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG
    - docker push $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG

And all I was seeing for about 30 mins was this:

Restarting the job or the whole runner did not help.

Finally, removing before_script and after_script in .gitlab-ci.yml unclogged the runner, thanks to this issue:

image: docker:latest

services:
    - docker:dind

build:
  stage: build
  script:
    - docker info
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

    - docker pull $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG || docker pull $CI_REGISTRY_IMAGE:master || true
    - docker build --tag=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --cache-from=$CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG --cache-from=$CI_REGISTRY_IMAGE:master --file=docker/Dockerfile .

    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG
    - docker push $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG

Running with gitlab-ci-multi-runner 9.0.0 (08a9e6f)
  on our-team-runner-2 (5acb0c2f)

This is being observed on GitHost.

Same here, it's maddening since our whole pipeline broke with this bug.

Also seeing the issue @kachkaev described on 9.0.0. Jobs have a status of "Running" but no output is ever displayed and the job remains running forever. I've had to downgrade to 1.11 until this bug is fixed as it broke a significant number of our pipelines.

duplicate of #2271

+1

I implemented a pipeline using after_script and it won't run and errors out with

gitlab-ci-multi-runner[24996]: time="2017-03-28T10:56:03+02:00" level=warning msg="Checking for jobs... failed" runner=6c430768 status="Error decoding json payload json: cannot unmarshal array into Go value of type string"

When I remove the after_script, everything runs fine. I'm not even doing something special:

after_script:
  - rm -rf .tmp/output

$ gitlab-runner --version
Version:      9.0.0
Git revision: 08a9e6f
Git branch:   9-0-stable
GO version:   go1.7.5
Built:        Wed, 22 Mar 2017 16:29:52 +0000
OS/Arch:      linux/amd64
```

closed

GitLab CE/EE 9.0.1 is out so problem of after_script should be resolved now :)

Tried to manually upgrade the runner on GitHost, but it still remained 9.0.0. Will probably need to wait for another day or so. Good news though!

@kachkaev It's GitLab CE/EE, not Runner, that need to be updated. The bug was in the GitLab's API, not in Runner 😉

@tmaczukin got it, thanks for the clarification! Will still have to wait till the next midnight when our GitHost instance is usually upgraded :–)