Failing Builds after upgrade: json: cannot unmarshal number

cc: @tmaczukin @ayufan seems like a 9.0 regression.

@kaystrobach Can you post your .gitlab-ci.yml that is causing that?

sure here you go:

stages:
  - install
  - test
  - distribute
  - deploy
  - screenshots

#before_script:
  # inject ssh host keys
  # - 'which ssh-agent || ( apt-get update -y --force-yes && apt-get install openssh-client -y --force-yes )'
  # - 'which git || ( apt-get update -y --force-yes && apt-get install git-core -y --force-yes )'
  # - eval $(ssh-agent -s)
  # - ssh-add <(echo "$SSH_PRIVATE_KEY")
  # - mkdir -p ~/.ssh
  # - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

composerInstall:
  image: docker.kay-strobach.de/docker/php:7.0
  stage: install
  variables:
    COMPOSER_CACHE_DIR: '.composer'
  script:
    - composer install --no-progress --optimize-autoloader --ignore-platform-reqs
  tags:
    - docker
  cache:
    untracked: true
    paths:
      - .composer
      - web
      - vendor
      - var
  artifacts:
    when: on_success
    expire_in: 1 hour
    paths:
      - .composer
      - web
      - vendor
      - var

runSecurityTest:
  image: docker.kay-strobach.de/docker/php:7.0
  stage: test
  script:
    - vendor/bin/security-checker security:check
  tags:
    - docker


#--------------------------------------------------------------------------------
# 3: Deployment Stage
#--------------------------------------------------------------------------------
.job_template: &standard_deployment  # Hidden job that defines an anchor named 'job_definition'
  image: docker.kay-strobach.de/docker/php:7.0
  stage: deploy
  before_script:
    # inject ssh host keys
    - 'which ssh-agent || ( apt-get update -y --force-yes && apt-get install openssh-client -y --force-yes )'
    - 'which git || ( apt-get update -y --force-yes && apt-get install git-core -y --force-yes )'
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$SSH_PRIVATE_KEY") || true
    - mkdir -p ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

    # install php zip extension
    - php -m|grep zip || (apt-get update && apt-get install -y --force-yes zlib1g-dev && docker-php-ext-install zip)

    # install composer if needed
    - which composer || (php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"; php composer-setup.php --install-dir=/usr/local/bin/ --filename=composer; php -r "unlink('composer-setup.php');")
    - export eval COMPOSER_CACHE_DIR=$CI_PROJECT_DIR/.composer
  script:
    - ./surf.phar deploy SurfBuildDescription --configurationPath=. -vv
  tags:
    - docker
  cache:
    paths:
      - Builds/surf/SurfBuildDescription/

publishToProduction:
  <<: *standard_deployment
  environment: production
  variables:
    PHP_BINARY: "php7-70LATEST-CLI -d zend_extension=/usr/local/lib/php_modules/7-70STABLE/ioncube_loader_lin_7.0.so"
    SURF_WORKSPACE: "Builds/surf/"
    TARGET_PATH: "/kunden/__hidden__/builds/shop/master/"
    TARGET_SHOP_URL: "https://www.__fqdn__.com/shop"
    REWRITE_PATH: "shop"
  only:
    - master
  when: manual

publishToShopTest:
  <<: *standard_deployment
  environment: shoptest
  variables:
    PHP_BINARY: "php7-70LATEST-CLI -d zend_extension=/usr/local/lib/php_modules/7-70STABLE/ioncube_loader_lin_7.0.so"
    SURF_WORKSPACE: "Builds/surf/"
    TARGET_PATH: "/kunden/__hidden__/builds/shop/masterShopTest/"
    TARGET_SHOP_URL: "https://www.__fqdn__.com/shoptest"
    REWRITE_PATH: "shoptest"
  only:
    - master

i removed the absolute pathes and the fqdn of the project

closed

i removed the absolute pathes and the fqdn of the project

reopened

Thanks @kaystrobach. I'll try to reproduce the error.

mentioned in issue gitlab-com/infrastructure#1430 (closed)

@tmaczukin please go ahead - if needed i can also create a repo on gitlab.com an try to reproduce it there, but we are using 9.0 CE.

@kaystrobach If you can then sure, please try to reproduce this on GitLab.com. At the moment I was unable to reproduce your problem on my local development version of GitLab on v9.0.0 tag. I'm currently going to install GitLab on a fresh host from the Omnibus package. Maybe there I will reproduce the error.

Got the same problem posting my .gitlab-ci.yaml too. There's any log/debug we can get that could be useful?

# GitLab CI configuration
# Example for a complete Puppet deployment workflow
# with separated pipelines for development, testing and production
# braches. Requires a fully configured PE-GitLab-CI runners
# environment to work.
#
stages:
  - checks
  - specs
  - version_check
  - diffs
  - integration_docker
  - integration_vagrant
  - live_runs
  - merge_request
  - promote
  - rollout
  - postcheck
  - sitedoc
  
after_script:
    - "bin/gitlab_after.sh"

# Da verificare il funzionamento
verify_code_deploy:
  stage: version_check
  script:
    - "bin/codemanager_check_deploy.sh ${CI_BUILD_REF_NAME}"
  tags:
    - puppetmaster
  when: on_success
  allow_failure: true
  environment:
    name: ${CI_BUILD_REF_NAME}
    url: https://server.domain/

# Development branch
syntax:
  stage: checks
  before_script:
    - "bin/gitlab_before.sh"
  script: "bin/puppet_check_syntax_fast.sh"
  cache:
    untracked: true
    paths:
    - modules/
  tags:
    - puppetdevel
  except:
    - testing
    - production
  only: 
    - branches

lint:
  stage: checks
  before_script:
    - "bin/gitlab_before.sh"
  script: "bin/puppet_lint.sh"
  cache:
    untracked: true
    paths:
    - modules/
  tags:
    - puppetdevel
  except:
    - testing
    - production
  only: 
    - branches
  allow_failure: true

#rake_site:
#  stage: specs
#  before_script:
#    - "bin/gitlab_before.sh"
#  script: "bin/puppet_check_rake.sh site bundle"
#  cache:
#    untracked: true
#    paths:
#    - modules/
#  tags:
#    - puppetdevel
#  only:
#    - development
#  allow_failure: true

#rake_modules:
#  stage: specs
#  before_script:
#    - "bin/gitlab_before.sh"
#  script: "bin/puppet_check_rake.sh modules bundle"
#  cache:
#    untracked: true
#    paths:
#    - modules/
#  tags:
#    - puppetdevel
#  only:
#    - development
#  allow_failure: true
# octocatalogdiff 
catalog_diff:
  stage: diffs
  before_script:
    - "bin/gitlab_before.sh"
  script: "bin/gitlab_catalog_diff.sh development"
  cache:
    untracked: true
    paths:
    - modules/
  tags:
    - puppetmaster
  only:
    - development
  allow_failure: true
  when: manual

catalog_preview:
  stage: diffs
  before_script:
    - "bin/gitlab_before.sh"
  script: "bin/gitlab_catalog_preview.sh development"
  cache:
    untracked: true
    paths:
    - modules/
  tags:
    - puppetmaster # We run this on Puppet Server to  compile catalogs
  only:
    - development
  allow_failure: true

docker_first:centos5:
  stage: integration_docker
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/docker_run.sh --image centos-5"
  cache:
    untracked: true
    paths:
    - modules/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development

docker_normal:centos5:
  stage: integration_docker
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/docker_run.sh --image centos-5 --fact FIRSTRUN=done"
  cache:
    untracked: true
    paths:
    - modules/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development

docker_first:centos6:
  stage: integration_docker
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/docker_run.sh --image centos-6"
  cache:
    untracked: true
    paths:
    - modules/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development

docker_normal:centos6:
  stage: integration_docker
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/docker_run.sh --image centos-6 --fact FIRSTRUN=done"
  cache:
    untracked: true
    paths:
    - modules/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development

docker_first:centos7:
  stage: integration_docker
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/docker_run.sh --image centos-7"
  cache:
    untracked: true
    paths:
    - modules/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development

docker_normal:centos7:
  stage: integration_docker
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/docker_run.sh --image centos-7 --fact FIRSTRUN=done"
  cache:
    untracked: true
    paths:
    - modules/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development


vagrant_destroy:
  stage: integration_vagrant
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/vagrant_node_test.sh testserver.testdomain gitlabci destroy"
  cache:
    untracked: true
    paths:
      #    - vagrant/environments/gitlabci/.vagrant/
    - modules/
    - tests/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development
  when: manual


# Da verificare il funzionamento
vagrant_checks:
  stage: integration_vagrant
  before_script:
    - "bin/gitlab_before.sh"
  script: 
    - "bin/vagrant_node_test.sh testserver.testdomain gitlabci setup"
    - "bin/vagrant_node_test.sh testserver.testdomain gitlabci drift"
  after_script:
    - "bin/gitlab_after.sh"
    - "bin/vagrant_node_test.sh testserver.testdomain gitlabci halt"
  cache:
    untracked: true
    paths:
    - modules/
    - tests/
  allow_failure: true
  tags:
    - puppetdevel
  only:
    - development
  when: manual

# Da verificare il funzionamento
#verify_code_deploy_development:
#  stage: version_check
#  script:
#    - "bin/codemanager_check_deploy.sh development"
#  tags:
#    - puppetmaster
#  when: on_success
#  only:
#    - development
#  allow_failure: true
#  environment:
#    name: development
#    url: https://testserver.testdomain

# In test should not fail
merge_request_testing:
  stage: merge_request
  script:
    - "bin/gitlab_create_merge_request.rb development testing"
  tags:
    - puppetmaster
  only:
    - development

merge_accept_testing:
  stage: promote
  script:
    - "bin/gitlab_accept_merge_request.rb development testing"
  tags:
    - puppetmaster
  only:
    - development

# On testing branch
run_puppet_on_testing:
  stage: live_runs
  before_script:
    - "bin/rundeck_job_run.sh testing"
  script:
    - "bin/puppetdb_env_query.sh testing"
  tags:
    - puppetmaster
  when: on_success
  only:
    - testing
  allow_failure: true

#verify_code_deploy_testing:
#  stage: version_check
#  script:
#    - "bin/codemanager_check_deploy.sh testing"
#  tags:
#    - puppetmaster
#  when: on_success
#  only:
#    - testing
#  allow_failure: true # TOREMOVE when fixed
#  environment:
#    name: testing
#    url: https://testserver.testdomain

# From Testing to Production it could fail as we can have more merge coming
merge_request_production:
  stage: merge_request
  script:
    - "bin/gitlab_create_merge_request.rb testing production"
  when: on_success
  tags:
    - puppetmaster
  only:
    - testing
  allow_failure: true

# Accept is manual
merge_accept_production:
  stage: promote
  script:
    - "bin/gitlab_accept_merge_request.rb testing production"
  tags:
    - puppetmaster
  when: manual
  only:
    - testing

# Production da sistemare
canary_run_on_production:
  stage: live_runs
  before_script:
    - "bin/rundeck_job_run.sh production"
  script:
    - "bin/puppetdb_env_query.sh production"
  tags:
    - puppetmaster
  when: on_success
  only:
    - production
  allow_failure: true # TOREMOVE when fixed

#verify_code_deploy_production:
#  stage: version_check
#  script:
#    - "bin/codemanager_check_deploy.sh production"
#  tags:
#    - puppetmaster
#  when: on_success
#  only:
#    - production
#  allow_failure: true # TOREMOVE when fixed
#  environment:
#    name: production
#    url: https://testserver.testdomain

# Da verificare
#run_puppet_on_production:
#  stage: rollout
#  before_script:
#    - "bin/puppet_job_run.sh production"
#  script:
#    - "bin/puppetdb_env_query.sh production"
#  tags:
#    - puppetmaster
#  when: on_success
#  only:
#    - production
#  allow_failure: true # TOREMOVE when fixed


test_post_run_on_production:
  stage: postcheck
  script:
    - "bin/puppetdb_env_query.sh production"
  tags:
    - puppetmaster
  only:
    - production
  allow_failure: true # TOREMOVE when fixed

# Creazione sito documentale
pages:
  stage: sitedoc
  script:
    - rm -rf doc public .yardoc
    - "bin/createdoc.sh"
      #    - cat README.md  |sed -e 's/\[\(.*\)\](docs\/\(.*\))/{file:\2 \1}/g' >docs/PuppetStringIndex.md
      #    - 'rake strings:generate'
      #    - 'rake strings:generate["site/**/*{.pp\,.rb} manifests/site.pp, false, false, markdown "]'
      #    - puppet strings generate site/profile/manifests/**/*.pp site/profile/functions/**/*.pp manifests/*.pp site/profile/manifests/**/**/*.pp
    - mv doc public
  tags:
    - puppetdevel
  artifacts:
    paths:
    - public
    expire_in: '1 day'
  only:
    - production

We have a similar issue with a build. But not all builds are failing.

WARNING: Checking for jobs... failed                runner=4167e060 status=Error decoding json payload json: cannot unmarshal array into Go value of type string

@fduranti Thanks for the .gitlab-ci.yml file.

I'm currently unable to reproduce the error and I think I will prepare an extended version on top of 9.0.0, which would print the json payload if the error occurs. This would help us much. But it will also print all secrets that are sent to Runner, so token, repository URL, secure variables so the output will need to be cleaned before it'll be published. I'll post an URL to the patched version in a few moments.

Hello folks, same problem with gitlab.com and I can provide a much simpler yml.

.gitlab-ci.yml

stages:
    - build
    - test

before_script:
    - pip install tox

after_script:
    - rm -rf .tox

test:
    stage: test
    script: 
        - tox -e coverage

Runner version:

Version:      9.0.0
Git revision: 08a9e6f
Git branch:   9-0-stable
GO version:   go1.7.5
Built:        Wed, 22 Mar 2017 16:29:52 +0000
OS/Arch:      linux/amd64

if there is any other useful information, just ask me.

Regards

@mbodock And this is it! I've got the error:

WARNING: Checking for jobs... failed                runner=9837b1f8 status=Error decoding json payload json: cannot unmarshal array into Go value of type string

Now I need to check why it's happening (but I still will post an URL to the patched version).

This change in our gitlab-ci config made the change between broken builds and working builds

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 74e1c62..11db26c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -54,8 +54,6 @@ before_script:
       --ignore="vendor" --extensions=php --runtime-set testVersion $(php -r "echo phpversion();" | cut -d"." -f1,2) . '
     - security-checker -n security:check
     - ./vendor/bin/phpunit
-  after_script:
-    - rm .env
 
 # Deploy templates
 .deploy: &deploy

For us a single job out of many cannot be run, it is the only job with an after_script:

lint:javascript:
  stage: test
  image: node:7
  tags:
    - docker
  script:
    - npm --silent run eslint
  after_script:
    - npm --silent run eslint-report
  artifacts:
    name: eslint-report
    expire_in: 1d
    paths:
      - eslint-report.html

gitlab-runner[5257]: time="2017-03-23T13:54:33+01:00" level=warning msg="Checking for jobs... failed" runner=f1b9ec48 status="Error decoding json payload json: cannot unmarshal array into Go value of type string" #012<nil>
gitlab-ci-multi-runner[5257]: time="2017-03-23T13:54:33+01:00" level=warning msg="Checking for jobs... failed" runner=f1b9ec48 status="Error decoding json payload json: cannot unmarshal array into Go value of type string"

Probably related: The job will stay in status "running" but should be set to failed whenever something like this happened. We realized something is wrong when to notification mails arrived about the pipeline status (as there was still this job running).

OK, I found the problem. after_script is encoded wrong:

    "steps": [
        {
            "allow_failure": false, 
            "name": "script", 
            "script": [
                "pip install tox", 
                "tox -e coverage"
            ], 
            "timeout": 3600, 
            "when": "on_success"
        }, 
        {
            "allow_failure": true, 
            "name": "after_script", 
            "script": [
                [
                    "rm -rf .tox"
                ]
            ], 
            "timeout": 3600, 
            "when": "always"
        }
    ],

script should be there an array of strings, and it is an array of arrays of strings. This will need a fix in GitLab CE/EE.

we did not use after_script - so maybe more problems?

Same issue here. If I remove after_script, build is running normally.

@kaystrobach Looking again on your initial report I now see that your issue is different than others. after_script generates invalid job payload and it fails the job even before anything is printed in trace (that's why everyone else is posting logs from Runner, not from job's trace).

Your error is related to docker executor and is raised while connecting to Docker engine (https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/v9.0.0/executors/docker/executor_docker.go#L964).

So we have two issues here. First will need a fix in GitLab CE/EE. Second will need a little more investigation and then fix in Runner or in your configuration/environment.

Can you post output of docker info and curl -s --unix-socket /var/run/docker.sock http://localhost/info? In both cases please hide the value of ID:.

Fix for the after_script problem added in gitlab-ce!10185

I confirm removing after_script from .gitlab-ci.yml does not fix issue here ; we use docker executor and get following errors :

any idea, when to expect a release? our autoupdate forced me to gitlab 9, so i'm totally screwed up :(

@kaystrobach @TheCapsLock Can you post data that I asked for at the end of https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2271#note_26060471? Your issue is still unexplained and gitlab-ce!10185 will not fix it.

@tmaczukin ; thanks for your reply ; requested command results are attached bellow :

$ docker info
Containers: 5
Images: 35
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 45
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 2
Total Memory: 1.469 GiB
Name: riboul
ID: <redacted>
WARNING: No memory limit support
WARNING: No swap limit support

My version of curl does not support option --unix-socket ; so I used following command :

$ echo -e "GET /info HTTP/1.1\r\n" | socat unix-connect:/var/run/docker.sock STDIO

giving followin result :

{"Containers":5,"Debug":0,"DockerRootDir":"/var/lib/docker","Driver":"aufs","DriverStatus":[["Root Dir","/var/lib/docker/aufs"],["Backing Filesystem","extfs"],["Dirs","45"],["Dirperm1 Supported","true"]],"ExecutionDriver":"native-0.2","ID":"<redacted>","IPv4Forwarding":1,"Images":35,"IndexServerAddress":"https://index.docker.io/v1/","InitPath":"/usr/lib/docker.io/dockerinit","InitSha1":"0ed07e86baf5b87a53f2a2a75730af8a2c24ba63","KernelVersion":"3.16.0-4-amd64","Labels":null,"MemTotal":1577287680,"MemoryLimit":0,"NCPU":2,"NEventsListener":0,"NFd":23,"NGoroutines":64,"Name":"riboul","OperatingSystem":"Debian GNU/Linux 8 (jessie)","RegistryConfig":{"IndexConfigs":{"docker.io":{"Mirrors":null,"Name":"docker.io","Official":true,"Secure":true}},"InsecureRegistryCIDRs":["127.0.0.0/8"]},"SwapLimit":0,"SystemTime":"2017-03-23T18:54:33.699996481+01:00"}

docker info:

Containers: 0
Images: 973
Storage Driver: devicemapper
 Pool Name: docker-202:0-657345-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 31.3 GB
 Data Space Total: 107.4 GB
 Data Space Available: 26.01 GB
 Metadata Space Used: 42.62 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.105 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 4.4.27-x86_64-jb1
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 2
Total Memory: 3.902 GiB
Name: git.4viewture.eu
ID: ---

curl -s --unix-socket /var/run/docker.sock http://localhost/info

curl: option --unix-socket: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

echo -e "GET /info HTTP/1.1\r\n" | socat unix-connect:/var/run/docker.sock STDIO

HTTP/1.1 200 OK
Content-Type: application/json
Job-Name: info
Date: Thu, 23 Mar 2017 18:36:57 GMT
Content-Length: 1403

{"Containers":0,"Debug":0,"DockerRootDir":"/var/lib/docker","Driver":"devicemapper","DriverStatus":[["Pool Name","docker-202:0-657345-pool"],["Pool Blocksize","65.54 kB"],["Backing Filesystem","extfs"],["Data file","/dev/loop0"],["Metadata file","/dev/loop1"],["Data Space Used","31.3 GB"],["Data Space Total","107.4 GB"],["Data Space Available","26.01 GB"],["Metadata Space Used","42.62 MB"],["Metadata Space Total","2.147 GB"],["Metadata Space Available","2.105 GB"],["Udev Sync Supported","true"],["Data loop file","/var/lib/docker/devicemapper/devicemapper/data"],["Metadata loop file","/var/lib/docker/devicemapper/devicemapper/metadata"],["Library Version","1.02.90 (2014-09-01)"]],"ExecutionDriver":"native-0.2","ID":"---","IPv4Forwarding":1,"Images":973,"IndexServerAddress":"https://index.docker.io/v1/","InitPath":"/usr/lib/docker.io/dockerinit","InitSha1":"0a6ed07baf5b8c273f2a2c75a573e880af8a24b3","KernelVersion":"4.4.27-x86_64-jb1","Labels":null,"MemTotal":4190269440,"MemoryLimit":1,"NCPU":2,"NEventsListener":0,"NFd":46,"NGoroutines":201,"Name":"git.4viewture.eu","OperatingSystem":"Debian GNU/Linux 8 (jessie)","RegistryConfig":{"IndexConfigs":{"docker.io":{"Mirrors":null,"Name":"docker.io","Official":true,"Secure":true}},"InsecureRegistryCIDRs":["127.0.0.0/8"]},"SwapLimit":1,"SystemTime":"2017-03-23T19:36:57.47235045+01:00"}

@TheCapsLock @kaystrobach Can you also post an output of docker version? In 9.0 we've changed docker library to official one (github.com/docker/docker/client/). Output of info is quite different that at mine personal computer and I'm starting to suspect that docker-engine version that you are using has older API than version supported by the library.

Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.3.3
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.3.3
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

@tmaczukin :

Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.3.3
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.3.3
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

OK, I think I know what is the problem. Your docker-engines are using API v1.18. Runner defines used API as v1.18 so requests are targeting a proper endpoint. However... https://docs.docker.com/engine/api/version-history/#v119-api-changes - here we can see, that in v1.19 version of API GET /info was changed:

GET /info The fields Debug, IPv4Forwarding, MemoryLimit, and SwapLimit are now returned as boolean instead of as an int.

In v1.18 these fields are returned as integers and that is confirmed in data posted by you above. Runner, through docker/docker/client library, is requesting /v1.18/info for the data and receive a JSON object with these fields as integers. And then it tries to decode this JSON data to a github.com/docker/docker/api/types.Info struct, where Debug, SwapLimit, MemoryLimit and IPv4Forwarding are defined as booleans!

If all changes after v1.18 would only add new fields, then JSON decoder should handle this without problems and just skip fields that are not send by docker engine. But in this case data is send, but in different format and that's why we see cannot unmarshal number into Go value of type bool.

The sad information is that I don't see an easy way to fix the issue other than upgrading docker-engine to at least 1.7 or switching back to Runner 1.11.x.

Thanks for the debug ; It would be nice if we can keep using versions provided by debian jessie at least

Is that compatible with gitlab 9???

But this will posibly hit all debian users ...

I'm currently trying to follow this guide to update my docker engine ...

https://docs.docker.com/engine/installation/linux/debian/#install-using-the-repository

docker version now prints out:

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:55:48 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:55:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

now i get different errors:

Running with gitlab-ci-multi-runner 9.0.0 (08a9e6f)
  on docker (c06ca919)
Using Docker executor with image docker.kay-strobach.de/docker/php:7.0 ...
Using docker image sha256:344bf826be5884d21a6fe8ebc1a0804910357a6c4825729835cd266ed87403ff ID=sha256:344bf826be5884d21a6fe8ebc1a0804910357a6c4825729835cd266ed87403ff for predefined container...
Pulling docker image docker.kay-strobach.de/docker/php:7.0 ...
Using docker image docker.kay-strobach.de/docker/php:7.0 ID=sha256:d778d4d40c05c502337a453102ca4c49a0d342482fb387209478498c3b5c73db for build container...
Running on runner-c06ca919-project-39-concurrent-0 via git.4viewture.eu...
Cloning repository...
Cloning into '/builds/flooridoo/Shop'...
warning: You appear to have cloned an empty repository.
Checking out 07ec8780 as master...
fatal: reference is not a tree: 07ec8780fba77ac299fe181d073b20dc54c46a73
ERROR: Job failed: exit code 1

@kaystrobach This one is known (https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2119). Run gitlab-ctl restart on your GitLab's host.

exactly ... builds are running again

so maybe you can add a check to omnibus / runner package and "magically fix" docker for people having these problems ... it's just a couple of shell commands

It would be nice if we can keep using versions provided by debian jessie at least

But this will posibly hit all debian users ...

I understand that. But we have no power to force Debian to update used provided Docker version, and for us using an old and non-official library to access Docker is not possible any more. This just doesn't scale :(

For Debian/Ubuntu (as also for RedHat/CentOS/Fedora) you can always switch to package repositories provided by Docker (as @kaystrobach linked here https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2271#note_26088059). I think this is the best way to go.

You can also stick to Runner 1.11.X. This one will be able to work with GitLab >= 9.0 up to August 2017. Until then we will still support old API used by Runner to talk with GitLab. After August API will be removed and since next GitLab release the only way to work will be to update Runner.

The root cause of this problem is in docker library and the breaking change in Docker API v1.19. The cleanest way would be to add fix in the library: if version v1.18 is used then JSON decoding should be done with translation from boolean to integer. I can open issue in docker issue tracker with such proposal, but I don't know if anyone would like to add such hack for support an API which is almost two years old

@kaystrobach What do you mean by "magically fix docker"?

As for the problem of "You appear to have cloned an empty repository" there is already discussion at omnibus-gitlab#2119 (closed) how to fix this issue.

As for the problem of Docker API v1.18 incompatibility - we can either inform that Runner needs to work with Docker at least 1.7 (and leave the code as it is with hope, that library will be updated) or switch change Docker API version used by Runner to 1.19 which should end with a clear error, that available docker-engine API is incompatible (this needs to be checked). An "automagic update" of docker-engine installed at users machine is out of scope of our package.

I mean if the wrong docker version is installed, fix it with the dependencies defined in the gitlab runner package ... that's it and nobody will recognize that problem, which is mainly cause due to often renaming the docker package in debian

@tmaczukin I agree with @kaystrobach ; I see only two correct solutions (letting it as-is will brake all gitlabs installed on those distros after a standard upgrade being run) :

prevent upgrade to gitlab v9 (by requiring specific docker engine version)
upgrade docker engine version

doing the last assumes docker engine is required only by gitlab-runner which is wrong for some of us : do we want to force adminsys guys ? I don't think so

mentioned in issue #2283 (closed)

mentioned in issue #2289 (closed)

I manually installed gitlab-runner, whats the link to manually install 1.11.1?

https://docs.gitlab.com/runner/install/linux-manually.html

@codenamemahi In the documentation page that you've linked you can find:

You can download a binary for every available version as described in Bleeding Edge - download any other tagged release.

As for 1.11.1 - you can find the list of all available packages and binaries at http://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/v1.11.1/index.html.

Helo, I'm having this issue as well and I have an up to date docker API

root@gitlab-runner01:~# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Tue Jan 31 23:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Tue Jan 31 23:35:14 2017
 OS/Arch:      linux/amd64

@dedsm Which error do you see? Because we were discussing two problems here:

There is no trace and in Runner's log after requesting for a job you can see Error decoding json payload json: cannot unmarshal array into Go value of type string. This is a bug in GitLab and it will be fixed with GitLab CE/EE 9.0.1 (gitlab-ce!10185).
Job starts but it fails with Error reading remote info: json: cannot unmarshal number into Go value of type bool. This is the problem of too old Docker API.

Since you have a quite new Docker version I suppose that you're hit by the first problem. In that case you need to wait for GitLab CE/EE 9.0.1 which should be released today or tomorrow.

mentioned in issue #2288 (closed)

Any update on when this is landing? It kind of hoses our CI pipeline.

As an aside, is the free service offered on gitlab.com considered stable? Is it the wrong place to host business-critical repos?

While we wait for a fix, this worked for me apt-get install gitlab-ci-multi-runner=1.11.1 (downgrade to the previous version)

Where can I see the release status for Gitlab-CE 9.0.1? Is this still on track for release today?

@codenamemahi 9.0.1 was tagged so it should be released today. Release will be notified with blog post on https://about.GitLab.com/blog/. For patches for current version (like this one) we're also updating GitLab.com each time a release is ready so you can follow https://twitter.com/gitlabstatus. We will mention there the start and finish of deployment.

GitLab CE/EE 9.0.1 is out so problem of after_script should be resolved now :)

Confirmed, working as expected again. Thank you!

mentioned in issue #2176

I have the same issue with GitLab 9.1.0 and a runner 9.1.1 with docker executor, but I have no after_script, just a before_script.

Hello,

I'm still having this issue with my project here: https://projects.adorsaz.ch/adrien/acme-dns-tiny/pipelines/150 and gitlab-ci-multi-runner 9.2.0.

I've even tried to set the gitlab-ci.yml file with only these lines:

jessietest:
  script:
    - "echo Hello world"

As you can see I've launched multiple tries and it only worked when I installed back the 1.1.11 gitlab-ci-multi-runner.

Maybe the error comes from another spot ? I've thought about the config.toml file, but I've still same error even removing my specific lines. I'm using these configurations:

concurrent = 1
check_interval = 0

[[runners]]
  name = "Debian Jessie dnspython3 1.11"
  url = "https://projects.adorsaz.ch/ci"
  token = "mytoken1"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "adt-jessie_dnspython3_1.11"
    privileged = false
    disable_cache = false
    volumes = ["/cache","/srv/opt/builds/acme-dns-tiny:/builds:rw"]
    pull_policy = "never"
  [runners.cache]

[[runners]]
  name = "Debian Jessie dnspython3 1.15-bpo"
  url = "https://projects.adorsaz.ch/ci"
  token = "mytoken2"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "adt-jessie_dnspython3_1.15-bpo"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    pull_policy = "never"
  [runners.cache]

These two docker images are local to my Debian Jessie server. I'm using these releases:

docker --version
Docker version 1.6.2, build 7c8fca2
apt show gitlab-ce
Version: 9.2.2-ce.0

Not sure if this is exactly the same issue I'm experiencing because my logs are slightly different:

Running with gitlab-ci-multi-runner 9.5.0 (413da38)
  on gitlabcinode (5a14a308)
Using Docker executor with image node:6 ...
ERROR: Preparation failed: Error reading remote info: json: cannot unmarshal number into Go struct field Info.Debug of type bool
Will be retried in 3s ...
Using Docker executor with image node:6 ...
ERROR: Preparation failed: Error reading remote info: json: cannot unmarshal number into Go struct field Info.Debug of type bool
Will be retried in 3s ...
Using Docker executor with image node:6 ...
ERROR: Preparation failed: Error reading remote info: json: cannot unmarshal number into Go struct field Info.Debug of type bool
Will be retried in 3s ...
ERROR: Job failed (system failure): Error reading remote info: json: cannot unmarshal number into Go struct field Info.Debug of type bool

Docker version 1.6.2, build 7c8fca2
gitlab-ci-multi-runner 9.5.0
GitLab 9.5.2

But all my builds have been failing for months. No upgraded has fixed this yet.

Failing Builds after upgrade: json: cannot unmarshal number

Summary

Steps to reproduce

Actual behavior

Expected behavior

Relevant logs and/or screenshots

Environment description

Used GitLab Runner version

Designs

Child items ...

Activity

Admin message

Admin message

Failing Builds after upgrade: json: cannot unmarshal number

Summary

Steps to reproduce

Actual behavior

Expected behavior

Relevant logs and/or screenshots

Environment description

Used GitLab Runner version

Activity