Permission denied when using Kubernetes executor and a non-root docker image

Summary

If you are using Kubernetes executor with a docker image that runs as a non-root user, you'll get permission denied.

Steps to reproduce

Just add a job with a docker image having

USER someone

specified in the Dockerfile to .gitlab-ci.yml

For example:

test:
  image: fuww/docker-meteor:build
  script:
    - echo ok

Reproduction repository:

https://gitlab.com/fuww/gitlab-ci-kubernetes-non-root-docker-bug

Actual behavior

Before running the script of the job, it fails with permission denied.

Expected behavior

It should just run the script fine, just as with the docker executor.

Relevant logs and/or screenshots

Cloning repository...
Checking out abcdef01 as master...
Skipping Git submodules setup
/bin/bash: line 6: /namespace/project.tmp/CI_SERVER_TLS_CA_FILE: Permission denied
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1

https://gitlab.com/fuww/gitlab-ci-kubernetes-non-root-docker-bug/-/jobs/21046305

Environment description

Using a runner with Kubernetes executor (gitlab/gitlab-runner:latest). Version: 9.3.0

Configuration:

concurrent = 16
check_interval = 0

[[runners]]
  name = "..."
  url = "https://gitlab.com/"
  token = "..."
  executor = "kubernetes"
[runners.cache]
  Insecure = false
[runners.kubernetes]
  namespace = "default"
  privileged = true
  image = "docker:latest"
  disable_cache = true
  image_pull_secrets = ["regsecret"]

Used GitLab Runner version

Version:      9.3.0
Git revision: 3df822b
Git branch:   9-3-stable
GO version:   go1.7.5
Built:        Thu, 22 Jun 2017 10:57:22 +0000
OS/Arch:      linux/amd64

changed the description

We are facing the same problem.

We have the same issue with the newer version.

Version:      9.4.0
Git revision: ef0b1a6
Git branch:
GO version:   go1.8.3
Built:        Sat, 22 Jul 2017 08:21:04 +0000
OS/Arch:      linux/amd64

That last release which doesn't produce this error is gitlab/gitlab-runner:v9.1.2. It works for a simple build, but it fails when I try to build a docker image in a job, getting this error:

Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I would guess this new error is because v9.1.2 doesn't have the feature which causes the "CI_SERVER_TLS_CA_FILE: Permission denied" bug.

I can confirm that it works in v9.1.3 but v9.2.0 is broken. Only when I use root user inside container I can proceed. That really should be fixed, because that this regression is seriously impacting security.

cc @bikebilly

added regression label

@tmaczukin This sounds like a regression. Can you investigate and prioritize?

changed milestone to %v9.2

added CI/CD label

added Next Patch Release label

It may be introduced in 9.2 by !157 (merged)

assigned to @nolith

mentioned in merge request !655 (merged)

mentioned in merge request !665 (closed)

changed milestone to %10.1

changed milestone to %9.4

changed milestone to %10.0

closed via merge request !655 (merged)