Kubernetes executor fails to find default credentials on GKE
Summary
The kubernetes executor fails to find the default credentials while running on GKE. See snippet from build logs below. This error only happens sometimes - I'd class it as a "transient error".
Steps to reproduce
- Register a gitlab runner and generate token, configure to use k8s executor, etc.
- Deploy runner to a GKE cluster using config below (fill in with token from above).
- Assign runner to a project
- Run project pipeline
Actual behavior
See build log below.
Expected behavior
Expect gitlab runner to communicate successfully with google auth metadata service to fetch instance service account credentials.
Relevant logs and/or screenshots
Beginning of build log:
Running with gitlab-ci-multi-runner 9.3.0 3df822b)
on Kubernetes Runner II (xxxxxx)
ERROR: Preparation failed: error connecting to Kubernetes: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Will be retried in 3s ...
ERROR: Preparation failed: error connecting to Kubernetes: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Will be retried in 3s ...
ERROR: Preparation failed: error connecting to Kubernetes: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Will be retried in 3s ...
ERROR: Job failed (system failure): error connecting to Kubernetes: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Environment description
Kubernetes executor. Dedicated runner on own GKE cluster. Kubernetes version 1.7.3
. Pod contents:
apiVersion: v1
kind: ConfigMap
metadata:
name: gitlab-runner
data:
config.toml: |
concurrent = 4
[[runners]]
name = "Kubernetes Runner II"
url = "https://gitlab.com/"
token = "xxxxxxxxxx"
executor = "kubernetes"
[runners.kubernetes]
[[runners.kubernetes.volumes.host_path]]
name = "docker-socket"
mount_path = "/var/run/docker.sock"
read_only = false
host_path = "/var/run/docker.sock"
privileged = true
cpu_request = "200m"
memory_request = "1Gi"
service_cpu_request = "500m"
service_memory_request = "1Gi"
helper_cpu_request = "200m"
helper_memory_request = "300Mi"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gitlab-runner
spec:
replicas: 1
selector:
matchLabels:
name: gitlab-runner
template:
metadata:
labels:
name: gitlab-runner
spec:
containers:
- args:
- run
image: gitlab/gitlab-runner:latest
imagePullPolicy: Always
name: gitlab-runner
volumeMounts:
- mountPath: /etc/gitlab-runner
name: config
restartPolicy: Always
volumes:
- configMap:
name: gitlab-runner
name: config
Used GitLab Runner version
9.3.0