kubernetes executor caches overwritten variables from gitlab-ci.yml file
Summary
Overwritten variables seems to be cached from one job to another.
Steps to reproduce
Deploy a kubernetes gitlab-runner with the bellow configuration:
metrics_server = "0.0.0.0:9252"
concurrent = 10
check_interval = 30
[[runners]]
name = "RUNNER_NAME"
output_limit = 40960
url = "https://GITURL/ci"
token = "TOKEN"
executor = "kubernetes"
[runners.cache]
Type = "s3"
ServerAddress = "s3.amazonaws.com"
AccessKey = "ACCESS_KEY"
SecretKey = "SECRET_KEY"
BucketName = "BUCKET"
BucketLocation = "AWS_REGION"
Shared = true
[runners.kubernetes]
host = ""
image = "alpine:latest"
namespace = "namespace-a"
namespace_overwrite_allowed = ".*"
privileged = false
cpu_limit = "500m"
memory_limit = "4Gi"
service_cpu_limit = "250m"
service_memory_limit = "500Mi"
helper_cpu_limit = "250m"
helper_memory_limit = "1Gi"
cpu_request = "250m"
memory_request = "1Gi"
service_cpu_request = "100m"
service_memory_request = "250Mi"
helper_cpu_request = "100m"
helper_memory_request = "500Mi"
terminationGracePeriodSeconds = 90
service_account = "tiller"
service_account_overwrite_allowed = ".*"
Create a gitlab-ci.yml file as follows:
stages:
- deploy_sandbox_retrofit
- deploy_sandbox_helm
retrofit:
stage: deploy_sandbox_retrofit
image: alpine:latest
tags:
- test.ash1
- kubernetes
script:
- echo Hello_World_from_stage_deploy_sandbox_retrofit
- echo 'This job should use namespace-a namespace and tiller SA'
gitlab_runner:
stage: deploy_sandbox_helm
image: alpine:latest
variables:
CLUSTER: test.ash1
KUBERNETES_NAMESPACE_OVERWRITE: kube-system
KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: default
tags:
- test.ash1
- kubernetes
script:
- echo "Hello world from deploy_sandbox_helm"
- echo "This job should run from kube-system and overwrite ns from namespace-a to kube-system"
- echo "and overwrite SA from tiller to default"
K8S deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
creationTimestamp: 2017-07-07T18:43:06Z
generation: 2
labels:
app: wondering-mongoose-gitlab-runner-rbac
chart: gitlab-runner-rbac-0.3.13
heritage: Tiller
release: wondering-mongoose
name: wondering-mongoose-gitlab-runner-rbac
namespace: prd357
resourceVersion: "3305976"
selfLink: /apis/extensions/v1beta1/namespaces/prd357/deployments/wondering-mongoose-gitlab-runner-rbac
uid: 1dfa0f9f-6344-11e7-bc41-180373f69c63
spec:
replicas: 1
selector:
matchLabels:
app: wondering-mongoose-gitlab-runner-rbac
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: wondering-mongoose-gitlab-runner-rbac
spec:
containers:
- command:
- /usr/bin/dumb-init
- /entrypoint
env:
- name: CI_SERVER_URL
value: https://URL-XXXX/ci
- name: REGISTRATION_TOKEN
valueFrom:
secretKeyRef:
key: runner-registration-token
name: wondering-mongoose-gitlab-runner-rbac
- name: KUBERNETES_IMAGE
value: alpine:latest
- name: KUBERNETES_NAMESPACE
value: prd357
- name: KUBERNETES_CPU_LIMIT
value: 500m
- name: KUBERNETES_MEMORY_LIMIT
value: 4Gi
- name: KUBERNETES_CPU_REQUEST
value: 250m
- name: KUBERNETES_MEMORY_REQUEST
value: 1Gi
- name: KUBERNETES_SERVICE_CPU_LIMIT
value: 250m
- name: KUBERNETES_SERVICE_MEMORY_LIMIT
value: 500Mi
- name: KUBERNETES_SERVICE_CPU_REQUEST
value: 100m
- name: KUBERNETES_SERVICE_MEMORY_REQUEST
value: 250Mi
- name: KUBERNETES_HELPER_CPU_LIMIT
value: 250m
- name: KUBERNETES_HELPER_MEMORY_LIMIT
value: 1Gi
- name: KUBERNETES_HELPER_CPU_REQUEST
value: 100m
- name: KUBERNETES_HELPER_MEMORY_REQUEST
value: 500Mi
- name: RUNNER_TAG_LIST
value: k8s-rbac-test.ash1, kubernetes, beta, test.ash1
- name: REGISTER_RUN_UNTAGGED
value: "false"
- name: RUNNER_NAME
value: prd357
- name: KUBERNETES_SERVICE_ACCOUNT
value: tiller
- name: CONFIG_FILE_RO
value: /tmp/gitlab-runner/config.toml
- name: KUBERNETES_SERVICE_ACCOUNT_OVERWRITE_ALLOWED
value: .*
- name: RUNNER_EXECUTOR
value: kubernetes
- name: CACHE_SHARED
value: "true"
- name: CACHE_TYPE
value: s3
- name: KUBERNETES_TERMINATIONGRACEPERIODSECONDS
value: "90"
- name: RUNNER_OUTPUT_LIMIT
value: "40960"
- name: S3_ACCESS_KEY
value: ACCESS_KEY
- name: S3_BUCKET_LOCATION
value: us-east-1
- name: S3_BUCKET_NAME
value: S3_BUCKET_FOO
- name: S3_SECRET_KEY
value: SECRET_KEY_XXXXXX
- name: S3_SERVER_ADDRESS
value: s3.amazonaws.com
- name: KUBERNETES_NAMESPACE_OVERWRITE_ALLOWED
value: .*
image: gitlab/gitlab-runner:alpine-v9.2.2
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /usr/bin/pgrep
- gitlab-ci-multi
failureThreshold: 3
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: wondering-mongoose-gitlab-runner-rbac
ports:
- containerPort: 9252
name: http-metrics
protocol: TCP
readinessProbe:
exec:
command:
- /usr/bin/pgrep
- gitlab-ci-multi
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 750m
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp/gitlab-runner
name: config
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: gitlab-runner
serviceAccountName: gitlab-runner
terminationGracePeriodSeconds: 60
volumes:
- configMap:
defaultMode: 420
items:
- key: config.toml
path: config.toml
name: wondering-mongoose-gitlab-runner-rbac
name: config
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2017-07-07T18:43:07Z
lastUpdateTime: 2017-07-07T18:43:07Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 2
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Actual behavior
Job: retrofit
Running with gitlab-ci-multi-runner 9.2.2 (893d769)
on namespace-a (914d7be3)
Using Kubernetes namespace: another-namespace
Using Kubernetes executor with image alpine:latest ...
Waiting for pod another-namespace/runner-914d7be3-project-16460-concurrent-0b115t to be running, status is Pending
Waiting for pod another-namespace/runner-914d7be3-project-16460-concurrent-0b115t to be running, status is Pending
Running on runner-914d7be3-project-16460-concurrent-0b115t via wondering-mongoose-gitlab-runner-rbac-705552118-krcwl...
Cloning repository...
Cloning into '/kubernetes/cluster-config'...
Checking out cfbf27fe as dev-jesus...
Skipping Git submodules setup
$ echo Hello_World_from_stage_deploy_sandbox_retrofit
Hello_World_from_stage_deploy_sandbox_retrofit
$ echo 'This job should use namespace-a namespace and tiller SA'
This job should use namespace-a namespace and tiller SA
Job succeeded
Somehow the executor is ignoring that the Namespace should be the default namespace-a
, the SA seems to be the correct one.
Its overwriting the namespace from another completely different pipeline run, that was overwriting the namespace to another-namespace
Job: gitlab_runner
Running with gitlab-ci-multi-runner 9.2.2 (893d769)
on namespace-a (914d7be3)
Overwritting configured namespace, from another-namespace to kube-system
Overwritting configured ServiceAccount, from tiller to default
Using Kubernetes namespace: kube-system
Using Kubernetes executor with image alpine:latest ...
Waiting for pod kube-system/runner-914d7be3-project-16460-concurrent-0mb9lq to be running, status is Pending
Waiting for pod kube-system/runner-914d7be3-project-16460-concurrent-0mb9lq to be running, status is Pending
Waiting for pod kube-system/runner-914d7be3-project-16460-concurrent-0mb9lq to be running, status is Pending
Running on runner-914d7be3-project-16460-concurrent-0mb9lq via wondering-mongoose-gitlab-runner-rbac-705552118-krcwl...
Cloning repository...
Cloning into '/kubernetes/cluster-config'...
Checking out cfbf27fe as dev-jesus...
Skipping Git submodules setup
$ echo "Hello world from deploy_sandbox_helm"
Hello world from deploy_sandbox_helm
$ echo "This job should run from kube-system and overwrite ns from namespace-a to kube-system"
This job should run from kube-system and overwrite ns from namespace-a to kube-system
$ echo "and overwrite SA from tiller to default"
and overwrite SA from tiller to default
Job succeeded
Again it looks that the namespace another-namespace
got cached somewhere.
Expected behavior
Executor pods should be scheduled according configuration.
Environment description
Tectonic 1.6 Kubernetes cluster, gitlab runner alpine:v9.2.2
Gitlab CE: 9.3.5