Prometheus in Omnibus is unable to perform Kubernetes SD
I manually attempted to enable Kubernetes SD on the packaged version of Prometheus, by editing prometheus.yml
and reloading the config. The config has taken effect, however I am getting errors in the log.
Is there some reason the packaged version of Prometheus is unable to retrieve environment variables?
2017-02-03_03:22:52.38914 time="2017-02-03T03:22:52Z" level=error msg="Cannot create Kubernetes discovery: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined" source="discovery.go:84"
In checking the environment of the container, these are defined:
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_HOST=10.15.240.1
The configuration is here:
global:
scrape_interval: 15s
scrape_timeout: 15s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
- job_name: node
static_configs:
- targets: ['localhost:9100']
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)