api-05 out of space

api-05 alerted for low disk space. I found that there was a /var/log/gitlab/gitlab-rails/api_json.log was 22GB. Nothing in the log was particularly interesting, it all seemed normal. Upon investigation it seems that logrotate has not been working for that log file, others on the same host were rotating fine. @briann noticed that logs for other things on the host were failing because it was out of disk space.

# ps -ax|grep svlog
 7053 ?        Ss     0:21 runsvdir -P /opt/gitlab/service log: ...................................................................................................................................................................svlogd: warning: out of disk space, delete: /var/log/gitlab/gitlab-workhorse/@4000000059c980af09d4903c.s ......svlogd: warning: out of disk space, delete: /var/log/gitlab/gitlab-workhorse/@4000000059c9ce84019e11f4.s ................

39736 ?        Ss     5:35 runsvdir -P /etc/service log:  directory: /var/log/prometheus/node_exporter: temporary failure svlogd: fatal: no functional log directories. svlogd: warning: unable to lock directory: /var/log/prometheus/node_exporter: temporary failure svlogd: fatal: no functional log directories. svlogd: warning: unable to lock directory: /var/log/prometheus/node_exporter: temporary failure svlogd: fatal: no functional log directories. 

Investigation is ongoing.

assigned to @ahanselka

Not unusual for a machine to run out of space, of course. But I don't see any warnings for this node prior to it reaching 100%. Either there was a failing in alerting or the partition filled up very quickly. I'm leaning towards the former.

I forgot to include that the most recent log before today was:

-rw-r--r-- 1 git  git   79M Sep 21 15:35 api_json.log.1
-rw-r--r-- 1 git  git  4.9M Sep 21 15:35 api_json.log.7.gz

In looking at that data again, I just now noticed that the log was immediately rotated after space was cleared.

We now only have:

-rw-r--r--  1 git  git   12M Oct  4 20:16 api_json.log
-rw-r--r--  1 git  git  1.8M Oct  4 20:07 api_json.log.1.gz

@briann and @ahanselka looks like we are missing and alert on api disk usage. @ahanselka please ad the API machines to alerting.

@sitschner This should have been resolved with https://gitlab.com/gitlab-com/runbooks/merge_requests/369 by @pcarranza. We got rid of the base-debian role, so I assume this affected the API servers.