Index gitaly logs in logsearch cluster

This is part of https://gitlab.com/gitlab-com/infrastructure/issues/2225

In order to start enabling cross cluster search feature in ES, we're going to start indexing gitaly logs on the logsearch cluster.

gitaly daily logs are around 25G per day:

green open gitaly-2017.09.29                 OR9YJWjXTP6_-VNEV6RZTA 6 0   69563368    0  24.5gb  24.5gb
green open gitaly-2017.09.30                 Ywikf72WQhyvUggrCfcLuw 6 0   57812524    0  20.1gb  20.1gb
green open gitaly-2017.10.01                 ptj_KVAMRoC7depDs7XDyQ 6 0   56363130    0  19.5gb  19.5gb
green open gitaly-2017.10.02                 BlD06NZETNCowcqOl8vmtw 6 0   67800025    0    24gb    24gb
green open gitaly-2017.10.03                 8HI6dWgjQCKjskypYIew0g 6 0   75420733    0  27.1gb  27.1gb
green open gitaly-2017.10.04                 KkyYaA94RM6Q3_cPErrOCQ 6 0   71719896    0  25.1gb  25.1gb
green open gitaly-2017.10.05                 _BlYZr7vRwiaZ0bvbVlCHg 6 0   70330365    0  24.8gb  24.8gb

In a week, that would be ~175GB of indices. If we split shards in 10GB, we get ~20 shards per week.

This is a better way of handling less shards that on a weekly basis. We have 6 now per day, so that is 42 per week.

We're going to adapt the template for gitaly to reflect this.

Changes done here: https://gitlab.com/gitlab-cookbooks/gitlab-elk/merge_requests/76

mentioned in merge request gitlab-cookbooks/gitlab-elk!77 (merged)

Applied some minor changes here too https://gitlab.com/gitlab-cookbooks/gitlab-elk/merge_requests/77

Template settings for the weekly gitaly index:

    "settings": {
        "index": {
            "number_of_shards": "20",
            "number_of_replicas": "1",
            "refresh_interval": "15s"
        }
    },

After applying the changes, we're now indexing on the logsearch cluster:

health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   gitaly-2017.40 Sv8GV5soT6KJ_zkRX26GhA  20   1    2512638            0        2gb            1gb

closed