Improve infrastructure documentation

Currently we are lacking quite a bit of documentation, which is slowing down the work on infrastructure in general. We need to fix this situation by just adding the right documentation in the chef repo so things are trivial to maintain (initially) and then to automate.

Missing documentation so far (that I can think of):

• Setup documentation for checkmk - including how to add plugins with a sample.
• Documentation for Elastic search
  • Setup
  • Enabling in production
  • Indexing
  • Troubleshooting
• Setup documentation for logstash, Kibana and ELK - General information on how these logs are being processed.
• General infrastructure sample with installation locations and where to get the logs when logstash is not working.
• Setup a general purpose VM in Azure
• Bootstrap a machine with chef (wherever it is running)
• Security setup (in general) including OAuth setup for all the services.
• Email documentation regarding how gmail actions work and how they are setup.
• How to troubleshoot Redis and Postgresql cluster problems
  • Where to get redis replication auth information
• RKHunter
• Document Grafana - https://gitlab.com/gitlab-com/operations/issues/167 @yorickpeterse
• Update HA documentation https://gitlab.com/gitlab-com/operations/issues/123
• Document how to create a custom package https://gitlab.com/gitlab-com/operations/issues/104
• Backup and backup restore documentation

Please add whatever else you think is tribal knowledge so we just fix it ASAP.

/cc @balameb

Added important label

@pcarranza totally agree!

Milestone changed to Scheduled

@sytses Thanks for that, if you remind any documentation missing please add it to the list.

Related: https://gitlab.com/gitlab-com/operations/issues/145

Marked the task Setup documentation for checkmk - including how to add plugins with a sample. as completed

CheckMK documentation -> https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/43

Marked the task Email documentation regarding how gmail actions work and how they are setup. as completed

Redis and Postgres -> https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/42

Thanks @tmaczukin !

Elastic search enabling documentation: https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/45

Reassigned to @jnijhof

@pcarranza @jnijhof: We should also update the GitLab.com cluster infrastructure diagram (I'm not sure is this a part of "General infrastructure sample with installation locations and where to get the logs when logstash is not working" point).

I agree @tmaczukin A quick guide on where to look for things is a must.

Marked the task Setup as completed

Marked the task Documentation for Elastic search as completed

Marked the task Setup documentation for logstash, Kibana and ELK - General information on how these logs are being processed. as completed

Marked the task Bootstrap a machine with chef (wherever it is running) as completed

I see ElasticSearch has been marked as completed. Will focus on Setup a general purpose VM in Azure for Friday 22nd.

Thanks @balameb - It looks like we have a good enough elasticsearch document: https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/elasticsearch.md

Reassigned to @pcarranza

@balameb did you had a chance to take a stab at it?

I am glad to see some improvements underway for ElasticSearch documentation. We have setup our EE instance with ELK, but are struggling to find the recommended method for keeping the data up-to-date.

Marked the task Setup a general purpose VM in Azure as completed

Milestone changed to Milestone-1

@northrup could you chime in with influxdb/grafana documentation?

Added documentation label

Reassigned to @jnijhof

@jnijhof I'm assigning this back to you so you can dedicate some time here.

Please add documentation on the CephFS setup.

Marked the task Where to get redis replication auth information as completed

Marked the task Backup and backup restore documentation as completed

@pcarranza what do you mean with General infrastructure sample with installation locations?

Didn't start on HA documentation yet but will focus on that next sprint.

Good question, let me try to explain myself.

We have one image that describes how things are built in general, but later changes have added complexity to our stack.

Take logging for example, we added rsyslog which ships data to the log host, which then is crawled by logstash and injected into elastic, etc, etc.

I miss something that explains this flow of data, a high level view, same with our chef setup and our monitoring tools, both influx/grafana and checkmk.

This could also cover how are we securing our network, etc, etc.

The whole stack (or parts of the stack in slices) in just one place so anybody new can easily find his way around the infrastructure.

Maybe that is what I meant by that.

Is this clearer?

Milestone changed to %Milestone-3

@pcarranza yes thanks!

Mentioned in issue infrastructure#727 (closed)

Marked the task Document Grafana - https://gitlab.com/gitlab-com/operations/issues/167 @yorickpeterse as completed

Marked the task Update HA documentation https://gitlab.com/gitlab-com/operations/issues/123 as completed

Marked the task Document how to create a custom package https://gitlab.com/gitlab-com/operations/issues/104 as completed

@briann could I assign you to document our security implementation (OAuth and such) so we have some form of documentation of it? no need to make it super awesome, just some pointers would be great.

That said, I'm closing this issue as most of it is done.

Status changed to closed