Provision pre-prod infrastructure and ensure successful chef runs

Validate and merge the preprod infrastructure/chef changes:

• https://gitlab.com/gitlab-com/gitlab-com-infrastructure/merge_requests/169
• https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/1175

Ensure that the preprod site functions and that it is reporting metrics to prometheus with our stock dashboards.

added pre label

changed title from [PreProd] Provision pre-prod infrastructure and ensure successful chef runs to Provision pre-prod infrastructure and ensure successful chef runs

removed assignee

changed milestone to %WoW

assigned to @jarv

The tf apply run is successful. Chef runs are currently failing because of the infamous syslog_client/_default is encrypted for you, but your private key failed to decrypt the contents. because of the multiple provisioning attempts.

Clearing out all of the hosts is not going to take too long, I think the best bet will be switch to an environment on the syslog_client vault.

We are still seeing some convergence errors, here is the current status:

$ knife status "chef_environment:pre"
418692 hours ago, web-03.sv.pre.gitlab.com.
418692 hours ago, registry01.sv.pre.gitlab.com.
418692 hours ago, altssh01.lb.gitlab.com.
418692 hours ago, redis-cache-01.db.pre.gitlab.com.
418692 hours ago, pages01.pre.gitlab.com.
418692 hours ago, web-02.sv.pre.gitlab.com.
418692 hours ago, consul-02.inf.pre.gitlab.net.
418692 hours ago, backend01.lb.pre.gitlab.com.
418692 hours ago, pgbouncer-01.db.pre.gitlab.com.
418692 hours ago, vault01.pre.gitlab.com.
418692 hours ago, fe01.pre.gitlab.com.
418692 hours ago, postgres01.db.pre.gitlab.com.
418692 hours ago, sidekiq-realtime-01.sv.pre.gitlab.com.
418692 hours ago, mailroom-01.sv.pre.gitlab.com.
418692 hours ago, redis01.pre.gitlab.com.
418692 hours ago, api-internal01.sv.pre.gitlab.com.
418692 hours ago, consul-01.inf.pre.gitlab.net.
418692 hours ago, registry02.sv.pre.gitlab.com.
418692 hours ago, vault02.pre.gitlab.com.
418692 hours ago, consul-03.inf.pre.gitlab.net.
418692 hours ago, postgres02.db.pre.gitlab.com.
41 minutes ago, deploy.pre.gitlab.com, ubuntu 16.04.
28 minutes ago, web-01.sv.pre.gitlab.com, ubuntu 16.04.
26 minutes ago, api-01.sv.pre.gitlab.com, ubuntu 16.04.
25 minutes ago, git-01.sv.pre.gitlab.com, ubuntu 16.04.
25 minutes ago, sidekiq-asap-01.sv.pre.gitlab.com, ubuntu 16.04.
25 minutes ago, sidekiq-pipeline-01.sv.pre.gitlab.com, ubuntu 16.04.
25 minutes ago, sidekiq-besteffort-01.sv.pre.gitlab.com, ubuntu 16.04.

mentioned in issue #2941