[These are not the runbooks you are looking for](https://gitlab.com/gitlab-com/runbooks)
## I'm here to see how the GitLab.com infrastructure is made
This project is really only used as an issue tracker for the
[infrastructure team](https://about.gitlab.com/handbook/infrastructure/) at GitLab.
You should read the [production architecture](production_architecture.md) then.
## But I'm here to see how the GitLab.com infrastructure is made
## Where and how to look for data
You should read the [production architecture](https://about.gitlab.com/handbook/infrastructure/production-architecture/) then.
### General System Health
### Where and how to look for data
#### Blackbox Monitoring
*[GitLab Web Status](http://performance.gitlab.net/dashboard/db/gitlab-web-status): front end perspective of GitLab. Useful to understand how GitLab.com looks from the user perspective. Use this graph to quickly troubleshoot what part of GitLab is slow.
*[GitLab Git Status](http://performance.gitlab.net/dashboard/db/gitlab-git-status): front end perspective of GitLab ssh access.
#### Public Whitebox Monitoring
We offer a monitoring infrastructure site that is publicly accessible.
This monitoring site is updated hourly with any change we make in the private one, so it is a 1:1 copy of the private dashboards.
There are some metrics that are not visible in this public site because we do not keep a copy of metrics obtained through influxdb.
*[Fleet overview](http://monitor.gitlab.net/dashboard/db/fleet-overview): useful to see the fleet status from the inside of GitLab.com. Use this graph to quickly see if the workers or the database are under heavy load, and to check load balancer bandwidth.
*[Postgres Stats](http://monitor.gitlab.net/dashboard/db/postgres-stats): useful to understand how is the database behaving in depth. Use this graph to review if we have spikes of exclusive locks, active or idle in transaction processes
*[Postgres Queries](http://monitor.gitlab.net/dashboard/db/postgres-queries) use this dashboard to understand if we have blocked or slow queries, dead tuples, etc.
*[Storage Stats](http://monitor.gitlab.net/dashboard/db/storage-stats) use this dashboard to understand storage use and performance.
#### Private Whitebox Monitor
*[Host Stats](http://performance.gitlab.net/dashboard/db/host-stats): useful to dive deep into a specific host to understand what is going on with it. Select a host from the dropdown on the top.
*[Business Stats](http://performance.gitlab.net/dashboard/db/business-stats): shows many pushes, new repos and CI builds.
*[Daily overview](http://performance.gitlab.net/dashboard/db/daily-overview): shows endpoints with amount of calls and performance metrics. Useful to understand what is slow generally.
Take a look at information about our infrastructure monitoring in the
[Source](https://docs.google.com/drawings/d/1MqoemFRdoLm3_p5aKBhzblZM872F1R-tWdoOR5xMQpE/edit), GitLab internal use only
## Azure
The main portion of GitLab.com is hosted on Microsoft Azure. We have
the following servers there.
* 5 HAProxy load balancers for GitLab.com
* 2 HAProxy load balancers for GitLab Pages
* 2 HAProxy nodes for altssh.GitLab.com
* 22 front-end nodes of which:
* 4 are Web nodes
* 8 are API nodes
* 10 are Git nodes
* 10 Sidekiq nodes
* 4 PostgreSQL servers
* 5 Redis servers
* 3 Prometheus servers
* 5 NFS servers
Note that these numbers can fluctuate to adapt to the platform needs.
We also use availability sets to ensure that a minimum number of servers in each
group are available at any given time. This ensures that Azure will not reboot
all instances in the same availability set at the same time for anything that
is planned.
All our servers run the latest Ubuntu LTS unless there is a specific need to do otherwise. Every server is configured with a fully fledged set of firewall rules for increased security.
### Load Balancers
We utilize Azure load balancers in front of our HAProxy nodes. This allows us to leverage on the Azure infrastructure for HA as well as [taking advantage of the power of HAProxy](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy).
Additionally, we utilize an Azure load balancer to manage PostgreSQL failovers.
* The GitLab.com load balancer pool serves git over ssh, git over https, http and https traffic.
* The GitLab Pages load balancer serves http and https.
* The AltSSH load balancer serves [git on port 443](https://about.gitlab.com/2016/02/18/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/) and translates it to port 22 on the back-end.
### Service nodes
Different services have different resource utilization patterns so we use a variety of instance types across our service nodes that are consistent for each group. We have recently isolated traffic by type on dedicated pools of nodes. We hope you noticed the performance improvement.
## Digital Ocean
Digital Ocean houses several servers that do not need to directly interact
with our main infrastructure. There are many of these that do a variety of
things, however not all will be listed here.
The primary things on Digital Ocean at this time are:
* Chef Configuration Management Servers
* Blackbox monitoring servers
* Shared runner managers
* Runner cache servers
* ELK servers
## AWS
We host our DNS with route53 and we have several EC2 instances for various
purposes. The servers you will interact with most are listed Below
* Version
* Mattermost
* License
## Google Cloud
We are currently investigating Google Cloud.
# Technology at GitLab
We use a lot of cool ([but boring](https://about.gitlab.com/handbook/#values))
technologies here at GitLab. Below is a non-exhaustive list of tech we use here.
*[Ruby](https://www.ruby-lang.org/)(probably goes without saying)
*[Chef](https://www.chef.io/chef/)
*[Prometheus](https://prometheus.io/)
*[PostgreSQL](https://www.postgresql.org/)
*[Redis](https://redis.io/)
*[ELK Stack](https://www.elastic.co/products)
*[Terraform](https://www.terraform.io)
*[Consul](https://www.consul.io)
This page has been moved to the [infrastructure handbook](https://about.gitlab.com/handbook/infrastructure/production-architecture/).
