Functional Group Update - 16/8/17
Infrastructure Functional Group Update - 16/8/17
@pcarranza @yorickpeterse @briann @andrewn
The Infrastructure Group is up for a Functional Group Update this Friday.
This will be my first presentation so I will need some extra help getting ready.
Please could you respond to this issue with your team's
- Accomplishments
- Concerns
- Stalled/need help
- Plans (As a reminder, our last FGU was 7nd July)
I'll also discuss with each of you so that I fully understand these items and don't misrepresent us.
Thanks @andrewn for the issue template
Activity
Database
Accomplishments:
- After months of hiring we managed to finally hire a 2nd database specialist
- Query timings of project issues pages were reduced by 10x, and total response timings of these pages by (only) 2x
- Loading times of the "Explore Projects" page have been significantly reduced (personal note: still waiting on the deploy to get the actual numbers, but a 2 sec reduction is expected)
- The "events" table is in the process of being migrated, allowing us to save roughly 140 GB of disk space and potentially a large amount of space used for PostgreSQL buffers
- With these changes querying event feeds can be, in the best cases, 66 times faster than before
- Sidekiq and Unicorn database connections are separated in pgbouncer. This means that a spike in Sidekiq activity won't result in Unicorn not having any available connections, improving availability
- We have an MR template for DB changes (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12355), which will hopefully make reviewing DB related changes much easier
- The database team handbook page (https://about.gitlab.com/handbook/infrastructure/database/) has been significantly improved
Concerns:
- Our failover system is still not up and running, and it's not entirely clear (to me personally at least) when this will be available
- The disks of the primary are causing problems when under high load: https://gitlab.com/gitlab-com/infrastructure/issues/2168
- pgbouncer is still running in our temporary setup instead of being based on omnibus pgbouncer
- InfluxDB is incapable of handling 25 hours of data, which means we can have accurate p95s/etc on a per day basis; this is a bit annoying when planning what to work on
Stalled/need help:
- https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2628 needs to be implemented before we can move over to omnibus pgbouncer
Plans:
- Get Greg up to speed
- Make sure our primary disks are up to snuff (though this is more of a production thing)
- Get a working failover system in production
- Set up pgbouncer using omnibus so we can get rid of our temporary setup
- Continue working on important controllers to make them faster
Gitaly
Accomplishments
-
All git clones/fetches on GitLab.com have been using Gitaly for the past week
- Will update the issue with some graphs showing performance and NFS traffic changes
-
Gitaly-Ruby: we've been focusing on running ruby code directly on the File Servers. This code is running experimentally in GL 9.5. Gitaly-Ruby saves us the effort of porting from Ruby/Rugged to Go and helps increase our velocity so that GitLab.com can be running without NFS sooner.
-
We're well on track for exceeding our OKR goal of 25 migrations in the quarter
- Will update the issue with graphs
Concerns
-
Resourcing: GitLab needs Gitaly to be delivered faster. We're considering bringing more people onto the project.
-
File Server Vertical Scaling. Gitaly is handling more operations with every release, but until we're off NFS, we're unable to scale horizontally. This means that git operations are being concentrated on our 12 file servers. Our concern is that they have enough capacity to handle all git traffic until we can turn off NFS. Once NFS is off, we can start scaling horizontally, so this issue will resolve itself.
Plans
- Migrations, migrations, migrations, migrations! Deliver migrations faster, deliver more migrations
-
Security
Accomplishments
- Package signing (active on 9.5): https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2537
- Remote access now requires using the VPN.
- Wider deployment of Intrusion Detection.
- GitLab static-analysis gem merged to GitLab CE+EE: https://gitlab.com/gitlab-org/gitlab-ce/issues/25513
- External security audits executed and found a critical vulnerability in
git: https://gitlab.com/gitlab-org/gitlab-ce/issues/35212 - Protections against denial-of-service via Regex: https://gitlab.com/gitlab-org/gitlab-ce/issues/24570
Concerns
- Increasing levels of CI abuse.
- Large security issues, those that require extensive development work, are languishing.
- Host headers: https://gitlab.com/gitlab-org/gitlab-ce/issues/17877
- Content Security Policy: https://gitlab.com/gitlab-org/gitlab-ce/issues/27094
- Private tokens: https://gitlab.com/gitlab-org/gitlab-ce/issues/29810, https://gitlab.com/gitlab-org/gitlab-ce/issues/28442
- Disabling inline images: https://gitlab.com/gitlab-org/gitlab-ce/issues/27304
Plans
- Auditing improvements.
- Investigations of incidents such as account lockouts, repository access, group membership changes, etc. should not take more than a few minutes.
- Improvements have been made to auditing inside the application, now let's improve external logging.
- Any question of the form "Who did what, when?" should be much easier to answer.
- Big improvements to our Vulnerability Scanning infrastructure.
- Automated, thorough, dynamic web scanning for each release.
Edited by Brian Neel@briann thanks for adding; @sitschner please do include Security for this iteration. But feel free to ask Brian to present that part :-) I think it makes sense to have a separate FGU for Security when it is a larger team. For now, Infra + Security makes sense.
Thanks for putting the FGU together @sitschner, and thanks everyone for input! To clean up the slides prior to publication (and as a note for future updates), can each team lead please make sure to include links to the relevant issues in the slides?
@ernstvn relevant issues updated on slides. Let me know if you have any other recommendations/feedback/requirements for publication.
