Build a staging environment for developers only

On Azure.

This should have the following:

• 2 worker nodes
• 1 Database server (that can be a bit smaller that the one we have)
• 1 small NFS/Ceph server/cluster (we could use the one that is in dev.gitlab.org)
• A restore of the database backup (consider restoring every weekend automatically, so we also test database backup restore)
• The git data of gitlab-com and gitlab-org in the storage up to date with the database restore
• performance metrics wired to it so we can track progress and behaviour
• Some way to deploy in this environment that is easy for developers - I can take the idea of having only release managers doing it.

cc/ @jnijhof @stanhu @DouweM @jschatz1

mentioned in issue #198 (closed)

mentioned in issue #336 (closed)

• Created roles for staging.gitlab.com:

-rw-rw-r-- 1 jeroen jeroen  983 Jun 13 10:44 gitlab-staging-backend.json
-rw-rw-r-- 1 jeroen jeroen 4453 Jun 13 10:46 gitlab-staging-base.json
-rw-rw-r-- 1 jeroen jeroen 1615 Jun 13 10:48 gitlab-staging-db.json
-rw-rw-r-- 1 jeroen jeroen 1090 Jun 13 10:50 gitlab-staging-lb.json
-rw-rw-r-- 1 jeroen jeroen 1130 Jun 13 10:48 gitlab-staging-redis.json
-rw-rw-r-- 1 jeroen jeroen 2600 Jun 13 10:47 gitlab-staging-worker.json

Let's use "East US" for staging location.

• Created network in "East US" called staging with subnet staging 192.168.0.0/24
• Create normal storage account in "East US" called gitlab1staging
• Creating nodes without any staging roles, we will apply roles after they are created:

./azure-create-node --size Standard_D4_v2 --storage gitlab1staging --net staging lb1.staging.gitlab.com
./azure-create-node --size Standard_D5_v2 --storage gitlab1staging --net staging worker1.staging.gitlab.com
./azure-create-node --size Standard_D5_v2 --storage gitlab1staging --net staging worker2.staging.gitlab.com
./azure-create-node --size Standard_D5_v2 --storage gitlab1staging --net staging db1.staging.gitlab.com
./azure-create-node --size Standard_D5_v2 --storage gitlab1staging --net staging redis1.staging.gitlab.com
./azure-create-node --size Standard_D5_v2 --storage gitlab1staging --net staging backend1.staging.gitlab.com

Awesome! @jnijhof

Please document how to restore the DB so we can then automate it.

Marked the task 2 worker nodes as completed

Marked the task 1 Database server (that can be a bit smaller that the one we have) as completed

Marked the task 1 small NFS/Ceph server/cluster (we could use the one that is in dev.gitlab.org) as completed

Marked the task Some way to deploy in this environment that is easy for developers - I can take the idea of having only release managers doing it. as completed

Waiting for SSL certificate for staging.gitlab.com

Deploy documentation: https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/staging.md

SSL certificate is in place. I will wait for the morning to restore database backup from production and to rsync gitlab-org and gitlab-com data.

To take care of scrubbing private data we could automatically run the following query after every restore:

UPDATE users SET encrypted_password = NULL, reset_password_token = NULL, authentication_token = NULL, encrypted_otp_secret = NULL, encrypted_otp_secret_iv = NULL, encrypted_otp_secret_salt = NULL, otp_backup_codes = NULL, unlock_token = NULL WHERE email NOT LIKE '%@gitlab.com'

This basically nukes private data for non @gitlab.com users. However, this does mean that if you use a different Email address your data is nuked. Alternatively we can whitelist a number of usernames, but this means we'll have to update that list every time people join/leave the company. Another option is to not nuke data for those in the gitlab-com group.

Some random thoughts:

• We should probably set RAILS_ENV=staging and add a config/environments/staging.rb file to disable things like sending emails.
• The Gitlab.com? method will return false on staging.gitlab.com, which will cause non-parity with production. I think this is only used by Devise emails right now so it's not a big deal, but something to think about.

@rspeicher that's a good one, could you open an issue for the GitLab.com? to review behaviour and use?

@pcarranza: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4637

mentioned in issue support#54 (closed)

Marked the task A restore of the database backup (consider restoring every weekend automatically, so we also test database backup restore) as completed

Marked the task The git data of gitlab-com and gitlab-org in the storage up to date with the database restore as completed

Actually I will just remove the email settings so we will not get any mail

I will focus on other urgent issues now since we do have a staging environment and documentation how to refresh the data and deploy. We should come up with a more easier way to deploy for developers and to refresh data automatically.

Assignee removed

Reassigned to @jnijhof

@jnijhof this looks like it is done, whatever is left or is an improvement to the issue just create a separate issue and close this one.

For what I read we need to setup performance metrics gathering (which can be done by any other person) and an automated way of refreshing the data.

We can live without those. So consider this delivered and close the issue

Great job!

Milestone changed to %Milestone-3

Ok well the database recovery takes to much time so I will rsync the repository and database data which is much faster after initial sync. I will setup rsync in a way so we can automate this later on.

It is an interesting test to run the whole restoration of the database to have a metric of how much time would it take to rebuild GitLab.com from backups in case of a full crash.

• Where is the cookbook for staging? It's not in https://dev.gitlab.org/groups/cookbooks/
• I created a separate Sentry project for staging.gitlab.com -- where is the DSN stored so I can change it?
• We need to disable WebHooks. @stanhu mentioned we were still triggering them for people from staging.

I ran WebHook.destroy_all on staging. Some users on IRC were wondering why they were receiving WebHooks.

We should also disable all active project services.

@rspeicher I think that the cookbooks are the same we use for production, there is not a specific one for staging.

If we need to make a change to use a different sentry (or performance) then we need to review how is that setup (and maybe write somewhere how to do it?)

DNS should be handled in route53.

Regarding the last point, could you provide a script tho manage what should be changed/destroyed before allowing everybody to log in and setting staging as available?

I would assume:

• users real names.
• users hashed password.
• users tokens.
• other applications or hooks tokens.
• webhooks in general.

Anything else?

cc/ @stanhu

@yorickpeterse I think you had a query in the works for clearing out sensitive data for people not in the gitlab-com group?

Edit: Derp, https://gitlab.com/gitlab-com/operations/issues/324#note_12428254

Right now it looks like the authorized_keys list isn't setup for git users to test push/pull. Can we use a special authorized_keys for devs?

@stanhu if we block users that are not from GitLab, this will also affect git pushes, right?

@pcarranza Yes.

I ran this script:

User.find_each do |user|
  is_employee = user.all_emails.grep(/gitlab.com$/).count > 0
  if is_employee
    puts "Skipping #{user.username}"
  else
    user.block
  end
end

We probably want to grep on /@gitlab\.com$/ otherwise something like mypersonalgitlab.com would match.

Alternatively, fetch the gitlab-com group members, and block any users that aren't in it.

@pcarranza the restore took 4 hours and some minutes...way too long

So basically after every data refresh the following should be run before staring gitlab:

WebHook.destroy_all

User.find_each do |user|
  user.block
end
Group.find_by(name: 'GitLab.com').group_members.each do | member |
  user = User.find_by(id: member.user_id)
  user.activate
end

settings = Gitlab.current_application_settings
settings.sentry_dsn = 'bla bla sentry bla'
settings.save

Cool, that's good data.

Did you checked this? @jnijhof https://gitlab.com/gitlab-com/infrastructure/issues/129

It looks like some email has been going out of staging :(

Next steps:

• Add performance monitoring
• Get a full cleaning script
• Automate backup restore and cleanup in staging

mentioned in issue infrastructure#132 (closed)

I'm closing this because we already have a staging environment in use.

Status changed to closed

Mentioned in commit pfjason/gitlab-ce@65187efa