Revamp the deployment procedure using Terraform
Building on top of using terraform to provision new hosts bootstrapped with chef we need to change our deployment procedure such as:
- We install the new version in the deployment host.
- We run migrations in this host.
- In parallel, we spin up a new fleet with the new version of the package.
- We validate that there are no remaining migrations.
- We direct the traffic in the front end load balancers to the new fleet.
- We wait until all the connections to the old fleet are drained.
- We stop mailing and sidekiq in the old fleet.
- We enable sidekiq and mailing in the new fleet.
- We execute post-deployment migrations.
- We destroy the old fleet.
Multiple advantages here:
- We have simple and immediate rollback plan.
- We will not be attacked by unpackaging or apt issues anymore.
- The switchover happens at LB level, so it's immediate
- Any ongoing connection will just continue.
- This could open the door to have canary deployment with our current setup (if the database model is the same)
Disadvantages:
- We need to scale connection to keep 2 fleets running at any time.
- We will be spending a bit more money while keeping 2 front end fleets at the same time.
cc/ @gl-infra
Edited by username-removed-274314