Auto/Integrated DevOps
Having an integrated product brings emergent benefits. We should provide best practices in an easy and default way, and market these under the Auto DevOps brand. The following features can be part of this:
- Auto create: https://gitlab.com/gitlab-org/gitlab-ce/issues/26388
-
Auto build (with buildpack): https://docs.gitlab.com/ee/ci/autodeploy/index.html -
Auto deploy (deploy): https://docs.gitlab.com/ee/ci/autodeploy/index.html -
Auto CI (using Heroku CI): https://gitlab.com/gitlab-org/gitlab-ce/issues/26941 -
Auto quality (Codeclimate feature made by DZ): https://gitlab.com/gitlab-org/gitlab-ce/issues/4044 - Auto Parallelization (split up long running code): https://gitlab.com/gitlab-org/gitlab-ce/issues/21480, https://gitlab.com/gitlab-org/gitlab-ee/issues/1519
-
Auto retry (retry failed tests 3 times automatically, we need this): https://gitlab.com/gitlab-org/gitlab-ce/issues/3442 -
Auto monitoring (autodetect metrics): https://gitlab.com/gitlab-org/gitlab-ce/issues/30369 - Auto coverage: https://gitlab.com/gitlab-org/gitlab-ee/issues/2781
- Auto correlation (show relevant graph of metrics in MR)
- Auto rollout (incremental rollouts): https://gitlab.com/gitlab-org/gitlab-ee/issues/1660
-
Auto scale runners (of runners) - Auto link (issue with MR based on branch name, create MR from issue)
- Auto secure/scan (for vulnerabilities): Scan (https://gitlab.com/gitlab-org/gitlab-ee/issues/2592)
- Auto license (white-list/blacklist dependencies): (https://gitlab.com/gitlab-org/gitlab-ee/issues/2592)
- Auto Browser Testing (Selenium): https://gitlab.com/gitlab-org/gitlab-ce/issues/6065
- Auto update (roll out security updates, debian example)
- Auto rebuild (rebuild all containers with one command)
-
Auto start (framework templates): https://gitlab.com/gitlab-org/gitlab-ce/issues/32420 -
Auto review apps (provision a review app automatically) -
Auto registry (container registry just works with CI) - Auto artifact (easily linked artifact repository)
- Auto opsview (ops views): https://gitlab.com/gitlab-org/gitlab-ee/issues/1788
- Auto a/b (automatically a/b test MR's for GitLab Pages, 'Merge if better')
- Auto overview/flow/sequence (cross linking of CI with triggers and dependecies)
- Auto Kube (link projects to Kubernetes if deployed on Kubernetes)
- Auto scale apps (more instances when load increases)
- Auto log (aggregation): https://gitlab.com/gitlab-org/gitlab-ce/issues/37769
- Auto exception (looking for better word, handle exceptions in GitLab): https://gitlab.com/gitlab-org/gitlab-ce/issues/38092
- Auto chatops (we already have chatops that just work)
- Auto config: (shadow .gitlab-ci.yml)
-
Auto web terminal: https://docs.gitlab.com/ce/ci/environments.html#web-terminals - Auto SLO: revert changes that don't meet the Service Level Objectives
Some overall goals/priorities:
- GitLab-controlled (this is not for enterprise admins to set company defaults; we don't want to deal with merge conflicts as we improve Auto DevOps)
- Same implied
.gitlab-ci.yml
config as Auto Deploy++ - Deprecate Auto Deploy button
- Easy to “grow up” (start from implied
.gitlab-ci.yml
, edit and commit) - No (bad, opaque) "magic"
- Should do limited, but valuable pipeline if no k8s configured
- Shouldn’t be horrible if don’t have runners at all
Questions
- What about non-web apps? If someone has k8s configured, but the project is not an app, how do we fail gracefully? Build and test might still work, but deploy would be a waste and likely fail, but certainly wouldn't "work".
- Should we default to using Canaries? (Mark: No)
- Should we default to deploying to production or staging? (Mark: Production, skip staging)
- Should we support Openshift? (Mark: No)
- Do people still need CI/CD Pipelines enabled just to make external integrations work? (Kamil: No)
- Should we skip CD if
KUBE_DOMAIN
isn't specified? (Ingress and Auto Deploy will fail without it) (Mark: Yes)
Flow
- User creates project
- User pushes code to repo
- User gets web notification that Pipeline passed if have UI open or see pipeline status on commit/branch/MR (or email notification if pipeline failed)
- User adds k8s cluster creds to project
- User adds Prometheus to k8s cluster
- User adds prometheus to project
- User re-runs pipeline, production app is created
- User creates topic branch and pushes code
- User gets notification that CI passed, sees code quality delta and review app in MR widget
Proposal
Current plan for the first iteration of Auto DevOps that will be shipped in %10.0 is at #37115.
Old plan
For first iteration, ship:
* Auto Build (with buildpack): https://docs.gitlab.com/ee/ci/autodeploy/index.html
* Auto Code Quality: https://docs.gitlab.com/ee/ci/examples/code_climate.html
* Auto Deploy: https://docs.gitlab.com/ee/ci/autodeploy/index.html
* Auto Review Apps: https://docs.gitlab.com/ee/ci/review_apps/index.html
* Auto Monitoring: https://docs.gitlab.com/ce/ci/environments.html#monitoring-environments
* Stretch Auto CI: https://gitlab.com/gitlab-org/gitlab-ce/issues/26941
The modifications needed are:
-
Implied .gitlab-ci.yml
(https://gitlab.com/gitlab-org/gitlab-ce/issues/34777) -
Make Kubernetes deploy actions optional: https://gitlab.com/gitlab-org/gitlab-ce/issues/34785 -
Add Code Quality: https://gitlab.com/gitlab-org/gitlab-ce/issues/33266 -
Get Docker-in-Docker working on Kubernetes (with identical .gitlab-ci.yml
config as for .com shared runners) -
Stretch Get Auto Build working on GitLab.com shared runners -
Deprecate/remove the existing Auto Deploy functionality (but include template in normal .gitlab-ci.yml
template list as a way to switch to Auto DevOps when you already have a CI/CD configuration) -
Stretch Rename codeclimate to codequality: https://gitlab.com/gitlab-org/gitlab-ee/issues/2783 -
Stretch Add Auto CI (using Heroku CI): https://gitlab.com/gitlab-org/gitlab-ce/issues/26941
Future iterations
- Add Auto Coverage: https://gitlab.com/gitlab-org/gitlab-ee/issues/2781
- etc.
Note that Auto CI is a Stretch, since we can ship without it. It's hard to imagine Auto DevOps without CI, but even if all they get is Auto Code Quality, that's some value.
Links
- Implied
.gitlab-ci.yml
: https://gitlab.com/gitlab-org/gitlab-ce/issues/34777 - Do our own language detection for Auto Deploy (and other Auto DevOps features): https://gitlab.com/gitlab-org/gitlab-ce/issues/33537
- Demo for Auto DevOps: https://gitlab.com/gitlab-com/www-gitlab-com/issues/1452
- Video: https://www.youtube.com/watch?v=KGrJguM361c and its script https://gitlab.com/gitlab-com/www-gitlab-com/issues/1452
- Part of: https://gitlab.com/gitlab-org/gitlab-ce/issues/32638
- Move Kubernetes from service to page: https://gitlab.com/gitlab-org/gitlab-ce/issues/35616
Edited by username-removed-1243277