Auto/Integrated DevOps
Having an integrated product brings emergent benefits. We should provide best practices in an easy and default way, and market these under the Auto DevOps brand. The following features can be part of this:
- Auto create: https://gitlab.com/gitlab-org/gitlab-ce/issues/26388
- Auto build (with buildpack): https://docs.gitlab.com/ee/ci/autodeploy/index.html
- Auto deploy (deploy): https://docs.gitlab.com/ee/ci/autodeploy/index.html
- Auto CI (using Heroku CI): https://gitlab.com/gitlab-org/gitlab-ce/issues/26941
- Auto quality (Codeclimate feature made by DZ): https://gitlab.com/gitlab-org/gitlab-ce/issues/4044
- Auto Parallelization (split up long running code): https://gitlab.com/gitlab-org/gitlab-ce/issues/21480, https://gitlab.com/gitlab-org/gitlab-ee/issues/1519
- Auto retry (retry failed tests 3 times automatically, we need this): https://gitlab.com/gitlab-org/gitlab-ce/issues/3442
- Auto metrics (autodetect metrics): https://gitlab.com/gitlab-org/gitlab-ce/issues/30369
- Auto coverage: https://gitlab.com/gitlab-org/gitlab-ee/issues/2781
- Auto correlation (show relevant graph of metrics in MR)
- Auto rollout (incremental rollouts): https://gitlab.com/gitlab-org/gitlab-ee/issues/1660
- Auto scale runners (of runners)
- Auto link (issue with MR based on branch name, create MR from issue)
- Auto secure/scan (for vulnerabilities): Scan (https://gitlab.com/gitlab-org/gitlab-ee/issues/2592)
- Auto license (white-list/blacklist dependencies): (https://gitlab.com/gitlab-org/gitlab-ee/issues/2592)
- Auto update (roll out security updates, debian example)
- Auto rebuild (rebuild all containers with one command)
- Auto start (framework templates): https://gitlab.com/gitlab-org/gitlab-ce/issues/32420
- Auto review apps (provision a review app automatically)
- Auto registry (container registry just works with CI)
- Auto artifact (easily linked artifact reposity)
- Auto opsview (ops views): https://gitlab.com/gitlab-org/gitlab-ee/issues/1788
- Auto a/b (automatically a/b test MR's for GitLab Pages, 'Merge if better')
- Auto overview/flow/sequence (cross linking of CI with triggers and dependecies)
- Auto Kube (link projects to Kubernetes if deployed on Kubernetes)
- Auto scale apps (more instances when load increases)
- Auto log (aggregation) https://gitlab.com/gitlab-org/gitlab-ce/issues/37769
- Auto exception (looking for better word, handle exceptions in GitLab)
- Auto chatops (we already have chatops that just work)
- Auto config: (shadow .gitlab-ci.yml)
- Auto web terminal: https://docs.gitlab.com/ce/ci/environments.html#web-terminals
- Auto SLO: revert changes that don't meet the Service Level Objectives
Some overall goals/priorities:
- GitLab-controlled (this is not for enterprise admins to set company defaults; we don't want to deal with merge conflicts as we improve Auto DevOps)
- Same implied
.gitlab-ci.ymlconfig as Auto Deploy++ - Deprecate Auto Deploy button
- Easy to “grow up” (start from implied
.gitlab-ci.yml, edit and commit) - No (bad, opaque) "magic"
- Should do limited, but valuable pipeline if no k8s configured
- Shouldn’t be horrible if don’t have runners at all
Questions:
- What about non-web apps? If someone has k8s configured, but the project is not an app, how do we fail gracefully? Build and test might still work, but deploy would be a waste and likely fail, but certainly wouldn't "work".
- Should we default to using Canaries? (Mark: No)
- Should we default to deploying to production or staging? (Mark: Production, skip staging)
- Should we support Openshift? (Mark: No)
Proposal
For first iteration, ship:
- Auto Build (with buildpack): https://docs.gitlab.com/ee/ci/autodeploy/index.html
- Auto Code Quality: https://docs.gitlab.com/ee/ci/examples/code_climate.html
- Auto Deploy: https://docs.gitlab.com/ee/ci/autodeploy/index.html
- Auto Review Apps: https://docs.gitlab.com/ee/ci/review_apps/index.html
- Auto Monitoring: https://docs.gitlab.com/ce/ci/environments.html#monitoring-environments
The modifications needed are:
-
Implied
.gitlab-ci.yml(https://gitlab.com/gitlab-org/gitlab-ce/issues/34777) - Make Kubernetes deploy actions optional: https://gitlab.com/gitlab-org/gitlab-ce/issues/34785
- Add Code Quality: https://gitlab.com/gitlab-org/gitlab-ce/issues/33266
-
Get Docker-in-Docker working on Kubernetes (with identical
.gitlab-ci.ymlconfig as for .com shared runners) - Get Auto Build working on GitLab.com shared runners
-
Deprecate/remove the existing Auto Deploy functionality (but include template in normal
.gitlab-ci.ymltemplate list as a way to switch to Auto DevOps when you already have a CI/CD configuration) - Stretch Rename codeclimate to codequality: https://gitlab.com/gitlab-org/gitlab-ee/issues/2783
- Stretch Add Auto CI (using Heroku CI): https://gitlab.com/gitlab-org/gitlab-ce/issues/26941
Future iterations can then:
- Add Auto Coverage: https://gitlab.com/gitlab-org/gitlab-ee/issues/2781
- etc.
Note that Auto CI is a Stretch, since we can ship without it. It's hard to imagine Auto DevOps without CI, but even if all they get is Auto Code Quality, that's some value.
Links
- Implied
.gitlab-ci.yml: https://gitlab.com/gitlab-org/gitlab-ce/issues/34777 - Do our own language detection for Auto Deploy (and other Auto DevOps features): https://gitlab.com/gitlab-org/gitlab-ce/issues/33537
- Demo for Auto DevOps: https://gitlab.com/gitlab-com/www-gitlab-com/issues/1452
- Video: https://www.youtube.com/watch?v=KGrJguM361c and its script https://gitlab.com/gitlab-com/www-gitlab-com/issues/1452
Designs
Relates to
- gitlab-foss #26388
- gitlab-foss #2694110.0
- gitlab-foss #40449.38
- gitlab-foss #34429.5
- gitlab-foss #324209.5
- gitlab-foss #3477710.0
- #7668Next 2-3 months
Activity
@bikebilly I totally agree.
This is a great list! Obviously it's more indicative rather than prescriptive. Some of these items aren't automatic at all, and some perhaps shouldn't be. But as a direction, it's great. I'd just like to add that "auto" isn't exactly a goal in itself, but the underlying goal is providing best practices in an easy and default way. If we just provide something automatically, but it's not actually a best practice, then it's not really delivering value. That may be obvious, but sometimes obvious things get lost.
@markpundsack good point, I added Auto build
Involving relevant team members:
- Auto create: @mydigitalself see the issue linked (note: it's assigned to you)
- Auto log: @mydigitalself our current centralised loggin
- Auto start: @mydigitalself I think this is the template repo we spoke about.
- Auto chatops: @victorwu
For auto exception, I imagine this would happen in issue. What is your thinking here @sytses?
@joshlambert Do you have a link for Auto Metrics?
Auto CI has at least three levels:
- Getting Herokuish to use CI testpacks
- Add Auto CI to Auto Deploy templates
- Enable implied
.gitlab-ci.ymlfor all projects w/o.gitlab-ci.yml(or all open source projects on gitlab.com)
As an alternative route to getting to "auto" faster, we could consider Auto Code Climate as the first implied
.gitlab-ci.yml.@ayufan also brought up a good idea for having group-level
.gitlab-ci.ymlconfigurations that would be applied to all projects in the group, unless they have their own.gitlab-ci.yml. That kind of turns the "implied" CI on its head, and makes it explicit, but default at the group level. This could go further into the instance level, where gitlab.com has a default.gitlab-ci.ymlthat runs Auto Deploy (or some portion thereof), but self-hosted installations could have different default CI.Since one of the goals of Auto DevOps, and certainly Onboarding in general, is to enable CI/CD for as many people as possible, the easiest path might be:
Since gitlab.com has shared runners, and Auto Code Quality should work for the vast majority of projects, and nice support has already started in 9.3 and will improve in subsequent releases, it provides an easier entry point to Auto DevOps.
Imagine a default, implied
.gitlab-ci.ymlthat has a single job to check code quality. Every project gets it by default. If you disable GitLab CI/CD or if you have your own.gitlab-ci.yml, then it won't run. One hiccup is that the fancy code quality reporting only works for EE, so on gitlab.com, that would be Bronze subscriptions and above. I'm not sure if we'd want to still run the analysis, but leave it as a raw JSON artifact for non-subscribers, or if we should consider moving Auto Code Quality to be a CE feature, or if we should consider giving Bronze (EES) features to all open source projects on .com.At any rate, once Auto CI ships, we can add it to the implied
.gitlab-ci.yml.When we provide k8s cluster for everyone, we can add Auto Deploy to the implied
.gitlab-ci.ymlas well.Edited by Mark Pundsackchanged milestone to %Next 2-3 months
added devops label
Added item for auto license and auto security scan. (https://gitlab.com/gitlab-org/gitlab-ee/issues/2592)
Edited by Joshua Lambertwhen clicking on
Set up CIin the project page, show the implied config instead of a blank entryThat's an interesting idea. We'll have to make it clear what exactly we're showing people, but it bridges nicely from having the magic to being able to edit it. From that perspective, maybe the button should just be "Edit CI/CD" since it's already "set up".
So then the only confusion is if people are like "wait, I didn't have a
.gitlab-ci.ymlfile in this repo, what the heck is this?"mentioned in issue #2781
mentioned in merge request !2328 (merged)
@joe.s Mark proposed Integrated DevOps in https://gitlab.com/gitlab-org/gitlab-ce/issues/34568 I'm open to that.
For now we'll keep calling this feature set Auto DevOps https://gitlab.com/gitlab-org/gitlab-ce/issues/34568#note_34291648
A couple of concerns have come up.
-
The value of Auto DevOps is to have it enabled for everyone, without configuration, so having a group-level
.gitlab-ci.ymldoesn't really solve that, if the group still needs to configure it. Also, this gives the opportunity for teams to mess up the implied configuration, and keeping it updated with future changes by us becomes really hard. For this reason, we should focus on implied.gitlab-ci.ymlrather than group or instance level.gitlab-ci.yml. -
Starting with Auto Code Quality might be cool and easy, but results in us having the implied
.gitlab-ci.ymlbe different from the Auto Deploy.gitlab-ci.yml. This is bad. Not horrible, but bad, and should be avoided.
-
Implied
.gitlab-ci.yml: https://gitlab.com/gitlab-org/gitlab-ce/issues/34777One path forward is to implement implied
.gitlab-ci.yml(https://gitlab.com/gitlab-org/gitlab-ce/issues/34777) with a modified configuration based on Auto Deploy, and then remove the existing Auto Deploy functionality.The modifications needed are:
- Make deploy actions that depend on Kubernetes optional
- Add Code Quality
Future iterations can then:
- Add CI tests
- Add Coverage
- etc.
For "Make deploy actions that depend on Kubernetes optional", I'm considering adding a capability to filter jobs using
onlyandexceptbased on existence (or value) of environment variables such asKUBE_CONFIGwhich is only present when the kubernetes service is enabled (and thus we know we have a deployable platform available). Thus, we could do something like:production: stage: production script: - some deploy script environment: name: production url: http://$CI_PROJECT_PATH_SLUG.$KUBE_DOMAIN when: manual only: - master - $KUBECONFIGEdited by Mark Pundsack@bikebilly I Created "Skip jobs based on variable existence/value": https://gitlab.com/gitlab-org/gitlab-ce/issues/34785 for discussion.
mentioned in issue #2708 (closed)
marked this issue as related to #1519
marked this issue as related to #2781
marked this issue as related to #1660
marked this issue as related to #2592
marked this issue as related to #2592
marked this issue as related to #1788
marked this issue as related to gitlab-com/www-gitlab-com#1452 (closed)
marked this issue as related to gitlab-com/www-gitlab-com#1452 (closed)
@markpundsack suggested Integrated Devops and I like it a lot. What is important to me is that we call the features Integrated X. We already have some Auto X features but I think we can rename those. In that case the list would be as below. I think that works.
- Integrated create: https://gitlab.com/gitlab-org/gitlab-ce/issues/26388
- Integrated build (with buildpack): https://docs.gitlab.com/ee/ci/Integrateddeploy/index.html
- Integrated deploy (deploy): https://docs.gitlab.com/ee/ci/Integrateddeploy/index.html
- Integrated CI (using Heroku CI): https://gitlab.com/gitlab-org/gitlab-ce/issues/26941
- Integrated quality (Codeclimate feature made by DZ): https://gitlab.com/gitlab-org/gitlab-ce/issues/4044
- Integrated Parallelization (split up long running code): https://gitlab.com/gitlab-org/gitlab-ce/issues/21480, https://gitlab.com/gitlab-org/gitlab-ee/issues/1519
- Integrated retry (retry failed tests 3 times Integratedmatically, we need this): https://gitlab.com/gitlab-org/gitlab-ce/issues/3442
- Integrated metrics (Integrateddetect metrics): https://gitlab.com/gitlab-org/gitlab-ce/issues/30369
- Integrated coverage: https://gitlab.com/gitlab-org/gitlab-ee/issues/2781
- Integrated correlation (show relevant graph of metrics in MR)
- Integrated rollout (incremental rollouts): https://gitlab.com/gitlab-org/gitlab-ee/issues/1660
- Integrated scale runners (of runners)
- Integrated link (issue with MR based on branch name, create MR from issue)
- Integrated secure/scan (for vulnerabilities): Scan (https://gitlab.com/gitlab-org/gitlab-ee/issues/2592)
- Integrated license (white-list/blacklist dependencies): (https://gitlab.com/gitlab-org/gitlab-ee/issues/2592)
- Integrated update (roll out security updates, debian example)
- Integrated rebuild (rebuild all containers with one command)
- Integrated start (framework templates): https://gitlab.com/gitlab-org/gitlab-ce/issues/32420
- Integrated review apps (provision a review app Integratedmatically)
- Integrated registry (container registry just works with CI)
- Integrated artifact (easily linked artifact reposity)
- Integrated opsview (ops views): https://gitlab.com/gitlab-org/gitlab-ee/issues/1788
- Integrated a/b (Integratedmatically a/b test MR's for GitLab Pages, 'Merge if better')
- Integrated overview/flow/sequence (cross linking of CI with triggers and dependecies)
- Integrated Kube (link projects to Kubernetes if deployed on Kubernetes)
- Integrated scale apps (more instances when load increases)
- Integrated log (aggregation)
- Integrated exception (looking for better word, handle exceptions in GitLab)
- Integrated chatops (we already have chatops that just work)
- Integrated config: (shadow .gitlab-ci.yml)
- Integrated web terminal: https://docs.gitlab.com/ce/ci/environments.html#web-terminals
mentioned in issue gitlab-com/blog-posts#359
Cloud Foundry is a very popular PaaS and fast growing ecosystems which also employs a “buildpack” to start the application. Cloudfoundry is natively available on Azure, GCE and AWS in the mean time therefore its more important your have deep support for that. With project Kubo (https://pivotal.io/partners/kubo) the integration of kubernetes (CaaS) workloads is forced by Pivotal and Google which is already using bosh to manage their kubernetes environments. There is Spring boot/cloud (Java/Kotlin/Groovy + Maven/Gradle) which is the most used (http://redmonk.com/fryan/2017/06/22/language-framework-popularity-a-look-at-java-june-2017/) and battle proven microservice cloud native stack (http://projects.spring.io/spring-cloud/ ) , which is also running on cloudfoundry out of the box - https://start.spring.io/ . I am missing a auto-build (pack) for spring boot/cloud and a auto deploy for cloudfoundry in your roadmap. For the devops experience we would highly appreciate such integration as an USP against existing CD toolchains out there.
Edited by username-removed-1359340changed milestone to %10.0
added meta label
@bikebilly @ayufan I updated the description with the proposal for the first iteration. Note that I made Auto CI Stretch as we could get away without it and we might need that breathing room in order to get this in in time.
@joshlambert Note that I'm including Auto Monitoring in the first iteration, but I believe existing functionality is sufficient. The changes (https://gitlab.com/gitlab-org/gitlab-ce/issues/35266) already scheduled for 9.5 will make it better.
@dimitrieh @sarrahvesselov We've tried to make the proposal in this issue as small as possible, but it would be great to get your thoughts on what we could do to improve the UX in future iterations. Perhaps it's nothing, as the best UX is one that doesn't exist (meaning there's zero configuration; it just works). But I imagine there are failure conditions or other situations that could be improved.
For example, if you don't have Kubernetes configured (via a service or env variable), then we'll silently not do any deploys. This could be confusing for people if they are expecting deploys.
One potential option could be to have dismissible notices for projects without kubernetes telling them about how they could enable X, Y, and Z if they enable Kubernetes.
Or how do we manage the surprise when this is suddenly turned for someone? Should we let them know that CI/CD has been configured for them, with a dismissible message?
Do we need to include something in notification emails for failures on implied CI?
I know there's a lot we can do, but we should start by looking for high bang-for-buck items; things we need to do to make this feature viable.
Perhaps create new issues for any proposals.
Totally understand why containers are the first focus, but would hope that EEP customers like me could also have a variant of AutoDevOps when using vagrant/virtualbox for build machines and bare-metal for test machines and deployment machines.
Would love to have AutoDevOps for kernel loadable modules / linux drivers where build, test and deploy depend on the same kernel version being in use across the board, whether VM for building or PM (physical machine) for test & deploy.
If you imagine the rate at which kernels get patched across N > 5 distros, and that toolchains get updated, etc. we have a fairly high burden simply verifying that running our existing code on somebody else's latest bits quite frequently even before we talk about testing our own next commit id on an existing "latest from somebody else" ... "the universe" is committing "towards us" faster than a small shop like ours commits towards "the universe" as it were.
@MikeWalsh It was moved to CE. It's easy to miss when there are comments after the move.
THIS ISSUE HAS MOVED TO: https://gitlab.com/gitlab-org/gitlab-ce/issues/35712
Edited by Mark Pundsack
@markpundsack Thanks!
marked the checklist item Implied
.gitlab-ci.yml(https://gitlab.com/gitlab-org/gitlab-ce/issues/34777) as completedmarked the checklist item Make Kubernetes deploy actions optional: https://gitlab.com/gitlab-org/gitlab-ce/issues/34785 as completed
marked the checklist item Make Kubernetes deploy actions optional: https://gitlab.com/gitlab-org/gitlab-ce/issues/34785 as incomplete
marked the checklist item Add Code Quality: https://gitlab.com/gitlab-org/gitlab-ce/issues/33266 as completed
marked the checklist item Make Kubernetes deploy actions optional: https://gitlab.com/gitlab-org/gitlab-ce/issues/34785 as completed
marked the checklist item Stretch Rename codeclimate to codequality: https://gitlab.com/gitlab-org/gitlab-ee/issues/2783 as completed
marked the checklist item Stretch Add Auto CI (using Heroku CI): https://gitlab.com/gitlab-org/gitlab-ce/issues/26941 as completed
