Merge GitLab CI into GitLab CE

CI Coordinator*

Becasue:

we duplicate a lot of configuration between apps
we can not efficiently use CI database in CE
we must cross-link CI and CE and setup OAuth which bring additional barrier for users

Pros:

It would be great to have it in 8.0
It will make Zero Config really simple
It will fix a lot of SSL/Nginx issues when connecting CI with CE
We could effectively reuse code for implemented Services
No need to port CI UI anymore
It will nicely integrate with GitLab CE UI: commit statuses, charts, builds view

@JobV @dzaporozhets @ayufan I added a description

I propose we:

Make a mockup of the UI
Estimate how much more memory this would use

I can see it as 2 big steps

Step 1 (8.0..8.2)

So far the simplest way I can see it is just take whole Coordinator app and mount it under /ci in CE. Coordinator database tables can be prefixed with ci_ and added to CE.

Step 2 (9.0)

Remove duplicating data. Remove CI projects table. Integrate CI in GitLab as issues and milestones

@dzaporozhets That looks doable.

Estimate how much more memory this would use

I think it would be near same if no CI activity. Dependencies are same. Few thousand of lines of code should not make much difference.

Yes. It can be done.

@sytses @ayufan awesome. I put 8.0 milestone to it.

@ayufan please ping me when you start so we can check how make first step faster and easier.

We can also consider to develop lightweight CI version on CE side because we don't need most of the pages. Most of the load in CI comes up in the API and API web server can be written in Golang (success story - http://blog.parse.com/learn/how-we-moved-our-api-from-ruby-to-go-and-saved-our-sanity/). So user can choose which server he wants to use).

Milestone changed to 8.0

@vsizov makes sense for future step. We do similar with git-over-http for CE.

Additional advantages:

Less memory usage if you want to run both CE and CI (for example on a Raspberry Pi
This would mean we can add enterprise features to CI without creating another repo (CI-EE?)

@sytses both are true!

I agree with @dzaporozhets that extra memory usage will be neglectable.

Comment from @dzaporozhets in core room: "I think its a future. We can throw away 50% of CI code if merged and make CI setup easy as press button"

I like the release style of Ember, they add features in point releases and clean up the code in major releases.

In 7.14 we ship Mattermost and gitlab-git-http-server so we can make them beta and the default in 8.0.

But maybe it is a stretch to add this to 7.14, I'll let @ayufan comment.

Anyway, we should delete the old database and files in a different release than the one we use to add CI.

@sytses I like idea of cleanup in major releases.

I have one concern: how we will migrate data from GitLab CI to GitLab CE?

@ayufan Database data? Just copy the tables and add ci_ as Dmitriy proposed?

@sytses I need around 3 days to merge the codebase, but I'm not available next week (vacations). I don't expect to have enough time to do that in London before 7.14 release. So I'll have enough time do it for 8.0 release.

@ayufan OK, thanks!

Disadvantage is a large write load to the database due to the build logs being posted.

@sytses FYI Build logs are stored in text files since 7.13.

@ayufan Lol, I didn't know that, great news.

@jwrede Build logs not being stored in the DB should help make large telco prospect considering many CI Runners happy, can you tell them that build logs are stored outside the database, thereby preventing a single point of write contention? Please comment in the internal issue.

Disadvantages:

Running GitLab CI in developments and tests will be much slower.
Using GitLab with Jenkins (which 90% of our customers are doing) will get harder, the UI won't be as good and we'll never get a proper commit status api

I also wonder how the UI will look. Commit status can be added as part of the normal commits page, charts to the charts page, webhooks, triggers, runners and variables to the settings page. Settings to the settings page under 'build'. Events should go into the audit log.

👍 for merging CI in to CE.

Using GitLab with Jenkins (which 90% of our customers are doing) will get harder, the UI won't be as good and we'll never get a proper commit status api

@sytses definitely agree this is a concern. There will still need to be some sort of CI API for Jenkins and other CI to use. I don't think people will be happy if the Jenkin plugin stops working.

@bbodenmiller We won't deprecate existing API. It is just that we will not add a new commit status API.

Additional advantages:

This will make GitLab CI more popular
Easier to deeply integrate the results in the merge request (show code coverage in diff)

I've added all items to the description for a future blog post.

Will CI integration be "always on" or will there be the option for an administrator to disable it for the whole instance? If an install is already using another CI tool then they might not want to confuse users by adding another option within GitLab.

@stevenorman It will be always on in the sense that if you add .gitlab-ci.yml it will work for the project. If you do not add runners it will obviously not execute the tests. But people can add their own runners for the own projects. This type of organic adoption has been a strong point of GitLab and we hope to see it continue. We're open to considering disabling options at some point, after hearing people running into problems. I think the confusion will be limited but maybe there are security considerations. But I'm worried about a blanket 'we use Jenkins here, ask the CI team if you want tests' ban on using GitLab CI. I have the feeling that in many organizations smaller repo's are not tested because it is too hard to set up Jenkins and many times people are not able to do so themselves.

@sytses 👍

@sytses Perhaps it is more an EE option to disable as they are the ones more likely to have a specific CI infrastructure. My main concern is the way that runners poll for new jobs in CI, I think every minute?, rather than being triggered when there is actually a build to happen. Scale that up with a large number of runners that you don't have any control of and your GitLab instance could easily be seeing a lot more load when the bulk of them aren't actually doing anything. What effect will that have for unicorn runners as I think the default of 5(?) runners might easily be swamped based on load testing that has been done on my installation.

This is a bit like configuring Jenkins to poll gitlab every minute for changes rather than using a project webhook (which is arguably the better way to do it but use, but no matter how much information we give users they just don't use it until they are pushed that way). It is easy for people to setup polling and for it to quickly snowball as you add more projects that poll.

@stevenorman We have many polling runners at ci.GitLab.com but few problems, it runs on a single server. But I think it makes sense to add this as an EE option when people run into problems.

Hi there, great idea!

I often see on http://feedback.gitlab.com, people want to be able to write plugins. This was the first thing which came into my mind when reading this story about integrating GitLab CI. Why not shipping it as a plugin for GitLab?

Just some thoughts:

You reach your goal by reducing code
You can still run fast tests (without plugin)
You can still ship a lightweight solution without plugins
Enterprise can still use their trusted jenkins and don't have to use the plugin
You get a full featured plugin api
You target new markets by opening this to the public

@widerin This has to integrate deep into the codebase, database and UI in many places. Having it as a plugin is very likely to break it in the future.

Besides GitLab CI there are many other tools people like to use. Can we allow other applications to post in the GitLab commit status database table via an API? This would allow people to get input about their code from multiple sources. There will always be tests we don't run ourselves (Coverity static analysis, AIX platform, etc.).

@ayufan "So I'll have enough time do it for 8.0 release." You'll be in London until halfway September, that is the same date as the 8.0 code freeze isn't it?

@sytses Yes, I plan to do it during that time, the beginning of September to be exact.

@ayufan I don't think you'll have time for it, you'll be pair programming full time in London. But let's see how the first week in London goes and evaluate then.

@sytses Yes, we will get back to it after first week.

Having a commit status API would be great I think. This way lots of different existing CI tools could be used w/o problems (even if you integrate GitLab CI into CE).

We also heard from CodeShip they would love a commit status API, I think we should do it.

@ayufan will have little time for this since he is in London. @dzaporozhets will also join the effort as soon he is done with the satellite work.

I think we should introduce a commit status api with this change. There is no way 100% of the people can use GitLab exclusively for CI and CD, there will always be other tools and we need to integrate well with them. Assuming a commit status api the advantages and disadvantages become the following:

The advantages are:

Automatic enabling of CI for new projects that contain .gitlab-ci.yml (zero-config)
Reduce duplicate project information
Reduce duplicate code for storing secrets
We prevent duplicating a lot of configuration between apps
We can stop cross-linking CI and CE via OAuth, removing a barrier
Reduce the number of normal backups and backing up secrets
Interface updates will be simultaneous, no effort to improve CI UI separately
This will make GitLab CI more popular
Easier to deeply integrate the results in the merge request (show code coverage in diff)
We can throw away 50% of CI code if merged
Less memory usage if you want to run both CE and CI (for example on a Raspberry Pi) since it is only a few thousand lines of code, almost of of the gems are shared
We can add enterprise features to CI without creating another repo like CI-EE
It will fix a lot of SSL/Nginx issues when connecting CI with CE
We could effectively reuse code for implemented Services

Disadvantages are:

More memory used by CE (will only be a little)
If you runners overload your CI system now both go down
Database will be larger, although build logs are stored in text files now
The CE interface will get more cluttered with CI settings and status
It takes longer to boot development
It takes longer to run tests
It takes longer to understand and explore the project
You can't host CI on different machines than CE

I think the advantages are still there.

The only real disadvantage that concerns me at all is performance/overhead based on runner traffic. Perhaps you could configure all api calls CI and CE/EE, to be handled by a separate host, but funneled through the main NGINX, which would be setup in a load-balancer configuration.

If we could make this happen, we could probably even add the ability to auto-scale both UI and the api hosts, or simply allow for multiple api hosts to be added to the gitlab.rb. The api hosts wouldn't need gitlab installed, just NGINX.

So from what I understand, and correct me if I'm wrong, ci will move from being a "subdomain" style URL to a different endpoint? For example: projects.mydomain.net and ci.mydomain.net would become projects.mydomain.net and projects.mydomain.net/ci?

Once the projects are merged, I don't know how hard it would be to serve ci as ci.mydomain.net instead of the /ci endpoint, but I prefer that method because I think it is clearer.

Perhaps you could provide a configuration option provide a way to reverse-proxy ci.mydomain.net to projects.mydomain.net/ci?

@LongLiveCHIEF If you're autobalancing there is no need to separate UI and api hosts, they can both handle each request. Having one pool is more efficient.

@techwiz24 You understand the endpoint change correctly. I'm not sure about the configuration option, the nginx config is already a bit messy and stuff like this is confusing to users.

@sytses really about the endpoint change? I was hoping to still have a domain for my ci & have it link to my ci dashboard.

@bbodenmiller I'm not sure what the impact is exactly, if you know more please let me know. What do you mean with linking it to your ci dashboard?

It really depends on how the CI UI is going to change going forward. Ideally if I visit ci.gitlab.com I still want it to show the build status for each of my projects.

@bbodenmiller What about seeing the build status in your project overview in GitLab?

On the side bar? Maybe but that might get cluttered. I think some mock ups and a blog post with discussion would be beneficial for what sounds like significant UI changes to CI.

Interface changes:

Commits status can be added where commits are shown
Charts can be added to Graphs
Events can go into the Audit Events
Services can be added to the existing services under Settings
Project settings will need to add: Runners, Variables, CI settings
Web Hooks can be added as 'CI web hooks' in settings or better move them .gitlab-ci.yml so you can define them per type/stage
In the admin screen of GitLab we need to add CI settings, a Builds and a Runners page

Conclusion, the left side project menu will not have to change, except under settings where we have space for the 3-4 new items.

@DouweM Will work on this coming week. He will make a draft here of the general idea (including the commit status API and maybe some screenshots) and then start the work without waiting for feedback, hopefully most of the feedback can be incorporated during the work. /cc @dzaporozhets

@sytses @DouweM awesome

@dzaporozhets :) Please note if there is anything he needs to take into account.

@DouweM if your plan differs from https://gitlab.com/gitlab-org/gitlab-ce/issues/2164#note_1810476 can you please describe steps you want to follow

Quick update: The first step is, as @dzaporozhets already determined, to merge the CI and CE code bases into one big code base. Only after that can we start looking at removing duplication and integration of CI features into the CE UI. At that point the commit status will also become relevant, so that CE remains relatively agnostic of the specific CI tool being used.

The initial idea was to simply mount the GitlabCi app under Gitlab, using either use in config.ru or mount in config/routes.rb, but that ended up not being feasible, since a lot of naming conflicts exist between both apps, for example the Project model or the ApplicationController, and both apps use some global state, like Settings, Rails.application, Rails.root and things like Sidekiq configuration.

What I've opted to do as a first step, is to move all CI files into CE, placed under ci directories in the standard Rails directory structure, so we have things like app/assets/images/ci, app/controllers/ci, app/models/ci, app/views/ci, lib/ci etc. I'm currently working on going through all of these files one by one, adding module Ci around classes, changing class references like Project to Ci::Project, changing routes from projects_path to ci_projects_path and changing file references from Rails.root.join('etc') to Rails.root.join('ci/etc').

The first goal is to get CI working within the CE code base, albeit as a separate app for all practical purposes.

@DouweM great!

@DouweM 👍

mentioned in merge request !1204 (merged)

WIP MR for the groundwork can be found at !1204 (merged).

@DouweM awesome ⚡

@DouweM just mentioned that everything he clicks seems to work. @rspeicher will work on upgrading the tests from rspec2 to 3 so the tests can be run. Migration will be complex.

typo "since and we'll not make"

I agree with @bbodenmiller

Using GitLab with Jenkins (which 90% of our customers are doing) will get harder, the UI won't be as good and we'll never get a proper commit status api

@sytses definitely agree this is a concern. There will still need to be some sort of CI API for Jenkins and other CI to use. I don't think people will be happy if the Jenkin plugin stops working.

If this change goes ahead, I am afraid I will be unable to upgrade onto any Gitlab distribution until some sort of CI API is available. I rely on Jenkins, and it provides services to me that Gitlab CI never can. I feel that many users will feel the same.

Also, I would prefer not to be running an application that I don't use; hence wasting resources on my server.

Did you considered making CI a rails engine? The benefits are you can still opt in or out and keep code isolated while integrated.

Did you considered making CI a rails engine?

From my experience it never works good if your engine relies on some data/models from original application. You loose advantages of standalone application and still fight problems that you would not have in monolithic application

The benefits are you can still opt in or out and keep code isolated while integrated.

But we dont need it to be separated. CI and CE have a lot of duplicate code. Merging it into one application will allow us to remove it.

Reading comments above I don't understand how this change affects Jenkins. Jenkins works via services in GitLab. It will continue work as usual. This issue is about integrating CI into CE code.

Also, I would prefer not to be running an application that I don't use; hence wasting resources on my server.

@ReidWeb its not running an application. It just GitLab will have more CI oriented features inside. I dont see any waste of resources. Its like other GitLab features that you might not use. For example I run GitLab but I don't use LDAP feature. That does not mean I am against people implementing LDAP feature in GitLab.

@dzaporozhets 👍

@ReidWeb It's not really clear from the issue description, but we did land on adding a Commit Status API after all. See https://gitlab.com/gitlab-org/gitlab-ce/issues/2164#note_1847420 and the commits after that. We will work on improving the experience for users that use other services than GitLab CI.

Does this mean if Gitlb-CE is merged into Gitlab-CE we will be able to use the same services or will they be locked off.

if so.. awesome 👍 .. the services in the gitlab-ce are quite lacking ;)

@freibuis after the merge it will be easier to neatly integrate other services (Jenkins, Codeship, etc.) with GitLab since we'll have a nice commit status api.

Hello,

I am using GitLab and GitLab CI on two different machines (https://[git|ci].list.lu). I really like to have GitLab installed on a very light server (with the strict minimum of software installed on it).

Of course the server on which the CI is running has a lot of things installed on it (java jre, maven, nodejs, etc.). And I do not want to install all those software on the same server.

Will I be able to solve this problem with docker? I have not yet tested docker with GitLab CI.

@DouweM Ahh okay; great. Best of luck. Looking forward to the new release then.

@cedricbonhomme you're doing wrong if you run your runners on your CI server.

Merging CI into GitLab has nothing to do with the location or servers your runners are installed on. You should never run test-executors on your GitLab or CI machine.

See Architecture here: https://about.gitlab.com/gitlab-ci/

@widerin , thank you. I know this architecture. So, it is the time to fix my mistake.

Actually we have a runner on the CI (which is not really used). I think that I'll use this machine for the runners only. The other machine will be for GitLab onnly.

@DouweM Did you hand over the work on this to @dzaporozhets as we discussed?

@dzaporozhets Can you pick this up tomorrow and try to finish the work @DouweM started?

@sytses I will start it tomorrow but half-time because I am working on UI with Andriy.

@vsizov I propose we do this issue together. It has plenty of work and we can separate it.

Reassigned to @dzaporozhets

@dzaporozhets Sure, I will start working on it today after my current task is done.

@vsizov awesome. Please make a brief review on which stage the migration is and call me so we can discuss next steps

@sytse Completely forgot to a proper handover, sorry about that.

@dzaporozhets @vsizov If you have any questions about how I approached the migration, I'm available. The main things I've done is merging the Gemfiles (everything now has ~> version, which makes it easier to keep track of the two sides), merging the file trees (everything CI is in /ci/ folders in the appropriate places), and adding Ci:: namespaces to all controllers, models, helpers etc. @rspeicher worked on porting the specs from RSpec 2 to 3 at gitlab-org/gitlab-ci!246.

The first thing you'd need to do is probably update CI-inside-CE to CI master; it's currently based on gitlab-ci@3f0679ab, so it's somewhat behind (most of that is the RSpec 3 changes).

Rspec 3 upgrade is done in https://gitlab.com/gitlab-org/gitlab-ci/merge_requests/246

We just had a call and discussed:

We would like to see a commit status API
The integrated GitLab CI should use the same data model as the commit status API
It will be very hard to make this in 8.0 so it will likely land in 8.1
If it doesn't land in 8.0 we can announce it for 8.1
It would have been better to have both at 8.0 but the priority is to ship something that works
The integration of CI is more disruptive to customers than any migrations to convert the data model
@dzaporozhets and @vsizov will work on this issue this week (Dmitriy 50%), @DouweM will help next week

As for the API: We can fix (ci_external_url) with nginx and do redirects, for API it will be sufficient to pass that to GitLab.

mentioned in commit cccd269d

Will there be any rc release of the 8.0 version? I would really like to test the whole upgrade process from 7.14 to 8.0 with my "from source" installation on ARM to give some feedback :)

@stefan-it for sure there will be, @rspeicher knows more about the timing of the rc

@stefan-it Late tonight/early tomorrow.

mentioned in issue #2560 (closed)

Thanks :) The link in the offical doc/update/7.14-to-8.0.md guide needs to be fixed from:

https://gitlab.com/gitlab-org/gitlab-ce/blob/migrate_ci_to_ce/README.md

to

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/migrate_ci_to_ce/README.md

:)

@stefan-it Actually that looks like a case that the relative link filter didn't account for. See !1352 (merged) - thanks for reporting it!

Status changed to closed

I am closing this issue since CI is merged in CE. All bugs and improvements should be reported as separate issues.

mentioned in merge request !1578 (merged)

mentioned in issue #2969 (closed)

mentioned in commit afdc0285

mentioned in merge request !1762 (closed)

mentioned in issue #4211 (moved)

mentioned in issue #12691 (closed)

mentioned in issue #17567 (moved)

Mentioned in commit pfjason/gitlab-ce@f3980e23

mentioned in issue gitlab#4891

Merge GitLab CI into GitLab CE

Designs

Child items 0

Activity

Step 1 (8.0..8.2)

Step 2 (9.0)

Admin message

Admin message

Merge GitLab CI into GitLab CE

Activity

Step 1 (8.0..8.2)

Step 2 (9.0)