Daily builds of GitLab / GitLab CI / gitlab-shell master

I propose we use GitLab CI's 'daily build' feature to do daily omnibus builds. This could be done in several steps.

1. Create a script that updates config/software/gitlab-{ci,rails,shell}.rb to use the master commit of the respective repositories and run this build daily, with email notifications on failure. This helps omnibus maintainers catch issues caused by upstream projects earlier.
2. Upload these daily builds to a special repo on packages.gitlab.com. This would allow us GitLab B.V. and community daredevils to run 'bleeding edge' GitLab via omnibus packages.

cc @marin @sytses

@jacobvosmaer does GitLab CI have a daily build feature?

@sytses good question. It does, sort of, but not like I thought.

CI can repeat the last build but in this case we would not know if that was the build we wanted.

@jacobvosmaer Indeed, can we easily purge old builds? In that case we can build every commit.

build every commit

You mean every push? That may take a bigger build farm than we have now (need enough builders to keep up with pushes * 6 platforms) but it could be doable. Expensive storage-wise though, and nobody is going to install all of those pushes. Daily builds seem more useful to me.

can we easily purge old builds

Doable; yanking packages from packages.gitlab.com is scriptable via a REST API. I hesitate to wave it away as 'easy' though.

OK, maybe add function to CI to ignore pushes and build periodically

nightly builds of the master branch is fine!

Here is a more specific idea on how to do this. We create a new project, gitlab-integrator or something like that. It contains a script that

• clones omnibus-gitlab master
• edits the cloned omnibus-gitlab to build the latest commit SHA's for gitlab-{ci,rails,shell}
• start an omnibus build
• yanks old packages from packages.gitlab.com

We could then use the existing CI 'repeat last build' feature to keep repeating the 'gitlab-integrator' build. The integrator build would run on the same boxes as the omnibus-gitlab build.

It's a bit of a hack (compared to having just the right CI features to do what we want) why not start like this?

Having an integrator that 'pulls', instead of building each 'push' to gitlab-rails OR gitlab-ci OR gitlab-shell seems like a less chaotic solution to me.

@jacobvosmaer I don't understand completely, but other people must have this need, lets build it in CI so everyone profits.

@vsizov Can you add a 'build every x hours' irrespective of pushes function to GitLab CI? Consider using Sidetiq /cc @dzaporozhets

@sytses I created issue https://dev.gitlab.org/gitlab/gitlab-ci/issues/229

Thanks @vsizov

@sytses @jacobvosmaer Currently we use "whenever" gem to run builds periodically. It does not depend on pushes, at least it is triggered independently. As I understand we need to create some feature that will pull recent master branch, right? Current design does not allow to implement such kind of logic. CI coordinator does not have access to repos itself. It requires to make changes on both sides: coordinator and runner.

@vsizov This seems to complex. Maybe we should build every commit but deploy no more than once every 24 hours. So before uploading to the package server it checks if there is an existing package with todays date and aborts if needed. What do you think @jacobvosmaer ?

@vsizov Lets wait with this until 7.12 is out.

What does 'build every commit' mean? The omnibus-gitlab packages integrate three different GitLab repositories: gitlab-ce, gitlab-ci and gitlab-shell. So how do we turn a commit in e.g. gitlab-shell into a build for omnibus-gitlab?

I think the simplest way to have daily builds is to use a cron job and forget about building 'every commit'.

• add a 'make latest' target to the omnibus-gitlab Makefile
• in a cron job pull omnibus-gitlab master and run 'make latest'
• once a day, on dev, install the latest package from a cron job

We don't need gitlab-ci for this, I think it is not the right tool.

@jacobvosmaer Makes sense, lets use cron

@marin Did you say you where working on this?

@sytses I started on it, yes. cc @jacobvosmaer

Reassigned to @marin

Milestone changed to 7.14

General idea for this is as follows:

1. Have a "place" which will run a cron task
2. Cron task will run a script that will update refs for CI, GitLab and gitlab-shell.
3. Script will commit and push the change to the separate branch

So the proposal is:

1. "The place" should be dev.gitlab.org. This will allow us to only care about one server(and all security around it is already in place)
2. Cron task will run a shell script which will checkout omnibus-gitlab bare repository from the disk. We don't have to worry about additional keys to clone omnibus-gitlab repo.
3. The script will then run set-revisions on the cloned repository in a new branch and commit the change
4. The script will then push to the origin which is a bare repo on the disk(instant push)
5. This will trigger all hooks on the repository which means that our new branch will be pushed to dev.gitlab.org which will also trigger a CI build. This is important because pushing to CI will put it in the build queue and we don't have to worry about interrupting any possible existing builds

Few issues:

1. Because all hooks are triggered on push to bare repo, GitLab is checking for GL_ID. => This can be solved by either using an existing users credentials(not perfect) or creating a new Nightly Build user which will only be added to the omnibus-gitlab repository on dev.gitlab.org.(I prefer this)
2. set-revisions script is using dev.gitlab.org to fetch latest refs. This will require access and keys for the user so defeats the purpose of having this on dev.gitlab.org => For now I see two options: Set the remotes to gitlab.com (not really a solution) or adapt set-revisions to accept arguments which would allow us to do another bare clone on dev.gitlab.org(I prefer this)

@jacobvosmaer We discussed parts of this so I am curious what you think about the issues. As I mentioned I prefer having a separate user who will only have access to this one repo and also adapting set-revisions to accept different remotes which should allow us to use the bare repo already on the disk.

I now realize we need dependent builds as a feature in CI for CloudFoundry in August. Having a dependent build in .gitlab-ci.yml will cause GitLab CI to periodically check an external git repo for updates. If there are updates a build will be triggered. At the end of a build we can automatically set a tag.

The only think that we maybe need to do is reduce the check time to once a day. I'm OK with making the check time for dependent builds configurable.

@marin does this sound like a solution to what we want? I'm sorry to bring this up now but it is a recent development. I think it would be wasteful to make something special on dev.gitlab.org when we can spend time making a feature that we will need anyway in a month from now.

@ayufan can you start an issue for dependent builds and one for tag setting on the GitLab CI issue tracker?

@marin I think we can override the remote in set-revisions so that is not a problem.

@marin separate user: sounds like a good idea.

@sytses I created meta issue for dependant builds some time ago: https://dev.gitlab.org/gitlab/gitlab-ci/issues/328.

Having a dependent build in .gitlab-ci.yml will cause GitLab CI to periodically check an external git repo for updates. If there are updates a build will be triggered. At the end of a build we can automatically set a tag. The only think that we maybe need to do is reduce the check time to once a day. I'm OK with making the check time for dependent builds configurable. @marin does this sound like a solution to what we want?

@sytses It does, if it is configurable time then there is no reason to continue working on this until we see how the CI stuff looks and works. I will stop with this and wait for CI if this is not urgent although I am close to having this running on dev.gitlab.org.

@marin Thanks, sorry for only mentioning this now.

CI is receiving triggers in 7.14 so will continue working on this after the release.

Milestone changed to 8.0

mentioned in merge request !444 (merged)

• Merged !444 (merged)
• Updated our infrastructure: https://dev.gitlab.org/cookbooks/gitlab-server/commit/94133c6cee31ac9719068b8132c904195c6db8e9 and https://dev.gitlab.org/cookbooks/chef-repo/compare/2a9307239d8e8bb5d17a28909f28b589239046d4...90e45a393dad2b90605cd44522a4db0f175e5571
• Created https://packages.gitlab.com/gitlab/nightly-builds

Double checking if everything was as expected tomorrow before closing the issue.

@marin Awesome work!

Nightly builds are working but we are still being hit by 504s when uploading to packagecloud. Still, this is working.

Status changed to closed

@marin do you have a link? I am not seeing the nightlies on ci.gitlab.org.

@jacobvosmaer https://packages.gitlab.com/gitlab/nightly-builds for the packages, https://ci.gitlab.org/projects/55/refs/master/commits/6a40008e205b8ce04b43ef6a8eb1457b12e275e4 for the build.

Thanks @marin, I overlooked them because I did not expect 7.14.1 to be built after the 'nightly'.

Out of curiosity, how is the omnibus package version set? What happens when gitlab master goes to 7.15-pre? I am wondering if we can expect a correct sequence of packages when just using 'apt-get upgrade'