Nightly builds aren't creating a new version when CE is updated

From https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1413#note_13868892:

I'm noticing that nightly builds aren't being updated because the omnibus version looks the same (e.g. https://dev.gitlab.org/gitlab/omnibus-gitlab/builds/133381), and the S3 uploads are failing as a result. But for nightly builds, isn't it possible the omnibus version remains the same but the CE/EE SHA is updated? Should the omnibus package include the CE/EE SHA as well?

@stanhu you are right (apart from uploading to S3, the failure is uploading to packages.gitlab.com).

We have a 2 fold problem here:

1. The mirroring for gitlabhq from GitLab.com to dev.gitlab.org is broken. I already raised an issue about this but I cannot find it now.
2. Removing the timestamp causes package with the same name to fail uploading

The 1. needs to be solved in EE, and 2. needs to be solved here.

@twk3 already mentioned this problem in the MR that stripped the timestamp but we only tied it to different branches and forgot to include the nightly builds as a special case.

We are again hitting the problem of how versions of the package are formed #864 (closed) . The best way forward is to properly resolve the problem described in #864 (closed)

Realised that there is a a workaround once again.

We already have the package in S3, why not try to query S3 for the existence of the file and if it exists, delete it from package server and continue with the build. Obviously, this would be only applied to the nightly build.

Does that mean we will only have the most recent copy of the nightly build? I would think we'd want to preserve at least N days worth.

@stanhu Do we need that? From omnibus-gitlab side the sha is the same which means there is no recent new commit.

Ah, but new master would be pulled for CE/EE. Not sure what the case would be to preserve that though.

Is there a way we can include the SHA of the CE/EE build in the filename? I think that would be helpful for people trying to install local branches etc.

@stanhu That wouldn't help too. Currently the repository mirroring to dev is a bit of a hit and miss so we end up with the situation where we have a "stale" sha of gitlab-rails which will in the end produce the same situation.

Kamil suggested using CI_PIPELINE_ID variable which I think is a good idea but we have no way of passing that into the version without changing omnibus to acomodate that.

I also found BUILD_ID in omnibus which Chef is using to pull timestamp from Jenkins. Sadly, our CI doesn't have any timestamp like var that we could reuse so that won't work for us.

So all of this leads back to us changing build_version and the way we are creating it now.

mentioned in merge request !947 (merged)

So I did !947 (merged) but I don't feel good about it. It will work and it will build a version like gitlab-ce_8.1.0+git.1267.4d22d39.20910-rc1.ce.0 (where 20910 is the Pipeline ID) for the nightly build.

However, at this point I am a bit tired of hacks and quickfixes and I really think we should devote some time to writing tests and rethinking how we do versioning. The downside is that we have NO time to do this so the hack will allow us to go forward. What do you think @stanhu ?

I think we should move forward with the pipeline ID (e.g. it's a build number). What is the 1267 before it? Does it increase with every version? It seems to make sense to put the pipeline ID before the SHA since the SHA isn't actually a monotonically increasing number.

1267 is the number of commits since the tag was created. This whole sequence git.1267.4d22d39 we have no control over without rewriting the way we do versions, so I suggest to keep it this way if we are moving forward with it since it makes no difference where in the sequence is that number.

mentioned in commit 35f0e2ea

Lets try it like this and we will improve it as part of the version overhaul we will have to do at some point.

Added infrastructure label

mentioned in merge request !949 (merged)

mentioned in commit cba58d54

Working ok, closing for now.

Status changed to closed