CI: connectivity issues all over the place in F-Droid repos

For the past few days, CI has been failing very often on all three of our repos:

• https://gitlab.com/fdroid/fdroiddata/builds
• https://gitlab.com/fdroid/fdroidserver/builds
• https://gitlab.com/fdroid/fdroidclient/builds

It's very obvious in fdroiddata, the most active of the three. It's also the one with the simplest CI script.

Sometimes the container won't start:

https://gitlab.com/fdroid/fdroidclient/builds/1083499

Sometimes there are weird connectivity issues inside the container:

https://gitlab.com/fdroid/fdroiddata/builds/1083556

Hm, actually, fdroidserver is failing for a mistake of ours. We didn't notice amidst all the other CI failures that weren't our fault (hopefully).

Now also including our friend unrecognised input header:

https://gitlab.com/mvdan/fdroidserver/builds/1084630

@ayufan @grzesiek Can you take a look at this again, please?

@dblessing It looks more like an infrastructure issue, @tmaczukin can you take a look?

@mvdan Are you using shared runners from GitLab.com or some specific provided by yourself?

@tmaczukin public runners. We use our own docker images too.

Also, this is still happening. Even a single clone that we run dies with a 500 nearly every single time: https://gitlab.com/fdroid/fdroiddata/builds/1110350

@mvdan Results of my investigation:

1. In logs we can find something like that:

   Apr 18 09:38:27 production.log: Started POST "/fdroid/fdroidserver.git/git-upload-pack" for X.X.X.X at 2016-04-18 09:38:27 +0000
   2016-04-18_09:38:27.90519 gitlab-workhorse: fatal: The remote end hung up unexpectedly
   2016-04-18_09:38:27.90623 gitlab-workhorse: 2016/04/18 09:38:27 error: handlePostRPC wait for [git upload-pack --stateless-rpc /var/opt/gitlab/git-data/repositories/fdroid/fdroidserver.git]: exit status 128
   2016-04-18_09:38:27.90627 gitlab-workhorse: gitlab.com @ - - [2016-04-18 09:38:27.731301767 +0000 UTC] "POST /fdroid/fdroidserver.git/git-upload-pack HTTP/1.1" 500 78 "" "git/2.1.4" 0.174936
   Apr 18 09:38:27 gitlab_access.log: 10.1.0.33 - - [18/Apr/2016:09:38:27 +0000] "POST /fdroid/fdroidserver.git/git-upload-pack HTTP/1.1" 500 78 "-" "git/2.1.4"

2. I've searched what exit status 128 means and as for now I only know it means "git command exited with an error".

3. I've tried the command git clone https://gitlab.com/fdroid/fdroidserver.git on my laptop and it worked without any problems.

4. I've realized that in your build the command has --depth=1 option so I executed:

   $ git clone https://gitlab.com/fdroid/fdroidserver.git --depth=1
   Cloning into 'fdroidserver'...
   error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error
   fatal: The remote end hung up unexpectedly

So the problem is the --depth=1 option but I don't know yet why. I will search the cause, but you can temporarily remove the --depth=1 option to use your builds.

I can also confirm that this is failing for any git clone --depth=1 over HTTP (on any project). And it's working just fine when using SSH protocol instead.

Nice find! I'll apply that temporary workaround for now.

Working now, thanks! Please ping me once that issue with --depth is fixed.

I'm triggering build retries for the other repos to see if things are any better.

Status changed to closed

@mvdan The issue is related to Git HTTP protocol handling and it was already mentioned: gitlab-org/gitlab-workhorse#36 (closed).

Problem will be fixed with going back to GitLab Workhorse 0.7.1. The MR changing the version of Workhose was merged and it was picked into stable branch so it should be fixed on GitLab.com after the next deploy.

We will probably deploy some RC version before Friday but if not, then changes should be on GitLab.com on release day (22.04).

Please follow https://twitter.com/gitlabstatus to know when the deploy is done. I will close this issue. If the problem will exists after the deploy, then feel free to reopen it :)

Thanks! The docker-related issues seem to have gone away too.

Docker issues are occurring from time to time in different builds. But for this we have also separate issue, which you saw already :)

Thanks for your report @mvdan!