Distributed Cache on AWS S3 only for US East (N. Virginia)

I configure my docker runner as documentation says :
https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#the-runnerscache-section

I configure my S3 endpoint to Ireland http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

config.toml

concurrent = 2

[[runners]]
  name = "#{ENV.fetch('NAME', 'CPL-DEFAULT-AWS-RUNNER-DOCKER')}"
  url = "http://xxx"
  token = "xxx"
  tls-ca-file = ""
  executor = "docker"
  environment = ["GITHUB_TOKEN=xxxx"]
  [runners.cache]
        Type = "s3"
        ServerAddress = "s3-eu-west-1.amazonaws.com"
        AccessKey = "xxxx"
        SecretKey = "xxxx"
        BucketName = "gitlab-cpl-cache"
        Insecure = false
  [runners.docker]
        image = "scratch:latest"
        disable_cache = true
        privileged = false
        volumes = ["/root/.composer/cache:/root/.composer/cache", "/root/.npm:/root/.npm", "/usr/local/bin/ci-helper:/usr/local/bin/ci-helper:ro", "/etc/ssl/certs:/etc/ssl/certs:ro", "/usr/share/ca-certificates:/usr/share/ca-certificates:ro", "/cache", "/root/.ssh-gitlabrunner:/root/.ssh:ro"]

But I have got this error in syslog

./kern.log:Apr  6 07:35:54 ip-172-31-18-12 gitlab-runner[2219]: Amazon S3 endpoint should be 's3.amazonaws.com'.
./kern.log:Apr  6 07:35:54 ip-172-31-18-12 gitlab-runner[2219]: Amazon S3 endpoint should be 's3.amazonaws.com'.
./kern.log:Apr  6 07:35:54 ip-172-31-18-12 gitlab-runner[2219]: Amazon S3 endpoint should be 's3.amazonaws.com'.
./kern.log:Apr  6 07:35:54 ip-172-31-18-12 gitlab-runner[2219]: Amazon S3 endpoint should be 's3.amazonaws.com'.

Perhaps due to this code https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/3da3eae5eb013ea881202e99955e6a4f172d40ae/Godeps/_workspace/src/github.com/minio/minio-go/utils.go#L192 https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/3da3eae5eb013ea881202e99955e6a4f172d40ae/Godeps/_workspace/src/github.com/minio/minio-go/utils.go#L165

Use s3.amazonaws.com as the error suggests. Minio should discover where your bucket is placed:

• https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/Godeps/_workspace/src/github.com/minio/minio-go/api.go#L464
• https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/Godeps/_workspace/src/github.com/minio/minio-go/s3-endpoints.go

And please tell us if this worked. If yes, then we will update documentation, so it will be clear for other users :)

Ok : I made the change and error disappears.

But S3 Bucket does not fill. It still empty. I am going to analyse what happens

mentioned in issue #1214 (closed)

I am seeing the same issue, I was wondering if it could be that the shared cache is only used for docker.machine runner?

I have two shell runners on osx that i am trying to get the cache shared for but my S3 bucket is empty, and IAM shows the the keys have never been used that I setup for this.

I dont see any errors reported, even when running gitlab-runner with verbose

My bucket is also in eu-west-1

mentioned in issue #1151

Milestone changed to v1.2

@tmaczukin with the latest stable runner (1.1.3, IIRC) setting the S3 URL to either s3-us-west-2.amazonaws.com or s3.amazonaws.com results in 0 errors and nothing being uploaded to s3.

Using the current bleeding edge runner 1.2.0~beta.44.g8a5080a using s3.amazonaws.com doesn't seem to work:

Creating cache test/ci-cache...
vendor/: found 14257 matching files                
Uploading cache.zip                                
WARNING: Retrying...                               
Uploading cache.zip                                
WARNING: Retrying...                               
Uploading cache.zip    
WARNING: Put https://mybucket.s3.amazonaws.com/runner/runner-token/project/98/test/ci-cache?X-Amz-
Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWS_ID%2F20160504%2Fus-east-1%2Fs3%2Faws4_
request&X-Amz-Date=20160504T045319Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-
Signature=SIGNATURE: write tcp 172.17.0.3:46556->54.231.184.17:443: write: broken pipe 

But neither does s3-us-west-2.amazonaws.com:

Creating cache test/ci-cache...
vendor/: found 14257 matching files

My S3 bucket is still empty.

@gmarchand @mcfedr @BM5k We've released 1.1.4 with a fix related to S3 caching. Could you try to update GitLab Runner and test it again?

@tmaczukin This is my test

I add theses lines at the beginning of .gitlabci.yml

cache:
  untracked: true

I check (with AWS CLI) my AWS S3 connection with credentials configured in config.toml and also with EC2 Instance Profile

The result of build

HEAD is now at d415cca... Test S3 Cache
Checking cache for build_js/fix/cache...
WARNING: Received: 400 Bad Request                 
Downloading artifacts for prepare_js (3537)...
...
Creating cache build_js/fix/cache...
untracked: found 43024 files                       
Uploading cache.zip                                
WARNING: Received: 400 Bad Request           

Nothing into logs about this bad request. But I have got this line, I don't know why

docker[372]: time="2016-05-16T08:28:10.023433015Z" level=error msg="Handler for GET /v1.18/images/gitlab-runner-cache:9e2fd1a/json returned error: No such image: gitlab-runner-cache:9e2fd1a"

I have updated on my docker based runner, not seeing this working - my bucket is still empty.

Creating cache test_and_publish/master...
node_modules: found 36662 matching files           
Uploading cache.zip                                
WARNING: Received: 400 Bad Request                 

In my config:

[runners.cache]
Type = "s3"
ServerAddress = "s3.amazonaws.com"
AccessKey = "****"
SecretKey = "****"
BucketName = "ek-gitlab-ci-cache"
Insecure = false

Bucket is in eu-west-1 region

Hmmm... Those 400 Bar Requests look like responses from docker hub. That's strange because the gitlab-runner-cache image is included into Runner and it should be unpacked from it.

/cc @ayufan

Installed 1.1.4 (9e2fd1a) and I'm not seeing any errors

Creating cache test/ci-cache...
vendor/: found 12488 matching files

Build succeeded

But my S3 bucket is still completely empty.

Double checked both s3.amazonaws.com and s3-us-west-2.amazonaws.com; same result.

Disregard my last message; apparently I had another runner that wasn't paused (oops) and it picked up the jobs. I am re-testing.

First actual run on 1.1.4:

Creating cache test/ci-cache...
vendor/: found 12394 matching files                
Uploading cache.zip                                
WARNING: Received: 400 Bad Request 

With ServerAddress=s3.amazonaws.com

And with ServerAddress=s3-us-west-2.amazonaws.com

Creating cache test/ci-cache...
vendor/: found 12394 matching files                

Build succeeded

Either way, my S3 bucket is still empty

OK, some progress. I found & set BucketLocation='us-west-2' with ServerAddress=s3.amazonaws.com and my bucket is no longer empty. Going to run a couple more test builds tomorrow to verify that the cache is being used.

BucketLocation also got files uploaded, looking promising.

As other people have I think pointed out, it seems strange that its in a folder runner/{id}- suggests its not going to be shared by other runners...

Found where it was mentioned in https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1226

mentioned in merge request !170 (merged)

mentioned in issue #1310 (closed)

@BM5k @mcfedr Thanks for the tests and for the MR! I'll try to confirm them this week and if this will be working then we will finally have the solution :)

Reassigned to @tmaczukin

Seems the folder per running issue is still a problem

@mcfedr

I made it this way, because usually the runners don't work on the same environments or on the same architectures.

@mcfedr Cache feature was introduced to speed-up the subsequent executions of the same build in a project. Also - like @ayufan wrote - if you are using different runners then it's possible that they will have a different OS-es or architectures. That's why cache is locked with a runner-project-build namespace.

And S3 based caching was created for auto-scaling mode, where your runner may be each time provisioned on a different machine, so it needs a one, central point to store its cache on it.

I hope that our answers are enough clear :)

@BM5k Have you done your tests? Can we confirm that setting BucketLocation in Amazon's S3 resolves the problem? I would like to close this issue finally :)

@tmaczukin yes, sorry for not posting here sooner. As far as I can tell, setting BucketLocation was the missing piece.

@BM5k Excellent! Thank you both for testing and working on finding the solution for this issue!

Status changed to closed