dev.gitlab.org backup upload to S3 fails
We upload our dev.gitlab.org backups to S3. This started failing recently with:
Uploading backup archive to remote storage backup-dev-gitlab-org ... rake aborted!
Excon::Errors::BadRequest: Expected(200) <=> Actual(400 Bad Request)
excon.error.response
:body => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>IllegalLocationConstraintException</Code><Message>The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.</Message><RequestId>CC48B38C92B06841</RequestId><HostId>=</HostId></Error>"
:headers => {
"Connection" => "close"
"Content-Type" => "application/xml"
"Date" => "Wed, 08 Jun 2016 08:44:51 GMT"
"Server" => "AmazonS3"
"x-amz-id-2" => "*"
"x-amz-request-id" => "*"
}
:local_address => "10.2.0.7"
:local_port => 33250
:reason_phrase => "Bad Request"
:remote_ip => "54.231.130.163"
:status => 400
:status_line => "HTTP/1.1 400 Bad Request\r\n"
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/expects.rb:6:in `response_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/response_parser.rb:8:in `response_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:372:in `response'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:236:in `request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:256:in `rescue in request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:204:in `request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:256:in `rescue in request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:204:in `request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:256:in `rescue in request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/excon-0.45.4/lib/excon/connection.rb:204:in `request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-core-1.35.0/lib/fog/core/connection.rb:81:in `request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-xml-0.1.2/lib/fog/xml/connection.rb:9:in `request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-aws-0.8.1/lib/fog/aws/storage.rb:611:in `_request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-aws-0.8.1/lib/fog/aws/storage.rb:606:in `request'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-aws-0.8.1/lib/fog/aws/requests/storage/put_bucket.rb:28:in `put_bucket'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-aws-0.8.1/lib/fog/aws/models/storage/directory.rb:102:in `save'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-core-1.35.0/lib/fog/core/collection.rb:51:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:51:in `upload'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:36:in `block in pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `chdir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'