Skip to content
Snippets Groups Projects
New issue look: Off

Backup to AWS S3 fails

    • Closed Issue created by username-removed-91883 @shirshir

    Running Gitlab Omnibus 7.7.1 on Ubuntu 14.04, trying to upload a 7GB .tar. This is the log from Amazon:

    1234567890 bucketname [20/Jan/2015:01:24:31 +0000] <ip address> arn:aws:iam::000000000000:user/bucketuser E0020120120202 REST.PUT.OBJECT 1421716901_gitlab_backup.tar "PUT /1421716901_gitlab_backup.tar HTTP/1.1" 400 EntityTooLarge 367 - 10 - "-" "fog/1.21.0" -

    Gitlab backup log snippet:

    Creating backup archive: 1421716901_gitlab_backup.tar ... rake aborted!
Excon::Errors::SocketError: Connection reset by peer (Errno::ECONNRESET)

Errno::ECONNRESET: Connection reset by peer

Excon::Errors::SocketError: Connection reset by peer (Errno::ECONNRESET)

Errno::ECONNRESET: Connection reset by peer

Excon::Errors::SocketError: Connection reset by peer (Errno::ECONNRESET)

Errno::ECONNRESET: Connection reset by peer

Excon::Errors::SocketError: Connection reset by peer (Errno::ECONNRESET)

Errno::ECONNRESET: Connection reset by peer

Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)

    It appears that Fog can be instructed to use multipart uploads:

    https://github.com/fog/fog-aws/blob/master/tests/models/storage/file_tests.rb#L70

    Designs

      Child items ...

      2. Activity

      • username-removed-14082
        username-removed-14082 @toricls ·

        +1
        I have a similar issue on GitLab 7.5.3 installation from source on Amazon Linux.
        My tar is nearly 6GB.

        Here is the log:

        Creating backup archive: 1422904007_gitlab_backup.tar ... done
Uploading backup archive to remote storage <bucket-name> ... rake aborted!
Excon::Errors::SocketError: Connection reset by peer (Errno::ECONNRESET)
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/socket.rb:120:in `write'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:189:in `request_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/mock.rb:42:in `request_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/instrumentor.rb:22:in `request_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:15:in `request_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:15:in `request_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:15:in `request_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:265:in `request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/idempotent.rb:12:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:10:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:10:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:288:in `rescue in request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:225:in `request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/idempotent.rb:12:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:10:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:10:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:288:in `rescue in request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:225:in `request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/idempotent.rb:12:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:10:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/middlewares/base.rb:10:in `error_call'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:288:in `rescue in request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/excon-0.32.1/lib/excon/connection.rb:225:in `request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/fog-core-1.21.1/lib/fog/core/connection.rb:56:in `request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/fog-1.21.0/lib/fog/xml.rb:24:in `request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/fog-1.21.0/lib/fog/aws/storage.rb:548:in `request'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/fog-1.21.0/lib/fog/aws/requests/storage/put_object.rb:32:in `put_object'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/fog-1.21.0/lib/fog/aws/models/storage/file.rb:211:in `save'
/home/git/gitlab/.bundle/ruby/2.0.0/gems/fog-core-1.21.1/lib/fog/core/collection.rb:51:in `create'
/home/git/gitlab/lib/backup/manager.rb:45:in `upload'
/home/git/gitlab/lib/backup/manager.rb:29:in `pack'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:15:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)

        S3 bucket log:

        0a96ba55b24a209397be04f9f0131ea0f42988514de09a496fc5a81a9ddd8a3b <bucket-name> [02/Feb/2015:19:48:56 +0000] <ip-address> arn:aws:iam::000000000000:user/<user-name> DAF1780549358B6C REST.GET.BUCKET - "GET / HTTP/1.1" 200 - 566 - 648 647 "-" "fog/1.21.0" -
0a96ba55b24a209397be04f9f0131ea0f42988514de09a496fc5a81a9ddd8a3b <bucket-name> [02/Feb/2015:19:48:58 +0000] <ip-address> arn:aws:iam::000000000000:user/<user-name> 10250852F9EC5692 REST.PUT.OBJECT 1422906310_gitlab_backup.tar "PUT /1422904007_gitlab_backup.tar HTTP/1.1" 400 EntityTooLarge 367 - 14 - "-" "fog/1.21.0" -
      • username-removed-5353
        username-removed-5353 @mmonge ·

        I have the same problem with Gitlab 7.7.2 (Omnibus) uploading the backup to Rackspace Cloud Files.

        I found some info about the issue related with the Linux Kernel: http://scie.nti.st/2008/3/14/amazon-s3-and-connection-reset-by-peer/

        There will be a fix for this on Gitlab? or should we try to fix it with kernel params?

      • username-removed-14082
        username-removed-14082 @toricls ·

        Quick patch. I tested below on GitLab 7.7.2 98ef3925

        diff --git lib/backup/manager.rb lib/backup/manager.rb
index ab8db4e..ad6ac52 100644
--- lib/backup/manager.rb
+++ lib/backup/manager.rb
@@ -41,7 +41,7 @@ module Backup

       connection = ::Fog::Storage.new(connection_settings)
       directory = connection.directories.get(remote_directory)
-      if directory.files.create(key: tar_file, body: File.open(tar_file), public: false)
+      if directory.files.create(key: tar_file, body: File.open(tar_file), public: false, multipart_chunk_size: 104857600)
         $progress.puts "done".green
       else
         puts "uploading backup to #{remote_directory} failed".red

        I guess the library fog supports multipart upload but the upload method is not using that currently.

        S3 Document: http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
        fog PR: https://github.com/fog/fog/pull/776

      • username-removed-138453
        username-removed-138453 @j0sh3rs ·

        +1 -- I had to implement this in my rails installation, which is currently running latest build of 7.10.4 EE.

      • username-removed-29582 mentioned in merge request !789 (merged)

        mentioned in merge request !789 (merged)

      • username-removed-444 mentioned in commit 8eadfd2d

        mentioned in commit 8eadfd2d

      • username-removed-14082
        username-removed-14082 @toricls ·

        @jnijhof Great fix! I think we can close this issue :)

      • username-removed-2900
        username-removed-2900 @razer6 ·
        Maintainer

        This is fixed now.

      • username-removed-2900 Status changed to closed

        Status changed to closed

      • Nikola Milojevic mentioned in merge request !14785

        mentioned in merge request !14785

      Please register or sign in to reply