Enabling maximum file size limit in repository causes pushes to fail

ZD: https://gitlab.zendesk.com/agent/tickets/82084

Steps to reproduce:

1. In the Push Rules settings under repository (settings/repository), set the max file limit to 5 MB.
2. Attempt to push the GitLab EE repository.

In the /internal/allowed endpoint, Unicorn will eventually gobble a lot of memory, spin, and get killed after exceeding the 60 s timeout:

$ git push stanhu-test --mirror
Counting objects: 757161, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (155822/155822), done.
Writing objects: 100% (757161/757161), 305.04 MiB | 31.21 MiB/s, done.
Total 757161 (delta 596177), reused 753066 (delta 592636)
remote: Resolving deltas: 100% (596177/596177), done.
remote: Checking connectivity: 757161, done.

remote: GitLab: Failed to authorize your Git request: internal API unreachable
To git@stanhu-geo-primary.gitlap.com:root/gitlab-ee-test3.git
 ! [remote rejected] master -> master (pre-receive hook declined)
 ! [remote rejected] v0.9.4 -> v0.9.4 (pre-receive hook declined)
 ! [remote rejected] v0.9.5 -> v0.9.5 (pre-receive hook declined)
 ! [remote rejected] v0.9.6 -> v0.9.6 (pre-receive hook declined)
<snip>
...

What's interesting is the strace output looks like an endless re-reading of gitconfig files:

10819 stat("/var/opt/gitlab/git-data/repositories/root/gitlab-ee-test3.git/config", 0x7fffb6168200) = 0
10819 open("/var/opt/gitlab/git-data/repositories/root/gitlab-ee-test3.git/config", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = true\n", 66) = 66
10819 close(21)                         = 0
10819 stat("/var/opt/gitlab/.gitconfig", {st_mode=S_IFREG|0644, st_size=306, ...}) = 0
10819 open("/var/opt/gitlab/.gitconfig", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "# This file is managed by gitlab-ctl. Manual changes will be\n# erased! To change the contents below, edit /etc/gitlab/gitlab.rb\n# and run `sudo gitlab-ctl reconfigure`.\n\n[user]\n        name = GitLab\n        email = gitlab@stanhu-geo-primary.gitlap.com\n[core]\n        autocrlf = input\n[gc]\n        auto = 0\n", 306) = 306
10819 close(21)                         = 0
10819 stat("/etc/gitconfig", {st_mode=S_IFREG|0644, st_size=126, ...}) = 0
10819 open("/etc/gitconfig", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "[filter \"lfs\"]\n\tclean = git-lfs clean -- %f\n\tsmudge = git-lfs smudge -- %f\n\tprocess = git-lfs filter-process\n\trequired = true\n", 126) = 126
10819 close(21)                         = 0
10819 stat("/var/opt/gitlab/git-data/repositories/root/gitlab-ee-test3.git/config", {st_mode=S_IFREG|0644, st_size=66, ...}) = 0
10819 open("/var/opt/gitlab/git-data/repositories/root/gitlab-ee-test3.git/config", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = true\n", 66) = 66
10819 close(21)                         = 0
10819 stat("/var/opt/gitlab/.gitconfig", {st_mode=S_IFREG|0644, st_size=306, ...}) = 0
10819 open("/var/opt/gitlab/.gitconfig", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "# This file is managed by gitlab-ctl. Manual changes will be\n# erased! To change the contents below, edit /etc/gitlab/gitlab.rb\n# and run `sudo gitlab-ctl reconfigure`.\n\n[user]\n        name = GitLab\n        email = gitlab@stanhu-geo-primary.gitlap.com\n[core]\n        autocrlf = input\n[gc]\n        auto = 0\n", 306) = 306
10819 close(21)                         = 0
10819 stat("/etc/gitconfig", {st_mode=S_IFREG|0644, st_size=126, ...}) = 0
10819 open("/etc/gitconfig", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "[filter \"lfs\"]\n\tclean = git-lfs clean -- %f\n\tsmudge = git-lfs smudge -- %f\n\tprocess = git-lfs filter-process\n\trequired = true\n", 126) = 126
10819 close(21)                         = 0
10819 sendto(12, "B\0\0\0\31\0a11\0\0\1\0\0\0\1\0\0\0\00219\0\1\0\0D\0\0\0\6P\0E\0\0\0\t\0\0\0\0\0S\0\0\0\4", 48, MSG_NOSIGNAL, NULL, 0) = 48
10819 poll([{fd=12, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
10819 recvfrom(12, "2\0\0\0\4T\0\0\0\34\0\1one\0\0\0\0\0\0\0\0\0\0\27\0\4\377\377\377\377\0\0C\0\0\0\rSELECT 0\0Z\0\0\0\5I", 16384, 0, NULL, NULL) = 54
10819 stat("/var/opt/gitlab/git-data/repositories/root/gitlab-ee-test3.git/config", {st_mode=S_IFREG|0644, st_size=66, ...}) = 0
10819 open("/var/opt/gitlab/git-data/repositories/root/gitlab-ee-test3.git/config", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = true\n", 66) = 66
10819 close(21)                         = 0
10819 stat("/var/opt/gitlab/.gitconfig", {st_mode=S_IFREG|0644, st_size=306, ...}) = 0
10819 open("/var/opt/gitlab/.gitconfig", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "# This file is managed by gitlab-ctl. Manual changes will be\n# erased! To change the contents below, edit /etc/gitlab/gitlab.rb\n# and run `sudo gitlab-ctl reconfigure`.\n\n[user]\n        name = GitLab\n        email = gitlab@stanhu-geo-primary.gitlap.com\n[core]\n        autocrlf = input\n[gc]\n        auto = 0\n", 306) = 306
10819 close(21)                         = 0
10819 stat("/etc/gitconfig", {st_mode=S_IFREG|0644, st_size=126, ...}) = 0
10819 open("/etc/gitconfig", O_RDONLY|O_CLOEXEC) = 21
10819 read(21, "[filter \"lfs\"]\n\tclean = git-lfs clean -- %f\n\tsmudge = git-lfs smudge -- %f\n\tprocess = git-lfs filter-process\n\trequired = true\n", 126) = 126

/cc: @lbot

In pushing a new EE repository, I see that a new push for master, there are 73,969 commits that need to be scanned by Gitlab::Checks::ChangeAccess. The push rule then scans every delta in each commit to determine whether any file exceeds the maximum limit.

Keep in mind this is just one ref. There are hundreds of others.

These file size limits won't scale for large pushes. At some point, I think we need to limit the of commits and refs we can scan.

What do you think, @eReGeBe @yorickpeterse?

Ok, wow. The file size and project size limits are being bit by this Rugged bug: https://github.com/libgit2/rugged/issues/695

Essentially if you iterate through lots of diffs, memory usage will skyrocket unless you call Rugged::Repository#close, which does NOT actually close the repo (http://www.rubydoc.info/github/libgit2/rugged/Rugged%2FRepository%3Aclose):

Frees all the resources used by this repository immediately. The repository can still be used after this call. Resources will be opened as necessary.

This is what I did to test that this, in fact, was happening. In a Rails console with the GitLab CE repository pushed to a repo, enable file size limits and then run:

# Monkey patch `RevList` to get back all refs
module Gitlab
  module Git
    class RevList
      def new_refs
        execute([*base_args, newrev, '--all'])
      end
    end 
  end
end

project = Project.find_by_full_path('root/gitlab-ce-test')
access = Gitlab::GitAccess.new(User.find(1), project, 'ssh', authentication_abilities: [:push_code])
changes = ['0' * 40 + ' 0cd1563fb66e9ba533346fe3957912225e27000f']
access.check('git-receive-pack', changes)

Let that run for a while and watch memory climb like crazy. If you hit CTRL-C, the memory doesn't come back, even after running GC.start.

But if you run project.repository.raw.rugged.close, memory gets freed.

mentioned in issue #2990

@stanhu Apart from scanning over everything that is pushed the one alternative I can think of is to check the size retroactively. That is, if we're not over the size limit yet we'll allow the push. Once the push is done we calculate the new repository size (async), and lock it if it exceeds the size. A big downside is that this would allow you to push e.g. 1 TB in the first commit even if your size limit is 10 GB.

Unfortunately I'm not aware of getting the file sizes and such in a commit without having to go through Rugged.

@mydigitalself @DouweM I see we are adding more push rules in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2913. Right now I think all of our push rules need some attention because:

1. They can make initial pushes fail for unknown reasons
2. They gobble up memory like crazy

In general, we tell our customers to turn off all push rules if they are having trouble.

changed milestone to %Next 3-6 months

changed milestone to %Next 2-3 months