post_receive job never finishes

@brodock Please could you take a look?

added Platform label

marked this issue as related to #13282 (moved)

marked this issue as related to #18663 (moved)

added bug label

@marcfinet Hi Marc, I think a real fix here will require a few Merge Requests. GitLab tries to do in a single execution the whole post_receive processing, which works well enough for small to mid size changes, but can be problematic when pushing something as big as the linux kernel in a single shot.

I think the temporary workaround here is to try to temporary increase even more the SIDEKIQ_MEMORY_KILLER_MAX_RSS (4Gb?).

[pid 18184] 14:05:59.808361 stat("/home/gitlab/git-data/repositories/<REDACTED>.git/config", {st_mode=S_IFREG|0644, st_size=66, ...}) = 0
[pid 18184] 14:05:59.808388 open("/home/gitlab/git-data/repositories/<REDACTED>.git/config", O_RDONLY|O_CLOEXEC) = 45
[pid 18184] 14:05:59.808409 read(45, "[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = true\n", 66) = 66

the bunch of similar operations like this one can be because we shell out to binary git cli for a few operations that would not be fast enough or would leak memory when using rugged (the C-based library to operate the git repository from ruby).

we could probably reduce the frequency of SELECT FEATURES with some caching per worker execution (similar to RequestCache).

Also building Hooks payload may take a lot of time and memory as we are adding to memory a lot of expensive datastructures to then dump a JSON from it.

We could do this differently when amount of changes is > then X:

Build the hook data only when/if needed (this means using lazy evaluation structures), perhaps building the json in an async way so instead of huge amount of hashes fo strings, we build a small amount of then, and flush to JSON, then build another batch and flush to json etc).

Also the execution loop should be resumable/idempotent, we should store the last processed change, for example, and when it fails, the next time we just need to resume it.

(The lazy processing of references would also help achieve this).

I'm also not aware if gitaly could provide any help in this type of situation, or the burden is more on the ruby side and how we choose to do things.

cc @DouweM @stanhu @andrewn @yorickpeterse

added memory usage label

An issue with similar symptoms was reported at https://gitlab.com/gitlab-org/gitlab-ce/issues/35531, so hopefully https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13132 will help here too.

@marcfinet To make sure I understand what happened:

Did the user try to push the entire linux kernel repo, including all commits, trees, blobs, or "just" 200 tags pointing to commits that were already in the repo?
What GitLab version are you currently running?
Does manually applying the patch in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13132 help?

@brodock I think the memory issue happens here: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13132/diffs#d9183678fb9fe8649cc75764316de010bd369746_69_75. We were iterating over all pushed commits, and filling the paths set with paths of all files that were touched in those commits. See https://gitlab.com/gitlab-org/gitlab-ce/issues/35531#note_36058891 and https://gitlab.com/gitlab-org/gitlab-ce/issues/35531#note_36061180

marked this issue as related to #35531 (closed)

@DouweM based on environment info (it's collapsed in the issue), it's using GitLab version 9.4.2.

@brodock Ah, right, missed that. Hopefully that patch will help at least somewhat.

I agree in general that there's a lot we can do to improve PostReceive.

Thanks for updates. Here are mines:

@brodock: bumping to 4GiB made post_receive finished its jobs
@DouweM:
meaning that I could not test the aforementioned patch (i'll reduce memory back to 2GiB and wait for a linux kernel push...)
version was 9.4.2 (and a bump to 9.4.5 didn't fix the issue though)
the user pushed the whole linux kernel tree, as done in #35531 (closed) (that I failed to find, probably because neither post_receive nor PostReceive was in issue description, only in discussion).

We might close this issue as duplicate of #35531 (closed). Just a few things:

I just hope that activity flood is not an issue
this use-case (pushing a linux-kernel tree) used to work (according to my user, but I won't blindly rely on his saying since he might have pushed only a snapshot, not a whole git tree), and I guess that gitlab.com already hosts a few linux-kernel mirrors.

I just hope that activity flood is not an issue

@marcfinet What do you mean exactly? It's a bug, and annoying for your users, but otherwise it won't cause any problems.

this use-case (pushing a linux-kernel tree) used to work (according to my user, but I won't blindly rely on his saying since he might have pushed only a snapshot, not a whole git tree), and I guess that gitlab.com already hosts a few linux-kernel mirrors.

I think it used to work; the poorly performing code that I optimized as part of https://gitlab.com/gitlab-org/gitlab-ce/issues/35531 had been added relatively recently.

I just hope that activity flood is not an issue

@marcfinet What do you mean exactly? It's a bug, and annoying for your users, but otherwise it won't cause any problems.

I was thinking of the volume.

Great for the think that has been patch is related to recent changes. It's was thinking about, how could this "bug" not having been noticed previously.

closed

post_receive job never finishes

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

Designs

Child items 0

Activity

Admin message

Admin message

post_receive job never finishes

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

Relates to

Activity