Skip to content
Snippets Groups Projects
New issue look: Off

Gitlab cannot handle large commits.

    • Closed Issue created by gitlab-qa-bot @gitlab-qa-bot

    Created by: dubcanada

    Hey,

    I have been using Gitlab in production for a while now and recently I brought in some repos with a crapload of PDF files. And gitlab is either 500 erroring on view or displaying 0 changed files.

    I'm not sure if it is something that can be fixed or not (it might be a Grit issue, but I don't see any issues with anyone mentioning large commit issues), but it is an issue that needs to be looked at.

    Designs

    Child items 0

    • No child items are currently open.

    Activity

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dubcanada

      I also did a commit with a 36 mb pack file (approx 4000 files (images, php files, text, database schemes, all text and images)) and it won't load the commit information.

      It may be a good idea to load the commit page then ajax load the commit information, to prevent the 500 error's and massively long waiting times for fairly large commits.

      By Administrator on 2012-04-04T18:55:47 (imported from GitLab project)

      By Administrator on 2012-04-04T18:55:47 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dubcanada

      The very first commit should also not show a commit history, as chances are it is too large to display. Or don't display the diff only, just the files added.

      Gitorious for example has options for Raw Diff (without graphics) and Raw Patch for very first commit. And doing the same 36 mb pack file upload on their works, but the Raw Diff and Raw Patch report

      This Git object is too large to be displayed in the browser Consider cloning the repository locally and look at the object there

      Which is true. And which is acceptable.

      Unless we are somehow able to duplicate GitHub, which manages to display the diff fine (slightly slow, but still works).

      Showing 4,022 changed files with 674,249 additions and 0 deletions.

      It also seems to only show 50% of those (probably about 500-1000 diffs) and the rest display.

      205 additions, 0 deletions not shown.

      By Administrator on 2012-04-04T19:25:53 (imported from GitLab project)

      By Administrator on 2012-04-04T19:25:53 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dzaporozhets

      I think GitHub has a bigger git_max_size option. In GitLab Grit raise error cause commit is too big. I'm trying to get only stat with files changed in this case. I'll try to do something with it

      By Administrator on 2012-04-04T19:52:18 (imported from GitLab project)

      By Administrator on 2012-04-04T19:52:18 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dzaporozhets

      removing this monkey patch resolve problem with big commits. but this will cause encoding errors

      By Administrator on 2012-04-04T20:56:15 (imported from GitLab project)

      By Administrator on 2012-04-04T20:56:15 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dzaporozhets

      Can you test this commt? 80ddd2c0

      By Administrator on 2012-04-04T22:52:35 (imported from GitLab project)

      By Administrator on 2012-04-04T22:52:35 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dubcanada

      It did help slightly, but it is still doesn't display the large commit (4000+ files). But it didn't timeout either, it just kind of crashed my browser.

      The problem seems is that it is trying to get the blob contents for every single file (minus the binary) and display it.

      Is there a way just for my testing sake to limit the amount of times it gets the blob contents?

      By Administrator on 2012-04-05T02:19:02 (imported from GitLab project)

      By Administrator on 2012-04-05T02:19:02 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dzaporozhets

      Browser crash mean that now we have enough memory to keep dif - its great. So now we can do some limitations for commit output.

      Ex.

      1. Suppress file diff > 1000 lines (new big file added)
      2. Limit diff for files to 500 files.

      In most cases people avoid large commits so it will be useful only in one case - commiting whole project (ex. migrate from svn etc)

      By Administrator on 2012-04-05T08:02:35 (imported from GitLab project)

      By Administrator on 2012-04-05T08:02:35 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dzaporozhets

      This one should help you 86676476. I close task cause now gitlab handle big commits well enough.

      By Administrator on 2012-04-05T22:16:44 (imported from GitLab project)

      By Administrator on 2012-04-05T22:16:44 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: dubcanada

      Just to bring this back up. I was still experiencing a 500 error when viewing a few commits (I did a few more). Upon removing the charlock holmes stuff it works perfectly fine.

      This fix will work for me because I work with same charset for everything. But this won't work for those who need charlock holmes.

      I am wondering if maybe rather then loading every single file transcoded at the same time, could you perhaps show a loading thing and do maybe 500 at a time or something? Just something to queue it a bit more rather then all in a big giant chunk to let charlock holmes not run out of memory. Because if it is trying to reencode 4000 files I can easily see how it would crash. It could also be that my 4 year old server does not have enough horsepower to process it.

      By Administrator on 2012-04-06T15:43:36 (imported from GitLab project)

      By Administrator on 2012-04-06T15:43:36 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: hpcarlos

      I try the same problem in my case I have a repository and update was made about 2000 files is giving the the same problem that @dubcanada.

      However I could see that my prorblema comes down to memory, leaving him even if I use more than 500MB of memory, he kept asking for more memory.

      The use of memory for processing the files is too high.

      By Administrator on 2012-04-16T21:08:31 (imported from GitLab project)

      By Administrator on 2012-04-16T21:08:31 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: davvid

      We're seeing this issue. When we hit a large commit puma starts spinning at 100% CPU (waiting on a futex) and never comes back. We've increased our git max_size to 16MB (the "big" commit is 2MB, and added binary files).

      Anybody else run into puma issues? Running top shows this:

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                  
39863 gitlab    20   0 1170m 530m 4164 S 99.6  3.3   1:37.75 ruby /home/gitlab/vendor/bundle/ruby/1.9.1/bin/puma -e production -C /home/gitlab/config/puma.rb

      strace output for puma

      $ sudo strace -p 35374
Process 35374 attached - interrupt to quit
futex(0xf555b4, FUTEX_WAIT_PRIVATE, 1, NULL
^C <unfinished ...>
Process 35374 detached

      and the gdb backtrace while it's hung:

      #0  0x0000003121c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005238a9 in native_cond_wait (cond=<value optimized out>, mutex=<value optimized out>) at thread_pthread.c:306
#2  0x0000000000528d8b in native_sleep (th=0xf55520, timeout_tv=0x0) at thread_pthread.c:906
#3  0x0000000000529380 in sleep_forever (arg=140736919621088) at thread.c:855
#4  thread_join_sleep (arg=140736919621088) at thread.c:688
#5  0x00000000004150ab in rb_ensure (b_proc=0x5292c0 <thread_join_sleep>, data1=140736919621088, e_proc=0x5234f0 <remove_from_join_list>, data2=140736919621088) at eval.c:744
#6  0x000000000052708b in thread_join (target_th=0x7aeb6a0, delay=<value optimized out>) at thread.c:721
#7  0x00000000005271d6 in thread_join_m (argc=0, argv=0x7f72ff7cb0f8, self=<value optimized out>) at thread.c:802
#8  0x00000000005120f7 in vm_call_cfunc (th=<value optimized out>, cfp=0x7f72ff8cad50, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x106e2e0, 
    recv=123250000) at vm_insnhelper.c:404
#9  vm_call_method (th=<value optimized out>, cfp=0x7f72ff8cad50, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x106e2e0, recv=123250000)
    at vm_insnhelper.c:530
#10 0x00000000005147ad in vm_exec_core (th=0xf55520, initial=<value optimized out>) at insns.def:1018
#11 0x0000000000518828 in vm_exec (th=0xf55520) at vm.c:1236
#12 0x0000000000518e76 in rb_iseq_eval (iseqval=25877360) at vm.c:1464
#13 0x000000000056ed65 in rb_load_internal (fname=25879720, wrap=<value optimized out>) at load.c:310
#14 0x000000000056eecb in rb_f_load (argc=<value optimized out>, argv=<value optimized out>) at load.c:384
#15 0x00000000005120f7 in vm_call_cfunc (th=<value optimized out>, cfp=0x7f72ff8caf08, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x105bf10, 
    recv=16484080) at vm_insnhelper.c:404
#16 vm_call_method (th=<value optimized out>, cfp=0x7f72ff8caf08, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x105bf10, recv=16484080)
    at vm_insnhelper.c:530
#17 0x00000000005147ad in vm_exec_core (th=0xf55520, initial=<value optimized out>) at insns.def:1018
#18 0x0000000000518828 in vm_exec (th=0xf55520) at vm.c:1236
#19 0x0000000000518c45 in rb_iseq_eval_main (iseqval=25863120) at vm.c:1478
#20 0x0000000000415732 in ruby_exec_internal (n=0x18aa3d0) at eval.c:204
#21 0x00000000004177d6 in ruby_exec_node (n=<value optimized out>) at eval.c:251
#22 ruby_run_node (n=<value optimized out>) at eval.c:244
#23 0x0000000000414929 in main (argc=4, argv=0x7fffde19d888) at main.c:38
(gdb) 
#0  0x0000003121c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005238a9 in native_cond_wait (cond=<value optimized out>, mutex=<value optimized out>) at thread_pthread.c:306
#2  0x0000000000528d8b in native_sleep (th=0xf55520, timeout_tv=0x0) at thread_pthread.c:906
#3  0x0000000000529380 in sleep_forever (arg=140736919621088) at thread.c:855
#4  thread_join_sleep (arg=140736919621088) at thread.c:688
#5  0x00000000004150ab in rb_ensure (b_proc=0x5292c0 <thread_join_sleep>, data1=140736919621088, e_proc=0x5234f0 <remove_from_join_list>, data2=140736919621088) at eval.c:744
#6  0x000000000052708b in thread_join (target_th=0x7aeb6a0, delay=<value optimized out>) at thread.c:721
#7  0x00000000005271d6 in thread_join_m (argc=0, argv=0x7f72ff7cb0f8, self=<value optimized out>) at thread.c:802
#8  0x00000000005120f7 in vm_call_cfunc (th=<value optimized out>, cfp=0x7f72ff8cad50, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x106e2e0, 
    recv=123250000) at vm_insnhelper.c:404
#9  vm_call_method (th=<value optimized out>, cfp=0x7f72ff8cad50, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x106e2e0, recv=123250000)
    at vm_insnhelper.c:530
#10 0x00000000005147ad in vm_exec_core (th=0xf55520, initial=<value optimized out>) at insns.def:1018
#11 0x0000000000518828 in vm_exec (th=0xf55520) at vm.c:1236
#12 0x0000000000518e76 in rb_iseq_eval (iseqval=25877360) at vm.c:1464
#13 0x000000000056ed65 in rb_load_internal (fname=25879720, wrap=<value optimized out>) at load.c:310
#14 0x000000000056eecb in rb_f_load (argc=<value optimized out>, argv=<value optimized out>) at load.c:384
#15 0x00000000005120f7 in vm_call_cfunc (th=<value optimized out>, cfp=0x7f72ff8caf08, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x105bf10, 
    recv=16484080) at vm_insnhelper.c:404
#16 vm_call_method (th=<value optimized out>, cfp=0x7f72ff8caf08, num=<value optimized out>, blockptr=0x0, flag=<value optimized out>, id=<value optimized out>, me=0x105bf10, recv=16484080)
    at vm_insnhelper.c:530
#17 0x00000000005147ad in vm_exec_core (th=0xf55520, initial=<value optimized out>) at insns.def:1018
#18 0x0000000000518828 in vm_exec (th=0xf55520) at vm.c:1236
#19 0x0000000000518c45 in rb_iseq_eval_main (iseqval=25863120) at vm.c:1478
#20 0x0000000000415732 in ruby_exec_internal (n=0x18aa3d0) at eval.c:204
#21 0x00000000004177d6 in ruby_exec_node (n=<value optimized out>) at eval.c:251
#22 ruby_run_node (n=<value optimized out>) at eval.c:244
#23 0x0000000000414929 in main (argc=4, argv=0x7fffde19d888) at main.c:38

      We're using the default puma settings. Should we change the thread count numbers? perhaps it's trying to go down to zero? (the default is 0, 16).

      By Administrator on 2013-08-06T21:13:36 (imported from GitLab project)

      By Administrator on 2013-08-06T21:13:36 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: davvid

      Ahh.. I'm also seeing this in our logs:

      /home/gitlab/gitlab/vendor/bundle/ruby/1.9.1/gems/gitlab-grit-2.5.1/lib/grit/git-ruby.rb:119: warning: conflicting chdir during another chdir block

      repeated many, many times.

      Ooh! This verifies the hunch I had earlier while investigating #4668 -- gitlab is doing File.chdir() behind the scenes (e.g. the Grit repo.refs() function) and that can be very bad.

      For example, the satellite merge request code depends on the current directory being correct. Why? Grit passes --git-dir to git but does not pass --work-tree. That's a recipe for disaster. If you don't specify the --work-tree then git will use the current directory, whatever that may be, and use it. So if you do something like "git clean -fdx" or "git reset --hard" and someone else does chdir() behind its back, then git will go and totally clobber whatever directory was chdir()'d to.

      This is really a problem in Grit. It should use both --git-dir and --work-tree. Using one without the other is a recipe for disaster.

      Anyways, so there are multiple things going on here. This issue is specifically about large commits, and I think the problem is puma.

      By Administrator on 2013-08-06T21:20:35 (imported from GitLab project)

      By Administrator on 2013-08-06T21:20:35 (imported from GitLab)

    • gitlab-qa-bot
      gitlab-qa-bot @gitlab-qa-bot ·
      Author Owner

      Created by: davvid

      I retract "blame puma" -- we upped our git timeout to 10 minutes and also changed our Apache configuration to have "Timeout 600" and we're now able to see large commits. Granted, they do take a long time to render (6 minutes for a diff with ~100K changed lines), but at least we see them now. Anyways, mentioning it here in case anyone else is also running gitlab through a proxy (in our case, that's apache).. you'll also need to up your proxy timeout limits.

      By Administrator on 2013-08-06T21:58:55 (imported from GitLab project)

      By Administrator on 2013-08-06T21:58:55 (imported from GitLab)

    Please register or sign in to reply