7.10.1 regression - Unable to render an issue

Right after upgrading to 7.10.1, I started to get "GitLab is not responding." messages.

According to the log, I see following:

Started GET "/e/bnv/issues/2222" for 127.0.0.1 at 2015-05-05 10:00:25 -0400
Processing by Projects::IssuesController#show as HTML
  Parameters: {"namespace_id"=>"xxx", "project_id"=>"xxx", "id"=>"2222"}

in unicorn.stderr.log I see following:

I, [2015-05-05T10:10:10.611953 #5873]  INFO -- : worker=0 ready
E, [2015-05-05T10:13:34.794670 #2718] ERROR -- : worker=0 PID:5873 timeout (61s > 60s), killing
E, [2015-05-05T10:13:34.817961 #2718] ERROR -- : reaped #<Process::Status: pid 5873 SIGKILL (signal 9)> worker=0
I, [2015-05-05T10:13:34.824679 #5884]  INFO -- : worker=0 ready

And then nothing in browser I will get timeout.

mentioned in issue #1438 (closed)

I've traced this down to GitLab taking much longer time rendering very heavy markdown table. Increasing timeout for unicorn from 60 seconds to 90 solved it, but in version 7.9.1 rendering took about 9-12 seconds according to historical logs.

Large table: 12 columns, 154 rows.

As permanent fix I moved that table into wiki and it takes 4 seconds there to render it.

@sashk can you give us a test repo (maybe here on GitLab.com) to show this off? I don't think this is a regression until confirmed.

Fine. Try viewing issue I just created.

Here is how it looks in my browser's window:

IMHO, this is potentially DoS issue, if I decide to post many posts like this, it will use up all your workers and GitLab is done.

Another problem I see, I posted single issue in the project, but GitLab ended up creating 4 more.

Added ~16800 reproduced on GitLab.com labels

Another issue related to markdown rending, which is regression in 7.10.1: Try rendering code block inside of the item list:

1. Code block bellow was rendered properly in 7.9 and before, but fails in 7.10.1:

def test_xxx(): assert xxx() == 1

def test_yyy(): assert yyy() == 2

Both functions should have been rendered in the single code block, and this text is no longer part of the code block.

mentioned in issue #1438 (closed)

@JobV - ping - could you please take look?

Thanks @sashk, we're looking into it

Notable changes in MD since 7.9:

• https://gitlab.com/gitlab-org/gitlab-ce/commit/cc29ce491786d631586c3b0d0da310b8b790a673

@rspeicher @DouweM could you have a look at this?

I see that @rspeicher contributed some changes since last release.

@sashk I took a quick look into both things you mentioned.

First, tables. My personal GitLab installation runs 7.9.4 on a pretty modest VPS and it rendered a 12-column, 400-row Markdown table in less than five seconds. My local development machine, running in development not production, on the latest master, rendered the same table in a little over six seconds. So I'm not sure why a smaller (though still large) table on your installation is taking so long. And it's odd to me that it would work quickly in a Wiki page but not an Issue, since they both render Markdown through the same markdown helper in GitLab.

However my test table was just simple text inside the cells. Can you let me know a bit more about the text in your table? Does it have references to issues, users, merge requests, etc.? That would slow things down considerably in 7.9 and 7.10, I believe. I think my reference changes went into the upcoming 7.11 release.

Second, code blocks in a list. My 7.9.4 installation also rendered it incorrectly, unfortunately I don't have an earlier version running to test there. I'll look into this more. I think this is just an edge case that Redcarpet doesn't handle correctly.

@rspeicher I've created test issue here on gitlab.com - as you can see, when you try to access it - sashk/test#1 (closed) - Gitlab times out after about 60 seconds (or whatever your timeout is on gitlab.com for unicorn). Table in that issue has 12 columns of random 7 digit numbers. There is smaller issue - sashk/test#6 (closed) - which you can

As to rendering of code blocks - this code was working as intended in 7.9.1, but fails to render properly in 7.10.1.

I never had 7.9.4 - I was upgrading from 7.9.1 to 7.10.1.

My personal GitLab installation runs 7.9.4 on a pretty modest VPS and it rendered a 12-column, 400-row Markdown table in less than five seconds.

Yes, it's how it was in 7.9.1 for us.

My local development machine, running in development not production, on the latest master, rendered the same table in a little over six seconds.

Were any changes made to markdown between 7.10.x and current master?

Were any changes made to markdown between 7.10.x and current master?

Yes. I'm not sure in which version 96c2b940 was merged, but it was a major Markdown processing overhaul. I'm confident this will get better for you in 7.11 at the absolute latest.

I see it only exists in master and other branches, but not in x.xx-stable, so I guess will need to wait for 7.11 :-(

But in any case - if there is quick fix which can go into 7.10.x - it will help us.

Not fixed in 7.10.4.

Sorry @sashk, we need to have ample time to build, test and release a patch release.

We typically don't ship huge changes in patches, such as the commit Robert linked, as that would risk bringing other issues. I'll note this issue in case of another patch release, but that doesn't mean it'll get fixed with a patch release.

Can confirm, that large issue rendering seems to be fixed on gitlab.com now, that it runs 7.11.0.rc2, but issue with rendering code blocks still present.

mentioned in issue #1641 (closed)

@sashk Glad to hear the tables have gotten better.

Rendering code blocks in lists is an issue with our Markdown renderer, which is currently Redcarpet. I'll look into fixing this in the future but right now it's a lower priority.

@rspeicher - thank you for update. I'm wondering if renderer was upgraded for 7.10, as I did not have such issues in 7.9.4, but started seeing them in the 7.10.

ie if it was, may be quickest and easiest fix to revert that upgrade?

I'm wondering if renderer was upgraded for 7.10, as I did not have such issues in 7.9.4, but started seeing them in the 7.10.

Possibly. If you want to help pin it down, you can install different versions of Redcarpet and run the text you're trying through each one to see where the regression began. GitLab 7.9.0 required Redcarpet 3.1.2, so you can probably start there.

Edit: Keep in mind Redcarpet was upgraded for a security issue, so we can't simply roll it back.

Status changed to closed

@sashk Please open a new issue for the code-in-list bug, going to close this one as fixed.

mentioned in issue #1653 (closed)

Opened #1653 (closed).

mentioned in issue #1653 (closed)