Performance issues with > 1000 projects in a group

• SF link: https://na34.salesforce.com/00Q6100000BHKa3EAH

A potential customer has a use case where they need to have 10000-50000 projects per group. They mentioned performance issues.

@dzaporozhets didn't we test for this some months ago?

@yorickpeterse can you set up a test case and see what is slow?

Performance issues with what exactly? Viewing issues of individual projects? Issues per group? Viewing milestones? Also do these projects have lots of issues per project?

@yorickpeterse I asked for more detail. I don't know at this time.

Updated: 10k to 50k projects per group. Still waiting for further info.

Here is the missing text he sent me,“Though we haven’t examined the source code, our experiments show significant performance problems when the number of projects (repos) in a group exceeds 1000. One problem is with the UI which attempts to display all projects in a group on one page. But there also appear to be database operations that bog down. We think that all that would be required is pagination of the group browser section of the UI and some tuning of the database indices.”

Sending email with another document they sent for this.

If it would be best for him, myself, and one of you to have a call, let me know and I can set it up. Thanks again for looking into this.

I am having a call with him and his co-worker this Thursday at 10am PST. I am inviting one of the Service Engineer's, or if anyone else would like to join let me know. They are wondering what our interest level in this is, so if it would be helpful to talk to them directly let me know.

@yorickpeterse @JobV Do you still have a test environment for this?

@balameb I didn't set anything up to test the exact amounts, but if needed I think that should be doable via the API. Alternatively they could use the upcoming performance monitoring system, though for the time being they'd have to set up InfluxDB and the likes themselves.

@JobV Is there any specific information Jose and I should find out from them that would help us determine if we are interested in doing this? Our call is tomorrow morning California time.

@dblessing @jwrede it would be great if they could provide / if you could investigate exactly what is slow and what isn't the more we know about what bothers them, the easier it'll be to fix.

In theory GitLab should work fine with any amount.

@job a report from them was shared through @jwrede. Not sure if we can make it public though. Maybe move to a dev issue?

Thanks @JobV. And thanks for mentioning that @balameb. I sent it in an e-mail after removing from here, but maybe we should put this in dev so I can link it there again. I think it explains the issues they found.

I'll take care of that @jwrede

Reassigned to @dblessing

Assigning to myself for testing.

@yorickpeterse @joshfng The requirement from the customer is that we support 50,000 projects in a single group with reasonable performance on key pages, like group page. Once I have the demo with all the projects can we do a quick hangout to discuss some metrics we should grab (early next week)? I want to make sure we have all the data we need to make improvements.

Pagination should be implemented regardless of how many projects we want to support. In my opinion 50 000 or 10 billion projects shouldn't matter, with pagination pages should load consistently and fast (enough).

Another note: the projects list of a group should only load data from the database. If it ends up doing any Git stuff secretly we have to get rid of this. There's no point in checking branches or whatever when we just want to list the existing projects. I don't know if there's currently any Git code involved.

Assignee removed

Thanks for your work on this @joshfng and @yorickpeterse It's much appreciated.

I have the same performance issue when I mirror about 2000 my company projects into one Gitlab group.

It's basically just timeout in following cases:

1. get group info from group API (which will contains project list)
2. '/groups/[group_name]' page when the group own all those projects
3. '/dashboard/projects' page when login user own all those projects

the third one is slowest one, it need takes minutes to load page (where I have to set long timeout for nginx)

Hope you can solve it! Then I might have chance to host all company projects on Gitlab.

Thank you!

Milestone changed to 8.6

Is this still a problem or did we resolve it?

Milestone changed to 8.7

Status changed to closed