https://gitlab.com/explore error 502

Can't open page https://gitlab.com/explore

https://gitlab.com/help: GitLab Enterprise Edition 7.12.2-ee 836505e

Added ~16800 ~50251 labels

/cc @jacobvosmaer

Looks like a Unicorn timeout, something is taking too long.

happens whether you are signed in or not

the page is slow in the console

irb(main):013:0> app.get '/explore'
Started GET "/explore" for 127.0.0.1 at 2015-07-16 14:34:32 +0000
Processing by Explore::ProjectsController#trending as HTML
Completed 200 OK in 33080ms (Views: 32368.8ms | ActiveRecord: 631.5ms)
=> 200

but now it loads.

maybe it's a cache issue? if the cache is cold the page load takes too long?

I think this SQL query cause 30 seconds execution - https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/finders/trending_projects_finder.rb#L9-11

@dzaporozhets I think I see a new challenge for you :)

Specifically this line:

select("projects.*, count(notes.id) as ncount").

Basically we get all projects from gitlab.com (which is a lot)

I think I see a new challenge for you :)

I will take care of this issue

Reassigned to @dzaporozhets

Milestone changed to 7.13

@dzaporozhets I think if you add a check to only get projects that had activity in the last month to that statement it should already be much faster :)
We have the field last_activity_at in the project so it's a simple where statement.

@haynes good idea. I also thought about sonething like

Select count(project_id) from notes group by project_id

Insted of quering projects with join

Ok its not about SQL. Although sql is not super efficient it takes only 1% of page load time. The rest is taken by template. Possibly by commit_count method. Recently few projects appeared on trending page with 200K commits and it was a reason of timeout. It was git trying to count all commits of 20 projects on trending page.

@dzaporozhets can we change it to use git rev-list HEAD --count to get the commit count of a repo?
That command seems to have a cache. I tested it on the linux mainline repo with 495570 commits.
The first run took ~20 seconds, all following calls took ~5 seconds.

Another option would be to change the text a bit and limit it to commits this year with something like : git rev-list HEAD --count --after="2014-07-27 13:37"
That command takes less than a second for the linux kernel for me and still returns ~30000 commits.

edit:
Personaly I like my second suggestion more. For trending projects you don't really need to see the total commits.
It would be enough to show the commits in one year to see how active a project is.

The first run took ~20 seconds

@haynes thats the problem we had at explore page. 2 repos were enough to get timeout

Another option would be to change the text a bit and limit it to commits this year

makes no sense to me. I want to know total commit count.

That command seems to have a cache

Yes but it get cleared on each push and not build until user visit page. I am going to implement cache build in background after each push. In this case even if it takes 20 seconds - its ok. And user will get cached information

Progress in !986 (merged)

@dzaporozhets I understand.
Another option would be to save the total number of commits in the project model on push.
Not sure if that still makes sense when the cache is implemented.
Won't the cache get pretty big when it caches a lot of big repos?

Won't the cache get pretty big when it caches a lot of big repos?

@haynes Thats why cache exists. To be big in compare to database with persistent data

Another option would be to save the total number of commits in the project model on push.

Just did it in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/986/diffs

mentioned in merge request !986 (merged)

Status changed to closed by commit 915f9855

mentioned in merge request !989 (merged)