Loading the labels page is slow

Loading GitLab CE's label list takes on average 28 seconds time to first byte. It takes a similar amount of time to load the second page. This makes the label list pretty much unusable. Even on a much smaller project, it still takes 15 seconds to load the page.