Fix performance regression when displaying builds and pipelines
What does this MR do?
Fixes performance regression introduced after merging https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10462, after noticing performance degredation on dev.gitlab.org: https://performance.gitlab.net/dashboard/db/rails-controllers?orgId=1&from=now-7d&to=now&var-action=Projects::PipelinesController%23index.json&var-database=dev.gitlab.org.
The chosen solution: https://gitlab.com/gitlab-org/gitlab-ce/issues/27377#note_27110678