The refs hash is used to determine what branches and tags have a commit
as head in the network graph. The previous implementation depended on
Rugged#references. The problem with this implementation was that it
depended on rugged, but also that it iterated over all references and
thus loading more data than needed if for example the project uses CI/CD
environments, Pipelines, or Merge Requests.

Given only refs are checked the network cares about the GraphHelper#refs
method has no need to reject those, simplifying the method.

Closes gitlab-org/gitaly#880

Closes https://github.com/gitlabhq/gitlabhq/issues/9657

Because model shouldn't know about view logic.