Follow-up from "Build and configure gitaly-ruby". Cleanup build directories from individual gems with native extensions

The following discussion from !1798 (merged) should be addressed:

• @marin started a discussion: (+5 comments)

  @jacobvosmaer-gitlab Did we think on how we will keep the gem versions in sync with GitLab Rails project?

@gitlab-build-team Not complicated to do but very rewarding task :)

mentioned in merge request !1798 (merged)

marked this issue as related to #1778

added Scheduled label