disallow irrelevant pages by defaults in robots

Can't show this merge request because the source branch github/fork/bbodenmiller/robots.txt is missing from project import-export-testing-group/gitlabhq_1662339666. Close this merge request or update the source branch.