Document and clarify user blocking (Rack::Attack)

Overview

We receive support requests for a number of users that see a forbidden 401 message when attempting to access gitlab.com. This is likley due to a rack::attack ban. This has been discussed a number of times https://gitlab.slack.com/archives/support/p1468300023003378

We have initializers/rack_attack.rb

 Rack::Attack.throttle('protected paths', limit: 10, period: 60.seconds) 

Finding active bans

• Login to a production redis box (redis1)

• Start a redis console - see https://gitlab.com/gitlab-com/runbooks/blob/master/howto/staging-environment.md#run-a-redis-console-in-staging-environment

• List the rack::attack keys

  • keys *rack::attack*

1. "cache:gitlab:rack::attack:24471669:allow2ban:count:0.0.0.0"
2. "cache:gitlab:rack::attack:24471669:allow2ban:count:0.0.0.0"
3. "cache:gitlab:rack::attack:24471669:allow2ban:count:0.0.0.0"
4. "cache:gitlab:rack::attack:24471669:allow2ban:count:0.0.0.0"

### Actions

+ [ ] Clarify what user actions can result in a ban
+ Number of incorrect `git clone` auths
+ Visiting a number of protected paths  

+ [ ] - Document ban times and user actions

+ [ ] - Improve ban logging https://gitlab.com/gitlab-org/gitlab-ce/issues/15612



cc// @markglenfletcher @balameb @stanhu @cabargas @abuango @Arihantar 

It also seems admins need better visibility when a Rack Attack ban occurs.

Made the issue visible

Related: https://gitlab.com/gitlab-org/gitlab-ce/issues/22527

Thanks Chris for linking the related issue. Quick summary: I received a ban due to a large git LFS commit when using a ssh remote, but it looks like we added support for this a while ago https://gitlab.com/gitlab-org/gitlab-ce/issues/3589

This is being worked on by @briann in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9363 - We just need documentation

Hi all,

Just wondering if we can add the ability to throttle requests to "child pages" and whether it could be added to this issue.

For example (customer question):

As you can see, we’re calling projects/<id>/builds – so I didn’t think we’d have to block off the builds URL as well. Does the rack_attack not take into child pages? For example, if I blocked:

/api/#{API::API.version}/projects

Would that mean it does NOT block this?

/api/#{API::API.version}/projects/<id>/builds

So if a request hits the builds API it should be throttled at the same rate as the projects API as each build depends on the project.

This is an issue because there can be a limitless number of requests to the build endpoint for each request to the projects endpoint.

i.e.

for project in projects:
  for build in project:  
    do  ...  # This could hit the API very hard and fast

This is in response to: ZD: https://gitlab.zendesk.com/agent/tickets/70632

I was on a call today with the same customer @amulvany is referencing above. I looked in the code and see that we are using a regex for the path so it should include 'child' pages. However, we also only throttle 'POST' requests, not 'GET', which is why this wasn't helping the customer.

Rack::Attack.throttle('protected paths', limit: <%= @rate_limit_requests_per_period %>, period: <%= @rate_limit_period %>.seconds) do |req|
    if req.post? && req.path =~ paths_regex
      req.ip
    end
  end

We may want to consider removing the if req.post? bit as this would allow us to specify any URL to throttle.