Ensure external users are not able to clone disabled repositories.

EE MR for gitlab/gitlabhq!2017

See merge request !506

Signed-off-by: Rémy Coutable <remy@rymai.me>

Ensure external users are not able to clone disabled repositories.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/23788



See merge request !2017

Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Code improvements, bug fixes, finish documentation and specs

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

1. Builds on dd26f4e7

1. A protected branch access level `belongs_to` a group in addition to a
   user. When access for a user is checked, see if the user is present
   in the access level's group, if any.

2. If a group is added to a project with `:reporter` access or lower,
   adding a group to a proteted branch does _not_ grant access, since
   the `push_code` ability is still checked. This is tested with a few
   hundred new tests in `git_access_spec`.

It uses a user activity table instead of a column in users.
Tested with mySQL and postgreSQL

This commit adds a number of _html columns and, with the exception of Note,
starts updating them whenever the content of their partner fields changes.

Note has a collision with the note_html attr_accessor; that will be fixed later

A background worker for clearing these cache columns is also introduced - use
`rake cache:clear` to set it off. You can clear the database or Redis caches
separately by running `rake cache:clear:db` or `rake cache:clear:redis`,
respectively.

Changes include:

- Ensure Member.add_user is not called directly when not necessary
- New GroupMember.add_users_to_group to have the same abstraction level as for Project
- Refactor Member.add_user to take a source instead of an array of members
- Fix Rubocop offenses
- Always use Project#add_user instead of project.team.add_user
- Factorize users addition as members in Member.add_users_to_source
- Make access_level a keyword argument in GroupMember.add_users_to_group and ProjectMember.add_users_to_projects
- Destroy any requester before adding them as a member
- Improve the way we handle access requesters in Member.add_user
  Instead of removing the requester and creating a new member,
  we now simply accepts their access request. This way, they will
  receive a "access request granted" email.
- Fix error that was previously silently ignored
- Stop raising when access level is invalid in Member, let Rails validation do their work

Signed-off-by: Rémy Coutable <remy@rymai.me>

Cycle Analytics: first iteration

## What does this MR do?

- Implement the first iteration of the "Cycle Analytics" feature.

## What are the relevant issue numbers?

- Closes #21170 

## Screenshots

![cycle_analytics_screencast.gif](/uploads/d23c3c912caa6935fd47b53ca3a56b97/cycle_analytics.gif)

## Backend Tasks

- [x]  Implementation
    - [x]  Phases
        - [x]  Issue (Tracker)
        - [x]  Plan (Board)
        - [x]  Code (IDE)
        - [x]  Test (CI)
        - [x]  Review (MR)
        - [x]  Staging (CD)
        - [x]  Production (Total)
    - [x]  Make heuristics more modular
    - [x]  Scope to project
    - [x]  Date range (30 days, 90 days)
    - [x]  Access restriction
- [x]  Test
    - [x]  Find a better way to test these phases
    - [x]  Phases
        - [x]  Issue (Tracker)
        - [x]  Plan (Board)
        - [x]  Code (IDE)
        - [x]  Test (CI)
        - [x]  Review (MR)
        - [x]  Staging (CD)
        - [x]  Production (Total)
    - [x]  Test for "end case happens before start case"
    - [x]  Consolidate helper
- [x]  Miniboss review
- [x]  Performance testing with mock data
- [x]  Improve performance
    - [x]  Pre-calculate "merge requests closing issues
    - [x]  Pre-calculate everything else
- [x]  Test performance against 10k issues
- [x]  Test all pre-calculation code
    - [x]  Ci::Pipeline -> build start/finish
    - [x]  Ci::Pipeline#merge_requests
    - [x]  Issue -> record default metrics after save
    - [x]  MergeRequest -> record default metrics after save
    - [x]  Deployment -> Update "first_deployed_to_production_at" for MR metrics
    - [x]  Git Push -> Update "first commit mention" for issue metrics
    - [x]  Merge request create/update/refresh -> Update "merge requests closing issues"
- [x]  Remove `MergeRequestsClosingIssues` when necessary
- [x]  Changes to unblock Fatih
    - [x]  Add summary data
    - [x]  `stats` should be array
    - [x]  Let `stats` be `null` if all `stats` are null
- [x]  Indexes for "merge requests closing issues"
- [x]  Test summary data
- [x]  Scope everything to project
    - [x]  Find out why tests were passing
- [x]  Filter should include issues/MRs which have made it to production within the range
- [x]  Don't create duplicate `MergeRequestsClosingIssues`
- [x]  Fix tests
- [x]  MySQL median
- [x]  Assign to Douwe for review
- [x]  Fix conflicts
- [x]  Implement suggestions from Yorick's review
    - [x]  Test on PG
    - [x]  Test on MySQL
- [x]  Refactor
    - [x]  Cleanup
        - [x]  What happens if we have no data at all?
        - [x]  Extract common queries to methods / scopes
    - [x]  Remove unused queries
    - [x]  Downtime for foreign key migrations
    - [x]  Find a way around "if issue.metrics.present?" all over the place
    - [x]  Find a way around "if merge_request.metrics.present?" all over the place
    - [x]  Test migrations on a fresh database
        - [x]  MySQL
        - [x]  Pg
- [x]  Access issues
    - While the project is public and the visibility is set to "Everyone with access", you cannot visit the cycle analytics page when signed out.
- [x]  CHANGELOG
- [x]  Implement suggestions from Douwe's review
    - [x]  First set of comments
    - [x]  Second set of comments
    - [x]  Third set of comments
    - [x]  Fourth set of comments
- [x]  Make sure build is green
- [ ]  Make issue for "polish"
- [ ]  EE MR


See merge request !5986

Fix broken spec due to last_activity_at updates being throttled

In https://gitlab.com/gitlab-org/gitlab-ce/builds/4218398, the build failed because the last_activity_at column was only being updated once per hour. We
can fix this spec by stubbing out the throttling and adjusting the spec to test the right event timestamp.

See merge request !6424

In https://gitlab.com/gitlab-org/gitlab-ce/builds/4218398, the build failed
because the last_activity_at column was only being updated once per hour. We
can fix this spec by stubbing out the throttling and adjusting the spec to
test the right event timestamp.

MySQL could not have support for millisecond precision, depends on the MySQL version 
so we just create issues in different seconds in a deterministic way

Use `update_all` to only require one query per discussion to
update the notes resolved status. Some changes had to be made to
the discussion spec to accout for the fact that notes are not
individually updated now

Restrict pushes / merges to a protected branch to specific people

- Closes #674
- Related to #179

![2016-08-04_16-19-12](/uploads/c81032fe262949f3084974c4e622e9eb/2016-08-04_16-19-12.png)
![](https://gitlab.com/gitlab-org/gitlab-ee/uploads/342f06078b07b39c2c817b12c81b30fe/Screen_Shot_2016-07-27_at_6.50.56_PM.png)
![](https://gitlab.com/gitlab-org/gitlab-ee/uploads/52fe1ae3027d491270ea4845ae7ce54b/Screen_Shot_2016-07-27_at_6.51.02_PM.png)

- [ ]  ee#674 !581 Restrict merges to specific people
    - [x]  Implementation
        - [x]  Model changes
            - [x]  Protected branch `has_many` access levels
        - [x]  Frontend
            - [x]  How to add new users / roles?
            - [x]  Dropdown should include users
            - [x]  Dropdown shouldn't include users / roles that have already been selected
            - [x]  Allow removing users / roles
        - [x]  Removing a user from the project should remove their access (?)
    - [x]  Test/refactor
        - [x]  Extract common from {Merge,Push}AccessLevel models
        - [x]  Clean up code that's removing users/roles that are already selected
        - [x]  AccessLevelController?
        - [x]  Fix build
        - [x]  Add more tests
    - [x]  Non `:push_code` users can't push even if added to an access level
    - [x]  Remove access levels when a user is removed from a project
    - [x]  Rebase off EE master instead of the "no one can push" feature branch
        - [x]  Fix create for roles
        - [x]  Fix update for roles
        - [x]  Fix delete
        - [x]  Fix create for users
        - [x]  Fix update for users
        - [x]  Fix defaults
    - [x]  Verify
        - [x]  API
            - [x]  For a developer user
                - [x]  When they are granted access specifically to
                    - [x]  Merge
                    - [x]  Push
            - [x]  For a reporter user
                - [x]  When they are granted access specifically to
                    - [x]  Merge
                    - [x]  Push
        - [x]  Default branch protection
    - [x]  CHANGELOG
    - [x]  Screenshots
    - [x]  Only show `:push_code` users in the dropdown
    - [x]  Send email when user is added/removed from a protected branch
    - [x]  Assign to {mini,end}boss
    - [x]  Fix build
    - [x]  Implement @dbalexandre's review comments
    - [x]  EE should _add_ to CE
    - [x]  Wait for [build](https://gitlab.com/gitlab-org/gitlab-ee/commit/61edf43d31452a0b972dc880e277c825d59f764f/builds) to pass
    - [x]  Test by hand
    - [x]  Assign to endboss
    - [x]  Create CE MR to backport changes
    - [x]  Implement @Douwe's comments
        - [x]  access_level#humanize
        - [x]  Blank line in _protected_branch.html.haml
        - [x]  move stuff into shared partials? (_protected_branch_ee.html.haml)
        - [x]  Can we move stuff into shared partials? (_create_protected_branch_ee.html.haml)
        - [x]  The CE version has a bunch of extra attributes here, do we need those?
        - [x]  We can't indent this section just in EE (protected_branches/show.html.haml)
        - [x]  In EE, try not to add stuff to existing views. (protected_branches/show.html.haml)
        - [x]  If we're gonna change multiline blocks to single-line blocks, we need to so in CE too. (factory)
        - [x]  Split up `ProtectedBranches#show`
    - [x]  Wait for [build](https://gitlab.com/gitlab-org/gitlab-ee/commit/3943254d5f92c9be520f685044d23bf75282d42a/builds)
    - [x]  Implement Douwe's suggestions
    - [x]  Add uniqueness validation
    - [x]  Wait for @alfredo's UI enhancements
    - [x]  Remove `show` page access controls
    - [x]  Add more feature specs
    - [ ]  Wait for review for @alfredo's work
    - [ ]  Wait for merge

See merge request !581