Search query is especially slow if a user searches a generic string
which matches many records, in such case search can take tens of
seconds or time out. To speed up the search query, we search only for
first 1000 records, if there is >1000 matching records we just display
"1000+" instead of precise total count supposing that with such amount
the exact count is not so important for the user.

Because for issues even limited search was not fast enough, 2-phase
approach is used for issues: first we use simpler/faster query to get
all public issues, if this exceeds the limit, we just return the limit.
If the amount of matching results is lower than limit, we re-run more
complex search query (which includes also confidential issues).
Re-running the complex query should be fast enough in such case because the
amount of matching issues is lower than limit.

Because exact total_count is now limited, this patch also switches to
to "prev/next" pagination.

Related #40540

A file containing /:\d+:/ in its contents would break the search results if
those contents were part of the results, because we were splitting on colons,
which can't work with untrusted input.

Changing to use the null byte as a separator is much safer.

Douwe Maan authored 7 years ago and Bob Van Landuyt :neckbeard: committed 7 years ago

Respect project features in wiki and blob search

See merge request !2089

See !8028 and #24833

Was proposed in #24833

Fix missing Note access checks in by moving Note#search to updated NoteFinder

Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867

## Which fixes are in this MR?

 - Potentially untested  
 - No test coverage  
 - Test coverage of some sort exists (a test failed when error raised)  
 - Test coverage of return value (a test failed when nil used)  
 - Permissions check tested

### Note lookup without access check

- [x]  app/finders/notes_finder.rb:13 :download_code check
- [x]  app/finders/notes_finder.rb:19 `SnippetsFinder`
- [x]  app/models/note.rb:121 [`Issue#visible_to_user`]
- [x]  lib/gitlab/project_search_results.rb:113
  - This is the only use of `app/models/note.rb:121` above, but importantly has no access checks at all. This means it leaks MR comments and snippets when those features are `team-only` in addition to the issue comments which would be fixed by `app/models/note.rb:121`.
  - It is only called from SearchController where `can?(current_user, :download_code, @Project)` is checked, so commit comments are not leaked.

### Previous discussions
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#b915c5267a63628b0bafd23d37792ae73ceae272_13_13 `: download_code` check on commit
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#b915c5267a63628b0bafd23d37792ae73ceae272_19_19 `SnippetsFinder` should be used
  - `SnippetsFinder` should check if the snippets feature is enabled -> https://gitlab.com/gitlab-org/gitlab-ce/issues/25223

###  Acceptance criteria met?
- [x] Tests added for new code
- [x] TODO comments removed
- [x] Squashed and removed skipped tests
- [x] Changelog entry
- [ ] State Gitlab versions affected and issue severity in description
- [ ] Create technical debt issue for NotesFinder.
  - Either split into `NotesFinder::ForTarget` and `NotesFinder::Search` or consider object per notable type such as `NotesFinder::OnIssue`. For the first option could create `NotesFinder::Base` which is either inherited from or which can be included in the other two.
  - Avoid case statement anti-pattern in this finder with use of `NotesFinder::OnCommit` etc. Consider something on the finder for this? `Model.finder(user, project)`
  - Move `inc_author` to the controller, and implement `related_notes` to replace `non_diff_notes`/`mr_and_commit_notes`

See merge request !2035

This reverts commit 3e991230.

# Conflicts:
#	app/models/project.rb

This ensures some other methods such as the "issues" method still work.

Previously this class would be given a project ID which was then used to
retrieve the corresponding Project object. However, in all cases the
Project object was already known as it was used to grab the ID to pass
to ProjectSearchResults. By just passing a Project instead we remove the
need for an extra query as well as the need for some other complexity
in this class.

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Include the log messages of recent commits in project-level search
results, providing functionality similar to 'git log --grep'.

Update repository model rspec tests to validate the output of
Repository#commits_with_log_matching.

There is no need to check if `repository_ref` is present as:
```
      @repository_ref = if repository_ref.present?
                          repository_ref
                        else
                          nil
                        end
```

is as same as doing:

```
@repository_ref = repository_ref
```

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Gitlab::SearchResults and Gitlab::ProjectSearchResults are libraries we
are going to use to get search results based on query, enitity type and
pagination.

It will allow us to get only issues from project #23 where title or
description includes 'foo'.

Ex:

search_results = Gitlab::ProjectSearchResults.new(project.id, 'foo', 'issues')

search_results.objects => # [<Issues #23>, <Issues #34>]
search_results.issues_count => 2
search_results.total_count => 12 (it includes results from comments and
merge requests too)

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>