Improve diff lines count efficiency

This fixes a high frequency N+1 issue:

`RoutableActions#find_routable!` is used across many controllers to
retrieve e.g. the Project or Namespace by path. The `#find_routable!`
method calls `#ensure_canonical_path` which in turn retrieves
`#full_path` from the given Routable.

This in turn triggers a lookup on `routes`, leading to a high frequency
of these queries:

```sql
SELECT  "routes".* FROM "routes" WHERE "routes"."source_id" = $1 AND
"routes"."source_type" = $2 LIMIT $3
```

This is unnecessary as we already join `routes` in
`Routable#find_by_full_path` anyways.

Etienne Baqué authored 5 years ago and Andreas Brandl committed 5 years ago

Modified schema via migrations.
Added one-to-one relationship between the two models.
Added changelog file

This commit adds pipeline.type key to PipelineEntity.
This key will be used in MR widget in the next iteration.

When updating group and project members, new system hooks
`user_update_for_group` and `user_update_for_team` will be executed.

Fixes https://gitlab.com/gitlab-org/gitlab-ee/issues/12252

Massimeddu Cireddu authored 5 years ago and Nick Thomas committed 5 years ago

Wrong format on MS Teams integration push events with multi line commit messages

Manoj M J authored 5 years ago and Mayra Cabrera committed 5 years ago

This change limits the number of emails
for new access requests notifications to
10 most recently active owners/maintainers

This does not support upgrading from earlier versions

In order to lookup a Project or Namespace by path, we prefer an exact
match (case-sensitive) but in absence of that, we'd also take a
case-insensitive match.

The case-insensitive matching with preference for the exact match is a
bit more involved in SQL as the exact lookup. Yet, the majority of cases
will be an exact match. The thinking here is that we can optimize the
lookup by performing an exact match first and only if there is no
result, we perform the case-insensitive lookup.

Data for GitLab.com:
* We have about 15M records in routes table
* About 2,500 routes exist where there's more than one record
  with the same `lower(path)`

It is possible for a user to craft requests that would always trigger
the 2-step search (e.g. we have a route for `/foo/bar`, the request is
always for `/FOO/bar`). In this case, the change at hand is not
beneficial as it would run an additional query.

However, based on the data, it is highly likely that the vast majority
of requests can be satisfied with an exact match only.

The context for this change is
https://gitlab.com/gitlab-org/gitlab-ce/issues/64590#note_208156463.

rails_helper.rb's only logic was to require
spec_helper.rb.

This reverts commit 0eff75fa.

Persists if a board list is collapsed for each user.

Adam Hegyi authored 5 years ago and Andreas Brandl committed 5 years ago

This change sets NOT NULL constraint to users.private profile.

closes https://gitlab.com/gitlab-org/gitlab-ce/issues/57538

Heinrich Lee Yu authored 5 years ago and Patrick Derichs committed 5 years ago

Also change test URL sequest to .test TLD

When a issue is moved from one project to another, all associated
Markdown text is rewritten in the context of the new project. If the
note contained a link to a commit URL, `CommitRewriter#rewrite` would
fail because `Commit#link_reference_pattern` would match `nil` `commit`
values in the HTML generated from the Markdown. These `nil` values were
passed along to `Project#commits_by` because `Commit#reference_valid?`
was always returning `true`.

To prevent this issue from happening, we tighten up the check for
`Commit#reference_valid?` to look for valid SHA values.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/66666

totalNotes is only used to prerender a number of
skeleton containers until real notes are loaded

issuable.discussions makes multiple requests, so
too expensive for this

This commit uses mere notes for this and sends
actual totalNotes number if it's less than 10;
otherwise it sends 10 - it allows us to avoid
bunch of skeleton prerenderings, which are not
necessary since they doesn't fit into the whole
screen and disappear quite fast

Only displays the todo body if the todo has a note.
This is to avoid redundant Issue or Merge Request titles
displayed both in the Todo title and body.

Also change test URL sequest to .test TLD

Removed unused method for name setter method

The flag defaulted to true, so there's no change unless users turned it
off. Given there's a lack of issues regarding object pools, this should
be OK.

User images and videos will get proxied through
the Camo server in order to keep malicious
sites from collecting the IP address of users.

Adam Hegyi authored 5 years ago and Mayra Cabrera committed 5 years ago

- Introducting StageEvents to define the available events
- Define the event pairing rules, since some events are not compatible
- Express default Cycle Analytics stages with the event structure

Alessio Caiazza authored 5 years ago and Mayra Cabrera committed 5 years ago

Root namespaces have storage statistics.
This commit allows namespace owners to get those stats via GraphQL
queries like the following one

{
  namespace(fullPath: "a_namespace_path") {
    rootStorageStatistics {
      storageSize
      repositorySize
      lfsObjectsSize
      buildArtifactsSize
      packagesSize
      wikiSize
    }
  }
}

Prefer the deployable user over the deployment user.

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/66037.

This makes BuildQueueService to force refresh runners
that are considered to have recent queue.

Such runners are the ones that connected within online
interval + time to expire runner cache.

Limiting the size of issuable description and comments to 1_000_000,
which is close to ~1MB of ASCII characters, which represents 99.9% of
all descriptions and comments we have in DB at the moment. This should
help prevent DoS attacks when comments contain refference strings.

Also this change updates regexp matching the namespaces paths by
limiting the namespaces paths to Namespace::NUMBER_OF_ANCESTORS_ALLOWED,
as we allow 20 levels deep groups.

see https://gitlab.com/gitlab-org/gitlab-ce/issues/61974#note_191274234

This commits avoiding conflicts between ArchiveTraceWorker
and ArchiveTracesCronWorker by changing the target of the
latter worker.

Devise checks before updating any of the authentication_keys if it
needs to clear the reset_password_tokens.

This should fix:
https://gitlab.com/gitlab-org/gitlab-ce/issues/42733 (Weak
authentication and session management)

Adding, destroying and toggling emoji previously lacked services and
instead were performed through methods called on Awardable models.

This led to inconsistencies where relevant todos would be marked as done
only when emoji were awarded through our controllers, but not through
the API. Todos could also be marked as done when an emoji was being
removed.

Behaviour changes

- Awarding emoji through the API will now mark a relevant Todo as done
- Toggling an emoji off (destroying it) through our controllers will no
  longer mark a relevant Todo as done

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/63372

Kubeclient uses rest-client. We hack into to access the net/http object
so that we can patch to connect to the resolved IP + set
hostname_override.

Add specs for discord. The discord integration also uses rest-client, so
since we patched rest-client, spec that the DNS rebinding protection
works

Stan Hu authored 5 years ago and Douwe Maan committed 5 years ago

Previously `ProjectCacheWorker` would be scheduled once per ref, which
would generate unnecessary I/O and load on Sidekiq, especially if many
tags or branches were pushed at once. `ProjectCacheWorker` would expire
three items:

1. Repository size: This only needs to be updated once per push.
2. Commit count: This only needs to be updated if the default branch
   is updated.
3. Project method caches: This only needs to be updated if the default
   branch changes, but only if certain files change (e.g. README,
   CHANGELOG, etc.).

Because the third item requires looking at the actual changes in the
commit deltas, we schedule one `ProjectCacheWorker` to handle the first
two cases, and schedule a separate `ProjectCacheWorker` for the third
case if it is needed. As a result, this brings down the number of
`ProjectCacheWorker` jobs from N to 2.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046

This allows us to check inclusion for the *_exists? methods without
downloading the full list of branch names, which is over 100KiB in size
for gitlab-ce at the moment.

Adam Hegyi authored 5 years ago and Mayra Cabrera committed 5 years ago

This change lays the foundation for customizable cycle analytics stages.
The main reason for the change is to extract the event definitions to
separate objects (start_event, end_event) so that it could be easily
customized later on.

Nick Thomas authored 5 years ago and Mayra Cabrera committed 5 years ago

Prior to 12.1, rebase status was looked up directly from Gitaly. In
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14417 , a DB
column was added to track the status instead. However, we couldn't stop
looking at the gitaly status immediately, since some rebases may been
running across the upgrade.

Now that we're in 12.3, it is safe to remove the direct-to-gitaly
lookup. This also happens to fix a 500 error that is seen when viewing
an MR for a fork where the source project has been removed.

We still look at the Gitaly status in the service, just in case Gitaly
and Sidekiq get out of sync - I assume this is possible, and it's a
relatively cheap check.

Since we atomically check and set `merge_requests.rebase_jid`, we
should never enqueue two `RebaseWorker` jobs in parallel.

- Adds UI to configure in group and project settings
- Removes notification configuration for users when
disabled at group or project level

Bob Van Landuyt :neckbeard: authored 5 years ago and Douwe Maan committed 5 years ago

**Prevention of running 2 simultaneous updates**

Instead of using `RemoteMirror#update_status` and raise an error if
it's already running to prevent the same mirror being updated at the
same time we now use `Gitlab::ExclusiveLease` for that.

When we fail to obtain a lease in 3 tries, 30 seconds apart, we bail
and reschedule. We'll reschedule faster for the protected branches.

If the mirror already ran since it was scheduled, the job will be
skipped.

**Error handling: Remote side**

When an update fails because of a `Gitlab::Git::CommandError`, we
won't track this error in sentry, this could be on the remote side:
for example when branches have diverged.

In this case, we'll try 3 times scheduled 1 or 5 minutes apart.

In between, the mirror is marked as "to_retry", the error would be
visible to the user when they visit the settings page.

After 3 tries we'll mark the mirror as failed and notify the user.

We won't track this error in sentry, as it's not likely we can help
it.

The next event that would trigger a new refresh.

**Error handling: our side**

If an unexpected error occurs, we mark the mirror as failed, but we'd
still retry the job based on the regular sidekiq retries with
backoff. Same as we used to

The error would be reported in sentry, since its likely we need to do
something about it.

This commit fixes the project avatar images that are rendered in the
footer of Slack pipeline notifications.  Previously, the image URLs
provided to Slack were relative URLs; now they are absolute.

Previously each tag in a push would invoke the Gitaly `FindAllTags` RPC
since the tag cache would be invalidated with every tag.

We can eliminate those extraneous calls by expiring the tag cache once
in `PostReceive` and taking advantage of the cached tags.

Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/65795