Skip to content
Snippets Groups Projects
Commit a89cb5cb authored by GitLab Bot's avatar GitLab Bot
Browse files

Add latest changes from gitlab-org/gitlab@master

parent 0d6fa033
No related branches found
No related tags found
No related merge requests found
Showing
with 374 additions and 78 deletions
Loading
Loading
@@ -371,7 +371,7 @@ constraints(::Constraints::ProjectUrlConstrainer.new) do
end
end
 
resources :container_registry, only: [:index, :destroy],
resources :container_registry, only: [:index, :destroy, :show],
controller: 'registry/repositories'
 
namespace :registry do
Loading
Loading
# frozen_string_literal: true
class AddIndexWebHooksOnGroupId < ActiveRecord::Migration[5.2]
include Gitlab::Database::MigrationHelpers
DOWNTIME = false
disable_ddl_transaction!
def up
add_concurrent_index :web_hooks, :group_id, where: "type = 'GroupHook'"
end
def down
remove_concurrent_index :web_hooks, :group_id, where: "type = 'GroupHook'"
end
end
Loading
Loading
@@ -4477,6 +4477,7 @@ ActiveRecord::Schema.define(version: 2020_02_05_143231) do
t.string "encrypted_token_iv"
t.string "encrypted_url"
t.string "encrypted_url_iv"
t.index ["group_id"], name: "index_web_hooks_on_group_id", where: "((type)::text = 'GroupHook'::text)"
t.index ["project_id"], name: "index_web_hooks_on_project_id"
t.index ["type"], name: "index_web_hooks_on_type"
end
Loading
Loading
Loading
Loading
@@ -385,7 +385,7 @@ Parameters:
- `skip_confirmation` (optional) - Skip confirmation - true or false (default)
- `external` (optional) - Flags the user as external - true or false (default)
- `avatar` (optional) - Image file for user's avatar
- `private_profile` (optional) - User's profile is private - true or false (default)
- `private_profile` (optional) - User's profile is private - true, false (default), or null (will be converted to false)
- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **(STARTER)**
- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **(STARTER)**
 
Loading
Loading
@@ -423,7 +423,7 @@ Parameters:
- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user
- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user
- `avatar` (optional) - Image file for user's avatar
- `private_profile` (optional) - User's profile is private - true or false (default)
- `private_profile` (optional) - User's profile is private - true, false (default), or null (will be converted to false)
- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **(STARTER)**
- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **(STARTER)**
- `note` (optional) - Admin notes for this user **(STARTER)**
Loading
Loading
Loading
Loading
@@ -83,6 +83,7 @@ Complementary reads:
- [Cycle Analytics development guide](cycle_analytics.md)
- [Issue types vs first-class types](issue_types.md)
- [Application limits](application_limits.md)
- [Redis guidelines](redis.md)
 
## Performance guides
 
Loading
Loading
Loading
Loading
@@ -19,9 +19,6 @@ The current stages are:
<https://gitlab.com/gitlab-org/gitlab-foss>.
- `prepare`: This stage includes jobs that prepare artifacts that are needed by
jobs in subsequent stages.
- `quick-test`: This stage includes test jobs that should run first and fail the
pipeline early (currently used to run Geo tests when the branch name starts
with `geo-`, `geo/`, or ends with `-geo`).
- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs.
- `post-test`: This stage includes jobs that build reports or gather data from
the `test` stage's jobs (e.g. coverage, Knapsack metadata etc.).
Loading
Loading
@@ -138,54 +135,109 @@ duplicating the `if` conditions and `changes` patterns lists since they cannot b
**If you update an `if` condition or `changes`
patterns list, make sure to mass-update those across all the CI config files (i.e. `.gitlab/ci/*.yml`).**
 
### Canonical commits only
### Canonical/security namespace merge requests only
 
This condition limits jobs creation to commits under the `gitlab-org/` top-level group
on GitLab.com only. This is similar to the `.only:variables-canonical-dot-com` CI definition:
This condition limits jobs creation to merge requests under the `gitlab-org/` top-level group
on GitLab.com only (i.e. this won't run for `master`, stable or auto-deploy branches).
This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [merge_requests]`
CI definitions.
 
```yaml
.if-canonical-gitlab: &if-canonical-gitlab
if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/'
```
The definition for `if-canonical-dot-com-gitlab-org-groups-merge-request` can be
seen in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml>.
 
### Canonical merge requests only
### Canonical/security namespace tags only
 
Same as the "Canonical commits only" condition above but further limits jobs creation
to merge requests only (i.e. this won't run for `master`, stable or auto-deploy branches).
This is similar to the `.only:variables-canonical-dot-com` + `.except:refs-master-tags-stable-deploy`
CI definitions:
This condition limits jobs creation to tags under the `gitlab-org/` top-level group
on GitLab.com only.
This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [tags]` CI definition:
 
```yaml
.if-canonical-gitlab-merge-request: &if-canonical-gitlab-merge-request
if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/ && $CI_MERGE_REQUEST_IID'
```
The definition for `if-canonical-dot-com-gitlab-org-groups-tag` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cng.gitlab-ci.yml>.
### Canonical namespace `master` only
This condition limits jobs creation to `master` pipelines for the `gitlab-org` top-level group
on GitLab.com only.
This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [master]` CI definition:
The definition for `if-canonical-dot-com-gitlab-org-group-master-refs` can be
seen in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/pages.gitlab-ci.yml>.
### Canonical namespace schedules only
This condition limits jobs creation to scheduled pipelines for the `gitlab-org` top-level group
on GitLab.com only.
This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [schedules]` CI definition:
The definition for `if-canonical-dot-com-gitlab-org-group-schedule` can be seen
in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>.
### Not canonical/security namespace
This condition matches if the project isn't in the canonical/security namespace.
Useful to **not** create a job if the project is a fork, or in other words, when
a job should only run in the canonical projects.
The definition for `if-not-canonical-namespace` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
### Not EE
This condition matches if the project isn't EE. Useful to **not** create a job if
the project is GitLab, or in other words, when a job should only run in the GitLab
FOSS project.
The definition for `if-not-ee` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
### Default refs only
This condition is the equivalent of `.default-only`.
The definition for `if-default-refs` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
### `master` refs only
This condition is the equivalent of `only:refs: [master]`.
The definition for `if-master-refs` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
 
### Code changes patterns
 
Similar patterns as for `.only:changes-code`:
 
```yaml
.code-patterns: &code-patterns
- ...
```
The definition for `code-patterns` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>.
 
### QA changes patterns
 
Similar patterns as for `.only:changes-qa`:
 
```yaml
.qa-patterns: &qa-patterns
- ...
```
The definition for `qa-patterns` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>.
### Docs changes patterns
Similar patterns as for `.only:changes-docs`:
The definition for `docs-patterns` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml>.
 
### Code and QA changes patterns
 
Similar patterns as for `.only:changes-code-qa`:
 
```yaml
.code-qa-patterns: &code-qa-patterns
- ...
```
The definition for `code-qa-patterns` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/review.gitlab-ci.yml>.
### Code, backstage and QA changes patterns
Similar patterns as for `.only:changes-code-backstage-qa`:
The definition for `code-backstage-qa-patterns` can be seen in
<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
 
## Directed acyclic graph
 
Loading
Loading
@@ -195,21 +247,21 @@ execute jobs out of order for the following jobs:
```mermaid
graph RL;
A[setup-test-env];
B["gitlab:assets:compile pull-push-cache<br/>(master only)"];
C[gitlab:assets:compile pull-cache];
B["gitlab:assets:compile pull-push-cache<br/>(canonical master only)"];
C["gitlab:assets:compile pull-cache<br/>(canonical default refs only)"];
D["cache gems<br/>(master and tags only)"];
E[review-build-cng];
F[build-qa-image];
G[review-deploy];
G2["schedule:review-deploy<br/>(master only)"];
H[karma];
I[jest];
I["karma, jest, webpack-dev-server, static-analysis"];
I2["karma-foss, jest-foss<br/>(EE default refs only)"];
J["compile-assets pull-push-cache<br/>(master only)"];
J2["compile-assets pull-push-cache foss<br/>(EE master only)"];
K[compile-assets pull-cache];
L[webpack-dev-server];
K2["compile-assets pull-cache foss<br/>(EE default refs only)"];
M[coverage];
N[pages];
O[static-analysis];
N["pages (master only)"];
Q[package-and-qa];
S["RSpec<br/>(e.g. rspec unit pg9)"]
T[retrieve-tests-metadata];
Loading
Loading
@@ -220,58 +272,55 @@ subgraph "`prepare` stage"
C
F
K
K2
J
J2
T
end
 
subgraph "`test` stage"
D --> |needs| A;
H -.-> |needs and depends on| A;
H -.-> |needs and depends on| K;
D -.-> |needs| A;
I -.-> |needs and depends on| A;
I -.-> |needs and depends on| K;
I2 -.-> |needs and depends on| A;
I2 -.-> |needs and depends on| K;
L -.-> |needs and depends on| A;
L -.-> |needs and depends on| K;
O -.-> |needs and depends on| A;
O -.-> |needs and depends on| K;
S -.-> |needs and depends on| A;
S -.-> |needs and depends on| K;
S -.-> |needs and depends on| T;
downtime_check --> |needs and depends on| A;
db:* --> |needs| A;
gitlab:setup --> |needs| A;
downtime_check --> |needs and depends on| A;
graphql-docs-verify --> |needs| A;
L["db:*, gitlab:setup, graphql-docs-verify, downtime_check"] -.-> |needs| A;
end
subgraph "`post-test` stage"
M --> |happens after| S
end
 
subgraph "`review-prepare` stage"
E --> |needs| C;
X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| C;
E -.-> |needs| C;
E2["schedule:review-build-cng<br/>(master schedule only)"] -.-> |needs| C;
end
 
subgraph "`review` stage"
G
G2
G --> |happens after| E
G2 --> |happens after| E2
end
 
subgraph "`qa` stage"
Q --> |needs| C;
Q --> |needs| F;
review-qa-smoke -.-> |needs and depends on| G;
review-qa-all -.-> |needs and depends on| G;
review-performance -.-> |needs and depends on| G;
X2["schedule:review-performance<br/>(master only)"] -.-> |needs and depends on| G2;
dast -.-> |needs and depends on| G;
Q -.-> |needs| C;
Q -.-> |needs| F;
QA1["review-qa-smoke, review-qa-all, review-performance, dast"] -.-> |needs and depends on| G;
QA2["schedule:review-performance<br/>(master only)"] -.-> |needs and depends on| G2;
end
 
subgraph "`post-test` stage"
M
end
subgraph "`post-qa` stage"
PQA1["parallel-spec-reports"] -.-> |depends on `review-qa-all`| QA1;
end
 
subgraph "`pages` stage"
N -.-> |depends on| C;
N -.-> |depends on| H;
N -.-> |depends on karma| I;
N -.-> |depends on| M;
N --> |happens after| PQA1
end
```
 
Loading
Loading
# Redis guidelines
GitLab uses [Redis](https://redis.io) for three distinct purposes:
- Caching via `Rails.cache`.
- As a job processing queue with [Sidekiq](sidekiq_style_guide.md).
- To manage the shared application state.
Every application process is configured to use the same Redis servers, so they
can be used for inter-process communication in cases where [PostgreSQL](sql.md)
is less appropriate, for example, transient state or data that is written much
more often than it is read.
If [Geo](geo.md) is enabled, each Geo node gets its own, independent Redis
database.
## Key naming
Redis is a flat namespace with no hierarchy, which means we must pay attention
to key names to avoid collisions. Typically we use colon-separated elements to
provide a semblence of structure at application level. An example might be
`projects:1:somekey`.
Although we split our Redis usage into three separate purposes, and those may
map to separate Redis servers in a [Highly Available](../administration/high_availability/redis.md)
configuration, the default Omnibus and GDK setups share a single Redis server.
This means that keys should **always** be globally unique across the three
purposes.
It is usually better to use immutable identifiers - project ID rather than
full path, for instance - in Redis key names. If full path is used, the key will
stop being consulted if the project is renamed. If the contents of the key are
invalidated by a name change, it is better to include a hook that will expire
the entry, instead of relying on the key changing.
We don't use [Redis Cluster](https://redis.io/topics/cluster-tutorial) at the
moment, but may wish to in the future: [#118820](https://gitlab.com/gitlab-org/gitlab/issues/118820).
This imposes an additional constraint on naming: where GitLab is performing
operations that require several keys to be held on the same Redis server - for
instance, diffing two sets held in Redis - the keys should ensure that by
enclosing the changeable parts in curly braces, such as, `project:{1}:set_a` and
`project:{1}:set_b`.
Loading
Loading
@@ -19,10 +19,15 @@ module API
end
params do
use :pagination
optional :search, type: String, desc: 'Search for a protected branch by name'
end
# rubocop: disable CodeReuse/ActiveRecord
get ':id/protected_branches' do
protected_branches = user_project.protected_branches.preload(:push_access_levels, :merge_access_levels)
protected_branches =
ProtectedBranchesFinder
.new(user_project, params)
.execute
.preload(:push_access_levels, :merge_access_levels)
 
present paginate(protected_branches), with: Entities::ProtectedBranch, project: user_project
end
Loading
Loading
Loading
Loading
@@ -52,7 +52,7 @@ module API
optional :external, type: Boolean, desc: 'Flag indicating the user is an external user'
# TODO: remove rubocop disable - https://gitlab.com/gitlab-org/gitlab/issues/14960
optional :avatar, type: File, desc: 'Avatar image for user' # rubocop:disable Scalability/FileUploads
optional :private_profile, type: Boolean, default: false, desc: 'Flag indicating the user has a private profile'
optional :private_profile, type: Boolean, desc: 'Flag indicating the user has a private profile'
all_or_none_of :extern_uid, :provider
 
use :optional_params_ee
Loading
Loading
# frozen_string_literal: true
module Gitlab
module Alerting
class Alert
include ActiveModel::Model
include Gitlab::Utils::StrongMemoize
include Presentable
attr_accessor :project, :payload
def gitlab_alert
strong_memoize(:gitlab_alert) do
parse_gitlab_alert_from_payload
end
end
def metric_id
strong_memoize(:metric_id) do
payload&.dig('labels', 'gitlab_alert_id')
end
end
def title
strong_memoize(:title) do
gitlab_alert&.title || parse_title_from_payload
end
end
def description
strong_memoize(:description) do
parse_description_from_payload
end
end
def environment
strong_memoize(:environment) do
gitlab_alert&.environment || parse_environment_from_payload
end
end
def annotations
strong_memoize(:annotations) do
parse_annotations_from_payload || []
end
end
def starts_at
strong_memoize(:starts_at) do
parse_datetime_from_payload('startsAt')
end
end
def starts_at_raw
strong_memoize(:starts_at_raw) do
payload&.dig('startsAt')
end
end
def ends_at
strong_memoize(:ends_at) do
parse_datetime_from_payload('endsAt')
end
end
def full_query
strong_memoize(:full_query) do
gitlab_alert&.full_query || parse_expr_from_payload
end
end
def alert_markdown
strong_memoize(:alert_markdown) do
parse_alert_markdown_from_payload
end
end
def status
strong_memoize(:status) do
payload&.dig('status')
end
end
def firing?
status == 'firing'
end
def resolved?
status == 'resolved'
end
def gitlab_managed?
metric_id.present?
end
def valid?
payload.respond_to?(:dig) && project && title && starts_at
end
def present
super(presenter_class: Projects::Prometheus::AlertPresenter)
end
private
def parse_environment_from_payload
environment_name = payload&.dig('labels', 'gitlab_environment_name')
return unless environment_name
EnvironmentsFinder.new(project, nil, { name: environment_name })
.find
&.first
end
def parse_gitlab_alert_from_payload
return unless metric_id
Projects::Prometheus::AlertsFinder
.new(project: project, metric: metric_id)
.execute
.first
end
def parse_title_from_payload
payload&.dig('annotations', 'title') ||
payload&.dig('annotations', 'summary') ||
payload&.dig('labels', 'alertname')
end
def parse_description_from_payload
payload&.dig('annotations', 'description')
end
def parse_annotations_from_payload
payload&.dig('annotations')&.map do |label, value|
Alerting::AlertAnnotation.new(label: label, value: value)
end
end
def parse_datetime_from_payload(field)
value = payload&.dig(field)
return unless value
Time.rfc3339(value)
rescue ArgumentError
end
# Parses `g0.expr` from `generatorURL`.
#
# Example: http://localhost:9090/graph?g0.expr=vector%281%29&g0.tab=1
def parse_expr_from_payload
url = payload&.dig('generatorURL')
return unless url
uri = URI(url)
Rack::Utils.parse_query(uri.query).fetch('g0.expr')
rescue URI::InvalidURIError, KeyError
end
def parse_alert_markdown_from_payload
payload&.dig('annotations', 'gitlab_incident_markdown')
end
end
end
end
# frozen_string_literal: true
module Gitlab
module Alerting
class AlertAnnotation
include ActiveModel::Model
attr_accessor :label, :value
end
end
end
Loading
Loading
@@ -57,6 +57,8 @@ dependency_scanning:
PIP_REQUIREMENTS_FILE \
MAVEN_CLI_OPTS \
BUNDLER_AUDIT_UPDATE_DISABLED \
BUNDLER_AUDIT_ADVISORY_DB_URL \
BUNDLER_AUDIT_ADVISORY_DB_REF_NAME \
) \
--volume "$PWD:/code" \
--volume /var/run/docker.sock:/var/run/docker.sock \
Loading
Loading
Loading
Loading
@@ -3,7 +3,7 @@
module QA
context 'Plan', :reliable do
describe 'check xss occurence in @mentions in issues', :requires_admin do
it 'user mentions a user in comment' do
it 'mentions a user in a comment' do
QA::Runtime::Env.personal_access_token = QA::Runtime::Env.admin_personal_access_token
 
unless QA::Runtime::Env.personal_access_token
Loading
Loading
Loading
Loading
@@ -18,7 +18,7 @@ module QA
push_commit('Initial commit')
end
 
it 'user closes an issue by pushing commit' do
it 'closes an issue by pushing a commit' do
push_commit("Closes ##{issue_id}", false)
 
issue.visit!
Loading
Loading
Loading
Loading
@@ -17,7 +17,7 @@ module QA
end
end
 
it 'user collapses and expands reply for comments in an issue' do
it 'collapses and expands reply for comments in an issue' do
Page::Project::Issue::Show.perform do |show|
one_reply = "1 reply"
 
Loading
Loading
Loading
Loading
@@ -9,7 +9,7 @@ module QA
Resource::Issue.fabricate_via_api!.visit!
end
 
it 'user comments on an issue and edits the comment' do
it 'comments on an issue and edits the comment' do
Page::Project::Issue::Show.perform do |show|
first_version_of_comment = 'First version of the comment'
second_version_of_comment = 'Second version of the comment'
Loading
Loading
Loading
Loading
@@ -7,7 +7,7 @@ module QA
Flow::Login.sign_in
end
 
it 'user creates an issue' do
it 'creates an issue' do
issue = Resource::Issue.fabricate_via_browser_ui!
 
Page::Project::Menu.perform(&:click_issues)
Loading
Loading
@@ -27,7 +27,7 @@ module QA
Resource::Issue.fabricate_via_api!.visit!
end
 
it 'user comments on an issue with an attachment' do
it 'comments on an issue with an attachment' do
Page::Project::Issue::Show.perform do |show|
show.comment('See attached banana for scale', attachment: file_to_attach)
 
Loading
Loading
Loading
Loading
@@ -9,7 +9,7 @@ module QA
Resource::Issue.fabricate_via_api!.visit!
end
 
it 'user filters comments and activities in an issue' do
it 'filters comments and activities in an issue' do
Page::Project::Issue::Show.perform do |show|
my_own_comment = "My own comment"
made_the_issue_confidential = "made the issue confidential"
Loading
Loading
Loading
Loading
@@ -13,7 +13,7 @@ module QA
end.project.visit!
end
 
it 'user sees issue suggestions when creating a new issue' do
it 'shows issue suggestions when creating a new issue' do
Page::Project::Show.perform(&:go_to_new_issue)
Page::Project::Issue::New.perform do |new_page|
new_page.add_title("issue")
Loading
Loading
Loading
Loading
@@ -20,7 +20,7 @@ module QA
end.visit!
end
 
it 'user mentions another user in an issue' do
it 'mentions another user in an issue' do
Page::Project::Issue::Show.perform do |show|
at_username = "@#{@user.username}"
 
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment