[Meta] Security/permissions of CD

Most bang/buck seems to be in providing environment-specific variables and then allowing some kind of restriction on (at least) production environment(s). This would naturally limit manual actions to those environments you have access to.

An alternative might be to leverage existing restrictions on branches, and use branch-variables instead.

At the very least, we should restrict developers from running manual actions on restricted branches.

/cc @ayufan

Apropos from https://www.blackhat.com/docs/eu-15/materials/eu-15-Mittal-Continuous-Intrusion-Why-CI-Tools-Are-An-Attackers-Best-Friend.pdf page 58:

"Restrict privileges which allow configuration of build steps" - the equivalent of edit access to .gitlab-ci.yml
"Do not provide read privileges to Anonymous users" - the equivalent to our disabling build log access by default, even for public projects

Added meta label

@markpundsack great overview. I propose we add these considerations to the documentation. So that other people realize the security tradeoffs and can alert us if we overlooked something. Add as a page in the ci documentation and link from http://docs.gitlab.com/ce/security/README.html

@sytses Good point! There's nothing in there currently about CI security.

@axil Any objections? Should CI security be under the CI docs, or more interspersed in regular documentation?

@markpundsack

My idea to tighten the security is to connect that with user permissions that we are now evaluating. Then it will make no difference who and how modifies the .gitlab-ci.yml.

I actually have a few extensions in mind:

make variables to be accessible by specific access level: ex. only masters,
make runners to be accessible by specific access level: ex. only owner of the projects,
make environments to be deployed by specific access level,

Variables, because can have credentials to access environments. Runners, because someone may only allow one runner to access production network and do deployment to production cluster. Environments, because at some point we will allow to use Variables in context of environments and we should not allow unconditionally anyone to change the last deployment.

This allows us to automatically know who has the permission to execute what, especially manual actions and deployments. For example person with developer permissions would not be able to use special runner or use deployment tokens needed for pushing application to production.

This also makes the system to restrict manual. If we would tie that with runner or environment we could automatically see if developer can run manual action or we need a user with bigger permissions.

If developer changes .gitlab-ci.yml it will not affect the security of the system, because developer will not be able to access the credentials/environments or runners by doing that, since the access level restrictions are on much lower level.

@ayufan I really like focusing on securing the system such that it's safe regardless of what's in .gitlab-ci.yml. I'm glad you added access levels for runners, since that was one loophole that wasn't covered before.

I haven't seen a proposal yet for how to handle a job definition that includes variables, runners, or environments that you don't have access to. Does the Pipeline fail? Does it succeed, but then halt on the forbidden job? Should an admin be able to then finish the pipeline? Does it try to run the job, but fail? For variables, the job may not even know that it's going to fail because it might work without the privileged variables. For runner tags and environments, it's easier to detect because those are hard declarations in the job.

Mentioned in issue #22431 (moved)

I've been running into these issues described in a bunch of these linked issues as I have been deploying GitLab for my company & working to implement CI + CD for our developers who are currently using a snowflake jenkins box they stood up to deploy. I think a lot of the issues hit most of our concerns around Secret variable visibility, ability to deploy to production, etc.

One thought I had after hitting this meta issue, a lot of these issues are around secrets management in general. Has there been any talk/issues in GitLab for integrating some kind of secrets management platform more deeply into GitLab? First thing that comes to mind is Hashicorp Vault.

I've been trying gitlab for a day now and already found out for myself that in the current form continuous deployment is just unusable and totally broken by design.

It's totally fine to just run whatever there is in .gitlab-ci.yml at any moment to perform testing, since tests don't change anything outside the runner and they just pass or not. However, when it comes to deployment, this is no longer valid at all - what's written in the currently evaluated .gitlab-ci.yml can completely ruin the outside world. And I'm not even considering malicious intents here - a developer merely wanting to change how a deployment is performed may create a merge request, and an error in his new draft version of .gitlab-ci.yml may completely ruin everything the moment he merely creates that merge request.

In a short analogy, re-using .gitlab-ci.yml for CD is like re-using GET requests for state-changing operations.

With that much PR I've found and read about how nicely integrated and awesome CD in gitlab is, I'm full of doubt about the whole platform in general right now. How out of this world can one be to consider anything as broken as CD in gitlab as production ready right now? Had it never occurred to gitlab developers that there's something wrong with the current CD design? Not even once during the whole development phase? And if so, can I trust the rest of the features in gitlab? I was really happy about gitlab at first, but this state of affairs just made me sad.

@dragonroot asks a good question: How can this have been so completely missed by the dev team? This is not just a big deal - it's a deal breaker.

Secret variables are an integral part of the CI/CD capabilities and Gitlab clearly states that "you can use them for passwords, secret keys or whatever you want" (see the Variables page for any project). However, it doesn't even attempt at protecting them. This is misleading at best.

Mistakes not malice

If we are talking about the on-premise/self-hosted version of Gitlab, protection against malicious users trying to exploit the system is, I believe, a different problem. You need to have some degree of trust in your developers, and in the mechanisms that ensure that only they can access the system.

Most of the issues around security here are actually not about malicious intent but about the ability for trusted users to mess things up by mistake. We are human, and we make mistakes. Systems should be designed to steer us away from making them and, when we do anyway, mitigate the negative effects. In the current design the CD system does the complete opposite - it amplifies mistakes to potentially catastrophic levels.

Branch protection

@markpundsack offers as one possible solution to extend branch protection to variables:

An alternative might be to leverage existing restrictions on branches, and use branch-variables instead.

I think (https://gitlab.com/gitlab-org/gitlab-ce/issues/23861#note_17494152) this is a great idea, as it addresses most of the issues in a simple way. If we extend branch restriction to apply to variables, it means we do not need to complicate things by worrying about runners, roles, environments or even .gitlab-ci.yml changes. We stop the problem at its source: the secrets.

What it could look like

I'm sure the implementation will require some thinking and lots of work. It's not obvious to me what the best way to achieve it is, but here's an idea:

Log in to Gitlab as the Master or Owner of a project and select Variables in the project settings. From there select a branch on which you want to create the variable. Alternatively, you can set project-wide variables (available to all branches) or wildcard variables, available to a subset of regex matching branches (for example version-.* or hotfix-*).

Before Gitlab CI runs a build, it checks which branch the build is running on, and exports the corresponding variables to the environment.

What it solves

Here's how branch-restriction would address the concerns in the original post:

All project variables are accessible by any developer. If one developer adds secure variables to a project with deployment capable creds, any developer could write a trivial CI script to use or output those variables.

As long as Merge Requests are reviewed, changes that output variables would not make it into protected branches. Builds on non-protected branches would not have access to the credentials, so the output would be empty.

Even if you restrict deploy actions to the master branch, a developer could overwrite that restriction in a branch's .gitlab-ci.yml to force a deploy of unvetted code.

With branch-specific variables, the master branch credentials would never be exported to feature branches, and any attempt to force deploy would fail at the first API call that uses the credentials.

Even if you restrict secrets to a dedicated, tagged runner, a developer could overwrite tags in a branch's .gitlab-ci.yml to run arbitrary code on the dedicated runner.

IMHO trying to patch up security on the runner means we've already failed. The runners should only execute the builds, and not worry about the contents. With branch-specific secret variables, any build executed on the runner will have (or not have) it's credentials set up, based on which branch it is running on.

What it doesn't solve

These concerns are not addressed directly:

Any developer can trigger a manual action. So while the master branch could be protected from pushes, any developer could trigger the deploy job in a pipeline.

Any developer can rollback.

However, we could extend branch protection further, to include pipeline actions. If you have Developer access to a project, you should not be able to push to master. By extension you should not be able to trigger manual actions on master builds: such as deploy or rollback.

What do you think? Am I missing anything here?

I'd just add that in some cases it's the runners which have the power - for instance, a runner which has direct password-less access to production (e.g. via a mounted fs or by other means). Such runners can be made project-specific, but there's currently no way to make them branch-specific.

@dragonroot That is how I handled some runners used to deploy to AWS services (instance profile). It removes the ability of users removing/using keys outside of the CI/CD Pipeline, but doesn't control the fact that users could just move whatever they want to do through the pipeline.

Mentioned in issue #23902 (moved)

Mentioned in issue #23262 (closed)

@ayufan I'd like to pick up #20261 (closed), #21358 (closed), #17739 (moved), and #24194 (closed) asap.

The MVP for #17739 (moved) and #24194 (closed) might be a simple checkbox to "protect" variables and runners to limit them to users with Master permission, rather than specifying branches. Obviously being able to have different variable values for different branches is a beneficial feature as well, but that shouldn't get in the way of solving the most immediate security concern. The verbiage of "protecting" the variable has a nice parallel with protected branches too.

/cc @ayufan @dimitrieh

Mentioned in issue #24196 (closed)

Created #24196 (closed) for MVP proposal above.

Mentioned in issue #17739 (moved)

Thanks again @markpundsack for consolidating all the issues. I feel that you've captured the current concerns well in the issue description.

It would be great if you could also clarify this bit:

...a simple checkbox to "protect" variables and runners to limit them to users with Master permission, rather than specifying branches.

Will that mean that the "protected" variables will be exported to a GitLab CI build only if that specific build was triggered by a commit made by a Master level user?

Mentioned in issue #24208 (moved)

@jonatanblue Yes, that's almost exactly what it means. Technically it's not about who made the commit, but who pushed the commit to GitLab.

@nick.thomas, relevant to a conversation we had on IRC.

I'll add another vote to branch-specific variables/pipelines - seems like it would fold in very nicely with protected branches. Of course, there's no way then to add user-specific keys, but my personal feeling is that those aren't as important.

Even if you restrict deploy actions to the master branch, a developer could overwrite that restriction in a branch's .gitlab-ci.yml to force a deploy of unvetted code.

We've described another solution for this problem: #24794 (moved) . Needs your review and the go-ahead to implement.

@markpundsack

able to have different variable values for different branches is a beneficial feature as well

Could this be done by having an extra input field?

It will be perfect ! (from my point of view)

@dimitrieh Yeah, something like that, but it would be great if it supported wildcards so you could do review/* and share deploy creds with all your review apps, but nowhere else, perhaps like we do for the protected branch UI. Perhaps this should be discussed on https://gitlab.com/gitlab-org/gitlab-ce/issues/20367 though.

Hmm. We have build tags. We use that only for runners now, but maybe we should also extend that to variables too?

test:
  tags:
  - review-app

Then instead of specifying branch or review/* you would just configure your job to get variables for these tags.

It works well with:

environments, review or not review,
branches,
tags,
(merge requests if implemented).

mentioned in issue #23966 (moved)

@ayufan I think the idea of "tags" misses the point: it's insecure for .gitlab-ci.yml to be able to decide which variables its jobs get, because anyone can edit it (and push to an unprotected branch and submit an MR, triggering a pipeline).

For example, suppose that I've set up a MY_SECRET_DEPLOY_KEY variable with the "secret" tag, and .gitlab-ci.yml looks like:

deploy:
  script: deploy something benign with ${MY_SECRET_DEPLOY_KEY}
  only:
    - master
  when:
    - manual
  variable_tags: # your proposed "tags" option
    - secret

Nothing is stopping me, a Developer, from pushing this change to a branch and triggering a pipeline:

deploy:
  script: deploy something malicious with ${MY_SECRET_DEPLOY_KEY}
  variable_tags:
    - secret

The point of restricting variables to a (protected) branch is that regardless of how .gitlab-ci.yml is edited, it will only ever have access to MY_SECRET_DEPLOY_KEY when run within that branch. It's impossible for me to do anything malicious with MY_SECRET_DEPLOY_KEY on any other branch since my pipeline won't have access to that variable. My change would have to be merged into a protected branch to take effect.

For the record, my plans, in order of priority are:

Respect protected branches for manual actions: https://gitlab.com/gitlab-org/gitlab-ce/issues/20261
Protect variables and runners: https://gitlab.com/gitlab-org/gitlab-ce/issues/24196
Environment-specific variables: https://gitlab.com/gitlab-org/gitlab-ce/issues/20367
Protected environments: https://gitlab.com/gitlab-org/gitlab-ce/issues/20368

EDIT: This is out of date. See description instead

mentioned in issue #20826 (moved)

marked the task Secret variables can be inadvertently disclosed, via a screencast or looking over the shoulder. (#21358 (closed)) as completed

Actually, realizing "Block pipelines on protected branches: https://gitlab.com/gitlab-org/gitlab-ce/issues/30634" needs to be in the list too, so added that and moved plan to the description.

removed Product work label

mentioned in issue #20261 (closed)

marked the checklist item #18994 (closed) as completed

mentioned in issue #14925 (moved)

marked the checklist item Respect protected branches for manual actions: https://gitlab.com/gitlab-org/gitlab-ce/issues/20261 as completed

marked the checklist item #20261 (closed) as completed

changed the description

marked the checklist item Any developer can trigger a manual action. So while the master branch could be protected from pushes, any developer could trigger the deploy job in a pipeline. (#20261 (closed)) as completed

changed milestone to %Backlog

changed the description

Updated the issue for "Protected environments" which was moved to EE

marked the checklist item Block pipelines on protected branches: https://gitlab.com/gitlab-org/gitlab-ce/issues/30634 as completed

marked the checklist item Environment-specific variables: https://gitlab.com/gitlab-org/gitlab-ee/issues/2302 as completed

marked the checklist item https://gitlab.com/gitlab-org/gitlab-ee/issues/2302 as completed

marked the checklist item https://gitlab.com/gitlab-org/gitlab-ee/issues/2302 as incomplete

marked the checklist item #30634 (closed) as completed

marked the checklist item https://gitlab.com/gitlab-org/gitlab-ee/issues/2302 as completed

changed the description

changed milestone to %Next 3-6 months

mentioned in issue gitlab#4931

moved to gitlab#6127

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

mentioned in issue gitlab#6906

[Meta] Security/permissions of CD

Priorities as of 4/7/17

Issues

Designs

Child items ...

Activity

Mistakes not malice

Branch protection

What it could look like

What it solves

What it doesn't solve

Frequently Asked Questions

Admin message

Admin message

[Meta] Security/permissions of CD

Priorities as of 4/7/17

Issues

Activity

Mistakes not malice

Branch protection

What it could look like

What it solves

What it doesn't solve

Frequently Asked Questions