Skip to content
Snippets Groups Projects
Unverified Commit 35fe98b3 authored by Andrew Newdigate's avatar Andrew Newdigate
Browse files

docs: improve doc layout

parent 70990d58
No related branches found
No related tags found
No related merge requests found
Pipeline #13625041 passed
Stage: validate
Hide whitespace changes
Inline Side-by-side
Showing
with 605 additions and 525 deletions
.editorconfig
+ 4
− 0
View file @ 35fe98b3
@@ -9,3 +9,7 @@ insert_final_newline = true
charset = utf-8
indent_style = space
indent_size = 2
[*.md]
indent_style = unset
indent_size = unset
README.md
+ 25
− 525
View file @ 35fe98b3
@@ -2,6 +2,10 @@
 
This project includes reuseable GitLab-CI jobs for the GitLab Dedicated Project
 
## Developer Setup
Looking to setup your workstation? Follow the documentation at: [`docs/developer-setup.md`](docs/developer-setup.md).
## Warning about Refs
 
It is possible to use these templates without a `ref` tag when including them in your project, but this is not recommended as it may break your build process and make it non-repeatable. For example, if you revert a change, the templates will still reference the latest template which may no longer be compatible. For this reason, it's recommended you always use refs for your templates.
@@ -10,533 +14,29 @@ It is possible to use these templates without a `ref` tag when including them in
 
It is preferable to use templates over individual CI jobs as it reduces clutter, increases standardization and improves consistency across projects. Additionally, new functionality can be added to projects with only an upgrade to the `ref`.
 
### [`templates/standard.yml`](./templates/standard.yml)
This template should be included in all Infrastructure projects. It includes standard checks, gitlab-scanners, validations and release processes common to all projects using this template library.
Includes the following tasks:
1. [`gitlab-scanners.yml`](#gitlab-scanners): adds various GitLab SAST, Dependency Scanner, Secret Detection, Licence Scanning, Container Scanning and IAC Scanner tools.
1. [`editorconfig-check.yml`](#editorconfig-check): ensures that the repository adheres to the [EditorConfig](https://editorconfig.org/) rules for the project.
1. [`semantic-release.yml`](#semantic-release): runs [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/) on your repository.
1. [`asdf-tool-versions.yml`](#asdf-tool-versions): ensures that `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml, that all the plugins are declared in `./scripts/install-asdf-plugins.sh` and that ASDF is generally working.
1. [`shfmt.yml`](#shfmt): validates that shell-scripts use a canonical formatting.
1. [`shellcheck.yml`](#shellcheck): performs linting for shell scripts using [shellcheck](https://www.shellcheck.net/).
```yaml
# Requires stages validate and release
stages:
- validate
- release
- renovate_bot
include:
- local: .gitlab-ci-asdf-versions.yml
# This template should be included in all Infrastructure projects.
# It includes standard checks, gitlab-scanners, validations and release processes
# common to all projects using this template library.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesstandardyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/standard.yml
```
### [`templates/golang.yml`](./templates/golang.yml)
This template should be used for Golang projects.
Includes the following tasks:
1. [`go-mod-tidy.yml`](#go-mod-tidy): ensures that [`go mod tidy`](https://go.dev/ref/mod) is up-to-date and `go.mod` and `go.sum` are tidy.
1. [`go-unittests.yml`](#go-unittests): runs unit tests to ensure and emits a JUnit XML report for GitLab.
1. [`golangci-lint.yml`](#golangci-lint): runs [golangci-lint](https://github.com/golangci/golangci-lint) on the project.
1. [`goreleaser.yml`](#goreleaser): builds a binary release of the project using [GoReleaser](https://goreleaser.com/).
```yaml
# Requires stages validate and release
stages:
- validate
- release
# Better to define these through .gitlab-ci-asdf-versions.yml
variables:
GL_ASDF_GOLANG_VERSION: ...
GL_ASDF_GOLANGCI_LINT_VERSION: ...
GL_ASDF_GORELEASER_VERSION: ...
include:
- local: .gitlab-ci-asdf-versions.yml
# Runs golang standard tests, including tests, goreleaser, golangci-lint and go-mod-tidy
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesgolangyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/golang.yml
```
### [`templates/terraform.yml`](./templates/terraform.yml)
This template should be used for Terraform projects. It performs standard validity checks against the Terraform files in the project.
Ensure that the project has a valid [`.tflint.hcl`](https://github.com/terraform-linters/tflint/blob/v0.34.1/docs/user-guide/config.md) file in the root directory.
Includes the following tasks:
1. [`terraform-format.yml`](#terraform-format): runs [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to ensure that all Terraform files are correctly formatted.
1. [`terraform-validate.yml`](#terraform-validate): runs [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure that all Terraform files are valid.
1. [`tflint.yml`](#tflint): runs [`tflint`](https://github.com/terraform-linters/tflint) across all directories that contain `*.tf` files.
```yaml
# Requires validate stage
stages:
- validate
# Better to define these through .gitlab-ci-asdf-versions.yml
variables:
GL_ASDF_TERRAFORM_VERSION: ...
GL_ASDF_TFLINT_VERSION: ...
include:
- local: .gitlab-ci-asdf-versions.yml
# Runs Terraform validations, including tflint, terraform validate and terraform formatting checks
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesterraformyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/terraform.yml
```
| Template Include | Description |
|-|-|
| [`templates/standard.yml`](./templates/standard.md) | This template should be included in all Infrastructure projects. It includes standard checks, gitlab-scanners, validations and release processes common to all projects using this template library.
| [`templates/golang.yml`](./templates/golang.md) | This template should be used for Golang projects. |
| [`templates/terraform.yml`](./templates/terraform.md) | This template should be used for Terraform projects. It performs standard validity checks against the Terraform files in the project. |
 
## CI Jobs
 
Alternatively to using the templates above, you can include the following individual jobs.
 
### `gitlab-scanners`
This adds various GitLab SAST, Dependency Scanner, Secret Detection, Licence Scanning, Container Scanning and IAC Scanner tools.
* Default Stage: `validate`
```yaml
stages:
- validate
include:
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'gitlab-scanners.yml'
```
### [`golangci-lint`](./golangci-lint.yml)
Runs [golangci-lint](https://github.com/golangci/golangci-lint) on the project.
* Default Stage: `validate`
```yaml
stages:
- validate
include:
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'golangci-lint.yml'
```
### [`editorconfig-check`](./editorconfig-check.yml)
This job ensures that the repository adheres to the [EditorConfig](https://editorconfig.org/) rules for the project.
* Default Stage: `validate`
* Expects config file: `.editorconfig`
```yaml
stages:
- validate
include:
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'editorconfig-check.yml'
```
### [`semantic-release`](./semantic-release.yml)
This job will run [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/) on your repository. This allows the repository to be automatically tagged based on [conventional commit](https://www.conventionalcommits.org/en/v1.0.0/) messages.
* Default Stage: `release` (make sure this stage is one of the last in your pipeline)
You will also need a file called `.releaserc.json` included in your project. Copy this one if you are unsure.
```json
{
"branches": ["main"],
"plugins": [
[
"@semantic-release/commit-analyzer",
{
"preset": "conventionalcommits"
}
],
[
"@semantic-release/release-notes-generator",
{
"preset": "conventionalcommits"
}
],
"@semantic-release/gitlab"
]
}
```
Next, create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and `write_repository` scope and make it available in your CI environment via the `GITLAB_TOKEN` environment variable through the CI/CD Variables settings.
```yaml
stages:
- release
include:
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'semantic-release.yml'
```
### [`goreleaser`](./goreleaser.yml)
Including this template will build a binary release of the project using [GoReleaser](https://goreleaser.com/).
It runs on tag pipelines.
It requires a `.goreleaser.yml` file in the root of the project, which looks something like this:
* Default Stages: `validate`, `release`
```
before:
hooks:
- go mod tidy
builds:
- id: "<binaryname>"
binary: "<binaryname>"
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ldflags:
- -s -w -X '{{.ModulePath}}/cmd.version={{.Version}}' -X '{{.ModulePath}}/cmd.commit={{.Commit}}' -X '{{.ModulePath}}/cmd.date={{.Date}}'
archives:
- replacements:
darwin: Darwin
linux: Linux
amd64: x86_64
dockers:
- image_templates:
- "{{ .Env.CI_REGISTRY_IMAGE }}:latest"
- "{{ .Env.CI_REGISTRY_IMAGE }}:{{ .Tag }}"
- "{{ .Env.CI_REGISTRY_IMAGE }}:v{{ .Major }}"
goos: linux
goarch: amd64
release:
gitlab:
owner: gitlab-com/gl-infra # GitLab Group
name: <projectname> # GitLab Project
checksum:
name_template: 'checksums.txt'
snapshot:
name_template: "{{ .Tag }}-next"
changelog:
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
gitlab_urls:
api: '{{ .Env.CI_SERVER_URL }}'
download: '{{ .Env.CI_SERVER_URL }}'
```
Next, create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and make it available in your CI environment via the `GITLAB_TOKEN` environment variable through the CI/CD Variables settings.
```yaml
stages:
- validate
- release
variables:
GL_ASDF_GORELEASER_VERSION: ...
include:
# build binary release artifacts with goreleaser
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: goreleaser.yml
```
### [`asdf-tool-versions`](./asdf-tool-versions.yml)
Checks that the `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml, that all the plugins are declared in `./scripts/install-asdf-plugins.sh` and that ASDF is generally working.
* Default Stages: `validate`
Setup process:
1. Ensure that the file [`scripts/install-asdf-plugins.sh`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/install-asdf-plugins.sh) exists in the repository. Tailor for the `asdf` configuration of the project.
1. Ensure that the files [`scripts/update-asdf-version-variables.sh`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/update-asdf-version-variables.sh) exists in the repository.
1. If the job will need SSH access to GitLab using a [deploy key](https://docs.gitlab.com/ee/user/project/deploy_keys/), create the deploy key for the project to which access will be required. Add the private key to the `VALIDATE_ASDF_TOOL_VERSIONS_DEPLOY_KEY_FILE` CI/CD variable.
1. Note that the SSH Agent may fail to parse the private key if a final newline is omitted.
1. To generate the key, follow the instructions at <https://docs.gitlab.com/ee/user/ssh.html#generate-an-ssh-key-pair>.
1. TL;DR is `ssh-keygen -t rsa -b 2048 -C "asdf-validation deploy key for <project>"`
1. Add the include to your `.gitlab-ci.yml` file:
```yaml
include:
- local: .gitlab-ci-asdf-versions.yml
# Checks that the `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml,
# that all the plugins are declared in `./scripts/install-asdf-plugins.sh`
# and that ASDF is generally working
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#asdf-tool-versions
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: asdf-tool-versions.yml
```
### [`tflint`](./tflint.yml)
Runs [`tflint`](https://github.com/terraform-linters/tflint) across all directories that contain `*.tf` files.
Setup process:
1. Ensure that a [`.tflint.hcl`](https://github.com/terraform-linters/tflint/blob/master/docs/user-guide/config.md) file exists in the root of the project.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
1. Ensure that the `GL_ASDF_TFLINT_VERSION` version for tflint is configured.
1. Directories can be excluded from tflint using the `TFLINT_EXCLUDE_REGEX` variable. See the example below.
1. The task will generate a [junit test output file](https://docs.gitlab.com/ee/ci/unit_test_reports.html) for any failed linter checks.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TFLINT_VERSION: ...
GL_ASDF_TERRAFORM_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Runs tflint on all terraform module directories
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: tflint.yml
```
### [`terraform-format`](./terraform-format.yml)
Runs [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to ensure that all Terraform files are correctly formatted.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TERRAFORM_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensures that all terraform files are correctly formatted
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: terraform-format.yml
```
### [`terraform-validate`](./terraform-validate.yml)
Runs [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure that all Terraform files are valid.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TERRAFORM_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensures that all terraform files are syntactically valid
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: terraform-validate.yml
```
### [`renovate-bot`](./renovate-bot.yml)
Runs [`renovatebot`](https://docs.renovatebot.com/) against the project to automatically upgrade dependencies.
1. Ensure that a `renovate_bot` stage exists in your `.gitlab-ci.yml` configuration
1. Create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and `write_repository` scope and make it available in your CI environment via the `RENOVATE_GITLAB_TOKEN` environment variable through the CI/CD Variables settings. Make sure the variable is **Protected** and **Masked**.
1. Lookup the GitHub Personal Access token from [1Password Production Vault](https://gitlab.1password.com/vaults/7xbs54owvjux3cypztlhyetej4/allitems/53z2zuf7urh7hoy3nqeqsei27e) and save it into the CI environment variable `RENOVATE_GITHUB_TOKEN`. Make sure the variable is **Protected** and **Masked**.
1. Note that you can use Group Access Tokens and Group CI/CD variables, instead of Project-level ones should you choose.
1. Create a CI Pipeline Schedule called `Renovatebot` with a daily schedule, eg `0 1 * * *`. Ensure that the CI Pipeline schedule includes a variable, `RENOVATE_SCHEDULED` with a value of `1`.
1. Create a `renovate.json` file in the root of the project. See below for an example configuration:
```json
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:base"],
"requireConfig": true,
"prConcurrentLimit": 5,
"ignorePaths": []
}
```
```yaml
stages:
- renovate_bot
include:
# Upgrades dependencies on a schedule
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#renovate-bot
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: renovate-bot.yml
```
### [`go-unittests`](./go-unittests.yml)
Runs unit tests to ensure and emits a JUnit XML report for GitLab.
1. Ensure that the `GL_ASDF_GOLANG_VERSION` version for Go is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_GOLANG_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Runs Go unit tests
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#go-unittests
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: go-unittests.yml
```
### [`go-mod-tidy`](./go-mod-tidy.yml)
Ensures that [`go mod tidy`](https://go.dev/ref/mod) is up-to-date and `go.mod` and `go.sum` are tidy.
1. Ensure that the `GL_ASDF_GOLANG_VERSION` version for Go is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_GOLANG_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Perform `go mod tidy` and ensure that go.mod and go.sum are tidy.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#go-mod-tidy
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: go-mod-tidy.yml
```
### [`shfmt`](./shfmt.yml)
Validates that shell-scripts use a canonical formatting.
1. Ensure that the `GL_ASDF_SHFMT_VERSION` version for Go is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_SHFMT_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensure that all shell-scripts are formatted according to a
# standard canonical format
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: shfmt.yml
```
### [`shellcheck`](./shellcheck.yml)
Performs linting for shell scripts using [Shellcheck](https://www.shellcheck.net/).
1. Ensure that the `GL_ASDF_SHELLCHECK_VERSION` version is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
1. Arguments for Shellcheck can optionally be passed through the variable `SHELLCHECK_ARGS`
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_SHELLCHECK_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
# Any arguments for shellcheck
SHELLCHECK_ARGS: --source-path=. --source-path=./test/ --source-path=./test/lib/ --source-path=./test/integration/ --external-sources
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensure that all shell-scripts are formatted according to a
# standard canonical format
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: shellcheck.yml
```
| Include | Description |
|-|-|
[`gitlab-scanners.yml`](./gitlab-scanners.md) | This adds various GitLab SAST, Dependency Scanner, Secret Detection, Licence Scanning, Container Scanning and IAC Scanner tools. |
| [`asdf-tool-versions`](./asdf-tool-versions.md) | Checks that the `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml, that all the plugins are declared in `./scripts/install-asdf-plugins.sh` and that ASDF is generally working. |
| [`editorconfig-check.yml`](./editorconfig-check.md) | This job ensures that the repository adheres to the [EditorConfig](https://editorconfig.org/) rules for the project. |
| [`go-mod-tidy.yml`](./go-mod-tidy.md) | Ensures that [`go mod tidy`](https://go.dev/ref/mod) is up-to-date and `go.mod` and `go.sum` are tidy. |
| [`go-unittests.yml`](./go-unittests.md) | Runs unit tests to ensure and emits a JUnit XML report for GitLab. |
| [`golangci-lint.yml`](./golangci-lint.md) | Runs [golangci-lint](https://github.com/golangci/golangci-lint) on the project. |
| [`goreleaser.yml`](./goreleaser.md) | Including this template will build a binary release of the project using [GoReleaser](https://goreleaser.com/).|
| [`renovate-bot.yml`](./renovate-bot.md) | Runs [`renovatebot`](https://docs.renovatebot.com/) against the project to automatically upgrade dependencies. |
| [`semantic-release.yml`](./semantic-release.md) | This job will run [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/) on your repository. This allows the repository to be automatically tagged based on [conventional commit](https://www.conventionalcommits.org/en/v1.0.0/) messages. |
| [`shellcheck.yml`](./shellcheck.md) | Performs linting for shell scripts using [Shellcheck](https://www.shellcheck.net/). |
| [`shfmt.yml`](./shfmt.md)| Validates that shell-scripts use a canonical formatting. |
| [`terraform-format.yml`](./terraform-format.md)| Runs [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to ensure that all Terraform files are correctly formatted. |
| [`terraform-validate.yml`](./terraform-validate.md) | Runs [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure that all Terraform files are valid. |
| [`tflint.yml`](./tflint.md) | Runs [`tflint`](https://github.com/terraform-linters/tflint) across all directories that contain `*.tf` files. |
asdf-tool-versions.md 0 → 100644
+ 29
− 0
View file @ 35fe98b3
# [`asdf-tool-versions`](./asdf-tool-versions.yml)
Checks that the `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml, that all the plugins are declared in `./scripts/install-asdf-plugins.sh` and that ASDF is generally working.
* Default Stages: `validate`
Setup process:
1. Ensure that the file [`scripts/install-asdf-plugins.sh`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/install-asdf-plugins.sh) exists in the repository. Tailor for the `asdf` configuration of the project.
1. Ensure that the files [`scripts/update-asdf-version-variables.sh`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/update-asdf-version-variables.sh) exists in the repository.
1. If the job will need SSH access to GitLab using a [deploy key](https://docs.gitlab.com/ee/user/project/deploy_keys/), create the deploy key for the project to which access will be required. Add the private key to the `VALIDATE_ASDF_TOOL_VERSIONS_DEPLOY_KEY_FILE` CI/CD variable.
1. Note that the SSH Agent may fail to parse the private key if a final newline is omitted.
1. To generate the key, follow the instructions at <https://docs.gitlab.com/ee/user/ssh.html#generate-an-ssh-key-pair>.
1. TL;DR is `ssh-keygen -t rsa -b 2048 -C "asdf-validation deploy key for <project>"`
1. Add the include to your `.gitlab-ci.yml` file:
```yaml
include:
- local: .gitlab-ci-asdf-versions.yml
# Checks that the `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml,
# that all the plugins are declared in `./scripts/install-asdf-plugins.sh`
# and that ASDF is generally working
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/asdf-tool-versions.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: asdf-tool-versions.yml
```
docs/developer-setup.md 0 → 100644
+ 51
− 0
View file @ 35fe98b3
# Environment Setup
This page describes the common process for managing dependencies using `asdf` that is used across many Infrastructure projects.
## Setup
### Step 1: Install `asdf`
Installation instructions for `asdf` can be found at <https://asdf-vm.com/guide/getting-started.html#_1-install-dependencies>.
If you're running on macos, the recommended approach is to use Homebrew:
```shell
brew install asdf
```
Linux users should follow the instructions for their package manager in [the ASDF documentation](https://asdf-vm.com/guide/getting-started).
### Step 2: Install development dependencies
Install all the plugins by running:
```shell
./scripts/install-asdf-plugins.sh
```
This will install required `asdf` plugins, and install the correct versions of the development tools.
Note that after pulling changes to the repository, you may sometimes need to re-run `./scripts/install-asdf-plugins.sh` to update your locally installed plugins and tool-versions.
## Updating Tool Versions
We use CI checks to ensure that tool versions used in GitLab-CI and on developer instances don't go out of sync.
### Keeping Versions in Sync between GitLab-CI and `asdf`.
`asdf` (and `.tool-versions` generally) is the SSOT for tool versions used in this repository.
To keep `.tool-versions` in sync with `.gitlab-ci.yml`, there is a helper script,
`./scripts/update-asdf-version-variables`.
#### Process for updating a Tool Version
```shell
./scripts/update-asdf-version-variables
```
1. Update the version in `.tool-versions`
1. Run `asdf install` to install latest version
1. Run `./scripts/update-asdf-version-variables` to update a refresh of the `.gitlab-ci-asdf-versions.yml` file
1. Commit the changes
1. A CI job (see [`asdf-tool-versions.md`](../asdf-tool-versions.md)) will validate the changes.
editorconfig-check.md 0 → 100644
+ 19
− 0
View file @ 35fe98b3
# [`editorconfig-check`](./editorconfig-check.yml)
This job ensures that the repository adheres to the [EditorConfig](https://editorconfig.org/) rules for the project.
* Stages: `validate`
* Expects config file: `.editorconfig`
```yaml
stages:
- validate
include:
# validate .editorconfig
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/asdf-tool-versions.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'editorconfig-check.yml'
```
gitlab-scanners.md 0 → 100644
+ 16
− 0
View file @ 35fe98b3
# `gitlab-scanners`
This adds various GitLab SAST, Dependency Scanner, Secret Detection, Licence Scanning, Container Scanning and IAC Scanner tools.
* Stages: `validate`
```yaml
stages:
- validate
include:
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/gitlab-scanners.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'gitlab-scanners.yml'
```
go-mod-tidy.md 0 → 100644
+ 24
− 0
View file @ 35fe98b3
# [`go-mod-tidy`](./go-mod-tidy.yml)
Ensures that [`go mod tidy`](https://go.dev/ref/mod) is up-to-date and `go.mod` and `go.sum` are tidy.
1. Ensure that the `GL_ASDF_GOLANG_VERSION` version for Go is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_GOLANG_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Perform `go mod tidy` and ensure that go.mod and go.sum are tidy.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/go-mod-tidy.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: go-mod-tidy.yml
```
go-unittests.md 0 → 100644
+ 24
− 0
View file @ 35fe98b3
# [`go-unittests`](./go-unittests.yml)
Runs unit tests to ensure and emits a JUnit XML report for GitLab.
1. Ensure that the `GL_ASDF_GOLANG_VERSION` version for Go is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_GOLANG_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Runs Go unit tests
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/go-unittests.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: go-unittests.yml
```
golangci-lint.md 0 → 100644
+ 18
− 0
View file @ 35fe98b3
# [`golangci-lint`](./golangci-lint.yml)
Runs [golangci-lint](https://github.com/golangci/golangci-lint) on the project.
* Stages: `validate`
```yaml
stages:
- validate
include:
# Runs golangci-lint on the project.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/golangci-lint.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'golangci-lint.yml'
```
goreleaser.md 0 → 100644
+ 83
− 0
View file @ 35fe98b3
# [`goreleaser`](./goreleaser.yml)
Including this template will build a binary release of the project using [GoReleaser](https://goreleaser.com/).
It runs on tag pipelines.
It requires a `.goreleaser.yml` file in the root of the project, which looks something like this:
* Default Stages: `validate`, `release`
```
before:
hooks:
- go mod tidy
builds:
- id: "<binaryname>"
binary: "<binaryname>"
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ldflags:
- -s -w -X '{{.ModulePath}}/cmd.version={{.Version}}' -X '{{.ModulePath}}/cmd.commit={{.Commit}}' -X '{{.ModulePath}}/cmd.date={{.Date}}'
archives:
- replacements:
darwin: Darwin
linux: Linux
amd64: x86_64
dockers:
- image_templates:
- "{{ .Env.CI_REGISTRY_IMAGE }}:latest"
- "{{ .Env.CI_REGISTRY_IMAGE }}:{{ .Tag }}"
- "{{ .Env.CI_REGISTRY_IMAGE }}:v{{ .Major }}"
goos: linux
goarch: amd64
release:
gitlab:
owner: gitlab-com/gl-infra # GitLab Group
name: <projectname> # GitLab Project
checksum:
name_template: 'checksums.txt'
snapshot:
name_template: "{{ .Tag }}-next"
changelog:
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
gitlab_urls:
api: '{{ .Env.CI_SERVER_URL }}'
download: '{{ .Env.CI_SERVER_URL }}'
```
Next, create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and make it available in your CI environment via the `GITLAB_TOKEN` environment variable through the CI/CD Variables settings.
```yaml
stages:
- validate
- release
variables:
GL_ASDF_GORELEASER_VERSION: ...
include:
# build binary release artifacts with goreleaser
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/goreleaser.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: goreleaser.yml
```
renovate-bot.md 0 → 100644
+ 34
− 0
View file @ 35fe98b3
# [`renovate-bot`](./renovate-bot.yml)
Runs [`renovatebot`](https://docs.renovatebot.com/) against the project to automatically upgrade dependencies.
1. Ensure that a `renovate_bot` stage exists in your `.gitlab-ci.yml` configuration
1. Create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and `write_repository` scope and make it available in your CI environment via the `RENOVATE_GITLAB_TOKEN` environment variable through the CI/CD Variables settings. Make sure the variable is **Protected** and **Masked**.
1. Lookup the GitHub Personal Access token from [1Password Production Vault](https://gitlab.1password.com/vaults/7xbs54owvjux3cypztlhyetej4/allitems/53z2zuf7urh7hoy3nqeqsei27e) and save it into the CI environment variable `RENOVATE_GITHUB_TOKEN`. Make sure the variable is **Protected** and **Masked**.
1. Note that you can use Group Access Tokens and Group CI/CD variables, instead of Project-level ones should you choose.
1. Create a CI Pipeline Schedule called `Renovatebot` with a daily schedule, eg `0 1 * * *`. Ensure that the CI Pipeline schedule includes a variable, `RENOVATE_SCHEDULED` with a value of `1`.
1. Create a `renovate.json` file in the root of the project. See below for an example configuration:
```json
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:base"],
"requireConfig": true,
"prConcurrentLimit": 5,
"ignorePaths": []
}
```
```yaml
stages:
- renovate_bot
include:
# Upgrades dependencies on a schedule
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/renovate-bot.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: renovate-bot.yml
```
semantic-release.md 0 → 100644
+ 43
− 0
View file @ 35fe98b3
# [`semantic-release`](./semantic-release.yml)
This job will run [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/) on your repository. This allows the repository to be automatically tagged based on [conventional commit](https://www.conventionalcommits.org/en/v1.0.0/) messages.
* Default Stage: `release` (make sure this stage is one of the last in your pipeline)
You will also need a file called `.releaserc.json` included in your project. Copy this one if you are unsure.
```json
{
"branches": ["main"],
"plugins": [
[
"@semantic-release/commit-analyzer",
{
"preset": "conventionalcommits"
}
],
[
"@semantic-release/release-notes-generator",
{
"preset": "conventionalcommits"
}
],
"@semantic-release/gitlab"
]
}
```
Next, create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and `write_repository` scope and make it available in your CI environment via the `GITLAB_TOKEN` environment variable through the CI/CD Variables settings.
```yaml
stages:
- release
include:
# Analyze commits to determine whether to cut a release
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/semantic-release.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'semantic-release.yml'
```
shellcheck.md 0 → 100644
+ 31
− 0
View file @ 35fe98b3
# [`shellcheck`](./shellcheck.yml)
Performs linting for shell scripts using [Shellcheck](https://www.shellcheck.net/).
1. Ensure that the `GL_ASDF_SHELLCHECK_VERSION` version is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
1. Arguments for Shellcheck can optionally be passed through the variable `SHELLCHECK_ARGS`
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_SHELLCHECK_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
# Any arguments for shellcheck
SHELLCHECK_ARGS: --source-path=. --source-path=./test/ --source-path=./test/lib/ --source-path=./test/integration/ --external-sources
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensure that all shell-scripts are formatted according to a
# standard canonical format
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/shellcheck.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: shellcheck.yml
```
shfmt.md 0 → 100644
+ 27
− 0
View file @ 35fe98b3
# [`shfmt`](./shfmt.yml)
Validates that shell-scripts use a canonical formatting.
1. Ensure that the `GL_ASDF_SHFMT_VERSION` version for Go is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_SHFMT_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensure that all shell-scripts are formatted according to a
# standard canonical format
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: shfmt.yml
```
templates/golang.md 0 → 100644
+ 31
− 0
View file @ 35fe98b3
[`templates/golang.yml`](./templates/golang.yml)
This template should be used for Golang projects.
Includes the following tasks:
1. [`go-mod-tidy.yml`](../go-mod-tidy.md): ensures that [`go mod tidy`](https://go.dev/ref/mod) is up-to-date and `go.mod` and `go.sum` are tidy.
1. [`go-unittests.yml`](../go-unittests.md): runs unit tests to ensure and emits a JUnit XML report for GitLab.
1. [`golangci-lint.yml`](../golangci-lint.md): runs [golangci-lint](https://github.com/golangci/golangci-lint) on the project.
1. [`goreleaser.yml`](../goreleaser.md): builds a binary release of the project using [GoReleaser](https://goreleaser.com/).
```yaml
# Requires stages validate and release
stages:
- validate
- release
# Better to define these through .gitlab-ci-asdf-versions.yml
variables:
GL_ASDF_GOLANG_VERSION: ...
GL_ASDF_GOLANGCI_LINT_VERSION: ...
GL_ASDF_GORELEASER_VERSION: ...
include:
- local: .gitlab-ci-asdf-versions.yml
# Runs golang standard tests, including tests, goreleaser, golangci-lint and go-mod-tidy
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesgolangyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/golang.yml
```
templates/standard.md 0 → 100644
+ 30
− 0
View file @ 35fe98b3
# [`templates/standard.yml`](./templates/standard.yml)
This template should be included in all Infrastructure projects. It includes standard checks, gitlab-scanners, validations and release processes common to all projects using this template library.
Includes the following tasks:
1. [`asdf-tool-versions.yml`](../asdf-tool-versions.md): ensures that `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml, that all the plugins are declared in `./scripts/install-asdf-plugins.sh` and that ASDF is generally working.
1. [`editorconfig-check.yml`](../editorconfig-check.md): ensures that the repository adheres to the [EditorConfig](https://editorconfig.org/) rules for the project.
1. [`gitlab-scanners.yml`](../gitlab-scanners.md): adds various GitLab SAST, Dependency Scanner, Secret Detection, Licence Scanning, Container Scanning and IAC Scanner tools.
1. [`semantic-release.yml`](../semantic-release.md): runs [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/) on your repository.
1. [`shellcheck.yml`](../shellcheck.md): performs linting for shell scripts using [shellcheck](https://www.shellcheck.net/).
1. [`shfmt.yml`](../shfmt.md): validates that shell-scripts use a canonical formatting.
```yaml
# Requires stages validate and release
stages:
- validate
- release
- renovate_bot
include:
- local: .gitlab-ci-asdf-versions.yml
# This template should be included in all Infrastructure projects.
# It includes standard checks, gitlab-scanners, validations and release processes
# common to all projects using this template library.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesstandardyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/standard.yml
```
templates/terraform.md 0 → 100644
+ 30
− 0
View file @ 35fe98b3
# [`templates/terraform.yml`](./templates/terraform.yml)
This template should be used for Terraform projects. It performs standard validity checks against the Terraform files in the project.
Ensure that the project has a valid [`.tflint.hcl`](https://github.com/terraform-linters/tflint/blob/v0.34.1/docs/user-guide/config.md) file in the root directory.
Includes these tasks:
1. [`terraform-format.yml`](../terraform-format.md): runs [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to ensure that all Terraform files are correctly formatted.
1. [`terraform-validate.yml`](../terraform-validate.md): runs [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure that all Terraform files are valid.
1. [`tflint.yml`](../tflint.md): runs [`tflint`](https://github.com/terraform-linters/tflint) across all directories that contain `*.tf` files.
```yaml
# Requires validate stage
stages:
- validate
# Better to define these through .gitlab-ci-asdf-versions.yml
variables:
GL_ASDF_TERRAFORM_VERSION: ...
GL_ASDF_TFLINT_VERSION: ...
include:
- local: .gitlab-ci-asdf-versions.yml
# Runs Terraform validations, including tflint, terraform validate and terraform formatting checks
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesterraformyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/terraform.yml
```
terraform-format.md 0 → 100644
+ 24
− 0
View file @ 35fe98b3
# [`terraform-format`](./terraform-format.yml)
Runs [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to ensure that all Terraform files are correctly formatted.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TERRAFORM_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensures that all terraform files are correctly formatted
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/terraform-format.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: terraform-format.yml
```
terraform-validate.md 0 → 100644
+ 27
− 0
View file @ 35fe98b3
# [`terraform-validate`](./terraform-validate.yml)
Runs [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure that all Terraform files are valid.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TERRAFORM_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensures that all terraform files are syntactically valid
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/terraform-validate.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: terraform-validate.yml
```
tflint.md 0 → 100644
+ 35
− 0
View file @ 35fe98b3
# [`tflint`](./tflint.yml)
Runs [`tflint`](https://github.com/terraform-linters/tflint) across all directories that contain `*.tf` files.
Setup process:
1. Ensure that a [`.tflint.hcl`](https://github.com/terraform-linters/tflint/blob/master/docs/user-guide/config.md) file exists in the root of the project.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
1. Ensure that the `GL_ASDF_TFLINT_VERSION` version for tflint is configured.
1. Directories can be excluded from tflint using the `TFLINT_EXCLUDE_REGEX` variable. See the example below.
1. The task will generate a [junit test output file](https://docs.gitlab.com/ee/ci/unit_test_reports.html) for any failed linter checks.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TFLINT_VERSION: ...
GL_ASDF_TERRAFORM_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Runs tflint on all terraform module directories
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/tflint.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: tflint.yml
```
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment