Skip to content
Snippets Groups Projects
Unverified Commit 35fe98b3 authored by Andrew Newdigate's avatar Andrew Newdigate
Browse files

docs: improve doc layout

parent 70990d58
No related branches found
No related tags found
No related merge requests found
Pipeline #13625041 passed
Loading
Loading
@@ -9,3 +9,7 @@ insert_final_newline = true
charset = utf-8
indent_style = space
indent_size = 2
[*.md]
indent_style = unset
indent_size = unset
This diff is collapsed.
# [`asdf-tool-versions`](./asdf-tool-versions.yml)
Checks that the `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml, that all the plugins are declared in `./scripts/install-asdf-plugins.sh` and that ASDF is generally working.
* Default Stages: `validate`
Setup process:
1. Ensure that the file [`scripts/install-asdf-plugins.sh`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/install-asdf-plugins.sh) exists in the repository. Tailor for the `asdf` configuration of the project.
1. Ensure that the files [`scripts/update-asdf-version-variables.sh`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/update-asdf-version-variables.sh) exists in the repository.
1. If the job will need SSH access to GitLab using a [deploy key](https://docs.gitlab.com/ee/user/project/deploy_keys/), create the deploy key for the project to which access will be required. Add the private key to the `VALIDATE_ASDF_TOOL_VERSIONS_DEPLOY_KEY_FILE` CI/CD variable.
1. Note that the SSH Agent may fail to parse the private key if a final newline is omitted.
1. To generate the key, follow the instructions at <https://docs.gitlab.com/ee/user/ssh.html#generate-an-ssh-key-pair>.
1. TL;DR is `ssh-keygen -t rsa -b 2048 -C "asdf-validation deploy key for <project>"`
1. Add the include to your `.gitlab-ci.yml` file:
```yaml
include:
- local: .gitlab-ci-asdf-versions.yml
# Checks that the `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml,
# that all the plugins are declared in `./scripts/install-asdf-plugins.sh`
# and that ASDF is generally working
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/asdf-tool-versions.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: asdf-tool-versions.yml
```
# Environment Setup
This page describes the common process for managing dependencies using `asdf` that is used across many Infrastructure projects.
## Setup
### Step 1: Install `asdf`
Installation instructions for `asdf` can be found at <https://asdf-vm.com/guide/getting-started.html#_1-install-dependencies>.
If you're running on macos, the recommended approach is to use Homebrew:
```shell
brew install asdf
```
Linux users should follow the instructions for their package manager in [the ASDF documentation](https://asdf-vm.com/guide/getting-started).
### Step 2: Install development dependencies
Install all the plugins by running:
```shell
./scripts/install-asdf-plugins.sh
```
This will install required `asdf` plugins, and install the correct versions of the development tools.
Note that after pulling changes to the repository, you may sometimes need to re-run `./scripts/install-asdf-plugins.sh` to update your locally installed plugins and tool-versions.
## Updating Tool Versions
We use CI checks to ensure that tool versions used in GitLab-CI and on developer instances don't go out of sync.
### Keeping Versions in Sync between GitLab-CI and `asdf`.
`asdf` (and `.tool-versions` generally) is the SSOT for tool versions used in this repository.
To keep `.tool-versions` in sync with `.gitlab-ci.yml`, there is a helper script,
`./scripts/update-asdf-version-variables`.
#### Process for updating a Tool Version
```shell
./scripts/update-asdf-version-variables
```
1. Update the version in `.tool-versions`
1. Run `asdf install` to install latest version
1. Run `./scripts/update-asdf-version-variables` to update a refresh of the `.gitlab-ci-asdf-versions.yml` file
1. Commit the changes
1. A CI job (see [`asdf-tool-versions.md`](../asdf-tool-versions.md)) will validate the changes.
# [`editorconfig-check`](./editorconfig-check.yml)
This job ensures that the repository adheres to the [EditorConfig](https://editorconfig.org/) rules for the project.
* Stages: `validate`
* Expects config file: `.editorconfig`
```yaml
stages:
- validate
include:
# validate .editorconfig
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/asdf-tool-versions.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'editorconfig-check.yml'
```
# `gitlab-scanners`
This adds various GitLab SAST, Dependency Scanner, Secret Detection, Licence Scanning, Container Scanning and IAC Scanner tools.
* Stages: `validate`
```yaml
stages:
- validate
include:
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/gitlab-scanners.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'gitlab-scanners.yml'
```
# [`go-mod-tidy`](./go-mod-tidy.yml)
Ensures that [`go mod tidy`](https://go.dev/ref/mod) is up-to-date and `go.mod` and `go.sum` are tidy.
1. Ensure that the `GL_ASDF_GOLANG_VERSION` version for Go is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_GOLANG_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Perform `go mod tidy` and ensure that go.mod and go.sum are tidy.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/go-mod-tidy.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: go-mod-tidy.yml
```
# [`go-unittests`](./go-unittests.yml)
Runs unit tests to ensure and emits a JUnit XML report for GitLab.
1. Ensure that the `GL_ASDF_GOLANG_VERSION` version for Go is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_GOLANG_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Runs Go unit tests
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/go-unittests.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: go-unittests.yml
```
# [`golangci-lint`](./golangci-lint.yml)
Runs [golangci-lint](https://github.com/golangci/golangci-lint) on the project.
* Stages: `validate`
```yaml
stages:
- validate
include:
# Runs golangci-lint on the project.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/golangci-lint.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'golangci-lint.yml'
```
# [`goreleaser`](./goreleaser.yml)
Including this template will build a binary release of the project using [GoReleaser](https://goreleaser.com/).
It runs on tag pipelines.
It requires a `.goreleaser.yml` file in the root of the project, which looks something like this:
* Default Stages: `validate`, `release`
```
before:
hooks:
- go mod tidy
builds:
- id: "<binaryname>"
binary: "<binaryname>"
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ldflags:
- -s -w -X '{{.ModulePath}}/cmd.version={{.Version}}' -X '{{.ModulePath}}/cmd.commit={{.Commit}}' -X '{{.ModulePath}}/cmd.date={{.Date}}'
archives:
- replacements:
darwin: Darwin
linux: Linux
amd64: x86_64
dockers:
- image_templates:
- "{{ .Env.CI_REGISTRY_IMAGE }}:latest"
- "{{ .Env.CI_REGISTRY_IMAGE }}:{{ .Tag }}"
- "{{ .Env.CI_REGISTRY_IMAGE }}:v{{ .Major }}"
goos: linux
goarch: amd64
release:
gitlab:
owner: gitlab-com/gl-infra # GitLab Group
name: <projectname> # GitLab Project
checksum:
name_template: 'checksums.txt'
snapshot:
name_template: "{{ .Tag }}-next"
changelog:
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
gitlab_urls:
api: '{{ .Env.CI_SERVER_URL }}'
download: '{{ .Env.CI_SERVER_URL }}'
```
Next, create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and make it available in your CI environment via the `GITLAB_TOKEN` environment variable through the CI/CD Variables settings.
```yaml
stages:
- validate
- release
variables:
GL_ASDF_GORELEASER_VERSION: ...
include:
# build binary release artifacts with goreleaser
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/goreleaser.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: goreleaser.yml
```
# [`renovate-bot`](./renovate-bot.yml)
Runs [`renovatebot`](https://docs.renovatebot.com/) against the project to automatically upgrade dependencies.
1. Ensure that a `renovate_bot` stage exists in your `.gitlab-ci.yml` configuration
1. Create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and `write_repository` scope and make it available in your CI environment via the `RENOVATE_GITLAB_TOKEN` environment variable through the CI/CD Variables settings. Make sure the variable is **Protected** and **Masked**.
1. Lookup the GitHub Personal Access token from [1Password Production Vault](https://gitlab.1password.com/vaults/7xbs54owvjux3cypztlhyetej4/allitems/53z2zuf7urh7hoy3nqeqsei27e) and save it into the CI environment variable `RENOVATE_GITHUB_TOKEN`. Make sure the variable is **Protected** and **Masked**.
1. Note that you can use Group Access Tokens and Group CI/CD variables, instead of Project-level ones should you choose.
1. Create a CI Pipeline Schedule called `Renovatebot` with a daily schedule, eg `0 1 * * *`. Ensure that the CI Pipeline schedule includes a variable, `RENOVATE_SCHEDULED` with a value of `1`.
1. Create a `renovate.json` file in the root of the project. See below for an example configuration:
```json
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:base"],
"requireConfig": true,
"prConcurrentLimit": 5,
"ignorePaths": []
}
```
```yaml
stages:
- renovate_bot
include:
# Upgrades dependencies on a schedule
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/renovate-bot.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: renovate-bot.yml
```
# [`semantic-release`](./semantic-release.yml)
This job will run [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/) on your repository. This allows the repository to be automatically tagged based on [conventional commit](https://www.conventionalcommits.org/en/v1.0.0/) messages.
* Default Stage: `release` (make sure this stage is one of the last in your pipeline)
You will also need a file called `.releaserc.json` included in your project. Copy this one if you are unsure.
```json
{
"branches": ["main"],
"plugins": [
[
"@semantic-release/commit-analyzer",
{
"preset": "conventionalcommits"
}
],
[
"@semantic-release/release-notes-generator",
{
"preset": "conventionalcommits"
}
],
"@semantic-release/gitlab"
]
}
```
Next, create a [Project Access Token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) with `api` and `write_repository` scope and make it available in your CI environment via the `GITLAB_TOKEN` environment variable through the CI/CD Variables settings.
```yaml
stages:
- release
include:
# Analyze commits to determine whether to cut a release
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/semantic-release.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: 'semantic-release.yml'
```
# [`shellcheck`](./shellcheck.yml)
Performs linting for shell scripts using [Shellcheck](https://www.shellcheck.net/).
1. Ensure that the `GL_ASDF_SHELLCHECK_VERSION` version is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
1. Arguments for Shellcheck can optionally be passed through the variable `SHELLCHECK_ARGS`
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_SHELLCHECK_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
# Any arguments for shellcheck
SHELLCHECK_ARGS: --source-path=. --source-path=./test/ --source-path=./test/lib/ --source-path=./test/integration/ --external-sources
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensure that all shell-scripts are formatted according to a
# standard canonical format
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/shellcheck.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: shellcheck.yml
```
shfmt.md 0 → 100644
# [`shfmt`](./shfmt.yml)
Validates that shell-scripts use a canonical formatting.
1. Ensure that the `GL_ASDF_SHFMT_VERSION` version for Go is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_SHFMT_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensure that all shell-scripts are formatted according to a
# standard canonical format
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: shfmt.yml
```
[`templates/golang.yml`](./templates/golang.yml)
This template should be used for Golang projects.
Includes the following tasks:
1. [`go-mod-tidy.yml`](../go-mod-tidy.md): ensures that [`go mod tidy`](https://go.dev/ref/mod) is up-to-date and `go.mod` and `go.sum` are tidy.
1. [`go-unittests.yml`](../go-unittests.md): runs unit tests to ensure and emits a JUnit XML report for GitLab.
1. [`golangci-lint.yml`](../golangci-lint.md): runs [golangci-lint](https://github.com/golangci/golangci-lint) on the project.
1. [`goreleaser.yml`](../goreleaser.md): builds a binary release of the project using [GoReleaser](https://goreleaser.com/).
```yaml
# Requires stages validate and release
stages:
- validate
- release
# Better to define these through .gitlab-ci-asdf-versions.yml
variables:
GL_ASDF_GOLANG_VERSION: ...
GL_ASDF_GOLANGCI_LINT_VERSION: ...
GL_ASDF_GORELEASER_VERSION: ...
include:
- local: .gitlab-ci-asdf-versions.yml
# Runs golang standard tests, including tests, goreleaser, golangci-lint and go-mod-tidy
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesgolangyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/golang.yml
```
# [`templates/standard.yml`](./templates/standard.yml)
This template should be included in all Infrastructure projects. It includes standard checks, gitlab-scanners, validations and release processes common to all projects using this template library.
Includes the following tasks:
1. [`asdf-tool-versions.yml`](../asdf-tool-versions.md): ensures that `.tool-versions` file is synced with .gitlab-ci-asdf-versions.yml, that all the plugins are declared in `./scripts/install-asdf-plugins.sh` and that ASDF is generally working.
1. [`editorconfig-check.yml`](../editorconfig-check.md): ensures that the repository adheres to the [EditorConfig](https://editorconfig.org/) rules for the project.
1. [`gitlab-scanners.yml`](../gitlab-scanners.md): adds various GitLab SAST, Dependency Scanner, Secret Detection, Licence Scanning, Container Scanning and IAC Scanner tools.
1. [`semantic-release.yml`](../semantic-release.md): runs [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/) on your repository.
1. [`shellcheck.yml`](../shellcheck.md): performs linting for shell scripts using [shellcheck](https://www.shellcheck.net/).
1. [`shfmt.yml`](../shfmt.md): validates that shell-scripts use a canonical formatting.
```yaml
# Requires stages validate and release
stages:
- validate
- release
- renovate_bot
include:
- local: .gitlab-ci-asdf-versions.yml
# This template should be included in all Infrastructure projects.
# It includes standard checks, gitlab-scanners, validations and release processes
# common to all projects using this template library.
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesstandardyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/standard.yml
```
# [`templates/terraform.yml`](./templates/terraform.yml)
This template should be used for Terraform projects. It performs standard validity checks against the Terraform files in the project.
Ensure that the project has a valid [`.tflint.hcl`](https://github.com/terraform-linters/tflint/blob/v0.34.1/docs/user-guide/config.md) file in the root directory.
Includes these tasks:
1. [`terraform-format.yml`](../terraform-format.md): runs [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to ensure that all Terraform files are correctly formatted.
1. [`terraform-validate.yml`](../terraform-validate.md): runs [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure that all Terraform files are valid.
1. [`tflint.yml`](../tflint.md): runs [`tflint`](https://github.com/terraform-linters/tflint) across all directories that contain `*.tf` files.
```yaml
# Requires validate stage
stages:
- validate
# Better to define these through .gitlab-ci-asdf-versions.yml
variables:
GL_ASDF_TERRAFORM_VERSION: ...
GL_ASDF_TFLINT_VERSION: ...
include:
- local: .gitlab-ci-asdf-versions.yml
# Runs Terraform validations, including tflint, terraform validate and terraform formatting checks
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/README.md#templatesterraformyml
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: templates/terraform.yml
```
# [`terraform-format`](./terraform-format.yml)
Runs [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to ensure that all Terraform files are correctly formatted.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TERRAFORM_VERSION: ...
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensures that all terraform files are correctly formatted
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/terraform-format.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: terraform-format.yml
```
# [`terraform-validate`](./terraform-validate.yml)
Runs [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure that all Terraform files are valid.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TERRAFORM_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Ensures that all terraform files are syntactically valid
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/terraform-validate.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: terraform-validate.yml
```
# [`tflint`](./tflint.yml)
Runs [`tflint`](https://github.com/terraform-linters/tflint) across all directories that contain `*.tf` files.
Setup process:
1. Ensure that a [`.tflint.hcl`](https://github.com/terraform-linters/tflint/blob/master/docs/user-guide/config.md) file exists in the root of the project.
1. Ensure that the `GL_ASDF_TERRAFORM_VERSION` version for terraform is configured.
1. Ensure that the `GL_ASDF_TFLINT_VERSION` version for tflint is configured.
1. Directories can be excluded from tflint using the `TFLINT_EXCLUDE_REGEX` variable. See the example below.
1. The task will generate a [junit test output file](https://docs.gitlab.com/ee/ci/unit_test_reports.html) for any failed linter checks.
1. Supports the `COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP` variable, for excluding paths from validation.
```yaml
stages:
- validate
# Not needed if .gitlab-ci-asdf-versions.yml is included...
variables:
GL_ASDF_TFLINT_VERSION: ...
GL_ASDF_TERRAFORM_VERSION: ...
# Exclude vendor and files directories from validation
COMMON_TASK_VALIDATIONS_EXCLUDES_REGEXP: '^\.(/vendor/|/files/)'
include:
# Not required, but recommended
- local: .gitlab-ci-asdf-versions.yml
# Runs tflint on all terraform module directories
# see https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/tflint.md
- project: 'gitlab-com/gl-infra/common-ci-tasks'
ref: v1.18.0 # renovate:managed
file: tflint.yml
```
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment