gitlab-runner merge requestshttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests2016-09-19T06:11:55Zhttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/53Runner Autoscale through docker-machine2016-09-19T06:11:55ZKamil TrzcińśkiRunner Autoscale through docker-machine## What is Docker Machine?
> Machine lets you create Docker hosts on your computer, on cloud providers, and inside your own data center. It automatically creates hosts, installs Docker on them, then configures the docker client to tal...## What is Docker Machine?
> Machine lets you create Docker hosts on your computer, on cloud providers, and inside your own data center. It automatically creates hosts, installs Docker on them, then configures the docker client to talk to them. A “machine” is the combination of a Docker host and a configured client.
Source: https://docs.docker.com/machine/
## Why Docker Machine?
Is easy to use. Is well documented. Is well supported and constantly extended. It supports almost any cloud provider or virtualization infrastructure. We need minimal amount of changes to support Docker Machine: machine enumeration and inspection. We don't need to implement any "cloud specific" features.
## GitLab Runner and Docker Machine sitting on tree
This MR adds support for Docker Machine in GitLab Runner allowing to automatically spin-up Docker-enabled hosts on any provider supported by Docker Machine: DigitalOcean, VirtualBox, Azure...
This allows to have automatically scalable cluster of runners managed by one GitLab Runner instance. You can set to have a maximum of 100 jobs running in parallel. The Docker Machine integration will create Cloud Instances on demand to accommodate the load. After defined period (it can be 1h) the machines will be destroyed and a new one will be created when needed.
We use one GitLab Runner to manage all jobs.
## How it works?
1. The GitLab Runner requests the Job from coordinator
2. The GitLab Runner checks available docker machines and takes the first one free
3. If no machine is found a new one is created
4. The Job is run on provisioned machine
5. After the Job is run the disk space is checked and if it goes below certain level the machine is removed
6. Otherwise the machine is added to list of machines that can be reused for other jobs
## How to enable Docker Machine configuration
The simplest `config.toml`:
```
concurrent = 50
[[runners]]
url = "https://gitlab.com/ci"
token = "XYZ"
name = "YZX"
executor = "docker+machine"
limit = 10
[runners.docker]
image = "ruby:2.1"
[runners.machine]
IdleCount = 5
IdleTime = 600
MaxBuilds = 100
MachineDriver = "digitalocean"
MachineName = "auto-scale-%s"
MachineOptions = [
"digitalocean-image=coreos-beta",
"digitalocean-ssh-user=core",
"digitalocean-access-token=DO_ACCESS_TOKEN",,
"digitalocean-region=nyc2",
"digitalocean-size=4gb",
"digitalocean-private-networking",
"engine-registry-mirror=http://10.128.11.79:34723"
]
```
The above configuration will run up to 10 jobs and it will remove machines after 600 seconds of inactivity. It leave at most 5 nodes waiting to process the jobs. The machines are created on DigitalOcean, using CoreOS image. The boot time of CoreOS VM is between 40-60s.
Other MachineOptions:
```
docker-machine create -d digitalocean --help
```
## Limitations
The runner cache is tied to one node. It can be solved by exposing cache as internal runner HTTP service.
**No longer true: if this is enabled: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/88**
## Considerations
The provisioned machines should not hold any data. The machines should be disposable if not needed. Having to provision a new node should not make a difference for a running a next stage build.
v1.1Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/73Change unregister command to accept runner name2016-09-19T06:12:03Zusername-removed-164769Change unregister command to accept runner nameAs suggested in #56. Unregister now takes a single parameter `-n` or `--name` and then proceed to look up the correct runner credentials from the configuration.
Closes #1149 As suggested in #56. Unregister now takes a single parameter `-n` or `--name` and then proceed to look up the correct runner credentials from the configuration.
Closes #1149 v1.2Tomasz Maczukintomasz@gitlab.comTomasz Maczukintomasz@gitlab.comhttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/80WIP: Download build artifacts from previous stages and restore them in contex...2017-09-03T21:04:27ZKamil TrzcińśkiWIP: Download build artifacts from previous stages and restore them in context of the build (Technology Preview)This is Technology Preview of artifacts passing between stages.
As for now the artifacts from previous stages are unconditionally extracted in context of current build.
**How it works?**
The approach is naive:
1. We archive art...This is Technology Preview of artifacts passing between stages.
As for now the artifacts from previous stages are unconditionally extracted in context of current build.
**How it works?**
The approach is naive:
1. We archive artifacts and upload them to coordinator.
2. When asking for a new build we receive from coordinator information about the builds on which current one depends on with information about artifacts.
3. We download artifacts archive.
4. We unpack artifacts archive.
It's not the fastest (requires to upload/download), but allows to have artifacts passing between different runners and can later be optimised to use local artifacts.
**Requirements:**
GitLab 8.4
**Test builds:**
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-linux-386
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-linux-amd64
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-linux-arm
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-darwin-386
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-darwin-amd64
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-windows-386.exe
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-windows-amd64.exe
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-freebsd-386
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-freebsd-amd64
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/download-artifacts/binaries/gitlab-ci-multi-runner-freebsd-arm
v1.1Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/88Shared caching (S3-compatible)2016-09-23T14:24:58ZKamil TrzcińśkiShared caching (S3-compatible)This implements support for external caching server, it has to be S3-compatible server.
The GitLab Runner when restoring and archiving cache will ask S3-server and download or upload the archive.
This allows to use separate machine...This implements support for external caching server, it has to be S3-compatible server.
The GitLab Runner when restoring and archiving cache will ask S3-server and download or upload the archive.
This allows to use separate machines (auto-scaling: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/53) and have a full caching support.
To enable caching define that in `config.toml`:
```
[runners.cache]
Type = "s3"
ServerAddress = "address.com"
AccessKey = "access-key"
SecretKey = "secret-key"
BucketName = "runner"
Insecure = false
```
As S3-compatible server you can use: https://www.minio.io/:
# Create a new VM, you can also use that with Web interface or use any already provision Docker server
docker-machine create -d digitalocean \
--digitalocean-image=coreos-beta \
--digitalocean-ssh-user=core \
--digitalocean-access-token=digital-ocean-token \
--digitalocean-region=nyc2 \
--digitalocean-size=8gb \
--digitalocean-private-networking \
gitlab-runner-cache-server
docker-machine ssh gitlab-runner-cache-server
# Start minio S3-compatible server
# You can replace 34233 with any port
docker run -it --restart always -p 34233:9000 -v /.minio:/.minio -v /export:/export --name minio minio/minio:latest server /export
# Check Access Key and Secret Key
sudo cat /.minio/config.json
# Create bucket
sudo mkdir /export/runner
# Check IP address of server
ifconfig eth0
# or
ifconfig eth1
# Put the IP in GitLab Runner config.toml:
[runners.cache]
Type = "s3"
ServerAddress = "ip-address:34233"
AccessKey = "access-key"
SecretKey = "secret-key"
BucketName = "runner"
Insecure = false
v1.1Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/89Simplify logging2016-09-23T14:24:58ZKamil TrzcińśkiSimplify loggingRefactor code related to sending build trace.
It moves all logic outside of `common/build.go` and `executors/*` to `network/build*.go`.
It makes the implementation simpler and adds missing tests for testing the build trace support....Refactor code related to sending build trace.
It moves all logic outside of `common/build.go` and `executors/*` to `network/build*.go`.
It makes the implementation simpler and adds missing tests for testing the build trace support.
/cc @tmaczukin v1.1Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/97Update golang.org/x/crypto/ssh2016-11-14T13:16:57ZKamil TrzcińśkiUpdate golang.org/x/crypto/sshFixes https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1009
Fixes https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1009
v1.1Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/106Add options to runner configuration to specify commands executed before code ...2017-02-24T04:32:59Zusername-removed-347329Add options to runner configuration to specify commands executed before code clone and buildThis is somewhat similar to !20 but meets a need that I have in my CI setup. I'm running some of my builds on cloud VMs using the new autoscale feature. However, those are across a WAN from my GitLab instance. Some of the repositories th...This is somewhat similar to !20 but meets a need that I have in my CI setup. I'm running some of my builds on cloud VMs using the new autoscale feature. However, those are across a WAN from my GitLab instance. Some of the repositories that I do builds on are quite large, so the time required to just clone the code adds large overhead to the build process.
I've mitigated this by setting up a caching proxy server on the cloud network using [gitpod](https://github.com/sitaramc/gitpod). This works great, but I need a way to configure those runners to use the proxy Git server instead.
That's where this merge request comes in. I added a new per-runner option that allows the user to inject commands before the Git clone/fetch occurs. This allows me to adjust the Git global configuration to rewrite the fetch URLs that the CI script uses to point to my proxy server instead. This seems to work and seems to be a pretty elegant solution.
I then had to add a second option that allows the specification of commands before the build occurs. This was because my project has submodules in it, which are not currently cloneable by `gitlab-runner`. In my `.gitlab-ci.yml`, I have some commands to insert an SSH private key from a secure variable into the environment and subsequently clone the code. Since I use the Docker executor, the `git clone` occurs in a different container than the subsequent build commands, so my pre-clone commands don't have any effect on the build. Therefore, I added the ability to configure commands run at this point as well.
I know there was some opposition to adding functionality that was like this in !20, but I really believe this is the best place to add this feature. Any chance of this getting merged?v1.6https://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/111Add support for cloning VirtualBox VM snapshots, linked clones2016-09-19T06:12:12Zusername-removed-434855Add support for cloning VirtualBox VM snapshots, linked clonesThis commit adds two new configuration options for VirtualBox runners:
- "base_snapshot": The name or UUID of a snapshot of the VM to clone. If
this is empty or excluded, the current state of the VM will be cloned
(as is the exi...This commit adds two new configuration options for VirtualBox runners:
- "base_snapshot": The name or UUID of a snapshot of the VM to clone. If
this is empty or excluded, the current state of the VM will be cloned
(as is the existing behavior).
- "linked_clone": Whether or not a linked clone of a VM snapshot should
be created when initially cloning the VM. Using linked clones can
significantly reduce cloning time as well as the disk space needed for
the clone, as the VM disk image does not need to be copied when
cloning. This is ignored if "base_snapshot" is empty or unspecified.v1.4Tomasz Maczukintomasz@gitlab.comTomasz Maczukintomasz@gitlab.comhttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/113Allow to define a name of uploaded artifacts archive and a list of dependent ...2016-09-23T07:36:33ZKamil TrzcińśkiAllow to define a name of uploaded artifacts archive and a list of dependent buildsThis implements needed changes for this to make work: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3182.
cc @tmaczukin This implements needed changes for this to make work: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3182.
cc @tmaczukin v1.1https://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/147Extend version output2016-09-23T11:08:20ZTomasz Maczukintomasz@gitlab.comExtend version outputThis MR is changeing the `--version` flag output to provide an extended version information:
```bash
# Before changes
$ gitlab-ci-multi-runner --version
gitlab-ci-multi-runner-linux-amd64 version 1.2.0~beta.71.gad1c29f (ad1c29f)
...This MR is changeing the `--version` flag output to provide an extended version information:
```bash
# Before changes
$ gitlab-ci-multi-runner --version
gitlab-ci-multi-runner-linux-amd64 version 1.2.0~beta.71.gad1c29f (ad1c29f)
# After changes
$ gitlab-ci-multi-runner --version
Version: 1.2.0~beta.71.gad1c29f
Git revision: ad1c29f
GO version: go1.5.3
Built: Mon, 09 May 2016 17:16:51 +0200
OS/Arch: linux/amd64
```
Closes #223v1.2Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/153Add support for after_script2016-09-19T06:12:01ZKamil TrzcińśkiAdd support for after_scriptThis brings support for after_script. The amount of changes is quite large, but in order to implement that I needed to make Executor more dumber that it was in fact moving some of the executor code up to Build. Build is now responsible f...This brings support for after_script. The amount of changes is quite large, but in order to implement that I needed to make Executor more dumber that it was in fact moving some of the executor code up to Build. Build is now responsible for executing build plan, the plan is run with help of executor. This is how the logic supposed to be.
During all this refactoring I did fix our famous `call :buildscript` issue. Right now all steps of build plan are executed as separate commands. This makes it simpler to track what and where is happening.
The introduced changes will allow us to easily add artifacts for failed builds. This will also make the plugin system super-easy to implement.
**Test builds:**
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-linux-386
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-linux-amd64
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-linux-arm
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-darwin-386
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-darwin-amd64
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-windows-386.exe
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-windows-amd64.exe
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-freebsd-386
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-freebsd-amd64
* https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/finally-script/binaries/gitlab-ci-multi-runner-freebsd-arm
cc @tmaczukin v1.2Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/157Add support for TLS client authentication2017-08-21T08:43:08Zusername-removed-525672Add support for TLS client authenticationAllow gitlab-ci to connect to a gitlab host using TLS client authentication
(mutual authentication). Adds configuration and support for using TLS client
certificates when using go's TLS transport layer and also sets git enviromental
v...Allow gitlab-ci to connect to a gitlab host using TLS client authentication
(mutual authentication). Adds configuration and support for using TLS client
certificates when using go's TLS transport layer and also sets git enviromental
variables for runners.
See also #1291 and !86 v9.2Tomasz Maczukintomasz@gitlab.comTomasz Maczukintomasz@gitlab.comhttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/169Add posibility to specify CpusetCpus, Dns & DnsSearch when creating docker co...2016-09-19T06:12:03Zusername-removed-356725Add posibility to specify CpusetCpus, Dns & DnsSearch when creating docker containerIntroduce new parameters in docker section of `config.toml`. Detailed explanation of parameters can be found in [docker manual](https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/#create-a-container)
- **cpuset_cpus** ...Introduce new parameters in docker section of `config.toml`. Detailed explanation of parameters can be found in [docker manual](https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/#create-a-container)
- **cpuset_cpus** - String value containing the cgroups CpusetCpus to use (e.g., 0-3, 0,1).
- **dns** - A list of DNS servers for the container to use.
- **dns_search** - A list of DNS search domains
Example:
```bash
[runners.docker]
cpuset_cpus = "0,1,4-6"
dns = ["8.8.8.8"]
dns_search = [""]
```
v1.3https://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/183Set custom User-Agent header with version number and runtime information2016-09-23T11:09:51ZTomasz Maczukintomasz@gitlab.comSet custom User-Agent header with version number and runtime informationThis can be useful for troubleshooting, for statistics and also for decisions.This can be useful for troubleshooting, for statistics and also for decisions.v1.3https://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/188Allow experimentally to specify GIT_DEPTH2016-11-01T11:35:29ZKamil TrzcińśkiAllow experimentally to specify GIT_DEPTHThis allows to shallow clone repository. The value is passed to `git fetch` and `git clone`.
It doesn't introduce any new syntax, it only requires to pass `GIT_DEPTH` variable to build environment. I think of this as an experimental f...This allows to shallow clone repository. The value is passed to `git fetch` and `git clone`.
It doesn't introduce any new syntax, it only requires to pass `GIT_DEPTH` variable to build environment. I think of this as an experimental feature that we can later extend with .gitlab-ci.yml syntax:
```
variables:
GIT_DEPTH: "20"
```
@tmaczukin What do you think?
v1.3https://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/203Log docker machine IP address2016-09-19T06:12:05ZKamil TrzcińśkiLog docker machine IP addressWhen using auto-scaling this will print to stdout and syslog following message:
**at the start**:
```
Starting docker-machine build... build=1840675 created=2016-06-16 12:46:07.165254115 +0200 CEST docker=tcp://10...When using auto-scaling this will print to stdout and syslog following message:
**at the start**:
```
Starting docker-machine build... build=1840675 created=2016-06-16 12:46:07.165254115 +0200 CEST docker=tcp://107.170.102.232:2376 name=runner-e0089f1d-kamil-auto-scale-runners-1466013997-901c5a03 project=100 runner=e0089f1d usedcount=0
```
and **at the end**:
```
Finished docker-machine build: exit code 129 build=1840675 created=2016-06-16 12:46:07.165254115 +0200 CEST docker=tcp://107.170.102.232:2376 name=runner-e0089f1d-kamil-auto-scale-runners-1466013997-901c5a03 project=100 runner=e0089f1d usedcount=0
```
This fixes: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1391
cc @tmaczukinv1.3https://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/214Feature/use ci environments2016-09-19T06:12:06ZTomasz Maczukintomasz@gitlab.comFeature/use ci environmentsThis MR refactorizes `.gitlab-ci.yml` and adds environment settings to it so we can track what is the latest version for Bleeding Edge and for Stable Release.
Closes #1426
/cc @ayufan This MR refactorizes `.gitlab-ci.yml` and adds environment settings to it so we can track what is the latest version for Bleeding Edge and for Stable Release.
Closes #1426
/cc @ayufan v1.4Kamil TrzcińśkiKamil Trzcińśkihttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/215add security_opt config option in docker executor.2016-09-19T06:12:12Zusername-removed-414746add security_opt config option in docker executor.this matches the --security-opt flag in docker run, and takes a
list of ':' separate key/value pairs.
check for more info:
https://docs.docker.com/engine/reference/run/#/security-configuration
---
Closes #1475this matches the --security-opt flag in docker run, and takes a
list of ':' separate key/value pairs.
check for more info:
https://docs.docker.com/engine/reference/run/#/security-configuration
---
Closes #1475v1.4https://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/218Improve logging2016-09-19T06:12:10ZKamil TrzcińśkiImprove logging- Introduce BuildError to generate soft failure on build error and generate hard failure on system error
- Use WithField instead of pushing data as arguments of most of logging functions
- Introduce BuildLogger which removes logging fu...- Introduce BuildError to generate soft failure on build error and generate hard failure on system error
- Use WithField instead of pushing data as arguments of most of logging functions
- Introduce BuildLogger which removes logging functionality from AbstractExecutor
- Use BuildLogger and write notice message as early as possible
- Remove finish notice/error from AbstractExecutor
- Sending and appending build trace is now debug message by default
cc @tmaczukin
v1.4Tomasz Maczukintomasz@gitlab.comTomasz Maczukintomasz@gitlab.comhttps://staging.gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/231Add integration tests2016-09-19T06:12:11ZKamil TrzcińśkiAdd integration testsThis tries to add black box tests for most common scenarios for a few executors.
It fixes two small bugs.
This tries to add black box tests for most common scenarios for a few executors.
It fixes two small bugs.
Tomasz Maczukintomasz@gitlab.comTomasz Maczukintomasz@gitlab.com