gitlab-runner exec: easily test builds locally

Awesome.

Maybe we can export env variables from the UI as a .something files, to be stored outside of version control, but found by gitlab-runner exec

@JobV We can make it better: fetch it from GitLab, but as first first approach we don't need it :)

You can always pass the env variables manually, using command line.

Title changed from gitlab-runner exec: easily test builds to gitlab-runner exec: easily test builds locally

Awesome , thank you @ayufan for the great work !

Status changed to closed

@ayufan "You can always pass the env variables manually, using command line."

how? You have an example for me plz?

EDIT: Answer is a variable definition in the yml file.

for example: variables: DATABASE_URL: "postgres://postgres@postgres/my_database"

@jan-philipp-bucher

gitlab-runner exec shell job --help

gitlab-runner exec shell job --env VARIABLE=VALUE

@ayufan where is this documented? I want to link it from about.gitlab.com/gitlab-ci I found https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/executors/README.md but it doesn't seem to talk about the local use case that much, also it isn't linked from http://doc.gitlab.com/ce/ci/

@sytses See https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/commands/README.md#gitlab-runner-exec

Is it possible to override this variable if it already exists in the .gitlab-ci.yml file? It doesn't appear so. It would be nice to be able to override variables used for databases and other services when running tests locally.

mentioned in issue #1221 (closed)

@ayufan I'm new to gitlab-ci and I've started to setup my gitlab-ci on my own gitlab-ce server 3 days ago. "gitlab-runner exec" is really useful thanks! My environment: gitlab-ce 8.7.3 on Debian 7, gitlab-runner 1.1.3 on Debian 7. I have 2 questions:

It seems "gitlab-runner exec shell myjob" doesn't take the "variables:" keyword into account (if "variables:" is used inside myjob or outside of it). It does work when the build is run by the gitlab-runner service but it doesn't when I use "gitlab-runner exec shell". Should I create an issue about this ? (I've not found an existing issue about this)
I'd like to re-use the .gitlab-ci.yml in another context than gitlab-ci. On a developer workstation where the git repository of the project is already cloned in "/home/khelkun/myproject", I wish I could run "/home/khelkun/myproject/.gitlab-ci.yml" to build the project without cloning a new fresh copy of the HEAD of the repository, e.g: build the sources which already exist in "/home/khelkun/myproject". It doesn't seems to me that gitlab-runner can do that. Am I wrong ? Is there another way I could do it ?

It seems "gitlab-runner exec shell myjob" doesn't take the "variables:" keyword into account (if "variables:" is used inside myjob or outside of it). It does work when the build is run by the gitlab-runner service but it doesn't when I use "gitlab-runner exec shell". Should I create an issue about this ? (I've not found an existing issue about this)

Yes. Please create an issue.

I'd like to re-use the .gitlab-ci.yml in another context than gitlab-ci. On a developer workstation where the git repository of the project is already cloned in "/home/khelkun/myproject", I wish I could run "/home/khelkun/myproject/.gitlab-ci.yml" to build the project without cloning a new fresh copy of the HEAD of the repository, e.g: build the sources which already exist in "/home/khelkun/myproject". It doesn't seems to me that gitlab-runner can do that. Am I wrong ? Is there another way I could do it ?

gitlab-runner exec does that.

@ayufan as far as I see "variables:" works at the global level but not at the job level, please see: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1302

About my question 2., when I run:

~/dev/srv-git/myproject$ gitlab-runner exec shell build-release:myjob

The output starts with:

Using Shell executor...
Running on khelkun-workstation...
Cloning repository...
Cloning in '/home/khelkun/dev/srv-git/myproject/builds/0/project-1'...

Can I skip the cloning step ? And tell "gitlab-runner exec" to run 'build-release:myjob' directly on the '/home/khelkun/dev/srv-git/myproject/' instead of '/home/khelkun/dev/srv-git/myproject/builds/0/project-1' ?

@khelkun This will be fixed by: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/160

It has not been fixed by !160 (merged) for me, @ayufan:

gdubicki@mbp-greg:~/git/myproject$ ~/gitlab-runner --version
Version:      1.3.0~beta.20.g36963db
Git revision: 36963db
Git branch:   master
GO version:   go1.6.2
Built:        Mon, 13 Jun 2016 17:33:02 +0000
OS/Arch:      darwin/amd64

gdubicki@mbp-greg:~/git/myproject$ ~/gitlab-runner exec shell test
WARNING: You most probably have uncommitted changes.
WARNING: These changes will not be tested.
gitlab-ci-multi-runner 1.3.0~beta.20.g36963db (36963db)
Using Shell executor...
Running on mbp-greg...
Cloning repository...
Cloning into '/Users/gdubicki/git/myproject/builds/0/project-1'...
done.

- it's still cloning.

@grzegorz-dubicki Clonning will always happen.

My !160 (merged) were to for variables support at job level.

Hi there, just wanted to mention that I'd also be a user of an option to mount the current folder in the runner image instead of cloning. In my case, the .gitlab-ci.yml file calls external scripts, so if I need to iterate on these scripts then I need to push them... Which kinds of defeats the purpose of local testing :)

@grzegorz-dubicki and @khelkun are right, if cloning is unavoidable, it defeats whole purpose of local testing. Then, we are back into "commit and push to test" paradigm which is not the way development should be done.

Vote for https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1359 then , @jmatsushita @rubycut .

Local automated testing is just that, workstation local. That is valuable in itself, since pushing would no longer be required, avoiding history writing and potentially improving the initial quality of pushed commits. Reducing time cost for testing is an independent objective. I agree that's an important objective as well.

I've just started playing with this great feature and I've seen in the docs that the cache and artefacts may and may not work. I'm wondering if I can end up on the "may" side though.

The ultimate goal is to build a node app via a gitlab-ci task locally before pushing to make sure there are no surprises later. For this I need node_modulesto be cached between gitlab-runner exec calls somehow - the process takes ages otherwise.

Here is a simplified.gitlab-ci.yml:

build:
  image: node:6.5.0
  cache:
    key: '42' # attempt to cache the folder 'globally' for even more simplicity
    paths:
    - node_modules/
  script:
    - ls -la
    - mkdir node_modules
    - touch node_modules/hi
    - ls -la

After running gitlab-runner exec docker --docker-privileged --docker-cache-dir /tmp/gitlabrunner build for the second time, I'd expect to see node_modules in the first ls, but this does not happen no matter how I modify various options of the run script. According to the output, the cache is being attempted to be restored and saved, but this has no effect:

Running with gitlab-ci-multi-runner 1.5.2 (76fdacd)
Using Docker executor with image node:6.5.0 ...
Pulling docker image node:6.5.0 ...
Running on runner--project-1-concurrent-0 via machinename...
Cloning repository...
Cloning into '/builds/project-1'...
done.
Checking out b316f21c as dev...
Checking cache for 42...
###
### output from ls, mkdir, touch and ls
###
Creating cache 42...
node_modules/: found 2 matching files              
Build succeeded

I suspect that some caching may be achieved with docker volumes, but it's not quite clear to me how to make work while keeping .gitlab-ci.yml both good for local execution and remote CI. If anyone has succeed in this challenge, could you please share a hint?

P.S.: all is happening on Ubuntu, i.e. docker is local.

Looks like I got it working, yay!

gitlab-runner exec docker --docker-privileged --cache-dir=/tmp/gitlabrunner --docker-volumes /tmp/gitlabrunner:/tmp/gitlabrunner build

Directory name inside docker runner can be any as long as it mounts to smth meaningful on the host machine:

gitlab-runner exec docker --docker-privileged --cache-dir=/dir/inside/docker --docker-volumes /tmp/gitlabrunner:/dir/inside/docker whatevertask

The reason why cache does not work out of box as I understand it now is because each time you call exec a new runner machine is created and then the container gets removed. Unless you mount the cache to the host, all the data you preserve within that container gets wiped. This peculiarity of how exec works might be worth noting in the docs as it was not very clear in the beginning. A lot of users may want local caching, I'm sure!

I'll additionally add that if you use a variable for your cache name then it has to be passed in

  cache:
    key: "$CI_PROJECT_NAME"
    paths:
      - .m2/repository/

Then you need to pass in --env CI_PROJECT_NAME=anything

I can't make dependencies and artifacts work with the local runner. It's a pain to setup a proper deployment pipeline without the ability to test changes locally.

Gitlab 8.9.6, gitlab-runner 1.5.3

Note to self/knowledge sharing: to pass more than one env var use multiple --env parameters, like this: --env VAR1=val1 --env VAR2=val2 (...)

Mentioned in issue #1853

If you want artifacts out of a container, be sure to use Docker's volume mounting capabilities to mount a local host directory into the container.

gitlab-ci-multi-runner exec docker --docker-volumes `pwd`/build-output:/build/my-project/build-output build

For shared cache and artifacts between locally jobs run I use:

gitlab-ci-multi-runner exec docker --cache-dir /cache --docker-volumes `pwd`/build-output:/cache JOB1
gitlab-ci-multi-runner exec docker --cache-dir /cache --docker-volumes `pwd`/build-output:/cache JOB2

I have my stages as 1) build 2) test. How do I locally trigger runner to execute test after build?

gitlab-ci-multi-runner exec shell test_job

If I just run test job, it will not trigger build first. This simply gives me error as some required files that should be generated in build stages is missing. Could we kick off the whole sequential build, test, deployment, etc locally?

It's not working for me with artifacts. If I try to mount the build dir, I'll get:
rm: can't remove '/builds/project-1': Resource busy see: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1293 an additional --artifact-dir Parameter would be awesome.

My tests assumes that project path's basename equals repository name. Can gitlab-runner clone project ~/projects/my-name to /build/project1/my-name?

mentioned in issue #2226

Is it possible to execute the whole pipeline defined in .gitlab-ci.yml not just a single job this way for testing purposes?

This works great - one small thing that tripped me up is that exec doesn't read docker pull policy from config.toml so you have to pass it in manually:

sudo gitlab-runner exec docker master_build --docker-pull-policy never

@Blackhex : check my issue #2226 regarding your request to run a full pipeline...

Hello, I have created a dummy proof-of-concept implementation of artifacts for exec docker subcommand. This is by no means intended as a merge request, but you can use it for your local development until more systematic approach and implementation is developed. You can find it in my fork and branch td-gitlab-ci-exec-artifacts.

Usage:

Build the gitlab-ci-multi-runner binary
Use it with docker executor

Example:

# .gitlab-ci.yml
stages:
  - s1
  - s2
  - s3

image: busybox

s1:
  stage: s1
  script:
  - cat foo || true

s2:
  stage: s2
  script:
  - echo `date` > foo
  artifacts:
    paths:
    - foo
    name: "custom"

s3:
  stage: s3
  script:
  - cat foo

$ gitlab-ci-multi-runner exec docker s1
...
$ cat foo || true # note that this is expected to fail!
cat: can't open 'foo': No such file or directory 
...

$ gitlab-ci-multi-runner exec docker s2
...
$ echo `date` > foo
...
Uploading artifacts...
foo: found 1 matching files

$ gitlab-ci-multi-runner exec docker s3
...
Downloading artifacts for s2 (0)...
$ cat foo
Sun Apr 30 18:41:49 UTC 2017
...

Limitations

for sure does not support artifacts:when nor artifacts:expire_in

Enjoy!

Has anyone (apart from @rpgillespie) experienced issues related to the config.toml and running local builds (using exec)? I'm currently having issues with the exec command ignoring both dns and extra_host settings.

In fact I've tested this under 3 debian based machines and it fails on all three. But it works under an Arch environment (using same docker and gitlab-runner versions), and that really misleads me.

mentioned in issue #2438

Has anyone succeeded in using gitlab-runner exec docker on Windows? I am having issue #1775 which seems to make this unusable on Windows, but it seems surprising to assume that this has never worked on Windows and this lack of support was not documented.

In my process, I have three steps: "docker build" > build project > test project
To be able to do it locally vs gitlab.com I have to use the DinD (Docker in Docker) approach.
Sample here: https://gitlab.com/Mizux/gitlabci

If someone manage to do it without DinD (i.e. privileged docker)...
Since it's a DinD, thus docker "image" are lost locally (i.e. "docker image ls" will be empty) (in fact you can retrieve it in the zip cache file but...)
gitlab could provide docker registry attached to your repo but i didn't manage to "simulate" it locally (i.e. use a local host registry instead of the gitlab.com/project one)

my 2 cents...

It seems this is going to be deprecated? https://gitlab.com/gitlab-org/gitlab-runner/commit/e2443e9f782b092951ac99431557e1bda9f8fd62 (as committed by @tmaczukin)

I hope not, or at least not without a better replacement. Being able to run CI locally is a feature that really sets Gitlab CI appart from other choices.

I agree with @Place1, please do not deprecate this feature, it's incredibly useful!

Update: https://gitlab.com/gitlab-org/gitlab-runner/issues/2797 (a replacement is being thought of, although hasn't started yet)

I'm concerned that this feature is deprecated before there's a replacement, so we're kind of in a limbo as to supported ways to run locally (super important for testing new configs, etc).

I would say, show your concerns on the new issue by upvoting. I also tweeted: https://twitter.com/gitlab/status/914493071533989888

gitlab-runner exec: easily test builds locally

How to use it

Using `exec` with Docker

Limitations:

Designs

Child items ...

Activity

Usage:

Example:

Limitations

Admin message

Admin message

gitlab-runner exec: easily test builds locally

How to use it

Using exec with Docker

Limitations:

Activity

Usage:

Example:

Limitations

Using `exec` with Docker