Allow overriding ENTRYPOINT from .gitlab-ci.yml

Currently, the only way to make use of a Docker image that sets ENTRYPOINT seems to be to create and publish another Docker image based off the original, but setting ENTRYPOINT to an empty array. (as discussed in gitlab-org/gitlab-ci-multi-runner!21) This becomes an issue when you have several third-party images you need to use (e.g., the 5 most recent releases of Node.js), especially if you also want to stay up to date with the latest versions of those images.

Instead, I'd like the ability to override ENTRYPOINT with an option in .gitlab-ci.yml. Similar functionality is already provided by docker-compose (using the entrypoint option), and docker run supports overriding ENTRYPOINT using the --entrypoint command-line option.

It looks like this is still relevant and quite important. /cc @tmaczukin @ayufan

Added feature proposal label

cc @nick.thomas

Another report: gitlab-org/gitlab-ce#22434

+1

I'm currently working on a build pipeline -- and I can see several more clones of this -- that takes the following path:

1. Build and publish container with fairly substantial python environment
2. Execute script(s) on that container to reconfigure deployment environment (e.g. database migrations)
3. Deploy container to cluster

The container uses a combination of ENTRYPOINT and CMD to provide default arguments and allow the deployment to pass/override arguments at runtime. I don't want to have to specify the entire command line on my deployments if I want to provide extra arguments and I don't want to build and publish multiple versions of the container when there is already a method (within Docker) for overriding ENTRYPOINT.

+1

Maybe 90% of official Docker images have custom entrypoint, this is so frustrating ;-(

We're also having this issue: https://github.com/Codeception/Codeception/pull/3739#issuecomment-261810192

i'll share my solution too, i had to build custom composer image to solve this issue:

having struggled with gitlab-ci-multi-runner#1421, it's fine to override ENTRYPOINT if you emulate the interface GitLab CI requires:

ENTRYPOINT ["/composer.sh"]

# cat composer.sh 
#!/bin/sh

case "$1" in
        *sh)
                exec "$@"
                exit 1
        ;;
esac

exec composer --ansi "$@"

It is really a show stopper, most of the images have custom entry points from maven to kubectl to node... We needed kubectl, and since it supports config from standard input, we could use a docker image and use the kubectl via docker run. But that was a lucky case, you can not do that with maven for example...

@ayufan @tmaczukin I think this is very important issue, and we should fix this problem. Is this a difficult change?

Would be really nice, if we could get an ETA for this, since the only current solution is to build a custom image.

@nick.thomas What do you think about it?

What about implement all Docker run parameters, in a generic way?

@ebuildy I would be very hesitant to implement all docker run parameters, as they could be used to escape the docker container. They should probably be implemented individually.

@grzesiek I'm afraid I don't really have the context on docker to make an informed call here :/

/cc @markpundsack

Don't want to be smartass, but I think fixing the entrypoint first and think about other options later would be a perfect Christmas gift for the people struggling with this.

Makes sense to me. What would the syntax in .gitlab-ci.yml might look like?

image: ruby:2.2 key=value

For example:

image: ruby:2.2 ENTRYPOINT=/bin/bash

@stanhu We were thinking about image: ruby:2.2 being a simplification of

image: 
  name: ruby:2.2
  entrypoint: bin/something

Just like we have something similar with environments

environment: production

environment:
  name: production
  external_url: example.gitlab.com
  # ...

where both variants are correct.

@grzesiek Sounds good to me!

Part of me really hates replicating all these pieces in docker compose (and kubernetes, and thousands of other places) to make yet another way to run Docker. I wish we could literally just use docker compose for this. Or worst case, make it syntax compatible.

Just throwing this out there, could we have a job specify a docker-compose.yml file to start?

@markpundsack Making it possible to embed docker compose YAML into .gitlab-ci.yml is very interesting concept. Unfortunately it probably would require much more work on the GitLab Runner side. What do you think @ayufan?

Just want to share some examples with docker-compose...

• It's basically working out-of-the-box: https://gitlab.com/t-munk/docker-yii2-app/blob/feature/build-gitlab.com/.gitlab-ci.yml
• Build: https://gitlab.com/t-munk/docker-yii2-app/pipelines/5066031

Currently it's a bit slow and can for sure be optimized...

We've also build a custom runner (requires custom VM, no dind) with docker-compose support.

I think @grzesiek is talking about a new executor "docker-compose" rather than execute shell commands, isnit?

@ebuildy Not sure yet how we would like to implement it.

Sure, I imagine this is a very big word to do ^^

Any progress on this? It's blocking a number of other issues :(

/cc @ayufan

mentioned in issue #1724

+1. Creating clones of docker images for entry point override is a waste of time.

changed milestone to %Backlog

added executordocker executordocker-machine labels

In GigLab 9.0 and Runner 9.0 we're changing API in the way that will make much easier to implement such change. After 9.0 will be released I'll look at this problem and probably implement it together with entrypoint for services and services aliases.

mentioned in merge request !21 (closed)

mentioned in issue #1641 (closed)

When in doubt, copy Kubernetes. They've thought through this stuff more than I (and maybe you?) ever will.

mentioned in issue #1501 (closed)

@markpundsack @bikebilly This issue problem seems to be important if we want to ship this direction issue https://gitlab.com/gitlab-org/gitlab-ce/issues/4044#note_28709005.

@grzesiek Yeah, perhaps it's time. :)

BTW, it doesn't seem like people should have to specify an override for the entry point in their .gitlab-ci.yml. For example, if you're trying to run:

codeclimate:
  image: codeclimate/codeclimate
  script: codeclimate analyze

but there's an entry point defined, it would seem logical to use

codeclimate:
  image: codeclimate/codeclimate
  script: analyze

But the problem is that we're not just passing the script as a command; we're explicitly telling it to run sh and there's no codeclimate sh command.

So don't we need to override the entry point for all images? e.g. rather than:

docker run <image> /bin/sh <script>

do:

docker run --entrypoint /bin/sh <image> <script>

Except, perhaps, not using a full path to sh. (I have no idea how the runner actually works, the above is meant to be semantically correct, not factually correct.)

@markpundsack yes, your analyzis is correct, runner invokes all commands expecting (posix) shell to be there.

see how i hacked my composer image (it's currently gitlab-ci-runner compatible, so hack is no longer needed, but still).

so indeed this is what is needed:

docker run --entrypoint /bin/sh <image> <script>

@nick.thomas Is there an easy solution here?

@markpundsack I've not used ENTRYPOINT in anger so far, but from what I know of the script-generation code, overriding the entrypoint for all images should be pretty easy - we expect it to be sh, and there's a flag to set it.

Supporting arbitrary interpreters, as in your second codeclimate example, is considerably harder. Right now we generate a large sh script based on the script directives, but also containing lots of important setup, and all that would get in the way of using codeclimate as the entrypoint.

I expect we'd have to add a new directive (args:?) to be used instead of script. The image would still lack the setup though.

To get that working, perhaps we could switch to building a custom image on top of the specified one... but we'd need to do this for every build.

@ayufan please schedule this issue for 9.3

I see it like this. We extend image: definition to support new syntax:

image:
  name: my-image
  entrypoint: /bin/sh

We should not overwrite unconditionally entrypoint.

@tmaczukin Could you do that? This would be generic mechanism for all images and services.

changed milestone to %9.3

The syntax @ayufan proposes is similar to what is requested in #1284 (closed), just with the entrypoint instead of the hostname.

@rabbitfang I just wanted write the same. We already have plan (moved from 9.2 to 9.3) to allow configuring a service alias. Work is ongoing in gitlab-ce!8578 and #1466 (closed). Having changes in GitLab CE/EE and support for it in Runner, extending the support on image: setting and adding entrypoint: parameter should be easy.

@bikebilly My understanding is that no one is depending on existing entrypoints being respected because we almost explicitly break them. We prefix the script with bin/sh or something similar, so unless your entrypoint was expecting to wrap bin/sh, it'll fail. It would be a different story if we were respecting the entry point and passing on the script unaltered. e.g. you have an ENTRYPOINT of bin/codeclimate and then have a script of analyze, with the effect being to run bin/codeclimate analyze. But we don't.

@ayufan In what circumstances would you need to specify an entrypoint other than /bin/sh? As @nick.thomas pointed out, we prepend some set commands to the script, so we depend on the entrypoint understanding at least some shell syntax. I suppose someone might prefer /bin/bash instead of /bin/sh, or even /bin/zsh or whatever. Or perhaps you installed sh in a different directory. Or perhaps you're running Docker on Windows, which we don't actually support yet, but could, and would likely require a different entry point. But those feel like edge cases unrelated to the crux of the current issue, which is that any image with an ENTRYPOINT defined just fails. And we can fix the primary issue by just ignoring any entrypoint set in the image.

@markpundsack

Yes, we are looking mostly at allowing you to overwrite it to /bin/sh, but also just because this mechanism will be generic to services you will be able to run service command with different params. For image: in almost all cases, you will want to use /bin/sh|bash.

@ayufan So what I'm suggesting is that rather than making a new keyword that lets you work around the problem, how about just ignoring all entrypoints (or forcing entrypoint to /bin/sh) so it works for (nearly) all cases by default. Then as a separate issue, consider adding an explicit entrypoint keyword, but as a much lower priority issue. i.e. wait until someone actually has one of the mythical use cases for custom entrypoint.

I'm afraid that by focusing on the keyword, you're still requiring people to know to override the entrypoint, and you're pushing the knowledge and work onto the user, when that can be avoided.

@markpundsack

Yeah, but in some cases, usage of entrypoint is just a good way to drop you into preconfigured shell. This is what happens with docker:git where when you are dropped into the shell you can effectively execute commands in that context before.

Making our entrypoint to be /bin/sh now is breaking change, and we should not do it. The problem is probably different and we should probably focus next on solving that problem is our usage of docker containers.

Maybe we look at executing default entrypoint, but with a custom command, as this is usually what you want to do, and also this is the workflow that you see in readmes when you have docker run. Not running default entrypoint usually forces you to replicate all this internal setup that usually happens in entrypoint, as it is put on purpose.

I wonder if we don't want to actually solve that problem differently, by executing a sequence of containers (with their default entrypoints, or overwritten ones) with commands instead of script:.

codeclimate:
  steps:
  - image: codeclimate # use `docker run` style of execution
    command: run-my-arguments
  - image: alpine # use current shell execution model
    script:
    - echo Hello World

Yeah, but in some cases, usage of entrypoint is just a good way to drop you into preconfigured shell. This is what happens with docker:git where when you are dropped into the shell you can effectively execute commands in that context before.

@ayufan Can you elaborate on this? I understand the value of entrypoints for CLI-based docker run usage, but my understanding is that with GitLab CI, any image with an entrypoint just breaks.

Making our entrypoint to be /bin/sh now is breaking change, and we should not do it.

What exactly would it break? Maybe I'm missing something, but I haven't seen anything that would break.

It would break images that depend on a custom entry point to start system services. (for example a virtual display driver in ci for testing with chrome).

That custom entry point script just passes all parameters to /bin/sh after launching the services - so you overwriting that would break it. Quite a few other containers use this way to start required services too. This is in conflict with setting everything to /bin/sh - so the only way to not break anything, and allow us to use images with a entrypoint that not passes commands to /bin/sh is to give us an option to define a custom entry point. This is the only solution which would work for all images.

Thanks for clarifying @PhilippHeuer!

mentioned in merge request !596 (merged)

assigned to @tmaczukin

changed milestone to %9.4

My team is currently looking to use the standard tools for CI/CD with GitLab and I'm doing some research to see if it meets our needs. I came across this issue and thought I would give our use case and the issue this would provide us.

This would be a problem for us if a user could override an ENTRYPOINT in the gitlab-ci.yml. Our use case is to provide shared runners to our teams that will perform common services. These runners are running on AWS EC2 instances that have permissions that we wouldn't want the users to be able to call directly. So if they could override the entrypoint the could potentially perform AWS calls to resources they shouldn't be able to access.

If this is pursued, I would think there should be a way to disable it through the runner/executor for docker.

It's the entrypoint of the docker container. Docker containers are isolated from the host and have no way to access/run commands on the host (AWS EC2).

Or are you somehow passing your access credentials to each container / have a custom container to access aws resources?

There is nothing special being done. We have Docker installed on the EC2 instance and are running the multi-runner on that instance. The containers use the EC2 instances Instance Profile as credentials. I've verified that scripts run in a container use the EC2 instance profile as the default credentials.

@gregory.mcdaniel interesting. You could mount some volumes with your shared runners that would give the runners access file such as creds but you would have to intentionally set this up. I am actually wanting dedicated runners with authorization to hit the AWS API.

What about network reachability from the runner to http://169.254.169.254/latest/meta-data/ that is reachable from the EC2 host - this is where the SDK gets its token from IAM Service Roles. Maybe you need to have shared runner config override this (and maybe passing this on is my solution too).

I'm not sure I'm completely follow. I'm not an expert with Docker but it is definitely able to hit the meta-data and use the instance profile. We don't use long lived credentials in AWS for security purposes so we couldn't use a credential file for the purpose of our tasks. Plus if we did it would have the same permissions, how would I be able to prevent the team from using them since they would still need mounted?

Thank you for sharing the experience - that the meta URL is reachable from the shared runner. This is good news for me. In my case the EC2 will only have the limited permissions that I want the build to have.

In you case, perhaps the runners need to be split on to different EC2 based on which service roles you want the builds to have access to? The runners can be configured to only accept jobs from registered projects.

Are you using ECS? I wonder if the runner can be a service task and then you can use IAM to give different service roles per runner directly. If you are doing something custom maybe you could adapt kube2iam

Anyway, prob hijacking this thread and there are other issues discussing this such as #2451 . I think the AWS perms is unique to how we run the runner in AWS and not whether or not we can override the entry point.

No problem. If we were going to go down the role of creating specific AWS resources to call, etc. We would probably go down one of two routes, either writing our own app and api to call from a service or writing a custom runner altogether.

We liked the current functionality of being about to specify an ENTRYPOINT for the docker image and throwing out the script, that way we control our common service and it's access to resources that we need to provide the service. We already have it configured so only certain jobs are accepted by the runners but if the users can override the entrypoint then they can do whatever they want on the runner.

Here is what we followed and liked about the Docker executor: https://docs.gitlab.com/runner/executors/docker.html#the-entrypoint

It provided that functionality and allowing the override would break this.

closed via merge request !596 (merged)

I’m having issues with this too but I don’t believe simply overriding the entrypoint is the solution. Take for instance my luacheck image. The entrypoint is the luacheck executable and the default CMD is ., so that people can run this container simply by mounting code to /data (because that’s the default WORKDIR).

What I would like to do on .gitlab-ci.yml is something like this:

lint:
  image: yangm97/luacheck
  tags:
    - docker
  stage: lint

Notice how there’s no script: I expect gitlab-ci to run the default one instead of forcing me to use this directive.

Should I open another issue?