Leverage Heroku CI in buildpacks for zero-configuration CI

Description

Heroku CI augments standard buildpacks with a couple CI-specific steps (bin/test-compile and bin/test). They have about 6 languages supported currently and I'm sure it will take a while for third-party buildpacks to support these new steps, but if it takes off, it could be an easy way to get people started with CI without having to configure anything themselves. We already leverage buildpacks for auto deploy, how about leveraging it for zero-configuration CI?

Note, Heroku CI also augments app.json with scripts.test-setup and scripts.test, and adds an environments:test way to declare addons needed during tests. We don't currently listen to app.json at all, but we could, if we wanted to have compatibility with Heroku CI for people who have gone beyond vanilla buildpack usage. app.json has a lot of the same goals as our .gitlab-ci.yml.

Proposal

• Update herokuish with latest buildpacks
• Add test commands to herokuish (Note there already is a test command, with a different purpose.)
  • Make sure handling of "no tests" is reasonable (probably should no-op succeed rather than error)
• Add test commands to our auto deploy image
• Add test commands to Auto Deploy template
• Add test commands to Auto DevOps template

Links / references

• Herokuish issue: https://github.com/gliderlabs/herokuish/issues/257
• Herokuish PR: https://github.com/gliderlabs/herokuish/pull/286

Test projects

So far successfully tested on next projects:

1. Clojure, no db -> https://gitlab.com/dzaporozhets/clj-hello-world
2. Rails, Postgres -> https://gitlab.com/dzaporozhets/rails-hello-world
3. Nodejs, Express -> https://gitlab.com/dzaporozhets/express-hello-world
4. Java, Spring -> https://gitlab.com/dzaporozhets/spring-hello-world
5. Scale, Play -> https://gitlab.com/dzaporozhets/play-hello-world

Build fails

1. Python, django -> https://gitlab.com/dzaporozhets/django-hello-world. Buildpack does not support test command
2. Ruby, no tests -> https://gitlab.com/dzaporozhets/sample-ruby-app/-/jobs/30607321

/cc @ayufan @dlap

made the issue confidential

Makes sense for me. Smart-deploys with smart-tests basically :)

mentioned in issue #28351 (moved)

made the issue visible to everyone

added idea-to-production label

Heroku CI is now GA. https://blog.heroku.com/heroku-ci-now-available

changed the description

/cc @dzaporozhets

mentioned in issue #32638 (moved)

changed the description

mentioned in issue #32829 (closed)

changed milestone to %Next 3-6 months

I am going to work on this in Q3

mentioned in issue #34777 (closed)

mentioned in issue #35547 (moved)

mentioned in issue #35712 (moved)

I start working on it tomorrow

assigned to @dzaporozhets

marked this issue as related to #32829 (closed)

changed milestone to %10.0

added Stretch label

We already leverage buildpacks for auto deploy,

@markpundsack could please point me to the place where we already use herokuish?

Add test commands to our auto deploy image

@markpundsack is it https://gitlab.com/gitlab-examples/kubernetes-deploy?

could please point me to the place where we already use herokuish?

@dzaporozhets https://gitlab.com/gitlab-examples/kubernetes-deploy/blob/master/build#L62-80

Currently, kubernetes-deploy is our auto deploy image, but we're in the process of removing it with https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/80, which @ayufan swears he'll finish for 9.5. :)

The idea is that we'll take all the custom scripts we put in the kubernetes-deploy Docker image, and shove it into the .gitlab-ci.yml template directly, so the logic is more transparent and editable by end-users.

@markpundsack not sure if huge .gitlab-ci.yml is good for new users. Having command deploy vs 20 lines of shell code is much easier to understand. Additionally, the more code you have - the easier it is to make a configuration mistake you hardly find.

@dzaporozhets It's an iteration; a step along a path. For example, this might be a future step.

For now, I'm arguing that 20 lines of shell script you can see is easier to understand than 20 lines of shell script buried in a Docker image. command deploy is easier to understand semantically, but nearly impossible to understand at a technical level, or modify when necessary.

@markpundsack I agree. It's just dropping all code into one file makes it impossible for me to visually parse it. https://gitlab.com/markpundsack/auto-deploy-mockup/blob/master/.gitlab-ci.yml looks good btw.

In meantime, I try to bring test command into herokuish https://github.com/gliderlabs/herokuish/pull/271

Awesome to see a PR already, @dzaporozhets!

@markpundsack I made some progress with herokuish. test command is working now. It means I can pass a repository to my herokuish fork image and it will run tests for me without any configuration. Tested on Ruby and Clojure so far.

If you want to try - you can do it with next code:

git clone git@gitlab.com:dzaporozhets/ruby-sinatra.git
docker run --rm -v "$(pwd)/ruby-sinatra:/tmp/app" registry.gitlab.com/dzaporozhets/herokuish /bin/herokuish buildpack test

Awesome @dzaporozhets! I'm on vacation, but maybe @bikebilly can try it out.

BTW, did you see any reason we'd want Herokuish to do test-compile separately from test? The buildpacks have them separate, but not really sure why. Maybe to retry tests without rebuilding...

I pushed a sample project with Ruby app and next .gitlab-ci.yml

image: docker:latest

services:
  - docker:dind

test:
  script:
    - docker run --rm -v "$(pwd):/tmp/app" registry.gitlab.com/dzaporozhets/herokuish /bin/herokuish buildpack test
  only:
    - branches

Link with green tests -> https://gitlab.com/dzaporozhets/ruyb-sinatra-zero-config-ci/-/jobs/28013206

BTW, did you see any reason we'd want Herokuish to do test-compile separately from test? The buildpacks have them separate, but not really sure why. Maybe to retry tests without rebuilding...

@markpundsack you mean Heroku. Probably because they cache build directory after test-compile before test when possible. In the end when you do git push heroku they may do single compile and use it for both deploy and test.

For now, I use both test-compile and test inside herokuish buildpack test. Because test without test-compile is useless until you cache everything installed by test-compile. I will see what Herokuish maintainers think about this approach. If they insist I will split the feature into 3 commands:

• buildpack test-compile
• buildpack test
• command combining both test-compile and test.

I see we do build and start separately for deployment here https://gitlab.com/gitlab-examples/kubernetes-deploy/blob/master/build#L62-80. I am not sure it makes sense for tests. You want to test often (every pushed commit) and there is only one test command execution vs deploying on multiple servers.

I raised a question about commands separation in PR https://github.com/gliderlabs/herokuish/pull/271#issuecomment-320254900

I created https://gitlab.com/gitlab-org/herokuish and will ship my changes there for now.

We will use our own docker image for zero-config CI until official herokuish repo has test feature.

mentioned in merge request herokuish!1 (closed)

mentioned in merge request gitlab-examples/kubernetes-deploy!21 (merged)

mentioned in issue #27255 (closed)

marked this issue as related to #27255 (closed)

marked the checklist item Update herokuish with latest buildpacks as completed

marked the checklist item Add test commands to herokuish (Note there already is a test command, with a different purpose.) as completed

changed the description

marked the checklist item Add test commands to our auto deploy image as completed

I will now test herokuish on several projects to detect possible fail cases

So far sucessfully tested on next projects:

1. Clojure, no db - https://gitlab.com/dzaporozhets/clj-hello-world
2. Rails, Postgres - https://gitlab.com/dzaporozhets/rails-hello-world

Auto CI already included in Auto-DevOps template https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/88

changed the description

@dzaporozhets Should we create a new issue to track switching to herokuish:latest when it ships?

mentioned in issue #37168 (closed)

marked this issue as related to #37168 (closed)

@markpundsack yep, created https://gitlab.com/gitlab-org/gitlab-ce/issues/37168

Few more projects sucessfully tests:

• Nodejs, Express -> https://gitlab.com/dzaporozhets/express-hello-world
• Java, Spring -> https://gitlab.com/dzaporozhets/spring-hello-world

Ruby app with no tests (and Rakefile) mark build as failed for auto test: https://gitlab.com/dzaporozhets/sample-ruby-app/merge_requests/1

changed the description

@bikebilly thanks, I updated the description.

@dzaporozhets if you test others add them there please :)

changed the description

So far I see next cases when build will fail:

1. buildpack is not detected (language not supported)
2. buildpack does not support test command (not all buildpacks have test command yet)
3. application has no tests and no test framework in dependencies.

cc @bikebilly

changed the description

@bikebilly anything else I can do related to this feature for 10.0?

marked the checklist item Add test commands to Auto DevOps template as completed

Closed by https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/88

closed