WIP Add support for testing the cookbook before building the package.

Adding support for testing the internal cookbook validity before building the package.

Reason(Pros):

• While almost all changes are tested before accepting them, it is extremely time consuming to test cookbook changes on all platforms. When chasing deadline this gets especially problematic.
• It has happened in the past that we release a package with broken internal cookbook.
• With the introduction of package server built packages are pushed to everyone as soon as they are built so even catching errors as soon as the package is built causes issues for end users.

Cons:

• Adds another step to already time consuming build
• Depending on yet another set of external resources(test-kitchen, kitchen-docker, docker)

Resolving the cons:

• For now, only enabled on branch builds which means that tagged packages build is not slower
• While depending on external resources can be time consuming, it is also time consuming fixing the issues after they are released to the ether

What has been also tried:

Instead of going down the test-kitchen route and testing only the cookbook, I've tried testing the whole package after it was built.

Pros of this approach:

• Testing the whole package installation, not only the cookbook

Cons of this approach

• Tests would have been executed at the very end of the build process so catching errors is again time consuming as it would take 2 long build processes to confirm if the package is working
• Still depends on external sources, mostly docker
• The installation itself was taking much(much) longer so there was no time justification for this approach anymore

Left to do

If this gets introduced as is, all omnibus builders need to have support for docker. This support will also need to enable non-sudo docker access which is always controversial.

Fixes #852 (closed)

@jacobvosmaer I really need some input on this from another angle, I feel strongly that we have a need for testing the convergence of the internal cookbook.

You probably want #!/bin/sh here (no exit needed).

@marin I am not sure about testing the cookbook before the package is built. It is supposed to be used after the package is built, isn't it? How would we get all the stuff that is built during the build ready for the package?

This sounds like something we could use the 'build pipeline' feature planned for GitLab CI for. Build the package, give it to a docker container that installs it and do tests in the container.

I am not sure about testing the cookbook before the package is built.

Look at it as rspec-like test for the cookbooks and recipes(the main ones). I would rather have the build fail if there are syntax or run errors than wait for the whole build to finish in order to find out.

How would we get all the stuff that is built during the build ready for the package?

That is why I created test/cookbooks to stub the calls to the things package provides.

This sounds like something we could use the 'build pipeline' feature planned for GitLab CI for.

Sure, but would be nice to not wait for that. It is also nice to receive this quicker feedback rather than wait for all compilation of the package to be done.

@marin @jacobvosmaer For GitLab Pivotal Release we did https://gitlab.com/gitlab-pivotal/gitlab-ee-bosh-release/tree/master/spec, with the use of docker image we could easily build the specs to test actual package installation and configuration change, allowing us to have behaviour testing.

Status changed to closed

mentioned in issue gitlab-com/operations#198 (closed)

Mentioned in issue #1695 (closed)