Investigate how to speed up asset-precompile

Our asset precompile takes a long time to run, 786s in one of the latest builds.

We should research if we can find a way to speed this up.

The asset:precompile task should be smart enough to only compile assets that changed. However, we purge the whole gitlab-rails definition and thus loose the previous state.

I think we should look into adding a way of packing those assets as cache which would be reused in the next build.

Maybe splitting asset compilation into a separate definition could help?

/cc @gitlab-build-team

since the frontend assets are not dependent at all on the underlying system for which the omnibus packages are built, I think it would be far more efficient to compile the frontend assets one time and inject the result into each omnibus build. this would also ensure that the frontend assets will be identical for all omnibus distributions of the same GitLab version.

there also may be ways in which we can add some caching layers to webpack that could improve build times. some babel transformations applied to code that hasn't changed between versions might be re-usable, et cetera.

@marin @mikegreiling

a way of packing those assets as cache which would be reused in the next build this would also ensure that the frontend assets will be identical for all omnibus distributions of the same GitLab version.

I may be missing some context ... this issue seems like a perfect candidate for something along the lines of pushing a precompiled AUFS layer into a registry. (solve from bottom-up filesystem perspective rather than top-down tooling) -- would be fairly easy if the pre-compile sources and artifacts are in the same "container volume" rather than "host volume" on a Docker container.

thoughts?

@mikegreiling "Precompile-once, inject to all" seems to be a good idea. We can possibly add a prior stage in our pipeline which does this and pass the result as artifacts. The build logic can then be updated to check the existence of it before attempting a precompile.

I think we should look into adding a way of packing those assets as cache which would be reused in the next build.

The problem with CI cache is that it doesn't always guarantee "latest version of cache", or so I was told earlier. But since we use an S3 bucket for storing it, I don't think it applies to us. So, we can try to copy asset folder to a general location and add it to CI cache.

Omnibus's Git cache always has the risk of something - even a single character change - dirtying it. So I am not sure if making asset precompilation a separate software definition will be of much help.

"Precompile-once, inject to all" seems to be a good idea. We can possibly add a prior stage in our pipeline which does this and pass the result as artifacts.

I think that adds a lot more complexity and ends up duplicating work. We have to install all dependencies for the Rails app so we can run the rake task. I don't see that ultimately saving time.

So I am not sure if making asset precompilation a separate software definition will be of much help.

Depends on how we organise it. And yes, I think that if we copy that path and cache it, we might end up still saving time.

mentioned in issue #2454 (closed)

added Scheduled label

changed milestone to %10.0