Autoscale with AWS

Description

Runners in AWS is important for IAM AWS has huge market share and many organizations are quite AWS centric with AWS IAM everywhere. Runners in AWS can gain access to AWS APIs via Service Roles - a familiar and desired pattern.

Current option is slow. We have one option for installing multi-runner with auto scale in AWS - Docker Machine. BUT this subjects us slow start times of EC2 (better on DO). Current documentation describes a custom crafted process of setting up and configuring rather than IaC or cattle approach. K8S is great, but not every org is willing to do K8S until there is an AWS branded version with seamless IAM.

AWS CodeBuild. Fully managed, docker builds on demand. Custom images can be used, stock images allow DinD. Jenkins can use (via plugin of course ).

Proposal

Build in native support for AWS CodeBuild just as with docker-machine and k8s. The Jenkins plugin uses S3 but the buildspec and env vars can be set via API and start with fetching from GitLab.

A CodeBuild job runs in a project. The integration will need to create the CodeBuild Project - akin to registering a runner with a project? so we would want to parallel it and be ale to create multiple CodeBuld Projects (maybe in different AWS accounts). We can let users do this on their own or provide a Terraform plan / docker image that could be run early in the pipeline or a button that does it all magically.

The GitLab Project would be configured with the CodeBuild Project ARN?

The GitLab-CI job that has the tag (the arn?) would transform the job's definition to a buildspec format and start a build with the pipeline's env vars and the generated buildspec string. There would be the handling of artifacts added to the buildspec and maybe we just support DinD so the buildspec can get the creds to pull the job's image.

Some good ideas here. Perhaps @ayufan and @tmaczukin can help you here.

/cc: @bikebilly

Would it not be easier to integrate GitLab with Aws CodeBuild? Keep the pipeline information on GitLab, but just run the jobs on codebuild.

added feature proposal label

@rodrigo.avila A CodeBuild runner? Interesting for sure and even more so if I can spec the runner per job #32323 . I'm not sure it is easier or harder​ but I know CodePipeline does not support git tags and neither CodePipeline or CodeBuild support git submodules so we would need to make sure everything was provided via artifacts to the CodeBuild S3. We would also want to make sure everything could be handled from gitlab and a user doesn't have to separately got to aws to find out what is going on. Without GitLab integration, we can do it right now just by shipping a source bundle to an S3 bucket.

I'd be willing to switch this issue's focus to CodeBuild if we can have fine integration - even parse the job def to define the CodeBuild and have an way for CodeBuild to return results.

@rodrigo.avila have you used CodeBuild and GitLab-CI this way before? Any repos or pointers to share?

I would still prefer to focus on AWS Scaling Group as this is an easier solution for us to test.

Interestingly probably the easiest way nowadays for having auto-scaled GitLab Runner is:

1. docker-machine approach described in documentation,
2. kubernetes executor talking to auto-scaled kubernetes cluster.

Kubernetes I would say is the preferred approach as it gives a lot of flexibility and visibility into how Cluster is behaving.

We use ECS as our scheduler and it would be fantastic if we could use our existing cluster for running builds, although we are probably in the minority compared to your two existing auto-scaling options.

mentioned in issue #1421 (closed)

I would also like to have the builds happen in an ECS cluster. My group already has an ECS cluster running some low-demand services that could share those resources with builds. Especially with the ECS cluster being configured to auto scale itself. I don't particularly want end up supporting both an ECS cluster and a Kubernetes cluster.

AWS CodeBuild could work, that I'm not entirely familiar with the inner workings of this service. My biggest concern would be where the resulting artifacts end up. My company has dozens of VPC's and we cannot assume that users of GitLab would have access to my VPC to get their artifacts. I suppose that's the beauty of the gitlab-ci.yml file, the build results can get configured to be uploaded to anything. I just want to make sure if this the approach that it's not assumed GitLab users have access to the VPC the configured CodeBuild is in (assuming it's a shared runner across the whole instance).

Can we also get the customer label adding to this request?

@rodrigo.avila 's direction is right on. Now that I have more experience with CodeBuild, I think what we should do is add CodeBuild to multi-runner just as k8s.

Users of GitLab CI would be indifferent... other than CodeBuild runners would have the AWS env vars set and access to the metadata URL and VPC resources.

I would be able to id which runners to use on a job with the tags ...

FYI

CodeBuild now allows custom docker images with DinD: http://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html#sample-docker-custom-image-running

Jenkins has CodeBuild integration: http://docs.aws.amazon.com/codebuild/latest/userguide/jenkins-plugin.html

changed title from Autoscale multi-runner via AWS ECS to Autoscale with AWS

changed the description