Skip to content

Add export project script & pipeline job

Matthias Käppler requested to merge 208100-collect-export-project-metrics into master

This MR extends our existing pipeline with a new export job, similarly to the import job we already have.

Job definitions

To improve fault isolation, I decided to run it in its own job, so I broke up the CI config to have a base job config describing the script and schedule, which is then inherited by both import-benchmark and export-benchmark jobs. I also decided to hard-code less values here and move them to the env, which means that once we merge this MR, we need to make sure these are all defined in the pipeline config of this project.

New export script

I added a new script bin/export-project. I ended up running into so much duplication tho that I decided to combine this with a refactor that extracts the following common concers that are now shared between the 2 scripts:

  • talking to the GL API (lib/gitlab_api.rb)
  • running and awaiting job results (lib/job.rb)
  • talking to Slack (lib/slack.rb)

Apart from that, the new logic is fairly simple, since less work is required to trigger an export; we only need to POST to the export endpoint and await the result, which contains a link to a web page where we can download the archive if needed.

Testing

I tested this againts my local env, both the new script as well as the much refactored import script. I tested Slack integration as well, and it all seemed to work fine. I don't know if there is a way to test the pipeline logic locally though, we might have to just land this and see.

Edited by Matthias Käppler

Merge request reports