Track long term performance changes of a branch

Description

Building a fast and responsive application is not a one time effort, it is an on-going journey. Over time, new features are added, bugs are fixed, and typical user behavior may change. In short, it's a moving target and requires diligence and a constant eye.

To help customers keep this watchful eye, we can provide tools to track the performance changes on a single branch, over time. For example, we could quickly display the performance trends on master, and provide insight as to whether progress is being made.

Proposal

With the ability to run reliable performance testing against a specific merge request (https://gitlab.com/gitlab-org/gitlab-ee/issues/3173), we have the ability to know the performance characteristics of a branch at a given point in time.

We can leverage this, to then track code performance in a repeatable and consistent fashion.

This data can then be presented in a few different ways:

• One is on a commit by commit basis, which provides a detailed look and is great for organizations using CD
• We could also display on a tag by tag basis, which would remove some noise during development, and focus specifically on released builds

The goal of this feature, versus the production dashboard (https://gitlab.com/gitlab-org/gitlab-ee/issues/3541), is to provide a look at the code performance in a sanitized environment over time. In short, are we making progress towards increasing the speed of our code or not?

Production is of course the ultimate test, but you could with rapidly increasing user counts, changes in scale and hardware, and other confounding factors. In this case, we are interesting in just showing the performance over time of the code base against a static set of tests.