Merge branch 'feature/improve-async-pipeline-processing' into 'master'

Improve asynchronous pipeline processing

## What does this MR do?

This MR improves asynchronous processing of pipeline.

## Why was this MR needed?

It eliminates some race conditions and improves performance.

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing

## What are the relevant issue / merge request numbers?

Related merge request:  !6410  
Extracted from !6411

See merge request !6650