Upgrade to Docker Distribution 2.6.1 broke AWS S3 uploads

After deploying 9.2RC3, users complained that pushes to the registry stopped working:

• https://gitlab.com/gitlab-org/gitlab-ce/issues/32458
• https://gitlab.com/gitlab-org/gitlab-ce/issues/32457

The registry logs show:

s3aws: NotImplemented: A header you provided implies functionality that is not implemented\n\tstatus code: 501

Once we reverted the /opt/gitlab/embedded/bin/registry binary from 2.6.1 to 2.4.1, things worked:

Other references: https://github.com/openshift/docker-distribution/blob/master/docs/storage-drivers/s3.md

/cc: @ayufan, @northrup, @rspeicher

If we had the full error message from AWS, we could probably tell which headers were inserted into the AWS request to get this message.

We suspect that we are either missing a field in the S3 config or are somehow inserting an extraneous header.

@MrChrisW Can you see what's going on here?

changed the description

FYI, I ran into a possibly related issue with GET redirecting not properly redirecting to S3: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11429

I think the problem is this: https://github.com/aws/aws-sdk-go/issues/1058 which was fixed in the go aws sdk here: https://github.com/aws/aws-sdk-go/pull/991

But current that change is only in docker distribution 2.7 which isn't released yet

2.6.1 is on a much older version

I propose we:

• Repro the problem on a test machine
• Fork the distribution, drop in a newer aws-sdk-go as they have already done in master: (https://github.com/docker/distribution/commit/44dfd1b76600b865725402f9c6c4278f3544e8b1, https://github.com/docker/distribution/pull/2154/files)
• Use the fork in omnibus, confirm it works on test machine
• Log the issue for a backport to 2.6.x in docker distribution,
• import the fork to the dev mirror
• merge omnibus registry with fork into master and pick to 9.2 stable

changed milestone to %9.2

cc\ @gitlab-build-team

assigned to @twk3

Confirmed that this is also happening on dev

I have forked and bumped aws-sdk-go dependency here for now: https://gitlab.com/twk3/docker-distribution/commit/dd544a82d93e8d39cec3d3d45117c3f486365589

Currently building a package to test if it fixes the problem.

Confirmed that fixes the problem

Here is a build against what is in master with the fix: https://dev.gitlab.org/gitlab/omnibus-gitlab/pipelines/49401

Great, thanks @twk3.

FYI, @selfup and @timothyandrew. This is important to fix. There is a workaround that we can apply on the production on-call if we deploy another RC without this fix.

mentioned in merge request !1563 (merged)

Logged upstream as: https://github.com/docker/distribution/issues/2278

And I've opened the MR for the fix here: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1563

closed via merge request !1563 (merged)

mentioned in commit 1c804dc4

mentioned in commit 35d8b153

mentioned in commit fd2d8f79

mentioned in issue gitlab-qa#49