WIP: Sign packages (with Rake task added)

This is intended to rebase and replace !922 (closed).

Stemming from https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2537, we decided that it would be cleaner and faster to implement full package signing than creating, transfering, and posting hashes of packages to a server outside of PackageCloud.

Problem

• Packages need signed.
• Omnibus supports signing .rpm files
• Omnibus does not support signing .deb files.

Path forward

RPM

For RPM files, the answer is simple. Enable it in Omnibus, and make necessary infrastructure and information available to dev in order to sign the packages. @marin had conveniently already done the work, but PackageCloud didn't support it at the time. Pull the changes from @marin's !922 (closed), rebase onto master, and try again.

DEB

For DEB files, we're on the short end of the stick. Currently, support is not present in Omnibus itself. A community member had done a significant source of the work, but had not submitted an MR as it was reliant on debsigs and they had a drawback in that they did not have a clean way to pass the signing key's passphrase to the called program. As a result of the issues, the omnibus community decided that revisiting the efforts with a mind to replace the debsigs calls with pure Ruby implementation was in order. Nothing progressed from that point.

For the time being, I have implemented a Rake task (repository:sign) that will take the packages and sign them prior to the upload to AWS. This is done by injecting a step in the Makefile prior to the sync step that uploads them to S3.

Thankfully, the .deb format is relatively simple in how it adds the signatures of the package, and we can use a simple series of calls to ar mixed with the GPGME Gem in order to sign.

Future

The work done by the Rake tasks should not be in the Rake task.

• We should get this logic moved into Omnibus, and allow the greater community to also have this capability.
• The added fact is that it will also reduce the complexity of our codebase here, which is always wanted.

cc @briann

Well, we'll have to move the key sync, as that breaks rubocop

mentioned in issue #2537 (closed)

Deprecating this for moving to directly into Omnibus @ https://gitlab.com/gitlab-org/omnibus

Some notes that I want to put here:

• GPGME was originally chosen for two reasons
• Ruby implementation including API calls to libgpgme, so no need to deal with shelling out to gpg, headaces
• GPGME is already being added to GitLab CE (LGPLv2.1+) for signed commits.
• Shelling out to gpg has distinct pitfalls, in how you manage to get the passphrase into play.
• Do you use PTY and catch the prompt?
• If you use --batch you can use the --passphrase* options, but are they portable, and will the logs expose any of the information?

Previous external work by another author, that was never MR'd to Omnibus upstream:

• https://github.com/chef/omnibus/issues/402
• https://github.com/SarahKowalik/omnibus/commit/72ca84f9611c6676f1893a300d60ac80bf991c91

Notations on the behaviors needed for signing DEB files:

• https://purplefloyd.wordpress.com/2009/02/05/signing-deb-packages/
• https://gitlab.com/debsigs/debsigs/blob/master/debsigs.txt#L103-124

MR for direct Omnibus addition coming soon.

Thanks to @marin for convincing me to do the direct implementation, based on the work I already have here.

closed

mentioned in merge request !1771 (merged)