Add Debian (.deb) package signing to Omnibus

What

Add Debian package signing, via methodology describe in debsigs documentation https://gitlab.com/debsigs/debsigs

How

Addition of a sign_deb_file function to Packager::DEB, after create_deb_file. The essential concepts of what is required to sign a .deb with with a type: origin signature is delineated per the link to debsigs above. There is no current functionailty built into dpkg scripting akin to rpm --addsign. Since the .deb file format is simple, we extract the contents of the archvice (ar x), sign the concatenated (specifically ordered) contents, and then append the created signature to the archive (ar rc debfile _gpgorigin).

These steps could have been accomplished in pure Ruby with the addition of several modules (GPGME, libarchive) except for two concerns: age & maintenance, fakeroot requirements.

Tests have been added to attempt to cover the behavior correctly.

External Program Requirements

• gpg : This is already an existing requirement of Packager::RPM due to the use of rpmsign
• ar : Most systems that attempt to build Debian packages will have the ar command, and it has been confirmed that MacOS also has this utility.

Adding ar to the required tools compiled by Omnibus may be required.

Relates to https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2537

mentioned in merge request !6 (closed)

changed target branch from gitlab_omnibus to gitlab-omnibus-ad5d3f98

I've opted to shy away from GPGME Gem for the sake of reduced footprint, and least changes required to Omnibus itself. Omnibus already requires gpg by way of RPM support as it is.

The handle of gpg via shellout!() calls has been ... interesting. I need to programmatically send the passphrase, in a secure method, but not a logged/exposed manner. Originally, I was going to attempt this in a call, but that can get logged. Then an environment, but it turns out Omnibus might log that too.

Thankfully, though not easily found, the root of the Omnibus shellout! function is actually Mixlibs::shellout which supports :input that will be used as STDIN on the called process. This should allow the use of gpg --batch --passphrase-fd 0 which will cause gpg to fetch the passphrase from the supplied STDIN.

changed the description

I've found the issue with Debian 9: it seems that the inappropriate ioctl for device stems from the addition gpg-agent when installing gpg, which aliases out to gnupg2. Debian 8 includes gpnug out box (version 1.4.18), and has gpnupg2 available.

Distribution	Version	Installed	Available	Symlink with v2?
Debian 7	1.4.12	gnupg	gnupg2(2.0.19)	no symlink
Debian 8	1.4.18	gnupg	gnupg2(2.0.26)	no symlink
Debian 9	x	x	gpg => gnupg => gnupg2(2.1.18)	gpg2 -> gpg
Ubuntu 14.04	1.4.16	gnupg	gnupg2(2.0.21)	no symlink
Ubuntu 16.04	1.4.20	gnupg	gunpg2(2.1.11)	no symlink
SLES 12 SP2	2.0.24	gpg2	gpg2(2.0.24)	gpg -> gpg2
CentOS 6	2.0.14	gnupg2	gnupg2(2.0.14)	gpg -> gpg2
CentOS 7	2.0.22	gnupg2	gnupg2(2.0.22)	gpg -> gpg2
OpenSUSE 24.2	2.0.24	gpg2	gpg2(2.0.24)	gpg -> gpg2
Rasbpian 8	1.4.18	gnupg	gnupg2(2.0.26)	no symlink

Our problem has, so far, only been encountered on Debian 9, but in theory, could occur on any version using gnupg v2+. According to the man entires, supposedly, I would point out, that extra handling has been done in Omnibus codebase to handle the PTY to input the passphrase for rpm --addsign calls.

According to recent manual entries, the gpg-agent will always be required, but has only bitten us here.

--no-use-agent
       This is dummy option. gpg2 always requires the agent.

cc @gitlab-build-team @joshlambert @briann

I believe this can be bypassed by gpg --batch --no-tty --import. Tested on the gitlab-omnibus-builder:debian_9-0.0.6 image with and without the additional arguments for key import. Adding the additional arguments removed the gpg-agent prompting for the password.

https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/1253763

One down, one more to go.

:successkid: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/1253865

However, that ended up breaking distros with older gpg

mentioned in merge request omnibus-gitlab!1718 (closed)

mentioned in merge request omnibus-gitlab!922 (closed)

mentioned in issue omnibus-gitlab#1054 (closed)

I've identified the likely culprit, and that comes down to the function / requirement for --pinentry-mode function due to underlying differences in version & supported/required flags. I will add in a check for the requirement of that flag, based on if gpg|gpg2(see previous table) actually supports it.

https://dev.gitlab.org/gitlab/omnibus-gitlab/pipelines/55756

Now on to testing & documenting the output packages with Debian/Ubuntu flavors, from the repository.

unmarked as a Work In Progress

assigned to @marin

Steps for testing the .deb's signature (beyond the repository signature)

# mkdir /usr/share/debsig/keyrings/66D26543C0207D21/
-- download pubkey --
# gpg --no-default-keyring --keyring /usr/share/debsig/keyrings/66D26543C0
207D21/debsig.gpg --import gitlab-test-signing-331E7725DC169E52.pub.gpg
# cat /etc/debsig/policies/66D26543C0207D21/gitlab.pol
<?xml version="1.0"?>
<!DOCTYPE Policy SYSTEM "http://www.debian.org/debsig/1.0/policy.dtd">
<Policy xmlns="http://www.debian.org/debsig/1.0/">

  <Origin Name="GitLab Inc" id="66D26543C0207D21" Description="GitLab.com"/>

  <Selection>
    <Required Type="origin" File="debsig.gpg" id="66D26543C0207D21"/>
  </Selection>

   <Verification MinOptional="0">
    <Required Type="origin" File="debsig.gpg" id="66D26543C0207D21"/>
   </Verification>
</Policy>
# debsig-verify gitlab-ce_8.1.0+git.3216.1f52045-rc1.ce.0_amd64.deb
debsig: Verified package from `GitLab.com' (GitLab Inc)

mentioned in issue omnibus-gitlab#2537 (closed)

resolved all discussions

resolved all discussions

Merging after the discussion in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2537 reaches a conclusion.

resolved all discussions

merged

@WarheadsSE Merged. Please create a MR bumping the version of omnibus in omnibus-gitlab.

mentioned in merge request omnibus-gitlab!1771 (merged)