Skip to content
Snippets Groups Projects
New issue look: Off

[Meta] Trim the package size

    • Open Issue created by Marin Jankovski @marin

    This is a holding issue with the topic of trimming unnecessary parts from the package.

    There are already a couple of issues touching this topic: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1537, https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1643, https://gitlab.com/gitlab-org/omnibus-gitlab/issues/988

    Since we are putting more things in the package with Prometheus, the time has come to add some resources towards removing unnecessary parts of the package.

    Prime candidates to save a few KB's are:

    • All software definitions cloned from git can have specs removed (some of them also docs). GitLab Rails still needs the doc directory
    • Gems with native extensions need to have a build cleanup after
    • Further investigate which software definitions can be removed as they are used as a build dependency and not a runtime dependency
    • More...

    cc @gitlab-build-team @stanhu

    Designs

    Child items ...

    • Activity

    • Marin Jankovski added P1 label

      added P1 label

    • Marin Jankovski changed milestone to %8.15

      changed milestone to %8.15

    • Marin Jankovski mentioned in merge request !1140 (closed)

      mentioned in merge request !1140 (closed)

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      For 8.14, the total uncompressed size: 1.1 GB

      Top consumers:

      |Path|Size (MB)|Notes| |------|-----|------|------| |/opt/gitlab/embedded/service/gem|313|Lots of gems take 1-2 MB, but rugged is 40 MB, gemojione is 15 MB, ace-rails-ap is 12 MB| |/opt/gitlab/embedded/lib/python3.4|122| |/opt/gitlab/embedded/lib/ruby/gems|87|

      A quick scan shows that the /opt/gitlab/embedded/lib/python3.4/test directory is 40.5 MB that can probably be cleaned.

    • Marin Jankovski assigned to @marin

      assigned to @marin

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      I will start on this.

      I will first start by going through the software definitions and check which libraries are runtime dependencies and which can be removed and added to the build images as build dependencies.

    • Marin Jankovski mentioned in merge request !1170 (merged)

      mentioned in merge request !1170 (merged)

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      I've submitted !1170 (merged) which shrinks the Ubuntu 14.04 nightly package from 290MB to 247MB.

    • Marin Jankovski mentioned in merge request !1173 (merged)

      mentioned in merge request !1173 (merged)

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      I've submitted https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1173 which shrinks the Ubuntu 14.04 package to 230MB.

    • Marin Jankovski mentioned in merge request !1174 (merged)

      mentioned in merge request !1174 (merged)

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      I've submitted https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1174 which shrinks the Ubuntu 14.04 package to 224MB.

      Edited by Marin Jankovski
    • Balasankar C mentioned in issue #1712 (closed)

      mentioned in issue #1712 (closed)

    • Marin Jankovski changed milestone to %8.16

      changed milestone to %8.16

    • Marin Jankovski mentioned in issue #1824 (closed)

      mentioned in issue #1824 (closed)

    • Marin Jankovski mentioned in issue #1823

      mentioned in issue #1823

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      We also have libruby-static.a which should only be necessary during build time, not on runtime. Checking whether it can be safely removed.

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      Didn't find a conclusive answer anywhere but I did find https://bugs.ruby-lang.org/issues/12845 . Linked to this I found https://github.com/ruby/ruby/commit/9bd24907851e390607d0d85365d0f00ed47a2a16#diff-3b3a6ec97232deb43dc14319a73872c1 which suggests that it is safe to remove it. It also means that with ruby 2.4 this file will no longer be included with a build.

    • Marin Jankovski mentioned in merge request !1193 (merged)

      mentioned in merge request !1193 (merged)

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      I've submitted !1193 (merged) which shrinks the Ubuntu 14.04 package down to 220MB.

    • username-removed-163557
      username-removed-163557 @glensc ·
      Contributor

      got link from 8.16 release blog, i noticed new messages appearing with gitlab-ctl reconfigure:

      root@soa:/opt/gitlab/embedded/lib/ruby/gems/2.3.0/specifications# gitlab-ctl reconfigure
Ignoring RedCloth-4.3.2 because its extensions are not built.  Try: gem pristine RedCloth --version 4.3.2
Ignoring allocations-1.0.5 because its extensions are not built.  Try: gem pristine allocations --version 1.0.5
Ignoring bcrypt-3.1.11 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.11
Ignoring charlock_holmes-0.7.3 because its extensions are not built.  Try: gem pristine charlock_holmes --version 0.7.3
Ignoring escape_utils-1.1.1 because its extensions are not built.  Try: gem pristine escape_utils --version 1.1.1
Ignoring ffi-1.9.17 because its extensions are not built.  Try: gem pristine ffi --version 1.9.17
Ignoring ffi-1.9.10 because its extensions are not built.  Try: gem pristine ffi --version 1.9.10
Ignoring ffi-yajl-2.3.0 because its extensions are not built.  Try: gem pristine ffi-yajl --version 2.3.0
Ignoring hamlit-2.6.1 because its extensions are not built.  Try: gem pristine hamlit --version 2.6.1
Ignoring http_parser.rb-0.6.0 because its extensions are not built.  Try: gem pristine http_parser.rb --version 0.6.0
Ignoring kgio-2.10.0 because its extensions are not built.  Try: gem pristine kgio --version 2.10.0
Ignoring libyajl2-1.2.0 because its extensions are not built.  Try: gem pristine libyajl2 --version 1.2.0
Ignoring RedCloth-4.3.2 because its extensions are not built.  Try: gem pristine RedCloth --version 4.3.2
Ignoring allocations-1.0.5 because its extensions are not built.  Try: gem pristine allocations --version 1.0.5
Ignoring bcrypt-3.1.11 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.11
Ignoring charlock_holmes-0.7.3 because its extensions are not built.  Try: gem pristine charlock_holmes --version 0.7.3
Ignoring ffi-yajl-2.3.0 because its extensions are not built.  Try: gem pristine ffi-yajl --version 2.3.0
Ignoring libyajl2-1.2.0 because its extensions are not built.  Try: gem pristine libyajl2 --version 1.2.0
Ignoring ffi-1.9.17 because its extensions are not built.  Try: gem pristine ffi --version 1.9.17
Ignoring ffi-1.9.10 because its extensions are not built.  Try: gem pristine ffi --version 1.9.10
Ignoring ffi-1.9.17 because its extensions are not built.  Try: gem pristine ffi --version 1.9.17
...

      while i like that it does not attempt to build extensions again (some performance boost), this may be confusing for some users.

      ps: two ffi versions installed is also weird, i wonder how they don't conflict, ruby can't load afaik two versions of same gem at the same time like nodejs can.

    • Gabriel Mazetto
      Gabriel Mazetto @brodock ·
      Maintainer

      @glensc probably a cache issue (gems were in cache before package build). @marin shall we execute bundle clean after a bundle install?

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      @glensc Thanks for reporting, interesting that I don't see that when I run reconfigure. Do you get that when you run reconfigure or in some other log?

    • Marin Jankovski mentioned in issue #1884 (closed)

      mentioned in issue #1884 (closed)

    • username-removed-163557
      username-removed-163557 @glensc ·
      Contributor

      @marin not sure what you mean, but those are emitted with any other bundler related commands as well:

      root@soa:/# gitlab-rake 
Ignoring RedCloth-4.3.2 because its extensions are not built.  Try: gem pristine RedCloth --version 4.3.2
Ignoring allocations-1.0.5 because its extensions are not built.  Try: gem pristine allocations --version 1.0.5
Ignoring bcrypt-3.1.11 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.11
Ignoring charlock_holmes-0.7.3 because its extensions are not built.  Try: gem pristine charlock_holmes --version 0.7.3
Ignoring RedCloth-4.3.2 because its extensions are not built.  Try: gem pristine RedCloth --version 4.3.2
Ignoring allocations-1.0.5 because its extensions are not built.  Try: gem pristine allocations --version 1.0.5
Ignoring bcrypt-3.1.11 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.11
Ignoring charlock_holmes-0.7.3 because its extensions are not built.  Try: gem pristine charlock_holmes --version 0.7.3
D, [2017-01-23T18:06:47.250945 #28045] DEBUG -- sentry: ** [Raven] cannot load such file -- haml_lint/rake_task excluded from capture due to environment or should_capture callback
rake aborted!
LoadError: cannot load such file -- haml_lint/rake_task
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/haml-lint.rake:2:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/Rakefile:10:in `<top (required)>'
(See full trace by running task with --trace)
D, [2017-01-23T18:06:47.355735 #28033] DEBUG -- sentry: ** [Raven] ["rake", "brakeman"] failed! excluded from capture due to environment or should_capture callback
rake aborted!
["rake", "brakeman"] failed!
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/test.rake:13:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/test.rake:12:in `each'  
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/test.rake:12:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/test.rake:5:in `block in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:test
(See full trace by running task with --trace)
root@soa:/#

      note the above failure is because --without tests version of the gitlab-ce package was produced. ps can you point to MR where actual size trimming changes landed for 8.16?

    • username-removed-163557
      username-removed-163557 @glensc ·
      Contributor

      i've solved in my own gitlab-ce packaging by defaulting -T if no args passed for gitlab rake: https://github.com/pld-linux/gitlab-ce/commit/f7036eeed0efea688cbcf576650c654aa2e0e398

    • Marin Jankovski
      Marin Jankovski @marin ·
      Author Maintainer

      I don't see any of those errors when I run any of the commands. But apprently you are doing some custom packaging of your own so that could be the issue. All the MR's are referenced in this issue.

    • username-removed-163557
      username-removed-163557 @glensc ·
      Contributor

      @marin seems 8.16.1 fixed something. i checked the deb and indeed 8.16.0 had duplicate gem installs, 8.16.1 has just one ffi gem, while 8.16.0 had two. so with 8.16.1 gitlab-ctl reconfigure does not complain about gems, but gitlab-rake still does. i'm sure removing /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems dir has something to do with that, i.e that gem extensions are now installed only to /opt/gitlab/embedded/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0 not both places.

      however, gitlab-rake without arg invoking itself twice once for production then for test and then failng is happening in 8.15 as well, so filing new issue about that

    • username-removed-163557 mentioned in merge request !1254

      mentioned in merge request !1254

    • username-removed-163557 mentioned in issue #1901

      mentioned in issue #1901

    • Douwe Maan changed milestone to %9.0

      changed milestone to %9.0

    • Marin Jankovski changed milestone to %9.3

      changed milestone to %9.3

    • username-removed-100770
      username-removed-100770 @tnir ·
      Contributor

      Just repeated for 9.3 release to update @stanhu 's excellent work above:

      For 9.3 (gitlab-ce_9.3.6-ce.0_amd64.deb 339MB), the total uncompressed size: 1.2 GB

      Top consumers:

      Path Size (MB) Notes
      opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems 371 Lots of gems take 1-2 MB, but rugged is 46 MB (including libgit2 30MB), grpc is 30MB (including src 20MB), chef is 17MB, gemojione is 16MB
      opt/gitlab/embedded/service/gitlab-rails 151 public/assets is 84MB (including public/assets/webpack 50MB)
      opt/gitlab/embedded/service/mattermost 137 webapp/dist/files is 15MB
      opt/gitlab/embedded/service/gitlab-shell 55
      opt/gitlab/embedded/lib/python3.4 53
      opt/gitlab/embedded/share/icu/57.1 35
    • username-removed-163557
      username-removed-163557 @glensc ·
      Contributor

      in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1643 i referenced script i use to clean vendor for pld linux distro packaging

    • username-removed-100770 mentioned in issue #2585

      mentioned in issue #2585

    • Marin Jankovski removed milestone

      removed milestone

    • Marin Jankovski added Backlog label

      added Backlog label

    • Marin Jankovski removed assignee

      removed assignee

    • Balasankar C marked this issue as related to #2642

      marked this issue as related to #2642

    • Marin Jankovski mentioned in commit d8aa7e2c

      mentioned in commit d8aa7e2c

    Please register or sign in to reply