Slow startup & Huge CPU usage

@ArianeLu Please could you provide information about the problematic processes?

@markglenfletcher Run sudo gitlab-ctl restart

+1

i visit "/dashboard/issues", then choose the label. gitlab will eat too many cpu resource. like this:

sometimes the kswapd0 process will eat 99% cpu. the free output:

          total        used        free      shared  buff/cache   available

Mem: 2048248 1896360 72648 39168 79240 7960

Swap: 3145724 269504 2876220

and sometimes the postgres process eats 39% cpu.

the hardware config is: 1 cpu, 2G RAM, 2G swap

i only install gitlab on this server

We are in the process of revising the hardware requirements:

https://gitlab.com/gitlab-org/gitlab-ce/issues/27833

Just a data point. On a VM with 2 cores, 4GB RAM with ruby 2.3.4p301 and after a bit of file cache warm up, on v9.1.0 ( c3f0f143 )

after monkey patching require:

root@git:/home/git/gitlab# bin/rails console production                                                                                                                              [78/9459]
0.6849976740777493 bundler/setup
0.14603838697075844 active_support
0.10097765550017357 yaml
0.1064574271440506 active_support/core_ext/time
0.13443092443048954 active_support/time
0.14214144181460142 active_support/core_ext/hash/conversions
0.18809638544917107 active_support/core_ext/object/conversions
0.2546364860609174 active_support/core_ext/object
0.26308039482682943 rails/configuration
0.2762715667486191 rails/railtie
0.28406856674700975 rails/engine
0.40706746093928814 rails/application
0.6443128129467368 rails
0.15627573430538177 arel
0.1320050274953246 active_record/connection_adapters/abstract_adapter
0.3252107501029968 active_record
0.103883589617908 action_controller
0.1390782529488206 action_controller/railtie
0.4698595991358161 active_record/railtie
0.10095592960715294 concurrent/configuration
0.26991893257945776 concurrent
0.2769208671525121 sprockets/manifest
0.4330629436299205 sprockets
0.18891711067408323 nokogiri
0.2197495773434639 loofah
0.23294059839099646 rails-html-sanitizer
0.23562453128397465 action_view/helpers/sanitize_helper
0.23933506198227406 action_view/helpers/text_helper
0.25142063945531845 action_view/helpers/form_tag_helper
0.2612722460180521 action_view/helpers/form_helper
0.344114163890481 action_view/helpers
0.3472602516412735 sprockets/rails/context
0.8079922934994102 sprockets/railtie
1.9863380985334516 rails/all
0.11924577597528696 devise/rails/routes
0.12257525976747274 devise/rails
0.10378104727715254 bindata
0.11052167974412441 json/jwe
0.18630370311439037 json/jwt
0.11145874299108982 oauth2
0.14903715904802084 omniauth/strategies/oauth2
0.15092411916702986 omniauth-oauth2
0.1527448082342744 omniauth/strategies/auth0
0.1005856217816472 onelogin/ruby-saml/saml_message
0.1879465188831091 onelogin/ruby-saml
0.1885893363505602 ruby-saml
0.1909448355436325 omniauth/strategies/saml
0.15949796605855227 httpclient
0.20863566920161247 active_record/base
0.21070900466293097 /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb
0.22346592973917723 attr_encrypted                                                                                                                                                   [27/9459]
0.24364088848233223 devise_two_factor/models/two_factor_authenticatable
0.24769986886531115 devise_two_factor/models
0.11037989240139723 omniauth-ldap/adaptor
0.2255648747086525 mime/types
0.3194393524900079 linguist/blob_helper
3.211084270849824 linguist/language
0.1674419203773141 virtus
0.14264837838709354 hamlit/engine
0.13172418531030416 fog/core
0.13909394945949316 fog/aws
0.24535205122083426 carrierwave/storage
0.10024033673107624 sass/script
0.2947955094277859 sass/engine
0.3500557681545615 sass
0.352139794267714 sass/rails/helpers
0.36463931761682034 sass/rails
0.13561067916452885 action_controller/base
0.2280623121187091 gon/helpers
0.1047702906653285 action_controller/test_case
0.10632121842354536 gon/spec_helpers
0.1845730049535632 rake
0.18735326081514359 raven/integrations/rake
0.24960803240537643 raven
11.296308410353959 /home/git/gitlab/config/application
0.17141323629766703 active_record/connection_adapters/postgresql_adapter
0.573069741949439 /home/git/gitlab/lib/gitlab/metrics
0.10949910711497068 mail/field
0.2300125639885664 mail
0.2787422416731715 action_mailer/base
0.11144427582621574 rdoc/markdown
0.13906196877360344 rdoc/text
0.1489139273762703 rdoc/markup/to_html
0.1535168346017599 github/markup/rdoc
0.2074959110468626 github/markup
0.2926411144435406 gollum-lib
0.20782167371362448 /home/git/gitlab/lib/api/v3/notes
0.17943862546235323 /home/git/gitlab/lib/api/pipelines
0.41490440629422665 /home/git/gitlab/lib/api/services
1.8353657918050885 /home/git/gitlab/lib/api/api
0.1744550634175539 /home/git/gitlab/app/models/user
0.1201958479359746 /home/git/gitlab/app/controllers/application_controller
0.13012953288853168 /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/devise-4.2.0/app/controllers/devise_controller
0.1432049060240388 /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/devise-4.2.0/app/controllers/devise/confirmations_controller
0.20125990360975266 /home/git/gitlab/app/models/project
0.10604811832308769 /home/git/gitlab/app/models/project_services/builds_email_service
0.12052568420767784 /home/git/gitlab/lib/generators/rails/post_deployment_migration/post_deployment_migration_generator

12.803566852584481 /home/git/gitlab/config/environment.rb
Loading production environment (Rails 4.2.8)
...

root@git:/home/git/gitlab# echo "exit" | time bin/rails console production
Loading production environment (Rails 4.2.8)
Switch to inspect mode.
exit
21.91user 3.51system 0:25.92elapsed 98%CPU (0avgtext+0avgdata 310252maxresident)k
0inputs+8outputs (0major+138049minor)pagefaults 0swaps

So every little gem takes ~200 ms, and the whole rails startup takes ~20 sec.

So the whole gitlab restart easily takes about half a minute. Which feels slow, but actually not completely insanely unreasonable.

root@git:/home/git/gitlab# time systemctl restart gitlab.service 

real    0m31.950s
user    0m0.004s
sys     0m0.000s

moved to gitlab#9190

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

Slow startup & Huge CPU usage

Summary

Hardware:

Resource Usage

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Results of GitLab environment info

System information

GitLab information

GitLab Shell

Designs

Child items ...

Activity

Frequently Asked Questions

Admin message

Admin message

Slow startup & Huge CPU usage

Summary

Hardware:

Resource Usage

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Results of GitLab environment info

System information

GitLab information

GitLab Shell

Activity

Frequently Asked Questions