Out of memory on Geo testbed
2017-10-07_12:27:42.94899 sync postgresql: LOG: database system is shut down
2017-10-07_12:27:43.96071 sync postgresql: FATAL: could not map anonymous shared memory: Cannot allocate memory
2017-10-07_12:27:43.96080 sync postgresql: HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 8118992896 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2017-10-07_12:27:43.96084 sync postgresql: LOG: database system is shut downIn order of memory usage:
Tasks: 275 total, 2 running, 273 sleeping, 0 stopped, 0 zombie
%Cpu(s): 11.1 us, 2.2 sy, 0.0 ni, 86.3 id, 0.2 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 30880416 total, 6582504 free, 7134740 used, 17163172 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 7072252 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28219 git 20 0 2708348 880860 9016 S 0.7 2.9 57:40.53 sidekiq 5.0.4 gitlab-rails [0 of 25 busy]
5817 git 20 0 981856 644048 8996 S 0.0 2.1 0:09.77 unicorn worker[3] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5829 git 20 0 965336 642548 3164 S 0.0 2.1 0:07.33 unicorn worker[7] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5810 git 20 0 959328 635824 9064 S 0.0 2.1 0:08.38 unicorn worker[1] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5813 git 20 0 957280 626072 10400 S 0.0 2.0 0:11.61 unicorn worker[2] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5807 git 20 0 961376 625280 9068 S 0.0 2.0 0:10.97 unicorn worker[0] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5731 git 20 0 828112 538120 1876 S 0.0 1.7 19:28.46 unicorn master -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/c+
5820 git 20 0 785248 518736 8996 S 0.0 1.7 0:08.96 unicorn worker[4] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5823 git 20 0 701280 446340 10428 S 0.0 1.4 0:00.94 unicorn worker[5] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5832 git 20 0 692944 438292 2420 S 0.0 1.4 0:00.30 unicorn worker[8] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
5826 git 20 0 692944 435880 8 S 0.0 1.4 0:00.30 unicorn worker[6] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rail+
8326 influxdb 20 0 1556148 383196 7060 S 1.7 1.2 594:58.77 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
22802 git 20 0 313652 150260 12868 R 99.7 0.5 0:05.20 ruby /opt/gitlab/embedded/service/gitlab-rails/bin/geo_log_cursor
5587 gitlab-+ 20 0 4012804 105896 105260 S 0.0 0.3 0:00.45 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/geo-postgresql/data
5691 gitlab-+ 20 0 123564 94972 2500 S 0.7 0.3 7:07.52 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
5589 gitlab-+ 20 0 4012916 63544 62812 S 0.0 0.2 0:10.18 postgres: checkpointer process
28462 gitlab-+ 20 0 4024264 61920 59904 S 0.0 0.2 0:17.91 postgres: gitlab_geo gitlabhq_geo_production [local] idle
28452 gitlab-+ 20 0 4024264 51896 49900 S 0.0 0.2 0:07.46 postgres: gitlab_geo gitlabhq_geo_production [local] idle
5695 root 20 0 133984 51460 4004 S 0.0 0.2 15:30.16 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/remote_syslog --no-detach --debug-level DEBUG -c /var/opt/gitlab/re+
17695 gitlab-+ 20 0 4024268 50084 48036 S 0.0 0.2 0:00.18 postgres: gitlab_geo gitlabhq_geo_production [local] idle
14965 gitlab-+ 20 0 4024268 50028 47980 S 0.0 0.2 0:00.22 postgres: gitlab_geo gitlabhq_geo_production [local] idle
11336 gitlab-+ 20 0 4024264 49984 47936 S 0.0 0.2 0:00.15 postgres: gitlab_geo gitlabhq_geo_production [local] idle
12736 gitlab-+ 20 0 4024264 49692 47644 S 0.0 0.2 0:00.21 postgres: gitlab_geo gitlabhq_geo_production [local] idle
12778 gitlab-+ 20 0 4024264 49268 47220 S 0.0 0.2 0:00.14 postgres: gitlab_geo gitlabhq_geo_production [local] idle
716 root 20 0 84832 45944 42428 S 0.0 0.1 12:32.81 /lib/systemd/systemd-journald
5621 git 20 0 2573964 34428 0 S 0.3 0.1 0:36.39 bin/gitaly-ruby 5596 /tmp/gitaly-ruby096178330/socket
28471 gitlab-+ 20 0 4024264 33892 31872 S 0.0 0.1 0:17.94 postgres: gitlab_geo gitlabhq_geo_production [local] idle
28473 gitlab-+ 20 0 4024264 33888 31872 S 0.0 0.1 0:17.77 postgres: gitlab_geo gitlabhq_geo_production [local] idle
28467 gitlab-+ 20 0 4024264 33860 31844 S 0.0 0.1 0:17.83 postgres: gitlab_geo gitlabhq_geo_production [local] idle
28441 gitlab-+ 20 0 4024264 33844 31812 S 0.0 0.1 0:17.63 postgres: gitlab_geo gitlabhq_geo_production [local] idle
28450 gitlab-+ 20 0 4024264 33784 31768 S 0.0 0.1 0:17.90 postgres: gitlab_geo gitlabhq_geo_production [local] idle
28443 gitlab-+ 20 0 4024264 33772 31752 S 0.0 0.1 0:17.72 postgres: gitlab_geo gitlabhq_geo_production [local] idle
28446 gitlab-+ 20 0 4024264 33756 31740 S 0.0 0.1 0:17.90 postgres: gitlab_geo gitlabhq_geo_production [local] idle
5590 gitlab-+ 20 0 4012804 31936 31292 S 0.0 0.1 0:00.32 postgres: writer process