Increase number of database connections and/or pool size for prod/sync.geo
We're now processing the Geo sync at a greatly increased rate on sync.geo.gitlab.com - however, some sidekiq jobs are failing with this error:
2017-09-27_08:16:58.64797 sync sidekiq: 2017-09-27T08:16:58.646Z 1523 TID-osegsxdpo WARN: {"context":"Job raised exception","job":{"class":"Geo::ProjectSyncWorker","args":[1938772,"2017-09-27 08:16:53 +0000"],"retry":3,"queue":"geo","dead":false,"jid":"8ec578d776729ac5c5cc7f2d","created_at":1506500213.3983622,"enqueued_at":1506500213.4040353,"error_message":"could not obtain a database connection within 5.000 seconds (waited 5.060 seconds)","error_class":"ActiveRecord::ConnectionTimeoutError","failed_at":1506500218.5461242,"retry_count":0},"jobstr":"{\"class\":\"Geo::ProjectSyncWorker\",\"args\":[1938772,\"2017-09-27 08:16:53 +0000\"],\"retry\":3,\"queue\":\"geo\",\"dead\":false,\"jid\":\"8ec578d776729ac5c5cc7f2d\",\"created_at\":1506500213.3983622,\"enqueued_at\":1506500213.4040353}"}
This indicates that either the pool size in config/database{,_geo}.yml
is too low, or the postgresql database doesn't have enough connections, or both.
Can we increase them?
/cc @jarv