Issues with postfix emailing - Omnibus
Hi. I have postfix installed, using Internet Site. It uses the machines IP, localhost for settings: System mail name: example.com
Root and postmaster mail recipient: <admin_user_name>
Other destinations for mail: server1.example.com, example.com, localhost.example.com, localhost
Force synchronous updates on mail queue?: No
Local networks: 127.0.0.0/8
Gitlab.rb has been configured to use "gitlab@" machine's IP address. My previous installation was on DigitalOcean's one-click install and I had no issues with emailing. But I've set the Omnibus insallation myself and mail is giving me trouble.
Looking at Sidekiq logs I'm getting these errors and failed jobs it seems:
2016-01-27_11:36:54.97981 2016-01-27T11:36:54.979Z 3104 TID-2kfcg ActionMailer::DeliveryJob JID-9d9f4d75bcda2d9ea067d520 INFO: start
2016-01-27_11:36:54.98736 2016-01-27T11:36:54.987Z 3104 TID-rfujo ActionMailer::DeliveryJob JID-aea95e9dcad19805cbfcd5e5 INFO: start
2016-01-27_11:36:55.06867 2016-01-27T11:36:55.068Z 3104 TID-2mdnk ProjectServiceWorker JID-998755551d025a6f7f7f05b4 INFO: start
2016-01-27_11:36:55.44108 2016-01-27T11:36:55.441Z 3104 TID-2kfcg ActionMailer::DeliveryJob JID-9d9f4d75bcda2d9ea067d520 INFO: fail: 0.461 sec
2016-01-27_11:36:55.44122 2016-01-27T11:36:55.441Z 3104 TID-2kfcg WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"ActionMailer::DeliveryJob", "queue"=>"mailers", "args"=>[{"job_class"=>"ActionMailer::DeliveryJob", "job_id"=>"0047236e-c5d4-41b7-b9ac-7ca86681683a", "queue_name"=>"mailers", "arguments"=>["Notify", "note_issue_email", "deliver_now", 1, 2540], "locale"=>"en"}], "retry"=>true, "jid"=>"9d9f4d75bcda2d9ea067d520", "created_at"=>1453894614.9749062, "enqueued_at"=>1453894614.975, "error_message"=>"Circular dependency detected while autoloading constant Gitlab::IncomingEmail", "error_class"=>"RuntimeError", "failed_at"=>1453894615.440479, "retry_count"=>0}
2016-01-27_11:36:55.44125 2016-01-27T11:36:55.441Z 3104 TID-2kfcg WARN: RuntimeError: Circular dependency detected while autoloading constant Gitlab::IncomingEmail
2016-01-27_11:36:55.44130 2016-01-27T11:36:55.441Z 3104 TID-2kfcg WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:492:in `load_missing_constant'
2016-01-27_11:36:55.44131 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:184:in `const_missing'
2016-01-27_11:36:55.44131 /opt/gitlab/embedded/service/gitlab-rails/app/mailers/notify.rb:107:in `mail_thread'
2016-01-27_11:36:55.44131 /opt/gitlab/embedded/service/gitlab-rails/app/mailers/notify.rb:144:in `mail_answer_thread'
2016-01-27_11:36:55.44131 /opt/gitlab/embedded/service/gitlab-rails/app/mailers/emails/notes.rb:20:in `note_issue_email'
2016-01-27_11:36:55.44132 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:198:in `process_action'
2016-01-27_11:36:55.44132 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
2016-01-27_11:36:55.44132 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
2016-01-27_11:36:55.44132 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
2016-01-27_11:36:55.44133 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
2016-01-27_11:36:55.44133 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
2016-01-27_11:36:55.44133 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in `process'
2016-01-27_11:36:55.44133 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in `process'
2016-01-27_11:36:55.44133 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:596:in `block in process'
2016-01-27_11:36:55.44134 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
2016-01-27_11:36:55.44134 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2016-01-27_11:36:55.44134 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
2016-01-27_11:36:55.44134 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:593:in `process'
2016-01-27_11:36:55.44135 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:584:in `initialize'
2016-01-27_11:36:55.44135 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:25:in `new'
2016-01-27_11:36:55.44135 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:25:in `__getobj__'
2016-01-27_11:36:55.44135 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:34:in `message'
2016-01-27_11:36:55.44135 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:85:in `deliver_now'
2016-01-27_11:36:55.44136 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/delivery_job.rb:10:in `perform'
2016-01-27_11:36:55.44136 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:32:in `block in perform_now'
2016-01-27_11:36:55.44136 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
2016-01-27_11:36:55.44136 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
2016-01-27_11:36:55.44137 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
2016-01-27_11:36:55.44137 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:36:55.44138 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:36:55.44138 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
2016-01-27_11:36:55.44138 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
2016-01-27_11:36:55.44138 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
2016-01-27_11:36:55.44839 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'
2016-01-27_11:36:55.44839 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
2016-01-27_11:36:55.44840 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
2016-01-27_11:36:55.44840 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
2016-01-27_11:36:55.44841 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
2016-01-27_11:36:55.44841 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
2016-01-27_11:36:55.44841 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
2016-01-27_11:36:55.44842 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
2016-01-27_11:36:55.44842 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:36:55.44843 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:36:55.44843 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
2016-01-27_11:36:55.44843 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
2016-01-27_11:36:55.44844 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
2016-01-27_11:36:55.44844 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `call'
2016-01-27_11:36:55.44844 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
2016-01-27_11:36:55.44845 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
2016-01-27_11:36:55.44845 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2016-01-27_11:36:55.44846 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
2016-01-27_11:36:55.44846 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
2016-01-27_11:36:55.44846 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `block in tag_logger'
2016-01-27_11:36:55.44847 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
2016-01-27_11:36:55.44847 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:26:in `tagged'
2016-01-27_11:36:55.44849 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `tagged'
2016-01-27_11:36:55.44849 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `tag_logger'
2016-01-27_11:36:55.44849 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
2016-01-27_11:36:55.44850 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
2016-01-27_11:36:55.44850 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
2016-01-27_11:36:55.44851 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
2016-01-27_11:36:55.44851 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
2016-01-27_11:36:55.44851 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
2016-01-27_11:36:55.44852 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
2016-01-27_11:36:55.44852 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:36:55.44852 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:36:55.44857 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
2016-01-27_11:36:55.44857 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_perform_callbacks'
2016-01-27_11:36:55.44858 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
2016-01-27_11:36:55.44858 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:31:in `perform_now'
2016-01-27_11:36:55.44859 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:21:in `execute'
2016-01-27_11:36:55.44859 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
2016-01-27_11:36:55.44859 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
2016-01-27_11:36:55.44860 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
2016-01-27_11:36:55.44860 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
2016-01-27_11:36:55.44861 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
2016-01-27_11:36:55.44862 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:36:55.44864 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
2016-01-27_11:36:55.44864 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:36:55.44865 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2016-01-27_11:36:55.44865 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:36:55.44865 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2016-01-27_11:36:55.44866 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in `with_context'
2016-01-27_11:36:55.44866 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2016-01-27_11:36:55.44867 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:36:55.44867 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `call'
2016-01-27_11:36:55.44867 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'
2016-01-27_11:36:55.44867 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in `block in process'
2016-01-27_11:36:55.44867 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in `stats'
2016-01-27_11:36:55.44867 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in `process'
2016-01-27_11:36:55.44868 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in `process_one'
2016-01-27_11:36:55.44868 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in `run'
2016-01-27_11:36:55.44868 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in `watchdog'
2016-01-27_11:36:55.44868 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in `block in safe_thread'
2016-01-27_11:36:55.73015 2016-01-27T11:36:55.730Z 3104 TID-rfujo ActionMailer::DeliveryJob JID-aea95e9dcad19805cbfcd5e5 INFO: done: 0.743 sec
2016-01-27_11:36:55.98100 2016-01-27T11:36:55.980Z 3104 TID-2mdnk ProjectServiceWorker JID-998755551d025a6f7f7f05b4 INFO: done: 0.912 sec
2016-01-27_11:37:13.72314 2016-01-27T11:37:13.723Z 3104 TID-2mdnk ActionMailer::DeliveryJob JID-9d9f4d75bcda2d9ea067d520 INFO: start
2016-01-27_11:37:13.83286 2016-01-27T11:37:13.832Z 3104 TID-2mdnk ActionMailer::DeliveryJob JID-9d9f4d75bcda2d9ea067d520 INFO: done: 0.11 sec
2016-01-27_11:55:36.69562 2016-01-27T11:55:36.695Z 3104 TID-rfujo ActionMailer::DeliveryJob JID-0251a302be2d1eccec0c9fef INFO: fail: 0.012 sec
2016-01-27_11:55:36.69580 2016-01-27T11:55:36.695Z 3104 TID-rfujo WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"ActionMailer::DeliveryJob", "queue"=>"mailers", "args"=>[{"job_class"=>"ActionMailer::DeliveryJob", "job_id"=>"2e2711e5-4303-47cd-8e79-5fa1dad2c1c7", "queue_name"=>"mailers", "arguments"=>["Notify", "project_access_granted_email", "deliver_now", 45], "locale"=>"en"}], "retry"=>true, "jid"=>"0251a302be2d1eccec0c9fef", "created_at"=>1453895736.6829226, "enqueued_at"=>1453895736.6830623, "error_message"=>"Couldn't find ProjectMember with 'id'=45 [WHERE \"members\".\"type\" IN ('ProjectMember') AND \"members\".\"source_type\" = ?]", "error_class"=>"ActiveRecord::RecordNotFound", "failed_at"=>1453895736.6926756, "retry_count"=>0}
2016-01-27_11:55:36.69586 2016-01-27T11:55:36.695Z 3104 TID-rfujo WARN: ActiveRecord::RecordNotFound: Couldn't find ProjectMember with 'id'=45 [WHERE "members"."type" IN ('ProjectMember') AND "members"."source_type" = ?]
2016-01-27_11:55:36.69595 2016-01-27T11:55:36.695Z 3104 TID-rfujo WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:324:in `raise_record_not_found_exception!'
2016-01-27_11:55:36.69595 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:444:in `find_one'
2016-01-27_11:55:36.69596 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:423:in `find_with_ids'
2016-01-27_11:55:36.69596 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:71:in `find'
2016-01-27_11:55:36.69596 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:3:in `find'
2016-01-27_11:55:36.69596 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/core.rb:131:in `find'
2016-01-27_11:55:36.69596 /opt/gitlab/embedded/service/gitlab-rails/app/mailers/emails/projects.rb:4:in `project_access_granted_email'
2016-01-27_11:55:36.69597 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:198:in `process_action'
2016-01-27_11:55:36.69597 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
2016-01-27_11:55:36.69597 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
2016-01-27_11:55:36.69597 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
2016-01-27_11:55:36.69598 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
2016-01-27_11:55:36.69598 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
2016-01-27_11:55:36.69598 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in `process'
2016-01-27_11:55:36.69598 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in `process'
2016-01-27_11:55:36.69598 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:596:in `block in process'
2016-01-27_11:55:36.69599 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
2016-01-27_11:55:36.69599 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2016-01-27_11:55:36.69599 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
2016-01-27_11:55:36.69599 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:593:in `process'
2016-01-27_11:55:36.69600 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:584:in `initialize'
2016-01-27_11:55:36.69600 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:25:in `new'
2016-01-27_11:55:36.69601 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:25:in `__getobj__'
2016-01-27_11:55:36.69601 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:34:in `message'
2016-01-27_11:55:36.69602 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:85:in `deliver_now'
2016-01-27_11:55:36.69602 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/delivery_job.rb:10:in `perform'
2016-01-27_11:55:36.69602 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:32:in `block in perform_now'
2016-01-27_11:55:36.69602 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
2016-01-27_11:55:36.69603 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
2016-01-27_11:55:36.69603 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
2016-01-27_11:55:36.69603 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:36.69603 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:36.69603 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
2016-01-27_11:55:36.69605 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
2016-01-27_11:55:36.69605 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
2016-01-27_11:55:36.69605 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'
2016-01-27_11:55:36.69605 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
2016-01-27_11:55:36.69606 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
2016-01-27_11:55:36.69606 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
2016-01-27_11:55:36.69606 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
2016-01-27_11:55:36.69606 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
2016-01-27_11:55:36.69607 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
2016-01-27_11:55:36.69607 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
2016-01-27_11:55:36.69607 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:36.69607 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:36.69607 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
2016-01-27_11:55:36.69608 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
2016-01-27_11:55:36.69608 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
2016-01-27_11:55:36.69608 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `call'
2016-01-27_11:55:36.69609 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
2016-01-27_11:55:36.69609 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
2016-01-27_11:55:36.69609 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2016-01-27_11:55:36.69609 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
2016-01-27_11:55:36.69609 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
2016-01-27_11:55:36.69610 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `block in tag_logger'
2016-01-27_11:55:36.69610 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
2016-01-27_11:55:36.69610 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:26:in `tagged'
2016-01-27_11:55:36.69610 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `tagged'
2016-01-27_11:55:36.69611 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `tag_logger'
2016-01-27_11:55:36.69611 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
2016-01-27_11:55:36.69611 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
2016-01-27_11:55:36.69611 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
2016-01-27_11:55:36.69611 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
2016-01-27_11:55:36.69612 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
2016-01-27_11:55:36.69612 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
2016-01-27_11:55:36.69613 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
2016-01-27_11:55:36.69613 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:36.69613 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:36.69614 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
2016-01-27_11:55:36.69614 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_perform_callbacks'
2016-01-27_11:55:36.69614 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
2016-01-27_11:55:36.69614 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:31:in `perform_now'
2016-01-27_11:55:36.69614 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:21:in `execute'
2016-01-27_11:55:36.69614 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
2016-01-27_11:55:36.69615 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
2016-01-27_11:55:36.69615 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
2016-01-27_11:55:36.69616 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
2016-01-27_11:55:36.69616 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
2016-01-27_11:55:36.69616 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:36.69616 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
2016-01-27_11:55:36.69616 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:36.69617 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2016-01-27_11:55:36.69617 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:36.69617 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2016-01-27_11:55:36.69617 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in `with_context'
2016-01-27_11:55:36.69618 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2016-01-27_11:55:36.69618 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:36.69618 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `call'
2016-01-27_11:55:36.69618 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'
2016-01-27_11:55:36.69618 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in `block in process'
2016-01-27_11:55:36.69618 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in `stats'
2016-01-27_11:55:36.69619 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in `process'
2016-01-27_11:55:36.69619 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in `process_one'
2016-01-27_11:55:36.69619 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in `run'
2016-01-27_11:55:36.69619 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in `watchdog'
2016-01-27_11:55:36.69620 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in `block in safe_thread'
2016-01-27_11:55:22.25719 2016-01-27T11:55:22.257Z 3104 TID-2l2u0 WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"ActionMailer::DeliveryJob", "queue"=>"mailers", "args"=>[{"job_class"=>"ActionMailer::DeliveryJob", "job_id"=>"a22ac4e2-dbdf-42e4-9aad-aad68bd49e72", "queue_name"=>"mailers", "arguments"=>["Notify", "new_user_email", "deliver_now", 9, "eBAgb624GbD36WP_x3yX"], "locale"=>"en"}], "retry"=>true, "jid"=>"334838f57df9f4c9a6410c03", "created_at"=>1453895722.2498167, "enqueued_at"=>1453895722.2499185, "error_message"=>"Couldn't find User with 'id'=9", "error_class"=>"ActiveRecord::RecordNotFound", "failed_at"=>1453895722.2566762, "retry_count"=>0}
2016-01-27_11:55:22.25724 2016-01-27T11:55:22.257Z 3104 TID-2l2u0 WARN: ActiveRecord::RecordNotFound: Couldn't find User with 'id'=9
2016-01-27_11:55:22.25730 2016-01-27T11:55:22.257Z 3104 TID-2l2u0 WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:324:in `raise_record_not_found_exception!'
2016-01-27_11:55:22.25731 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:444:in `find_one'
2016-01-27_11:55:22.25731 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:423:in `find_with_ids'
2016-01-27_11:55:22.25731 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/relation/finder_methods.rb:71:in `find'
2016-01-27_11:55:22.25732 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:3:in `find'
2016-01-27_11:55:22.25732 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/core.rb:131:in `find'
2016-01-27_11:55:22.25732 /opt/gitlab/embedded/service/gitlab-rails/app/mailers/emails/profile.rb:4:in `new_user_email'
2016-01-27_11:55:22.25732 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:198:in `process_action'
2016-01-27_11:55:22.25733 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
2016-01-27_11:55:22.25733 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
2016-01-27_11:55:22.25733 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
2016-01-27_11:55:22.25734 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
2016-01-27_11:55:22.25734 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
2016-01-27_11:55:22.25734 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in `process'
2016-01-27_11:55:22.25734 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in `process'
2016-01-27_11:55:22.25734 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:596:in `block in process'
2016-01-27_11:55:22.25736 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
2016-01-27_11:55:22.25737 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2016-01-27_11:55:22.25737 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
2016-01-27_11:55:22.25737 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:593:in `process'
2016-01-27_11:55:22.25737 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/base.rb:584:in `initialize'
2016-01-27_11:55:22.25737 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:25:in `new'
2016-01-27_11:55:22.25738 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:25:in `__getobj__'
2016-01-27_11:55:22.25738 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:34:in `message'
2016-01-27_11:55:22.25738 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/message_delivery.rb:85:in `deliver_now'
2016-01-27_11:55:22.25738 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionmailer-4.2.5.1/lib/action_mailer/delivery_job.rb:10:in `perform'
2016-01-27_11:55:22.25739 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:32:in `block in perform_now'
2016-01-27_11:55:22.25739 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
2016-01-27_11:55:22.25739 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
2016-01-27_11:55:22.25739 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
2016-01-27_11:55:22.25739 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:22.25740 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:22.25740 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
2016-01-27_11:55:22.25741 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
2016-01-27_11:55:22.25741 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
2016-01-27_11:55:22.25741 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'
2016-01-27_11:55:22.25742 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
2016-01-27_11:55:22.25742 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
2016-01-27_11:55:22.25742 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
2016-01-27_11:55:22.25742 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
2016-01-27_11:55:22.25742 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
2016-01-27_11:55:22.25743 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
2016-01-27_11:55:22.25743 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
2016-01-27_11:55:22.25744 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:22.25744 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:22.25744 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
2016-01-27_11:55:22.25744 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `call'
2016-01-27_11:55:22.25745 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
2016-01-27_11:55:22.25745 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `call'
2016-01-27_11:55:22.25745 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
2016-01-27_11:55:22.25745 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
2016-01-27_11:55:22.25746 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2016-01-27_11:55:22.25746 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
2016-01-27_11:55:22.25746 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
2016-01-27_11:55:22.25746 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `block in tag_logger'
2016-01-27_11:55:22.25746 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
2016-01-27_11:55:22.25746 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:26:in `tagged'
2016-01-27_11:55:22.25747 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/tagged_logging.rb:68:in `tagged'
2016-01-27_11:55:22.25747 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:43:in `tag_logger'
2016-01-27_11:55:22.25747 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
2016-01-27_11:55:22.25747 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `instance_exec'
2016-01-27_11:55:22.25748 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
2016-01-27_11:55:22.25748 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `call'
2016-01-27_11:55:22.25748 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:342:in `block in simple'
2016-01-27_11:55:22.25748 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
2016-01-27_11:55:22.25749 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
2016-01-27_11:55:22.25750 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:22.25750 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
2016-01-27_11:55:22.25750 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
2016-01-27_11:55:22.25750 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_perform_callbacks'
2016-01-27_11:55:22.25751 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
2016-01-27_11:55:22.25751 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:31:in `perform_now'
2016-01-27_11:55:22.25751 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/execution.rb:21:in `execute'
2016-01-27_11:55:22.25751 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activejob-4.2.5.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
2016-01-27_11:55:22.25752 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
2016-01-27_11:55:22.25752 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
2016-01-27_11:55:22.25752 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
2016-01-27_11:55:22.25752 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
2016-01-27_11:55:22.25752 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:22.25753 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
2016-01-27_11:55:22.25753 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:22.25753 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2016-01-27_11:55:22.25753 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:22.25754 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2016-01-27_11:55:22.25754 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in `with_context'
2016-01-27_11:55:22.25754 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2016-01-27_11:55:22.25754 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-01-27_11:55:22.25754 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `call'
2016-01-27_11:55:22.25755 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'
2016-01-27_11:55:22.25755 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in `block in process'
2016-01-27_11:55:22.25755 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in `stats'
2016-01-27_11:55:22.25755 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in `process'
2016-01-27_11:55:22.25756 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in `process_one'
2016-01-27_11:55:22.25756 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in `run'
2016-01-27_11:55:22.25756 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in `watchdog'
2016-01-27_11:55:22.25756 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in `block in safe_thread'
2016-01-27_11:55:36.68667 2016-01-27T11:55:36.683Z 3104 TID-rfujo ActionMailer::DeliveryJob JID-0251a302be2d1eccec0c9fef INFO: start
2016-01-27_11:55:36.69562 2016-01-27T11:55:36.695Z 3104 TID-rfujo ActionMailer::DeliveryJob JID-0251a302be2d1eccec0c9fef INFO: fail: 0.012 sec
2016-01-27_11:26:22.91858 2016-01-27T11:26:22.918Z 819 TID-u8bzg ActionMailer::DeliveryJob JID-af2855fa6a1586a78ec8bdbd INFO: fail: 0.647 sec
2016-01-27_11:26:22.93902 2016-01-27T11:26:22.938Z 819 TID-u8bzg WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"ActionMailer::DeliveryJob", "queue"=>"mailers", "args"=>[{"job_class"=>"ActionMailer::DeliveryJob", "job_id"=>"302be9c7-c07c-4388-86a7-5dac0df879d0", "queue_name"=>"mailers", "arguments"=>["Notify", "note_issue_email", "deliver_now", 8, 2538], "locale"=>"en"}], "retry"=>true, "jid"=>"af2855fa6a1586a78ec8bdbd", "created_at"=>1453893982.2626197, "enqueued_at"=>1453893982.263419, "error_message"=>"Circular dependency detected while autoloading constant Gitlab::IncomingEmail", "error_class"=>"RuntimeError", "failed_at"=>1453893982.9180193, "retry_count"=>0}
2016-01-27_11:26:22.93905 2016-01-27T11:26:22.939Z 819 TID-u8bzg WARN: RuntimeError: Circular dependency detected while autoloading constant Gitlab::IncomingEmailSMTP is not really necessary, basic sendmail is good enough but I can't seeem to get it working. :/
I have followed "steps" in https://gitlab.com/gitlab-org/gitlab-ce/issues/10448