Skip to content

Geo: clocks must be in sync

For JWT authentication work correctly, clocks between all involved machines must be in sync. We say in the docs we require NTP to be setup in the machines, but we currently have no code that monitors that nor warns the user if the clocks are not in sync.

An unsync clock will cause a few issues:

Gitlab:Geo:FileTransfer: Unsuccessful download: 500 Internal Server Error
Started POST "/api/v4/jobs/request" for 10.139.9.176 at 2017-05-23 15:23:20 +0800
Error decoding Geo request: Invalid iat

NoMethodError (undefined method `[]' for true:TrueClass):
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/geo/file_uploader.rb:28:in `matches_requested_model?'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/geo/file_uploader.rb:23:in `valid?'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/geo/file_uploader.rb:15:in `execute'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/geo/file_upload_service.rb:17:in `execute'
  /opt/gitlab/embedded/service/gitlab-rails/lib/api/geo.rb:18:in `block (2 levels) in <class:Geo>'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:59:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:59:in `block (2 levels) in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:166:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:58:in `block in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:262:in `block in run'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:166:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:240:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:309:in `block in build_stack'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/base.rb:31:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/base.rb:31:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/error.rb:34:in `block in call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/error.rb:33:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/error.rb:33:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:224:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/endpoint.rb:218:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router/route.rb:72:in `exec'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router.rb:119:in `process_route'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router.rb:74:in `block in identity'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router.rb:93:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router.rb:72:in `identity'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router.rb:57:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router.rb:135:in `with_optimization'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/router.rb:56:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/api.rb:119:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/api.rb:45:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grape-0.19.1/lib/grape/api.rb:40:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/routing/mapper.rb:51:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/journey/router.rb:43:in `block in serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/journey/router.rb:30:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/journey/router.rb:30:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/routing/route_set.rb:817:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:93:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:14:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:16:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:10:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:35:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-cors-0.4.0/lib/rack/cors.rb:80:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/etag.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/conditionalget.rb:25:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/readonly_geo.rb:30:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/flash.rb:260:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/cookies.rb:560:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/query_cache.rb:36:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/rack/logger.rb:38:in `call_app'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/rack/logger.rb:20:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/tagged_logging.rb:68:in `block in tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/tagged_logging.rb:26:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/tagged_logging.rb:68:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/rack/logger.rb:20:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:18:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/request_id.rb:21:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/lock.rb:17:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sentry-raven-2.4.0/lib/raven/integrations/rack.rb:50:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/engine.rb:518:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:165:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `public_send'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `method_missing'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:66:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:132:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
  /opt/gitlab/embedded/bin/unicorn:22:in `load'
  /opt/gitlab/embedded/bin/unicorn:22:in `<top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
  /opt/gitlab/embedded/bin/bundle:22:in `load'
  /opt/gitlab/embedded/bin/bundle:22:in `<main>'

The amount of time we allow clocks to be off from each other is hard-coded in the lib/gitlab/geo/jwt_request_decoder.rb:

module Gitlab
  module Geo
    class JwtRequestDecoder
      IAT_LEEWAY = 60.seconds.to_i

Based on the discussion here: https://github.com/zquestz/omniauth-google-oauth2/issues/197 I believe we can increase the IAT_LEEWAY to a bigger number without compromising security. Let's say we decide from a value in the 2 - 10 minutes range. This will reduce the possibility of machines not using NTP to show up the issue too often.

Another thing we may do is try to provide clock / timezone information in the status API call for Geo, and figure out if clocks are out of sync.

We can also add an NTP call to the gitlab:geo:check rake task to check if clocks are off by more than the IAT_LEEWAY and help system administrators fix the issue. (I will cross-link this suggestion with the effort concentration in #727).

cc @stanhu @dbalexandre @rspeicher @xiaogang_gitlab

Edited by Gabriel Mazetto