Merge branch 'cleanup-config-settings' into 'master'

Move gitlab.rb config library to the packages cookbook

Closes #2662

See merge request !1822

CHANGELOG.md

@@ -9,7 +9,8 @@ omnibus-gitlab repository.
 - Remove sensitive params from the NGINX access logs 6983fe59
 - Add option to pass EXTERNAL_URL during installation d0f30ef2
   * Saves users from manually editing gitlab.rb just to set the URL and hence
-	makes installation process easier
+    makes installation process easier
+- Moved the settings handling into the package cookbook and reduced code duplication in settings
 - Remove geo_bulk_notify_worker_cron 44def4b5
 9.5.5

doc/development/new-services.md

@@ -60,29 +60,24 @@ now.
 In order for user to be able to configure your service from `/etc/gitlab/gitlab.rb`
 you will need to add a top level Mash for the service.
-In `files/gitlab-cookbooks/gitlab/libraries/gitlab.rb` you will find the list of
-`Mash.new calls`.
-Add your service to the bottom, using an underscore to seperate words, **even if you
+In `files/gitlab-cookbooks/package/libraries/config/gitlab.rb` you will find the list of
+`attribute` methods.
+If your service exists within the attributes for the gitlab cookbook, you should
+add it within the `attribute_block('gitlab')` block.  Otherwise, if your service
+has its own cookbook, add it above.
+
+Add your service as an attribute, using an underscore to separate words, **even if you
 used a hyphen in the default attributes.**
 ```ruby
-best_service Mash.new
+attribute('best_service')
 ```
-You also need to add the service a little lower down in the `generate_hash` method.
-Add your service into the array. If you service exists within the attributes for
-the gitlab cookbook, you should add it in the array that gets appended to the `gitlab`
-results, the first array. Otherwise, if your service has its own cookbook, add it
-to the second array.
+For an EE only attribute, use `ee_attribute` instead.
 ```ruby
-[
-  "best_service"
-].each do |key|
-  rkey = key.tr('_', '-')
-  results['gitlab'][rkey] = Gitlab[key]
-end
+ee_attribute('best_service')
 ```
 ### Add service configuration to the settings template
@@ -113,7 +108,7 @@ In order to allow the service to be easily enable/disabled within the recipes, i
 should be added to the [services list](../architecture/README.md#services)
 and given appropriate groups.
-In the `files/gitlab-cookbooks/package/libraries/services.rb` file, add the
+In the `files/gitlab-cookbooks/package/libraries/config/services.rb` file, add the
 service to the appropriate Config class, Base or EE depending on whether the
 service is only for GitLab EE.
@@ -240,21 +235,18 @@ end
 We then need to have the gitlab config call your parse_variables method.
-Go into `files/gitlab-cookbooks/libraries/gitlab.rb` and all a `require_relative`
-for the library.
+Go into `files/gitlab-cookbooks/package/libraries/config/gitlab.rb` and update
+your attribute to use the library.
 ```ruby
-require_relative 'best-service.rb'
+attribute('best_service').use { BestService }
 ```
-Then below, in the `generate_config` method. Add a call to your `parse_variables`,
-above the call to `generate_hash`.
+Note that sequence for parsing variables matters. So if your library expects to
+be parsed after another service's library, you need to update your attribute with
+a `sequence` value that comes later. (The default `sequence` value is `20`)
 ```ruby
-BestService.parse_variables
+attribute('expected_service').use { ExpectedService }
+attribute('best_service', sequence: 25).use { BestService }
 ```
-
-Not that order in the `generate_config` method matters. So any config that the
-new service library expects to have been parsed should come first, and any other
-config parsing that depends on the new service's library should be parsed later
-in the list. Place the new service's parse method with that in mind.

files/gitlab-cookbooks/gitlab-ee/libraries/gitlab-ee.rb

@@ -15,39 +15,7 @@
 # limitations under the License.
 #
-require_relative 'sidekiq_cluster.rb'
-require_relative 'gitlab_geo.rb'
-
-module GitlabEE
-  class << self
-    def generate_hash
-      # NOTE: If you are adding a new service
-      # and that service has logging, make sure you add the service to
-      # the array in parse_udp_log_shipping.
-      #
-      # Add to the list below any service that has additional EE specific
-      # behavior or is impacted by an EE role definition
-      results = { 'gitlab' => {} }
-      [
-        'sidekiq_cluster',
-        'geo_secondary',
-        'geo_postgresql',
-        'geo_logcursor',
-        'postgresql', # impacted by role
-        'gitlab_rails' # impacted by role
-      ].each do |key|
-        rkey = key.tr('_', '-')
-        results['gitlab'][rkey] = Gitlab[key]
-      end
-
-      results
-    end
-
-    def generate_config
-      SidekiqCluster.parse_variables
-      GitlabGeo.parse_variables
-      # The last step is to convert underscores to hyphens in top-level keys
-      generate_hash
-    end
-  end
+# Override the GitLab edition to be EE
+module Gitlab
+  edition :ee
 end

files/gitlab-cookbooks/gitlab/libraries/gitlab.rb

-#
-# Copyright:: Copyright (c) 2012 Opscode, Inc.
-# Copyright:: Copyright (c) 2016 GitLab Inc.
-# License:: Apache License, Version 2.0
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# The Gitlab module in this file is used to parse /etc/gitlab/gitlab.rb.
-#
-# Warning to the reader:
-# Because the Ruby DSL in /etc/gitlab/gitlab.rb does not accept hyphens in
-# section names, this module translates names like 'gitlab_rails' to the
-# correct 'gitlab-rails' in the `generate_hash` method. This module is the only
-# place in the cookbook where we write 'gitlab_rails'.
-
-require 'mixlib/config'
-require 'chef/json_compat'
-require 'chef/mixin/deep_merge'
-require 'securerandom'
-require 'uri'
-
-require_relative 'gitlab_mattermost.rb'
-require_relative 'gitlab_pages.rb'
-require_relative 'gitlab_rails.rb'
-require_relative 'gitlab_workhorse.rb'
-require_relative 'incoming_email.rb'
-require_relative 'logging.rb'
-require_relative 'nginx.rb'
-require_relative 'postgresql.rb'
-require_relative 'redis.rb'
-require_relative 'registry.rb'
-require_relative 'unicorn.rb'
-require_relative 'gitaly.rb'
-require_relative 'prometheus.rb'
-
-module Gitlab
-  extend(Mixlib::Config)
-
-  bootstrap ConfigMash.new
-  omnibus_gitconfig ConfigMash.new
-  manage_accounts ConfigMash.new
-  manage_storage_directories ConfigMash.new
-  runtime_dir nil
-  user ConfigMash.new
-  postgresql ConfigMash.new
-  redis ConfigMash.new
-  gitlab_rails ConfigMash.new
-  gitlab_ci ConfigMash.new
-  gitlab_shell ConfigMash.new
-  unicorn ConfigMash.new
-  sidekiq ConfigMash.new
-  sidekiq_cluster ConfigMash.new
-  gitlab_workhorse ConfigMash.new
-  gitlab_git_http_server ConfigMash.new # legacy from GitLab 7.14, 8.0, 8.1
-  pages_nginx ConfigMash.new
-  registry_nginx ConfigMash.new
-  mailroom ConfigMash.new
-  nginx ConfigMash.new
-  mattermost_nginx ConfigMash.new
-  logging ConfigMash.new
-  remote_syslog ConfigMash.new
-  logrotate ConfigMash.new
-  high_availability ConfigMash.new
-  web_server ConfigMash.new
-  mattermost ConfigMash.new
-  gitlab_pages ConfigMash.new
-  registry ConfigMash.new
-  node_exporter ConfigMash.new
-  prometheus ConfigMash.new
-  redis_exporter ConfigMash.new
-  postgres_exporter ConfigMash.new
-  gitlab_monitor ConfigMash.new
-  sentinel ConfigMash.new
-  node nil
-  external_url nil
-  pages_external_url nil
-  mattermost_external_url nil
-  registry_external_url nil
-  git_data_dirs ConfigMash.new
-  gitaly ConfigMash.new
-  geo_secondary ConfigMash.new
-  geo_postgresql ConfigMash.new
-  geo_logcursor ConfigMash.new
-  prometheus_monitoring ConfigMash.new
-  pgbouncer ConfigMash.new
-  repmgr ConfigMash.new
-  consul ConfigMash.new
-
-  # Single-Service Roles
-  # When enabled, default enabled services are disabled
-  redis_sentinel_role ConfigMash.new
-  redis_master_role ConfigMash.new
-  redis_slave_role ConfigMash.new
-
-  SERVICE_ROLES ||= [
-    'redis_sentinel',
-    'redis_master',
-    'redis_slave',
-  ].freeze
-
-  # Behavior roles
-  # Used to configure different "defaults" based on intended behavior of the node.
-  geo_primary_role ConfigMash.new
-  geo_secondary_role ConfigMash.new
-
-  BEHAVIOR_ROLES ||= [
-    'geo_primary',
-    'geo_secondary',
-  ].freeze
-
-  class << self
-    def from_file(_file_path)
-      # Allow auto mash creation during from_file call
-      ConfigMash.auto_vivify { super }
-    end
-
-    def method_missing(method_name, *arguments)
-      # Give better message for NilClass errors
-      # If there are no arguements passed, this is a 'GET' call, and if
-      # there is no matching key in the configuration, then it has not been set (not even to nil)
-      # and we will output a nicer error above the exception
-      if arguments.length == 0 && !configuration.has_key?(method_name)
-        message = "Encountered unsupported config key '#{method_name}' in /etc/gitlab/gitlab.rb."
-        puts "\n  *ERROR*: #{message}\n"
-        Chef::Log.error(message)
-      end
-
-      # Parent method_missing takes care of setting values for missing methods
-      super
-    end
-
-    # guards against creating secrets on non-bootstrap node
-    def generate_secrets(node_name)
-      SecretsHelper.read_gitlab_secrets
-
-      # Blow up when the existing configuration is ambiguous, so we don't accidentally throw away important secrets
-      ci_db_key_base = Gitlab['gitlab_ci']['db_key_base']
-      rails_db_key_base = Gitlab['gitlab_rails']['db_key_base']
-
-      if ci_db_key_base && rails_db_key_base && ci_db_key_base != rails_db_key_base
-        message = [
-          "The value of Gitlab['gitlab_ci']['db_key_base'] (#{ci_db_key_base}) does not match the value of Gitlab['gitlab_rails']['db_key_base'] (#{rails_db_key_base}).",
-          "Please back up both secrets, set Gitlab['gitlab_rails']['db_key_base'] to the value of Gitlab['gitlab_ci']['db_key_base'], and try again.",
-          "For more information, see <https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update/README.md#migrating-legacy-secrets>"
-        ]
-
-        raise message.join("\n\n")
-      end
-
-      # Transform legacy key names to new key names
-      Gitlab['gitlab_rails']['db_key_base'] ||= Gitlab['gitlab_ci']['db_key_base'] # Changed in 8.11
-      Gitlab['gitlab_rails']['secret_key_base'] ||= Gitlab['gitlab_ci']['db_key_base'] # Changed in 8.11
-      Gitlab['gitlab_rails']['otp_key_base'] ||= Gitlab['gitlab_rails']['secret_token']
-
-      # Note: If you add another secret to generate here make sure it gets written to disk in SecretsHelper.write_to_gitlab_secrets
-      Gitlab['gitlab_rails']['db_key_base'] ||= SecretsHelper.generate_hex(64)
-      Gitlab['gitlab_rails']['secret_key_base'] ||= SecretsHelper.generate_hex(64)
-      Gitlab['gitlab_rails']['otp_key_base'] ||= SecretsHelper.generate_hex(64)
-      Gitlab['gitlab_rails']['jws_private_key'] ||= SecretsHelper.generate_rsa(4096).to_pem
-
-      Gitlab['gitlab_shell']['secret_token'] ||= SecretsHelper.generate_hex(64)
-
-      # gitlab-workhorse expects exactly 32 bytes, encoded with base64
-      Gitlab['gitlab_workhorse']['secret_token'] ||= SecureRandom.base64(32)
-
-      Gitlab['registry']['http_secret'] ||= SecretsHelper.generate_hex(64)
-      gitlab_registry_crt, gitlab_registry_key = Registry.generate_registry_keypair
-      Gitlab['registry']['internal_certificate'] ||= gitlab_registry_crt
-      Gitlab['registry']['internal_key'] ||= gitlab_registry_key
-
-      Gitlab['mattermost']['email_invite_salt'] ||= SecretsHelper.generate_hex(16)
-      Gitlab['mattermost']['file_public_link_salt'] ||= SecretsHelper.generate_hex(16)
-      Gitlab['mattermost']['sql_at_rest_encrypt_key'] ||= SecretsHelper.generate_hex(16)
-
-      SecretsHelper.write_to_gitlab_secrets
-    end
-
-    def generate_hash
-      # NOTE: If you are adding a new service
-      # and that service has logging, make sure you add the service to
-      # the array in parse_udp_log_shipping.
-      results = { "gitlab" => {} }
-      [
-        "bootstrap",
-        "omnibus_gitconfig",
-        "manage_accounts",
-        "manage_storage_directories",
-        "runtime_dir",
-        "user",
-        "redis",
-        "gitlab_rails",
-        "gitlab_ci",
-        "gitlab_shell",
-        "unicorn",
-        "sidekiq",
-        "sidekiq-cluster",
-        "gitlab_workhorse",
-        "mailroom",
-        "nginx",
-        "mattermost_nginx",
-        "pages_nginx",
-        "registry_nginx",
-        "logging",
-        "remote_syslog",
-        "logrotate",
-        "high_availability",
-        "postgresql",
-        "web_server",
-        "mattermost",
-        "external_url",
-        "mattermost_external_url",
-        "pages_external_url",
-        "gitlab_pages",
-        "gitaly",
-        "node_exporter",
-        "prometheus",
-        "redis_exporter",
-        "postgres_exporter",
-        "gitlab_monitor",
-        'prometheus_monitoring',
-        'pgbouncer',
-        "sentinel"
-      ].each do |key|
-        rkey = key.tr('_', '-')
-        results['gitlab'][rkey] = Gitlab[key]
-      end
-
-      %w(
-        consul
-        registry
-        repmgr
-        repmgrd
-      ).each do |key|
-        rkey = key.tr('_', '-')
-        results[rkey] = Gitlab[key]
-      end
-
-      results['roles'] = {}
-      (BEHAVIOR_ROLES + SERVICE_ROLES).each do |key|
-        rkey = key.gsub('_', '-')
-        results['roles'][rkey] = Gitlab["#{key}_role"]
-      end
-
-      results
-    end
-
-    def generate_config(node_name)
-      generate_secrets(node_name)
-      GitlabWorkhorse.parse_variables
-      GitlabShell.parse_variables
-      GitlabRails.parse_variables
-      Logging.parse_variables
-      Redis.parse_variables
-      Postgresql.parse_variables
-      Unicorn.parse_variables
-      IncomingEmail.parse_variables
-      GitlabMattermost.parse_variables
-      GitlabPages.parse_variables
-      Registry.parse_variables
-      Prometheus.parse_variables
-      # Parse nginx variables last because we want all external_url to be
-      # parsed first
-      Nginx.parse_variables
-      GitlabRails.disable_services
-      # The last step is to convert underscores to hyphens in top-level keys
-      generate_hash
-    end
-  end
-end

files/gitlab-cookbooks/gitlab/libraries/gitlab_mattermost.rb

@@ -24,6 +24,12 @@ module GitlabMattermost
       parse_gitlab_mattermost
     end
+    def parse_secrets
+      Gitlab['mattermost']['email_invite_salt'] ||= SecretsHelper.generate_hex(16)
+      Gitlab['mattermost']['file_public_link_salt'] ||= SecretsHelper.generate_hex(16)
+      Gitlab['mattermost']['sql_at_rest_encrypt_key'] ||= SecretsHelper.generate_hex(16)
+    end
+
     def parse_mattermost_external_url
       return unless Gitlab['mattermost_external_url']

files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb

@@ -38,6 +38,33 @@ module GitlabRails
       parse_repository_storage
     end
+    def parse_secrets
+      # Blow up when the existing configuration is ambiguous, so we don't accidentally throw away important secrets
+      ci_db_key_base = Gitlab['gitlab_ci']['db_key_base']
+      rails_db_key_base = Gitlab['gitlab_rails']['db_key_base']
+
+      if ci_db_key_base && rails_db_key_base && ci_db_key_base != rails_db_key_base
+        message = [
+          "The value of Gitlab['gitlab_ci']['db_key_base'] (#{ci_db_key_base}) does not match the value of Gitlab['gitlab_rails']['db_key_base'] (#{rails_db_key_base}).",
+          "Please back up both secrets, set Gitlab['gitlab_rails']['db_key_base'] to the value of Gitlab['gitlab_ci']['db_key_base'], and try again.",
+          "For more information, see <https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update/README.md#migrating-legacy-secrets>"
+        ]
+
+        raise message.join("\n\n")
+      end
+
+      # Transform legacy key names to new key names
+      Gitlab['gitlab_rails']['db_key_base'] ||= Gitlab['gitlab_ci']['db_key_base'] # Changed in 8.11
+      Gitlab['gitlab_rails']['secret_key_base'] ||= Gitlab['gitlab_ci']['db_key_base'] # Changed in 8.11
+      Gitlab['gitlab_rails']['otp_key_base'] ||= Gitlab['gitlab_rails']['secret_token']
+
+      # Note: If you add another secret to generate here make sure it gets written to disk in SecretsHelper.write_to_gitlab_secrets
+      Gitlab['gitlab_rails']['db_key_base'] ||= SecretsHelper.generate_hex(64)
+      Gitlab['gitlab_rails']['secret_key_base'] ||= SecretsHelper.generate_hex(64)
+      Gitlab['gitlab_rails']['otp_key_base'] ||= SecretsHelper.generate_hex(64)
+      Gitlab['gitlab_rails']['jws_private_key'] ||= SecretsHelper.generate_rsa(4096).to_pem
+    end
+
     def parse_external_url
       return unless Gitlab['external_url']
@@ -161,6 +188,10 @@ module GitlabRails
       end
     end
+    def load_role
+      disable_services
+    end
+
     def disable_services
       disable_services_roles if any_service_role_defined?
@@ -195,7 +226,7 @@ module GitlabRails
     private
     def any_service_role_defined?
-      Gitlab::SERVICE_ROLES.any? { |role| Gitlab["#{role}_role"]['enable'] }
+      Gitlab.roles.keys.any? { |role| Gitlab["#{role}_role"]['enable'] }
     end
     def disable_services_roles

files/gitlab-cookbooks/gitlab/libraries/gitlab_shell.rb

@@ -23,6 +23,10 @@ module GitlabShell
       parse_auth_file
     end
+    def parse_secrets
+      Gitlab['gitlab_shell']['secret_token'] ||= SecretsHelper.generate_hex(64)
+    end
+
     def parse_git_data_dirs
       git_data_dirs = Gitlab['git_data_dirs']
       git_data_dir = Gitlab['git_data_dir']

files/gitlab-cookbooks/gitlab/libraries/gitlab_workhorse.rb

@@ -17,20 +17,9 @@
 module GitlabWorkhorse
   class << self
-    def parse_variables
-      parse_gitlab_git_http_server
-    end
-
-    def parse_gitlab_git_http_server
-      Gitlab['gitlab_git_http_server'].each do |k, v|
-        Chef::Log.warn "gitlab_git_http_server is deprecated. Please use gitlab_workhorse in gitlab.rb"
-        if Gitlab['gitlab_workhorse'][k].nil?
-          Chef::Log.warn "applying legacy setting gitlab_git_http_server[#{k.inspect}]"
-          Gitlab['gitlab_workhorse'][k] = v
-        else
-          Chef::Log.warn "ignoring legacy setting gitlab_git_http_server[#{k.inspect}]"
-        end
-      end
+    def parse_secrets
+      # gitlab-workhorse expects exactly 32 bytes, encoded with base64
+      Gitlab['gitlab_workhorse']['secret_token'] ||= SecureRandom.base64(32)
     end
   end
 end

files/gitlab-cookbooks/gitlab/libraries/registry.rb

@@ -28,6 +28,13 @@ module Registry
       parse_registry_notifications
     end
+    def parse_secrets
+      Gitlab['registry']['http_secret'] ||= SecretsHelper.generate_hex(64)
+      gitlab_registry_crt, gitlab_registry_key = Registry.generate_registry_keypair
+      Gitlab['registry']['internal_certificate'] ||= gitlab_registry_crt
+      Gitlab['registry']['internal_key'] ||= gitlab_registry_key
+    end
+
     def parse_registry_external_url
       return unless Gitlab['registry_external_url']

files/gitlab-cookbooks/gitlab/recipes/config.rb

@@ -24,8 +24,3 @@ if File.exists?('/etc/gitlab/gitlab.rb')
 end
 node.consume_attributes(Gitlab.generate_config(node['fqdn']))
-
-# If is EE package, load EE config
-if defined?(GitlabEE) == 'constant'
-  node.consume_attributes(GitlabEE.generate_config)
-end

files/gitlab-cookbooks/gitlab/recipes/default.rb

@@ -144,8 +144,3 @@ include_recipe "gitlab::gitlab-healthcheck" if node['gitlab']['nginx']['enable']
 # Recipe which handles all prometheus related services
 include_recipe "gitlab::gitlab-prometheus"
-
-# Deprecated in favor of gitlab-workhorse since 8.2
-runit_service "gitlab-git-http-server" do
-  action :disable
-end

files/gitlab-cookbooks/gitlab/recipes/mattermost.rb

@@ -102,7 +102,7 @@ end
 ruby_block "populate mattermost configuration options" do
   block do
-    node.consume_attributes(Gitlab.generate_hash)
+    node.consume_attributes(Gitlab.hyphenate_config_keys)
   end
 end

files/gitlab-cookbooks/package/libraries/config/gitlab.rb

+#
+# Copyright:: Copyright (c) 2017 GitLab Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+require_relative '../helpers/settings_helper.rb'
+
+module Gitlab
+  extend(Mixlib::Config)
+  extend(SettingsHelper)
+
+  ## Attributes that don't get passed to the node
+  node nil
+  edition :ce
+  git_data_dirs ConfigMash.new
+
+  ## Roles
+  role('redis_sentinel').use { GitlabRails }
+  role('redis_master')
+  role('redis_slave')
+  role('geo_primary')
+  role('geo_secondary')
+
+  ## Attributes directly on the node
+  attribute('registry', priority: 20).use { Registry }
+  attribute('repmgr')
+  attribute('repmgrd')
+  attribute('consul')
+
+  ## Attributes under node['gitlab']
+  attribute_block 'gitlab' do
+    # EE attributes
+    ee_attribute('sidekiq_cluster', priority: 20).use { SidekiqCluster }
+    ee_attribute('geo_postgresql',  priority: 20).use { GitlabGeo }
+    ee_attribute('geo_secondary')
+    ee_attribute('geo_logcursor')
+
+    # Base GitLab attributes
+    attribute('gitlab_shell',     priority: 10).use { GitlabShell } # Parse shell before rails for data dir settings
+    attribute('gitlab_rails',     priority: 15).use { GitlabRails } # Parse rails first as others may depend on it
+    attribute('gitlab_workhorse', priority: 20).use { GitlabWorkhorse }
+    attribute('logging',          priority: 20).use { Logging }
+    attribute('redis',            priority: 20).use { Redis }
+    attribute('postgresql',       priority: 20).use { Postgresql }
+    attribute('unicorn',          priority: 20).use { Unicorn }
+    attribute('mailroom',         priority: 20).use { IncomingEmail }
+    attribute('mattermost',       priority: 20).use { GitlabMattermost }
+    attribute('gitlab_pages',     priority: 20).use { GitlabPages }
+    attribute('prometheus',       priority: 20).use { Prometheus }
+    attribute('nginx',            priority: 40).use { Nginx } # Parse nginx last so all external_url are parsed before it
+    attribute('external_url',            default: nil)
+    attribute('registry_external_url',   default: nil)
+    attribute('mattermost_external_url', default: nil)
+    attribute('pages_external_url',      default: nil)
+    attribute('runtime_dir',             default: nil)
+    attribute('bootstrap')
+    attribute('omnibus_gitconfig')
+    attribute('manage_accounts')
+    attribute('manage_storage_directories')
+    attribute('user')
+    attribute('gitlab_ci')
+    attribute('sidekiq')
+    attribute('mattermost_nginx')
+    attribute('pages_nginx')
+    attribute('registry_nginx')
+    attribute('remote_syslog')
+    attribute('logrotate')
+    attribute('high_availability')
+    attribute('web_server')
+    attribute('gitaly')
+    attribute('node_exporter')
+    attribute('redis_exporter')
+    attribute('postgres_exporter')
+    attribute('gitlab_monitor')
+    attribute('prometheus_monitoring')
+    attribute('pgbouncer')
+    attribute('sentinel')
+  end
+end

files/gitlab-cookbooks/package/libraries/services.rb → files/gitlab-cookbooks/package/libraries/config/services.rb

@@ -14,7 +14,7 @@
 # limitations under the License.
 #
-require_relative 'helpers/services_helper.rb'
+require_relative '../helpers/services_helper.rb'
 module Services
   # Define the services included in every GitLab Edition

files/gitlab-cookbooks/package/libraries/helpers/settings_helper.rb

+#
+# Copyright:: Copyright (c) 2017 GitLab Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+require 'mixlib/config'
+require 'chef/json_compat'
+require 'chef/mixin/deep_merge'
+require 'securerandom'
+require 'uri'
+
+require_relative '../config_mash.rb'
+
+module SettingsHelper
+  def self.extended(base)
+    # Setup getter/setters for roles and settings
+    class << base
+      attr_accessor :roles
+      attr_accessor :settings
+    end
+
+    base.roles = {}
+    base.settings = {}
+  end
+
+  # Change the default root location for node attributes
+  # Pass in the root (ie 'gitlab') and a block containing the attributes that should
+  # use that root.
+  # ex:
+  #   attribute_block('example') do
+  #     attribute('some_attribute')
+  #   end
+  #   This will convert Gitlab['some_attribute'] to node['example']['some-attribute']
+  def attribute_block(root = nil)
+    return unless block_given?
+    begin
+      @_default_parent = root
+      yield
+    ensure
+      @_default_parent = nil
+    end
+  end
+
+  # Create a new role with the given 'name' config
+  # Roles are configured as Gitlab['<name>_role'] and are added to the node as node['roles']['<name>']
+  # ex: some_specific_role['enable'] = true
+  #     will result in Gitlab['some_specific_role']['enable'] = true
+  #     and node['roles']['some-specific']['enable'] = true
+  def role(name, **config)
+    @roles[name] = HandledHash.new.merge!(config)
+    send("#{name}_role", Gitlab::ConfigMash.new)
+    @roles[name]
+  end
+
+  # Create a new attribute with the given 'name' and config
+  #
+  # config options are:
+  #  parent   - String name for the root node attribute, default can be specified using the attribute_block method
+  #  priority - Integer used to sort the settings when applying them, defaults to 20, similar to sysvinit startups. Lower numbers are loaded first.
+  #  ee       - Boolean to indicate that the variable should only be used in GitLab EE
+  #  default  - Default value to set for the Gitlab Config. Defaults to Gitlab::ConfigMash.new, should be set to nil config expecting non hash values
+  #
+  # ex: attribute('some_attribute', parent: 'gitlab', sequence: 10, default: nil)
+  #     will right away set Gitlab['some_attribute'] = nil
+  #     and when the config is generated it will set node['gitlab']['some-attribute'] = nil
+  def attribute(name, **config)
+    @settings[name] = HandledHash.new.merge!(
+      { parent: @_default_parent, priority: 20, ee: false, default: Gitlab::ConfigMash.new }
+    ).merge(config)
+
+    send(name.to_sym, @settings[name][:default])
+    @settings[name]
+  end
+
+  # Same as 'attribute' but defaults 'enable' to false if the GitlabEE module is unavailable
+  def ee_attribute(name, **config)
+    config = { ee: true }.merge(config)
+    attribute(name, **config)
+  end
+
+  def from_file(_file_path)
+    # Allow auto mash creation during from_file call
+    Gitlab::ConfigMash.auto_vivify { super }
+  end
+
+  def method_missing(method_name, *arguments) # rubocop:disable Style/MethodMissing
+    # Give better message for NilClass errors
+    # If there are no arguements passed, this is a 'GET' call, and if
+    # there is no matching key in the configuration, then it has not been set (not even to nil)
+    # and we will output a nicer error above the exception
+    if arguments.length.zero? && !configuration.key?(method_name)
+      breaktxt = '=' * 80
+      message = "Encountered unsupported config key '#{method_name}' in /etc/gitlab/gitlab.rb."
+      puts "\n#{breaktxt}\n  ERROR: #{message}\n#{breaktxt}\n"
+      Chef::Log.error(message)
+    end
+
+    # Parent method_missing takes care of setting values for missing methods
+    super
+  end
+
+  def hyphenate_config_keys
+    results = { "gitlab" => {}, "roles" => {} }
+
+    # Add the settings to the results
+    sorted_settings.each do |key, value|
+      raise "Attribute parent value invalid" if value[:parent] && !results.key?(value[:parent])
+      target = value[:parent] ? results[value[:parent]] : results
+
+      rkey = key.tr('_', '-')
+      target[rkey] = Gitlab[key]
+    end
+
+    # Add the roles the the results
+    @roles.each do |key, value|
+      rkey = key.tr('_', '-')
+      results['roles'][rkey] = Gitlab["#{key}_role"]
+    end
+
+    results
+  end
+
+  def generate_secrets(node_name)
+    # guards against creating secrets on non-bootstrap node
+    SecretsHelper.read_gitlab_secrets
+
+    # Parse secrets using the handlers
+    sorted_settings.each do |_key, value|
+      handler = value.handler
+      handler.parse_secrets if handler && handler.respond_to?(:parse_secrets)
+    end
+
+    SecretsHelper.write_to_gitlab_secrets
+  end
+
+  def generate_config(node_name)
+    generate_secrets(node_name)
+
+    # Parse all our variables using the handlers
+    sorted_settings.each do |_key, value|
+      handler = value.handler
+      handler.parse_variables if handler && handler.respond_to?(:parse_variables)
+    end
+
+    # Load our roles
+    @roles.each do |_key, value|
+      handler = value.handler
+      handler.load_role if handler && handler.respond_to?(:load_role)
+    end
+
+    # The last step is to convert underscores to hyphens in top-level keys
+    hyphenate_config_keys
+  end
+
+  private
+
+  # Sort settings by their sequence value
+  def sorted_settings
+    @settings.select { |_k, value| !value[:ee] || Gitlab['edition'] == :ee }.sort_by { |_k, value| value[:priority] }
+  end
+
+  # Custom Hash object used to add a handler as a block to the attribute
+  class HandledHash < Hash
+    attr_writer :handler
+
+    def use(&block)
+      @handler = block
+      self
+    end
+
+    def handler
+      @handler = @handler.call if @handler&.respond_to?(:call)
+      @handler
+    end
+  end
+end

spec/chef/recipes/config_spec.rb

@@ -34,7 +34,7 @@ describe 'gitlab::config' do
     expect do
       Gitlab.instance_eval('abc["def"]["hij"] = "top-level"')
     end.to raise_error(NoMethodError).and(
-      output(/\*ERROR\*: Encountered unsupported config key/).to_stdout
+      output(/ERROR: Encountered unsupported config key/).to_stdout
     )
   end

spec/chef_helper.rb

@@ -6,7 +6,7 @@ require 'knapsack'
 Knapsack::Adapters::RSpecAdapter.bind if ENV['USE_KNAPSACK']
 # Load our cookbook libraries so we can stub them in our tests
-Dir[File.join(__dir__, '../files/gitlab-cookbooks/package/libraries/*.rb')].each { |f| require f }
+Dir[File.join(__dir__, '../files/gitlab-cookbooks/package/libraries/**/*.rb')].each { |f| require f }
 Dir[File.join(__dir__, '../files/gitlab-cookbooks/gitlab/libraries/*.rb')].each { |f| require f }
 Dir[File.join(__dir__, '../files/gitlab-cookbooks/gitlab-ee/libraries/*.rb')].each { |f| require f }

spec/libraries/gitlab_spec.rb

+require 'chef_helper'
+
+describe Gitlab do
+  context 'when using an attribute_block' do
+    it 'sets top level attributes to the provided root' do
+      Gitlab.attribute_block('gitlab') do
+        expect(Gitlab.attribute('test_attribute')[:parent]).to eq 'gitlab'
+      end
+      expect(Gitlab['test_attribute']).not_to be_nil
+      expect(Gitlab.hyphenate_config_keys['gitlab']).to include('test-attribute')
+    end
+  end
+
+  it 'sets top level attributes when no parent is provided' do
+    Gitlab.attribute('test_attribute')
+    expect(Gitlab['test_attribute']).not_to be_nil
+    expect(Gitlab.hyphenate_config_keys).to include('test-attribute')
+  end
+
+  it 'properly defines roles' do
+    Gitlab.role('test_node')
+    expect(Gitlab['test_node_role']).not_to be_nil
+    expect(Gitlab.hyphenate_config_keys['roles']).to include('test-node')
+  end
+
+  it 'supports overriding attribute default configuration' do
+    attribute = Gitlab.attribute('test_attribute', parent: 'example', priority: 40, enable: false, default: '')
+    expect(Gitlab['test_attribute']).to eq('')
+    expect(attribute).to include(parent: 'example', priority: 40, enable: false)
+  end
+
+  it 'disables ee attributes when EE is not enabled' do
+    allow(Gitlab).to receive(:[]).and_call_original
+    allow(Gitlab).to receive(:[]).with('edition').and_return(:ce)
+    expect(Gitlab.ee_attribute('test_attribute')[:ee]).to eq true
+    expect(Gitlab['test_attribute']).not_to be_nil
+    expect(Gitlab.hyphenate_config_keys).not_to include('test-attribute')
+  end
+
+  it 'enables ee attributes when EE is enabled' do
+    allow(Gitlab).to receive(:[]).and_call_original
+    allow(Gitlab).to receive(:[]).with('edition').and_return(:ee)
+    expect(Gitlab.ee_attribute('test_attribute')[:ee]).to eq true
+    expect(Gitlab['test_attribute']).not_to be_nil
+    expect(Gitlab.hyphenate_config_keys).to include('test-attribute')
+  end
+
+  it 'sorts attributes by sequence' do
+    Gitlab.attribute('last', priority: 99)
+    Gitlab.attribute('other1')
+    Gitlab.attribute('first', priority: -99)
+    Gitlab.attribute('other2')
+
+    expect(Gitlab.send(:sorted_settings).first[0]).to eq 'first'
+    expect(Gitlab.send(:sorted_settings).last[0]).to eq 'last'
+  end
+
+  it 'filters ee settings when sorting' do
+    Gitlab.attribute('test_attribute1')
+    Gitlab.attribute('test_attribute2', ee: true)
+    allow(Gitlab).to receive(:[]).and_call_original
+    allow(Gitlab).to receive(:[]).with('edition').and_return(:ce)
+    expect(Gitlab.send(:sorted_settings).map(&:first)).to include('test_attribute1')
+    expect(Gitlab.send(:sorted_settings).map(&:first)).not_to include('test_attribute2')
+  end
+
+  it 'allows passing a block to the attribute use method' do
+    attribute = Gitlab.attribute('test_attribute').use { 'test' }
+    expect(attribute.handler).to eq('test')
+  end
+end