[Docker] "422 - The change you requested was rejected" when using SSL offloading
Problem description:
When an SSL-offloading load balancer service like Elastic Load Balancer from AWS is used, Devise (I think) doesn't seem to understand the endpoint is actually https, and so tries to verify requests based on http, which in turn fails the CSRF token authenticity check. This causes an error when trying to log in (see screen shot below).
Workaround
This can be worked around by manually modifying the generated nginx config in /var/opt/gitlab/nginx/conf/gitlab-http.conf
, and changing all occurrences of
proxy_set_header X-Forwarded-Proto http;
to
proxy_set_header X-Forwarded-Proto https;
However, note that after the next gitlab-ctl reconfigure
these changes will be gone.
Suggested solution
Introduce an extra boolean parameter nginx['ssl_offloaded']
, and modify the offending template to reflect 'https' as X-Forwarded-Proto either when SSL is enabled or SSL is offloaded.
Offending Chef template:
files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb
, lines 121, 146 and 169.
Screen shot
Setup description
Docker:
- version: lxc-docker 1.7.1 (official Docker Ubuntu packages, via https://get.docker.com/ubuntu/)
- image: gitlab/gitlab-ce:latest (as of 5 October, 01:00 GMT)
- ports: 0.0.0.0:2222:22, 0.0.0.0:80:80
gitlab.rb relevant config changes:
- external_url 'https://fqdn.of.gitlab.server'
- nginx['listen_https'] = false
- nginx['listen_port'] = 80
Output of gitlab-ctl show-config
(censored):
note: this was run from within the gitlab container
root@gitlab:~# gitlab-ctl show-config
Starting Chef Client, version 12.4.1
resolving cookbooks for run list: ["gitlab::show_config"]
Synchronizing Cookbooks:
- gitlab
- package
- runit
Compiling Cookbooks...
{
"gitlab": {
"bootstrap": {
},
"omnibus-gitconfig": {
},
"manage-accounts": {
"enable": false
},
"user": {
"git_user_email": "XXX@XXX"
},
"redis": {
},
"ci-redis": {
},
"gitlab-rails": {
"gitlab_email_enabled": true,
"gitlab_email_from": "XXX@XXX",
"gitlab_email_display_name": "Gitlab",
"gitlab_email_reply_to": "noreply@XXX",
"smtp_enable": true,
"smtp_address": "XXX",
"smtp_port": 465,
"smtp_user_name": "XXX",
"smtp_password": "XXX",
"smtp_domain": "XXX",
"smtp_authentication": "login",
"smtp_enable_starttls_auto": true,
"smtp_tls": true,
"secret_token": "XXX",
"gitlab_host": "XXX",
"gitlab_https": true,
"gitlab_port": 443,
"db_username": "gitlab",
"db_host": null,
"db_port": 5432
},
"gitlab-ci": {
"secret_token": null,
"secret_key_base": "XXX",
"db_key_base": "XXX",
"db_username": "gitlab_ci",
"db_host": null,
"db_port": 5432
},
"gitlab-shell": {
"secret_token": "XXX"
},
"unicorn": {
},
"ci-unicorn": {
},
"sidekiq": {
},
"ci-sidekiq": {
},
"gitlab-git-http-server": {
"auth_backend": "http://127.0.0.1:8080"
},
"nginx": {
"listen_https": false,
"listen_port": 80,
"ssl_certificate": "/etc/gitlab/ssl/XXX.crt",
"ssl_certificate_key": "/etc/gitlab/ssl/XXX.key"
},
"ci-nginx": {
"listen_port": 80
},
"mattermost-nginx": {
"listen_port": null
},
"logging": {
},
"remote-syslog": {
},
"logrotate": {
},
"high-availability": {
},
"postgresql": {
"shared_buffers": "1MB"
},
"web-server": {
},
"mattermost": {
"service_invite_salt": "XXX",
"service_public_link_salt": "XXX",
"service_reset_salt": "XXX",
"sql_at_rest_encrypt_key": "XXX",
"sql_data_source": "user=gitlab_mattermost host=/var/opt/gitlab/postgresql port=5432 dbname=mattermost_production",
"sql_data_source_replicas": [
"user=gitlab_mattermost host=/var/opt/gitlab/postgresql port=5432 dbname=mattermost_production"
]
},
"external-url": "https://XXX",
"ci-external-url": null,
"mattermost-external-url": null
}
}
Converging 0 resources
Running handlers:
Running handlers complete
Chef Client finished, 0/0 resources updated in 0.972578983 seconds
AWS Elastic Load Balancer setup:
- c4.2xlarge ec2 instance in VPC
- ELB load balancer with external IP
- TCP:22 -> TCP:2222
- HTTPS:443 -> HTTP:80
- Health check: HTTP:80/users/sign_in
Disclaimer
I am not very good with Chef, or I would have submitted a patch.