Skip to content
Snippets Groups Projects
New issue look: Off

ERROR -- omniauth: (ldapmain) Authentication failure! ldap_error: Errno::ECONNRESET, Connection reset by peer @ io_fillbuf - fd:18

    • Closed Issue created by username-removed-1209486 @johneismeier

    Summary

    (Summarize the bug encountered concisely)

    upgrade to 9.4.2 or 9.4.3 for EE only.

    ERROR -- omniauth: (ldapmain) Authentication failure! ldap_error: Errno::ECONNRESET, Connection reset by peer @ io_fillbuf - fd:18

    Steps to reproduce

    (How one can reproduce the issue - this is very important)

    Running 9.4.1 configuration works. Upgrade to 9.4.2 or 9.4.3 and: ERROR -- omniauth: (ldapmain) Authentication failure! ldap_error: Errno::ECONNRESET, Connection reset by peer @ io_fillbuf - fd:18

    Example Project

    (If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report)

    (If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)

    What is the current bug behavior?

    (What actually happens)

    I can't login with our LDAP auth. I know there was discussion and repairs on LDAP for the 9.4.3 release but it didn't work for our configuration.

    What is the expected correct behavior?

    (What you should see instead) I don't know? Please advise? I reverted to 9.4.1 release until this issue is repaired.

    Relevant logs and/or screenshots

    (Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

    Output of checks

    (If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

    Results of GitLab environment info

    Expand for output related to GitLab environment info 
    (For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)
    

    (For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

    Results of GitLab application Check

    Expand for output related to the GitLab application check 
    (For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
    

    (For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
    

    (we will only investigate if the tests are passing)

    Possible fixes

    (If you can, link to the line of code that might be responsible for the problem)

    Edited by username-removed-1144264

    Designs

      Child items ...

      2. Activity

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        noticed that others had issues with CE

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        https://gitlab.com/gitlab-org/gitlab-ce/issues/35752

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        gitlabee.9.4.3.bug.2.Aug.2017

      • username-removed-419655
        username-removed-419655 @markglenfletcher1 ·
        Developer

        @johneismeier Please can you tail the logs to attempt to retrieve a full stack trace for this error?

        sudo gitlab-ctl tail

        Whilst the tail is running please reproduce the error to acquire the relevant log file entries. Please ensure that any sensitive information is removed before posting here.

        /cc @mkozono

      • username-removed-419655 added Platform ldap labels

        added Platform ldap labels

      • username-removed-1144264 changed the description

        changed the description

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        I believe this is unrelated to https://gitlab.com/gitlab-org/gitlab-ce/issues/35752 since in that issue, a user could authenticate but git operations failed.

        This may be more like https://gitlab.com/gitlab-org/gitlab-ce/issues/972, which was a config issue.

        Could you please also post the ldap section of the config file (also being careful for sensitive info)?

      • username-removed-169538
        username-removed-169538 @aplymal ·

        We are also encountering this issue in QA during testing. Is there a workaround? image

        Edited by username-removed-169538
      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @aplymal I don't know of a workaround atm. We need more information to diagnose the problem.

        I wonder if you could post the ldap section of the config file (with sensitive info redacted)?

      • username-removed-169538
        username-removed-169538 @aplymal ·

        gitlab_rails['ldap_enabled'] = true gitlab_rails['active_directory'] = true gitlab_rails['ldap_host'] = '<AD URL>' gitlab_rails['ldap_port'] = 636 ``gitlab_rails['ldap_uid'] = 'sAMAccountName' gitlab_rails['ldap_method'] = 'ssl' gitlab_rails['ldap_bind_dn'] = 'CN=pcgit_qa_svc,OU=Services,OU=Accounts,etc..... ' gitlab_rails['ldap_password'] = '<password>' gitlab_rails['ldap_allow_username_or_email_login'] = true gitlab_rails['ldap_base'] = 'DC=<dc>,DC=<other dc>,DC=com'

        Edited by username-removed-169538
      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        I was finally able to add:

        gitlab_rails['verify_certificates'] = false

        to our gitlab.rb to repair the upgrade issue.

      • username-removed-1209486 closed

        closed

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @johneismeier Thank you for reporting back! The verify_certificates option was added in 9.4.2 to resolve a man-in-the-middle security issue, but it should default to false for the moment. We should be defaulting it to true in 10.0, but I am not sure why it didn't default properly to false. I will need to look into this further.

        @aplymal Does this solve your issue?

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        My issue is solved with verify_certificates set to false. I have a puppet controlled environment that probably over wrote the chef configuration that may have set this to false. My puppet environment has been updated to reflect this assignment to false. I will have to watch for a change going forward when I upgrade to find impact to our puppet/gitlab configuration.

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @johneismeier Oh, what I mean by default to false, is that we currently treat it as false when the verify_certificates key doesn't exist, so the fact that you have to add it would be a regression.

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        Oh. I had to add it (sorry).

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author 
        external_url "<%= @gitlab_external_url %>"
nginx['redirect_http_to_https'] = true
git_data_dir "<%= @git_data_dir %>"
<% if @ldap_host -%>
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_host'] = '<%= @ldap_host %>'
gitlab_rails['ldap_port'] = <%= @ldap_port %>
gitlab_rails['ldap_uid'] = '<%= @ldap_uid %>'
gitlab_rails['ldap_method'] = 'ssl'
gitlab_rails['ldap_bind_dn'] = '<%= @ldap_user %>'
gitlab_rails['ldap_password'] = '<%= @ldap_password %>'
gitlab_rails['ldap_allow_username_or_email_login'] = true
gitlab_rails['ldap_base'] = '<%= @ldap_base %>'
gitlab_rails['ldap_group_base'] = '<%= @group_base %>'
gitlab_rails['ldap_admin_group'] = '<%= @admin_group %>'
gitlab_rails['verify_certificates'] = false
<% end -%>

        is a snippet of the updated embedded ruby code to create the gitlab.rb file

        Are you sure my puppet didn't over ride your chef ?

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        :slight_smile: No need to be sorry, I mean it looks like it's our fault.

        But I still don't know how, since we do this:

        https://gitlab.com/gitlab-org/gitlab-ee/blob/5e93a42b5d3bfda8de54de7b703f52e8dab39ed8/config/initializers/1_settings.rb#L157

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        OK, best of luck then.

        something I noticed today now that 9.4.3 is deployed

        I, [2017-08-09T08:32:06.945581 #9137]  INFO -- : POST http://127.0.0.1:8080/api/v4/internal/allowed 0.02303
E, [2017-08-09T08:32:06.945705 #9137] ERROR -- : API call <POST http://127.0.0.1:8080/api/v4/internal/allowed> failed: 500 => <{"message":"500 Internal Server Error"}>.
I, [2017-08-09T08:32:08.563932 #9148]  INFO -- : POST http://127.0.0.1:8080/api/v4/internal/allowed 0.01815

        does this make sense?

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        Should I open a new issue on the 500 errors ? Should I revert back to 9.4.1 ? thanks.

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        I see more error in the logs?

        2017-08-09_08:32:14.13166 2017-08-09T08:32:14.131Z 8262 TID-orbqdygzo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-08-09_08:32:14.13174 2017-08-09T08:32:14.131Z 8262 TID-orbn7zfjo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-08-09_08:32:14.13174 2017-08-09T08:32:14.131Z 8262 TID-orbqdyghw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:17:in `watchdog'
2017-08-09_08:32:14.13181 2017-08-09T08:32:14.131Z 8262 TID-orbqdygzo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-08-09_08:32:14.13182 2017-08-09T08:32:14.131Z 8262 TID-orbn7zfjo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-08-09_08:32:14.13186 2017-08-09T08:32:14.131Z 8262 TID-orbqdyghw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:26:in `block in safe_thread'
      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        I think I need to revert back to 9.4.1.

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        Fresh login in Incognito mode with chrome is failing so I will revert.

      • username-removed-169538
        username-removed-169538 @aplymal ·

        @mkozono I still get the error even after adding the gitlab_rails['verify_certificates'] = false to the gitlab.rb file.

        I did a reconfigure and restart after I updated the file, still get the same error. Do I need to do something to force the upgrade again? It looks like it upgraded successfully.

        Edited by username-removed-169538
      • username-removed-1209486 reopened

        reopened

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        I had to revert back to 9.4.1 today. With a fresh Incognito logout/login with 9.4.3 with verify_certificates assigned to false I still can not authenticate in EE.

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @johneismeier I found another issue created for what looks like the same problem https://gitlab.com/gitlab-org/gitlab-ce/issues/34890. Please continue the discussion there if you agree, or open a separate issue if it seems to be different.

        Unfortunately I am unaware of a solution for that issue at this time, so reverting is definitely necessary in this case.

        @aplymal Your config looks ok on the surface.

        Do I need to do something to force the upgrade again? It looks like it upgraded successfully.

        I think the upgrade did happen successfully. A reconfigure and restart should be enough for a config change.

        But if you are still experiencing a broken LDAP connection, unfortunately I'd have to suggest reverting for now as well.

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @aplymal Oh! Before you revert, I wonder if you could run sudo gitlab-rake gitlab:check --trace or sudo gitlab-rake gitlab:ldap:check --trace to hopefully get us the full stacktrace.

      • username-removed-169538
        username-removed-169538 @aplymal ·

        @mkozono Sure can.. where would you like me to put that?

        The only thing that looks anything like an error is

        ** Invoke gitlab:ldap:check (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute gitlab:ldap:check Checking LDAP ... Server: ldapmain rake aborted! NoMethodError: undefined method 'to_sym' for nil:NilClass Did you mean? to_s

        Edited by username-removed-169538
      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @aplymal Thanks very much! That error line is a good clue, but I was really hoping for a full stacktrace.

        If you wouldn't mind posting the whole output of the check, just in case, I would appreciate it. Also note, for a block of code or logs, you can wrap the text in three backticks above and three backticks below to put it in a box.

      • username-removed-169538
        username-removed-169538 @aplymal · 
        ** Invoke gitlab:check (first_time)
** Invoke gitlab:gitlab_shell:check (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:gitlab_shell:check
Checking GitLab Shell ...

GitLab Shell version >= 5.3.1 ? ... OK (5.3.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
Unix Server Support / CAB ... ok
OMS / OMSC ... ok
ecommerce / ecommerce-env ... ok
ecommerce / ecommerce-tools ... ok
ecommerce / ecommerce-purge ... ok
ETL / Test ... repository is empty
ProductionControl / CA_DesktopClient_Update ... ok
April Merritt / dware_us ... repository is empty
Demandware / sapient-git-test ... ok
Demandware / sapient-migration ... ok
EnterpriseAPI / test ... repository is empty
OpenSystems / Powershell_Scripts ... ok
ecommerce / ecommerce-nativeapps ... repository is empty
Data_and_Insights / EDW ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

** Invoke gitlab:sidekiq:check (first_time)
** Invoke environment
** Execute gitlab:sidekiq:check
Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

** Invoke gitlab:incoming_email:check (first_time)
** Invoke environment
** Execute gitlab:incoming_email:check
Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

** Invoke gitlab:ldap:check (first_time)
** Invoke environment
** Execute gitlab:ldap:check
Checking LDAP ...

Server: ldapmain
rake aborted!
NoMethodError: undefined method `to_sym' for nil:NilClass
Did you mean?  to_s
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ldap/config.rb:194:in `translate_method'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ldap/config.rb:184:in `encryption_options'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ldap/config.rb:53:in `adapter_options'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ldap/adapter.rb:13:in `open'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:475:in `block in check_ldap'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:471:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:471:in `check_ldap'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:460:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:201:in `block in invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:199:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:199:in `invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:178:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:150:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:115:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:100:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:78:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:75:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-10.5.0/bin/rake:33:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:22:in `load'
/opt/gitlab/embedded/bin/rake:22:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:check => gitlab:ldap:check
      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @aplymal Ah, perfect, the stacktrace is there after all. This is very helpful.

        In your config, could you please try changing this

        gitlab_rails['ldap_method'] = 'ssl'

        to this?

        gitlab_rails['ldap_encryption'] = 'simple_tls'
      • username-removed-169538
        username-removed-169538 @aplymal ·

        @mkozono Changed that line, the error persists.

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        Ok, I was confused as to the existing configuration syntax, then I found this note at the bottom of the LDAP doc:

        Note: If you are using pre GitLab 7.4 configuration syntax like described in the old version README LDAP section be advised that it is deprecated.

        @aplymal It looks to me like that syntax may not work with 9.4.2 (@DouweM am I reading this right?), so what I would do now is move to the new syntax described in the doc: https://docs.gitlab.com/omnibus/settings/ldap.html

        Let me know if you run into any problems.

      • username-removed-169538
        username-removed-169538 @aplymal ·

        Yep I think that worked! I assume there wouldn't be any weirdness, either it lets you in or not, correct? I ran the trace again from above....

        ** Invoke gitlab:ldap:check (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:ldap:check
Checking LDAP ...

Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
..... put in a bunch of stuff here....
Checking LDAP ... Finished
      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @aplymal Awesome!! Yes I don't expect any weirdness.

      • username-removed-1144264 mentioned in issue #3144 (closed)

        mentioned in issue #3144 (closed)

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        Opened https://gitlab.com/gitlab-org/gitlab-ee/issues/3144 to discuss pre-7.4 syntax not working in 9.4.2.

        Edited by username-removed-1144264
      • username-removed-1144264 closed

        closed

      • username-removed-1144264 reopened

        reopened

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        From https://gitlab.com/gitlab-org/gitlab-ee/issues/3144#note_37302945:

        does this mean 9.4.4 with updated syntax should work for my deployment of EE? How and why doesn't this impact my CE deployment?

        @johneismeier Sorry, I should not have closed this issue since I still have not investigated why adding verify_certificates caused your LDAP config to work. The other 500 issue you had on 9.4.3 looked different, and I don't know if 9.4.4 should work for your deployment of EE or not.

        Let's continue the verify_certificates thing.

        Does your CE deployment also use LDAP? And does the LDAP config look like your EE LDAP config?

        If you are able to, please also do the check rake task to get a stacktrace:

        sudo gitlab-rake gitlab:check --trace
      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        Active users: ... 330

        Checking GitLab ... Finished

        ** Execute gitlab:check -bash-4.2$ echo

        ?0−bash−4.2? 0 -bash-4.2
        rpm -qa|grep gitlab gitlab-ce-9.4.3-ce.0.el7.x86_64

        from:

        -bash-4.2$ sudo gitlab-rake gitlab:check --trace ** Invoke gitlab:check (first_time) ** Invoke gitlab:gitlab_shell:check (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute gitlab:gitlab_shell:check Checking GitLab Shell ...

        GitLab Shell version >= 5.3.1 ? ... OK (5.3.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes

        and our CE LDAP is like EE with the snippet below:

        -bash-4.2$ sudo grep ldap gitlab.rb gitlab_rails['ldap_enabled'] = true gitlab_rails['ldap_host'] = 'x.x.x.x' gitlab_rails['ldap_port'] = 999 gitlab_rails['ldap_uid'] = 'sAMAccountName' gitlab_rails['ldap_method'] = 'ssl' gitlab_rails['ldap_bind_dn'] = 'myaccount' gitlab_rails['ldap_password'] = 'mypassword' gitlab_rails['ldap_allow_username_or_email_login'] = true gitlab_rails['ldap_base'] = 'DC=a,DC=b,DC=c' gitlab_rails['ldap_group_base'] = '' gitlab_rails['ldap_admin_group'] = ''

        I did take out what is Dell specific.

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author 
        -bash-4.2$ sudo grep ldap gitlab.rb
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_host'] = 'x.x.x.x'
gitlab_rails['ldap_port'] = 999
gitlab_rails['ldap_uid'] = 'sAMAccountName'
gitlab_rails['ldap_method'] = 'ssl'
gitlab_rails['ldap_bind_dn'] = 'myaccount'
gitlab_rails['ldap_password'] = 'mypassword'
gitlab_rails['ldap_allow_username_or_email_login'] = true
gitlab_rails['ldap_base'] = 'DC=a,DC=b,DC=c'
gitlab_rails['ldap_group_base'] = ''
gitlab_rails['ldap_admin_group'] = ''
      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author 
        Active users: ... 330

Checking GitLab ... Finished

** Execute gitlab:check
-bash-4.2$ echo $?
0
-bash-4.2$ rpm -qa|grep gitlab
gitlab-ce-9.4.3-ce.0.el7.x86_64
-bash-4.2$ cd /etc/gitlab

        tail of the command that seems fine in CE should I push the updated CE version?

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer 
        Active users: ... 330

Checking GitLab ... Finished

** Execute gitlab:check
-bash-4.2$ echo $?
0
-bash-4.2$ rpm -qa|grep gitlab
gitlab-ce-9.4.3-ce.0.el7.x86_64

from:

-bash-4.2$ sudo gitlab-rake gitlab:check --trace
** Invoke gitlab:check (first_time)
** Invoke gitlab:gitlab_shell:check (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:gitlab_shell:check
Checking GitLab Shell ...

GitLab Shell version >= 5.3.1 ? ... OK (5.3.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes

        I believe even if there is a failure, it would just print the error in red, but the check would still run without error. I know it's a huge block to paste in, but perhaps there are some clues in the whole check task output?

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        should I push the updated CE version?

        So this is a separate installation than the EE one? If the LDAP check doesn't complain, then the connection should work fine.

        Regardless, I personally would try the check with the new syntax, and if it works move forward with that.

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        Yes, the ldap check is 9.4.3 CE server where ldap is not a blocker. on EE server I can not upgrade without being blocked by ldap.

        New Syntax? So I really need to embed YAML into the ruby to get the current release with ldap working ?

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        I think we moved to the new config syntax to support multiple LDAP servers. Technically you could write out the YAML in Ruby, but it's easier to read as YAML.

      • username-removed-1144264 mentioned in merge request !2690 (merged)

        mentioned in merge request !2690 (merged)

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @johneismeier I found the likely culprit for your issue. If you'd like to upgrade without waiting for the fix (or you'd like to confirm the fix actually fixes your issue), and you don't mind editing one of the Rails files, please try commenting out the two lines in /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb as shown in this MR https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2690/diffs.

        This should work with the old LDAP config syntax.

        And I should have mentioned before, apologies for the inconvenience of this bug, and thank you for helping debug the issue.

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        @johneismeier Change of approach, if you look at the updated MR, instead of removing those lines, I moved one line to the end of the loop to avoid another issue.

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        Thanks Michael. Do you think this MR will be completed with a release this week?

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        Yes after it is merged into master (which hopefully is today or tomorrow), it will go into the next 9.4 patch release.

      • username-removed-1209486
        username-removed-1209486 @johneismeier ·
        Author

        sounds great! So the 9.4 release I just deploy and use my existing LDAP configuration that is under puppet control without issue? I am hoping to have dev servers soon so that I can test before deployment to all.

        Thanks again for working my issue.

      • username-removed-1144264
        username-removed-1144264 @mkozono ·
        Developer

        So the 9.4 release I just deploy and use my existing LDAP configuration that is under puppet control without issue?

        Yes, the existing LDAP config should work again without issue.

        I'm glad I could help!

      • Douwe Maan closed via merge request !2690 (merged)

        closed via merge request !2690 (merged)

      • Douwe Maan mentioned in commit 8d3330ec

        mentioned in commit 8d3330ec

      • Douwe Maan mentioned in commit 60001dba

        mentioned in commit 60001dba

      • Marin Jankovski mentioned in commit d37a3a57

        mentioned in commit d37a3a57

      Please register or sign in to reply