Upgrade from 9.1.3 to 9.1.4 Breaks New User LDAP Sign-In

Summary

After upgrading the server from 9.1.3-ce.0 to 9.1.3=4-ce.0, new users (signing in for the first time) could no longer sign in via LDAP.
Instead, they got the error: Could not authenticate you from Ldapmain because "Undefined method 'provider' for nil:nilclass"
Downgrading again fixed the issue instantly, so it is definitely an issue with the upgrade.

Note: I will help has best I can with testing things, but I was forced to keep the downgrade because we currently have a lot of new users (interns) coming in for the summer, and not being able to give them access to repos is not acceptable, so I will have to upgrade and then downgrade again when testing.

Steps to reproduce

Upgrade a Gitlab CE server that uses external LDAP for authentication from 9.1.3 to 9.1.4. Add a new user who has never logged in before to the access group and have them try to log in.

What is the current bug behavior?

User is not logged in and gets an error. They cannot log in no matter how many times they try. However, the user account is created as expected.

What is the expected correct behavior?

User account is created and user is logged in to Gitlab.

Relevant logs and/or screenshots

Screenshot of the error the user sees:

Logs from a failed login attempt by a new user:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/users/auth/ldapmain/callback" for [ip] at 2017-05-18 10:08:05 -0600
Processing by OmniauthCallbacksController#ldapmain as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"redacted", "username"=>"redacted", "password"=>"[FILTERED]"}
Completed 500 Internal Server Error in 207ms (ActiveRecord: 6.0ms)
Processing by OmniauthCallbacksController#failure as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"redacted=", "username"=>"redacted", "password"=>"[FILTERED]"}
Redirected to https://[server]/users/sign_in
Completed 302 Found in 2ms (ActiveRecord: 0.0ms)

==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
I, [2017-05-18T10:08:05.910487 #1730]  INFO -- omniauth: (ldapmain) Callback phase initiated.
E, [2017-05-18T10:08:06.179028 #1730] ERROR -- omniauth: (ldapmain) Authentication failure! ldap_error: NoMethodError, undefined method `provider' for nil:NilClass

==> /var/log/gitlab/gitlab-rails/application.log <==
May 18, 2017 10:08: No existing LDAP account was found in GitLab. Checking for cas3 account.
May 18, 2017 10:08: No user found using cas3 provider. Creating a new one.
May 18, 2017 10:08: Correct account has been found. Adding LDAP identity to user: redacted1.
May 18, 2017 10:08: (OAuth) Error saving user redacted (redacted@email.com): ["Email has already been taken"]
May 18, 2017 10:08: (LDAP) Error saving user uid=redacted (temp-email-for-oauth-user@gitlab.localhost): ["Email has already been taken"]

I checked for users with the same email and there were none before this login attempt.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Git Version:	2.11.1
Sidekiq Version:4.2.7
GitLab information
Version:	9.1.3
Revision:	2e4e522
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://redacted
HTTP Clone URL:	https://redacted/some-group/some-project.git
SSH Clone URL:	git@redacted:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers: cas3
GitLab Shell
Version:	5.0.2
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check - tests all pass

Checking GitLab Shell ...
GitLab Shell version >= 5.0.2 ? ... OK (5.0.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
10/5 ... ok
2/6 ... ok
10/9 ... ok
10/10 ... ok
10/12 ... ok
12/13 ... ok
8/14 ... ok
14/15 ... ok
82/17 ... ok
8/18 ... ok
10/20 ... repository is empty
13/21 ... repository is empty
83/24 ... ok
30/26 ... ok
10/27 ... ok
10/32 ... ok
10/34 ... ok
10/37 ... ok
40/42 ... ok
40/43 ... ok
40/44 ... ok
10/46 ... ok
10/50 ... ok
25/51 ... ok
43/53 ... ok
170/55 ... ok
20/56 ... ok
3/60 ... ok
10/61 ... ok
72/64 ... ok
73/65 ... ok
73/66 ... ok
73/67 ... ok
73/68 ... ok
73/69 ... ok
73/70 ... ok
73/71 ... ok
73/72 ... ok
73/73 ... ok
73/74 ... ok
73/75 ... ok
73/76 ... ok
73/77 ... ok
73/78 ... ok
73/79 ... ok
73/80 ... ok
73/81 ... ok
73/82 ... ok
73/83 ... ok
73/84 ... ok
73/86 ... ok
73/87 ... ok
73/88 ... ok
10/89 ... ok
170/90 ... ok
170/91 ... ok
72/93 ... ok
78/94 ... ok
72/96 ... ok
72/97 ... ok
72/98 ... ok
72/99 ... ok
72/100 ... ok
72/101 ... ok
72/102 ... ok
72/103 ... ok
72/104 ... ok
72/105 ... ok
72/106 ... ok
72/107 ... ok
72/108 ... ok
72/109 ... ok
65/110 ... ok
64/112 ... ok
72/113 ... ok
72/114 ... ok
72/115 ... ok
25/116 ... ok
85/117 ... ok
85/118 ... ok
85/119 ... ok
85/120 ... ok
86/121 ... ok
86/122 ... ok
86/123 ... ok
86/124 ... ok
86/125 ... ok
86/126 ... ok
72/127 ... ok
10/128 ... ok
10/129 ... ok
10/130 ... ok
10/131 ... ok
82/132 ... ok
82/133 ... ok
86/134 ... ok
65/135 ... ok
65/136 ... ok
6/139 ... ok
96/140 ... ok
8/141 ... ok
2/142 ... ok
96/143 ... ok
10/145 ... ok
96/149 ... ok
96/150 ... ok
96/151 ... ok
96/153 ... ok
96/154 ... ok
96/155 ... ok
10/157 ... ok
2/162 ... ok
5/163 ... ok
10/164 ... ok
106/165 ... ok
10/166 ... ok
107/167 ... ok
107/168 ... ok
10/169 ... ok
10/170 ... ok
10/171 ... ok
96/172 ... ok
158/173 ... ok
158/174 ... ok
115/175 ... ok
115/176 ... ok
10/177 ... repository is empty
10/178 ... ok
10/180 ... ok
10/181 ... ok
115/183 ... ok
20/184 ... ok
107/185 ... ok
107/186 ... ok
107/187 ... ok
107/188 ... ok
107/190 ... ok
107/191 ... ok
73/192 ... ok
86/193 ... ok
73/194 ... ok
3/199 ... ok
138/202 ... ok
73/205 ... ok
69/206 ... ok
2/207 ... ok
69/208 ... repository is empty
131/209 ... ok
32/210 ... ok
158/211 ... ok
158/212 ... ok
158/213 ... ok
10/214 ... ok
32/216 ... ok
128/217 ... ok
158/218 ... ok
128/220 ... ok
79/221 ... ok
32/222 ... ok
148/223 ... ok
73/225 ... ok
73/226 ... ok
32/227 ... ok
115/230 ... ok
115/231 ... ok
32/232 ... ok
115/234 ... ok
138/236 ... ok
115/237 ... ok
115/238 ... ok
8/239 ... ok
32/240 ... ok
158/241 ... ok
158/244 ... ok
148/249 ... repository is empty
158/250 ... ok
158/251 ... ok
75/252 ... ok
67/253 ... ok
115/255 ... ok
158/256 ... ok
24/257 ... repository is empty
148/260 ... ok
111/261 ... ok
148/262 ... ok
85/263 ... ok
75/264 ... repository is empty
165/265 ... repository is empty
111/266 ... ok
24/268 ... ok
138/269 ... ok
138/270 ... ok
138/271 ... ok
138/272 ... ok
25/273 ... ok
134/277 ... ok
126/280 ... ok
65/283 ... ok
72/284 ... ok
53/287 ... ok
24/290 ... ok
97/292 ... ok
85/293 ... ok
138/294 ... ok
10/295 ... ok
149/296 ... ok
174/298 ... ok
58/299 ... ok
15/301 ... repository is empty
15/302 ... ok
32/303 ... ok
128/305 ... ok
174/306 ... ok
2/307 ... ok
65/308 ... ok
18/309 ... ok
115/311 ... ok
53/312 ... ok
173/313 ... ok
53/314 ... ok
153/315 ... ok
10/319 ... ok
166/320 ... ok
160/322 ... ok
10/323 ... ok
53/325 ... ok
53/326 ... ok
160/327 ... ok
158/328 ... ok
10/329 ... ok
160/330 ... ok
160/332 ... ok
160/333 ... ok
174/335 ... ok
166/336 ... ok
10/337 ... ok
27/338 ... ok
126/339 ... ok
115/340 ... ok
115/341 ... ok
138/342 ... ok
174/343 ... ok
53/346 ... ok
158/347 ... ok
164/348 ... ok
2/349 ... repository is empty
10/350 ... ok
164/351 ... ok
163/352 ... repository is empty
128/353 ... ok
174/354 ... ok
158/355 ... ok
158/356 ... ok
6/357 ... ok
160/358 ... ok
193/359 ... ok
27/360 ... ok
128/361 ... ok
160/362 ... ok
128/363 ... ok
32/364 ... ok
154/365 ... ok
160/366 ... ok
174/368 ... ok
147/369 ... ok
6/371 ... ok
6/372 ... ok
6/373 ... ok
174/374 ... repository is empty
10/375 ... ok
120/378 ... ok
174/380 ... ok
174/381 ... ok
196/383 ... ok
196/384 ... ok
196/385 ... ok
196/386 ... ok
10/387 ... ok
115/388 ... ok
115/389 ... ok
6/391 ... ok
10/392 ... ok
194/393 ... ok
194/394 ... ok
201/396 ... ok
193/397 ... ok
10/398 ... ok
115/399 ... ok
172/400 ... ok
195/401 ... ok
72/402 ... ok
172/403 ... ok
195/404 ... ok
204/405 ... ok
111/406 ... ok
205/408 ... ok
196/409 ... ok
10/410 ... ok
174/411 ... ok
65/412 ... ok
174/413 ... ok
166/414 ... ok
174/415 ... ok
196/416 ... ok
53/417 ... ok
225/418 ... ok
172/419 ... ok
167/420 ... ok
194/421 ... ok
115/422 ... ok
174/423 ... ok
115/424 ... ok
115/425 ... ok
173/426 ... ok
174/427 ... ok
174/428 ... ok
194/429 ... ok
194/430 ... ok
194/433 ... ok
194/434 ... ok
10/435 ... ok
20/436 ... ok
174/437 ... ok
10/438 ... ok
228/439 ... ok
228/440 ... ok
173/442 ... ok
75/443 ... ok
227/444 ... ok
10/445 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
redacted
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
10/5 ... yes
2/6 ... yes
10/9 ... yes
10/10 ... yes
10/12 ... yes
12/13 ... yes
8/14 ... yes
14/15 ... yes
82/17 ... yes
8/18 ... yes
10/20 ... yes
13/21 ... yes
83/24 ... yes
30/26 ... yes
10/27 ... yes
10/32 ... yes
10/34 ... yes
10/37 ... yes
40/42 ... yes
40/43 ... yes
40/44 ... yes
10/46 ... yes
10/50 ... yes
25/51 ... yes
43/53 ... yes
170/55 ... yes
20/56 ... yes
3/60 ... yes
10/61 ... yes
72/64 ... yes
73/65 ... yes
73/66 ... yes
73/67 ... yes
73/68 ... yes
73/69 ... yes
73/70 ... yes
73/71 ... yes
73/72 ... yes
73/73 ... yes
73/74 ... yes
73/75 ... yes
73/76 ... yes
73/77 ... yes
73/78 ... yes
73/79 ... yes
73/80 ... yes
73/81 ... yes
73/82 ... yes
73/83 ... yes
73/84 ... yes
73/86 ... yes
73/87 ... yes
73/88 ... yes
10/89 ... yes
170/90 ... yes
170/91 ... yes
72/93 ... yes
78/94 ... yes
72/96 ... yes
72/97 ... yes
72/98 ... yes
72/99 ... yes
72/100 ... yes
72/101 ... yes
72/102 ... yes
72/103 ... yes
72/104 ... yes
72/105 ... yes
72/106 ... yes
72/107 ... yes
72/108 ... yes
72/109 ... yes
65/110 ... yes
64/112 ... yes
72/113 ... yes
72/114 ... yes
72/115 ... yes
25/116 ... yes
85/117 ... yes
85/118 ... yes
85/119 ... yes
85/120 ... yes
86/121 ... yes
86/122 ... yes
86/123 ... yes
86/124 ... yes
86/125 ... yes
86/126 ... yes
72/127 ... yes
10/128 ... yes
10/129 ... yes
10/130 ... yes
10/131 ... yes
82/132 ... yes
82/133 ... yes
86/134 ... yes
65/135 ... yes
65/136 ... yes
6/139 ... yes
96/140 ... yes
8/141 ... yes
2/142 ... yes
96/143 ... yes
10/145 ... yes
96/149 ... yes
96/150 ... yes
96/151 ... yes
96/153 ... yes
96/154 ... yes
96/155 ... yes
10/157 ... yes
2/162 ... yes
5/163 ... yes
10/164 ... yes
106/165 ... yes
10/166 ... yes
107/167 ... yes
107/168 ... yes
10/169 ... yes
10/170 ... yes
10/171 ... yes
96/172 ... yes
158/173 ... yes
158/174 ... yes
115/175 ... yes
115/176 ... yes
10/177 ... yes
10/178 ... yes
10/180 ... yes
10/181 ... yes
115/183 ... yes
20/184 ... yes
107/185 ... yes
107/186 ... yes
107/187 ... yes
107/188 ... yes
107/190 ... yes
107/191 ... yes
73/192 ... yes
86/193 ... yes
73/194 ... yes
3/199 ... yes
138/202 ... yes
73/205 ... yes
69/206 ... yes
2/207 ... yes
69/208 ... yes
131/209 ... yes
32/210 ... yes
158/211 ... yes
158/212 ... yes
158/213 ... yes
10/214 ... yes
32/216 ... yes
128/217 ... yes
158/218 ... yes
128/220 ... yes
79/221 ... yes
32/222 ... yes
148/223 ... yes
73/225 ... yes
73/226 ... yes
32/227 ... yes
115/230 ... yes
115/231 ... yes
32/232 ... yes
115/234 ... yes
138/236 ... yes
115/237 ... yes
115/238 ... yes
8/239 ... yes
32/240 ... yes
158/241 ... yes
158/244 ... yes
148/249 ... yes
158/250 ... yes
158/251 ... yes
75/252 ... yes
67/253 ... yes
115/255 ... yes
158/256 ... yes
24/257 ... yes
148/260 ... yes
111/261 ... yes
148/262 ... yes
85/263 ... yes
75/264 ... yes
165/265 ... yes
111/266 ... yes
24/268 ... yes
138/269 ... yes
138/270 ... yes
138/271 ... yes
138/272 ... yes
25/273 ... yes
134/277 ... yes
126/280 ... yes
65/283 ... yes
72/284 ... yes
53/287 ... yes
24/290 ... yes
97/292 ... yes
85/293 ... yes
138/294 ... yes
10/295 ... yes
149/296 ... yes
174/298 ... yes
58/299 ... yes
15/301 ... yes
15/302 ... yes
32/303 ... yes
128/305 ... yes
174/306 ... yes
2/307 ... yes
65/308 ... yes
18/309 ... yes
115/311 ... yes
53/312 ... yes
173/313 ... yes
53/314 ... yes
153/315 ... yes
10/319 ... yes
166/320 ... yes
160/322 ... yes
10/323 ... yes
53/325 ... yes
53/326 ... yes
160/327 ... yes
158/328 ... yes
10/329 ... yes
160/330 ... yes
160/332 ... yes
160/333 ... yes
174/335 ... yes
166/336 ... yes
10/337 ... yes
27/338 ... yes
126/339 ... yes
115/340 ... yes
115/341 ... yes
138/342 ... yes
174/343 ... yes
53/346 ... yes
158/347 ... yes
164/348 ... yes
2/349 ... yes
10/350 ... yes
164/351 ... yes
163/352 ... yes
128/353 ... yes
174/354 ... yes
158/355 ... yes
158/356 ... yes
6/357 ... yes
160/358 ... yes
193/359 ... yes
27/360 ... yes
128/361 ... yes
160/362 ... yes
128/363 ... yes
32/364 ... yes
154/365 ... yes
160/366 ... yes
174/368 ... yes
147/369 ... yes
6/371 ... yes
6/372 ... yes
6/373 ... yes
174/374 ... yes
10/375 ... yes
120/378 ... yes
174/380 ... yes
174/381 ... yes
196/383 ... yes
196/384 ... yes
196/385 ... yes
196/386 ... yes
10/387 ... yes
115/388 ... yes
115/389 ... yes
6/391 ... yes
10/392 ... yes
194/393 ... yes
194/394 ... yes
201/396 ... yes
193/397 ... yes
10/398 ... yes
115/399 ... yes
172/400 ... yes
195/401 ... yes
72/402 ... yes
172/403 ... yes
195/404 ... yes
204/405 ... yes
111/406 ... yes
205/408 ... yes
196/409 ... yes
10/410 ... yes
174/411 ... yes
65/412 ... yes
174/413 ... yes
166/414 ... yes
174/415 ... yes
196/416 ... yes
53/417 ... yes
225/418 ... yes
172/419 ... yes
167/420 ... yes
194/421 ... yes
115/422 ... yes
174/423 ... yes
115/424 ... yes
115/425 ... yes
173/426 ... yes
174/427 ... yes
174/428 ... yes
194/429 ... yes
194/430 ... yes
194/433 ... yes
194/434 ... yes
10/435 ... yes
20/436 ... yes
174/437 ... yes
10/438 ... yes
228/439 ... yes
228/440 ... yes
173/442 ... yes
75/443 ... yes
227/444 ... yes
10/445 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.11.1)
Active users: 141
Checking GitLab ... Finished

Possible fixes

The behavior is almost exactly what is described in this issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/3134
However, the issue is quite old and the fix had no effect.

I tried manually creating the user account (rather than letting Gitlab create it on login) and even that resulted in the same error.

/label bug

changed the description

I didn't think we touched LDAP in 9.1.4.

@markglenfletcher Are you able to reproduce this?

Looking at the information provided it appears that a user with the same UID already exists in the database. redacted1 represents the unique username that we attempted to generate because the original was already taken.

I wonder if the original user was/is missing an LDAP identity

added ~480950 ~17488 ~12980 ldap labels

@markglenfletcher Some extra strange details - the user, as far as I can see, has the LDAP uid field populated correctly. In fact, Gitlab even thinks they are logging in and ups their login count/date of last login. But they still are unable to get past the login page.

There is, as far as I can see, no user with their username, email, or LDAP ID before they attempt to login for the first time (my test cases are brand-new users).

This issue does not affect already existing accounts who have logged in at least once, only people logging in for the first time.

We had the same issue on GitLab Enterprise, updating to 9.0.7-ee.0 with the Omnibus packages. Existing LDAP users are fine, LDAP users logging in for the first time get an error message when before they were fine to.

We looked in logs, and we saw:

May 24, 2017 17:22: (LDAP) Error saving user cn=xxxxxxxxxxxxxxxx (temp-email-for-oauth-xxxxxxxxxxxxxx@gitlab.localhost): ["Email domain is not authorized for sign-up"]

We went to /admin/application_settings, section headed "Sign-up Restrictions". Noted "Sign-up enabled" was off, because we use LDAP right?

But we still had our 2 domains in "Whitelisted domains for sign-ups", 2 of them. So randomly, we added "gitlab.localhost" as a third white listed domain. And it worked!

Now LDAP users who had never signed in before could sign in.

moved to gitlab#9975

Hey! 👋

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue