Several recent reports of 404 for recent GitLab pages Deployments

There have been recent reports of GitLab page deployments returning a 404 for the latest deployment

Once this happens already deployed pages seem to still be accessible.

Reports:

• https://gitlab.com/gitlab-com/support-forum/issues/1479
• https://gitlab.com/gitlab-com/support-forum/issues/1471
• https://gitlab.com/gitlab-com/support-forum/issues/1470
• https://gitlab.com/gitlab-com/support-forum/issues/1468
• https://gitlab.com/gitlab-com/support-forum/issues/1464
• https://gitlab.com/gitlab-com/support-forum/issues/1460 (main issue)

It seems that recently the issues subside after a restart of the GitLab-Pages daemons only to return a few days later

/cc @stanhu @cpallares @ayufan

assigned to @jnijhof

@ayufan so it seems the problems were solved by restarting gitlab-pages daemons across the workers. Could it be that for some strange reason the pages daemons are not re-indexing?

assigned to @ayufan

Yes, it seems that we are being affected by that.

Not yet sure why. I asked @nick.thomas to create this issue to improve Pages Daemon: https://gitlab.com/gitlab-org/gitlab-pages/issues/41, but also introduce monitoring: https://gitlab.com/gitlab-org/gitlab-pages/issues/42

mentioned in issue support-forum#1460 (closed)

@markglenfletcher @ayufan Is the short-term solution to scan log messages to verify that the index is not being reloaded?

mentioned in issue support-forum#1464 (closed)

We're getting reports that this is happening again

Maybe this helps: I have to disable and subsequently re-enable the shared runners for any build to pass. Otherwise they get stuck in pending state with the message This build is stuck, because you don't have any active runners that can run this build.

@markglenfletcher FYI, you can see when gitlab-pages last updated by scanning the current and *.s log files:

# zgrep "Updated " /var/log/gitlab/gitlab-pages/current
# zgrep "Updated " /var/log/gitlab/gitlab-pages/@40000000588936452735aa94.s
2017-01-25_10:19:05.68580 worker1 gitlab-pages: 2017/01/25 10:19:05 Updated N domains in 1m58.5310298s Hash: [102 111 114 99 101 45 117 112 100 97 116 101 10]10]

Looks like it is indeed stuck.

strace is showing that the daemon is attempting to read the .update file:

[pid 44689] openat(AT_FDCWD, ".update", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 44689] read(29, "force-update\n", 525) = 13
[pid 44689] openat(AT_FDCWD, ".update", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 44689] <... read resumed> "force-update\n", 525) = 13
[pid 44689] openat(AT_FDCWD, ".update", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 44689] <... read resumed> "force-update\n", 525) = 13
[pid 44689] openat(AT_FDCWD, ".update", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 44689] <... read resumed> "force-update\n", 525) = 13

Is it possible that EE isn't writing to .update anymore?

@zj https://gitlab.com/gitlab-org/gitlab-pages/issues/42 is currently working on monitoring, so we should be able to enable it with next release.

@Bartcardi The message you are seeing seems to be different one that the problem of pages.

It seems that updates does happen, it takes some time to have them in effect, but they do work.

Maybe we should check if everything is working fine on Rails side: That pages are properly unpacked.

mentioned in issue support-forum#1475

mentioned in issue pages/plain-html#3 (closed)

@ayufan I'm not seeing the .update file change automatically. Isn't this a Rails issue?

mentioned in issue support-forum#1484 (closed)

I think we have a Rails scoping bug. I'm not sure why this isn't showing up in Sentry:

irb(main):018:0> update = Projects::UpdatePagesConfigurationService.new(proj); nil
=> nil
irb(main):019:0> update.pages_update_file
NoMethodError: undefined method `pages' for Projects::Settings:Module
        from /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/update_pages_configuration_service.rb:50:in `pages_update_file'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:157:in `pages_update_file'
        from (irb):19
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.7.1/lib/rails/commands/console.rb:110:in `start'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.7.1/lib/rails/commands/console.rb:9:in `start'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.7.1/lib/rails/commands.rb:17:in `<top (required)>'
        from bin/rails:9:in `require'
        from bin/rails:9:in `<main>'

Hack test:

irb(main):009:0> Projects::Settings = Settings 
(irb):9: warning: already initialized constant Projects::Settings
/opt/gitlab/embedded/service/gitlab-rails/app/controllers/projects/settings/integrations_controller.rb:2: warning: previous definition of Settings was here
=> Settings
irb(main):010:0> update.execute
=> {:status=>:success}

mentioned in issue support-forum#1485 (closed)

closed via merge request gitlab-org/gitlab-ee!1134

I believe things are working now with 8.16.3.

mentioned in commit test-nick/gitlab-ce@8249f918

mentioned in commit test-nick/gitlab-ce@44c93f5b

mentioned in commit test-nick/gitlab-ce@fcb9edeb