Creating "Schedule trigger" causes http-500 when accessing settings/ci_cd

We're facing the same problem with EE 9.1.0 (non-RC!) - installed using official Omnibus package.

It is also present in the final 9.1.0

added ~19173 Next Patch Release regression labels

Related Sentry issue: https://sentry.gitlap.com/gitlab/gitlabcom/issues/28230

added pipeline label

Is there a chance this will be fixed in 9.1.1?

/cc: @ayufan, @grzesiek

Would be nice. For us it's one of the most awaited features in 9.1

/cc @dosuken123

It appears that Rufus::Scheduler can return objects of a different type from parse

def self.parse(o, opts={})
  opts[:no_error] = true

  parse_cron(o, opts) ||
  parse_in(o, opts) || # covers 'every' schedule strings
  parse_at(o, opts) ||
  fail(ArgumentError.new("couldn't parse \"#{o}\""))
end

This is something we didn't take into account. It currently happens only when schedule configuration string is not a cron-like value.

changed milestone to %9.2

That is strange, because I used '0 1 * * *' as in the default value and still have same problem. This works as expected on Gitlab.com though

@phpb-com Strange, did you use a correct timezone like UTC?

Yes, UTC

The problem occurs when there is an ArgumentError raised in one of these validation in def initialize, see https://github.com/jmettraux/rufus-scheduler/blob/e9a6c8eb0a55bd6cd8f4a18afa98521d7efeaeb8/lib/rufus/scheduler/cronline.rb#L49

It also appears that we are using a really old version of that gem, which is not good!

Sorry for late reply. I'll dig in possible fix from now.

Omnibus docker CE 9.1.0 c3f0f143 worked properly. Should I offer a SQL query to remove the all scheduled triggers, at first? I assume that getting 500 when access settings/ci_cd is really painful.

It is still possible to access trigger configuration page to disable it, so, at least for me, there is no pain and I can remove it. The pain is that I cannot use it in my installation from source, and trying to figure-out what is missing.

This is something we didn't take into account. It currently happens only when schedule configuration string is not a cron-like value.

e.g. Cron: every 3h CronTimeZone: UTC (Rufus::Scheduler.parse("every 3h UTC")) gives 500. Time will be returned instead of CronLine.

@phpb-com Is there any string-based Cron in your configuration beside 0 1 * * *? If you have only 0 1 * * * and getting 500 in settings/ci_cd page, then it may be a different problem.

I have only cron like value as in the example '0 1 * * *' and UTC timezone

@phpb-com Thank you in advance for the information! I wonder if you are wrapping the 0 1 * * * by '(quotations), because I noticed that also causes 500. e.g.

This should be

I am on the iPhone so no backticks, and yes I am not using single quotes, it is exactly like you have in the second screenshot.

Maybe something is missing in non docker omnibus install steps, since @pnu is having issues with GL-EE

I am using https://gotfix.com/docker/gitlab docker image myself.

For Next Patch Release, Quick fix would be

Try catch @cron_line.next_time
Add Regex with try_parse_cron

However, before that, I should study rufus-scheduler more.

@phpb-com Oh, alright. I'll dig the problem further. Anyway, I could reproduce on Gitlab.com.

@dosuken123 thanks, I will see if I can catch the elusive difference that makes it work in omnibus docker when used with clean syntax.

But please note that I am using unquoted value and still get 500. So in my case, different to your screenshot, I do not have single or double or any quotes, just clean cron line 0 1 * * *

@phpb-com Yes, I'll dig it as well. You got the error on the source installation. @pnu got on an omnibus image. Hmm. Probably some related configuration is missing...?

You got the error on the source installation. @pnu got on an omnibus image. Hmm. Probably some related configuration is missing...?

This may be related to the Gitlab timezone. NOTE: Gitlab timezone and Cron timezone(for scheduled trigger) are different.

When Gitlab timezone is UTC.

[1] pry(main)> Time.zone
=> #<ActiveSupport::TimeZone:0x007fdde278a038 @name="UTC", @tzinfo=#<TZInfo::TimezoneProxy: Etc/UTC>, @utc_offset=nil>
[2] pry(main)> Rufus::Scheduler.parse("#{Settings.cron_jobs['trigger_schedule_worker']['cron']} #{Time.zone.name}")
=> #<Rufus::Scheduler::CronLine:0x007fdddf8e5138 @days=nil, @hours=#<Set: {0, 12}>, @minutes=#<Set: {0}>, @monthdays=nil, @months=nil, @original="0 */12 * * * UTC", @seconds=[0], @timezone="UTC", @weekdays=nil>

When Gitlab timezone is Eastern Time (US & Canada).

[5] pry(main)> Time.zone
=> #<ActiveSupport::TimeZone:0x007fdde06f5368 @name="Eastern Time (US & Canada)", @tzinfo=#<TZInfo::TimezoneProxy: America/New_York>, @utc_offset=nil>
[6] pry(main)> Rufus::Scheduler.parse("#{Settings.cron_jobs['trigger_schedule_worker']['cron']} #{Time.zone.name}")
=> 2017-04-12 00:00:00 +0900

As you can see, Rufus::Scheduler.parse returns Time instead of Date.

You can check timezone from config/gitlab.yml line#64. e.g.

time_zone: 'Eastern Time (US & Canada)'

This causes 500. Reproduced on my local machine.

@dosuken123 Bingo! Once I set timezone to UTC all started to work. I have my TZ set to Tokyo in the docker config for Gitlab and I was using UTC in the Schedule config. Thanks for finding why it did not work. In any case, I hope you will find solution for making it work in a different TZ setting.

@dosuken123 I also noted that next run time does not correspond to the schedule. If in my gitlab.yml I have trigger scheduled as:

    # Execute scheduled triggers
    trigger_schedule_worker:
      cron: "0 */12 * * *"

Does that mean that my scheduled builds will only be triggered every 12 hours?

If I wanted to have 1 minute granularity, would this:

    # Execute scheduled triggers
    trigger_schedule_worker:
      cron: "*/1 * * * *"

be the correct way to configure it?

Does that mean that my scheduled builds will only be triggered every 12 hours?

Yes

If I wanted to have 1 minute granularity, would this:

Yes

Please also check the scheduled trigger document https://docs.gitlab.com/ce/ci/triggers/README.html#using-scheduled-triggers

Just to inform you.. I've set time_zone in gitlab.rb to 'Berlin':

gitlab_rails['time_zone'] = 'Berlin'

And used UTC in the UI. That led to the 500 in our GL-EE (Omnibus) instance.

@phpb-com In Notes section...

Those triggers won't be executed precicely. Because scheduled triggers are handled by Sidekiq, which runs according to its interval. For exmaple, if you set a trigger to be executed every minute (* * * * *) and the Sidekiq worker performs 00:00 and 12:00 o'clock every day (0 */12 * * *), then your trigger will be executed only 00:00 and 12:00 o'clock every day. To change the Sidekiq worker's frequency, you have to edit the trigger_schedule_worker value in config/gitlab.yml and restart GitLab. The Sidekiq worker's configuration on GiLab.com is able to be looked up at here.

@pnu Thank you for the information!

@pnu you will need to set it to

gitlab_rails['time_zone'] = 'UTC'

to make it work. Then you can set your trigger to Berlin timezone.

What does the time_zone actually do? What does it change?

I fear that this renders all timestamps in the database "invalid"..?!

Another question: Why can i set the time_zone when it destroys features if i do?

@pnu from my experience, changing timezone form one to the other, did not cause any issues.

@pnu The time_zone affects rails Time.zone, which is used in everywhere in Gitlab code bases. Here is the official doc from Gitlab. https://docs.gitlab.com/ce/workflow/timezone.html

@dosuken123 Thanks. In that case it would be a bad idea to change the time_zone parameter.

@pnu Yes, I think you don't need to change time_zone in config files for scheduled trigger. Please don't touch the time_zone(I assume UTC is default) and set your timezone at here.

Berlin syntax seems invalid, please use Europe/Berlin. (You can check syntax at here)

mentioned in merge request !10948 (merged)

gitlab-rake time:zones:all shows me this:

Depending on the docs and this list, Berlin should be perfectly valid. Do i miss something?

Urm. I changed from Berlin to Europe/Berlin in gitlab.rb - despite the docs and my understanding of the list. That fixed the problem somehow. Thanks @dosuken123 !

Still there's another problem: The time that's displayed as the next run time is wrong:

It should be 2017-04-27 11:00:00 +0200..

@pnu

If you want to change Gitlab-omnibus time zone, then please change gitlab_rails['time_zone'] = 'Berlin'.

If you want to use scheduled trigger, then please set Europe/Berlin in Cron Timezone.

This is because the processes of parsing the string-time-zone are different. Anyway, that's a good point! Maybe we should consider aligning timezone syntax.

@dosuken123 That's what i did:

I changed gitlab_rails['time_zone'] from 'Berlin' to 'Europe/Berlin' as you suggested. (and did a gitlab-ctl reconfigure of course) - that fixed the 500!
I set Europe/Berlin in Cron Timezone.

The result can be read here: https://gitlab.com/gitlab-org/gitlab-ce/issues/31274#note_28340358

@pnu That's because of https://gitlab.com/gitlab-org/gitlab-ce/issues/31274#note_28288617. That is described in the scheduled trigger documentation https://docs.gitlab.com/ce/ci/triggers/README.html#using-scheduled-triggers. If you have time, please take a look.

@pnu Oh, sorry, my explanation might be confusing. What you're doing is correct!

@dosuken123 i've tried to set

gitlab_rails['trigger_schedule_worker_cron'] = "*/5 * * * *"

in gitlab.rb. That leads to a restart loop of sidekiq with a strange error message in /var/log/gitlab/sidekiq/current log:

2017-04-27_08:36:23.26949 (<unknown>): did not find expected alphabetic or numeric character while scanning an alias at line 171 column 13

What's wrong with that cron definition?

@pnu I think the cron syntax is correct. Does it work with 0 */12 * * *, but */5 * * * *?

Yep. Works with 0 */12 * * *, but not with */5 * * * *.. :(

I've tried some other options:

0 */12 * * *: works
0 * * * *: works
* * * * *: doesn't work
*/5 * * * *: doesn't work
0,5,10,15,20,25,30,35,40,45,50,55 * * * *: works

@pnu !?!?!? That's weird. I'll figure out why it happens.

Just as an additional datapoint, I have tried */1 * * * * and */5 * * * * on the source installation, and both worked as expected. This might be related to omnibus only.

Maybe a problem in the chef scripts?

@phpb-com Thanks! Maybe so!

@pnu If you're not bothered, I'd like to know what will happen if other worker's cron changed, for exmaple, gitlab_rails['expire_build_artifacts_worker_cron'] = "* * * * *". If it does not get an error, probably some configuration is missing for trigger_schedule_worker_cron.

Yep. Same problem if i change the config like this.

@pnu Thank you for the cooperation! Probably something went wrong in omnibus side as you assumed. I'll create an issue for catch it up.

No problem. You're welcome.

mentioned in issue omnibus-gitlab#2284 (closed)

mentioned in issue #31554 (closed)

mentioned in merge request !10976 (merged)

closed via merge request !10948 (merged)

mentioned in commit d88b9ac7

mentioned in commit 85d5ca37

Creating "Schedule trigger" causes http-500 when accessing settings/ci_cd

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

Designs

Child items ...

Activity

Admin message

Admin message

Creating "Schedule trigger" causes http-500 when accessing settings/ci_cd

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

Activity