Error 502 when loading autocomplete_sources

cc @rspeicher @DouweM

This method causes the timeout when the cache is cold or disabled (as in a test environment). We need a better way to generate Emoji autocompletion. cc @vsizov

Added ~18308 label

@yorickpeterse Would appreciate your input on fixing this, see link in above note. I think we need to generate a static JSON file with all of the Emojis mapped to their relative /assets/*.png paths, and then the GfmAutoComplete javascript should prepend the server's URL.

@rspeicher I agree. Since the built-in emojis don't change between restarts (or after a certain period of time) caching this in Redis is the wrong approach. I agree with generating some list of JSON files (or whatever). Since the list rarely changes I think having to run some Rake task to update the files (as a developer) is the easiest way. Doing this ensures that when people install/run GitLab all the Emoji stuff is right there and good to go.

https://gitlab.com/snippets/15348 is waiting to be added to the project's Rake tasks, I just need to find some time to clean it up. That'd be a perfect place to also generate the JSON file.

Edit: This was added. https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3130

A number of us seem to be hitting this timeout over the last two weeks. It seems that the rake task requires ImageMagick to run. Should we just check in a static file to avoid that dependency?

Please fix this and avoid ImageMagick dependency. Also in future please make sure dev/test environment is not broken for 2 weeks.

Milestone changed to 8.6

We're not running a Rake task during a web request, don't be crazy.

The task isn't even loaded unless the user manually adds two gems to their Gemfile. I said the Rake task would be a good place to generate a static JSON file if we ended up going that route, that's all.

@rspeicher yep sorry. I am quite sensitive about adding ImageMagick . Still 502 in development/test for 2 weeks is something we should not allow.

@rspeicher Random idea: we convert the JSON to a Ruby Hash and just require that file when booting GitLab. This way we don't have to parse any JSON or whatever. You basically get something like https://github.com/YorickPeterse/oga/blob/master/lib/oga/html/entities.rb (this file was generated from a JSON blob).

Milestone changed to 8.7

Random idea: we convert the JSON to a Ruby Hash and just require that file when booting GitLab

@yorickpeterse sounds good to me!

@DouweM I assing this to you since you are Dev lead. Please find someone to fix this issue immediately. 502 errors are not acceptable.

Reassigned to @DouweM

I'll take a look at this.

Reassigned to @yorickpeterse

@yorickpeterse thanks!

mentioned in issue #14423 (closed)

@yorickpeterse please do something with it. 26 days with 502 makes me mad

@rspeicher It appears the autocomplete_emojis method is slow due to Rails' image_url method. For example, this snippet of code takes forever to complete:

AwardEmoji.emojis.map { |name, emoji| { name: name, path: ActionController::Base.helpers.image_url("#{emoji['unicode']}.png") } }

This however is instant:

AwardEmoji.emojis.map { |name, emoji| { name: name, path: "#{emoji['unicode']}.png" } }

I'm not sure what the best way around this would be. It's ridiculous that generating image URLs is so slow but I don't see a way around it as URLs can differ between environments (e.g. because of digests being added). Loading this when booting Rails (without making this faster) is only going to slow down the bootup process (which will annoy developers).

Thinking out loud: one solution to the above notes is to eager load emoji in production when booting up Rails and lazy load it during development. This however does mean the bootup phase will be slower.

@yorickpeterse Interesting. Another thing to consider is that this method has also been the source of some Poltergeist timeouts in the test environment.

The solution should be for both dev/test env too. Otherwise we will get timeouts on tests that open issue page with @javascript.

I am ok with loading it during bootup. At least as temporary solution. Basically any solution will be better then what we have now :(

My current idea is as follows:

Move the image URL generation out of the controller to a separate method, memoize the results in an instance variable.
For production and CI (if we can detect when we're running in a CI environment) we'll call the above method when booting, resulting in the data being eager loaded

This removes the need for generating any Ruby code (parsing of the JSON files isn't the problem) and should make loading the Emoji autocomplete list much faster since it just has to serialize a Ruby Array to JSON.

Move the image URL generation out of the controller to a separate method, memoize the results in an instance variable.

@yorickpeterse how moving image generation to separate method make it work for dev env? If I understood correctly it will cause timeout anyway (at least for first loading)

@dzaporozhets During development one can either increase the Unicorn timeout (if they're using Unicorn) or just use no timeout at all (e.g. bundle exec rails s uses Thin which doesn't have a timeout as far as I can tell). I'm really not a fan of slowing down the bootup phase during development even further, it's already annoyingly slow.

Another solution is that we use a sprite for all the emoji images instead of storing them as individual files. Doing this we only need to generate 1 URL, though we'd have to keep track of the position of every emoji so we can set the right offsets in CSS. The downside of this is having to load one rather big sprite file.

Another solution is that we use a sprite for all the emoji images instead of storing them as individual files.

@yorickpeterse can we make a JSON file with path to images and generate it once via rake task?

During development one can either increase the Unicorn timeout

Really 502 timeouts for autocomplet on dev are bad. But increasing timeout so you can wait 90 seconds for autocomplete load first time is also annoying

After a call with @dzaporozhets we decided to go with the following:

The emoji URLs are in the following format: HOST/assets/EMOJI_NAME-SHA1.png. The Emoji names are known, the SHA1 hashes can be generated based on the image content. Using a Rake task we can generate some JSON that contains the same output we generate dynamically at the moment. During development no SHA1 is appended to the file name but this is something we can quite easily handle.

For production and CI (if we can detect when we're running in a CI environment)

We can. See https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/support/capybara.rb#L6.

During development one can either increase the Unicorn timeout (if they're using Unicorn) or just use no timeout at all (e.g. bundle exec rails s uses Thin which doesn't have a timeout as far as I can tell).

Yep, I set the Unicorn timeout ridiculously high so that it stopped killing my pry debugging sessions.

I'm really not a fan of slowing down the bootup phase during development even further, it's already annoyingly slow.

to https://gitlab.com/gitlab-org/gitlab-ce/issues/14009#note_4500679

mentioned in commit 5830d80b

Status changed to closed by merge request !3458 (merged)

thanks for fixing this!

Yes Thanks a lot!!

Mentioned in commit pfjason/gitlab-ce@5830d80b

mentioned in commit test-nick/gitlab-ce@06252fd5

Error 502 when loading autocomplete_sources

Designs

Child items ...

Activity

Admin message

Admin message

Error 502 when loading autocomplete_sources

Activity