Projects::NotesController#create is slow

Dev: https://dev.gitlab.org/gitlab/gitlabhq/issues/2463

It takes almost a second. Check for performance issues

Yorick

How was this measured as being slow? Is it slow because of the database, because of Azure, or something else? I took a quick look at the Note model but can't find anything that immediately stands out. The same goes for the controller code. Having said all that I'm not sure if creating notes taking a second is a really big problem, certainly not bigger than improving generic page loading times for example.

Job

it's one of the most done queries, afaik. And it has a very high annoyance ratio, as it's the one thing we hit many times a day. Shame you don't see obvious improvements. I agree that page load in general should be low.

Dmitriy

Is it slow because of the database, because of Azure, or something else? Yorick, its not related to GitLab.com specific. I think it makes a lot of sql queries in https://dev.gitlab.org/gitlab/gitlabhq/blob/master/app/services/notes/create_service.rb.

Having said all that I'm not sure if creating notes taking a second is a really big problem

We render new comment only after POST request succeed. That means after I press Add comment button I need to wait 1+ second to see my comment posted. That's bad for user experience.

cc/ @JobV @dzaporozhets @yorickpeterse

Reassigned to @yorickpeterse

I won't have the time to look into this before 8.3, moving this to 8.4.

Milestone changed to 8.4

I'm looking into https://gitlab.com/gitlab-org/gitlab-ce/issues/3507 first, then I'll take a peek at this one.

Having looked into this there are 3 sources as to why creating notes takes a while:

1. Extracting participants (to send Emails to) takes up most of the time. Moving participants to a table (#3965 (closed) ) will most likely cut down the time quite a bit.
2. The NotificationService class spends about 35% of its time (though this is only about 100 ms for a local issue with 6 participants) in queueing Emails to send. Having looked around it seems ActionMailer has no support for batch processing. The current setup means that the more participants you have, the slower it becomes.
3. The NotificationService class spends the remaining 65% of the time spent in this class in a variety of methods/lines. These bits of code are quite fast individually (considering their purpose), but when added together add up quite a bit. I don't see any easy ways of resolving this without completely rethinking on how to send out Emails. One option would be to run NotificationsService in a Sidekiq worker, but this may complicate things too much.

In other words, I can't seem to find something that I can change now that would have a noticeable impact on the load time.

Timing information from Sherlock:

Time (inclusive) 	 Count 	 Path
205.75 ms        	 1     	 app/services/notification_service.rb
146.64 ms        	 1     	 lib/banzai/reference_extractor.rb
137.83 ms        	 1     	 app/models/concerns/mentionable.rb
128.12 ms        	 1     	 lib/gitlab/reference_extractor.rb
114.21 ms        	 1     	 lib/banzai/pipeline/base_pipeline.rb
110.77 ms        	 1     	 lib/gitlab/sherlock/query.rb
109.79 ms        	 1     	 lib/banzai/renderer.rb
109.48 ms        	 1     	 app/models/concerns/participable.rb
85.53 ms         	 1     	 app/services/notes/create_service.rb
53.78 ms         	 3     	 app/views/projects/notes/_note.html.haml
52.69 ms         	 1     	 lib/gitlab/sherlock/transaction.rb
43.49 ms         	 1     	 lib/banzai/filter/abstract_reference_filter.rb
43.48 ms         	 1     	 lib/gitlab/sherlock/location.rb
43.13 ms         	 1     	 app/controllers/application_controller.rb
42.65 ms         	 1     	 app/controllers/projects/notes_controller.rb
36.11 ms         	 1     	 lib/banzai/filter/reference_filter.rb
32.12 ms         	 1     	 app/models/project.rb
31.36 ms         	 1     	 app/controllers/projects/application_controller.rb
31.0 ms          	 1     	 lib/gitlab/current_settings.rb
20.98 ms         	 4     	 app/views/projects/notes/_edit_form.html.haml
19.15 ms         	 1     	 app/helpers/gitlab_markdown_helper.rb
19.1 ms          	 1     	 lib/banzai.rb
19.02 ms         	 1     	 app/models/project_team.rb
18.25 ms         	 1     	 app/services/event_create_service.rb
18.15 ms         	 1     	 app/models/ability.rb
17.62 ms         	 1     	 app/mailers/notify.rb
17.35 ms         	 1     	 lib/banzai/filter/user_reference_filter.rb
13.67 ms         	 1     	 app/models/note.rb
13.46 ms         	 1     	 lib/banzai/filter/external_issue_reference_filter.rb
10.99 ms         	 1     	 lib/gitlab/note_data_builder.rb

mentioned in issue #3965 (closed)

Added blocked label

Milestone changed to 8.6

@joshfng great idea to make that a background job /cc @dzaporozhets

mentioned in commit 22772871

Status changed to closed

Fixed by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2631

@yorickpeterse feel free to reopen if I missed something

mentioned in merge request !14785