Improve performance of creating projects
Merge request reports
Activity
Added 1 commit:
- a3af032e - Added benchmark for Projects::CreateService
Added 1 commit:
- 389f3637 - Added concern for bulk inserting database rows
TODO:
-
Check if the bulk insertion of services needs to run any
after_savehooks or not. If these hooks need to be run bulk insertion won't be possible. -
Check if we can maybe use Rugged instead of gitlab-shell for creating repositories, possibly making this process faster.Too much of a fuzz for now
Edited by yorickpeterse-staging-
Check if the bulk insertion of services needs to run any
mentioned in issue #3064 (closed)
@yorickpeterse Did you take a look at https://github.com/zdennis/activerecord-import?
It looks like it was created exactly for what you're building here.@haynes Yes, there were a few things I didn't like about it:
- Looking at the code it jumps all over the place for something that's fairly simple.
- It does weird things such as modifying input (https://github.com/zdennis/activerecord-import/issues/203)
- It doesn't seem terribly actively maintained
Having said all that, I'm still not 100% sure about the bulk importing. Creating the services only takes a relatively small amount of time compared to creating the repository, wiki, etc, while it does come at the cost of not running validations.
I'll let this sit for a day or two to think about it.
Seeing how at least the build succeeds I think this should be good to go, but I'd like an extra pair of eyes on this just in case I missed something.
Last reply by yorickpeterse-staging94 94 @project.team << [current_user, :master, current_user] 95 95 end 96 96 97 @project.update_column(:last_activity_at, @project.created_at) -
@yorickpeterse does it set somewhere else?
-
@DouweM @yorickpeterse @rspeicher if we have performance problem with creating many services maybe we should change logic instead: create service only after user actually edited it and press save button? Each project has ~ 20 service objects created now even if it use 0 of it.
I think that's a good idea because:
- It has a side benefit of allowing admins to configure the service templates and then activating the service with those templated settings even after the project has been created.
- Most projects don't need most of the services, so this spares the database of unnecessary rows.
@DouweM @yorickpeterse @rspeicher if we have performance problem with creating many services maybe we should change logic instead: create service only after user actually edited it and press save button? Each project has ~ 20 service objects created now even if it use 0 of it.
I think this is a good idea.
- Last reply by yorickpeterse-staging
OK so to recap:
- Bulk insertion of services will be replaced with only inserting the services when the user actually activates them.
- The BulkInsertable concern will stay as it's still quite useful (especially for benchmarks that need lots of data).
- I'll keep the index/timestamp and other small changes in this MR since they are still useful on their own.
What's not entirely clear to me is what needs to be done for step 1 other than (probably) removing
Project#build_missing_services. Do we need to do anything extra at all?Edited by yorickpeterse-stagingAdded 167 commits:
Toggle commit listI'm going to scrap the BulkInsertable concern for the time being as well. Having tested it for some of my benchmarks it's not as useful as I thought. Inserting larger chunks of data doesn't seem to have any affect on my benchmarks, at the cost of (apparently) using quite a bit of CPU/memory (probably because it's not a super efficient implementation of bulk insertion).
I'll rebase the concern out of this MR but keep it around separately in case I never need it again.
Added 1 commit:
- 29b3ce56 - Removed extra activity update for new projects
@yorickpeterse Sounds good.
mentioned in commit 6d91ee00
mentioned in issue gitlab#3895
