Optimize project creation
ZenDesk: https://gitlab.zendesk.com/agent/tickets/78488
Customer is testing the GitLab project endpoints on a local MacOS and AWS. On a local Mac, they are seeing these numbers:
| API Endpoint | Min time |
|---|---|
| POST /api/v4/projects | 1.6 s |
| POST /api/v4/projects/${projectId}/repository/commits | 629 ms |
In both these cases, the work is handled synchronously, so the endpoints block until they finish.
They are using JMeter to test how well GitLab can handle these scenarios. They are finding that as they test with a fixed number of concurrent users (e.g. 100), eventually the response times look like:
They have 5 unicorn workers on a 2-core machine allocated by Kubernetes.
It seems to me that what is happening is that requests get backed up and eventually timeout. For example, if it takes one unicorn worker to create a project a full second, that means on a 5-unicorn worker we can handle up to 5 transactions/s.
Next steps:
- See if we can replicate these numbers with simple POST requests (e.g. using curl).
- Try these requests with JMeter on a test instance.
- Profile the endpoints and see if we can optimize something.
Ideally, it should take less than 100 ms to do either operation. We may consider moving these to background tasks (or even crazier, Workhorse) to improve perceived performance.