gitaly issueshttps://staging.gitlab.com/gitlab-org/gitaly/-/issues2017-10-13T20:37:44Zhttps://staging.gitlab.com/gitlab-org/gitaly/-/issues/652dddd2017-10-13T20:37:44ZAndrew Newdigateandrew@gitlab.comddddhttps://staging.gitlab.com/gitlab-org/gitaly/-/issues/651Gitaly does not start2017-10-08T21:05:07Zusername-removed-955034Gitaly does not startHi, I hope I'm posting this in the right place...
after updating my gitlab on arch (via repo) from 9 to 10, Gitaly refuses to start. I have not edited the config.toml at all and it is at its repo default.
This is the error I get:
`INFO...Hi, I hope I'm posting this in the right place...
after updating my gitlab on arch (via repo) from 9 to 10, Gitaly refuses to start. I have not edited the config.toml at all and it is at its repo default.
This is the error I get:
`INFO[0000] Starting Gitaly version="Gitaly, version 0.43.0, built 20171006.023852"
WARN[0000] git path not configured. Using default path resolution resolvedPath=/usr/bin/git
FATA[0000] load config config_path=/etc/gitlab-gitaly/config.toml error="exit status 127; stderr: \"\""`
This is the repo default config.toml:
`# Example Gitaly configuration file
socket_path = "/var/lib/gitlab/sockets/gitlab-gitaly.socket"
# # Optional: listen on a TCP socket. This is insecure (no authentication)
# listen_addr = "localhost:9999"
#
# # Optional: export metrics via Prometheus
# prometheus_listen_addr = "localhost:9236"
#
# # Git executable settings
# [git]
# bin_path = "/usr/bin/git"
[[storage]]
name = "default"
path = "/var/lib/gitlab/repositories"
# # You can optionally configure more storages for this Gitaly instance to serve up
#
# [[storage]]
# name = "other_storage"
# path = "/mnt/other_storage/repositories"
#
# # You can optionally configure Gitaly to output JSON-formatted log messages to stdout
# [logging]
# format = "json"
# # Additionally exceptions can be reported to Sentry
# sentry_dsn = "https://<key>:<secret>@sentry.io/<project>"
# # You can optionally configure Gitaly to record histogram latencies on GRPC method calls
# [prometheus]
# grpc_latency_buckets = [0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]
[gitaly-ruby]
# The directory where gitaly-ruby is installed
dir = "/usr/share/webapps/gitlab-gitaly/ruby"
[gitlab-shell]
# The directory where gitlab-shell is installed
dir = "/usr/share/webapps/gitlab-shell"
# # You can adjust the concurrency of each RPC endpoint
# [[concurrency]]
# rpc = "/gitaly.RepositoryService/GarbageCollect"
# max_per_repo = 1`
Uncommenting `[git] > bin_path` does remove the warning, but Gitaly doesn't give me any indicator as to what's wrong.
How can I find out why the start fails? It does say it's the config, but I can't find the error...https://staging.gitlab.com/gitlab-org/gitaly/-/issues/650Gitaly quota2017-10-07T18:21:14ZSid SijbrandijGitaly quotaIntroduction
1. This is a proposal to implement quota's to Gitaly.
1. It is a generalized version of our per user quota https://gitlab.com/gitlab-org/gitaly/issues/429 (pretty cool that that issue number matches the status code for rate...Introduction
1. This is a proposal to implement quota's to Gitaly.
1. It is a generalized version of our per user quota https://gitlab.com/gitlab-org/gitaly/issues/429 (pretty cool that that issue number matches the status code for rate limiting)
1. It is inspired by Gitmon https://www.youtube.com/watch?v=f7ecUqHxD7o&feature=youtu.be&t=8m37s
1. We need quota's for the file servers because they are stateful, applications servers are stateless and we can autoscale them when there is more demand and spread the load among them, file servers contain specific repositories and have a finite capacity.
1. The fileservers run git operations, these are very CPU, memory, and network intensive. A user consuming a lot of resources causes a service degradation for everyone else.
1. If this was about request rate limiting we could probably use something like Envoy https://github.com/envoyproxy/envoy or Istio https://istio.io/ but we need to measure resource consumption here, a certain request can require 1000x the CPU time of another one.
1. GitHub eventually moved to Spokes https://githubengineering.com/building-resilience-in-spokes/ that had multiple fileservers for the same repository. We will probably not need that. We run networked storage in the cloud where the public cloud provider is responsible for the redundancy of the files. To do cross availability zone failover we'll use GitLab Geo.
Features
1. During every request checks if the account is over quota for a resource, if so Gitaly doesn't do anything and returns a 429 https://httpstatuses.com/429 and logs the request
1. Resources: CPU, memory, and network usage
1. Accounts: repo, user, and client IP address
1. If the user is over quota the Rails application server does a retry later (possibly with exponential backoff) or fails the request.
1. The resource usage of every request is added to the total usage of the user in the last minute.
1. Every Gitaly server has a current maximum limit per resource per account (100 seconds of CPU time per repo per minute).
1. Every second the Gitaly server adjust the limit per resource.
1. If there is low resource usage (<50% CPU) the limit is doubled (up to a maximum).
1. If there is high resource usage (>80% CPU) the limit is halved (down to 1).
1. It is possible to see the current limits for each resource in prometheus.
First Iteration
1. Only measure the most important resource (CPU).
1. Fixed quota per user instead of a dynamically adjusted one.
1. A per repo limit.
Storage options for the user resource consumption
1. In the Go program (simple and fast)
1. In a local Redis (large binary and fast)
1. In a local SQLite (small binary and slow)
1. In a central Redis (slow due to network round trip but you can do real per user/IP address limits instead of per fileserver)
Difference from https://gitlab.com/gitlab-org/gitaly/issues/429 in the first iteration
1. Don't limit the number of requests but limit the resource usage.
1. When over limit don't queue requests but decline them, this prevents memory from blowing up due to a big queue and allows the Rails application to make better decisions (back off or show error).
/cc @edjdev @jacobvosmaer-gitlab @andrewnhttps://staging.gitlab.com/gitlab-org/gitaly/-/issues/649Add a generic RPC that calls a json in, json out ruby method2017-10-06T16:12:15ZJacob Vosmaer (GitLab)Add a generic RPC that calls a json in, json out ruby methodThis is an idea on how to make it easier for developers outside the Gitaly team to build new Git-based features. Once the migration is done, all Git features will have to be implemented in Gitaly. Our current process is quite involved an...This is an idea on how to make it easier for developers outside the Gitaly team to build new Git-based features. Once the migration is done, all Git features will have to be implemented in Gitaly. Our current process is quite involved and requires knowledge of Protobuf, gRPC and Go in addition to Ruby.
Git features run the fastest if they involve just one Gitaly call. So to build interesting new Git features in GitLab, we want to make it easy to add high-level Gitaly calls that use Rugged inside gitaly-ruby. If you need to be a Gitaly expert to add a Gitaly call it becomes a lot harder to ship new features.
What if we create a generic RPC that allows us to call a unary method in gitaly-ruby.
```
service GenericRubyService {
rpc GenericRubyUnaryCall(GenericRubyUnaryCallRequest) returns (GenericRubyUnaryCallResponse) {}
}
message GenericRubyUnaryCallRequest {
Repository repository = 1;
string call_name = 2;
// JSON data decoded by ruby
bytes request = 3;
}
message GenericRubyUnaryCallResponse {
bytes response = 1;
}
```
Then in gitaly-ruby we'd have a hash that maps call names to objects, and the objects have a `perform` method that takes a repository and a Ruby hash.
Then a developer adding a Git feature would add a class in gitaly-ruby, add their class to the 'call name' -> 'object' map, and be able to call it from gitlab-ce.
We can create metrics per 'call name' to see how these things behave. If something has special needs (stream responses, a Go implementation) we can create a custom RPC later.
This is just a sketch, I hope it's not too specific. Came up after discussion with @DouweM .https://staging.gitlab.com/gitlab-org/gitaly/-/issues/648RepositoryService::GetArchive Acceptance testing2017-10-05T22:18:56Zusername-removed-367626RepositoryService::GetArchive Acceptance testing~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/533
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#acceptance-testing-acceptance-testing)
for more informati...~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/533
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#acceptance-testing-acceptance-testing)
for more information on the Acceptance Testing stage of the process.
## Details
- **Feature Toggle Name**: `gitaly_workhorse_archive`
- **GRPC Service**: `RepositoryService::GetArchive`
- **Required Gitaly Version**: `v0.43.0`
- **Required GitLab Version**: `v10.1`
--------------------------------------------------------------------------------
## 1. Preparation
- [ ] **Routes**: what routes use this migration?
1. `/[group]/[project]/repository/[revision]/archive.[zip|tar|tar.bz|tar.gz2]`
## 2. Development Trial
#### Check Dev Server Versions
- [ ] Gitaly: [Gitaly Dev Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-dev)
- [ ] GitLab: https://dev.gitlab.org/help
#### Enable on `dev.gitlab.org`:
- [ ] `!feature-set GITALY_FEATURE_NAME true` in [`#dev-gitlab`](https://gitlab.slack.com/messages/C6WQ87MU3)
Then leave running while monitoring and performing some testing through web, api or SSH.
#### Monitor (initially )
- [ ] **Monitor Grafana** feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-dev&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.GRPC_SERVICE_NAME%2FGRPC_METHOD_NAME)
- [ ] **Check for errors** in [GitLab Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set GITALY_FEATURE_NAME false` in [`#dev-gitlab`](https://gitlab.slack.com/messages/C6WQ87MU3) otherwise leave running and proceed proceed to next stage.
## 3. Staging Trial
#### Check Staging Server Versions
- [ ] Gitaly: [Gitaly Staging Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-staging)
- [ ] GitLab: https://staging.gitlab.com/help
#### Enable on `staging.gitlab.com`
- [ ] `!feature-set GITALY_FEATURE_NAME true` in [`#development`](https://gitlab.slack.com/messages/C02PF508L/)
Then leave running while monitoring for at least **15 minutes** while performing some testing through web, api or SSH.
#### Monitor (at least every 5 minutes, preferably real-time)
- [ ] **Monitor Grafana** feature dashboard on staging: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-nfs-staging&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20hostname:nfs5'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20hostname:nfs5%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Staging Sentry](https://sentry.gitlap.com/gitlab/staginggitlabcom-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.GRPC_SERVICE_NAME%2FGRPC_METHOD_NAME)
- [ ] **Check for errors** in [GitLab Staging Sentry](https://sentry.gitlap.com/gitlab/staginggitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using `!feature-set GITALY_FEATURE_NAME false` in [`#development`](https://gitlab.slack.com/messages/C02PF508L/) otherwise leave running and proceed to next stage.
## 4. Production Server Version Check
- [ ] Gitaly: [Gitaly Production Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-production)
- [ ] GitLab: https://gitlab.com/help
## 5. Initial Impact Check
- [ ] Create an issue in the infrastructure tracker: [Create issue now](https://gitlab.com/gitlab-com/infrastructure/issues/new?issue[title]=Testing%20of%20Gitaly%20Feature%20GITALY_FEATURE_NAME&issue[description]=https%3A%2F%2Fgitlab.com%2Fgitlab-org%2Fgitaly%2Fissues%2FACCEPTANCE_TEST_ISSUE_NUMBER%0A%0A%2Flabel%20~gitaly%20~change)
- [ ] Set Gitaly to 1% using the command `!feature-set GITALY_FEATURE_NAME 1` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **15 minutes** while performing some testing through web, api or SSH.
#### Monitor (at least every 5 minutes, preferably real-time)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.GRPC_SERVICE_NAME%2FGRPC_METHOD_NAME)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set GITALY_FEATURE_NAME false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 6. Low Impact Trial
- [ ] Set Gitaly to 5% using the command `!feature-set GITALY_FEATURE_NAME 5` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **2 hours**.
#### Monitor (at least every 20 minutes)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.GRPC_SERVICE_NAME%2FGRPC_METHOD_NAME)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set GITALY_FEATURE_NAME false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 7. Mid Impact Trial
- [ ] Set Gitaly to 50% using the command `!feature-set GITALY_FEATURE_NAME 50` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **24 hours**.
#### Monitor (at least every few hours)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.GRPC_SERVICE_NAME%2FGRPC_METHOD_NAME)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set GITALY_FEATURE_NAME false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 8. Full Impact Trial
- [ ] Set Gitaly to 100% using the command `!feature-set GITALY_FEATURE_NAME 100` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **1 week**.
#### Monitor (at least every day)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.GRPC_SERVICE_NAME%2FGRPC_METHOD_NAME)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Success?
- [ ] Close this issue and mark the ~Conversation as ~"Migration:Opt-In"https://staging.gitlab.com/gitlab-org/gitaly/-/issues/647Namespace exists2017-10-06T07:13:43ZZeger-Jan van de Wegzegerjan@gitlab.comNamespace exists~Conversation: #385
## Details
- **Feature Toggle Name**: `namespace_exists`
- **GRPC Service**: `NamespaceService::NamespaceExists`
- **Required Gitaly Version**: `v0.42.0`
- **Required GitLab Version**: `v10.1`
----------------------...~Conversation: #385
## Details
- **Feature Toggle Name**: `namespace_exists`
- **GRPC Service**: `NamespaceService::NamespaceExists`
- **Required Gitaly Version**: `v0.42.0`
- **Required GitLab Version**: `v10.1`
--------------------------------------------------------------------------------
## 1. Preparation
- [X] **Routes**: what routes use this migration?
1. Creating a new namespace (user/project)
2. Removing a namespace (user/project)
## 2. Development Trial
#### Check Dev Server Versions
- [X] Gitaly: [Gitaly Dev Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-dev)
- [ ] GitLab: https://dev.gitlab.org/help -- expected 6th October
#### Enable on `dev.gitlab.org`:
- [ ] `!feature-set namespace_exists true` in [`#dev-gitlab`](https://gitlab.slack.com/messages/C6WQ87MU3)
Then leave running while monitoring and performing some testing through web, api or SSH.
#### Monitor (initially)
- [ ] **Monitor Grafana** feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=NamespaceExists&var-job=gitaly-dev&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [NamespaceExists invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:NamespaceExists%20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [NamespaceExists errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:NamespaceExists%20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set namespace_exists false` in [`#dev-gitlab`](https://gitlab.slack.com/messages/C6WQ87MU3) otherwise leave running and proceed proceed to next stage.
## 3. Staging Trial
#### Check Staging Server Versions
- [ ] Gitaly: [Gitaly Staging Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-staging)
- [ ] GitLab: https://staging.gitlab.com/help
#### Enable on `staging.gitlab.com`
- [ ] `!feature-set namespace_exists true` in [`#development`](https://gitlab.slack.com/messages/C02PF508L/)
Then leave running while monitoring for at least **15 minutes** while performing some testing through web, api or SSH.
#### Monitor (at least every 5 minutes, preferably real-time)
- [ ] **Monitor Grafana** feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=AddNamespace&var-job=gitaly-dev&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:NamespaceExists%20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FNamespaceExists)
- [ ] **Check for errors** in [GitLab Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using `!feature-set GITALY_FEATURE_NAME false` in [`#development`](https://gitlab.slack.com/messages/C02PF508L/) otherwise leave running and proceed to next stage.
## 4. Production Server Version Check
- [ ] Gitaly: [Gitaly Production Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-production)
- [ ] GitLab: https://gitlab.com/help
## 5. Initial Impact Check
- [ ] Create an issue in the infrastructure tracker: [Create issue now](https://gitlab.com/gitlab-com/infrastructure/issues/new?issue[title]=Testing%20of%20Gitaly%20Feature%20GITALY_FEATURE_NAME&issue[description]=https%3A%2F%2Fgitlab.com%2Fgitlab-org%2Fgitaly%2Fissues%2FACCEPTANCE_TEST_ISSUE_NUMBER%0A%0A%2Flabel%20~gitaly%20~change)
- [ ] Set Gitaly to 1% using the command `!feature-set GITALY_FEATURE_NAME 1` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **15 minutes** while performing some testing through web, api or SSH.
#### Monitor (at least every 5 minutes, preferably real-time)
- [ ] **Monitor Grafana** feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=AddNamespace&var-job=gitaly-dev&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set add_namespace false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 6. Low Impact Trial
- [ ] Set Gitaly to 5% using the command `!feature-set add_namespace 5` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **2 hours**.
#### Monitor (at least every 20 minutes)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [GRPC_METHOD_NAME invocations, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.GRPC_SERVICE_NAME%2FGRPC_METHOD_NAME)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set GITALY_FEATURE_NAME false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 7. Mid Impact Trial
- [ ] Set Gitaly to 50% using the command `!feature-set namespace_exists 50` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **24 hours**.
#### Monitor (at least every few hours)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=NamespaceExists&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [NamespaceExists invocations, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:NamespaceExists%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [NamespaceExists errors, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:NamespaceExists%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FNamespaceExists)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set GITALY_FEATURE_NAME false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 8. Full Impact Trial
- [ ] Set Gitaly to 100% using the command `!feature-set GITALY_FEATURE_NAME 100` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **1 week**.
#### Monitor (at least every day)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [NamespaceExists invocations, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:NamespaceExists%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [GRPC_METHOD_NAME errors, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:NamespaceExists%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FNamespaceExists)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Success?
- [ ] Close this issue and mark the ~Conversation as ~"Migration:Opt-In"https://staging.gitlab.com/gitlab-org/gitaly/-/issues/646Add namespace2017-10-06T13:18:59ZZeger-Jan van de Wegzegerjan@gitlab.comAdd namespace~Conversation: #385
## Details
- **Feature Toggle Name**: `add_namespace`
- **GRPC Service**: `Namespace::AddNamespace`
- **Required Gitaly Version**: `v0.42.0`
- **Required GitLab Version**: `v10.1`
-----------------------------------...~Conversation: #385
## Details
- **Feature Toggle Name**: `add_namespace`
- **GRPC Service**: `Namespace::AddNamespace`
- **Required Gitaly Version**: `v0.42.0`
- **Required GitLab Version**: `v10.1`
--------------------------------------------------------------------------------
## 1. Preparation
- [X] **Routes**: what routes use this migration?
1. Creating a new namespace (user/group)
1. Removing a namespace (user/group)
## 2. Development Trial
#### Check Dev Server Versions
- [X] Gitaly: [Gitaly Dev Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-dev)
- [X] GitLab: https://dev.gitlab.org/help
#### Enable on `dev.gitlab.org`:
- [ ] `!feature-set gitaly_add_namespace true` in [`#dev-gitlab`](https://gitlab.slack.com/messages/C6WQ87MU3)
Then leave running while monitoring and performing some testing through web, api or SSH.
#### Monitor (initially)
- [ ] **Monitor Grafana** feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=AddNamespace&var-job=gitaly-dev&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [AddNamespace invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [AddNamespace errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set GITALY_FEATURE_NAME false` in [`#dev-gitlab`](https://gitlab.slack.com/messages/C6WQ87MU3) otherwise leave running and proceed proceed to next stage.
## 3. Staging Trial
#### Check Staging Server Versions
- [ ] Gitaly: [Gitaly Staging Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-staging)
- [ ] GitLab: https://staging.gitlab.com/help
#### Enable on `staging.gitlab.com`
- [ ] `!feature-set gitaly_add_namespace true` in [`#development`](https://gitlab.slack.com/messages/C02PF508L/)
Then leave running while monitoring for at least **15 minutes** while performing some testing through web, api or SSH.
#### Monitor (at least every 5 minutes, preferably real-time)
- [ ] **Monitor Grafana** feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=AddNamespace&var-job=gitaly-dev&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [AddNamespace invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [AddNamespace errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using `!feature-set gitaly_add_namespace false` in [`#development`](https://gitlab.slack.com/messages/C02PF508L/) otherwise leave running and proceed to next stage.
## 4. Production Server Version Check
- [ ] Gitaly: [Gitaly Production Version Tracker Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1&var-job=gitaly-production)
- [ ] GitLab: https://gitlab.com/help
## 5. Initial Impact Check
- [ ] Create an issue in the infrastructure tracker: [Create issue now](https://gitlab.com/gitlab-com/infrastructure/issues/new?issue[title]=Testing%20of%20Gitaly%20Feature%20AddNamespace&issue[description]=https%3A%2F%2Fgitlab.com%2Fgitlab-org%2Fgitaly%2Fissues%2FACCEPTANCE_TEST_ISSUE_NUMBER%0A%0A%2Flabel%20~gitaly%20~change)
- [ ] Set Gitaly to 1% using the command `!feature-set gitaly_add_namespace 1` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **15 minutes** while performing some testing through web, api or SSH.
#### Monitor (at least every 5 minutes, preferably real-time)
- [ ] **Monitor Grafana** feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=AddNamespace&var-job=gitaly-dev&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [AddNamespace invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [AddNamespace errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set add_namespace false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 6. Low Impact Trial
- [ ] Set Gitaly to 5% using the command `!feature-set add_namespace 5` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **2 hours**.
#### Monitor (at least every 20 minutes)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GRPC_METHOD_NAME&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [AddNamespace invocations, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [AddNamespace errors, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set gitaly_add_namespace false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 7. Mid Impact Trial
- [ ] Set Gitaly to 50% using the command `!feature-set gitaly_add_namespace 50` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **24 hours**.
#### Monitor (at least every few hours)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=AddNamespace&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [AddNamespace invocations, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [AddNamespace errors, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Continue?
- [ ] On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with `!feature-set gitaly_add_namespace false` in [`#production`](https://gitlab.slack.com/messages/C101F3796/) otherwise leave running and proceed to next stage.
## 8. Full Impact Trial
- [ ] Set Gitaly to 100% using the command `!feature-set gitaly_add_namespace 100` in [`#production`](https://gitlab.slack.com/messages/C101F3796/)
Then leave running while monitoring for at least **1 week**.
#### Monitor (at least every day)
- [ ] **Monitor Grafana** feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=AddNamespace&var-job=gitaly-production&refresh=5m)
- [ ] **Inspect logs** in ELK:
- [AddNamespace invocations, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:GRPC_METHOD_NAME%20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [AddNamespace errors, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:AddNamespace%20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [ ] **Check for errors** in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.Namespace%2FAddNamespace)
- [ ] **Check for errors** in [GitLab Sentry](https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+gitaly)
#### Success?
- [ ] Close this issue and mark the ~Conversation as ~"Migration:Opt-In"https://staging.gitlab.com/gitlab-org/gitaly/-/issues/645Conversation WikiService::WikiFindFile2017-10-05T13:07:08ZJacob Vosmaer (GitLab)Conversation WikiService::WikiFindFile------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Crea...------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation)
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiFindFile`
**Known Client Routes**:
- Clone wiki repo, add image in root, push, load `http://gitlab.example/namespace/repo/wikis/img.png`
------------------------------------------------------------https://staging.gitlab.com/gitlab-org/gitaly/-/issues/644Conversation WikiService::WikiUpdatePage2017-10-05T12:59:55ZJacob Vosmaer (GitLab)Conversation WikiService::WikiUpdatePage------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Crea...------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation)
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiUpdatePage`
**Known Client Routes**:
- https://gitlab.com/gitlab-org/gitaly/wikis/home/edit -> submit
------------------------------------------------------------https://staging.gitlab.com/gitlab-org/gitaly/-/issues/643Conversation WikiService::WikiPreviewSlug2017-10-05T12:54:02ZJacob Vosmaer (GitLab)Conversation WikiService::WikiPreviewSlug------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Crea...------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation)
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiPreviewSlug`
**Known Client Routes**:
- https://gitlab.com/gitlab-org/gitaly/wikis/home -> New -> submit
------------------------------------------------------------https://staging.gitlab.com/gitlab-org/gitaly/-/issues/642Conversation WikiService::WikiFindPage2017-10-05T13:54:17ZJacob Vosmaer (GitLab)Conversation WikiService::WikiFindPage------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Crea...------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation)
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiFindPage`
**Known Client Routes**:
- https://gitlab.com/gitlab-org/gitaly/wikis/home (show page)
------------------------------------------------------------
Wraps https://gitlab.com/gitlab-org/gitlab-ce/blob/1fe7a42afca83fbbda638d9569d58a87039cd8f2/lib/gitlab/git/wiki.rb#L55-64 into an RPC.https://staging.gitlab.com/gitlab-org/gitaly/-/issues/641Conversation WikiService::WikiGetAllPages2017-10-05T13:08:00ZJacob Vosmaer (GitLab)Conversation WikiService::WikiGetAllPages------------------------------------------------------------
**Stages**:
- [x] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Crea...------------------------------------------------------------
**Stages**:
- [x] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation)
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiGetAllPages`
**Known Client Routes**:
- https://gitlab.com/gitlab-org/gitaly/wikis/pages
------------------------------------------------------------https://staging.gitlab.com/gitlab-org/gitaly/-/issues/640Conversation WikiService::WikiDeletePage2017-10-05T09:21:59ZJacob Vosmaer (GitLab)Conversation WikiService::WikiDeletePage------------------------------------------------------------
**Stages**:
- [x] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Crea...------------------------------------------------------------
**Stages**:
- [x] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation)
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiDeletePage`
**Known Client Routes**:
- https://gitlab.com/gitlab-org/gitaly/wikis/home/edit
------------------------------------------------------------https://staging.gitlab.com/gitlab-org/gitaly/-/issues/639Conversation WikiService::WikiGetPageVersions2017-10-06T14:10:32ZJacob Vosmaer (GitLab)Conversation WikiService::WikiGetPageVersions------------------------------------------------------------
**Stages**:
- [x] ~"RPC Design": gitlab-org/gitaly-proto!92
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=...------------------------------------------------------------
**Stages**:
- [x] ~"RPC Design": gitlab-org/gitaly-proto!92
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation)
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiGetPageVersions`
**Known Client Routes**:
- https://gitlab.com/gitlab-org/gitaly/wikis/home/history
------------------------------------------------------------Zeger-Jan van de Wegzegerjan@gitlab.comZeger-Jan van de Wegzegerjan@gitlab.comhttps://staging.gitlab.com/gitlab-org/gitaly/-/issues/638WikiWritePage Client Implementation2020-10-23T15:51:00Zusername-removed-506061WikiWritePage Client Implementation~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/635
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#client-implementation-client-implementation)
for more in...~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/635
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#client-implementation-client-implementation)
for more information on the Client Implementation stage of the process.2017-10-11 - Weekly Deliverablesusername-removed-506061username-removed-506061https://staging.gitlab.com/gitlab-org/gitaly/-/issues/637WikiWritePage Server Implementation2017-10-04T16:52:07Zusername-removed-506061WikiWritePage Server Implementation~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/635
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#server-implementation-server-implementation)
for more in...~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/635
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#server-implementation-server-implementation)
for more information on the Server Implementation stage of the process.2017-10-11 - Weekly Deliverablesusername-removed-506061username-removed-506061https://staging.gitlab.com/gitlab-org/gitaly/-/issues/636WikiWritePage RPC2017-10-04T16:52:07Zusername-removed-506061WikiWritePage RPC~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/635
## Expected Artefacts
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#rpc-design-rpc-design)
for more ...~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/635
## Expected Artefacts
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#rpc-design-rpc-design)
for more information on the RPC Design stage of the process.
- [ ] **`grpc-proto` RPC prototypes**: merged change2017-10-11 - Weekly Deliverablesusername-removed-506061username-removed-506061https://staging.gitlab.com/gitlab-org/gitaly/-/issues/635Conversation WikiService::WikiWritePage2017-10-05T09:22:45ZJacob Vosmaer (GitLab)Conversation WikiService::WikiWritePage------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": https://gitlab.com/gitlab-org/gitaly/issues/636
- [ ] ~"Server Implementation": https://gitlab.com/gitlab-org/gitaly/issues/637
- [ ] ~"Cli...------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": https://gitlab.com/gitlab-org/gitaly/issues/636
- [ ] ~"Server Implementation": https://gitlab.com/gitlab-org/gitaly/issues/637
- [ ] ~"Client Implementation": https://gitlab.com/gitlab-org/gitaly/issues/638
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `WikiService::WikiWritePage`
**Known Client Routes**:
- https://gitlab.com/gitlab-org/gitaly/wikis/home
------------------------------------------------------------
See https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/git/wiki.rb for the code to use in gitaly-ruby.username-removed-506061username-removed-506061https://staging.gitlab.com/gitlab-org/gitaly/-/issues/634RPC design OperationService::UserMergeBranch2017-10-04T14:41:00ZJacob Vosmaer (GitLab)RPC design OperationService::UserMergeBranch~Conversation: #633
## Expected Artefacts
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#rpc-design-rpc-design)
for more information on the RPC Design stage of th...~Conversation: #633
## Expected Artefacts
See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#rpc-design-rpc-design)
for more information on the RPC Design stage of the process.
- [ ] **`grpc-proto` RPC prototypes**: merged change2017-10-11 - Weekly DeliverablesJacob Vosmaer (GitLab)Jacob Vosmaer (GitLab)https://staging.gitlab.com/gitlab-org/gitaly/-/issues/633Conversation OperationService::UserMergeBranch2020-10-23T15:51:00ZJacob Vosmaer (GitLab)Conversation OperationService::UserMergeBranch------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Crea...------------------------------------------------------------
**Stages**:
- [ ] ~"RPC Design": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_RPC_Design)
- [ ] ~"Server Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Server_Implementation)
- [ ] ~"Client Implementation": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Client_Implementation) https://gitlab.com/gitlab-org/gitaly/merge_requests/394
- [ ] ~"Acceptance Testing": [Create Issue](https://gitlab.com/gitlab-org/gitaly/issues/new?issuable_template=Migration_Acceptance_Testing)
**RPC Endpoints**:
- `OperationService::UserMergeBranch`
**Known Client Routes**:
- Known client endpoints
------------------------------------------------------------Jacob Vosmaer (GitLab)Jacob Vosmaer (GitLab)