Test higher diff size limits

What are we going to do?

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11875 adds a feature flag to increase the diff size limits for displaying and collapsing the diffs respectively. We are going to test those new limits for a period of 4 hours.

For this we will enable the feature flag gitlab_git_diff_size_limit_increase by setting it to true.

---

Why are we doing it?

We never actually tested the previous limits, we just picked ones that seemed reasonable. These new limits also seem reasonable, but we should experiment first 🙂

We think we can increase the limits because we have since added other limits to diffs, based on the size of the entire diff, which should limit the impact while being more user-friendly for diffs where few files changed, but they changed a great deal.

---

When are we going to do it?

• Start time: 2017-06-27 10:00 UTC
• Duration: 4 hours
• Estimated end time: 2017-06-27 14:00 UTC

How are we going to do it?

1. Someone with admin access: set the gitlab_git_diff_size_limit_increase feature flag to true.
2. Me: monitor these dashboards:
3. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::MergeRequestsController%23diffs.json
4. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::MergeRequestsController%23new_diffs
5. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::MergeRequestsController%23new_diffs.json
6. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::MergeRequestsController%23diff_for_path.json
7. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::CompareController%23show
8. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::CompareController%23diff_for_path.json
9. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::CommitController%23show
10. https://performance.gitlab.net/dashboard/db/rails-controllers?var-action=Projects::CommitController%23diff_for_path.json
11. Someone with admin access: set the gitlab_git_diff_size_limit_increase feature flag to false.

---

How are we preparing for it?

Once 9.3.0-rc2 is deployed, we just need to pick a time that is convenient (any fires after the deploy put out, no subsequent deploy planned).

---

What can we check before starting?

The timings for these controller actions are very noisy, but we can pick specific public URLs - ideally on GitLab projects - and use those as our baselines. We'll specifically pick these:

• The current CE -> EE merge. This is typically a large MR with lots of changes.
• The most recent release post. This normally hits the existing collapse limit, but the overall diff isn't very big.
• https://gitlab.com/nrclark/dummy_project/commit/81ebdea5df2fb42e59257cb3eaad671a5c53ca36, which is a common 'stress test' of diffing.

The first and last shouldn't get slower. The middle one may get slightly slower - I'd accept a 5% increase in timings.

Prometheus metrics for these: https://prometheus.gitlab.com/graph?g0.range_input=2h&g0.expr=probe_duration_seconds%7Binstance%3D~%22gitlab.com%2Fgitlab-com%2Fwww-gitlab-com%2Fmerge_requests%2F5804%2Fdiffs.json%22%7D&g0.tab=0&g1.range_input=1h&g1.expr=probe_duration_seconds%7Binstance%3D~%22gitlab.com%2Fnrclark%2Fdummy_project%2Fcommit%2F81ebdea5df2fb42e59257cb3eaad671a5c53ca36%22%7D&g1.tab=0&g2.range_input=1h&g2.expr=probe_duration_seconds%7Binstance%3D~%22gitlab.com%2Fgitlab-org%2Fgitlab-ee%2Fmerge_requests%2F2178%2Fdiffs.json%22%7D&g2.tab=0

---

What can we check afterwards to ensure that it's working?

We can check the release post MR for an example. We should also pay close attention to the response timings for the above charts. We do expect all response timings to increase slightly - although in the worst case, not at all (because of the limits on the overall diff size).

The diff_for_path actions should see fewer, slower transactions as they will now only be called for larger diffs. These are called async anyway.

---

Impact

• Type of impact: client facing.
• What will happen: people will see more diffs rendered by default, at a cost in load times. Hopefully the cost in load times is small.
• Do we expect downtime? (set the override in pagerduty): no.

How are we communicating this to our customers?

• Tweet before and after the change.
• Do we need to set a broadcast banner?: no.

What is the rollback plan?

We will disable the feature flag at the end of the period. If we see problems caused by this during the period, we can disable it at any time.

---

Monitoring

• Graphs to check for failures:
  • See the above list of links.
• Graphs to check for improvements:
  • See the above list of links.
• Alerts that may trigger:
  • None.

---

Scheduling

Schedule a downtime in the production calendar twice as long as your worst duration estimate, be pessimistic (better safe than sorry)

When things go wrong (downtime or service degradation)

• Label the change issue as outage
• Perform a blameless post mortem

References

• Making changes to GitLab.com
• Infrastructure links
• On-Call Log
• Blameless Postmortems Guideline
• Monitoring

@pcarranza @ernstvn I like this new checklist much more ❤️

Can you take a look at my questions above please and just review the whole thing too? 🙂

added experiment label

assigned to @pcarranza

@smcgivern which is the feature flag? (so I can just call it with Marvin)

changed milestone to %WoW ending 2017-06-27

assigned to @smcgivern

gitlab_git_diff_size_limit_increase

@smcgivern

but we can pick specific public URLs - ideally on GitLab projects - and use those as our baselines.

How? can you explain how are going to measure this?

Tweet before and after the change. @pcarranza do we need this? If so, do I need to add it to the

Yes, just in case things go wrong, it's better to be communicative than sorry.

what do I schedule here? Just the four-hour period? Nothing? (No downtime estimated.) Eight hours?

I think that running the experiment for 8hs sounds like a plan, but if we can get good information in 4hs that's good enough for me.

Planning

I think we can run this test this week, just find a spot and we can do it. Shouldn't have a large impact at all.

changed the description

@smcgivern does it sound reasonable to do this next Monday? (I would like to avoid doing it in the release week honestly)

Yes, definitely. I'll try to answer the questions above in time for then 👍

changed the description

@pcarranza I updated the description with some answers!

Is Tuesday OK too? I have to validate another change I made on the Monday.

I think it is possible @smcgivern

Would it make sense to add a URL to measure before and after with black box monitoring? that way we can see the customer impact too in our metrics.

If so, could you give me the exact URL to monitor so I can add it?

@pcarranza yes, we should do at least these:

• https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/5804/diffs.json
• https://gitlab.com/nrclark/dummy_project/commit/81ebdea5df2fb42e59257cb3eaad671a5c53ca36
• https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2178/diffs.json

(It just occurred to me that these don't need to be open MRs, as diff loading is the same either way.)

I added them here https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/785

changed the description

I changed the times to afternoon as I have an appointment in the morning and put an event in the production calendar.

@smcgivern looks good to me 👍

changed the description

changed the description

We just tried this and reverted the change within 10 minutes. There were two problems:

1. This is totally my fault: large diffs within MRs were not expanded or collapsed. We made some other changes to the rendering, and it seems like it doesn't work with the configurable limits:

   

2. Response timings increased even for the diffs that were already at the limit:

   

I will take those MRs and try to reproduce those timings effects locally, and then come back to this experiment. We might need to reduce the overall limits, too.

@pcarranza would you rather I created a new issue for the next round, or keep this one open but unscheduled?

@smcgivern Let's create a new one, this experiment is done and we had some data samples from this event.

The next one should be basically a copy of this one.

closed

Therefore, I'm closing @smcgivern