Document issues surrounding Prometheus scrape interval configuration
The new embedded Prometheus server has a configurable scrape interval with a default of 15s.
It is very important that we document in-line that adjusting this configuration can break things. When we start to configure reporting graphs, dashboards, and alerts, we will be tuning our queries around this time interval. If an end user changes the scrape interval they could cause a cascading problem with all of the queries that need multiple samples to do accurate statistical reporting.
For example, if we want to have a detailed rate(metric[1m])
graph, 15s is recommended longest scrape_interval. We need to have extra samples in case of failed or delayed scrapes. It also can help with counter resets.
Option 2, we could remove the scrape interval configuration to avoid this dependency problem.