InfluxDB and Grafana

Closes https://gitlab.com/gitlab-org/gitlab-development-kit/issues/119

@yorickpeterse could you test this? cd influxdb && make

@jacobvosmaer-gitlab The compilation process appears to be working and the binary ends up being placed in ./influxdb/bin/influxd. Running ./influxdb/bin/influxd --help also works.

Thanks @yorickpeterse , that is what I was hoping for.

Now fighting with Grafana; it needs quite some time to build its Javascript part.

Added 1 commit:

• 1bc003a7 - Build grafana too, improve influxdb build

@yorickpeterse could you try the same for grafana?

Added 1 commit:

• 982ac1fd - Remove unfinished bogus rules

Added 1 commit:

• 04824137 - Gitignore, clean up makefile

Added 1 commit:

• 8016732a - make idempotence fixes

@jacobvosmaer-gitlab Compiling Grafana also works, so does running it with the --help option.

Great! I think the next step is putting influxdb in the procfile and seeding its database.

I want to specify the listen address (localhost:xyz) and the data directory. And if I remember correctly we need to seed the database. @yorickpeterse do you know how to do any/all of those things?

@jacobvosmaer-gitlab

I want to specify the listen address (localhost:xyz) and the data directory.

We can ship a configuration file for this. I have attached my local config file, all we'd need to change is any of the directories it stores data in. The UDP settings are already configured.

Config file: influxdb.conf

And if I remember correctly we need to seed the database.

InfluxDB needs to have a bunch of queries/retention policies set up. We can set this up by running the following queries in an Influx shell:

queries.txt

In this setup the raw data sticks around for 1 day (production uses 1 hour but for development it may be useful to keep it around a bit longer), downsampled data for 7 days.

Added 1 commit:

• a1ad668d - Add influxdb to Procfile and "run"

Thanks @yorickpeterse I have incorporated the config file. It seems influxdb opens an extra port (8088) for clustering and I am having a hard time find out how to turn that off. Not a big fan of opening lots of ports, port conflicts are frustrating for GDK users.

Will move on to the queries.txt file.

Added 1 commit:

• 18f3a5e7 - Bootstrap the influxdb database

@yorickpeterse I pushed code that will seed the database. It runs when you do make influxdb-setup at the top level. Could you give it a try?

It may fail if meta/meta.db already exists.

@jacobvosmaer-gitlab This fails with:

cd src/github.com/influxdata/influxdb && go install ./...
make[1]: Leaving directory '/home/yorickpeterse/Projects/gitlab/gitlab-development-kit/influxdb'
support/bootstrap-influxdb 8086
ERROR: Procfile does not exist.
support/bootstrap-influxdb: could not connect to InfluxDB at 127.0.0.1:8086
Makefile:170: recipe for target 'influxdb/meta/meta.db' failed
make: *** [influxdb/meta/meta.db] Error 1

I ran:

git clean -dfx # to remove all untracked files
make influxdb-setup

@yorickpeterse could you do a plain make and try again?

That will take a while because it will create GitLab development seed data. For people who are already using GDK, the plain make finishes instantly.

@jacobvosmaer-gitlab That seems to succeed with output something along the lines of:

...
To start only the app (assuming the DBs are already running):

    ./run app

To access GitLab you may now go to http://localhost:3000 in your
browser. The development login credentials are `root` and `5iveL!fe`.

You can override the port used by this GDK with a 'port' file.

    echo 4000 > port

If you want to work on GitLab CI you will need to install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner).

To enable the OpenLDAP server, see the OpenLDAP instructions in this readme.

END Post-installation
*********************************************
10:52:21 redis.1                 | exited with code 0
gitlab-development-kit $ 10:52:23 postgresql.1            | exited with code 0

OK, then I think InfluxDB is ready.

@yorickpeterse what are the setup tasks for Grafana?

@jacobvosmaer-gitlab For that we'd have to import the dashboards using https://gitlab.com/gitlab-org/grafana-dashboards/, but this repository doesn't support basic authentication just yet (only API token authentication at the moment). Basically what it has to do is:

POST to http://grafana-host/api/dashboards/db with the POST body being set to the JSON of a dashboard. The Content-Type header must be set to application/json.

Maybe we can script this out using the curl CLI? Something like:

for dashboard in dashboards/*.json
do
  curl http://grafana-host/api/dashboards/db -d "$(cat "$dashboard")" -H 'Content-Type: application/json'
done

@yorickpeterse we can do that but we can also put a Ruby stdlib-only script in support/. BTW we should not do that cat, Curl lets you write -d @filename.

@yorickpeterse do you have a grafana config file example?

@jacobvosmaer-gitlab Sure, my local config file is attached. A few things that should be changed:

• The paths (under paths)
• The HTTP port (under server -> http_port)
• Database settings (under database)
• Security settings (under security) such as the admin username/password

Everything else can be left as-is for a development environment

grafana.ini

@yorickpeterse thanks! I found this in the ini, can we use it to load dashboards?

;#################################### Dashboard JSON files ##########################
[dashboards.json]
;enabled = false
;path = /home/git/grafana/dashboards

@jacobvosmaer-gitlab I'll be honest, I have no idea as I have never tried it.

Added 1 commit:

• 5c802d2e - First steps to get grafana started

Fighting with starting grafana up. It boots faster than Postgres, and then dies because it cannot connect to postgres. I think I will create a wait wrapper that waits for PG to respond, hope that is realistic.

Will also have to 'CREATE DATABASE grafana' during bootstrapping.

@yorickpeterse OK I will look into it myself.

Do you know how to turn down the log level in influxdb? It is way too chatty as it is.

@jacobvosmaer-gitlab Yes:

• Under meta set logging-enabled to false
• Under data set wal-logging-enabled, data-logging-enabled, and query-log-enabled to false
• Under http set log-enabled to false
• Under continuous_queries set log-enabled to false

This should disable most (if not all) of the logging.

Added 1 commit:

• 0f3bb930 - More grafana setup, not done yet

Thank @yorickpeterse . To be continued.

Added 2 commits:

• 61b389d8 - Fix gitignore
• 17856ca9 - Add support/wait-postgresql-ready helper

@yorickpeterse seems like we can auto-import dashboard JSON files from disk!

Testing now if I can hook up GitLab to InfluxDB, this might be almost done.

@yorickpeterse how do I see in Grafana if everything is connected correctly? I see a lot of dashboards with hours worth of data in it, which cannot be right because I just turned GDK on.

I also get this popup:

Added 3 commits:

• 4a385bf9 - Reduce influxdb log output
• 93c7677b - Clone grafana dashboards
• 026dbcac - Use Grafana 3.0.2, import dasboards from disk

Added 1 commit:

• 88ffdfaf - Performance metrics documentation

Added 1 commit:

• 7105b227 - Simplify .gitignore

@yorickpeterse what version of Grafana are the dashboards in the gitlab-dashboards repo for?

This MR uses Grafana 3.0.2 which is relatively new. I wonder if that explains the error popups I am seeing. It should not be too difficult to downgrade Grafana here.

Unmarked this merge request as a Work In Progress

I removed WIP because all the parts that I understand seem to be working. I still don't know how to use Grafana (which is one of the reasons I started working on this) so I cannot see if data from GDK is really showing up there.

Added 1 commit:

• 63880b54 - Configure grafana data source during bootstrap

@yorickpeterse and I realized we still needed to configure a data source. It seems like it all works now!

mentioned in commit c7a09f8e

Status changed to merged

@jacobvosmaer this won't work.
The next command needs an existing Procfile. I get the follwing error otherwise: ERROR: Procfile does not exist.

Thanks @haynes , this should be fixed in c9163294