README.md 6.94 KB
Newer Older
Jacob Vosmaer's avatar
Jacob Vosmaer committed
1
# gitlab-workhorse
Jacob Vosmaer's avatar
Jacob Vosmaer committed
2

3
4
5
Gitlab-workhorse is a smart reverse proxy for GitLab. It handles
"large" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
6

7
8
9
10
11
12
13
14
15
16
17
18
19
20
## Quick facts (how does Workhorse work)

-   Workhorse can handle some requests without involving Rails at all:
    for example, Javascript files and CSS files are served straight
    from disk.
-   Workhorse can modify responses sent by Rails: for example if you use
    `send_file` in Rails then gitlab-workhorse will open the file on
    disk and send its contents as the response body to the client.
-   Workhorse can take over requests after asking permission from Rails.
    Example: handling `git clone`.
-   Workhorse can modify requests before passing them to Rails. Example:
    when handling a Git LFS upload Workhorse first asks permission from
    Rails, then it stores the request body in a tempfile, then it sends
    a modified request containing the tempfile path to Rails.
21
22
-   Workhorse can manage long-lived WebSocket connections for Rails.
    Example: handling the terminal websocket for environments.
23
-   Workhorse does not connect to Postgres, only to Rails and (optionally) Redis.
24
25
26
27
28
29
-   We assume that all requests that reach Workhorse pass through an
    upstream proxy such as NGINX or Apache first.
-   Workhorse does not accept HTTPS connections.
-   Workhorse does not clean up idle client connections.
-   We assume that all requests to Rails pass through Workhorse.

30
31
For more information see ['A brief history of
gitlab-workhorse'][brief-history-blog].
Jacob Vosmaer's avatar
Jacob Vosmaer committed
32

33
34
35
## Usage

```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
36
  gitlab-workhorse [OPTIONS]
37
38

Options:
39
  -apiCiLongPollingDuration duration
40
      Long polling duration for job requesting for runners (default 50s - enabled) (default 50ns)
Kamil Trzcinski's avatar
Kamil Trzcinski committed
41
  -apiLimit uint
42
      Number of API requests allowed at single time
Kamil Trzcinski's avatar
Kamil Trzcinski committed
43
  -apiQueueDuration duration
44
      Maximum queueing duration of requests (default 30s)
Kamil Trzcinski's avatar
Kamil Trzcinski committed
45
  -apiQueueLimit uint
46
      Number of API requests allowed to be queued
47
  -authBackend string
48
      Authentication/authorization backend (default "http://localhost:8080")
49
  -authSocket string
50
51
52
      Optional: Unix domain socket to dial authBackend at
  -config string
      TOML file to load config from
53
  -developmentMode
54
      Allow the assets to be served from Rails app
55
  -documentRoot string
56
      Path to static files content (default "public")
57
  -listenAddr string
58
      Listen address for HTTP server (default "localhost:8181")
59
  -listenNetwork string
60
      Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
61
  -listenUmask int
62
63
64
65
66
      Umask for Unix socket
  -logFile string
      Log file location
  -logFormat string
      Log format to use defaults to text (text, json, structured, none) (default "text")
67
  -pprofListenAddr string
68
69
70
      pprof listening address, e.g. 'localhost:6060'
  -prometheusListenAddr string
      Prometheus listening address, e.g. 'localhost:9229'
71
  -proxyHeadersTimeout duration
72
      How long to wait for response headers when proxying the request (default 5m0s)
73
  -secretPath string
74
      File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret")
75
  -version
76
      Print version and exit
77
78
```

Marco Vito Moscaritolo's avatar
Marco Vito Moscaritolo committed
79
The 'auth backend' refers to the GitLab Rails application. The name is
Jacob Vosmaer's avatar
Jacob Vosmaer committed
80
81
a holdover from when gitlab-workhorse only handled Git push/pull over
HTTP.
82

83
Gitlab-workhorse can listen on either a TCP or a Unix domain socket. It
84
85
86
can also open a second listening TCP listening socket with the Go
[net/http/pprof profiler server](http://golang.org/pkg/net/http/pprof/).

87
88
Gitlab-workhorse can listen on redis events (currently only builds/register
for runners). This requires you to pass a valid TOML config file via
89
90
`-config` flag.
For regular setups it only requires the following (replacing the string
91
with the actual socket)
92
93
94

### Redis

95
96
97
98
99
100
101
102
103
104
105
106
107
108
Gitlab-workhorse integrates with Redis to do long polling for CI build
requests. This is configured via two things:

-   Redis settings in the TOML config file
-   The `-apiCiLongPollingDuration` command line flag to control polling
    behavior for CI build requests

It is OK to enable Redis in the config file but to leave CI polling
disabled; this just results in an idle Redis pubsub connection. The
opposite is not possible: CI long polling requires a correct Redis
configuration.

Below we discuss the options for the `[redis]` section in the config
file.
109

110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
```
[redis]
URL = "unix:///var/run/gitlab/redis.sock"
Password = "my_awesome_password"
Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
SentinelMaster = "mymaster"
```

- `URL` takes a string in the format `unix://path/to/redis.sock` or
`tcp://host:port`.
- `Password` is only required if your redis instance is password-protected
- `Sentinel` is used if you are using Sentinel.
  *NOTE* that if both `Sentinel` and `URL` are given, only `Sentinel` will be used

Optional fields are as follows:
```
[redis]
127
128
129
DB = 0
ReadTimeout = "1s"
KeepAlivePeriod = "5m"
130
131
132
133
MaxIdle = 1
MaxActive = 1
```

134
135
136
- `DB` is the Database to connect to. Defaults to `0`
- `ReadTimeout` is how long a redis read-command can take. Defaults to `1s`
- `KeepAlivePeriod` is how long the redis connection is to be kept alive without anything flowing through it. Defaults to `5m`
137
138
139
- `MaxIdle` is how many idle connections can be in the redis-pool at once. Defaults to 1
- `MaxActive` is how many connections the pool can keep. Defaults to 1

140
141
142
143
144
145
146
147
148
149
### Relative URL support

If you are mounting GitLab at a relative URL, e.g.
`example.com/gitlab`, then you should also use this relative URL in
the `authBackend` setting:

```
gitlab-workhorse -authBackend http://localhost:8080/gitlab
```

Jacob Vosmaer's avatar
Jacob Vosmaer committed
150
151
## Installation

152
To install gitlab-workhorse you need [Go 1.8 or
Jacob Vosmaer (GitLab)'s avatar
Jacob Vosmaer (GitLab) committed
153
154
newer](https://golang.org/dl) and [GNU
Make](https://www.gnu.org/software/make/).
Jacob Vosmaer's avatar
Jacob Vosmaer committed
155

156
157
158
159
160
161
To install into `/usr/local/bin` run `make install`.

```
make install
```

162
163
164
165
166
To install into `/foo/bin` set the PREFIX variable.

```
make install PREFIX=/foo
```
167

168
169
On some operating systems, such as FreeBSD, you may have to use
`gmake` instead of `make`.
Jacob Vosmaer (GitLab)'s avatar
Jacob Vosmaer (GitLab) committed
170

171
172
173
174
175
176
## Error tracking

GitLab-Workhorse supports remote error tracking with
[Sentry](https://sentry.io). To enable this feature set the
GITLAB_WORKHORSE_SENTRY_DSN environment variable.

177
178
179
180
181
182
183
184
185
186
187
188
Omnibus (`/etc/gitlab/gitlab.rb`):

```
gitlab_workhorse['env'] = {'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'}
```

Source installations (`/etc/default/gitlab`):

```
export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
```

189
190
## Tests

Jacob Vosmaer's avatar
Jacob Vosmaer committed
191
Run the tests with:
Jacob Vosmaer's avatar
Jacob Vosmaer committed
192
193

```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
194
make clean test
Jacob Vosmaer's avatar
Jacob Vosmaer committed
195
196
```

Jacob Vosmaer's avatar
Jacob Vosmaer committed
197
### Coverage / what to test
198

Jacob Vosmaer's avatar
Jacob Vosmaer committed
199
200
201
202
203
Each feature in gitlab-workhorse should have an integration test that
verifies that the feature 'kicks in' on the right requests and leaves
other requests unaffected. It is better to also have package-level tests
for specific behavior but the high-level integration tests should have
the first priority during development.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
204

Jacob Vosmaer's avatar
Jacob Vosmaer committed
205
It is OK if a feature is only covered by integration tests.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
206

Jacob Vosmaer's avatar
Jacob Vosmaer committed
207
208
209
## License

This code is distributed under the MIT license, see the LICENSE file.
210
211

[brief-history-blog]: https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/