README.md 10.9 KB
Newer Older
Jacob Vosmaer's avatar
Jacob Vosmaer committed
1
# gitlab-workhorse
Jacob Vosmaer's avatar
Jacob Vosmaer committed
2

3
4
5
Gitlab-workhorse is a smart reverse proxy for GitLab. It handles
"large" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
6

7
8
9
## Quick facts (how does Workhorse work)

-   Workhorse can handle some requests without involving Rails at all:
Ben Bodenmiller's avatar
Ben Bodenmiller committed
10
    for example, JavaScript files and CSS files are served straight
11
12
13
14
15
16
17
18
19
20
    from disk.
-   Workhorse can modify responses sent by Rails: for example if you use
    `send_file` in Rails then gitlab-workhorse will open the file on
    disk and send its contents as the response body to the client.
-   Workhorse can take over requests after asking permission from Rails.
    Example: handling `git clone`.
-   Workhorse can modify requests before passing them to Rails. Example:
    when handling a Git LFS upload Workhorse first asks permission from
    Rails, then it stores the request body in a tempfile, then it sends
    a modified request containing the tempfile path to Rails.
21
22
-   Workhorse can manage long-lived WebSocket connections for Rails.
    Example: handling the terminal websocket for environments.
23
-   Workhorse does not connect to Postgres, only to Rails and (optionally) Redis.
24
25
26
27
28
29
-   We assume that all requests that reach Workhorse pass through an
    upstream proxy such as NGINX or Apache first.
-   Workhorse does not accept HTTPS connections.
-   Workhorse does not clean up idle client connections.
-   We assume that all requests to Rails pass through Workhorse.

30
31
For more information see ['A brief history of
gitlab-workhorse'][brief-history-blog].
Jacob Vosmaer's avatar
Jacob Vosmaer committed
32

33
34
35
## Usage

```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
36
  gitlab-workhorse [OPTIONS]
37
38

Options:
39
  -apiCiLongPollingDuration duration
40
      Long polling duration for job requesting for runners (default 50s - enabled) (default 50ns)
Kamil Trzcinski's avatar
Kamil Trzcinski committed
41
  -apiLimit uint
42
      Number of API requests allowed at single time
Kamil Trzcinski's avatar
Kamil Trzcinski committed
43
  -apiQueueDuration duration
44
      Maximum queueing duration of requests (default 30s)
Kamil Trzcinski's avatar
Kamil Trzcinski committed
45
  -apiQueueLimit uint
46
      Number of API requests allowed to be queued
47
  -authBackend string
48
      Authentication/authorization backend (default "http://localhost:8080")
49
  -authSocket string
50
      Optional: Unix domain socket to dial authBackend at
Heinrich Lee Yu's avatar
Heinrich Lee Yu committed
51
52
53
54
  -cableBackend string
      Optional: ActionCable backend (default authBackend)
  -cableSocket string
      Optional: Unix domain socket to dial cableBackend at (default authSocket)
55
56
  -config string
      TOML file to load config from
57
  -developmentMode
58
      Allow the assets to be served from Rails app
59
  -documentRoot string
60
      Path to static files content (default "public")
61
  -listenAddr string
62
      Listen address for HTTP server (default "localhost:8181")
63
  -listenNetwork string
64
      Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
65
  -listenUmask int
66
67
68
69
70
      Umask for Unix socket
  -logFile string
      Log file location
  -logFormat string
      Log format to use defaults to text (text, json, structured, none) (default "text")
71
  -pprofListenAddr string
72
73
74
      pprof listening address, e.g. 'localhost:6060'
  -prometheusListenAddr string
      Prometheus listening address, e.g. 'localhost:9229'
75
  -proxyHeadersTimeout duration
76
      How long to wait for response headers when proxying the request (default 5m0s)
77
  -secretPath string
78
      File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret")
79
  -version
80
      Print version and exit
81
82
```

Marco Vito Moscaritolo's avatar
Marco Vito Moscaritolo committed
83
The 'auth backend' refers to the GitLab Rails application. The name is
Jacob Vosmaer's avatar
Jacob Vosmaer committed
84
85
a holdover from when gitlab-workhorse only handled Git push/pull over
HTTP.
86

87
Gitlab-workhorse can listen on either a TCP or a Unix domain socket. It
88
89
90
can also open a second listening TCP listening socket with the Go
[net/http/pprof profiler server](http://golang.org/pkg/net/http/pprof/).

91
92
Gitlab-workhorse can listen on redis events (currently only builds/register
for runners). This requires you to pass a valid TOML config file via
93
94
`-config` flag.
For regular setups it only requires the following (replacing the string
95
with the actual socket)
96
97
98

### Redis

99
100
101
102
103
104
105
106
107
108
109
110
111
112
Gitlab-workhorse integrates with Redis to do long polling for CI build
requests. This is configured via two things:

-   Redis settings in the TOML config file
-   The `-apiCiLongPollingDuration` command line flag to control polling
    behavior for CI build requests

It is OK to enable Redis in the config file but to leave CI polling
disabled; this just results in an idle Redis pubsub connection. The
opposite is not possible: CI long polling requires a correct Redis
configuration.

Below we discuss the options for the `[redis]` section in the config
file.
113

114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
```
[redis]
URL = "unix:///var/run/gitlab/redis.sock"
Password = "my_awesome_password"
Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
SentinelMaster = "mymaster"
```

- `URL` takes a string in the format `unix://path/to/redis.sock` or
`tcp://host:port`.
- `Password` is only required if your redis instance is password-protected
- `Sentinel` is used if you are using Sentinel.
  *NOTE* that if both `Sentinel` and `URL` are given, only `Sentinel` will be used

Optional fields are as follows:
```
[redis]
131
132
133
DB = 0
ReadTimeout = "1s"
KeepAlivePeriod = "5m"
134
135
136
137
MaxIdle = 1
MaxActive = 1
```

138
139
140
- `DB` is the Database to connect to. Defaults to `0`
- `ReadTimeout` is how long a redis read-command can take. Defaults to `1s`
- `KeepAlivePeriod` is how long the redis connection is to be kept alive without anything flowing through it. Defaults to `5m`
141
142
143
- `MaxIdle` is how many idle connections can be in the redis-pool at once. Defaults to 1
- `MaxActive` is how many connections the pool can keep. Defaults to 1

144
145
146
147
148
149
150
151
152
153
### Relative URL support

If you are mounting GitLab at a relative URL, e.g.
`example.com/gitlab`, then you should also use this relative URL in
the `authBackend` setting:

```
gitlab-workhorse -authBackend http://localhost:8080/gitlab
```

154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
### Interaction of authBackend and authSocket

The interaction between `authBackend` and `authSocket` can be a bit
confusing. It comes down to: if `authSocket` is set it overrides the
_host_ part of `authBackend` but not the relative path.

In table form:

|authBackend|authSocket|Workhorse connects to?|Rails relative URL|
|---|---|---|---|
|unset|unset|`localhost:8080`|`/`|
|`http://localhost:3000`|unset|`localhost:3000`|`/`|
|`http://localhost:3000/gitlab`|unset|`localhost:3000`|`/gitlab`|
|unset|`/path/to/socket`|`/path/to/socket`|`/`|
|`http://localhost:3000`|`/path/to/socket`|`/path/to/socket`|`/`|
|`http://localhost:3000/gitlab`|`/path/to/socket`|`/path/to/socket`|`/gitlab`|

Heinrich Lee Yu's avatar
Heinrich Lee Yu committed
171
172
The same applies to `cableBackend` and `cableSocket`.

Jacob Vosmaer's avatar
Jacob Vosmaer committed
173
174
## Installation

175
To install gitlab-workhorse you need [Go 1.13 or
Jacob Vosmaer (GitLab)'s avatar
Jacob Vosmaer (GitLab) committed
176
177
newer](https://golang.org/dl) and [GNU
Make](https://www.gnu.org/software/make/).
Jacob Vosmaer's avatar
Jacob Vosmaer committed
178

179
180
181
182
183
184
To install into `/usr/local/bin` run `make install`.

```
make install
```

185
186
187
188
189
To install into `/foo/bin` set the PREFIX variable.

```
make install PREFIX=/foo
```
190

191
192
On some operating systems, such as FreeBSD, you may have to use
`gmake` instead of `make`.
Jacob Vosmaer (GitLab)'s avatar
Jacob Vosmaer (GitLab) committed
193

194
195
196
197
198
199
200
201
202
## Dependencies

### Exiftool

Workhorse uses [exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
removing EXIF data (which may contain sensitive information) from uploaded
images. If you installed GitLab:

-   Using the Omnibus package, you're all set.
203
204
    *NOTE* that if you are using CentOS Minimal, you may need to install `perl`
    package: `yum install perl`
205
206
207
208
209
210
211
212
213
214
-   From source, make sure `exiftool` is installed:

    ```sh
    # Debian/Ubuntu
    sudo apt-get install libimage-exiftool-perl

    # RHEL/CentOS
    sudo yum install perl-Image-ExifTool
    ```

215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
### GraphicsMagick (**experimental**)

Workhorse has an experimental feature that allows us to rescale images on-the-fly.
If you do not run Workhorse in a container where the `gm` tool is already installed,
you will have to install it on your host machine instead:

#### macOS

```sh
brew install graphicsmagick
```

#### Debian/Ubuntu

```sh
sudo apt-get install graphicsmagick
```

For installation on other platforms, please consult http://www.graphicsmagick.org/README.html.

Note that Omnibus containers already come with `gm` installed.

237
238
239
240
## Error tracking

GitLab-Workhorse supports remote error tracking with
[Sentry](https://sentry.io). To enable this feature set the
Ben Bodenmiller's avatar
Ben Bodenmiller committed
241
242
`GITLAB_WORKHORSE_SENTRY_DSN` environment variable.
You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable to
243
244
use the Sentry environment functionality to separate staging, production and
development.
245

246
247
248
Omnibus (`/etc/gitlab/gitlab.rb`):

```
249
250
251
252
gitlab_workhorse['env'] = {
    'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
    'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
}
253
254
255
256
257
258
```

Source installations (`/etc/default/gitlab`):

```
export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
259
export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
260
261
```

262
263
## Tests

Jacob Vosmaer's avatar
Jacob Vosmaer committed
264
Run the tests with:
Jacob Vosmaer's avatar
Jacob Vosmaer committed
265
266

```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
267
make clean test
Jacob Vosmaer's avatar
Jacob Vosmaer committed
268
269
```

Jacob Vosmaer's avatar
Jacob Vosmaer committed
270
### Coverage / what to test
271

Jacob Vosmaer's avatar
Jacob Vosmaer committed
272
273
274
275
276
Each feature in gitlab-workhorse should have an integration test that
verifies that the feature 'kicks in' on the right requests and leaves
other requests unaffected. It is better to also have package-level tests
for specific behavior but the high-level integration tests should have
the first priority during development.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
277

Jacob Vosmaer's avatar
Jacob Vosmaer committed
278
It is OK if a feature is only covered by integration tests.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
279

280
281
## Distributed Tracing

282
Workhorse supports distributed tracing through [LabKit][] using [OpenTracing APIs](https://opentracing.io).
283

284
By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags][build-tags]/[build constraints][build-tags]. This can be done by setting the `BUILD_TAGS` make variable.
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299

For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.

```shell
make BUILD_TAGS="tracer_static tracer_static_jaeger"
```

Once Workhorse is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.

For example:

```shell
GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
```

300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
## Continuous Profiling

Workhorse supports continuous profiling through [LabKit][] using [Stackdriver Profiler](https://cloud.google.com/profiler).

By default, the Stackdriver Profiler implementation is linked in the binary using [build tags][build-tags], though it's not
required and can be skipped.

For example:

```shell
make BUILD_TAGS=""
```

Once Workhorse is compiled with Continuous Profiling, the profiler configuration can be set via `GITLAB_CONTINUOUS_PROFILING`
environment variable.

For example:

```shell
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
```

More information about see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).

Jacob Vosmaer's avatar
Jacob Vosmaer committed
324
325
## License

Ben Bodenmiller's avatar
Ben Bodenmiller committed
326
This code is distributed under the MIT license, see the [LICENSE](LICENSE) file.
327
328

[brief-history-blog]: https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/
329
330
[LabKit]: https://gitlab.com/gitlab-org/labkit/
[build-tags]: https://golang.org/pkg/go/build/#hdr-Build_Constraints