Browse build artifact files with the GitLab UI

Milestone changed to 8.3

mentioned in issue #3624 (closed)

Added ~19173 label

Reassigned to @grzesiek

We had a call with @ayufan and @JobV today, and our proposition to handling this is something like:

We should migrate artifact data stored in database to separate model/table, something like Ci::Artifacts.

This will make it possible to store more information about artifacts. Then, later, it will be possible to introduce new features like storing more than single artifacts archive created during a build. Maybe even making release-artifacts would be possible, along with other types of artifacts for single build (release artifacts, pages artifacts, shared artifacts, etc.).
Then add a metadata attribute to artifacts, that will describe content of artifacts archive.

Metadata will be stored along with an archive created during a build. Metadata will store information about what is inside an archive, when it has been created, and other information can be also stored here (what would be nice to add here ?)
Generating tree view of artifacts content in GitLab UI using metadata instead of extracting an archive for this purpose.

This will make this implementation more efficient and will impose less stress on GitLab if user wants only to view contents, instead of downloading something.
Provided that we have artifacts metadata we can serve that through API.

It may be particularly interesting to give an access to artifacts metadata via API. It may be also used in hooks etc. Users would be able to consume this API endpoint / hooks, to check - for example - if artifacts are complete or what is the size of artifacts (many use cases here).
When user wants to download a single file or multiple files, it will be necessary to extract entire archive using Sidekiq job.

But we will do this only if it is mandatory to do so. Before that, user will see some kind of model/splash screen saying that he needs to wait for artifacts to decompress and then he will get access to files. He will also see information that it may be better to download entire archive, and it is preferred way, but if user decides to extract artifacts - it is fine.

It would be also possible to extract single file from an archive, but it is not really advantageous in terms of performance when using tar.gz. It would be more efficient if we where using zip format instead, but tar.gz has other benefits.
Cleanup extracted artifacts data (invalidate cache)

Extracted artifacts will be cached in case of further requests, and we will invalidate cache after some time (let's say, about an hour). Then everything will be cleaned up. In worst scenario, when users need to access artifacts continuously, artifacts will be extracted once in every hour.

Milestone changed to 8.4

mentioned in issue #3439 (closed)

Had a call with @ayufan, we simplified this solution (above) for now.

Marked the task Add artifacts metadata field to build as completed

Marked the task Add artifacts metadata field to build as incomplete

Marked the task Add artifacts metadata field to build as completed

cc @jacobvosmaer

Workhorse can extract a single file from a tar.gz to standard output with tar -Ozxf archive.tar.gz -- my/file.txt. Golang io.Copy can stream this into the HTTP repsonse body.

@jacobvosmaer: @ayufan said, that it may be particularly useful to generate metadata while receiving this archive file, instead of using Sidekiq job. I think that it is good idea (but don't know workhorse that much at this moment to say more). What do you think ?

I disagree. Why make the HTTP client wait while we parse a potentially large tar.gz file? @ayufan we have to think about NGINX / HAProxy timeouts. I think it is better to just store the file in workhorse, and inspect it later in Sidekiq.

Yesterday and today I talked a lot with @ayufan, @JobV and @jacobvosmaer about this feature, also done some TAR-related prototyping, looked how things are done in GitLab CI, and I think that we have managed to decide how this feature should be implemented.

@jacobvosmaer

We would do that online, while receiving the data (not after the full file is received). Gunzip will be faster than client sending the data to us. We will save on IOPS reading the full file later.

@ayufan that sounds nice in theory but looking at something like Packagecloud, where pushing a package takes forever because the web server processes it on the fly I am a little skeptical about how it will work out in practice.

On the other hand, the end result when we decorate the tar.gz in gitlab-workhorse might be simpler than having a Sidekiq worker and a "your artifacts are being processed" page.

I think doing it in Sidekiq is more natural but I am open to your "we have enough time to pipe into tar -tz argument.

@jacobvosmaer I'm not sure how it's done in packagecloud.

I would use gzip.Reader with tar.Reader. I don't see benefit to pass that to tar -tz.

@ayufan I think tar tz is better because it avoids inflating the memory footprint of gitlab-workhorse and it is not hard to parse its output.

Think of what libgit2 does for Unicorn/Sidekiq memory use. I feel pretty strongly about this.

Marked the task Ci Build fixture as completed

mentioned in merge request !2123 (merged)

Marked the task Artifacts fixtures as completed

Marked the task Mock metadata at the beginning (fixtures, seeds) as completed

I did tests of go-tar processing. The main advantage is that we have full metadata from tar archive, not only the file name.

Code: http://pastebin.com/raw/Dxxicavc

Test 1. One big archive with small amount of files:

Cost: 30-40kB of memory.

Test 2. Big archive with large amount of files: 37k and one big 800MB file:

The memory usage jumps up to 3.5MB, but GC does quite good work keeping it low. The tar tzf is faster (8s vs 19s).

Test 3. In-line processing: For slowish connection: 100kB every 100ms the memory cost is 50kB and is constant.

@ayufan Hmm, full metadata would be certainly useful. I'm currently implementing StringPath class that would ease iteration over such metadata-paths, and it would be useful to know what format of metadata we expect to have.

I have done some command profiling:

Archive I used:

grzesiek@debian: perf $ du -s linux-2.6.0.tar.gz 
40644	linux-2.6.0.tar.gz

Memory profiling done with valgrind, I/O operations counted by strace (all signals and calls included, but I/O are most common here).

Extracting a `tar.gz` with `tar -zxf`:

Heap allocations:

total heap usage: 96,586 allocs, 96,582 frees, 2,392,039 bytes allocated

Calls (including I/O operations)

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 41.09    0.002300           0     31804           write
 32.46    0.001817           0     15007           openat
  9.54    0.000534           0     19390           read
  6.79    0.000380           0       917           mkdirat
  4.25    0.000238           0     15924           utimensat
  3.18    0.000178           0     15029           fstat
  2.43    0.000136           0     15034           close
  2.55    0.000122           0     15029           fstat
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.004791                114260        19 total

Making a list of files with `tar -tzf`

Heap allocations

total heap usage: 95,673 allocs, 95,665 frees, 2,216,731 bytes allocated

Calls (including I/O operations)

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 79.68    0.001337           0     22121           read
  7.75    0.000130           0     15924           write
  6.91    0.000116         116         1           clone
  5.66    0.000095          95         1           wait4
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.001678                 38231        19 total

Extracting last file (`tar -zxf`)

Heap allocations

total heap usage: 79,756 allocs, 79,746 frees, 1,593,757 bytes allocated

Calls

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.23    0.000998           0     22875           read
  1.77    0.000018          18         1           wait4
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.001016                 23069        20 total

I will try to make same stats for ZIP and maybe other formats like XZ probably (today / tomorrow).

cc @ayufan @jacobvosmaer

Of course we do not distinguish TTY writes and disk writes here, but I gives general view on how things work.

I would love full metadata and think it would be very useful to users.

Marked the task Design mechanism that will make iterating metadata collection easier as completed

@ayufan Since we want full metadata and:

Full metadata can contain quite a large amount of information
Storing serialized metadata inside ci_builds.artifcats_metadata breaks 1NF property
We want to expire artifacts in the future
We want builds to be independent from artifacts (artifacts can be deleted, builds maybe not)
Full metadata can contain multiple attributes of given artifact entry, we can then use in our design

What do you think about creating Ci::Artifact -> ci_artifacts that would store a single metadata of artifact located in artifacts archive ?

Zip benchmarks on same archive:

Extracting zip with `-q` option

Heap allocations

total heap usage: 258,219 allocs, 258,219 frees, 175,340,313 bytes allocated

Calls

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 32.94    0.001655           0     15678           write
 24.46    0.001229           0     15014           open
 20.94    0.001052           0    104741     30931 stat
  5.35    0.000269           0      6156           read
  4.20    0.000211           0     15924           utime
  4.10    0.000206           0     30014     30014 lstat
  3.98    0.000200           0     15007           fchmod
  2.01    0.000101           0       917           mkdir
  1.75    0.000088           0     15014           close
  0.26    0.000013           0      1834           chmod
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.005024                220358     60949 total

Making a list of files

Heap allocations

total heap usage: 31,877 allocs, 31,877 frees, 704,978 bytes allocated

Calls

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 56.46    0.000682           0     47775           write
 27.32    0.000330           0     47775           ioctl
 16.23    0.000196           0     15925           stat
  0.00    0.000000           0       217           read
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.001208                111749         4 total

Extracting single file

Heap allocations

total heap usage: 15,972 allocs, 15,972 frees, 479,297 bytes allocated

Calls

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0       219           read
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   293        10 total

tar.xz benchmarks:

Extracting

Heap allocations

total heap usage: 96,590 allocs, 96,582 frees, 2,392,096 bytes allocated

Calls

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 29.67    0.002010           0     31778           write
 29.24    0.001981           0     27689           read
 23.37    0.001583           0     15007           openat
  8.49    0.000575         575         1           wait4
  2.46    0.000167           0     15027           fstat
  2.39    0.000162           0     15924           utimensat
  2.11    0.000143           0       917           mkdirat
  2.04    0.000138           0     15034           close
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.006775                122430        19 total

List of files

Heap allocations

total heap usage: 95,673 allocs, 95,665 frees, 2,216,731 bytes allocated

Calls

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 61.60    0.002280           0     32955           read
 22.21    0.000822         822         1           wait4
 16.18    0.000599           0     15924           write
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.003701                 49063        19 total

Extracting single file

Heap allocations

total heap usage: 79,757 allocs, 79,747 frees, 1,593,946 bytes allocated

Calls

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 81.88    0.002237           0     36311           read
 18.12    0.000495         495         1           wait4
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.002732                 36503        20 total

Summary (heap allocations / total calls):

Type	Extracting archive	Extracting single file	Making file list
tar.gz	2,392 kB / 114260	1,593 kB / 23069	2,216 kB / 38231
zip	175,340 kB / 220358	479 kB / 293	704 kB / 111749
tar.xz	2,392 kB / 122430	1,593 kB / 36503	2,216 kB / 49063

Sizes after compression:

grzesiek@debian: test-decompression $ du -sh *
40M	linux-2.6.0.tar.gz
27M	linux-2.6.0.tar.xz
49M	linux-2.6.0.zip

Marked the task Implement mechanism that will make iterating metadata collection easier as completed

Marked the task Create a new view, where content of artifacts will be displayed as completed

Things to consider when choosing ZIP format:

big files, big archive (over 4G)
extracting absolute path
support for UID/GID, Unix permissions
in-case sensitivity
CVE
ZIP implementation in Go

Marked the task Prepare benchmarks for ZIP / TAR GZIP / TAR XZ as completed

@creamzy @skyruler Does it look good for you ? Is it consistent with repository tree view ?

Marked the task Consult Frontend / UX as completed

It looks like we need to improve performance of this mechanism that is meant for metadata's paths traversal. I've used Linux kernel archive metadata and it works way too slow. I have some ideas how to optimize it so I will give it a try.

@grzesiek What specifically is slow? Do you have an implementation you can point to? I'd be happy to take a look and offer any suggestions for optimization.

Marked the task Tweak performance of metadata's paths traversal as completed

@jonathon-reinhart Thanks Jonathon, but I think that I can handle this :) (just pushed refactored code). If you want to help, take a look at new class, this MR introduces - StringPath which purpose is to simplify metadata paths traversal. It currently works well with linux-2.6.0.tar.gz metadata (~ 15k files), but I believe it can be even faster.

Marked the task Check for security vulnerabilities in ZIP (browse CVEs) as completed

Marked the task Check how ZIP handles SUID/SGID as completed

Marked the task Check how ZIP handles absolute paths as completed

Recent Info-ZIP CVEs

mentioned in issue #4263 (closed)

Marked the task Check how ZIP behaves with big files (> 4G) as completed

ZIP handles large files properly (CLI) if compiled with ZIP64 support:

Large Archives and Zip64. zip automatically uses the Zip64 extensions when files larger than 4 GB are added to an archive, an archive containing Zip64 entries is updated (if the resulting archive still needs Zip64), the size of the archive will exceed 4 GB, or when the number of entries in the archive will exceed about 64K.

Zip64 is also used for archives streamed from standard input as the size of such archives are not known in advance, but the option -fz- can be used to force zip to create PKZIP 2 compatible archives (as long as Zip64 extensions are not needed). You must use a PKZIP 4.5 compatible unzip, such as unzip 6.0 or later, to extract files using the Zip64 extensions.

Most of available ZIP binaries (including Debian package in stable have ZIP64 support included:

grzesiek@debian: ~ $ zip -v
[...]
Zip special compilation options:
	LARGE_FILE_SUPPORT   (can read and write large files on file system)
	ZIP64_SUPPORT        (use Zip64 to store large files in archives)
[...]

LARGE_FILE_SUPPORT - OS dependencies, OK after Linux 2.4
ZIP64_SUPPORT - implementation that makes it possible to read and store files > 4GB

This is something that we need to check if we want to use Go implementation.

Marked the task Check how ZIP handles UIDs/GIDs, UNIX permissions as completed

Marked the task Check how ZIP handle case sensitivity as completed

Simple Go implementation that iterates archive and prints a list of files (metadata), consumes around 270% more memory than CLI unzip implementation (using linux-2.6.0.tar.gz - 8080kB compared to 2968kB -- checked using GNU Time as valgrind doesn't work with Go very well), however Go implementation has very low impact on signals/calls, including IO operations (only about 14% of total calls/signals).

package main

import (
	"archive/zip"
	"flag"
	"fmt"
	"log"
)

func main() {
	flag.Parse()
	file := flag.Args()[0]
	ZipFileListReader(file)
}

func ZipFileListReader(file string) {
	archive, err := zip.OpenReader(file)
	if err != nil {
		log.Fatal(err)
	}
	defer archive.Close()

	for _, entry := range archive.File {
		fmt.Println(entry.Name)
	}
}

Marked the task Making a list of files as completed

We decided with @ayufan to not to cache artifacts_metadata at this point, and to generate metadata for each path in each subsequent request.

I have found a fast way to list only current directory to reduce the metadata size:

unzip -Z -l linux-master.zip 'linux-master/**' -x 'linux-master/*/**’

The linux-master/ is the directory we want to view.

It produces:

$ time unzip -Z  -l linux-master.zip 'linux-master/**' -x 'linux-master/*/**'                                                                                       [18:36:33]
-rw----     0.0 fat       31 tx       31 stor 21-Dec-15 01:06 linux-master/.get_maintainer.ignore
-rw----     0.0 fat     1280 tx      669 defN 21-Dec-15 01:06 linux-master/.gitignore
-rw----     0.0 fat     5537 tx     2634 defN 21-Dec-15 01:06 linux-master/.mailmap
-rw----     0.0 fat    18693 tx     7080 defN 21-Dec-15 01:06 linux-master/COPYING
-rw----     0.0 fat    97181 tx    43505 defN 21-Dec-15 01:06 linux-master/CREDITS
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/Documentation/
-rw----     0.0 fat     2622 tx      950 defN 21-Dec-15 01:06 linux-master/Kbuild
-rw----     0.0 fat      252 tx      181 defN 21-Dec-15 01:06 linux-master/Kconfig
-rw----     0.0 fat   336854 tx    85195 defN 21-Dec-15 01:06 linux-master/MAINTAINERS
-rw----     0.0 fat    54743 tx    16771 defN 21-Dec-15 01:06 linux-master/Makefile
-rw----     0.0 fat    18599 tx     7294 defN 21-Dec-15 01:06 linux-master/README
-rw----     0.0 fat     7485 tx     3152 defN 21-Dec-15 01:06 linux-master/REPORTING-BUGS
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/arch/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/block/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/certs/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/crypto/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/drivers/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/firmware/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/fs/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/include/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/init/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/ipc/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/kernel/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/lib/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/mm/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/net/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/samples/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/scripts/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/security/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/sound/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/tools/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/usr/
drwx---     0.0 fat        0 bx        0 stor 21-Dec-15 01:06 linux-master/virt/
unzip -Z -l linux-master.zip 'linux-master/**' -x 'linux-master/*/**'  0.02s user 0.00s system 90% cpu 0.024 total

It's also fast, because ZIP contains central file directory located at end of file.

mentioned in issue #4314 (closed)

Marked the task Check large file support (> 4G) as completed

Marked the task Extracting single file as completed

Marked the task Compare to CLI ZIP implementation as completed

Single file extract prototype:

package main

import (
	"archive/zip"
	"flag"
	"io"
	"log"
	"os"
)

func main() {
	flag.Parse()
	args := flag.Args()

	archive := args[0]
	file := args[1]

	ZipFileWriter(archive, file)
}

func ZipFileWriter(zarchive, zfile string) {

	archive, err := zip.OpenReader(zarchive)
	if err != nil {
		log.Fatal(err)
	}
	defer archive.Close()

	for _, entry := range archive.File {
		if entry.Name == zfile {
			file, err := entry.Open()
			if err != nil {
				log.Fatal(err)
			}
			defer file.Close()

			_, err = io.Copy(os.Stdout, file)
			if err != nil {
				log.Fatal(err)
			}

			break
		}
	}
}

Consumes similar amount of memory like file-lister (linux-2.6.0.zip ~8MB), ~3MB when using CLI implementation.

Conclusion

We will use Workhorse to extract single file (as we will benefit from streaming file content to user), and we will use Ruby backend to generate metadata for given path in ZIP archive using CLI implementation of Info-ZIP.

I've also checked ruby-zip gem.

Listing all files located inside the archive consumes about 64964 kB of memory. Compared to CLI implementation (3MB) and Go implementation (8MB) it is highly inefficient.

require 'zip'

Zip::File.open(ARGV[0]) do |zip_file|
  zip_file.each { |entry| puts entry.name }
end

Marked the task Check ruby-zip gem as completed

@dzaporozhets What do you thing about using CLI implementation of ZIP (unzip command) here? It looks like most reasonable option to generate artifacts metadata for each subsequent request in artifacts browser, but it is additional external dependency.

@grzesiek you mentioned also GO implementation. What are benefits of CLI compare to GO beside 2x memory?

@grzesiek I think using 8MB for linux in Go implementation is acceptable. I like that it the Go implementation has lower IOPS: "Go implementation has very low impact on signals/calls, including IO operations (only about 14% of total calls/signals)." And not adding a dependency would be great.

@sytses, @dzaporozhets Main advantage of CLI variant is lower implementation effort, and decoupling at least half of logic from Workhorse (workhorse will be used to download single artifact, it doesn't have to be used for browsing artifacts).

And info-zip makes it possible to generate only partial metadata (@ayufan gave an example above, but it seems to inflate as much memory as generating entire metadata for me).

Decoupling artifacts browser from Workhorse also makes it easier to maintain and lowers contribution barrier for other people.

Nonetheless I don't have strong opinion on using info-zip as it may have some security flaws (quite much CVS from 2015, link somewhere above), it is external dependency and using workhorse feels somewhat more idiomatic for me here (but I didn't have an opportunity to look at Workhorse yet, so this may be a false feeling now).

Tough decision.

Note that this 8MB is valid only for linux-2.6.0.zip archive I used for prototyping. Size of this file is 50MB and it contains about 15000 files. Amount of memory consumed may differ depending on size of central directory (see ZIP Structure), but I believe that this is quite a good sample.

@grzesiek if you dont have strong opinion on which we should use - I propose we go with less dependencies. Cc @sytses

@dzaporozhets I think that is a good direction to give a try to ZIP implementation in Workhorse.

I will eventually get to know Workhorse better :)

After investigating GitLab-Workhorse I think that using it to extract archive metadata adds quite much complexity to this solution, compared to using info-zip. I will try to talk to @marin today to ask him few questions about Workhorse.

I had a call with @marin today, to discuss unzip Go implementation in Workhorse.

It looks like implementing this feature in Workhorse would introduce much of complexity, and we would need to extend workhorse with connection to Redis (which will probably, eventually, happen in near feature).

We decided to go with simplest solution (as we value boring solutions) and to generate artifacts metadata using ci-runner, and send it along with artifacts archive.

This will add some redundancy, as we already have metadata inside the archive, but we will have full control over information, format and won't need to depend on additional, external utilities. It will be also more simple to serve metadata using API.

Then, in the future, it will be possible to change a format o metadata file to, for example, binary file using something like hash map to speed up lookups for entries relative to specific path only.

Another benefit is that we will be able to write better specs for artifacts browser, as - at this moment - we are not able to exercise entire GitLab stack with specs located in rails application because we do not preload Workhorse in specs now.

@grzesiek simple sounds good /cc @dzaporozhets

Marked the task Artifacts metadata fixture as completed

Another work that has been done today:

Building Gem with native extension written in Go.

Purpose of this was to prepare gemified library which would make it possible to generate ZIP archive metadata faster, with less memory footprint and to expose each node as Ruby object. I've used a Go native extensions compiled as C shared library, then binding methods using FFI (new feature in Go 1.5). This failed because I run into intermittent SIGSEGV issues, and wasn't able to debug it with gdb (no debug symbols present that would hopefully give a hint where the problem is) and I didn't want to spend too much time on it.
Preparing benchmarks on iterating big files in Ruby plain-text vs. gzipped. It looks like Zlib does quite a good job, and impact on resources being used is acceptable, given that it saves a lot of storage.

Summary: We will store metadata generated by Go application (Workhorse preferably, or CI runner) in .gz format, then iterate this file in rails backend on each subsequent request to artifacts. It is possible to add Redis cache (using redis-objects for example) later if this will not be efficient enough. With this approach it is possible to leverage full metadata, improve performance, and yet minimize complexity.

Go application that generates gzipped file with metadata: https://gitlab.com/gitlab-org/gitlab-ce/snippets/12770.

This will be used in Workhorse (preferably) or Runner.

Do we need compressed_size in metadata?

Marked the task Implement mechanism that will generate metadata as completed

@grzesiek awesome! and would indeed be great to have this in workhorse and keep the runner as simple as possible

Marked the task Implement mechanism that will generate metadata as incomplete

We should implement DoS mitigation mechanism in Workhorse (or Runner). DoS is possible when malicious artifacts archive is received (with more than 1 000 000 files inside). We should not generate metadata, when archive contains more than about 20 000 entries (should we make it configurable in Workhorse?).

Marked the task Check for possible vulnerabilities in handling arbitrary paths in metadata as completed

We just had a call with @marin, @jacobvosmaer and @ayufan, talking about unzip and generating metadata for artifacts and - what is important - for GitLab Pages (as this issue is connected).

Solution for now is:

Add unzip dependency to GitLab EE, to make it easier to extract pages artifacts
Generate artifacts metadata in Workhorse as previously decided
Go implementation for metadata generator is something like ~~https://gitlab.com/gitlab-org/gitlab-ce/snippets/12770~~ https://gitlab.com/grzesiek/artifacts-metadata
This should be placed in separate file in Workhorse, but in same package main, and executed as subcommand
With this approach we create a coupling between GitLab Rails -> Workhorse, so we have now two direction coupling
But it is easier to add little-application into Workhorse and expose a command than to introduce new mechanism for managing little-Go-application from Rails (@jacobvosmaer suggested a rake task for go get'ting Go apps, but then stated it is too complex for now, and we should go with something less complex).

@grzesiek thank you for logging all process in issue via comments. I am really glad you try to follow boring solutions

Status changed to closed by merge request !2123 (merged)

Marked the task Implement mechanism that will generate metadata as completed

Marked the task Change compression format used by CI Runner if decided on other than tar.gz as completed

Marked the task Add implementation for creating TAR/ZIP metadata as completed

Marked the task Prepare mechanisms that will extract single file from artifacts archive. This should be done by workhorse. as completed

mentioned in issue #8079 (closed)

mentioned in commit f03da18e

Status changed to reopened

Part of this feature is being implemented in gitlab-workhorse. It is not clear if that can be finished on time for 8.4. Considering it is not finished ATM, maybe we already know that it is not on time. cc @rspeicher

The code that is now in https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/28 does not have tests yet, it has a design that we are suspicious of (it does work in the main workhorse process that may cause memory leaks).

Marked the task Write documentation for artifacts browser as completed

Status changed to closed

I'm closing this issue as we managed to include it in 8.4

Browse build artifact files with the GitLab UI

Designs

Child items ...

Activity

Extracting a `tar.gz` with `tar -zxf`:

Heap allocations:

Calls (including I/O operations)

Making a list of files with `tar -tzf`

Heap allocations

Calls (including I/O operations)

Extracting last file (`tar -zxf`)

Heap allocations

Calls

Extracting zip with `-q` option

Heap allocations

Calls

Making a list of files

Heap allocations

Calls

Extracting single file

Heap allocations

Calls

Extracting

Heap allocations

Calls

List of files

Heap allocations

Calls

Extracting single file

Heap allocations

Calls

Conclusion

Admin message

Admin message

Browse build artifact files with the GitLab UI

Activity

Extracting a tar.gz with tar -zxf:

Heap allocations:

Calls (including I/O operations)

Making a list of files with tar -tzf

Heap allocations

Calls (including I/O operations)

Extracting last file (tar -zxf)

Heap allocations

Calls

Extracting zip with -q option

Heap allocations

Calls

Making a list of files

Heap allocations

Calls

Extracting single file

Heap allocations

Calls

Extracting

Heap allocations

Calls

List of files

Heap allocations

Calls

Extracting single file

Heap allocations

Calls

Conclusion

Extracting a `tar.gz` with `tar -zxf`:

Making a list of files with `tar -tzf`

Extracting last file (`tar -zxf`)

Extracting zip with `-q` option