Add timestamps/line numbers to job output

The job output is very nice, but it would be useful if it always had timestamps on the left-side, so you can see how long different commands took.

It would also be very useful to have line numbers in job outputs, I'm often trying to show someone a particular line in the job output, and have to guide them through finding it. If there was a line number, I could simply say "look at line 149". If those line numbers were link anchors, that would be even better because then you can just provide a link to the exact problem.

There is this gitlab-ce issue as well that proposes the same thing: https://gitlab.com/gitlab-org/gitlab-ce/issues/22745

Don't know which project it should be tracked in though.

@rymai and I discussed the need for this. Can we get this in soon?

/cc: @tmaczukin, @ayufan

/cc @bikebilly

@godfat IIRC you created an issue of linking to the build log line...

@dosuken123 This one? https://gitlab.com/gitlab-org/gitlab-ce/issues/28291 Thanks for bringing this up.

@stanhu @micah Having log lines should be relatively easy to add, and could be handled only on the GitLab's side. This is only a matter of printing a raw file in a proper way, as we are already doing for code files. I know there are issues already describing this idea.

With timestamps there is one problem - currently we're storing job's trace as a raw file on GitLab's filesystem. Doing incremental update with sending the PATH /api/v4/jobs/1234/trace request we're doing a simple comparison of files and file operations. When sending a full update with trace, we're just dropping a text blob into a file.

To have timestamps we would need to change a way that the file is stored to know which part was send with which timestamp. The easiest way would be to store such data in database, but we already migrated job traces from the database into files to make things faster and to take some load off the database.

Anyway I think that this idea could be also fully handled at GitLab's side without need to change anything on Runner's side. I would propose to:

1. Think how this could be handled in a clean way at GitLab, so we would have all data needed to print line timestamps,
2. Check if there are any additional information that need to be send from Runner to make things work.

@tmaczukin Adding timestamps to each lines would be nice, but I think if we could get timestamps for each commands would be great, too. Does that make sense to do in the runner then? Of course, we still need to update GitLab for storing and displaying the information, matching against each commands, for example.

Use the same approach as Travis CI. Do annotated output:

\travis_time:<unix_timestamp>\r\ANSI_CODE_TO_CLEAR_LINE

Fix support for that when rendering a build trace.

Still, I would prefer us to start sending json chunks with timestamps, and instead of storing full trace, storing a chunks, always.

mentioned in issue #2513

mentioned in issue #2535

While 3 months passed after labeled as %Next 2-3 months, any progress?