Show test result with web interface

@sgallerani It's not entirely clear what you're referring to. What are "test results"? In GitLab CI, you can see the standard output from whatever your build process does. There are several issues open discussing how to improve this build output (see #7141 (closed) and #14664 (moved)).

And with artifact collection, you can collect files produced by your build (like test reports, etc) and they will be kept on your GitLab server. Currently, you must download these artifacts to browse them, but this will hopefully be addressed soon (see #10982 (moved)).

I try to explain with more details my point.
Let's suppose that during the build process, the testing framework that concretely runs the tests, produces for every test a report in some format (e.g. junit xml or whatever), the reports contain the test name, test status (passed, failed, ...), duration, and if failed the failure message.
In my experience in Jenkins is possible to use this kind of test report to get statistics about build, so for instance, the number of test run, how many pass, how many failure, the duration of every test. Moreover is possible to get information about the evolution of the test status, for instance is possible to know if there is a test that in the previous build was passing, and now is failing, or even if new test have been added.
So it is a sort of post processing of the tests reports in order to get info about the tests and the story of the tests, and it would be visible from the web interface
(in this case would be gitlab ci).
I was wandering if gitlab ci allows to do something similar or is in the plane.
Thanks.

GitLab CI does not have such functionality today.

Check out these open issues, which hope to add this functionality in a generic way:

#13227 (moved) - CI Feature Request: Allow arbitrary build "results"
#13409 (closed) - Code coverage graphs and adding more metrics

I could think of a special yaml extension for artifacts, say something like "type: junit", which in turn triggers the rendering of a view in the pipeline to show the results. Sure, today you can zip and name the test results and extract them to your local place and then look into it. This is slightly inconvenient.

Added ~19173 feature proposal labels

As a workaround, you might want to consider Danger's JUnit plugin.

Mentioned in issue #22098 (closed)

This is absolutely crucial feature for our enterprise and #1 (closed) reason why we dont switch over from Bamboo. To be able to see failing tests is something our whole QA team is using extensively - (and in Bamboo they are linked to JIRA issues).

Is there any possibility that this item could become sooner on your roadmap (is it even planned)?
@markpundsack

@dragantomasevic Thanks for your feedback. This does seem like a fairly critical piece for a lot of users, but we've got a lot of other issues to work through for the next couple releases and we don't like to plan more than a release or two ahead, except for direction and other visionary issues. This also isn't likely to be easy. But we do understand that you and others need this functionality, and that does impact our planning.

Milestone changed to %Backlog

I just wanted to comment and say this is a very important feature for us as well. In addition to code coverage html reports, we would like to see support for any html reports for any stage of the build process. For example, we have a code quality stage in our pipeline that runs checkstyle against our code base. This tool spits out a simple html page to view the results of the checks. If this stage fails, developers will want to see the report to correct the failures. We also have a minimum code coverage check in place in another stage which will fail if that threshold isn't met. The html report makes it easy to determine what needs to be fixed. As of now, developers have to download the artifacts, unzip the file, dig through files and find the page they are interested in.

@markpundsack I think this could be closed as a duplicate of #13227 (moved).

To your statement, I agree that this (and thus #13227 (moved)) are not trivial. Having a shortcut/iframe/embedded content in the build view won't be easy to support for all use cases.

However, #10982 (moved) is (IMO) long over-due and much simpler. If that were done, people could at least view their HTML reports from the browser, like @rpipino wants. (rpipino, it may help to voice your concerns there.)

Mentioned in issue #10982 (moved)

@jonathon-reinhart My understanding is that this issue is about processing JUnit XML results and rendering them, so I don't consider this to be a duplicate of #13227 (moved). But I agree that showing HTML reports in artifacts would be a lot easier, and might work for many people, especially if they're already creating HTML reports from their JUnit results.

@markpundsack You are correct. I had only looked at the original post, and not the OP's clarifying comment.

The kinds of Unit Test outputs I'm aware of include:

HTML output. For example "reports\result.html" output by the Karma test runner.
XML and JSON report output, containing something like:

{ "tests_run": 100,
  "tests_skipped": 10,
  "tests_failed": 2,
  "failed_test_names": [
                       "should show names of failed tests",  
                        "divide by zero raises exception"]
}

Here is some more information on what I think would be great:

a plugin that I enable per job, in .gitlab-ci.yml so that the job has results associated with it.
As a very basic start, the JUNIT-style XML data capture parser should be the first plugin created.
Additional plugins for other formats are an obvious no brainer down the road.
A chart like the jenkins "Test Result Trend" chart below would be great.
A feature to show you the latest test result is very useful, when you want to be told "1191 tests passed, 0 failed, 17 were disabled/skipped", or other similar breakdowns that are pulled from key-value pairs in some XML or JSON.

Sample screenshot from Jenkins job which is reading a JUNIT-style XML unit test report, and generating a Test Result Trend, and also a list of recent Test Result failures.

Mentioned in issue #13227 (moved)

Mentioned in issue #13409 (closed)

Mentioned in issue #24883 (closed)

Frequently HTML report is the main feature of the build or particular stage. There are a lot of popular reports, for example Allure for web tests or Spoon for Android tests and etc. It's not necessary to parse something to publish such reports. This is just a directory which should be stored and linked to a build. Jenkins uses very simple HTML pulisher plugin that requires a path to report dir and path to index file. I think that it will be perfect to create Reports block with a links to HTML reports on corresponding stage's page.

I see a lot of space for it :)

mentioned in issue #27686 (closed)

I like the way TFS shows the results, although it needs usability improvements:

I would see the results in different ways/ from different viewpoints.

overall results from a merge request, test difference in compare to the baseline/ in compare to the target branch.
trend for a given branch
overall trend of a project.

http://allure.qatools.ru also interesting tool to learn from it besides Jenkins/FTS.

In gitlab spirit way of doing things, I would split the current issue in half.

First

We model and define the unit tests data that gitlab will need to leverage
We define a way to tell the CI to parse unit test results. For example, JUnit XML files. So we feed gitlab database.
We generate a simple report that's easy to read. We have that test coverage field we might simply add the test results close to it.
No history or trending
Report only for the current branch
One report per build only

Next steps

We introduce history / trend
We introduce inter-branch comparison
REST API for accessing Unit test data
Support for many other things we don't foresee right now...

@fboutin I suggest you look at #13227 (moved) which seems to be underway.

mentioned in merge request gitlab-com/www-gitlab-com!5803

Can I ask an honest question? Why is this not a much higher priority? I've been reading similar issues here that go back well over a year. Where do I go to find out what features are being worked on that are more important than reporting standard code quality issues?

I have a gradle build that generates reports for unit tests, integration tests, code coverage, findbugs (+ findsecbugs), animal sniffer (invalid bytecode access), and javadoc output, and it's going to get completely ignored by the devs on our team unless it's displayed in the build UI.

We'd also like to automate some nightly performance metrics to track regressions, but there also doesn't seem to be a way to report any sort of trend information across builds.

#13227 (moved) is also over a year old, and doesn't seem to be anywhere near releasing. I've never used a CI tool (and I've used most of them) that doesn't support any kind of integrated reporting of code metrics. The 'coverage' config doesn't count, as it doesn't allow you to see any details.

@lkeagle Don't worry though, we got Canary Deployments and Service Desk in 9.1

+1000 for this feature. if ci fails, why do I have to run the same build locally just to see where it failed?
Jenkins has gitlabs beat on this by far

@joshua.calloway because you always need build reproducibility? Also please people, stop comparing Gitlab CI with Jenkins. If I want to use Jenkins, I'll setup Jenkins. But I don't want Jenkins.

@fboutin This has very little to do with reproducibility. Even reproducible builds may fail, and then you want to see where it failed. And GitLab themselves encourage the comparison with Jenkins, see https://about.gitlab.com/comparison/gitlab-vs-jenkins.html for details.

@fboutin, it is very wise to compare a wonderful software like Gitlab (and Gitlab CI) with other wonderful softwares (like Jenkins) that do things in a similar or equal way, in order to identify points that should get better.

Of course, if somebody wants a software exactly equal to Jenkins, them, he/she should use Jenkins. But that is not the case here. Gitlab CI has some advantages over Jenkins, because of its integration with the code, so, if it can reproduce some of Jenkins features, it can replace Jenkins completely.

Gitlab team main goal probably is to make their client's lives easier. If a client could stop using Jenkins, and use only Gitlab, their objective would be met, since using only one tool is easier than using two or more.

In my opinion, this feature is blocking. Without it, me and my team won't migrate to GitlabCI. The same is true to many other teams.

@elygre of course you want to see and that's why you should rely on build output, including console and archived reports. BTW, because gitlab CE now supports Gitlab Pages you already can host your generated test reports, https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/

@bruno-eng-comp You sum it up all. You don't use gitlab-ci, period. And you mean you don't use it solely because of this missing feature? I doubt it.

@fboutin, here we have some pipelines. One of them is to generate our plugins. We build the plugin, test it, and if it is ok, we publish it in Artifactory. But we need an easy feedback to understand what exactly went wrong (low code coverage, failing test, or any other problem). So we won't migrate this specific pipeline because of this missing feature, since Jenkins IN THIS CASE fits better our needs.

Thanks for the link, I'll read it and see if it is what we are looking for.

But the important point is: a good community must be open to hear the feedback of their users. I really like Gitlab, and use it every day. However, I miss some features, including features that Jenkins has implemented already. As Gitlab-CE is an open source, people like me may contribute in many ways, including listing features that we have used in other softwares.

If the suggested features will be implemented, only the time will tell.

@bruno-eng-comp A good community must be able to argument openly even if we don't agree with each others, I agree with you on that. In fact, I'm only trying to help you here.

Is there any official indication from Gitlab Team that this is an accepted idea to come soon in future releases?

In case this is an option for anyone else:

We currently use a mix of GitLab CI and Jenkins; I see the platforms, in some respects as complementary. We build and test our software with GitLab CI and trigger a Jenkins job as part of our deployment to capture binaries/test results/code analysis back for fingerprinting/display/trending:

curl --insecure -X POST $jenkins_url/job/$CI_PROJECT_NAMESPACE/job/$CI_PROJECT_NAME-$CI_COMMIT_REF_NAME/buildWithParameters?pipeline_id=$CI_PIPELINE_ID --user $jenkins_user:$jenkins_api_token

Obviously I would prefer one tool that had all the functionality, but I really think we're getting the best of both worlds with minimal penalty using this approach and intend on using it for the foreseeable future.

@junalmeida In https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/5803#note_28659889, @JobV indicates that this is "Coming soon". I'm not sure what that means.

(The referred merge request is not to fix the problem, but to improve the comparison with jenkins on the web site)

mentioned in issue #4044 (closed)

added customer label

I was quite surprised to find that, even in the latest version, GitLab's CI doesn't parse and display JUnit XML files for itemized test results. Jenkins does it, Bamboo does it, and there are JUnit results formatters for most major testing tools (I just dropped it into my Jest tests, for example, expecting GitLab CI JUnit support to be a given). This seems like a significant feature gap that I hope gets bridged soon!

I'm not yet a GitLab customer, but if this feature were implemented, I would definitely consider it!!

We recently thought about that in a different context: https://gitlab.com/gitlab-org/gitlab-ce/issues/35379, but also https://gitlab.com/gitlab-org/gitlab-ce/issues/34102 actually makes it fully functional.

I'd like to point out that #34102 is (deliberately) scoped to only work for public projects. A solution covering only public projects is not usable for many of us, and it would not be accurate to call that "fully functional".

@ayufan - the ticket you mentioned seems entirely Java Specific. The JUnit XML reporting format is actually completely independent of the language that the program was written in, which is why e.g. Google Test uses it, which can be used for C/C++ testing.

And @elygre is correct - the feature should definitely not be limited to public projects.

@elygre You are correct it is current public limitation, but the idea is to figure out the plan to make it usable for everyone, and usable also for private projects.

I think this issue can be closed as it is covered by a few other issues:

Parsing JUnit: https://gitlab.com/gitlab-org/gitlab-ce/issues/17081
Viewing arbitrary HTML (public, private)
Tracking arbitrary build results: https://gitlab.com/gitlab-org/gitlab-ce/issues/13227

If there's anything in here that's not covered by those issues, feel free to re-open, or open a new issue.

closed

Show test result with web interface

Designs

Child items ...

Activity

First

Next steps

Admin message

Admin message

Show test result with web interface

Activity

First

Next steps