Time tracking: reports for Issues and MR

Harvest has a nice starting point for reports:

For this idea, we would need the ability to set a capacity (in hours per week) for each member of the team.

Mentioned in issue #1293

What about export to XLS / CSV?

@regisF yes yes yes, that is what I like :)

Additionally, I would include this on Graphs part/tab under it. Make sure you also open future opportunities to link existing graphs (commits, CI builds) to hours spent estimate - that might be interesting to see how much someone actually spent on each commit.

Mentioned in issue #1301

@landsman I can see that being useful. I've created an issue for that: https://gitlab.com/gitlab-org/gitlab-ee/issues/1301

@Letme we don't track time on individual commit at the moment. It's either on an issue, or on a merge request. If we do it at the commit level, we would need to parse commit messages to do it, which we've decided not to at the moment, because it's more complex.

@regisF I have just mentioned it for future development possibility.

@Letme I've actually created a separate issue for that. Feel free to add your thoughts on it #1302

Mentioned in issue #1265 (closed)

Report for members

Option A is centered towards "what has been done" in the time range, regardless of the day.
Option B is for knowing what people have done every day in the time range.

For both option, clicking on a row shows details about the member.

Option A

Option B

Your feedback would be much appreciated @victorwu @JobV @markpundsack

@regisF what about questions:

"How much time a member spent per project"?
"How much time all members spent per project"?

As discussed in person:

reports should have a button download report
I think we should leave out capacity for now and all related UI elements
Option A looks like the simplest MVP. Maybe exclude listing individual contributions, although if we can include that, that'd be great

Initial concepts look great.

Who are the initial users and basic scenarios at the beginning?

If the purpose is indeed for time tracking so that a manager can take formatted was-working-on-data of direct reports, and analyze it. Then likely the export functionality makes the most sense, especially if a manager plans to reviewing it in an spreadsheet setting, or even combine it with other employee performance data. In some cases, the data might even need to be imported to other systems. I'm not too certain what are the more detailed scenarios here, but I do know there's plenty of tools that people use where folks need to clock in and clock out. Not sure if that market is huge for people who use GitLab though. I suppose that maybe for small consultancies it's important to get these numbers and report on them to expense clients. In any case, if this is important, then nailing the export format is more important than presenting it in a dashboard I would say. So that they could use it along with other numbers, say in Microsoft Project for example. But again we have the trade off of integrating with existing user workflows and fleshing out the entire thing.

If the purpose is for a collective team to view their own progress and retro on it and improve as a team, I think team metrics make more sense, instead of individuals. Typically these may be grouped or filtered within a date range of sprint. For us, a sprint could be a milestone. But again, why would we need detailed individual metrics in this case. Team metrics are more important.

For my use case, I would need to be able to track time that was worked on R&D Proof of Concepts and time used for building new features. Currently, these types of worked are tracked by tags. So, it would be very helpful for having the ability to report time spent based on certain tags. We have to report this information for R&D Tax Credit quarterly.

@regisF As @victorwu alluded to, the purpose of these reports is unclear. I can't evaluate this proposal without knowing "why" you want these features. Please consider expanding the Description to include "why", expected use cases, problems to be solved, jobs to be done, etc. Just "visualizing all this data together" isn't really a valid goal. :)

Harvest looks pretty interesting, for example, because it looks like it helps consulting shops understand their billings and billing ratios per person. I can imagine use cases for your member report, but I can also imagine it just being used to check up on people and make sure they're working hard enough, which doesn't seem like a great goal (nor one that actually works).

The issues report kinda looks like it might help people understand when a project may finish, but then a burndown chart or gantt chart might do better there. So, personally, I'm struggling to see what someone might use this for.

Thanks @markpundsack and @victorwu.

Reports are aimed towards two types of people: developers and managers.

Time tracking is the mechanism that lets us track things. The natural concern for companies who use time tracking is: where have our team members spent their time. This is the question that keeps coming in the feedback we are receiving. This is answered by the first table.
The first table also answers the question: as a developer, where have I spent my time last week, when I was working on this project?
The second question that we want to answer is what has been worked on last week, for issues/MR which have time tracking data. There is currently no way in GitLab to know it accurately, with data we can export. This is especially useful for managers who want to know what are the issues/MR that have taken the most time of the team.

I'm going to refresh the body issue with updated visuals and this description.

@regisF @markpundsack @victorwu good points. Seems like we're all making assumptions now.

@regisF what has feedback been pointing towards? What did the potential customers would use this for?

I can imagine that the current implementation of time tracking is too limited a seed for proper feedback. I can also imagining the simplest visualization of time spent to be a better, so I'd support any way to move this feature forward to a stage where any kind of company would consider using time tracking if GitLab implements X or Y.

@JobV I agree that this is the interpretation of all the feedback we've received. People want reports for the things below, but they don't know which forms those reports should have (even when talking to them on the phone):

what have people done
which issues take the longest
what is the breakdown per issue (addressed here: https://gitlab.com/gitlab-org/gitlab-ee/issues/1347)

There are a million ways to visualize this data. At the project level. The group level. The member level. The instance level.

What about providing a nice exporter and let our users visualize the data how they want, in a way that makes sense to them?

The exporter would allow to:

choose issues, MR, or both
choose a specific member who entered time tracking data, or no one in particular
choose between the total of time spent (:one line per issue), or the breakdown (:multiple lines per entry)
choose a time range

The CSV would include all time tracking data related to these options combined. And should cover all the questions we could have, at the project level to begin with.

@regisF good idea. Maybe consider other integration points for time tracking as well and asking a potential customer what they'd want.

@warren.postma @ landsman can you let me know what you think about the idea above (export to CSV), so you can generate the reports you need? I'm asking you because I've seen you active on the time tracking topic in GitLab. I'd appreciate your feedback. Thanks!

I think that's a both-and kind of decision. JSON + CSV exports are going to be popular. I think at least, both a JSON and a CSV export, and a manual (web view) and automated (Rest api) endpoint should exist.

I personally would not be excited to build something new myself around CSV because I'm a developer and developer manager and I think CSV tools are a bit dated. But don't let me discourage the idea. I think I'm not the target market for it. I would be more likely to want to consume JSON.

I think it's safest to assume as little as possible about what the user wants to do with the data. I might not care so much about the total hours but more about whether or not the data is likely or unlikely to be accurate. So some people are logging 10 minute increments, some are logging in 4 hour increments, and some are logging 40 hour increments. Wow that's a big difference in precision. I wonder if the 40 hour increment data is accurate to within +/- 20 hours, or what?

So to start with, I would just provide the simplest possible UI (start date, end date, and scope combobox giving me a choice of Project, Site, or Single Repo). Then just dump the data to the user's choice of JSON or CSV, in the most high fidelity format you can.

@warren.postma thanks a lot.

Keep in mind that you can import CSV into Google Doc or Excel and create the report you want from it (with filters, sums, etc..). Moreover, new API methods for time tracking are coming to integrate with apps or your systems.

The new API methods look super useful.

I wonder if you could publish some RSS feed which Google Docs Spreadsheets could consume directly.

You can use "import-HTML" functions in Google docs, and there are other similar ways to directly "read web data":

http://www.labnol.org/internet/import-html-in-google-docs/28125/

=importHtml( "https://mygitlab.com/api/TimeOutput?apikey=ABC" )

People love CSV exports (https://news.ycombinator.com/item?id=13157062)

Oh gosh yes. Companies love their CSV exports! I think the reason is that it lets them make glue - they can take data from something, play with it, analyse it, produce graphs, using tools they understand (almost invariably Excel), and sometimes then import into another system.

mentioned in issue #1401 (closed)

changed title from Time tracking: reports to Time tracking: reports for Issues and MR

Reports about users, if limited to the project level, would give a false overview of time spent. If the goal is to know how much time was spent per user, we have to have the bigger picture: display the work done across all the projects. This becomes out of scope for this issue as we are talking about providing reports at the project level.

I've created a new issue for this https://gitlab.com/gitlab-org/gitlab-ee/issues/1406

Let's focus on time spent on issues/MR.

The second most popular time tracking addon on Atlassian's marketplace: https://marketplace.atlassian.com/plugins/jira-timesheet-plugin/server/overview

@awhildy two things

can we get some UX love for this?
where would we place that feature? I wonder if it's time to rename Graphs to Reports and put a tab inside it.

Hi guys, we are so close to considering purchase of Gitlab enterprise, we really would like to see a feature along the lines of... https://confluence.atlassian.com/agile/jira-agile-user-s-guide/using-a-board/using-reports/viewing-the-version-report

where perhaps we can visualise how far along we are to completing a milestone and what the worst case scenario and best case scenario as shown in the Jira link above.

Is this a coming feature, perhaps I can create an issue for this?

added ~126459 label

@regisF - sounds good. @tauriedavis: Do you have the time to help out with this (as you worked on the initial time tracking UX)?

I would support renaming Graphs to Reports and adding this as a tab underneath. Makes sense .

Yeah I will take a look at this @awhildy! Thanks for the ping. Will this be scheduled for 8.16?

@tauriedavis 8.17, so we have a bit of time

Great, thanks @regisF !

assigned to @tauriedavis

mentioned in issue #1490 (closed)

Don't forget me @tauriedavis

@regisF - Just a FYI - @tauriedavis is on vacation for the rest of the week. If something needs to happen this week, let me know. Thanks!

I won't forget you @regisF ! This has no milestone, should 8.17 be added?

changed milestone to %9.0

@tauriedavis 9.0, actually

We are working on removing the graphs tab in 8.17. See https://gitlab.com/gitlab-org/gitlab-ce/issues/18713

Due to this, I think we should rethink where this new table will live.

One option would be to have Time Tracking be it's own tab under Issues and Merge Requests. If we are thinking about adding a table for member data later, this might not work.

Another option would be to think of time tracking as activity and make it it's own section under activity. This doesn't work well with the All tab we have on this page because mixing time tracking with other activity info doesn't make much sense so we would need to look at splitting activity into event data and time data.

Since we are moving graphs under repository, maybe we add another tab to repository for time tracking.

I'm not sure these are the best suggestions so please feel free to offer more! cc @awhildy for your opinion here as well.

@tauriedavis

A major area in the future for EE will be reporting of all kinds. It's unclear yet if the reports will be accessible where the different features are, or if we will have a dedicated section for it. Food for thoughts.

That being said, time tracking is used both in issues and MR. Moreover, we'll provide at some point time tracking for members as well. Perhaps Activity would be the best option - I need some time to think.

@regisF @tauriedavis - Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/26348 is actually moving the activity tab under 'Project' (see this discussion https://gitlab.com/gitlab-org/gitlab-ce/issues/25386#note_20042859). So, under the activity tab may not work. But maybe then Time Tracking could be under the 'Project' tab?

@awhildy Is a design being worked on for https://gitlab.com/gitlab-org/gitlab-ce/issues/26348? It's hard to tell from the issue if frontend is jumping right in before UX. I'm not sure how the Project tab will look after that issue but it would help to know that when deciding whether time tracking makes sense there. I worry we might be throwing too much into the project tab - readme, activity, cycle analytics, and now maybe time tracking - without spending adequate time on the UX. I think it makes sense to include a lot of this in the new smart dashboard we've discussed but I'm not sure if that is happening quite yet. Are we taking the time to design a new project page that is coherent with all this new information in one view?

No. Unfortunately the smart dashboard is not in scope for 9.0, but this navigation work is. The goal is to prioritize work that may be a little bit of a change, and thus a little painful to adjust to (like the navigation work), because we can easily target features that are just clear wins (like the smart dashboard) in subsequent releases. That being said -- I would really like to push for it soon.

@dimitrieh - Would you be able to mock up the tabs for the different navigation work in the issues linked to from https://gitlab.com/gitlab-org/gitlab-ce/issues/26348. Can you look into what impact adding 'time tracking' would add to the Project tab and what your thoughts are on that? Thanks!

@awhildy posted a reply to this on https://gitlab.com/gitlab-org/gitlab-ce/issues/26348#note_22436184

@regisF - Is this work still scheduled for 9.0? If so, is it a Deliverable or a Stretch? Thanks!

changed milestone to %Backlog

@awhildy moved to backlog but I think we should do it for 9.1.

added ~874214 label

@victorwu I think we should plan the next iteration of time tracking. What do you think?

Unassigning from myself. @regisF If it is looking like this may be added to an upcoming release please let me know and I'll see if someone on the team is able to pick it up.

removed assignee

Is there any indication on when this might be available? I would also love to see this in the Enterprise Starter..

@bobvandevijver Thanks for checking in! It doesn't look like this is on the pipeline, currently, since it is not labeled with ~"coming soon" and has not been scheduled for our next release. We typically only schedule 1 milestone in advance. That said, it is in Backlog so it won't be lost.

cc @victorwu

I really hope you push this feature forward as I don't see current time tracking at all useful - if team leaders can't report the hours done by developers, it's obligatory to have some parallel time tracking solution in use and it's quite stupid to ask developers to fill hours in two separate systems.

Together with time tracking through commits, this is also something we would really like/need. Unfortunately, in the meantime, we are still dependent on Jira.

If someone knows a time tracking tool that can hook into GitLab (commits), that would also be highly appreciated.

If you are looking a tool that can generate reports based on gitlab time tracking records, you can take a look at this project(https://github.com/kriskbx/gitlab-time-tracker).

This seems to have been kicked down the road several times - what is the current plan for this?

Hi @jbrandhorst, thanks for asking. Currently this is in Backlog and has not yet been scheduled. We typically only schedule 1 milestone in advance.

Cc'ing product for eyes and scheduling when there is availability @JobV

Hi @tauriedavis, @JobV. We're going to implement reports and want to discuss that with someone from the team, to merge that into gitlab upstream. Who is a suitable person for this discussion?

P. S. Right now we're using the separate app, that query api/db and prepare some kind of reports.

@g3dinua : Are you saying you would like to contribute to GitLab as a community contribution?

For this particular feature discussed here, I think it makes sense to iterate on our existing CSV export ability first, before considering a dedicated UI inside of GitLab.

Would you be interested in contributing to expand the CSV export functionality (i.e. more columns with time tracking info?) The currently functionality is here. https://docs.gitlab.com/ee/user/project/issues/csv_export.html. It would be part of the EES tier of GitLab.

In particular, we already have this issue here: https://gitlab.com/gitlab-org/gitlab-ee/issues/1406. If you are interested, we can continue the conversation there.

@victorwu it does not matter so much, is that CE or EES/EEP. The main points of pain we have right now:

no export functionality/UI
no ability to log time for selected date.

So our idea is to finally do it, but we want not just make a fork & maintain it all time, but merge that back into upstream master, that's why we need some communication & feedback from Gitlab team. We can start with #1406 (closed), and discuss other improvements after we'll finish with that. Sounds like a plan?

Thanks @g3dinua . Added a comment to #1406 (closed) and pinged you there. Let's continue the convo there.

@victorwu do you have any task for "ability to log time for selected date". Pretty often developers forgot to log hours at the same date and there is no ability to log time for any other date different than today.

The leanest possible solution is to expand /spend quick command with part of functionality from /due quick command, that will help to detect data. Do you have some feedback about that idea?

@g3dinua : Time tracking and due dates are separate features (currently). Look at the docs here: https://docs.gitlab.com/ce/workflow/time_tracking.html

Basically you can keep updating the "estimate" of how much time an issue will/should take.
And then you can keep logging how much actual time was spent.
And there is UI to tell you the difference between the estimate and actual.

Note that the above is totally divorced from due date functionality.

Time tracking has no reference to actual points of time. Where as due dates do refer to actual points of time.

I think that design is nice right now since it's clean and uncomplicated. Regarding your request, if a developer logs any time spent, it simply goes to a bucket. It doesn't associate it with a particular date in which the logging happened. Does that work for your use cases? If you do indeed want to start tying those two concepts together, feel free to create a new issue and let's chat about it. (We don't have anything like that with ongoing discussions.)

@victorwu in the wireframes concept above there is a link between time and spend time (http://d.pr/i/y6eN8S), so I'm curious how that should be implemented if we don't have a link between time spend and any timestamps. If that's part is not implemented, let's create new subtask and discuss how to implement it.

@smcgivern : Correct me if I'm wrong. Time spent and time estimate are just two fields associated with an issue. They are unrelated to due dates correct?

@g3dinua : That's my understanding. So I think https://gitlab.com/gitlab-org/gitlab-ee/issues/1406 is clear right? And we can definitely talk about any other use cases separately.

@smcgivern @victorwu

Time spent and time estimate are not related to due date command. From the short investigation, it seems that time spent and time estimate are "timelog" records (https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/models/timelog.rb, correct me, if I'm wrong).

So next task we're going to work is to add the date to indicate which date should be used for timelog record. Seems like there is issue for that #1312, we can continue that part of the conversation here.

I'm managing a freelance team and gitlab time tracking is the optimal way to understand where effort is being allocated and how much we have to pay developers. Unfortunately, this information is currently dying within the issue/MR. Being able to generate a report out of it would be crucial for us to actually stop using external time-tracking tools.

issue/MR	issue/MR #	title	time spent entered in seconds	user #	username	full name	time spent so far in seconds	estimate in seconds	timestamp
Issue	123	Set up review apps for `gitlab-ce`	14382	313	luke	Luke Skywalker	2098249	9302938	2016-10-10
MR	453	Fix relative bug	34254	398	henri	Henri Troyat	324423	897487	2016-09-10

Time tracking: reports for Issues and MR

Description

Proposal

Wireframes

Designs

Child items ...

Activity

Report for members

Option A

Option B

Admin message

Admin message

Time tracking: reports for Issues and MR

Description

Proposal

Wireframes

Activity

Report for members

Option A

Option B