Paginate environments list

I believe !7196 (merged) will provide a reusable pagination component that we should be able to use in the environments list.

Maybe we could add this issue for the next release, @markpundsack ?

I'll leave it up to @ayufan to schedule. :) I'm not sure it's a practical concern yet.

@filipa I believe yes. We should solve that. Not sure how to fit into design of folders.

changed milestone to %8.16

assigned to @filipa

@ayufan

Not sure how to fit into design of folders.

Design as in UX?

Maybe we can keep the work done in pipelines, issues and merge requests. //cc @dimitrieh

Yes. I would say, what we should show if we have folders that are by design does not work very well with pagination.

Maybe we should return a some latest amount of environments and sort them based on deploy date.

@ayufan and @filipa is it correct that there needs to be a design here that paginates the environments within a review environments folder/group ? I believe it already should have functionality that if it exceeds a certain number of environments it will show a button to see all sub environments on their own view, with pagination not?

It seems this is not the case: https://gitlab.com/gitlab-com/www-gitlab-com/environments

I created a design for this here: https://gitlab.com/gitlab-org/gitlab-ce/issues/22539#note_16748043

Basically it shows you the first 10 results with a "show all..." button, which leads to a list with all sub environments plus pagination.

would this solve the issue?

@dimitrieh @ayufan @grzesiek I don't think it solves it, because the items inside a folder are already on the frontend, since they have to be changed into a new structure in order to be displayed as folders, so this would not help with the loading time/performance since it would be on the Frontend only.

It would solve if the "show all..." button would make a call to the API requesting more environments, which would not be the case with the code we have now.

We could make that work, but both Frontend and Backend would need some deeper changes:

The folder structure would have to be sent by the Backend instead of being calculated in the Frontend and for each folder we would need 2 new fields, one to know if there are more environments under that folder, and the other with the URL to make the request.
The view would have to be changes as well to support this, but is not a huge change.

However: I think we can make this work with a classic pagination.

When we render the view, the folders are closed. This means the number of visible environments will not match the number of environments received. But I don't think it is a problem.

Scenario:

We have a folder with 1000 environments

The user is on the first page and opens the folder, clicks to go to the second page. The second page would have the folder name and the environments sent in this second request.

TL;DR;

IMO, The prettiest more awesome solution would me a mix of the two

We need pagination, as we can see here https://gitlab.com/gitlab-com/www-gitlab-com/environments, I suggest we go for the classic pagination in a first iteration and if we realise it does not work/it won't scale, we can implement the mix of the two solutions.

What do you think?

@filipa its not entirely clear yet to me :) could you elaborate?

Scenario:

We have a folder with 1000 environments

The user is on the first page and opens the folder...

what happens here iyo? what will it show?

@dimitrieh @grzesiek

Scenario 1:

In the case the user has one folder with 1000 environments:

Will see one closed folder when he gets to the page.
When the user opens the folder, will see the first 30 (assuming we request 30 at a time)
When clicks next, will see the folder name and the following 30

Scenario 2:

In the case the user has several folders it could get trickier. But I think we can make it work if we order them by environment_type and date.

Scenario 3:

List of environments without any folder. both paginations would work, but wouldn't it be odd to have a different pagination here than in everywhere else?

Also, with the classic one we can share a link to the exact page. We can't do that with the 'show all ... ' button.

@filipa What do you think about Scenario 4:

there are hundreds of folders
there are hundreds of environments in each folder

Proposed solution:

there is a separate API call to get the list of the folders
we paginate folders the usual way
there is a separate API call to get environments inside folders
when someone opens a folders, there is an inner pagination as well

This is more like your proposal to mix frontend and backend changes here. This is a significant effort required to implement that, on both, backend and frontend sides. Any ideas how to make is more simple?

mentioned in issue #25419 (closed)

@grzesiek

I don't fully understand this:

there is a separate API call to get the list of the folders we paginate folders the usual way

is it the same as this?

The folder structure would have to be sent by the Backend instead of being calculated in the Frontend and for each folder we would need 2 new fields, one to know if there are more environments under that folder, and the other with the URL to make the request.

My suggestion is the API will send an array with both environments and folders with environments instead of doing that transformation in the frontend. Something like this:

[
  {
    "name": "foo",
    "type": "folder",
    "children": [
      {
        "name": "env 1",
        "type": "environment",
        "last_deployment": {},
        ...
      },
      {
        "name": "env w",
        "type": "environment",
        "last_deployment": {},
        ...
      },
    ]
  },
  {
    "name": "environment",
    "type": "environment",
    "last_deployment": {},
    ...
  }
]

We can do this in two iterations,

the first one with this structure and a classic pagination
And the second one where we add the 'show all' button inside the folder

For the second iteration we would need:

[
  {
    "name": "foo",
    "type": "folder",
    "hasMoreChildren": true,
    "children_endpoint": "api/...",
    "children": [
      {
        "name": "env 1",
        "url": "ssa",
        ...
      },
      {
        "name": "env w",
        "url": "ssa",
        ...
      },
    ]
  },
  {
    "name": "environment",
    "type": "environment",
    "url": "...",
    "last_deployment": {},
    ...
  }
]

What do you think @grzesiek ?

@filipa My point was to introduce two separate endpoints (one for getting the paginated list of folder, and second for getting the paginated list of environments inside the folder), because when you have folder with hundreds of environment you will need additional pagination anyway.

@grzesiek I believe we are on the same page then, correct?

@filipa I think we agree that this should be handled by backend

added ~901060 label

I think both my and @grzesiek proposal will work ;)

added Deliverable label

@grzesiek @filipa me and @pedroms talked this issue over and think the idea from https://gitlab.com/gitlab-org/gitlab-ce/issues/25499#note_19929799 can be considered best, as we were opposed to have double pagination within 1 view.

unlabeling this UX

removed ~19492 label

@dimitrieh just to make sure, we would have:

a 'load more' button under each folder that has more environments
a 'load more' button in the end of the table to load more folders or root level environments

I like this idea, but I have 2 concerns:

this would be the only table in the all site with a different pagination. Shouldn't we keep consistency?
traditional pagination makes it easier to share a page, we won't have that with the 'load more' button

//cc @grzesiek @pedroms

@filipa mmh yes let's make sure we are on the same page on this:

How I think it should be:

https://gitlab.com/gitlab-org/gitlab-ce/environments will list both static environments and dynamic environment folders (review apps)
if enough static environments and dynamic environment folders are present, it will create pagination to paginate those rows.
the folders are expandable
the folders will show 10 items max when expanded and a "load more" button if more sub environments are available
the "load more" button will load up a page specific to those sub environments with a unique url

this would be the only table in the all site with a different pagination. Shouldn't we keep consistency?

I don't see how this will be inconsistent? Is there any other page with nested items? having double pagination would be inconsistent as well correct?

traditional pagination makes it easier to share a page, we won't have that with the 'load more' button

The load more button will open up a page and unique URL dedicated to the subenvironments on that folder.. which has its own pagination

Also with the double pagination on the same page, you will loose unique sharable url to that specific page within the environment folder

please let me know if anything is still unclear

I agree with @dimitrieh's suggestion. Just to further clarify so there's no confusion:

if enough static environments and dynamic environment folders are present, it will create pagination to paginate those rows.

In …/environments there will be a pagination control (1, 2, 3, …) for static environments and dynamic environment folders only.

the "load more" button will load up a page specific to those sub environments with a unique url

Dynamic environment folders have their own page (e.g. …/environments/<folder>) if they have more than 10 dynamic environments inside them. On that page, the dynamic environments list for that folder has a pagination control (1, 2, 3, …).

@ayufan can we get someone from backend assigned to this please?

Can we write an API call that will return paginated root folder? I have concerns that there can be some ~Performance penalty when we will have hypothetical 1000 of environments.

@filipa to visualise whats discussed :)

normal env list with pagination and show all button:

Within a review app folder specific page (after you have clicked on "show all button")

clicking on environments would let you go back
note that there should be 20 on a page, not 3 like in the mockup
"review/" is gone, as we are now inside the folder

perhaps we could even do a dropdown with search like with projects:

Thank you @dimitrieh it look amazing!

@ayufan @grzesiek do we have any news regarding performance in this issue? :)

@filipa We might need to do some prototyping to measure performance impact using a prototyped code. I don't think anyone have done it yet.

I believe this won't be ready for 8.16

mentioned in issue #26259 (closed)

changed milestone to %8.17

Since we will be focusing on adding more information to this view we should improve the pagination as a first thing in the release.

@yorickpeterse To make you aware that we are working on fixing this.

Notes from call with @grzesiek, @filipa and @dimitrieh

Proposal for Frontend

In the first iteration, we will focus on having a list of environments and folders displayed on this page: https://gitlab.com/gitlab-org/gitlab-ce/uploads/ff023423bd39060cfb82d1504abe6b4e/pagination.png.
The review folder will not be expandable, but only clickable. It will move us to https://gitlab.com/gitlab-org/gitlab-ce/uploads/4914c5fcae10d8417f9efffd15a71794/image.png.
In the second iteration, because we did https://gitlab.com/gitlab-org/gitlab-ce/uploads/4914c5fcae10d8417f9efffd15a71794/image.png. This will allow us to later purely in Frontend to implement expandable folders that will be dynamically loaded when expanded because we will already have all API needed to do that improvement. This is a small regression to what we have now: we have expandable folders, but I don't think that this is the right thing to do that in the first iteration.

Proposal for API

We discussed what is needed to make the https://gitlab.com/gitlab-org/gitlab-ce/uploads/ff023423bd39060cfb82d1504abe6b4e/pagination.png and to happen: https://gitlab.com/gitlab-org/gitlab-ce/uploads/4914c5fcae10d8417f9efffd15a71794/image.png.

It seems that this is viable and doesn't impose performance impact.

The idea is to have a paginated API request for returning a "folders" list of environments. By using below query:

p.environments.available.group('COALESCE(environment_type, name)').order('name ASC').select('COALESCE(environment_type, name)', 'count(*)', 'MAX(id)')

We effectively receive a list of folders with information about the name of the folder (or name of the environment if this is not in a folder), a number of environments of the same type and the ID of the newest created one.

This would allow us to return this data in API calls. There are two examples:

1. Return a list of environments, with indication that some of them do have siblings:

list_of_environments = p.environments.available.group('COALESCE(environment_type, name)').order('name ASC').select('MAX(id)').page(params[:page])

p.environments.where(id: list_of_environments)

class Environment
	def siblings_count
		project.environments.where(environmnet_type: environment_type).count if environment_type
	end
end

{
	environments: [
		{ name: 'production', environment_type: nil, siblings_count: nil },
		{ name: 'review/my-review-app', environment_type: nil, siblings_count: 100 }
	]
}

Based on this data. The entry review/my-review-app could be presented as a folder. However, this makes it a little inconsistent as we mix data in the API result.

2. Return a list of items with indication what needs to be shown: folder or not.

list_of_environments = p.environments.available.group('COALESCE(environment_type, name)').order('name ASC').select('COALESCE(environment_type, name)', 'count(*)', 'MAX(id)').page(params[:page])

{
	items: [
		{ name: 'production', environments: 1, first_environment: { 
                     name: 'production, environment_type: nil } },
		{ name: 'review', environments: 100, first_environment: { 
                     name: 'review/my-review-app, environment_type: 'review' } }
	]
}

This is more consistent, as we return a name of the item and we evaluate whether we show folder or the first_item based on number returned by environments.

Final note for API

For me it seems that solution 2. is the better, but I'm leaving that up to @grzesiek and @filipa to decide.

Extra

Our API should also include information about a number of available and stopped environments:

{
   count: { available: 100, stopped: 5000 }
}

Paginate environments list

Description

Proposal

Links / references

Designs

Child items ...

Activity

Proposal for Frontend

Proposal for API

1. Return a list of environments, with indication that some of them do have siblings:

2. Return a list of items with indication what needs to be shown: folder or not.

Final note for API

Extra

Other discussed solutions

Admin message

Admin message

Paginate environments list

Description

Proposal

Links / references

Activity

Proposal for Frontend

Proposal for API

1. Return a list of environments, with indication that some of them do have siblings:

2. Return a list of items with indication what needs to be shown: folder or not.

Final note for API

Extra

Other discussed solutions