Sub groups

This seems like a good reason to reintroduce teams.

GitHub handles this nicely in organizations by providing a default "owners" team with complete access to the organization and it's repos, and allowing you to create other teams with users who may not be part of the organization to give very specific access to specific repos.

We could expand on that by allowing other groups to add teams from other organizations to their own repos access lists.

@dzaporozhets made the following proposal to solve this for Android repo's that was also in the feature request tracker, I like it:

The Android repo is basically hundred of GIT repos stored in subdirectories same way like it stored in https://android.googlesource.com/. Repo tool for Android source code development is actually 700 lines of python script which makes a wrapper around some git commands allowing you to push multiple repos with one command. If at certain point Android core team will decide switch to GitLab they just will update default remotes in cloned repos. Having nested groups can simplify their life only in way of keeping old paths like device/htc/passion-common.git instead of changing it to 1 level structure device/htc-passion-common.git.

Looking at our architecture of GitLab I think bringing sub-groups will cause a mess in application. It like creating users inside of users in Facebook. However we can consider bringing Directory entity in future:

Directory 
has_many :projects 
belongs_to :group

Group 
has_many :projects 
has_many :directories

Project 
belongs_to :group 
belongs_to :directory

Group/User 
* dir 1 
- project 1 
* dir 2 
- project 2 
- project 3

:group/:project.git 
:group/:dir/:project.git 
:group/:dir/:dir/:dir/:project.git

In this case we can have Directory entity exists between Group and Project just for organising projects. Both Group and Project can't be manipulated with nesting. Group/User is always top level directory and must exist. Project is always bottom level entity and cannot contain any resources inside.

Full path of project will be build using group path, all directories between (if any) and project path.

if project.directory 
project.group.path + project.directory.path + project.path 
else 
project.group.path + project.path

I really like @dzaporozhets's proposal. But how would we handle permissions and group-level features?

I think we should, at the very least, allow the same kind of permission / membership management as we do now by groups, otherwise there is little to gain. That does bring UX challenges with it (membership window).

@sytses thoughts?

@JobV Can't we just do without those things? If we just fix Android we solved 80% of the problem right?

@sytses I think we solved 50% of the problem without those features. The other half is additional granularity in assigning permissions, which is the main reason why some customers requested this.

Without being able to assign permissions to directories, you will be forced to step away from the directory model the moment you want to lock down part of your (formerly SVN) collection of repos. I'm sure with the introduction of directories without permissions, we'll get that request the next week.

They would be better off with 50% (vs 0), so I'm OK with doing this in iterations. It should be kept in mind that we want to expand the feature in the future though.

I'm trying to find a space in a milestone for the proposal by DZ.

I agree with @sytses lets go simple things. We can always increase complexity later

Something to keep in mind: some of our routing is handled by NGINX now: 'git clone' over http gets sent to gitlab-git-http-server. If we allow more variation in project paths (gitlab.example.com/foo/bar/nested/wheeee/project) then this may (or may not) cause trouble in the NGINX config.

I dislike the 'directory' proposal, for the same reasons that I objected to it in the original feature request;

Within our company, we have multiple "groups," representing different divisions of the company. Within that division, we have multiple 'teams' of people. Each team may be working on one or more 'products'. Within each 'product,' there may be one or more 'projects' (repositories).

I would like to structure our GitLab hierarchy to match this corporate structure, with the ability for each 'level' of the hierarchy to optionally inherit the members and permissions from the parent, or to manually override permissions and members at any given level.

While I appreciate that a "directory"-based solution quickly solves a small percentage of the overall desired solution, and complexity can be added later; I don't believe a 'directory' accurately represents the type of organization being requested. Nothing seems to really capture the (desired) requirements quite as well as "nested groups." "Team" abstractions come pretty close, though.

I'll use the Redmine issue tracker as a rough analog. Redmine is divided up into multiple "Projects." Each project represents a collection of users (with customizable permissions), and 'issues'. Optionally, a Redmine "Project" can specify a "Parent Project," from which it can inherit (and optionally override) its list of users and permissions. Thus, a simple hierarchical pattern is possible.

I don't see much difference in this analogy, between Redmine "Projects" and GitLab "Groups." What are the technical challenges of adding a new (optional) "Parent Group" property to the existing "Group" structure? If a Parent Group is specified (and the group is configured to inherit permissions), then recursively include the users and permissions from the parent group(s) with those defined in the current group. If the group has additional users and permissions specified, those will 'add or override' the inherited ones.

When constructing URLs; if the the group has a Parent Group specified, then recursively use the parent group(s) to create the URL path. A small check for name collisions would need to be made, to prevent a 'sub-group' and 'project' from resulting in the same/conflicting URL.

In the UX; any users and permissions that are inherited from a parent group are visually indicated as such by being 'grayed-out' in the permissions view for that group. Perhaps a tooltip could further indicate "Inherited from 'XXXXX'."

I see there's been no new discussion since my last post, a few weeks ago. I hope I didn't kill the conversation...

I consider myself a skilled developer, but sadly RoR is completely outside my area of expertise. That said, I started looking at Redmine source code. In my previous post, when I associated the proposed "Nested Group" concept to Redmine's "Nested Projects," I failed to point out that Redmine is also implemented in Ruby on Rails. By conceptually mapping Redmine "Projects" directly onto GitLab "Groups," we could feasibly copy a good percentage of the implementation details directly from the Redmine source code, as a starting point.

I've provided a few relevant links below, from the Redmine source code:

I think just bringing back teams would make it much simpler.

Projects are repos+stuff, with users [and teams] having project level permissions
Groups are groups of projects, with users [and teams] having group level permissions
Teams are groups of users, with users as members having team permissions (owner,management,member), that can be assigned in membership of groups and projects.

While true nested (groups of projects) would potentially be very capable, I don't think it's as necessary if you had the above.

I agree; "Teams" sounds much closer to the desired functionality than "Directories," as I understand them.

I know in my environment, I would have immediate use for teams of users to collect together people for grants to large numbers of projects+groups - for stuff like "Give this list of users ability to review all of these (X) projects, independent of their membership in the groups of those projects.

With a "Teams"-based solution, it sounds like we lose the ability to 'nest' repositories' URLs. Is that an accurate statement?

With a "Directory"-based solution, we get the conveniently-nested URLs, but we lose the customization of permissions.

I wonder if there's any room to implement both approaches - using Directories for nesting and Teams for flexible permissions. But at that point, it probably makes more sense to aim for the "True Nested Groups" approach.

As the original description indicates; it's definitely a very nuanced issue... personally, I think there's great value in aiming for "True Nested Groups", to get the best of both worlds.

Of the (very) large customers asking for this functionality, have we determined which of the 3 approaches so far best match their needs?

@JobV @dzaporozhets Can we continue this discussion, please? I like the idea of @dzaporozhets original directory approach. I also agree with @JobV that it gets us 50% of the way there. However, based on the later parts of this discussion it sounds like me may want to consider how we can apply permissions to this model ahead of time. Even if we don't implement the permissions portion now and least we can be relatively sure we can apply a good model later on. I hate to see us implement directories only to realize we can't add the permission model we need later.

As another approach, does it give us what we need if we allow groups to belong to another group? This gives us built-in 'directory' solution and also has a user container that we can cascade. For example, based on @dzaporozhets original approach:

Group 
has_many :projects 
has_many :groups
belongs_to :group

Project 
belongs_to :group 

Group/User 
* Group 1 
- project 1 
* Group 2 
- project 2 
- project 3

:group/:project.git 
:group/:group/:project.git 
:group/:group/:group/:project.git

The only thing above that I don't know is what happens when you create a circular relationship like where a Group has many Group and also potentially belongs to a Group.

@dblessing You've just perfectly described the proposed "True Nested Groups" approach, which I've advocated for since the issue first arrived. It solves 100% of the desired functionality, and just seems to 'make sense,' conceptually. It feels like the right choice, barring any obscene technical limitations which would prevent its implementation.

As a bonus, it mirrors a similar concept used in Redmine (an open-source, RoR-based issue-tracking system). A large chunk of the functionality could be copied directly from the Redmine source code implementation of "Parent Projects", and adapted for use with GitLab "Groups."

Milestone changed to 9.0

Set milestone to 9.0, which has no due date.

Implementing this in any way will be a very major change, reason for a major release.

@dblessing your group nesting proposal also makes a lot of sense.

I think we should go with group nesting proposal. This task is hard anyway so better get more advantages from it.

Now small list of problems with this task:

Routing

How detect gitlab-org/gitlab-ce is project in group and not group in group? How to properly route all this with Rails?

Also we now should bring extra validations so I can not create a group inside group with name of project that is already in this group

Backend

There are many places where namespace/project used. All this should be fixed to prevent group, project = path_with_namespace.split("/") because its no longer a case.

UI

Every place where we render group/project now can easily overflow because of group/group/group/project. We can just ignore it or adopt for something reasonable like group/group/project.

Mentions

We use @group mention for members. Not sure if we want to support @group/group/group mention.

Permissions

Now we need collect all user permissions from bottom to top to detect what access level should this person get

Its what came to my mind if 5 minutes. I am sure developer who will do this find x10 more problems. But looks like people really want this feature so lets do it. For 9.0 probably

"How detect gitlab-org/gitlab-ce is project in group and not group in group"

When you start looking at that parsing/disambiguation logic, it might also be worth considering if there would be any way to deal with the conflict between userid "fred" and group "fred" in the same pathing. I realize this isn't directly related, but as it stands now if you have a central directory of users and set an expectation of what the per-user namespace will look like, ANY other user on the system can preempt that by creating a group of the same name.

@dzaporozhets I like your ideas and points - lots to consider. I look forward to us exploring this for 9.0

@dzaporozhets RE Routing/backend: we can have a DB table with every full user, group and project path, with a target_type of User, Group or Project and a target_id, and use this for both routing and validation. Records would exist for both supergroup, supergroup/group and supergroup/group/project.

Since the project-part of the URL can now have more than 2 segments, we don't know if the 3rd segment is a project name within a group's subgroup or a controller route in scope of a simple group's project, like gitlab.com/company/product/issues, which could either be referring to a project issues within group company/project, or to all issues of project company/project. We would need to do a lookup for the path in the paths table that returns the records with the longest match of the full path company/product/issues, so it returns the correct record for either project company/product/issues or company/product.

RE Mentions: I think we definitely do want that. We're using @developers at dev.gitlab.org, but that doesn't make sense on gitlab.com since then we'd own the top-level developers group.

RE Routing/backend: we can have a DB table with every full user, group and project path, with a target_type of User, Group or Project and a target_id, and use this for both routing and validation. Records would exist for both supergroup, supergroup/group and supergroup/group/project.

interesting concept. worth investigation

mentioned in merge request !1771 (closed)

mentioned in issue #1256 (closed)

Disclaimer: I'm a fairly new GitLab user stumbling in from Google who has never read the source.

I get the sense that to do recursively nested groups cleanly, the path needs to be decoupled from the group concept. Effectively the path segments become pure directories, where each segment is associated with many groups and users. Permission checks start at the rightmost segment, recursively checking the segment's associated groups/users, then move to the left and repeat as needed. I prefer this recursive approach to both paths and groups over imposing any particular organization's structure in order to provide ultimate flexibility, and it feels simpler to implement to boot (see disclaimer). It eliminates routing ambiguity, and prevents group nesting changes from overflowing urls and breaking bookmarks.

Added ~122770 label

Added ~123454 label

Added feature proposal label

mentioned in issue #5940 (closed)

I think the easiest way to handle routing and any other ambiguity would be to prefix users, groups and projects. For example a 'g' for groups, 'p' for projects and 'u' for users. Then g_gitlab-org/p_gitlab-ce would be a project without any doubt. Downside is that it looks a bit ugly in the URL

mentioned in issue #10322 (closed)

A customer (https://gitlab.zendesk.com/agent/tickets/15082) mentions that having 'teams' with LDAP sync like we do in EE groups would resolve their issue. They have a different group/project structure and essentially need group sync at the project level. However, if it is at a team level they can easily add teams to the project and resolve this.

We are migrating thousands of repos from a filesystem-based git store into a new gitlab instance. Most of these projects won't care about the old path vs. the new one. But projects that use submodules often do. For example, some of the projects have submodules named "../vendor/kernel/embedded-linux-3.18.5-23.git" and so on. "vendor" and "kernel" are not variables; they're actual path elements. And some of the projects even have two levels of un-nesting with submodules named "../../other-project/libs/foo-lib.git", for example.

This relative path feature of submodules served us well as we moved from host to host and in different replicated environments. But now it seems like a curse since our new target, GitLab, does not allow it.

I hope you can keep this kind of raw path need in mind as you move forward.

2 more cents: the idea that paths==groups or some other special thing seems very unusual to me. Relying on it as a convention is one thing but requiring it to have special meaning seems a bit over the top.

From what I've read of this thread, it sounds like there are two good ideas that could be supported. The idea of nested groups has merit, that way a parent group will have permissions propagate down to child groups, but users can possibly only be assigned to a child group and don't have permissions in the parent group.

The idea of teams is also a good thought, but I don't think they should be tied to the nested groups. The ability to create a team of users that could be assigned to groups the way individual users are assigned to groups would definitely reduce administration overhead.

That's exactly what I'm thinking too!

Thirded! "Nested groups" is my primary desire; "Teams" functionality seems to be an entirely different feature, but one which is not mutually exclusive to the desires expressed in this thread!

It seems like naming conflicts can be avoided with three simple rules:

When creating a 'subgroup'; do not allow it to have a name which is already taken by any other group or project with the same 'parent' group.
When creating a 'project'; do not allow it to have a name which is already taken by any other group or project with the same parent.
When moving a 'subgroup' to a different 'parent group'; do not allow the move, if the new parent contains a project or subgroup with the same name.

mentioned in issue #14193 (moved)

to teams here. The subgroups, on the other hand is not as important.

A Teams concept is a big gap in Gitlab for our organization. Right now I create "project groups" and "user groups", and share the project groups with user groups. User groups are essentially teams in my world, representing an internal department/team.

The fact that I can't share a group with another group causes major headaches when needing to share all of a group's projects (which can be over 50 for some groups). I have to do the same tedious process 50+ times. Or remembering to add all pertinent groups when creating a new project in an existing group.

Also it leads to frequent confusion when training people why we have the different groups.

Example use-case to consider: https://twitter.com/Det_Conan_Kudo/status/716454915992891393

This seems like the best hierarchy to me:

Group
- Team
  - Project
  - Project
- Team
  - Project
- Team
  - Project

+1 to doing this with Rails, imagine the size of a Git repo with hundreds of repos in it D: Even with submodules that'd be a complete mess.

To expand on @connorshea's message, we at Unity Linux made the move from a GitHub monorepo to GitLab with per package repos. We did this to make it easier to connect GitLab to systems like Koji for building the Linux distribution.

However, due to the lack of nested groups, Unity Linux has to have six groups that don't necessarily appear related to each other:

Ideally, I'd like to have the following structure be possible:

Organization Group (Unity Linux)
- Category Group (Packages, Software)
  - Section Group (Core, CLI base, etc.)
    - Package/Software repositories

However, I'd settle for just being able to have the existing groups inside of a Unity Linux organization group if multiple layers of nested groups isn't possible.

It is key to note that being able to assign arbitrary users or groups of users to any of these levels and having it inherit from parent levels is important, as well as the ability to override the permissions of a user inherited from parent levels.

Also, slightly unrelated but I figure I should mention it, when implementing this, make it so that the dashboard can collapse groups of projects in the "Your Projects" view. I already go insane from having 300+ repositories in that page, making it hard to find anything I'm looking for there.

+1 for both nested groups AND teams. Here's an example:

I have clients, and those clients each have several different projects. I also have my infrastructure, tools and other source. I would like to do something like the following:

clients:
- client 1
  - project 1
  - project 2
- client 2
  - project
Tools:
- project 1
- project 2
Infrastructure:
- group 1
  - project 1
  - project 2

....

I'm sure that I'm not alone in wanting to create similarly organized nested groups. I do not see how teams alone can solve this particular use case.

Thanks!

May be I am getting confused by the example @connorshea provided, but for me teams in the sense of gitlab (grouping users) and nested groups are two independent concepts. From my point of view groups are purely used to group projects and put them into a hierarchy. Together with the ability to share project with other groups it's possible put the same project into multiple location similar the way symbolic links can be used on the linux file systems. Teams on the other side are used purely for grouping users and teams (nested teams) and can be assigned as member to groups just the same way like it is already done for individual users. Coming back @connorshea example, there is nothing wrong with that hierarchy, nothing will prevent one from having a nested group of each software team, but I like to also have the ability to group users independent from the project hierarchy (gitlab teams).

@sbrady, +1 for your suggestion, sounds good. Especially if you have several departments which want to have control over all of their projects, while you maintain control over everything on the server.

@heiko_boettger I'm thinking something like GitHub's Orgs and Teams features.

Jekyll is an interesting example (not publicly accessible, so a screenshot will have to do):

Using GitLab as an example: Essentially, the idea is that you'd have a Group (GitLab), which would have a bunch of Members (all our employees), a bunch of Projects/repos (GitLab CE, GitLab EE, GitLab Development Kit, GitLab Runner, www.gitlab.com, etc.), and a bunch of Teams.

The Teams would be Backend Engineering, Frontend Engineering, Design, Service Engineering, Marketing, Sales, etc. Each team would have access to a given set of repositories, e.g. Frontend Engineering, Backend Engineering, and Service Engineering would all have permissions to commit to non-Protected branches in the GitLab CE project. Teams wouldn't "own" repositories, which I should have been more clear about in my last comment, as each team would have access to a set of repositories. And of course members of the organization could be members of multiple teams.

I would also imagine you'd be able to @mention teams, e.g. @gitlab/performance for issues relating to performance. Teams could be used for a number of different things. Jekyll and other open source projects have a handful of Teams which anyone can request to join and be notified of relevant issues/discussions/PRs. So people interested in using Jekyll on Linux could join a Linux Team in the Jekyll organization, same for OS X users, etc.

Hopefully my wall of text hasn't confused you more :)

The biggest problem we have to solve with this feature is the competing use-cases of CE, EE, and GitLab.com. CE is generally for self-hosted open source projects or personal projects, EE is for larger companies (per about.gitlab.com, IBM, NASA, SpaceX, RedHat, etc.), and GitLab.com is for hosting open source projects as well as private projects that users don't want to have to use self-hosted CE for.

As such, Groups and/or Nested Groups and/or Teams needs to fit all three of these main use cases simultaneously, which is an interesting problem. It needs to work with open source projects, it needs to work with a company of 5-25 people, and it needs to work with a company with 5000 people.

Sidenote: I may have gotten the exact goals of GitLab CE/EE/.com wrong, feel free to correct me if I'm misunderstanding anything.

@sbrady see above, I think my suggestion would work for your use case?

Depending on if you're using GitLab CE/EE/.com, the way you're hosting this "Group" might differ, but essentially you'd have a Group called Consulting Company with the following teams and projects:

Teams:

Client 1 employees (e.g. a manager and developer from Client 1 who are added to the GitLab instance so they can see the relevant project(s) being worked on)
Client 2 employees (same as above)
Consulting Company Employees (Every employee at Consulting Company)
Consulting Company Team 1 (Team of X people that work together on a given client project)
Consulting Company Team 2
Consulting Company Team 3

Projects:

Client 1 Project A (Accessible by Consulting Company Team 1, Client 1 Employees)
Client 1 Project B (Accessible by Consulting Company Team 2, Client 1 Employees)
Client 2 Project A (Accessible by Consulting Company Team 3, Client 2 Employees)
Internal Tool Project A (Accessible by Consulting Company Employees)
Internal Tool Project B (Accessible by Consulting Company Employees)

My earlier post put this in a hierarchy, but that isn't really correct, since members of the Group can be added to multiple Teams and Projects can be accessible to multiple Teams. It'd be better described as a venn diagram.

Do you think that'd work for you?

Actually no, as my use case is for me alone and has nothing to do with user access - in fact my scenario assumes that I am the only user on the instance.

I use a nested structure like this to create "classes of service" for different services, different clients, etc. I have some clients where I have multiple projects installed (actual example: one client I have a web server, the associated site, several other web apps, a phone system and a business logic application, while another is just a phone system on a VM) and I need to keep track of the base templates as well as client customizations.

Yes it's a mess, but it's my mess, as I must give the guy with the VISA card what he wants :)

We don't need to solve the Android repo use case anymore. They'll move to submodules.

@connorshea I like the idea to have something similar to the GitHub Teams and Orgs, but I also want to have groups containing groups and may be teams containing teams. Teams containing teams can be easily solved by ldap and I am not sure if team is the right term in that case. Thinking about ldap and talking about different types of customers, I think teams are only needed for customers that don't have a 1:1 relation between ldap groups and teams or don't have a ldap server for that at all. In our company it's possible to have a ldap group for each team. As a result in my case the ability to add ldap groups to groups (which is already implemented) and projects (which is currently missing) would give me all the flexibility I need regarding access management.

To give summary I need the following features:

allow groups to contain groups
inherit access rights on groups it is currently done for projects
allow ldap groups synchronization as it's already done for groups on projects

@heiko_boettger I see a few potential problems with that. If you reference an issue from one of those repos, you'd end up with something like alphabet/google/android/camera#37. And then the header would have "Alphabet / Google / Android / Camera • Issues", or something ridiculously nested like that.

I think it'd be more complex and easy to break than it's worth. If you already have a GitLab EE instance, you already have your main organization, e.g. "Alphabet", or more realistically, "Google", as the "root" of every project.

I'm not particularly intimately familiar with LDAP, so someone else should be able to better respond to your specific use-case if I'm misunderstanding things.

We've been watching the nested-groups discussion crawl forward and the slow pace is very frustrating to us. We really need a solution which allows at-least 3 extra nested groups between today's existing group and existing project.

We have automated jobs which create both groups and projects automatically via GitLab's API. Groups of projects are measured in double-digit gigabytes for sizes and triple-digit numbers for number of repositories we create automatically. Not being able to further organize our groups of (thousands of) repositories is a huge problem for us.

What we'd like to see ASAP, similar to Heiko:

Allow groups to contain groups and projects
Inherit access rights on groups it is currently done for projects

http://gitlabserver.com/Group/Level1/Level2/Level3/Project.git

http://gitlabserver.com/Group/Level1/Level2/Project.git

http://gitlabserver.com/Group/Level1/Project.git

http://gitlabserver.com/Group/Project.git

http://gitlabserver.com/Project.git

For each level in the hierarchy, the permissions should apply downwards (inherit), just like today. For instance, if you have access to the group, you have access to everything below the group. If you have access to "Level1" you have access to everything contained within "Level1", both projects and groups. If you have access to "Level2" you have access to everything contained within "Level2", both projects and groups. And so on.

You can reply to me privately if you want to know further details.

Thank you.

@connorshea My opinion on long paths is, that this problem should be well known by everybody. You have the same problem when working with directories and URLs on webpages but no one . One solution for that might be providing an additional URI that just uses the project-id, a hash or guid. Thinking about what is possible with markups in the wiki and comments, it should be possible to create a reference which is translated from the identifier to the fullpath, project name, parent group name or whatever. Since guids and ids without additional text themself are not very userfriendly there might be a URL just using the project name without the path as a prefix followed by the project-id.

At the end nobody needs to be forces to create nested groups. As for your example, there is already the need for android/camera, imagine you have not only one camera chips but many. In that case you even might need:

android/camera/chipset/documentation.git
android/camera/chipset/testtool.git
android/camera/chipset/driver.git
android/display/chipset/documentation.git
android/display/chipset/testtool.git
android/display/chipset/driver.git

Well you could say that the company needs to setup a separate gitlab server for each android devices, but that doesn't make it better. Putting more things into the same project is also not possible since access rights cannot be restricted within a project.

My suggestion is to check whether these potential problems are realistic. Don't get me wrong, but shouldn't the customers be able to decide by themself when a path is to long?

cc: @JobV @DouweM @sytses any further thoughts on this? Seems quite a few organizations need it.

@dzaporozhets You've shared your thoughts here before, can you again based on the most recent discussion?

@DouweM sure.

Nesting should be done with Group entity. So Group.has_many :groups and Group.belongs_to :groups. I see some people want deep nesting without need to create Group for it but I am against it. Every / should represent directory on FS. So like directories - you can not create file inside something before creating directory.
Discussion about GitHub like teams is not relevant to nesting and should be moved to separate thread. FYI I am against introducing Teams.

@dzaporozhets What does a Group.belongs_to :groups mean? Multiple parents? +1 for discussing teams separately.

@heiko_boettger it means group can have many groups inside. But group can have only 1 parent group. Basically directory behaviour.

Nesting should be done with Group entity. So Group.has_many :groups and Group.belongs_to :groups. I see some people want deep nesting without need to create Group for it but I am against it. Every / should represent directory on FS. So like directories - you can not create file inside something before creating directory.

Unlike directories, though, this would require that I create a new group for each directory and then assign group ownership to that directory, and presumably group belongs_to-ship to the new group.

Consider traditional directories:

mkdir foo/subdir

Contrast with strict per-group directories:

addgroup foo-subdir --parent=foo
mkdir foo/subdir
chgroup foo-subdir foo/subdir

Now think how you would have to handle mkdir -p:

mkdir -p some/long/path/whose/history/you/neednt/concern/yourself/with/

Even worse than the complexity is the unnecessary explosion of groups needed to support arbitrary path elements.

I suppose one way to work around the problem is to add some rewriterule outside of gitlab which translates some other character to a forward-slash. That this horrendous kluge would be a welcome "feature" for many users should give you pause.

A suggestion:

One way to prevent difficulty for testing this area would be the arbitrarily limit the number of levels which is supported. So the test-cases for GitLab would test, for instance, 6 levels deep. That much could be supported for EE paying customers (like me) and would allow placing multiple copies of the entire Android source tree (all 400+ repositories multiple times) into GitLab.

Beyond the "supported levels" is a policy decision for GitLab to make whether additional levels are "allowed".

Thank you.

We thought about this some more:

We don’t think we require fully “nested groups”. We need the ability to have a project housed within a directory-tree under the same namespace (group). This would seem to be much simpler to implement than fully-nested groups.

We think if GitLab implemented support for pathnames (including multiple levels of slashes) preceding the project name, but after the group-name, that would be sufficient for our need. The mock-up of the “New Project” UI is shown below for what I’m talking about.

Access-checking would only be done at the Group and Project level, same as today.

Obviously there’s a bunch of error-checking which needs to take place before a project is created. For instance, creating a new project which would nest inside an existing project's pathname would need to be disallowed. But this is a reasonable set of rules to avoid conflicts on the server.

And more error-checking if a project is moved / renamed. And the GitLab API would need to be augmented to accept a “pathname” component when creating / renaming a project. And there are probably a vast number of thorny issues I'm overlooking.

But I'm guessing this would still be much simpler to implement than full-blown nested-groups of infinite depth.

I think full blown nested groups will be needed at the end of the day for a lot of people. I do completely agree with an arbitrary limit on the number of nested groups.

@JobV @DouweM can you add the "customer" label?

Added ~113220 label

mentioned in issue #15234 (moved)

Something we are 100% sure and we are not going to change

Project and group path in URL is equal to path on filesystem of repository. If project URL is foo/bar/project then FS path will be foo/bar/project.git. So if we are going to implement any kind of nesting for each / it will be a real directory. Second is we don't want / in project path (db field that represents project directory name) as it breaks every single place where we work with project path. So in GitLab database each directory in filesystem will have own database record (group, project or something else). That means for foo/bar/project we have next structure:

foo - group or user path
bar - ???
project - project path

Something we actually want to discuss

So we have project path foo/bar/project and we still thinking about what entity will be bar. And here are 2 choices: either it will be a group or some new entity that has representation on file system. And its important decision to make before we implement anything because if we decide that bar should be group - we can implement membership and other features for nested group later. If it will be something lightweight - that means no permissions.

Second thing is limitation in nesting we want to support. I think it should be lowest possible value because we don't want redesign every single place in UI where project full path is rendered. So I think about 1 or 2 subdirs between group dir and project dir.

About feature in general

Implementing nesting inside GitLab is quite complex task that require big amount of changes (and bugs as result). We are not going implement it without clear benefit or with hurt to application architecture. That's why I think about it more just just ability to avoid prefixes in project names by putting it into subdir. And that is the reason why I advice community and core team to consider use of Groups for nesting. That can serve as real benefit for one that wants not only subdir but hierarchy for projects and groups. And it will also prevent people from doubtful using of feature when user just put project under foo/bar/much/deep/nesting/project without reason. Fact that each directory in project URL will represent group should make people use only necessary amount of nesting.

P.S. I don't like this feature in general as its against my vision of application simplicity. So we either do it right or not do it at all.

I would avoid adding another element besides groups and projects and thus foo/bar/project would be two groups (foo and bar) and the project.

This has two main advantages:

There is no new building block that needs to be specified and maintained
Every current and future group feature is available on every level (permissions, #4037 (moved), ...)

mentioned in issue #18017 (closed)

Any update? Are you working on it?

@Baertierchen Since my company is still interested in a solution, I have to vote for your proposal, because it is directly what I like to see. +1

I would also agree with Steffan that it should be done through the use of nested groups. There's no point over-complicating the change by introducing a new entity type.

@dzaporozhets In terms of your limitation question: For myself I could easily make do with adding a single sub group level, though two would certainly afford me more flexibility in my structure. I would personally suggest going with two.

As long as people don't go crazy with the length of their group paths then that shouldn't be a problem. When I was briefly scrolling through the conversation I struggled to find anyone who wanted more than two additional directories, though I wouldn't be surprised if there's individuals who would want even more.

Even if there is any disagreement about the number of levels to introduce, I don't feel that's a reason to not proceed with the number that you feel comfortable proceeding with. At the end of the day, one or two extra levels is far better than none.

@Baertierchen yep probably re-using groups is best way to do it

@Rjs37 makes sense. So if we start doing this feature I think we can start with 1 sub group level since it might fit in current UI without much modifications.

@dzaporozhets I think you will see a lot of demand for a second sub group level, but as long as it's done in a way that would allow for further expansion (if there's enough demand / there's time to make the UI changes), then I still think it's worth proceeding.

The only quick UI change I'd probably suggest is scaling back the font size on the project list dropdown, that would be an easy win in my eyes.

Even with the Unity Linux use case provided by @Conan_Kudo (one of the more complex structures that I saw in this thread) he admitted he'd be able to make do with a single extra level if more weren't possible.

@dzaporozhets hey, just curious, how come you are against introducing teams as a way of managing "groups" of user's access to projects? :)

@lxsndl we had Team and Groups in version 5 and it was a mess with a lot of intersecting functionality . So based on past experience

Just realized I mucked up my initial tagging of my last response, edited it but guessing you'll have seen it now @dzaporozhets

I think teams could still be useful as long as it's only used in the same way as users i.e. adding a team of users to a group/project rather than the github way. But yeah that's for a separate discussion, not really relevant to how this discussion has progressed.

@lxsndl and @dzaporozhets Aren't teams a totally different feature? From my point of view managing group member (teams) and organizing projects into hierarchy (subgroups) are two different issues.

Yes, it is kind of unrelated .. that's why I opened another issue (#15234 (moved)) for teams ;)

Hey guys, it's been 9 months since this was first proposed.

How's it coming along?

mentioned in issue #17801 (closed)

Would really love to have nested groups for better organizational structure of my teams!

mentioned in issue #19477 (closed)

Hi all, we are also very interested on this feature. Is there any planed release date for it? thanks!

Added ~423634 label

mentioned in issue #19609 (closed)

Yes please, I just registered here to say that this feature is really important to me!

In my company we have multiple teams (software, electronics, mechanical, control, etc.) working together on a few projects. Each project is broken down into a number of systems, each system may have multiple sub-systems, and eventually each sub-system is built from multiple git repositories. We use git submodules a lot to track component versions independently and glue things together.

With GitLab we are forced to flatten this hierarchy and lose all the information embedded in it, or alternatively run hundreds of instances of GitLab in parallel, which is a pain to manage. We used to use a single gitolite instance (with a lot of duck-tape around it) and never had any issue with our deep hierarchy. I'm waiting for this feature to make the move to GitLab.

Please consider this feature in the next release!

This is the ONLY feature preventing us from migrating from Redmine to GitLib. Help!

@JobV This is how Deveo does nested groups for the Android project http://blog.deveo.com/how-to-create-local-android-open-source-project-mirror/

I think we can look at the Kallithea project (https://kallithea-scm.org/). They have implemented nested group. They have also add a unique ID to each repo and using this ID in the URL, it's possible to reference a repo regardless of which group the repo belongs to. With this approach, we don't have to worry about breaking references and cloning URLs when moving repos between different groups.

I think there should be two URLs referencing each repo: The current owner-slash-repo schema plus a new GUID based (hence owner-less) schema like "http://gitlab.com/repos/guid". While the first one apparently changes with renames, user changes or with group changes (like my-group/team-a/repo-one -> my-group/team-b/repo-one), the latter stays the same. So everybody can decide what the right URL forhis personal se case is.

Mentioned in issue #22147 (moved)

Mentioned in issue #14908 (moved)

Added major wins label

@dzaporozhets will look into this

Yes my plan is to start working on it next month.

This will be a boon.

Mentioned in issue #23687 (closed)

Reassigned to @dzaporozhets

First step is to make routing play nicely with nesting. So both group/subgroup/subsubgroup/project, group/subgroup/project and group/projectworks

Mentioned in merge request gitlab-shell!102 (merged)

Mentioned in merge request !7148 (merged)

Will we be able to move existing projects in the current group/project structure into subgroups after this feature is implemented?

Mentioned in commit b7d3fc15

Mentioned in commit gitlab-shell@0f8a3cbb

Mentioned in merge request !7200 (merged)

Mentioned in commit 6cd508d4

Added ~19492 label

Mentioned in merge request !7121 (merged)

Mentioned in merge request gitlab-workhorse!80 (merged)

Will we be able to move existing projects in the current group/project structure into subgroups after this feature is implemented?

@eaidan yes. Same like now you can move project between groups - it will be just more options in dropdown.

One thing that hasn't been discussed yet as far as I can see is the Pages integration in Gitlab EE. This defaults to hostnames like 'group.pages....', which are served via wildcard DNS.

I can't think of any reason why it would be incompatible with nested groups, it's just something to be aware of!

Mentioned in merge request !7459 (merged)

Mentioned in commit a7a9db80

Marked the task Allow nested groups in URL (routing) of rails application https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7459 as completed

1 step is complete (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7459): Application routing supports / in group/project name.

Next step is data level support by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7121

mentioned in commit 05efd19e

marked the task Add nested groups on data level (db, migration, lookup etc). https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7121, as completed

mentioned in merge request !8008 (merged)

mentioned in merge request !8011 (merged)

mentioned in merge request !8013 (merged)

mentioned in commit abad9a9b

mentioned in issue #25750 (moved)

mentioned in merge request !8071 (merged)

marked the task Update db, validations to allow creating groups with same path and different parent etc https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7976, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8008, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8011 as completed

marked the task UI: General https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8062 as completed

mentioned in merge request !8182 (merged)

mentioned in commit f8262e85

mentioned in commit fc4491c8

mentioned in issue #25917 (closed)

@tauriedavis: Would you be able to help out with nested groups from a UX perspective? Looking at the description above, there are 4 'UI' related tasks remaining. It would be good to have one person be able to think through all the touch points of nested groups in the experience and make sure they all make sense.

added ~874211 label

mentioned in merge request !8308 (merged)

removed ~123454 label

mentioned in merge request !8448 (merged)

mentioned in issue #26407 (moved)

mentioned in merge request gitlab-workhorse!80 (merged)

Groups as considered by GitLab are a mix between repository organization, as they define the workspace for some repositories, and permission management. Having these two different concepts tied in the same feature reduces the scalability potential of GitLab. How would a company divided in dozens of teams, of hundreds of developers each, handle their projects divided as well in back and front end sections for example?

In my opinion, these two concepts need to be clearly split from eachother to allow better control and organization. This is my proposal:

Legend:

[AI]: already implemented.
[PI]: partially implemented.
[NI]: not implemented.

Definitions:

Item is defined as either a Repository or a Directory.
Repository is defined as a software packages.
Directories are defined as a group of Items.
Actors are defined as either Users or Teams.
Users are defined as accounts representing a real life humans.
Teams are defined as groups of Actors.
Roles are sets of allowed actions that can be realised by a User.
Role caps are high bounds for sets of allowed actions that can be realised by a Team.

Rules:

A User can belong to multiple Teams with a Role assigned. [PI]: Groups, which are not exactly the same as Teams, exist and the one-to-many relation is already implemented. Roles assignment is already implemented.
Roles for each Item can be assigned to Users. [PI]: Directories are not implemented.
When a Role is assigned to a Team instead of a User, it becomes a Role cap. A Role cap sets the maximum Role that member of the Team can have. All higher Roles will be considered to be the Role cap for this purpose. [NI]:
When a User is granted two different Roles over an Item, the higher one applies. [NI]: as Groups can't be assigned roles, there's no crash in actual implementation.
A Repository resides inside one and only one Directory. [PI]: the concept of Directory does not exist, but the one-to-one relation is implemented.
Every Actor has its own root Directory with the Owner Role/Role cap. [PI]: the root Directory is the only one implemented.
Non-root Directories can be created inside other Directories, inheriting their parent's assigned Roles and Role caps. [NI]: Directories are not implemented, so inheritance isn't either.

As a clarification of Role caps, If a User is a Master inside a Team, when this Team is granted Developer permissions inside an Item he will be granted this Developer permissions, while a Reporter in the same Team will only be given Reporter permissions.

This features allow finer control over the organization and permissions, using Directories to group different Repositories (as it has been asked in #14908 (moved)) and Teams to group different Users both in a nested way.

NOTE: the full path to a Repository could keep the owner Actor namespace in the first place, followed by the Directory path and the Repository name itself.

mentioned in merge request !8743 (merged)

mentioned in merge request !8779 (closed)

mentioned in merge request !8786 (merged)

mentioned in merge request !8827 (merged)

mentioned in merge request !8835

removed milestone

changed milestone to %9.0

@dzaporozhets I've just noticed that our robots.txt needs updating to account for nested groups:

https://gitlab.com/robots.txt

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/public/robots.txt

@nick.thomas thanks. I added it to the todo list in the issue description

mentioned in merge request !8845 (closed)

mentioned in merge request !8979 (merged)

marked the task Cleanup: rename find_with_namespace with find_by_full_path everywhere as completed

marked the task Workhorse support https://gitlab.com/gitlab-org/gitlab-workhorse/issues/74 as completed

marked the task Inherit membership and permission from parent group - https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8071 as completed

mentioned in merge request !9000 (merged)

mentioned in merge request !9034 (merged)

mentioned in merge request !9061 (closed)

mentioned in merge request !9062 (merged)

marked the task Limit nesting level !9000 (merged) as completed

marked the task Cache full_name and full_path !8979 (merged) as completed

mentioned in merge request !9074 (merged)

marked the task API support https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9034 as completed

marked the task Allow create nested group in UI https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8786 as completed

mentioned in merge request !9136 (closed)

marked the task Show parent groups on members page https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9062 as completed

mentioned in merge request !9184 (merged)

mentioned in issue #28125 (closed)

mentioned in merge request !9206 (closed)

marked the task Nested groups GFM support https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9184 as completed

mentioned in merge request !9227 (merged)

marked the task Project features like transfer, import should work for nested groups https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9227 as completed

marked the task Show nested groups on dashboard, dropdown autocomplete etc https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8827, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9227 as completed

marked the task Search support for nested groups https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8827 as completed

mentioned in issue inkscape/inkscape-web#12 (closed)

mentioned in issue #27692 (closed)

mentioned in merge request !9288 (merged)

mentioned in issue #28761 (closed)

mentioned in merge request !9553 (merged)

mentioned in issue #28786 (closed)

mentioned in issue #28787 (closed)

mentioned in issue #28788 (closed)

mentioned in issue #28811 (moved)

This feature can't be pushed into production until we find a way to not impact current infrastructure scaling efforts: https://gitlab.com/gitlab-org/gitlab-ce/issues/28125#note_24648827

marked the task It should not be possible to share a project with a group that's an ancestor of the group the project is in https://gitlab.com/gitlab-org/gitlab-ce/issues/28788 as completed

@pcarranza who is responsible for making it happen?

I'm pushing for unambiguous prefix as the way to go @dzaporozhets who can we ping for implementation?

https://gitlab.com/gitlab-org/gitlab-ce/issues/28125#note_24863481

@pcarranza Commented on https://gitlab.com/gitlab-org/gitlab-ce/issues/28125#note_24864018

mentioned in merge request !9738 (merged)

marked the task Project Members list should list all members of all ancestor groups https://gitlab.com/gitlab-org/gitlab-ce/issues/28786 as completed

marked the task User doesn't have access to projects shared with a group if they're a member of one of that group's ancestors but not of the group itself directly https://gitlab.com/gitlab-org/gitlab-ce/issues/28787 as completed

Feature name will be Subgroups https://gitlab.com/gitlab-org/gitlab-ce/issues/28761#note_25166592

changed title from Nested groups to Sub groups

@dzaporozhets does this still need the 9.0 milestone? as it is a meta issue and now includes issue for beyond 9.0?

added meta and removed ~19492 labels

marked the task Documentation https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9783 as completed

does this still need the 9.0 milestone? as it is a meta issue and now includes issue for beyond 9.0?

Docs where the final piece, now merged :) I took the liberty and assigned %9.1 for the next iteration.

Great to see this thing happening!! Kudos!

#29702 (closed), wiki clone of repo not working in a project inside a subgroup

@arif-ali probably more useful to open a separate issue, yes please.

@dzaporozhets given the conversation and status of https://gitlab.com/gitlab-org/gitlab-ce/issues/29324 can the HA task towards the top of this meta issue be marked as completed?

GitLab.com HA compatible https://gitlab.com/gitlab-org/gitlab-ce/issues/28125#note_24973610

Quick question... For version 8.16,7 In our development (playground) environment, we created a group and assigned a parent id to it so it became a subgroup. Transferring a project to it seems to make it think the repository doesn't exist anymore. We noticed the files went to the group's folder, as it's supposed to, but the new "subgroup" is still in the git-data/repositories directory instead of git-data/repositories/parentgroup directory as it's looking for. Moving the files over didn't change the error.

Is there a refresh method somewhere we don't see that needs to be called in addition to the parent id being set, or is this something that wasn't working quite yet as of 8.16.7? Or is this a bug that wasn't noticed already?

@K73SK would you mind filing a separate issue for this please and we can investigate - if you link it here once you've done that, it would be very helpful. Thanks.

mentioned in issue #29718 (closed)

@mydigitalself done! :)

mentioned in issue #29725 (closed)

mentioned in issue #29919 (moved)

Are subgroups supported under user namespaces? If not, is this on the roadmap? If so, I can't find the button in the UI to do this when on my profile, how would I create one.

It would be really nice to be able to organise projects under a user namespace: for example, rather than have a bunch of repositories names docker-* create a docker subgroup and put all docker image in there. The way I currently do it is by using multiple organisations prefixed with my username but this feels pretty messy and clutters the global namespace. Happy to create a full feature proposal for this if required.

Org namespaces are free, why not just make an org and then your sub-groups under that for your docker images?

Thanks for that feature, have been waiting for it for a long time.
Just an idea for some improvement, what do you think of a new tab All Projects that show -- big surprise -- all projects under that (sub)group, so even those that are nested.

Shall I make a new issue for that, or is a comment here ok?

@flungo Right now only projects are created under your namespace, not groups. Meaning your namespace does not have it's own group page and this is why subgroups is not available. This would be a new feature proposal.

@bcersows Only groups are nested, not projects. The projects tab shows all projects under the group that you are viewing. I'm confused what you would see under All projects that isn't within Projects already. Can you clarify? An example might help me understand. Thanks!

@tauriedavis, sure.

Sorry for not beaing clear enough.
I got a GitLab with the following structure:

Group A
 |
 -> Project A1
 -> Project A2
 -> Project A3
 |
 --> Subgroup AB
   |
   -> Project AB1 
   -> Project AB2
   |
    --> Subsubgroup AD
     |
     -> Project AD1 
 --> Subgroup AC
   |
   -> Project AC1 
   -> Project AC2

What I'd like to have is that when I open the All Projects tab on group A (which is the master group, maybe for a team, whatever) is not only the direct children projects (A1-3), like it is now, but also the projects AB1/2, AD1, and AC1/2. That way there'd be an easy overview of and access to all groups that are available below that master group. Right now I'd have to check every subgroup (and all subgroups below that) to find a specific project I'm looking for.

This obviously could also be done recursively, so on AB you'd see AB1/2 and AD1.

Did this clear things up?

mentioned in issue #30343 (moved)

Yes, that is super clear, thanks @bcersows. I've made a separate issue for this request https://gitlab.com/gitlab-org/gitlab-ce/issues/30343

mentioned in merge request !10414 (merged)

marked the task GitLab.com HA compatible https://gitlab.com/gitlab-org/gitlab-ce/issues/28125#note_24973610 as completed

marked the task go import support? https://gitlab.com/gitlab-org/gitlab-ce/issues/1337 as completed

@dzaporozhets can we close this issue? It's still on the 9.0 milestone

@smcgivern not yet. Once I have issues for everything at https://gitlab.com/gitlab-org/gitlab-ce/issues/2772#after-90 I will close. For now I just move it to 9.1. OK?

changed milestone to %9.1

@dzaporozhets fine by me!

marked the task Cleanup: rename *_with_namespace with new methods https://gitlab.com/gitlab-org/gitlab-ce/issues/30786 as incomplete

mentioned in issue #30795 (moved)

I created issue for every task at https://gitlab.com/gitlab-org/gitlab-ce/issues/2772#after-90. Those issues will be scheduled by PM and backend, UX teams. I also made few technical debt issues that will be nice to fix in future. I am closing this issue.

cc @JobV @smcgivern @tauriedavis

closed

mentioned in issue #31032 (closed)

mentioned in issue gitlab#4672

mentioned in issue gitlab#7464

mentioned in issue gitlab#8546

mentioned in issue gitlab#8924

mentioned in merge request !14785

Sub groups

Progress

Before 9.0

9.0

After 9.0

Info

Reasons people want this:

Implementation questions

Next steps

Designs

Child items ...

Activity

Routing

Backend

UI

Mentions

Permissions

Something we are 100% sure and we are not going to change

Something we actually want to discuss

About feature in general

Admin message

Admin message

Sub groups

Progress

Before 9.0

9.0

After 9.0

Info

Reasons people want this:

Implementation questions

Next steps

Activity

Routing

Backend

UI

Mentions

Permissions

Something we are 100% sure and we are not going to change

Something we actually want to discuss

About feature in general