updated documentation for Geo

Enable Get and Update of ci_default_git_depth for
Project API.

Renaming Project#default_git_depth to :ci_default_git_depth
to give more context through the API usage.

Add API documentation

If the origin project has no default_git_depth set (i.e. nil) set the
fork's default_git_depth to 0

Krasimir Angelov authored 5 years ago and Fabio Pitino committed 5 years ago

Introduce default_git_depth in project's CI/CD settings and set it to
50. Use it if there is no GIT_DEPTH variable specified. Apply this
default only to newly created projects and keep it nil for old ones
in order to not break pipelines that rely on non-shallow clones.

default_git_depth can be updated from CI/CD Settings in the UI, must be
either nil or integer between 0 and 1000 (incl).

Inherit default_git_depth from the origin project when forking projects.

MR pipelines are run on a MR ref (refs/merge-requests/:iid/merge) and it
contains unique commit (i.e. merge commit) which doesn't exist in the
other branch/tags refs. We need to add it cause otherwise it may break
pipelines for old projects that have already enabled Pipelines for merge
results and have git depth 0.

Document new default_git_depth project CI/CD setting

Probably useful as we often move these files to "new" files.

When hashed storage is in use, it's helpful to have the project
name associated with the request.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1394

When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.

Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.

When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.

These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.

Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.

To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.

Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)

In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.

This removes the `ForkedProjectLink` model that has been replaced by
the `ForkNetworkMember` and `ForkNetwork` combination. All existing
relations have been adjusted to use these new models.

The `forked_project_link` table has been dropped.

The "Forks" count on the admin dashboard has been updated to count all
`ForkNetworkMember` rows and deduct the number of `ForkNetwork`
rows. This is because now the "root-project" of a fork network also
has a `ForkNetworkMember` row. This count could become inaccurate when
the root of a fork network is deleted.

A forked project stores its LFS objects in the `forked_from_project`.
So the LFS objects become inaccessible, and therefore delete them from
the database so they'll get cleaned up.

To be refactored when implementing
https://gitlab.com/gitlab-org/gitlab-ce/issues/39769

Direct disk access is done through Gitaly now, so the legacy path was
deprecated. This path was used in Gitlab::Shell however. This required
the refactoring in this commit.

Added is the removal of direct path access on the project model, as that
lookup wasn't needed anymore is most cases.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1111

Prior to this change, this method was called add_namespace, which broke
the CRUD convention and made it harder to grep for what I was looking
for. Given the change was a find and replace kind of fix, this was
changed without opening an issue and on another feature branch.

If more dynamic calls are made to add_namespace, these could've been
missed which might lead to incorrect bahaviour. However, going through
the commit log it seems thats not the case.

In this case the project will get the minimum between both
visibilities.

If that visibility is restricted, then a lower level will be picked.

The helper creates a fork of a project with all provided attributes,
but skipping the creation of the repository on disk.

When no fork network exists for the source projects, we create a new
one with the correct source

There are some redundancies in the validation steps, and that is to
preserve current error messages behavior

Also few specs have to be changed in order to fix madness in validation
logic.

The number of forks of a project doesn't change very frequently and
running a COUNT(*) every time this information is requested can be quite
expensive. We also end up running such a COUNT(*) query at least twice
on the homepage of a project.

By caching this data and refreshing it when necessary we can reduce
project homepage loading times by around 60 milliseconds (based on the
timings of https://gitlab.com/gitlab-org/gitlab-ce).

Remove superfluous lib: true, type: redis, service: true, models: true, services: true, no_db: true, api: true

Signed-off-by: Rémy Coutable <remy@rymai.me>

Projects::ForkService delegates to this service almost entirely, but needed
one small change so it would propagate create errors correctly.

CreateService#execute needs significant refactoring; it is now right at the
complexity limit set by Rubocop. I avoided doing so in this commit to keep the
diff as small as possible.

Several tests depend on the insecure behaviour of ForkService, so fi them up at
the same time.

This reverts commit 3e991230.

# Conflicts:
#	app/models/project.rb