During a project import, `LfsLinkService` attempts to link `LfsObjects`
that have not already been associated with a project.  It's possible for
a large repo to have thousands of OIDs, which can cause long database
query and parsing times.  By processing a batch of 1000 at a time, we
can reduce that time at the expense of a few more SQL queries.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/66274

This backports to CE changes that allow the recording of the
repository_type in the table lfs_objects_projects.

This is in order to allow future pruning of unreferenced LFS objects,
as we will need to know which repository to look in for the LFS pointer
file.

The EE MR that contains the original code and a full explanation of the
changes is
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13894

EE Issue https://gitlab.com/gitlab-org/gitlab-ee/issues/9490

Note that there was a lot of CE code changed in the EE MR because we
want to allow the wiki repository to also use LFS. See
https://gitlab.com/gitlab-org/gitlab-ce/issues/43721. As the wiki is
an unlicensed feature, a full backport is required to enable this.

Jasper Maes authored 6 years ago and Heinrich Lee Yu committed 5 years ago

Model.new.attributes now also returns encrypted attributes.

Add an index to the `file_store` column on `lfs_objects`. This makes
counting local objects faster.

Also, there is no longer need to check for objects with `file_store`
being `NULL`. See
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18557

---

### Query plans

#### Before & with `NULL`

```
Aggregate  (cost=113495.96..113495.97 rows=1 width=8) (actual time=1691.394..1691.394 rows=1 loops=1)
  ->  Seq Scan on lfs_objects  (cost=0.00..106415.50 rows=2832186 width=0) (actual time=0.012..1312.488 rows=2852607 loops=1)
        Filter: ((file_store = 1) OR (file_store IS NULL))
        Rows Removed by Filter: 131
Planning time: 0.077 ms
Execution time: 1691.433 ms
```

#### Before, without `NULL`

```
Aggregate  (cost=113495.96..113495.97 rows=1 width=8) (actual time=856.423..856.424 rows=1 loops=1)
  ->  Seq Scan on lfs_objects  (cost=0.00..106415.50 rows=2832186 width=0) (actual time=0.012..672.181 rows=2852607 loops=1)
        Filter: (file_store = 1)
        Rows Removed by Filter: 131
Planning time: 0.128 ms
Execution time: 856.470 ms
```

#### After & with `NULL`

```
Aggregate  (cost=68819.95..68819.96 rows=1 width=8) (actual time=583.355..583.355 rows=1 loops=1)
  ->  Index Only Scan using index_lfs_objects_on_file_store on lfs_objects  (cost=0.43..61688.35 rows=2852643 width=0) (actual time=0.028..399.177 rows=2852607 loops=1)
        Filter: ((file_store = 1) OR (file_store IS NULL))
        Rows Removed by Filter: 131
        Heap Fetches: 867
Planning time: 0.096 ms
Execution time: 583.404 ms
```

#### After, without `NULL`

```
Aggregate  (cost=68817.29..68817.30 rows=1 width=8) (actual time=490.550..490.551 rows=1 loops=1)
  ->  Index Only Scan using index_lfs_objects_on_file_store on lfs_objects  (cost=0.43..61685.68 rows=2852643 width=0) (actual time=0.040..311.760 rows=2852607 loops=1)
        Index Cond: (file_store = 1)
        Heap Fetches: 831
Planning time: 0.294 ms
Execution time: 490.590 ms
```

Closes https://gitlab.com/gitlab-org/gitlab-ee/issues/6067

This whitelists all existing places where we use "destroy_all".

Partially addresses #47424.

Old records have a null value of file_store column.
This causes the problems with current direct_upload implementation,
as this makes it to choose Store::REMOTE instead of Store::LOCAL.

This change moves the store save when change saving the object.

This reverts the addition of the "goldiloader" Gem and all use of it.
While this Gem is very promising it's causing a variety of problems on
GitLab.com due to it eager-loading too much data in places where we
don't expect/can handle this. At least for the time being this means we
have to go back to manually fixing N+1 query problems, but at least
those should not cause a negative impact on availability.

Goldiloader is great, but has several issues with has_many :through relations:

* https://github.com/salsify/goldiloader/issues/12
* https://github.com/salsify/goldiloader/issues/14
* https://github.com/salsify/goldiloader/issues/18

Rather than try to figure out which applies in each case, we should just do the
drudge work of manually disabling autoloading for all relations of this type. We
can always use regular preloading for specific cases, but this way we avoid
generating invalid queries through Goldiloader's magic.

Sean McGivern authored 7 years ago and Micael Bergeron committed 6 years ago

See merge request gitlab-org/gitlab-ee!4753

Move uploads to object storage

Closes #4163

See merge request gitlab-org/gitlab-ee!3867

Transfer job archives after creation

See merge request gitlab-org/gitlab-ee!3646

Can migrate LFS objects to S3 style object storage

Closes #2841

See merge request !2760

This is allowed for existing instances so we don't end up 76 offenses
right away, but for new code one should _only_ use this if they _have_
to remove non database data. Even then it's usually better to do this in
a service class as this gives you more control over how to remove the
data (e.g. in bulk).

This reverts commit e00fb2bdc2090e9cabeb1eb35a2672a882cc96e9.

# Conflicts:
#	.rubocop.yml
#	.rubocop_todo.yml
#	lib/gitlab/ci/config/entry/global.rb
#	lib/gitlab/ci/config/entry/jobs.rb
#	spec/lib/gitlab/ci/config/entry/factory_spec.rb
#	spec/lib/gitlab/ci/config/entry/global_spec.rb
#	spec/lib/gitlab/ci/config/entry/job_spec.rb
#	spec/lib/gitlab/ci/status/build/factory_spec.rb
#	spec/lib/gitlab/incoming_email_spec.rb

This reverts commit cb10b725c8929b8b4460f89c9d96c773af39ba6b.

This reverts commit 3e991230.

# Conflicts:
#	app/models/project.rb

In 8278b763 the default behaviour of annotation
has changes, which was causing a lot of noise in diffs. We decided in #17382
that it is better to get rid of the whole annotate gem, and instead let people
look at schema.rb for the columns in a table.

Fixes: #17382