Do backups really need to be version dependent?

Zendesk issue: https://gitlab.zendesk.com/agent/tickets/11362

Although tied to this Zendesk issue I have seen this issue more and more with customers and users. Is there a long term need to tie backup/restore to a single GitLab version? Can we add some simple version detection logic in here to say if it's restoring from an older version run migrations? Also, if the version of the backup is greater than current version, reject the restoration because it may cause problems. There are lots of use cases for restoring an older backup to a newer GitLab and it would eliminate a lot of confusion and extra steps.

I volunteer to take a look at this but I hope anyone will chime in if there are known reasons why we cannot/should not do this.

From looking at the code I can tell what one concern probably is -

If we add something to backup/restore in a newer version - like we did recently for backing up CI builds - this could be concerning. However, if a user is restoring a backup from an older version of GitLab we don't need to care about restoring builds (just skip it if the piece doesn't exist). There's nothing to lose because the data didn't ever exist to begin with.

@dblessing If the migration runs it probably is OK in a lot of cases. What do you think @dzaporozhets ?

I agree that if it is possible it would be great to have this.

Can we add some simple version detection logic in here to say if it's restoring from an older version run migrations?

Check db schema id (20151109100728) in GitLab vs db schema in backup. If same - allow restore.

Also, if the version of the backup is greater than current version, reject the restoration

correct

mentioned in issue #3208 (closed)

One caveat I discovered while working with a customer last week:

If we support importing to newer versions we need to be sure to change the restore process so we explicitly drop all tables before running SQL import. We ran in to an issue where restoring the SQL backup drops all tables that the backup knows about, but it was leave newer tables. This leads to migration failures like 'table X already exists'. Dropping all tables or the entire database explicitly will address this.

cc/ @jacobvosmaer-gitlab

Milestone changed to 8.9

Milestone changed to Backlog

@dblessing 's caveat has been addressed recently so I think we can make this change.

@dzaporozhets wrote:

Check db schema id (20151109100728) in GitLab vs db schema in backup. If same - allow restore.

@dzaporozhets I think we are talking about allowing the restore even if the DB schema is different.

I think we can even relax this to also allow importing a CE backup into EE. This makes it easier for admins who want to upgrade to EE and move to a new server at the same time.

• Parse backup and app versions: CE/EE, major/minor/patch
• Abort if backup is EE and app is CE: I think this is likely to be an admin mistake. All other CE/EE combinations are OK.
• Abort if backup major/minor/patch is newer than app
• After the backup manager finishes, invoke 'db:migrate'. This is idempotent and we already suffered the overhead of loading the Rails app.

@JobV any reason not to want this in 8.9?

@jacobvosmaer-gitlab it's only a matter of capacity. If the requirements are clear and there is capacity, I'm fine with doing it.

cc @DouweM @stanhu

mentioned in merge request !4343 (closed)

@jacobvosmaer-gitlab ok makes sense.

Added ~480950 label

added backup-restore label

mentioned in issue #19300 (closed)

At least it would be great, if the version of GitLab, which created the backup-file, would be part of the filename. Otherwise I have to track each installation of an update of GitLab (and there are often updates, which is good). Like <timestamp>_gitlab_backup_<version>.tar.

Or should I open an extra issue?

@lars.hanisch I agree having the gitlab version in the filename would be useful in its own right.

mentioned in issue #22407 (closed)

Consider offering this functionality for gitlab 7.0 also.

There should be an easy straight forward way to update from 7.0 to 8.15.1 for example.

When can we expect this functionality?

thanks

any news here?

Do you plan to work on this any time soon? Is there a way to update from 7.0 to latest version without updating to each incremental release?

Hi @domnulnopcea You shouldn't need to install every individual version.

@dblessing What's the upgrade path that you would suggest here? 7.0 -> 7.14 -> 8.0 -> 8.16?

why not 7.0 => 8.16?

@domnulnopcea We do not recommend upgrading straight across major versions. Your best bet is to take the path mentioned by @markglenfletcher. Be sure to take backups at each step so you can recover if something goes wrong. I don't anticipate any issues from 8.0 to 8.16 or 8.17, as we've been much more diligent about having solid migrations.

If you're using GitLab CE, be sure to migrate your CI data before you upgrade from 8.0 further.

The whole purpose of the update procedure it to easily take me from 6 to 9 for example... We do not have time to update incrementally. Are you going to offer this functionality?

@domnulnopcea As I mentioned in my last comment, you do not need to upgrade incrementally for each and ever minor release. You do need to upgrade from 6.x to 6.latest, 7.0 to 7.latest, 8.0 to latest. Even if we implemented what we're discussing here, we would not allow restoring from one major version to another. There are breaking changes in major releases and there is too much risk to allow this. I think it's acceptable to allow upgrades from a version like 8.0, which was released in the fall of 2015, to the latest 8.x version which will be released this month - that's a big timeline!

At least it would be great, if the version of GitLab, which created the backup-file, would be part of the filename. Otherwise I have to track each installation of an update of GitLab (and there are often updates, which is good). Like <timestamp>_gitlab_backup_<version>.tar.

MR for including the GitLab version in the filename in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10901

For the record, @stanhu started a MR to improve this: !4343 (closed). /cc @mydigitalself maybe you'd like to re-prioritize since there's already a start.

changed milestone to %Next 3-6 months

moved to gitlab#3908

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

mentioned in merge request !14785