Do backups really need to be version dependent?
Zendesk issue: https://gitlab.zendesk.com/agent/tickets/11362
Although tied to this Zendesk issue I have seen this issue more and more with customers and users. Is there a long term need to tie backup/restore to a single GitLab version? Can we add some simple version detection logic in here to say if it's restoring from an older version run migrations? Also, if the version of the backup is greater than current version, reject the restoration because it may cause problems. There are lots of use cases for restoring an older backup to a newer GitLab and it would eliminate a lot of confusion and extra steps.
I volunteer to take a look at this but I hope anyone will chime in if there are known reasons why we cannot/should not do this.
Activity
From looking at the code I can tell what one concern probably is -
If we add something to backup/restore in a newer version - like we did recently for backing up CI builds - this could be concerning. However, if a user is restoring a backup from an older version of GitLab we don't need to care about restoring builds (just skip it if the piece doesn't exist). There's nothing to lose because the data didn't ever exist to begin with.
@dblessing If the migration runs it probably is OK in a lot of cases. What do you think @dzaporozhets ?
I agree that if it is possible it would be great to have this.
Can we add some simple version detection logic in here to say if it's restoring from an older version run migrations?
Check db schema id (20151109100728) in GitLab vs db schema in backup. If same - allow restore.
Also, if the version of the backup is greater than current version, reject the restoration
correct
mentioned in issue gitlab-foss#3208 (closed)
One caveat I discovered while working with a customer last week:
If we support importing to newer versions we need to be sure to change the restore process so we explicitly drop all tables before running SQL import. We ran in to an issue where restoring the SQL backup drops all tables that the backup knows about, but it was leave newer tables. This leads to migration failures like 'table X already exists'. Dropping all tables or the entire database explicitly will address this.
Milestone changed to 8.9
Milestone changed to Backlog
@dblessing 's caveat has been addressed recently so I think we can make this change.
@dzaporozhets wrote:
Check db schema id (20151109100728) in GitLab vs db schema in backup. If same - allow restore.
@dzaporozhets I think we are talking about allowing the restore even if the DB schema is different.
I think we can even relax this to also allow importing a CE backup into EE. This makes it easier for admins who want to upgrade to EE and move to a new server at the same time.
- Parse backup and app versions: CE/EE, major/minor/patch
- Abort if backup is EE and app is CE: I think this is likely to be an admin mistake. All other CE/EE combinations are OK.
- Abort if backup major/minor/patch is newer than app
- After the backup manager finishes, invoke 'db:migrate'. This is idempotent and we already suffered the overhead of loading the Rails app.
@JobV any reason not to want this in 8.9?
@jacobvosmaer-gitlab it's only a matter of capacity. If the requirements are clear and there is capacity, I'm fine with doing it.
mentioned in merge request gitlab-foss!4343 (closed)
-
@jacobvosmaer-gitlab ok makes sense.
added backup-restore in GitLab FOSS label
mentioned in issue gitlab-foss#19300 (closed)
At least it would be great, if the version of GitLab, which created the backup-file, would be part of the filename. Otherwise I have to track each installation of an update of GitLab (and there are often updates, which is good). Like
<timestamp>_gitlab_backup_<version>.tar.Or should I open an extra issue?
@lars.hanisch I agree having the gitlab version in the filename would be useful in its own right.
mentioned in issue gitlab-foss#22407 (closed)
Consider offering this functionality for gitlab 7.0 also.
There should be an easy straight forward way to update from 7.0 to 8.15.1 for example.
When can we expect this functionality?
thanks
Edited by username-removed-24521
Hi @domnulnopcea You shouldn't need to install every individual version.
@dblessing What's the upgrade path that you would suggest here? 7.0 -> 7.14 -> 8.0 -> 8.16?
@domnulnopcea We do not recommend upgrading straight across major versions. Your best bet is to take the path mentioned by @markglenfletcher. Be sure to take backups at each step so you can recover if something goes wrong. I don't anticipate any issues from 8.0 to 8.16 or 8.17, as we've been much more diligent about having solid migrations.
If you're using GitLab CE, be sure to migrate your CI data before you upgrade from 8.0 further.
@domnulnopcea As I mentioned in my last comment, you do not need to upgrade incrementally for each and ever minor release. You do need to upgrade from 6.x to 6.latest, 7.0 to 7.latest, 8.0 to latest. Even if we implemented what we're discussing here, we would not allow restoring from one major version to another. There are breaking changes in major releases and there is too much risk to allow this. I think it's acceptable to allow upgrades from a version like 8.0, which was released in the fall of 2015, to the latest 8.x version which will be released this month - that's a big timeline!
At least it would be great, if the version of GitLab, which created the backup-file, would be part of the filename. Otherwise I have to track each installation of an update of GitLab (and there are often updates, which is good). Like
<timestamp>_gitlab_backup_<version>.tar.MR for including the GitLab version in the filename in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10901
For the record, @stanhu started a MR to improve this: gitlab-foss!4343 (closed). /cc @mydigitalself maybe you'd like to re-prioritize since there's already a start.
changed milestone to %Next 3-6 months in gitlab-foss
moved from gitlab-foss#3366 (moved)
