Restore scheduled deletions if the user loses group/project access

This change addresses an issue where users can schedule a group or
project for deletion and subsequently lose access to it. This can
occur in various scenarios, such as user deletion, membership
downgrade, blocking, or banning.

Currently, when Sidekiq cron workers attempt to load groups/projects
scheduled for deletion, the Sidekiq jobs fail indefinitely. This
happens because the deleting user no longer has access to the
group/project being deleted, as the `DestroyService`s include
permission checks.

I considered several solutions:

1. Skip the authorization check when calling the destroy services:
   * Pro: Simple implementation
   * Con: Highly destructive; risky if a bad actor schedules a
     group/project for deletion

2. Delete the project/group deletion schedule when the user loses
   access:
   * Pro: Removes the deletion schedule immediately upon user removal
   * Con: Complex implementation arises due to numerous ways users can
     lose access, including deletion, downgrade, blocking, and
     banning. Handling all cases, especially membership downgrades
     through direct, inherited, or shared membership removal, is
     challenging. This complexity makes processing in the `DELETE`
     request to the members' removal API inefficient. The
     `Members::DestroyService` still needs optimization due to timeout
     issues. While a separate worker could be used, it's not the most
     efficient solution and may cause race conditions when users are
     quickly removed and re-added, potentially leaving group/project
     schedules intact. Moreover, this approach doesn't address
     existing projects and groups that are failing deletion.

3. Check permissions in the cron worker deleting the scheduled
   projects/groups:
   * Pro: Covers all cases without needing a separate worker
   * Con: Deletion may succeed if a user regains access just before
     the scheduled time

We chose option 3 as the best solution. Although there's a minor risk
of deletion succeeding if a user regains access just before the
scheduled time, this aligns with our documentation stating that the
user should have access at the scheduled time for deletion to succeed.

Currently, we partially use option 2 for group schedule deletion upon
membership removal. However, it only deletes the group deletion
schedule if the user has direct membership to the group. For inherited
or shared membership, the group deletion schedule remains intact. This
new approach addresses these limitations and provides a more
comprehensive solution.

This implementation checks user permissions during the cron job
execution and restores the resource if the user doesn't have access,
ensuring that scheduled deletions are handled appropriately.

Changelog: changed
EE: true

.rubocop_todo/layout/line_length.yml

@@ -2089,7 +2089,6 @@ Layout/LineLength:
     - 'ee/spec/views/shared/credentials_inventory/personal_access_tokens/_personal_access_token.html.haml_spec.rb'
     - 'ee/spec/views/shared/promotions/_promotion_link_project.html.haml_spec.rb'
     - 'ee/spec/workers/active_user_count_threshold_worker_spec.rb'
-    - 'ee/spec/workers/adjourned_group_deletion_worker_spec.rb'
     - 'ee/spec/workers/adjourned_projects_deletion_cron_worker_spec.rb'
     - 'ee/spec/workers/analytics/cycle_analytics/consistency_worker_spec.rb'
     - 'ee/spec/workers/analytics/devops_adoption/create_all_snapshots_worker_spec.rb'

doc/user/group/index.md

@@ -213,7 +213,11 @@ You can also delete a group from the groups dashboard:
 On GitLab [Premium](https://about.gitlab.com/pricing/premium/) and [Ultimate](https://about.gitlab.com/pricing/ultimate/), this action adds a background job to mark a group for deletion. By default, the job schedules the deletion seven days in the future. You can modify this retention period through the [instance settings](../../administration/settings/visibility_and_access_controls.md#deletion-protection).
-If the user who set up the deletion is removed from the group before the deletion occurs, the group deletion job is canceled.
+If the user who scheduled the group deletion loses access to the group (for example, by leaving the group, having their role downgraded, or being banned from the group) before the deletion occurs,
+the deletion job will instead restore and unarchive the group, so the group will no longer be scheduled for deletion.
+
+   WARNING:
+   If the user who scheduled the group deletion regains Owner role or administrator access before the job runs, then the job removes the group permanently.
 ### View groups pending deletion

doc/user/project/working_with_projects.md

@@ -197,6 +197,12 @@ You can [view projects that are pending deletion](#view-projects-pending-deletio
 and use the Rails console to
 [find projects that are pending deletion](troubleshooting.md#find-projects-that-are-pending-deletion).
+If the user who scheduled the project deletion loses access to the project (for example, by leaving the project, having their role downgraded, or being banned from the project) before the deletion occurs,
+the deletion job will instead restore and unarchive the project, so the project will no longer be scheduled for deletion.
+
+   WARNING:
+   If the user who scheduled the project deletion regains Owner role or administrator access before the job runs, then the job removes the project permanently.
+
 ### Delete a project immediately
 DETAILS:

ee/app/services/ee/members/destroy_service.rb

@@ -16,7 +16,6 @@ def after_execute(member:, skip_saml_identity: false)
         end
         cleanup_group_identity(member) unless skip_saml_identity
-        cleanup_group_deletion_schedule(member) if member.source.is_a?(Group)
         cleanup_oncall_rotations(member)
         cleanup_escalation_rules(member) if member.user
         reset_seats_usage_callouts(member)
@@ -70,14 +69,6 @@ def cleanup_group_identity(member)
         saml_provider.identities.for_user(member.user).delete_all
       end
-      def cleanup_group_deletion_schedule(member)
-        deletion_schedule = member.source&.deletion_schedule
-
-        return unless deletion_schedule
-
-        deletion_schedule.destroy if deletion_schedule.deleting_user == member.user
-      end
-
       def cleanup_oncall_rotations(member)
         user = member.user

ee/app/services/namespaces/groups/adjourned_deletion_service.rb

+# frozen_string_literal: true
+
+# This service manages the group deletion process:
+# - Permanently deletes the group if user has sufficient permissions.
+# - Restores the group if user lacks necessary permissions.
+module Namespaces
+  module Groups
+    class AdjournedDeletionService < ::BaseGroupService
+      def execute
+        if can_current_user_remove_group?
+          delete_group
+        else
+          restore_group
+        end
+      end
+
+      private
+
+      def can_current_user_remove_group?
+        Gitlab::Auth::CurrentUserMode.optionally_run_in_admin_mode(current_user) do
+          current_user.can?(:remove_group, group)
+        end
+      end
+
+      def delete_group
+        GroupDestroyWorker.perform_in(params[:delay], group.id, current_user.id)
+      end
+
+      def restore_group
+        admin_bot = ::Users::Internal.admin_bot
+        Gitlab::Auth::CurrentUserMode.optionally_run_in_admin_mode(admin_bot) do
+          ::Groups::RestoreService.new(group, admin_bot).execute
+        end
+      end
+    end
+  end
+end

ee/app/services/projects/adjourned_deletion_service.rb

+# frozen_string_literal: true
+
+# This service manages the project deletion process:
+# - Permanently deletes the project if user has sufficient permissions.
+# - Restores the project if user lacks necessary permissions.
+module Projects
+  class AdjournedDeletionService < ::BaseProjectService
+    def execute
+      if can_current_user_remove_project?
+        delete_project
+      else
+        restore_project
+      end
+    end
+
+    private
+
+    def can_current_user_remove_project?
+      return false unless current_user
+
+      Gitlab::Auth::CurrentUserMode.optionally_run_in_admin_mode(current_user) do
+        current_user.can?(:remove_project, project)
+      end
+    end
+
+    def delete_project
+      ::Projects::DestroyService.new(project, current_user).async_execute
+    end
+
+    def restore_project
+      admin_bot = ::Users::Internal.admin_bot
+      Gitlab::Auth::CurrentUserMode.optionally_run_in_admin_mode(admin_bot) do
+        ::Projects::RestoreService.new(project, admin_bot).execute
+      end
+    end
+  end
+end

ee/app/workers/adjourned_group_deletion_worker.rb

@@ -24,7 +24,9 @@ def perform
       user = deletion_schedule.deleting_user
       with_context(namespace: group, user: user) do
-        GroupDestroyWorker.perform_in(delay, group.id, deletion_schedule.user_id)
+        Namespaces::Groups::AdjournedDeletionService
+          .new(group: group, current_user: user, params: { delay: delay })
+          .execute
       end
     end
   end

ee/app/workers/adjourned_project_deletion_worker.rb

@@ -14,9 +14,9 @@ def perform(project_id)
     project = Project.find(project_id)
     user = project.deleting_user
-    return unless user
-
-    ::Projects::DestroyService.new(project, user).async_execute
+    Projects::AdjournedDeletionService
+      .new(project: project, current_user: user)
+      .execute
   rescue ActiveRecord::RecordNotFound => error
     logger.error("Failed to delete project (#{project_id}): #{error.message}")
   end

ee/spec/services/ee/members/destroy_service_spec.rb

@@ -78,30 +78,6 @@
       include_examples 'sends streaming audit event'
     end
-    context 'group deletion schedule' do
-      context 'when member user has a scheduled deletion for the group' do
-        let!(:group_deletion_schedule) { create(:group_deletion_schedule, group: group, user_id: member_user.id, marked_for_deletion_on: 2.days.ago) }
-
-        it 'deletes the group deletion schedule' do
-          expect(group.reload.deletion_schedule).to eq(group_deletion_schedule)
-
-          destroy_service.execute(member)
-
-          expect(group.reload.deletion_schedule).to be nil
-        end
-      end
-
-      context 'when scheduled deletion for the group belongs to different user' do
-        let!(:group_deletion_schedule) { create(:group_deletion_schedule, group: group, user_id: current_user.id, marked_for_deletion_on: 2.days.ago) }
-
-        it 'does not delete the group deletion schedule' do
-          destroy_service.execute(member)
-
-          expect(group.reload.deletion_schedule).to eq(group_deletion_schedule)
-        end
-      end
-    end
-
     context 'on-call rotations' do
       let!(:project) { create(:project, group: group) }

ee/spec/services/namespaces/groups/adjourned_deletion_service_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+RSpec.describe Namespaces::Groups::AdjournedDeletionService, feature_category: :groups_and_projects do
+  let_it_be(:delay) { 1.hour }
+  let_it_be(:params) { { delay: delay } }
+  let(:group) { create(:group) }
+  let(:resource) { group }
+
+  subject(:service) { described_class.new(group: group, current_user: user, params: params) }
+
+  def ensure_destroy_worker_scheduled
+    expect(GroupDestroyWorker).to receive(:perform_in).with(delay, group.id, user.id)
+  end
+
+  include_examples 'adjourned deletion service'
+end

ee/spec/services/projects/adjourned_deletion_service_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+RSpec.describe Projects::AdjournedDeletionService, feature_category: :groups_and_projects do
+  let(:project) { create(:project, marked_for_deletion_at: 10.days.ago, marked_for_deletion_by_user_id: user&.id) }
+  let(:resource) { project }
+
+  subject(:service) { described_class.new(project: project, current_user: user) }
+
+  def ensure_destroy_worker_scheduled
+    expect(ProjectDestroyWorker).to receive(:perform_async).with(project.id, user.id, {})
+  end
+
+  include_examples 'adjourned deletion service'
+
+  context 'when user cannot remove the project', :sidekiq_inline do
+    context 'with deleted user' do
+      let(:user) { nil }
+
+      it_behaves_like 'user cannot remove'
+    end
+  end
+end

ee/spec/support/shared_examples/services/adjourned_deletion_service_shared_examples.rb

+# frozen_string_literal: true
+
+RSpec.shared_examples 'adjourned deletion service' do
+  before do
+    stub_licensed_features(adjourned_deletion_for_projects_and_groups: true)
+  end
+
+  shared_examples 'user can remove resource' do
+    it 'enqueues the resource destroy worker' do
+      ensure_destroy_worker_scheduled
+
+      service.execute
+    end
+
+    it 'removes the resource' do
+      service.execute
+
+      record_exists = resource.class.exists?(resource.id)
+      expect(record_exists).to be_falsey
+    end
+  end
+
+  shared_examples 'user cannot remove' do
+    it 'uses admin bot to restore the resource', :enable_admin_mode do
+      service.execute
+
+      expect(resource.reload.marked_for_deletion?).to eq(false)
+    end
+  end
+
+  context 'when user can remove resource', :sidekiq_inline do
+    context 'when user is an owner' do
+      let(:user) { create(:user) }
+
+      before do
+        resource.add_owner(user)
+      end
+
+      it_behaves_like 'user can remove resource'
+    end
+
+    context 'when user is an admin with admin mode enabled', :enable_admin_mode do
+      let(:user) { create(:admin) }
+
+      it_behaves_like 'user can remove resource'
+    end
+
+    context 'when user is an admin with admin mode disabled' do
+      let(:user) { create(:admin) }
+
+      before do
+        stub_application_setting(admin_mode: false)
+      end
+
+      it_behaves_like 'user can remove resource'
+    end
+  end
+
+  context 'when user cannot remove the resource', :sidekiq_inline do
+    let(:user) { create(:user) }
+
+    context 'when user is non-admin with admin mode enabled', :enable_admin_mode do
+      before do
+        resource.add_maintainer(user)
+      end
+
+      it_behaves_like 'user cannot remove'
+    end
+
+    context 'when user has maintainer access' do
+      before do
+        resource.add_maintainer(user)
+      end
+
+      it_behaves_like 'user cannot remove'
+    end
+
+    context 'when user is blocked' do
+      before do
+        user.block!
+      end
+
+      it_behaves_like 'user cannot remove'
+    end
+
+    context 'when user is banned' do
+      before do
+        user.ban!
+      end
+
+      it_behaves_like 'user cannot remove'
+    end
+  end
+end

ee/spec/workers/adjourned_group_deletion_worker_spec.rb

@@ -6,58 +6,125 @@
   describe "#perform" do
     subject(:worker) { described_class.new }
-    let_it_be(:user) { create(:user) }
-    let_it_be(:group_not_marked_for_deletion) { create(:group) }
-
-    let_it_be(:group_marked_for_deletion) do
-      create(
-        :group_with_deletion_schedule,
-        marked_for_deletion_on: 14.days.ago,
-        deleting_user: user
-      )
+    let(:user) { create(:user) }
+    let!(:group_not_marked_for_deletion) { create(:group) }
+    let!(:parent_group) { create(:group) }
+    let!(:group_marked_for_deletion) do
+      create(:group_with_deletion_schedule,
+        parent: parent_group,
+        marked_for_deletion_on: 15.days.ago,
+        deleting_user: user)
     end
-    let_it_be(:group_marked_for_deletion_for_later) do
-      create(
-        :group_with_deletion_schedule,
-        marked_for_deletion_on: 2.days.ago,
-        deleting_user: user
-      )
+    let!(:group_marked_for_deletion_for_later) do
+      create(:group_with_deletion_schedule, marked_for_deletion_on: 2.days.ago, deleting_user: user)
     end
     before do
       stub_application_setting(deletion_adjourned_period: 14)
+      stub_licensed_features(adjourned_deletion_for_projects_and_groups: true)
     end
-    it 'only schedules to delete groups marked for deletion on or before the specified `deletion_adjourned_period`' do
-      expect(GroupDestroyWorker).to receive(:perform_in).with(0, group_marked_for_deletion.id, user.id)
-      worker.perform
-    end
-    it 'does not schedule to delete a group not marked for deletion' do
-      expect(GroupDestroyWorker).not_to receive(:perform_in).with(0, group_not_marked_for_deletion.id, user.id)
-      worker.perform
-    end
-    it 'does not schedule to delete a group that is marked for deletion after the specified `deletion_adjourned_period`' do
-      expect(GroupDestroyWorker).not_to receive(:perform_in).with(0, group_marked_for_deletion_for_later.id, user.id)
-      worker.perform
+    context 'when deleting user has access to delete the group' do
+      before do
+        group_not_marked_for_deletion.add_owner(user)
+        group_marked_for_deletion.add_owner(user)
+        group_marked_for_deletion_for_later.add_owner(user)
+      end
+      it 'only schedules to delete groups marked for deletion on or before the specified `deletion_adjourned_period`' do
+        expect(GroupDestroyWorker).to receive(:perform_in).with(0, group_marked_for_deletion.id, user.id)
+        worker.perform
+      end
+      it 'does not schedule to delete a group not marked for deletion' do
+        expect(GroupDestroyWorker).not_to receive(:perform_in).with(0, group_not_marked_for_deletion.id, user.id)
+
+        worker.perform
+      end
+
+      it 'does not schedule to delete a group marked for deletion after the specified `deletion_adjourned_period`' do
+        expect(GroupDestroyWorker).not_to receive(:perform_in).with(0, group_marked_for_deletion_for_later.id, user.id)
+
+        worker.perform
+      end
+      it 'schedules groups 20 seconds apart' do
+        group_marked_for_deletion_2 = create(
+          :group_with_deletion_schedule,
+          marked_for_deletion_on: 14.days.ago,
+          deleting_user: user,
+          owners: user
+        )
+        expect(GroupDestroyWorker).to receive(:perform_in).with(0, group_marked_for_deletion.id, user.id)
+        expect(GroupDestroyWorker).to receive(:perform_in).with(20, group_marked_for_deletion_2.id, user.id)
+
+        worker.perform
+      end
     end
-    it 'schedules groups 20 seconds apart' do
-      group_marked_for_deletion_2 = create(
-        :group_with_deletion_schedule,
-        marked_for_deletion_on: 14.days.ago,
-        deleting_user: user
-      )
-      expect(GroupDestroyWorker).to receive(:perform_in).with(0, group_marked_for_deletion.id, user.id)
-      expect(GroupDestroyWorker).to receive(:perform_in).with(20, group_marked_for_deletion_2.id, user.id)
-      worker.perform
+    context 'with direct and indirect accesses to group', :sidekiq_inline do
+      shared_examples 'destroys the group' do
+        specify do
+          worker.perform
+
+          expect(Group.exists?(group_marked_for_deletion.id)).to be_falsey
+        end
+      end
+
+      context 'when user is a direct owner' do
+        before do
+          group_marked_for_deletion.add_owner(user)
+        end
+
+        it_behaves_like 'destroys the group'
+      end
+
+      context 'when user is an inherited owner' do
+        before do
+          parent_group.add_owner(user)
+        end
+
+        it_behaves_like 'destroys the group'
+      end
+
+      context 'when user is an owner through group sharing' do
+        before do
+          invited_group = create(:group, owners: user)
+          create(:group_group_link, :owner, shared_group: group_marked_for_deletion, shared_with_group: invited_group)
+        end
+
+        it_behaves_like 'destroys the group'
+      end
+
+      context 'when user is an owner through parent group sharing' do
+        before do
+          invited_group = create(:group, owners: user)
+          create(:group_group_link, :owner, shared_group: parent_group, shared_with_group: invited_group)
+        end
+
+        it_behaves_like 'destroys the group'
+      end
+
+      context 'when an admin deletes the group', :enable_admin_mode do
+        let_it_be(:user) { create(:admin) }
+
+        before do
+          group_marked_for_deletion.deletion_schedule.update!(deleting_user: user)
+        end
+
+        it_behaves_like 'destroys the group'
+      end
+    end
+    context 'when the deleting user does not have access to delete the group', :sidekiq_inline, :enable_admin_mode do
+      it 'restores the group' do
+        worker.perform
+        expect(group_marked_for_deletion.reload.marked_for_deletion?).to be_falsey
+      end
     end
   end
 end

ee/spec/workers/adjourned_project_deletion_worker_spec.rb

@@ -7,36 +7,82 @@
     subject(:worker) { described_class.new }
     let_it_be(:user) { create(:user) }
-    let_it_be(:project) { create(:project, deleting_user: user, owners: user) }
-    let(:service) { instance_double(Projects::DestroyService) }
-    shared_examples 'executes destroying project' do
-      specify do
-        expect(service).to receive(:async_execute)
-        expect(Projects::DestroyService).to receive(:new).with(project, user).and_return(service)
-        worker.perform(project.id)
+    let_it_be(:group) { create(:group) }
+    let_it_be(:project) { create(:project, group: group, marked_for_deletion_at: 8.days.ago, deleting_user: user) }
+    context 'when deleting user has access to remove the project', :sidekiq_inline do
+      shared_examples 'destroys the project' do
+        specify do
+          worker.perform(project.id)
+          expect(Project.exists?(project.id)).to be_falsey
+        end
       end
-    end
-    it_behaves_like 'executes destroying project'
-    context 'when an admin deletes the project', :enable_admin_mode do
-      let_it_be(:user) { create(:admin) }
-      before do
-        project.update!(deleting_user: user)
+      context 'when user is a direct owner' do
+        before_all do
+          project.add_owner(user)
+        end
+
+        it_behaves_like 'destroys the project'
+      end
+
+      context 'when user is an inherited owner' do
+        before_all do
+          group.add_owner(user)
+        end
+
+        it_behaves_like 'destroys the project'
+      end
+
+      context 'when user is an owner through project sharing' do
+        before_all do
+          invited_group = create(:group, owners: user)
+          create(:project_group_link, :owner, project: project, group: invited_group)
+        end
+
+        it_behaves_like 'destroys the project'
+      end
+      context 'when user is an owner through parent group sharing' do
+        before_all do
+          invited_group = create(:group)
+          create(:group_group_link, :owner, shared_group: group, shared_with_group: invited_group)
+          invited_group.add_owner(user)
+        end
+        it_behaves_like 'destroys the project'
       end
-      it_behaves_like 'executes destroying project'
+      context 'when an admin deletes the project', :enable_admin_mode do
+        let_it_be(:user) { create(:admin) }
+
+        before do
+          project.update!(deleting_user: user)
+        end
+
+        it_behaves_like 'destroys the project'
+      end
     end
-    it 'stops execution if user was deleted' do
-      project.update!(deleting_user: nil)
-      expect(Projects::DestroyService).not_to receive(:new)
-      worker.perform(project.id)
+    context 'when deleting user does not have access to remove the project', :sidekiq_inline do
+      shared_examples 'restores the project' do
+        specify do
+          worker.perform(project.id)
+          expect(project.reload.marked_for_deletion_at.present?).to eq(false)
+        end
+      end
+
+      it_behaves_like 'restores the project'
+      context 'when deleting user was deleted' do
+        before do
+          project.update!(deleting_user: nil)
+        end
+
+        it_behaves_like 'restores the project'
+      end
     end
   end
 end

spec/support/rspec_order_todo.yml

@@ -2168,7 +2168,6 @@
 - './ee/spec/services/ee/keys/destroy_service_spec.rb'
 - './ee/spec/services/ee/labels/promote_service_spec.rb'
 - './ee/spec/services/ee/members/create_service_spec.rb'
-- './ee/spec/services/ee/members/destroy_service_spec.rb'
 - './ee/spec/services/ee/members/import_project_team_service_spec.rb'
 - './ee/spec/services/ee/members/invite_service_spec.rb'
 - './ee/spec/services/ee/members/update_service_spec.rb'
@@ -2537,7 +2536,6 @@
 - './ee/spec/views/subscriptions/buy_storage.html.haml_spec.rb'
 - './ee/spec/views/subscriptions/new.html.haml_spec.rb'
 - './ee/spec/workers/active_user_count_threshold_worker_spec.rb'
-- './ee/spec/workers/adjourned_group_deletion_worker_spec.rb'
 - './ee/spec/workers/adjourned_projects_deletion_cron_worker_spec.rb'
 - './ee/spec/workers/admin_emails_worker_spec.rb'
 - './ee/spec/workers/analytics/code_review_metrics_worker_spec.rb'