Delete Groups which failed to be deleted

Also, this commit limits the amount of projects and groups we queue per
call of the worker. At this time we have over 18k projects marked as
pending_delete, so it wouldn't be wise to queue all these as removing a
project for example is quite compute intensive for the disk and for PG.


Former-commit-id: 1bb70e12

CHANGELOG.md

@@ -134,6 +134,37 @@ entry.
 - Fix and improve `Sortable.highest_label_priority`. !7165
 - Fixed sticky merge request tabs when sidebar is pinned. !7167
 - Only remove right connector of first build of last stage. !7179
+  - Backups do not fail anymore when using tar on annex and custom_hooks only. !5814
+  - Cronjob to delete projects and groups which failed previously !7067
+  - Adds user project membership expired event to clarify why user was removed (Callum Dryden)
+  - Trim leading and trailing whitespace on project_path (Linus Thiel)
+  - Prevent award emoji via notes for issues/MRs authored by user (barthc)
+  - Adds an optional path parameter to the Commits API to filter commits by path (Luis HGO)
+  - Fix extra space on Build sidebar on Firefox !7060
+  - Fix mobile layout issues in admin user overview page !7087
+  - Fix HipChat notifications rendering (airatshigapov, eisnerd)
+  - Add hover to trash icon in notes !7008 (blackst0ne)
+  - Fix sidekiq stats in admin area (blackst0ne)
+  - Removed delete branch tooltip !6954
+  - Escape ref and path for relative links !6050 (winniehell)
+  - Fixed link typo on /help/ui to Alerts section. !6915 (Sam Rose)
+  - Fix filtering of milestones with quotes in title (airatshigapov)
+  - Refactor less readable existance checking code from CoffeeScript !6289 (jlogandavison)
+  - Update mail_room and enable sentinel support to Reply By Email (!7101)
+  - Simpler arguments passed to named_route on toggle_award_url helper method
+  - Fix typo in framework css class. !7086 (Daniel Voogsgerd)
+  - New issue board list dropdown stays open after adding a new list
+  - Fix: Backup restore doesn't clear cache
+  - API: Fix project deploy keys 400 and 500 errors when adding an existing key. !6784 (Joshua Welsh)
+  - Replace jquery.cookie plugin with js.cookie !7085
+  - Use MergeRequestsClosingIssues cache data on Issue#closed_by_merge_requests method
+  - Fix Sign in page 'Forgot your password?' link overlaps on medium-large screens
+  - Show full status link on MR & commit pipelines
+  - Fix documents and comments on Build API `scope`
+  - Refactor email, use setter method instead AR callbacks for email attribute (Semyon Pupkov)
+
+## 8.13.2
+  - Fix builds dropdown overlapping bug !7124
 ## 8.13.1 (2016-10-25)

app/workers/cleanup_pending_delete_worker.rb

 class CleanupPendingDeleteWorker
   include Sidekiq::Worker
+  include CronjobQueue
+
+  MAX_NEW_JOBS = 256
   sidekiq_options retry: false # this job auto-repeats via sidekiq-cron
   def perform
     admin = User.find_by(admin: true)
-    Project.unscoped.where(pending_delete: true).find_each do |project|
+    Group.with_deleted.where.not(deleted_at: nil).limit(MAX_NEW_JOBS).order(:deleted_at).each do |group|
+      GroupDestroyWorker.perform_async(group.id, admin.id)
+    end
+
+    Project.unscoped.where(pending_delete: true).limit(MAX_NEW_JOBS).order(:updated_at).each do |project|
       ProjectDestroyWorker.perform_async(project.id, admin.id)
     end
   end

spec/workers/cleanup_pending_delete_worker_spec.rb

 require 'spec_helper'
 describe CleanupPendingDeleteWorker do
+  let!(:admin)  { create(:admin) }
   let(:project) { create(:empty_project, pending_delete: true) }
-  let(:admin)   { create(:admin) }
-  subject { CleanupPendingDeleteWorker.new }
+  let(:group)   { create(:group) }
+  subject { described_class.new }
   describe "#perform" do
+    it "queues groups for deletion" do
+      # We can't set deleted_at on 1.week.ago because of acts_as_paranoid
+      group.destroy
+
+      expect(GroupDestroyWorker).to receive(:perform_async).with(group.id, admin.id)
+
+      subject.perform
+    end
+
     it "queues projects for deletion" do
       expect(ProjectDestroyWorker).to receive(:perform_async).with(project.id, admin.id)
+
       subject.perform
     end
   end