Verify stages statuses migration correctness on production

We recently migrated some data regarding pipelines and stages. The code snippet below should be executed in the rails console, to check if migration has been successful.

Ci::Pipeline.order('RANDOM()').limit(200).map do |pipeline|
  pipeline.legacy_stages.map do |legacy_stage|
    persisted_stage = pipeline.stages.find_by(name: legacy_stage.name)

    if persisted_stage.nil?
      "Persisted stage #{legacy_stage.name} for pipeline " \
        "##{pipeline.id} not found!"
    elsif persisted_stage.status != legacy_stage.status
      "Stage `#{persisted_stage.name}` status mismatch: " \
        "#{legacy_stage.status} / #{persisted_stage.status}"
    end
  end
end.flatten.compact

What we need is the value returned by this value, other logs (like ActiveRecord stuff printed into the console) are not relevant.

Please execute it more than once if this is fast enough, sample of 1000 pipelines (execute this 5 times) should be enough.

Thanks in advance! 💛

/cc @ayufan

@omame Can you help me with this? 🙏 😸

changed the description

This should be done after deploying RC1 and all background migrations finished.

Right, thanks @ayufan!

mentioned in issue support#754

@omame @pcarranza Migration is now 100% complete on prod, so we can check the correctness! Thanks in advance! 💛

Lack of a feedback in this issue is a blocker for https://gitlab.com/gitlab-org/gitlab-ce/issues/34514, can someone take a look please? Thanks in advance! 💜

@grzesiek I tried to ping you a couple of times in slack but had no luck. Can you ping me when you've got time?

@omame Do you need my assistance to run this code snippet in the rails console?

assigned to @jtevnan

=> ["Persisted stage external for pipeline #10841545 not found!", "Persisted stage external for pipeline #9551107 not found!", "Persisted stage external for pipeline #12264653 not found!", "Persisted stage externa
l for pipeline #12208851 not found!"] 

=> ["Persisted stage external for pipeline #9621952 not found!", "Persisted stage deploy for pipeline #9616739 not found!", "Persisted stage deploy for pipeline #11145745 not found!", "Persisted stage deploy for p
ipeline #10425693 not found!", "Persisted stage external for pipeline #9902460 not found!"]

=> ["Persisted stage external for pipeline #11219876 not found!", "Persisted stage external for pipeline #9360292 not found!", "Persisted stage deploy for pipeline #11993928 not found!", "Persisted stage deploy fo
r pipeline #9269506 not found!", "Persisted stage external for pipeline #10174786 not found!"]

=> ["Persisted stage deploy for pipeline #9698564 not found!", "Persisted stage external for pipeline #11824120 not found!"]

=> ["Persisted stage external for pipeline #10581944 not found!", "Persisted stage external for pipeline #11097372 not found!"]

=> ["Persisted stage external for pipeline #12353465 not found!", "Persisted stage external for pipeline #9842360 not found!", "Persisted stage external for pipeline #10087174 not found!"]

Awesome! Thanks @jtevnan!

It appears that we still do not create a persisted stage in commit statuses API.

We are going to need yet another migration, so it means that we will have another delay in pipeline controller performance improvements. \cc @ayufan @bikebilly

closed

changed milestone to %WoW

added unblocks others label