Pipeline retries on projects where pipelines are disabled

added ~19173 ~901060 ~12980 reproduced on GitLab.com labels

@mrtux Thanks for the report! We will take a look at this problem! 🌲

Another extreme example is https://gitlab.com/tortoisegit/tortoisegit/commit/5556a42885b4524d8b776a9f2ac271bbe6d93ba7/builds where you can see 6 builds with several retries. Each retry causes the external service to run a completely new built, which is not necessaty and a waste of resources.

That's why I cancelled some of the extermal service, but now those are shown as "failed". Here it would also be perfect if those entries could be deleted by the project owner.

mentioned in issue #23638 (closed)

assigned to @grzesiek

changed milestone to %9.0

I believe that this is a duplicate of https://gitlab.com/gitlab-org/gitlab-ce/issues/28656, and should be fixed by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9618.

mentioned in merge request !9618 (merged)

closed via merge request !9618 (merged)

reopened

This is not fixed, I just noticed this again. There are four retries:

@mrtux It should be fixed 🤔 Can you provide some details please? How do you post commit status using commit status API, what JSON do you post?

I can't provide more information, please see AppVeyor.com

removed assignee

removed milestone

I can't find docs about AppVeyor's integration with GitLab and commit status API, but from the GitLab perspective, commit status change request will update existing external job only if these attributes are the same for each subsequent request:

SHA
ref (branch / tag name)
job name
same user communicating with the API

If one of these attributes changes, the external job will be retried.

I don't see why job name and especially ref needs to be the same... Doesn't the SHA count for the commit?

Hi guys,

Feodor from AppVeyor is here. We use this API method to create/update commit statuses in GitLab: https://docs.gitlab.com/ce/api/commits.html#post-the-build-status-to-a-commit

We populate only 4 fields:

            var body = new
            {
                state = state,
                target_url = targetUrl,
                description = description,
                context = pullRequest ? "continuous-integration/appveyor/pr" : "continuous-integration/appveyor/branch"
            };

and project id and sha come in URL.

Let me know if there are any issues with that call or you need more information.

@FeodorFitsner The code responsible for updating a commit status or creating a retry looks like:

        status = GenericCommitStatus.running_or_pending.find_or_initialize_by(
          project: @project,
          pipeline: pipeline,
          name: name,
          ref: ref,
          user: current_user
        )

This means that if name (or context) changed, status will be retried. Same for user and ref. Are you updating status using the same authenticated user each time? Note that this changed in 9.0. @mrtux are you using 9.0?

@grzesiek This issue is filed against gitlab.com, so: yes.

Why does the commit status api rely on ref? A commit status is just dependand on the SHA1, isn't it?
Why should the commit status API schedule retries at all (if no pipelines are enabled)?

@mrtux I wonder if we should clarify concepts, because there seems to be some confusion here.

Commit status API is used to post and read a commit status using an external service. In other words:

You can have an external service that reads a commit status.
You can have an external service that posts a commit status.

In GitLab you can see statuses posted by an external service like AppVeyor. The question is how does AppVeyor receive information about pushes / tags. Does it use webhooks? It appears that there is some AppVeyor-specific job that is retried, or maybe you are referring to retries visible in GitLab?

We use ref because it is possible to have the same SHA on multiple branches, and we need to know on to which merge request commit status belongs to.

AppVeyor is informed by a webhook (on Push Events and Tag Push Events).

Yeah, that is exactly the concept I thought of it is, but somehow it is tighly intertwined with pipelines (and retries..., it seems to me as if the webhook is re-executed, otherwise AppVeyor would not queue new jobs for same SHA/ref).

If a SHA is on multiple branches, what's the issue? Update all?!

@mrtux Commit statuses are tightly coupled with internal CI/CD pipelines in GitLab. We plan to decouple it a little, to make it possible to extend both feature is separate directions, but we do not have immediate plans to do it. For now, it looks for me like a AppVeyor internal problem, instead of GitLab. @mrtux does it make sense to close this issue then? We still can continue discussion here to find a good solution for AppVeyor.

@grzesiek Do you mean posting a commit status can (in some circumstances like when repo uses built-in CI/CD pipelines?) trigger repository webhook?

@mrtux Is it possible to create a repro of this issue in a clean public repository?

Do you mean posting a commit status can (in some circumstances like when repo uses built-in CI/CD pipelines?) trigger repository webhook?

@FeodorFitsner Not really. I mean that what I conclude after seeing this image

and from docs:

For every project AppVeyor will configure webhooks for its repository to automatically start a build when you push the changes. For every private project AppVeyor will add an SSH public key (deployment key) to the clone repository on the build machine.

is that the problem @mrtux describes is seeing multiple retried builds in AppVeyor, not in GitLab. Since Commit Status API is used to post status from AppVeyor to GitLab, it appears that this is not a problem in GitLab (or at least not in Commit Status API, maybe webhooks are a culprit?).

Is that correct?

Just to make it a bit more fuzzy: The builds are not re-tried everytime (see https://ci.appveyor.com/project/TortoiseGit/tortoisegit-06nst/history), the SHAs 9e5f65e7, 6d20f291, b37c98e9 and 2069682d were tested just once - whereas de738877 twice and 08fe5a5d five times...

OK, I think we need a simple repro to further work on this issue...

@FeodorFitsner Feel free to create a test repo.

Why not look into the logs and spot what happened for these rebuilds? - Whether several webhooks fired or what else happened?

After having faced the issue of multiple builds for a single push, too, I investigated a little and it seems the webhook request is indeed repeated multiple times by GitLab. See https://github.com/appveyor/ci/issues/1617 for details.

The only thing I noticed that could even remotely explain the behavior is that AppVeyor answers with HTTP status 204 instead of 200. Might this throw GitLab off track?

Also it seems "Recent Deliveries" is broken making it unsuitable to debug these kind of issues.

I'm consistently getting five builds for my fork of the Inkscape repo now (https://gitlab.com/Ede123/inkscape/ https://ci.appveyor.com/project/Ede123/inkscape/). If there's anything I can do to help track down the issue please let me know!

Could this issue be related to the webhook request timing out? (see #15 (closed), #2229 (closed), and #2779 (closed) for similar issues that could be solved by increasing the timeout by setting the webhook_timeout variable).

@grzesiek (or anybody else who has this information):

What is the webhook_timeout set for gitlab.com?
Do I understand correctly the webhook will be retried if the timeout is reached?
If yes, how often and with what delays will the webhook be re-tried? (I'm often seeing five duplicates so far, so maybe this is the maximum?)

I suppose GitLab and AppVeor both have logs, but noone seems to look into them...

moved to gitlab#7480

Hey! 👋

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

Pipeline retries on projects where pipelines are disabled

Summary

Steps to reproduce

Expected behavior

Actual behavior

Relevant logs and/or screenshots

Output of checks

Possible fixes

Designs

Child items 0

Activity

Frequently Asked Questions

Admin message

Admin message

Pipeline retries on projects where pipelines are disabled

Summary

Steps to reproduce

Expected behavior

Actual behavior

Relevant logs and/or screenshots

Output of checks

Possible fixes

Activity

Frequently Asked Questions