Mail listener ping-ponging with mailer-daemon

@Lozik What was the initial email that you sent? A reply by email comment for an issue?

added ~65372 label

In my specific case, the initial e-mail actually came from the gitlab installation. I added an user with an "@ example.com"-address whom I granted access to a repository. Gitlab then sent an e-mail to that not working address and got a reply by my mail server, that the e-mail could not be delivered. And starting there, the sequential responses never stopped.

Summary:

Gitlab -> Example.com: Access granted
Mailserver(mailer-daemon) -> Gitlab: Example.com not in service; could not deliver
Gitlab -> Mailserver(mailer-daemon): Email could not be parsed
Mailserver(mailer-daemon) -> Gitlab: Mailbox "mailer-daemon" not in service

Phase 3-4 repeated endlessly.

To follow up with this issue: I've just upgraded from Gitlab 8.16.4-ce to 8.17.0-ce. The issue seems to be non existent in this new version. There seems to be no 'could not be processed'-message from Gitlab anymore. However, I do not know if this was a known change in the upgrade or if this is just a coincidence. Maybe it would be necessary to check and verify this.

Is this missing of the 'could not be processed'-message a wanted behaviour? I would actually like such a message, since the goal is to submit an issue or respond to a comment without accessing a web page. At that point a feedback or at least a negative feedback would be great. Accessing the web page to confirm that the action has succeeded is maybe a bit contra productive.

@godfat Is this a known issue that has since been fixed?

@markglenfletcher I think this is a known issue and a duplicate of #18548 (moved). However I don't think we did anything directly related to this yet. Not sure why it's somehow fixed in this case, but since we did some tweaks to the email reply parsing codes, it could be that it's now recognized as a different issue and... maybe it silently failed it? Would be nice to investigate.

Having the same issue just now ... I think it started with replying to a failed pipeline process. I have GitLab 9.1.2. Any tips on how I can stop that ping pong game?

Stopping the mutual e-mail transmission can be done by temporarily disabling the gitlab "Reply by email" feature.

Useful information over here: https://docs.gitlab.com/ce/administration/reply_by_email.html

@Lozik That helped! Thank's a lot!

Maybe in a future version we should add a black list of email addresses not to reply to, like mailer-daemon@gmail.com, ...@yahoo.com and ...@microsoft.com, etc.

The alternative would be to parse incoming messages for 'Can't deliver mail' or something. That would probably have to happen in multiple languages, which makes the blacklist idea easier.

I just came to this same issue.

The best I can sort out is: On June 3 we added a new intern to Gitlab, who's email may or may not have been set up yet. Gitlab sent the Account was created for you email.

Shortly after, the battle begins. It looks like it started with a "message delayed" email, however I can't find that first response to the gitlab inbox. The first response I can find is Gitlab replying saying it can't understand, so I can't be certain this is in response to creating the new user account, but it makes sense as it has worked well until this, and someone above mentions something similar.

GitLab <gitlab@mycompany.com>
Jun 3

to mailer-daemon 
Unfortunately, your email message to GitLab could not be processed.

We couldn't figure out what the email is for. Please create your issue or comment through the web interface.

which received a reply from Mail Delivery Subsystem <mailer-daemon@googlemail.com> stating


Mail Delivery Subsystem
Jun 3

to me 
Error Icon
Message not delivered
There was a problem delivering your message to mailer-daemon@googlemail.com. See the technical details below, or try resending in a few minutes.
LEARN MORE
The response from the remote server was:
550 5.2.1 The user you are trying to contact is receiving mail at a rate that prevents additional messages from being delivered. For more information, please visit https://support.google.com/mail/?p=ReceivingRatePerm c75si25525920ioj.87 - gsmtp


Final-Recipient: rfc822; mailer-daemon@googlemail.com
Action: failed
Status: 5.2.1
Remote-MTA: dns; gmail-smtp-in.l.google.com. (2607:f8b0:4001:c0a::1a, the
 server for the domain googlemail.com.)
Diagnostic-Code: smtp; 550-5.2.1 The user you are trying to contact is receiving mail at a rate that
 550-5.2.1 prevents additional messages from being delivered. For more
 550-5.2.1 information, please visit
 550 5.2.1  https://support.google.com/mail/?p=ReceivingRatePerm c75si25525920ioj.87 - gsmtp
Last-Attempt-Date: Sat, 03 Jun 2017 09:22:01 -0700 (PDT)


---------- Forwarded message ----------
From: GitLab <gitlab@mycompany.com>
To: mailer-daemon@googlemail.com
Cc: 
Bcc: 
Date: Sat, 03 Jun 2017 11:15:01 -0500
Subject: [Rejected] Delivery Status Notification (Delay)
Unfortunately, your email message to GitLab could not be processed.

We couldn't figure out what the email is for. Please create your issue or comment through the web interface.

I uncovered this because Sentry was starting to report errors:

This had me go check out Sidekiq:

This is the dead jobs tab. You can't see the 'failed jobs' tab, so I'm not sure if those will represent a couple hundred more email replies to retry processing (that will fail) or what.

Either way, I'm going to take a stab at disabling/re-enabling reply-by-email.

After trawling through the emails a bit more, I notice another culprit that was created: ghost@example.com I'm not sure if this is adding to the party of reply loops, but it seems reasonable enough. I see this user creation mentioned here: https://gitlab.com/gitlab-org/gitlab-ce/issues/31325

I think we need to do something here even there's no clear action we could take yet.

added ~480950 ~12980 reply by email labels

changed milestone to %Backlog

I took a stab at disabling email integration based on the comment above. I left it off for about 24 hours and re-enabled it. Very shortly after I got back into a reply war.

Would using gmail filters help bypass the messages from getting picked up by GitLab? For example would using: create a filter from:mailer-daemon@googlemail.com -> Skip Inbox (archive it) help? Or would GitLab still pick those up?

@ttilberg I suppose, what you could try is disabling the reply by mail function, then deleting all emails in the git-mailbox (or marking them all as read, at least). Then you could reenable the reply by mail service. At that point, no pingpong emails should be send, until there is again a trigger e-mail address.

@godfat I think, there should be done something about it, too. I do not know a whole lot about the gitlab architecture but a bug like this could potentially block all the server's performance and even overload mail servers. It could also consume all server memory or even all disk space in installations with a lot of users.

@Lozik Yes, I agree.

@smcgivern What do you think? Perhaps we could find an easy trick to prevent this? Maybe throttling for the same receiver would do it?

@godfat we already raise AutoGeneratedEmailError and send a response to that (although apparently not in all cases - there are two specs which use the auto_reply.yml fixture).

We should probably just always raise that when an email contains an Auto-Submitted header, and then never reply in that case. Wdyt?

@smcgivern Oh, I was not aware (or forgot?) that we could detect it already. In that case, yes I think we should not reply it in all cases. I think the risk is quite low.

Sounds simple to do, I could probably take it. What do you think?

@smcgivern Reading above, this might not solve the issue here though. Do they have headers containing /auto-(generated|replied)/?

Also, our responses should probably also contain this header.

@godfat they should do! @Lozik can you confirm please?

We already set this header on our emails

Yes, I can confirm, at least for Postfix, that a header flag Auto-Submitted: auto-replied is set for "Undelivered Mail"-Notices from MAILER-DAEMON. I do, however, not know if this is the standard for other mail servers like MS Exchange.

On the 24th of February this year, the Emails sent by Gitlab did not (yet) have such a header flag. I suppose however, that this has been changed later, as @smcgivern mentioned.

It was added in 9.0: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9163

We should probably just always raise that when an email contains an Auto-Submitted header, and then never reply in that case. Wdyt?

I think this would be a good idea, since the whole bug could also occur when it is not the mailer daemon responding but simply an 'out of office'-autoresponder.

Edit: However there might be some autoresponders that do not set such an Auto-Submitted flag. In that case, it would be helpful to additionally surveil the situation and, with a timer, check for such bounce ping-pongs? That should eliminate the last bit of possibility of the situation to occur. Although, that solution seems to be more complex...

assigned to @godfat

added Discussion and removed Platform ~65372 labels

changed milestone to %9.5

mentioned in merge request !13254 (merged)

closed via merge request !13254 (merged)

Mail listener ping-ponging with mailer-daemon

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Possible fixes

Designs

Child items ...

Activity

Admin message

Admin message

Mail listener ping-ponging with mailer-daemon

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Possible fixes

Activity