Gmail spam filter and topic reply notifications

Anything about use of this forum : NOT about cycling
User avatar
freiston
Posts: 793
Joined: 6 Oct 2013, 10:20am
Location: Coventry

Gmail spam filter and topic reply notifications

Postby freiston » 4 Nov 2020, 3:12pm

Since about a couple of weeks ago (I could be a week or two out), the Gmail spam filter has filtered all topic reply notifications into my spam folder. Even though I flag them all as not spam, this persists with new topic reply notifications. Gmail is helpful enough to give the following information:
Screenshot from 2020-11-04 14-54-07.png

I don't know much about these things but I've tried examining the message source and comparing it to an earlier message that wasn't filtered as spam but cannot see anything which would make the later email be identified as spam but not the earlier one.

I haven't yet attempted to customise my Gmail spam filter because I thought I would ask here first if anyone is familiar with this issue and can give the required steps to take (rather than me doing it by trial and error). I also thought it might be handy to have a thread started in case anyone else has the issue but has not resolved it.
Disclaimer: Treat what I say with caution and if possible, wait for someone with more knowledge and experience to contribute. ;)


User avatar
freiston
Posts: 793
Joined: 6 Oct 2013, 10:20am
Location: Coventry

Re: Gmail spam filter and topic reply notifications

Postby freiston » 4 Nov 2020, 4:33pm

Cheers Jonathan.

I also found this page on a search:
https://clean.email/gmail-spam-filter - particularly the "Bypass Spam Filters for Messages from Addresses or Domains" sub-heading.

I haven't tried anything yet but I will report back when I do.
Disclaimer: Treat what I say with caution and if possible, wait for someone with more knowledge and experience to contribute. ;)

User avatar
Graham
Moderator
Posts: 6489
Joined: 14 Dec 2006, 8:48pm

Re: Gmail spam filter and topic reply notifications

Postby Graham » 4 Nov 2020, 7:46pm

Such message should be from "mbadmin@cyclinguk.org"

I'll have a quick look at the Admin Panel to see if I can change it there.

EDIT : I have found a couple of cases of the old email address being used behind the scenes. I have updated both.
Hopefully, this will work better ( if I haven't broken anything :? ).

User avatar
freiston
Posts: 793
Joined: 6 Oct 2013, 10:20am
Location: Coventry

Re: Gmail spam filter and topic reply notifications

Postby freiston » 4 Nov 2020, 10:00pm

Cheers Graham, I will wait for further notifications to see what happens before I change anything.
Disclaimer: Treat what I say with caution and if possible, wait for someone with more knowledge and experience to contribute. ;)

richardfm
Posts: 372
Joined: 15 Apr 2018, 3:17pm
Location: Cardiff, Wales

Re: Gmail spam filter and topic reply notifications

Postby richardfm » 4 Nov 2020, 10:21pm

freiston wrote:Cheers Graham, I will wait for further notifications to see what happens before I change anything.

Hopefully you won't have to wait too long

User avatar
freiston
Posts: 793
Joined: 6 Oct 2013, 10:20am
Location: Coventry

Re: Gmail spam filter and topic reply notifications

Postby freiston » 5 Nov 2020, 2:52pm

Thanks for the notification-inducing post, richardfm :D

Unfortunately, this does not seem to have resolved the issue - same "problem" reported with cyclinguk.org:
Screenshot from 2020-11-05 14-31-44.png

Here's an extract from the message source of a notification from the forum before I started having the problem:
message source of notification not flagged as spam wrote:ARC-Authentication-Results: i=1; mx.google.com;
spf=pass (google.com: domain of forumbounces@fonant.com designates 89.145.86.110 as permitted sender) smtp.mailfrom=forumbounces@fonant.com
Return-Path: <forumbounces@fonant.com>
Received: from elsie.fonant.com (elsie.fonant.com. [89.145.86.110])
by mx.google.com with ESMTPS id h9si1187036wrp.54.2020.09.01.05.11.25
for <freiston@theforum.com*>
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Tue, 01 Sep 2020 05:11:25 -0700 (PDT)
Received-SPF: pass (google.com: domain of forumbounces@fonant.com designates 89.145.86.110 as permitted sender) client-ip=89.145.86.110;
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of forumbounces@fonant.com designates 89.145.86.110 as permitted sender) smtp.mailfrom=forumbounces@fonant.com
Received: from elsie.fonant.com (localhost [127.0.0.1])

and here's an extract from the message source of the latest notification flagged as spam:
message source of notification flagged as spam wrote:ARC-Authentication-Results: i=1; mx.google.com;
spf=pass (google.com: domain of forumbounces@fonant.com designates 89.145.86.110 as permitted sender) smtp.mailfrom=forumbounces@fonant.com;
dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=QUARANTINE) header.from=cyclinguk.org
Return-Path: <forumbounces@fonant.com>
Received: from elsie.fonant.com (elsie.fonant.com. [89.145.86.110])
by mx.google.com with ESMTPS id d126si3619596wmf.33.2020.11.04.14.21.48
for <freiston@theforum.com*>
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Wed, 04 Nov 2020 14:21:48 -0800 (PST)
Received-SPF: pass (google.com: domain of forumbounces@fonant.com designates 89.145.86.110 as permitted sender) client-ip=89.145.86.110;
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of forumbounces@fonant.com designates 89.145.86.110 as permitted sender) smtp.mailfrom=forumbounces@fonant.com;
dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=QUARANTINE) header.from=cyclinguk.org
Received: from elsie.fonant.com (localhost [127.0.0.1])

*My email address altered

As I've mentioned previously, I don't really understand these things but does it look like this issue might be down to changes at the google end rather than the forum end?
Disclaimer: Treat what I say with caution and if possible, wait for someone with more knowledge and experience to contribute. ;)

User avatar
admin
Site Admin
Posts: 1242
Joined: 14 Dec 2006, 8:27pm
Location: Lancing, West Sussex
Contact:

Re: Gmail spam filter and topic reply notifications

Postby admin » 5 Nov 2020, 3:10pm

That's very useful, thanks!

Looks like the messages are now failing DMARC validation, because the message isn't signed by the outgoing mail server (my server, not Cycling UK's one).

Because cyclinguk.org now has a DMARC record, asking for failed messages to be quarantined, it ends up in the Spam folder.

I'll chat to Cycling UK's IT department, as I might be able to get the Forum to send outgoing mail via their mail server, which would fix the problem.

User avatar
freiston
Posts: 793
Joined: 6 Oct 2013, 10:20am
Location: Coventry

Re: Gmail spam filter and topic reply notifications

Postby freiston » 5 Nov 2020, 3:42pm

Thank you very much :D
Disclaimer: Treat what I say with caution and if possible, wait for someone with more knowledge and experience to contribute. ;)

Samuel D
Posts: 3079
Joined: 8 Mar 2015, 11:05pm
Location: Paris
Contact:

Re: Gmail spam filter and topic reply notifications

Postby Samuel D » 26 Nov 2020, 9:53am

This problem persists for me and is really quite a nuisance, because no matter how many times I tell Gmail that these messages ‘look safe’ and are ‘not spam’, it insists on throwing them in the bin. I’ve probably missed some of them in there.

This must be affecting a huge number of people even if they don’t realise it or complain about it (because they don’t check their spam folder before it’s automatically emptied). Consider how many people use the CyclingUK forum with a Gmail address for notifications.

User avatar
admin
Site Admin
Posts: 1242
Joined: 14 Dec 2006, 8:27pm
Location: Lancing, West Sussex
Contact:

Re: Gmail spam filter and topic reply notifications

Postby admin » 26 Nov 2020, 10:39am

I've contacted Cycling UK again, in the hope that we can sort this out.

Samuel D
Posts: 3079
Joined: 8 Mar 2015, 11:05pm
Location: Paris
Contact:

Re: Gmail spam filter and topic reply notifications

Postby Samuel D » 27 Nov 2020, 11:20am

Thanks, admin.

User avatar
deliquium
Posts: 2127
Joined: 9 Mar 2007, 3:40pm
Location: Eryri

Re: Gmail spam filter and topic reply notifications

Postby deliquium » 1 Dec 2020, 5:00pm

Same problem here - and even following Gmail's suggestion to create a filter that is supposed to stop mbadmin@cyclinguk.org forever going to the Gmail spam folder - doesn't work either.

Is the problem the via fonant.com ?

Gmail says:

Remove 'via' information from emails not sent through Gmail

Gmail checks whether the messages that you send are authenticated.

If you send messages with a bulk-mailing vendor or third-party affiliates, prevent your emails from being blocked by Gmail.
Publish an SPF record that includes the IPs of the vendor or affiliates which send your messages.
Sign your messages with a DKIM signature that is associated with your domain.
Make sure that the domain in the 'From:' address matches the domain that you're using to authenticate your emails.
Current pedalable joys

"you would be surprised at the number of people in these parts who nearly are half people and half bicycles"

Psamathe
Posts: 11932
Joined: 10 Jan 2014, 8:56pm

Re: Gmail spam filter and topic reply notifications

Postby Psamathe » 1 Dec 2020, 5:09pm

Not sure who controls what but wouldn't adding cyclinguk.org to the SPF record help?

Ian

User avatar
admin
Site Admin
Posts: 1242
Joined: 14 Dec 2006, 8:27pm
Location: Lancing, West Sussex
Contact:

Re: Gmail spam filter and topic reply notifications

Postby admin » 2 Dec 2020, 10:55am

The problem is as follows:

Emails from the Forum are sent "From" <mbadmin@cyclinguk.org>, as that's the best address to handle any replies.

Emails from the Forum are sent by <forumbounces@fonant.com> (the "Envelope sender" or "Return-Path") because that's the best address to handle any non-delivery notices. It also means that SPF tests pass, as the server is a fonant.com machine.

There are three main email tests that recipient mail servers can use: SPF, DKIM and DMARC.

SPF tests the "Return-Path" or "Envelope sender" (in this case <forumbounces@fonant.com>) against the list of authorised mail servers specified by the sending domain (in this case fonant.com). This passes.

DKIM tests the "From" address combined with a signature generated from a secret key held by the domain's mail servers (in this case cyclinguk.org mail servers), and the contents of the message. The test uses a public key published by the domain (in this case cyclinguk.org). This fails, because the Forum server cannot sign outgoing messages with the correct secret key.

DMARC tests two things: SPF and DKIM. The SPF tests pass, but the DKIM tests fail.

The possible fixes are:

1) Stop using DKIM for cyclinguk.org email.
2) Send Forum messages via a cyclinguk.org mail server.
3) Add the cyclinguk.org secret DKIM key to Fonant mail servers... not a safe idea.

I think the second option is the best, but will require the Forum to authenticate with the cyclinguk.org mail server. This might have security/fraud/impersonation implications.

TL;DR: Email security is complicated, DMARC is a nice idea but a right pain in the real world.