Notification mails don't arrive anymore

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
Notification mails don't arrive anymore
by on (#231318)
Recently, I don't get any notification mails anymore when somebody answered the topic.

The threads where people answered are definitely in my subsription options:
Attachment:
Subscriptions.png
Subscriptions.png [ 15.39 KiB | Viewed 6421 times ]


And just to make sure, I added a rule to my e-mails "if sender contains nesdev, then move to inbox and don't apply any more rules".

But I still don't get the notification mails anymore, even though it worked in the past and I didn't change my account settings.

Can you please check what's up here? Were the notifications of the two threads in the screenshot sent to me in the past days, so that I have to re-evaluate my e-mail program? Or did they never go out?
Re: Notification mails don't arrive anymore
by on (#231319)
I think the same goes for PM notifications now? Not sure when it started since those are sorted into a gmail folder i need to check manually anyway, but that would suggest the mail robot is down.
Re: Notification mails don't arrive anymore
by on (#231321)
The emails stopped going out after 12-31. Google likes flagging the account for "suspicious activity", usually it works again after I log in and re-verify. We'll see if that fixes it.

Thanks for the notice!
Re: Notification mails don't arrive anymore
by on (#231668)
In the meantime, it worked again, but now I have the issue again: Two forum replies and one private message without a notification.
Re: Notification mails don't arrive anymore
by on (#231712)
When I logged in Google gave me a stern warning about sending unsolicited emails, and suggested that I file a "my account is locked" appeal, which I did. I doubt it would be acted on, but whatever. Other than that, it let me log in normally and is back to sending messages again.

Lately we've had some massive spam attacks, with huge numbers of fake accounts passing the registration trivia questions, could be related to that. Or it could be Google changing things over there, not sure.
Re: Notification mails don't arrive anymore
by on (#231714)
Interesting. When I search something in Google, it sometimes tells me that I have to verify that I'm a human because apparently, suspicious actions came from my network. Maybe they're just paranoid in the moment.

This time, the notification mail arrived.
Re: Notification mails don't arrive anymore
by on (#231736)
DRW wrote:
Interesting. When I search something in Google, it sometimes tells me that I have to verify that I'm a human because apparently, suspicious actions came from my network. Maybe they're just paranoid in the moment.

They definitely are, and also, they want to force people to make google accounts, and for those having an account, they want to foce them to give them personal data such as phone number.
Re: Notification mails don't arrive anymore
by on (#232002)
It happened again: Got a PN, but no e-mail notification.

Maybe you should send the mails out with a different mail address if Google is being shitty in the moment?
Re: Notification mails don't arrive anymore
by on (#232004)
Changing the From: line, or the SMTP envelope From, will not fix this problem.
Re: Notification mails don't arrive anymore
by on (#232017)
For your reply, I got a mail again. It seems to fail arbitrarily.
Re: Notification mails don't arrive anymore
by on (#232019)
I tried switching it to send through membler-industries.com. Not sure if it's working yet, nothing is showing up in the "sent" box, but I'm not sure if it normally would. Let me know if you see it working or not.

edit: I was able to have it send a test message to myself. But in Gmail it showed up in the spam folder, if that happens to everyone else then it's not going to work very well.
Re: Notification mails don't arrive anymore
by on (#232132)
I didn't get a mail again for this reply here and for a thread where I wrote a post and where others posted.

Also, I most likely would have seen an NESDev mail in the spam forum, so I assume it never arrived.
Re: Notification mails don't arrive anymore
by on (#232555)
Aaand it happened again: Got a private message, but no notification.
Re: Notification mails don't arrive anymore
by on (#232556)
Speaking of which, can we change the header from "info" to "nesdev notification" or the like?
Re: Notification mails don't arrive anymore
by on (#232557)
For the past week it has been emailing through membler-industries.com, but I'm not sure I have the SPF record set properly for membler-industries.com, and it probably needs to be changed on nesdev.com as well, I'll need WhoaMan to do that. So while the emails are going out, in my gmail address they show up in the spam folder (with it reporting it couldn't verify the sender). I guess I'll have to switch it back to the old system for now.

FrankenGraphics wrote:
Speaking of which, can we change the header from "info" to "nesdev notification" or the like?

Surprisingly, phpBB won't accept the usual "Name <email@domain.com>" format as an email address. To include a name, it would have to be hacked into the code, that's really dumb.
Re: Notification mails don't arrive anymore
by on (#232569)
Memblers wrote:
For the past week it has been emailing through membler-industries.com, but I'm not sure I have the SPF record set properly for membler-industries.com, and it probably needs to be changed on nesdev.com as well, I'll need WhoaMan to do that.
Yeah, the domain that it is reportedly coming from has to say what mail servers are allowed to send mail on its behalf. And "mx" doesn't take a parameter...
Re: Notification mails don't arrive anymore
by on (#232575)
That would be an SPF DNS record.
Re: Notification mails don't arrive anymore
by on (#233553)
Recently, the notification mails appear at completely random times and out of order. Today, I got a mail for the first reply in a thread, despite having received multiple notifications for the other replies before.

Seriously, there is some major issue in your mailing system.
Re: Notification mails don't arrive anymore
by on (#233889)
A friend of mine is trying to register an account here, but he's not able to get the account activation email. He's tried a few different email addresses, and tried using the feature to re-send the activation email, but is still unsuccessful. Maybe that is related?

Also, if we could get his account authorized while this email issue is ironed out, that would be great! Just PM me for his account details. :)
Re: Notification mails don't arrive anymore
by on (#235073)
These have been working for me for several days now. Not sure if something changed or not, but thought I'd mention it. These are the actual headers (with some stuff modified, and a couple initial lines removed), to give folks some idea of what the mail flow is like:

Code:
From info@nesdev.com  Tue Feb 26 18:10:01 2019
Return-Path: <info@nesdev.com>
...
Received: from qproxy4-pub.mail.unifiedlayer.com (qproxy4-pub.mail.unifiedlayer.com [66.147.248.250])
        by mambo.koitsu.org (Postfix) with ESMTP id 30E8B4E7F0
        for <XXX>; Tue, 26 Feb 2019 18:05:40 -0800 (PST)
Received: from cmgw11.unifiedlayer.com (unknown [10.9.0.11])
        by qproxy4.mail.unifiedlayer.com (Postfix) with ESMTP id C2638A034F
        for <XXX>; Tue, 26 Feb 2019 19:05:38 -0700 (MST)
Received: from box609.bluehost.com ([70.40.220.109])
        by cmsmtp with ESMTP
        id yobKgpppND8RpyobKgDPU1; Tue, 26 Feb 2019 19:05:38 -0700
X-Authority-Reason: nr=8
X-Authority-Reason: s=1
X-Authority-Analysis: $(_cmae_reason
Received: from [208.71.141.54] (port=35686 helo=gen.8bitalley.com)
        by box609.bluehost.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256)
        (Exim 4.91)
        (envelope-from <info@nesdev.com>)
        id 1gyobK-000zYs-E3
        for XXX; Tue, 26 Feb 2019 19:05:38 -0700
Subject: XXX
To: XXX
From: <info@nesdev.com>
Reply-To: <info@nesdev.com>
Sender: <info@nesdev.com>
MIME-Version: 1.0
Message-ID: XXX
Date: Tue, 26 Feb 2019 19:04:02 -0700
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: phpBB3
X-MimeOLE: phpBB3
X-phpBB-Origin: phpbb://forums.nesdev.com
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - box609.bluehost.com
X-AntiAbuse: Original Domain - koitsu.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - nesdev.com
X-BWhitelist: no
X-Source-IP: 208.71.141.54
X-Source-L: No
X-Exim-ID: 1gyobK-000zYs-E3
X-Source:
X-Source-Args:
X-Source-Dir:
X-Source-Sender: (gen.8bitalley.com) [208.71.141.54]:35686
X-Source-Auth: nesdevforum@membler-industries.com
X-Email-Count: 3
X-Source-Cap: bWVtYmxlcmk7bWVtYmxlcmk7Ym94NjA5LmJsdWVob3N0LmNvbQ==
X-Local-Domain: no
Status: RO
Content-Length: 514
Lines: 17

Running full headers through a tool like this sheds some light on the topology and possible causes. The topology currently is this:

1. gen.8bitalley.com (nesdev box) --> bluehost.com (who is a general web/hosting provider); could be a service/company/thing running on BlueHost
2. bluehost.com --> something I can't exactly figure out, maybe local mail parsing of some kind, or some TCP/SSL tunnel, I don't know.
3. Somehow at this point the mail makes it to unifiedlayer.com
4. unifiedlayer sends the mail along to another mail server they themselves run
5. That server then sends it out onto the Internet (in this case, my own server that I use to manage my own Email)

I have no idea what the relationship is between all these things, but no wonder there are problems. Things could go missing, flagged, whatever -- and nobody has any insights into why because there appear to be 2 "intermediate" parties/providers involved (BlueHost and UnifiedLayer). That said, using said tool, we find something interesting:

During step #5, the tool reports "Blacklisted". I was like, "what? Surely not my box". Nope, not mine, but rather UnitedLayer!

https://mxtoolbox.com/SuperTool.aspx?ac ... ailheaders

They seem to be on several major/key DNSBL lists, which means they have a history of sending out spam. It doesn't mean nesdev is sending out spam -- they aren't -- it means that one of those intermediary servers does. Anyone running a mail server on the Internet that does DNSBL lookups and sees the IP of UnitedLayer mail servers (66.147.248.250 in this particular case, but I imagine they have tons) could have their email flat out rejected (i.e. you never get a copy), or possibly accepted but scored as spam (i.e. the mail arrives but in your spam folder).

This is from a SORBS DNSBL lookup:

Attachment:
Capture.PNG
Capture.PNG [ 50.49 KiB | Viewed 10833 times ]

This is really not good, guys.

Finally: nesdev.com -- the sending domain -- has no SPF (TXT) records of what SMTP servers are authoritative for sending mail out on its behalf. This isn't the reason for the above, I assure you, but it is certainly complicating matters too. In fact, maybe it lacks SPF/TXT records *because* of this whole BlueHost->UL thing, I don't know.

Code:
$ dig txt nesdev.com.

; <<>> DiG 9.12.3-P4 <<>> txt nesdev.com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31516
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: d8b654e3b3aafbb92633df695c7608e5370799386a39067d (good)
;; QUESTION SECTION:
;nesdev.com.                    IN      TXT

;; AUTHORITY SECTION:
nesdev.com.             10735   IN      SOA     ns1.dreamhost.com. hostmaster.dreamhost.com. 2018091600 17404 1800 1814400 14400

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Feb 26 19:49:57 PST 2019
;; MSG SIZE  rcvd: 131


Not bragging or egoing, I assure you, but Parodius did not work this way. When nesdev was hosted with me, the mail went directly from our box out onto the Internet. I ran a tight ship and did not use intermediary mail providers -- because all it takes is one customer sending spam via such a place and suddenly that mail provider is blacklisted. There's a lot of good reasons I did everything directly.

I kinda wonder if the community would actually prefer to have the nesdev sites hosted back on something that isn't set up this way, i.e. done the "old way" (without ads too) and with reliable Email and so on. I could probably do that in a matter of days, though more likely weeks given my full-time job and so on. I just have a feeling WhoaMan is full-time swamped with his job, runs this as a hobby that "mostly works", but when it breaks/misbehaves/issues like this happen, it's hard to get them fixed (given the model/topology).
Re: Notification mails don't arrive anymore
by on (#235098)
koitsu wrote:
Changing the From: line, or the SMTP envelope From, will not fix this problem.

Do not be so categorical.
Note: SMTP was originally defined by two separate documents, RFC-821, which defined the communication protocol between sending and receiving servers, and RFC-822. I fix it by using free software - that includes the function of "fast fix" SMTP-service and of all protocols. Work for me.
Re: Notification mails don't arrive anymore
by on (#235102)
ArtemisK wrote:
Do not be so categorical.
Note: SMTP was originally defined by two separate documents, RFC-821, which defined the communication protocol between sending and receiving servers, and RFC-822. I fix it by using free software - that includes the function of "fast fix" SMTP-service and of all protocols. Work for me.

I have literally no idea what this means (free software that uses the function of "fast fix" SMTP-service and of all protocols) -- this sounds almost like marketing material.

And I'm chuckling at someone throwing RFCs at me as if I don't actually have familiarity with them for 30 years. :-) I might point you to RFC 5321, for example, for a more present-day version of how SMTP behaves, and also point you to RFCs 2505, 5322, 4871, 6376, and 7208 (or 4408 for some history) (re: spam related).

Rephrased: if you think From/From: dictate what will/won't pass spam filters or DNSBL, then that's false. Those are involved, yes, but it's not just one thing. Even SPF/DKIM records aren't responsible for permit/deny or entire scoring. Here's the part a lot of people don't seem to realise:

A lot of spam software analyses Received: and if upon seeing an IP address from a particular CIDR (ex. 67.188.0.0/16 for Comcast) sending mail through a non-Comcast SMTP server, will flag it and/or score it negatively. It's why you can't just pick a separate SMTP relay host (sometimes called a "smart host") somewhere on the Internet and have mail no longer be flagged/scored negatively or even blocked -- you literally have to send mail through your ISP's mail server otherwise this happens. AWS IP space, for example, is often flagged this way if the mail isn't siphoned through AWS's SES service; Gsuite/Gmail is another one (though some of the big hosting provides like this actually rewrite Received: headers to help with this problem, which is both good and bad depending on how you see it). This is all stuff we (as people) have no control over -- sometimes it's in open-source software, other times it's in large-scale enterprise software (ex. Proofpoint). In either case, there's nothing someone can do about it -- that someone has no control over other people's anti-spam mail configurations.

Finally, you might think "use a VPN service!" Sometimes that can work, but you have to be very careful with your SMTP MTA configuration in this case, ensuring that IP addresses used for Internet-bound delivery utilise that of the local VPN tunnel endpoint, while for local mail only things like 127.0.0.1 are used; if there's an intermediary network that's outside of private-allocated CIDR space (10/8, 192.168/16, etc.) that shows up in Received: due to the MTA adding it then you're still subject to the same problem. You might think "well then disable the MTA entirely and just port forward outbound TCP port 25 traffic to whatever IP the VPN service offers for SMTP", which will work, except you now have locally-delivered mail going out onto the Internet -- things like failing cronjobs trying to be delivered to root@yourmachinename, making its way to your VPN provider's SMTP service, which is not what you want (and why running an MTA is a good thing). You just have to be very careful about MTA configuration.

In this case though, I think a big part of this is because UnifiedLayer is flagged for spam in several (at least 6) public DNSBLs. There's nothing nesdev admins can do about that, other than simplify the topology (read: don't use UnifiedLayer, use BlueHost directly -- if they even offer SMTP servers), and/or set up SPF and/or DKIM records denote what IPs *can* send out mail on behalf of nesdev.com.