Cloudflare DMARC Management: Buggy?

On April 8 I have deleted all API keys in Sendgrid, disabled “less secure apps” in Google account, purge from the Google account link to Gmass, paused warmup services and changed Gmail password. There is nothing what could send emails now from my side. Even after this I receive the following statistics from Cloudflare what is actually based on email providers reports. I really don’t understand who and why sends that messages and how it’s possible they pass DKIM while all Sendgrid API are deleted and Gmail account is isolated from 3rd parties.

Email messages are not always delivered instantaneously, and can be delayed, such as for example due to network connectivity issues.

A typical (default) set up is to re-try a message’s delivery for 120 hours (5 days).

So if you’re leaving an email provider, and send your latest email through them on e.g. April 8, I wouldn’t necessarily expect it to be completely gone from your DMARC reports for the first 120 hours (5 days), so eventually until April 13 or April 14 (depending on time zone, what exact time the last message was sent, … and so forth).

Deleting stuff in your (previous) email provider isn’t alone what you need to do.

You should also clean up your own DNS, if you’re leaving them.

You likely still have one or more _domainkey record(s) in your DNS.

People are unfortunately often forgetting to clean up their own DNS, when they leave one (or more) provider(s) behind, which can cause situations like this.

Alternatively, there may be the chance that you have set up a connection to some system at a third party, where this third party is using them to send messages on your behalf.

If you are clicking the individual name under “Source” in this list, you should be taken to a new page where you see the IP address(es) from that specific source.

→ What exact IP address do you see there?

Together with that, and to dig even further in to your issue, can you possibly share:

  1. What is your domain?

  2. What DNS record(s) do you have, of the type CNAME, NS, or TXT, where the name contains “_domainkey”?

  3. What DNS record(s) do you have, of the type TXT, where the content starts with “v=spf1”?

  4. What DNS record(s) do you have, of any type, where the content contains “sendgrid”?

1 Like

I have checked today and there is the same issue. So delays in delivery is not a reason.

I have also restore settings now as we have to send our newsletter today.

But on the time when I was testing it I have revoked Sendgrid API keys thus even if Sendgrid _domainkey record left it cannot be used if password is wrong.

We didn’t share any data with 3rd party except gmass plugin what was disabled for test purposes.

For SPF there is only IP address allowed to send from behalf of our domain.

After further discussions with our team and 3rd party specialists we strongly believe this is a glitch on Cloudflare side thuse we are not lookg for alternative solution to manager DMARC.

I am sorry for my typos I was rushing with typing.

We are now looking for alternative solution to manager DMARC.

If move our focus out of Sendgrid the most important is why and how Google, Namechip and Zoho was able to send emails and pass DKIM as we never shared this data with them.

Revoking API keys or whatever isn’t enough to invalidate email authentication.

You MUST delete those email authentication record(s) from your DNS.

I strongly believe you, and your team, as well as the 3rd party specialists you refer to, are misunderstanding DMARC, and how exactly it works.

Post the data I asked for above, and we will be able to dig in to the situation together.

As you see these three (Google, Namecheap and Zoho) are 100% DKIM aligned, it means that there is still valid email authentication, with a DKIM key, that would, as mentioned with point #2 above, be with a record whose name in your DNS contains “_domainkey”.

The failed SPF alignment for those three sources points towards the fact that the message(s) they refer to have been forwarded, by those mentioned organisations, to somewhere else, and that this somewhere else (recipient after forwarding) produced DMARC reports, that they sent your way.

1 Like

I also was thinking that these 3 might be a forwarded mail. When Gmail or Yahoo forward an email the end recipient receives it with DKIM verified? Because original email had DKIM verified, right?

1 Like

As long as the components of the message that were used to generate the signature are unaltered, the DKIM signature will verify.

I have the same issue. I see some sources in my report that shouldn’t be there or more specifically that I do not recognise. I am trying to get to the bottom but have not found anything useful so far either. I also think it might be something related to forwarded emails. Did you managed to get to the bottom of this, since last time you posted, please?

Can you possibly elaborate on the results of the DMARC pass, SPF aligned and DKIM aligned columns?

Perhaps a screenshot?

2 Likes

Hello! Yes, sure.

(Example 1)
Here is the record that I recognise and it’s normal. It’s from our friend SendGrid and envelop domain is from my newsletter provider ConvertKit.

(Example 2)
But then I’m finding records like this one where source is listed as Yahoo. Note the envelop domain is still a “correct” one although ConvertKit wasn’t able to explain this so far.

(Example 3)
And then I see records that are completely odd like this one

This started to happen virtually from day 1, so I doubt it’s spoofing and rather either a bug in Cloudflare system (I doubt it, either?) or just we(?) don’t understand what those reports mean and how they’re getting generated.
Any insight would be greatly appreciated.

As you indicate that you’re sending through ConvertKit, that appears to be using SendGrid as their outbound mail provider, things do indeed look normal in that regards.

I do however find it hard to concur with this statement though.

SendGrid has for more than a decade turned the blind eye to spam, phishing and malware egressing from their network, and never done anything to e.g. terminate bad customers.

The Swiss Government Computer Emergency Response Team did for example comment the role of those email provides, with a very specific mention of SendGrid over here:

“SpamGrid”, as it is referred to in various email communities, wouldn’t be a provider I put my faith in…

It shares the patterns with the “good” messages above, and therefore, I wouldn’t worry that much here.

As it is passing DKIM, you have two options:

  1. It was a legitimate message from you, which was forwarded.

  2. There may be a compromise somewhere.
    → The actual sender had access to one or more _domainkey record(s) on your domain, and could send out messages that way.

What can be a little bit interesting though, is that the reports from Yahoo claim that the message(s) were delivered to Yahoo by the two IP addresses 77.238.179.146 and 87.248.110.83.

These two IP addresses are actually two of Yahoo’s own IP addresses.

Normally, you would be performing sanity checks on the first IP address (in reverse order), that you do not trust.

So it could be a configuration mistake on Yahoo’s end, that they haven’t marked their own IP address(es) 77.238.179.146 and 87.248.110.83 as trusted in their systems, and such, some sanity “checks” may therefore be applied incorrectly.

Yahoo may have received the message on one of their servers (e.g. 77.238.179.146), and then have passed the message on to one of their other servers, within their own network, and since this other server was not configured to be trusting 77.238.179.146, it did believe the actual message source was 77.238.179.146, and not the actual source that delivered the message to 77.238.179.146.

In short words, it seems to be some sort of “internal” forwarding within Yahoo, where the source IP address(es) got messed up somewhere.

I would lean towards #1 above, by considering the message to have ben forwarded, and not be worried here.

I understand that it may look odd.

In terms of the email authentication you have set up for your domain, it is actually spoofing here.

Cloudflare retrieved a report from Microsoft (Office 365, Outlook, Hotmail, …), that they received one (1) message delivered to them from the IP address 3.231.237.226.

That IP address (and several other IP addresses) have the Reverse DNS (PTR) record set as ipw-outbound.inkyphishfence.com, however, the set up they have isn’t very consistent though.

It seems to be related to INKY (https://www.inky.com/), which appears to be a platform that aims to assist you in training your user(s) to understand to stay away from phishing messages.

I don’t know if you know of this platform, or have heard about it before?

Some further insights:

Your DMARC policy is just none, you would want to upgrade that one to reject, if you want to make sure that Google, Yahoo, and several others are refusing deliveries impersonating your domain name, which would include the one from Example 3.

Even when such messages are rejected though, you will STILL be seeing the information in the DMARC reports, exactly as you did in Example 3.

In addition, -

Your SPF record is currently:

v=spf1 include:sendgrid.net ~all include:spf.tutanota.de -all
                            ^^^^

You need to remove that “~all” from the middle.

Final result:

v=spf1 include:sendgrid.net include:spf.tutanota.de -all

If you are not using Tutanota any more, I would also suggest removing them from your SPF.

3 Likes

Hi @DarkDeviL,
Wow, this is a very helpful and comprehensive response. Thank you!

Thank you for confirming this. It is exactly what I was suspecting based on other records similar to this. Although, I would not discount a misconfiguration on Yahoo’s end either. But I’ll leave it up to them now and not pursue further.

:scream: I just hope you meant it in literal technical terms and not an actual malicious actor… I don’t recall hearing about inky specifically but I’m familiar with services of this sort. Several of my newsletter subscribers (companies, gov agencies) are using various email security solutions, including ones that are clicking in every single link in my newsletter, which is annoying when analysing subscriber engagement stats. They can also replace the links in the original email before placing them in the recipient’s inbox, but I’m not sure about the exact mechanics of that process (eg. is forwarding involved).

Yes, I changed it from p=quarantine for the time of troubleshooting this and one other deliverability issue with ConvertKit support. I will certainly change it back.

Once again, thank you for pointing this out. Much appreciated!

I do use Tutanota (or Tuta as they like to be called now) and so this is why I was expecting to only see SendGrid (spamgrid :face_with_hand_over_mouth: ) and kyberio GmbH in my reports. Now, as you clarified, I appreciate that I will also see the forwarded emails in this report.

Final question if I may. Is the spoofy-looking record (might be) related to Inky being used by one of my subscribers? Did I get the right?

Many thanks again for such a comprehensive answer!

1 Like

You’re welcome!

Happy to hear that my assistance and guidance is well appreciated!

It looks that way, especially when comparing it to e.g. Example 1, so yeah, chasing Example 2 in any way might only just be a waste of time. :wink:

It could be both ways actually.

Initially, when I was looking at INKY, I was thinking one of these companies that (hopefully, on your own request, or by the request of one of your employees) would be sending out phishing messages targetting your own organisation’s employees, in order to test your employees and see whether any of them would fall for the phishing message(s).

… However:

A such kind of “middleman” set up that replaces links, scans for viruses and so on, could technically be set up in a way, that would be quite equal to forwarding, and where the final destination (e.g. Microsoft Outlook here) will sending DMARC reports like you see.

With the replacements of e.g. links, you’re already that far where e.g. DKIM signatures typically wouldn’t be able to survive, due to the modified message, which could explain the failure there.

Their “Inbound Mail Protection” could indeed seem to be a such product, that may be rewriting messages in a such “severe” way, so that the DKIM signature (if existing) cannot be verified any more.

So it could technically be one of your recipients, that are using the products of INKY in a legitimate way.

“Unfortunately”, DMARC won’t be able to see through such (potential) legitimate use cases, and organisations that send out the aggregate reports will always notify you every time their system sees a message “pretending” to be from your domain name.

Hehe, maybe it would be time for them to “rebrand” their domain name, too. :thinking:

You can try looking up each of your subscriber’s domain names, and see if there is something pointing towards INKY in each domain name’s MX record(s).

It may not actually give you anything conclusive though, as a potential legitimate connection to INKY would be done after the MX record(s).

You’re welcome! :slight_smile:

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.