This week, MailChannels will enforce Domain Lockdown on legacy Workers users

When MailChannels rolled out Domain Lockdown on June 20th, we exempted Workers users who had already sent emails through our API to give them time to implement the new domain verification feature. Later this week, we will begin enforcing Domain Lockdown on these legacy users.

Here’s how to become compliant with Domain Lockdown in three steps:

  1. Find the CF-Worker header associated with your Worker, a header that identifies the zone in which your Worker lives. This header is included in every fetch() request by Cloudflare, and it’s how MailChannels identifies you to establish your reputation and track any abusive email you send. If you need help finding this value, send a GET fetch() from your worker to https://postman-echo.com/get, and it will return JSON containing your request headers like this:
export default {
  async fetch(request, env, ctx) {
    const url = "postman-echo.com/";

    async function gatherResponse(response) {
      const { headers } = response;
      const contentType = headers.get("content-type") || "";
      if (contentType.includes("application/json")) {
        return JSON.stringify(await response.json());
      }
      return response.text();
    }

    const init = {
      headers: {
        "content-type": "application/json;charset=UTF-8",
      },
    };

    const response = await fetch(url, init);
    const results = await gatherResponse(response);
    return new Response(results, init);
  },
};

This will return something like the following:

{
  "args": {},
  "headers": {
    ...
    "cf-worker": "myzone.workers.dev",
    "accept": "*/*"
  },
  "url": "https://postman-echo.com/get"
}

Look for the cf-worker line in the JSON response. The value of that key is your CF-Worker id. Keep in mind that the CF-Worker header does not identify the subdomain serving your Worker; rather, it’s the zone. In most cases, this means your domain name or your workers.dev string, which can be found on the dashboard for Workers.

  1. Create a Domain Lockdown record for each domain from which you want to send email. These TXT records specify your CF-Worker identity, telling MailChannels that you authorize that CF-Worker to send email from the domain. Following the fetch() code from above, if your domain is example.com and your CF-Worker id is myzone.workers.dev, then you would need to add a TXT record like this:
_mailchannels.example.com TXT "v=mc1 cfid=myzone.workers.dev

If you send out of multiple domains, each one will need its own _mailchannels record. Subdomains will also each need their own record.

  1. If for some reason we can’t find the TXT record or if your cfid field is wrong, you will get an error message from our API such as the following:
5.7.1 This sender is not authorized to send from mailchannels.com. See https://bit.ly/domain-lockdown. cfid=example.workers.dev

As you can see, our rejection error message gives you the cfid value to include in your Domain Lockdown record.

For further assistance, please reply here or contact our support team.

5 Likes

This is good to see, thank you.

What’s the plan for addressing the real issue here, where users can still spoof tonnes of your customers for the low price of $80?

UPDATE (Aug 13 2023): Two days after my DEFCON 31 talk, MailChannels silently decided to require a Domain Lockdown Record in order to send emails from Cloudflare Workers meaning this code doesn’t work anymore. However, because they just addressed a “symptom” and not the underlying issue (lack of sender idenitity verification) anyone can still signup on their website (80$) and use their “normal” SMTP relay to spoof all of their customer domains :man_shrugging:t2:

5 Likes

I appreciate your concern. MailChannels’ primary business is sending emails on behalf of web hosting providers. In the hosting realm, without MailChannels, there is very little filtering of outgoing emails for spam and phishing attacks, and domain verification is spotty to non-existent. MailChannels greatly improves things by implementing a huge variety of abuse detection measures. Over the past week, we blocked between 25 and 50% of all messages submitted to our service because they did not meet our high standards:

After blocking this stuff at the front door, our delivery success rate to the internet was about 80%:

Of the messages that did not deliver successfully, a median of 9% were rejected because of recipient address problems. Only 3% were rejected because the receiver did not like the message content.

We obsessively work to improve our filtering accuracy, tracking dozens of metrics so that we know how well we are at blocking spam and phishing to maintain a good reputation with email receivers. The above screenshot (with some redactions) was pasted from the dashboard in Datadog that tracks these metrics. If filtering metrics go outside of our expectations, alerts are generated, and our ops team investigates, just as they would investigate any service issue.

Reprocessing the accuracy data with some help from ChatGPT Code Interpreter, our filtering accuracy has been between 97-99% over the past 30 days. That is to say, 97-99% of the time, we reject the right messages. The huge challenge of spam filtering is to balance spam rejection against rejecting legitimate messages. Getting that formula right is essential to operate the kind of service MailChannels offers.

For the data nerds out there, much of the magic in the box at MailChannels is because of a very fast time-series reputation database that tracks per-sender reputation second by second. For instance, if you spool up a Cloudflare Worker and start sending spam, the first few messages might get delivered, but once we start seeing delivery failures and other signals associated with your Worker, automated rate limits will kick in. The total feedback loop happens in under one second and the reputation system supports tens of thousands of queries per second with queries taking much less than 1ms to resolve.

Fun fact: We tried replacing our time-series database with something off the shelf. We played with Clickhouse, Timescale, and a few others, but found that nothing met our needs better than the special-case software we built for this purpose many years ago. I’m sure giant internet companies and high-frequency trading outfits have much better things available, but for our specific purpose, we really need something special.

Regarding domain spoofing, we absolutely want to reduce or eliminate the abusive spoofing of emails on our platform. However, the nature of our service as a general email delivery endpoint for the hosting industry makes some straightforward-sounding domain verification controls hard to implement in practice. I’ll try to explain why this is the case.

One of the very common things people do with web hosting and email accounts is to forward emails to, say, a Gmail account. Some hosting providers do the right thing and wrap these emails using the Sender Rewriting Scheme (SRS), but many do not. As a result, many emails flow through MailChannels from major domains because it’s being legitimately forwarded. We use heuristics and machine learning to identify forwarded traffic, but a strong signal of forwarding is not always available.

Every major email provider, including Microsoft and Google, struggle to stop outgoing email abuse on their platform. A vast amount of spam and phishing originates from the big cloud platforms. Spam and phishing is also rampant on SendGrid and other transactional platforms. Even if you provide a robust domain verification requirement, when you operate at scale, you also have to have robust anti-abuse mechanisms in place to prevent email abuse

Even if there was a practical way for us to require tens of millions of users behind hundreds of web hosting companies to add Domain Lockdown records, forwarded traffic would still be blocked because people aren’t able to create records for domains they don’t own. Yet forwarding is quite a legitimate thing to do.

The proper solution here is for the email authentication standards to advance. The group developing the next version of the DMARC standard has been actively discussing what to do with SPF in particular, including the potential idea of removing SPF from the next version of DMARC.

As a final note, I want to say we appreciate criticism of our anti-abuse measures. It helps us improve our email filtering, which leads to our IP addresses having an even better reputation. Ultimately, that helps us deliver mail more reliably and to land more of our customers’ messages into inboxes.

3 Likes

What’s the plan for addressing the real issue here, where users can still spoof tonnes of your customers for the low price of $80?

We have no plan to force Domain Lockdown onto tens of millions of web hosting customers spread across hundreds of web hosting providers worldwide who send email through MailChannels, most of whom have no idea what “SPF” even means, let alone the technical knowhow to add a TXT record. However, we are working on some automations to help our customers roll Domain Lockdown out to their users.

As a first step, we are adding Domain Lockdown support to our cPanel plugin, which will make it easy for millions of web hosting users and their hosting providers to add a Domain Lockdown record with the push of a button. Because Domain Lockdown works in the DNS, our primary tool is to spread the knowledge of the capability to our customers, who can then automate Domain Lockdown roll-outs as they see fit.

We are also closely following the work of the people crafting the next version of DMARC. DMARC 1.0 was (in my opinion) lacking in its design to some degree. Specifically, if you have both an SPF record and a DKIM record, emails can pass DMARC verification if either of these mechanisms pass. In our opinion, DMARC 2.0 should either get rid of SPF entirely, or should provide a mechanism allowing domain owners to express that DKIM signing is required regardless of SPF status.

In practice, many email receivers examine the DKIM and SPF results and will tilt a filtering decision more heavily toward “reject” if they receive a message that is SPF-aligned but lacks a valid DKIM signature for a domain that typically signs its messages. These behind-the-scenes anti-abuse measures are often behind the black box of services like Gmail, but are very much discussed in industry forums such as M3AAWG.

In my opinion, SPF is trusted a bit too much, particularly given how DMARC works. For instance, Cloudflare’s SPF record includes the SPF records of Google, MailChimp, Zendesk, StatusPage, and Salesforce:

v=spf1 ip4:199.15.212.0/22 ip4:173.245.48.0/20 include:_spf.google.com include:spf1.mcsv.net include:spf.mandrillapp.com include:mail.zendesk.com include:stspg-customer.com include:_spf.salesforce.com -all

By including these records - as many companies do - Cloudflare is exposing itself to the risk that anyone using any IP address authorized by one of these services can send email from cloudflare.com. All it takes is one successful breach of the mechanisms used to vet domain ownership at one of these providers, and all bets are off. The messages will pass DMARC verification, which is doubtless not what the domain owner intended.

For this reason, I hope that the DMARC working group provides a way to remove SPF from DMARC alignment checks without forcing domain owners to delete their SPF record, which remains somewhat valuable in helping to detect abuse. Following their email discussions over the last few months, it seems likely that the next DMARC will enable this, among other improvements.

2 Likes

Looking forward to your BlackHat talk about this!

Until then, MailChannels was the only one with a vulnerability. Stop trying to hide it.

1 Like

These numbers are great, but none of the emails sent in testing this issue were ever blocked, including from high profile customers such as boston.gov.

Sure, that’s a great long-term goal. But in the short-term, you need to do what’s right and proactively restrict customers from sending from other’s domains. Basic API verification, unique SPF records, etc. are just some example ways competing services implement this.

I really hope you reconsider this approach. You need to be proactive here and educate them. It’s your responsibility to do this when your marketing talks so much about “secure”, and “trust”, and “improv[ing] your security”. What are you going to do when a large profile government domain starts being used for phishing or other acts? Wait for them report it and act afterwards? Maybe your anti-abuse mechanisms pick it up (unlikely from everything that’s been discussed/seen thus far)? At that point the damage might already be done.

This is not a great argument. It’s like saying, “someone could break into my house and steal my belongings, so why would I bother putting locks on my door”. It’s one step in a bigger security sandwich, sure, but dismissing this is really disappointing.

Sure, me too. We all hope for better security implementations of everything we use, but you can do better right now and implement the most basic of protections against this. Please stop deflecting with marketing and sales stuff. Please do better.

6 Likes

This is once again like saying, “in this neighbourhood crime rates are low, so you’re probably good to go without locks on your door”. That’s only (kinda) valid until someone gets broken into, people learn about the neighbourhood, and now everyone has an issue. Also forgive me if I don’t trust your own stats for “significant vector of abuse” measuring. It’s been shown by multiple people that these emails can be sent seemingly either without your knowledge, or worse if you do know, your care.

Okay, and I added cameras to my house to detect when people come in through my unlocked door so I can sound alarms. At that point, it might be too late, or maybe they wear a disguise, or sneak in through the window. All of these things you have are great IN ADDITION to basic API protections like sender verification, not instead of.


Please, stop deflecting and throwing around whataboutisms. Own this issue, alert your customers, and implement basic protections to prevent this. No one is asking you to solve the bigger problem of email spam; you’re simply being asked to protect your own customers in some of the most basic and trivial ways.

4 Likes

6888295605f63b49b30220ae703dfa47c1c89230b42628d655ee5a9e6a0d18e2

I can see why I’ve never bothered with MailChannels.

@ksimpson Can you please acknowledge publicly the severity of this issue?

And in addition, if and when you plan to roll out required sender domain verification for every outgoing email?

2 Likes

Thanks for the information. To make sure I understand, you DO intend to require sender domain verification for all outgoing email, there’s just not a timeline you can share yet?

And if so, until that timeline is hit, will you be endeavoring to educate your customers about this issue, and any steps they can take to mitigate it?

1 Like

Good to hear. Please keep us updated on the progress of this and when you’ll be enforcing sender domain verification for all of your customers.

1 Like

I added the record and still getting the same error
any help would be appreciated.


I host this worker on custom url of my domain
/cf-mail.php

Any updates on rolling this out, and the educational campaign?

How long do you cache TXT records for?

Also, 500 Internal Server Error is an inappropriate status code, you should probably use 403.

Any updates @ksimpson?

With Gmail introduces new requirements to fight spam via Your 2024 guide to Google and Yahoo’s new requirements for email senders | Postmark, it sounds like any emails that aren’t signed with DKIM are going to start landing in spam for popular email providers.

What’s the plan to resolve this @ksimpson? Are you going to finally do what everyone has been suggesting and implement some real and required sender domain verification?

Any updates yet @ksimpson ?