This week, MailChannels will enforce Domain Lockdown on legacy Workers users

When MailChannels rolled out Domain Lockdown on June 20th, we exempted Workers users who had already sent emails through our API to give them time to implement the new domain verification feature. Later this week, we will begin enforcing Domain Lockdown on these legacy users.

Here’s how to become compliant with Domain Lockdown in three steps:

  1. Find the CF-Worker header associated with your Worker, a header that identifies the zone in which your Worker lives. This header is included in every fetch() request by Cloudflare, and it’s how MailChannels identifies you to establish your reputation and track any abusive email you send. If you need help finding this value, send a GET fetch() from your worker to https://postman-echo.com/get, and it will return JSON containing your request headers like this:
export default {
  async fetch(request, env, ctx) {
    const url = "postman-echo.com/";

    async function gatherResponse(response) {
      const { headers } = response;
      const contentType = headers.get("content-type") || "";
      if (contentType.includes("application/json")) {
        return JSON.stringify(await response.json());
      }
      return response.text();
    }

    const init = {
      headers: {
        "content-type": "application/json;charset=UTF-8",
      },
    };

    const response = await fetch(url, init);
    const results = await gatherResponse(response);
    return new Response(results, init);
  },
};

This will return something like the following:

{
  "args": {},
  "headers": {
    ...
    "cf-worker": "myzone.workers.dev",
    "accept": "*/*"
  },
  "url": "https://postman-echo.com/get"
}

Look for the cf-worker line in the JSON response. The value of that key is your CF-Worker id. Keep in mind that the CF-Worker header does not identify the subdomain serving your Worker; rather, it’s the zone. In most cases, this means your domain name or your workers.dev string, which can be found on the dashboard for Workers.

  1. Create a Domain Lockdown record for each domain from which you want to send email. These TXT records specify your CF-Worker identity, telling MailChannels that you authorize that CF-Worker to send email from the domain. Following the fetch() code from above, if your domain is example.com and your CF-Worker id is myzone.workers.dev, then you would need to add a TXT record like this:
_mailchannels.example.com TXT "v=mc1 cfid=myzone.workers.dev

If you send out of multiple domains, each one will need its own _mailchannels record. Subdomains will also each need their own record.

  1. If for some reason we can’t find the TXT record or if your cfid field is wrong, you will get an error message from our API such as the following:
5.7.1 This sender is not authorized to send from mailchannels.com. See https://bit.ly/domain-lockdown. cfid=example.workers.dev

As you can see, our rejection error message gives you the cfid value to include in your Domain Lockdown record.

For further assistance, please reply here or contact our support team.

5 Likes

This is good to see, thank you.

What’s the plan for addressing the real issue here, where users can still spoof tonnes of your customers for the low price of $80?

UPDATE (Aug 13 2023): Two days after my DEFCON 31 talk, MailChannels silently decided to require a Domain Lockdown Record in order to send emails from Cloudflare Workers meaning this code doesn’t work anymore. However, because they just addressed a “symptom” and not the underlying issue (lack of sender idenitity verification) anyone can still signup on their website (80$) and use their “normal” SMTP relay to spoof all of their customer domains :man_shrugging:t2:

5 Likes

I appreciate your concern. MailChannels’ primary business is sending emails on behalf of web hosting providers. In the hosting realm, without MailChannels, there is very little filtering of outgoing emails for spam and phishing attacks, and domain verification is spotty to non-existent. MailChannels greatly improves things by implementing a huge variety of abuse detection measures. Over the past week, we blocked between 25 and 50% of all messages submitted to our service because they did not meet our high standards:

After blocking this stuff at the front door, our delivery success rate to the internet was about 80%:

Of the messages that did not deliver successfully, a median of 9% were rejected because of recipient address problems. Only 3% were rejected because the receiver did not like the message content.

We obsessively work to improve our filtering accuracy, tracking dozens of metrics so that we know how well we are at blocking spam and phishing to maintain a good reputation with email receivers. The above screenshot (with some redactions) was pasted from the dashboard in Datadog that tracks these metrics. If filtering metrics go outside of our expectations, alerts are generated, and our ops team investigates, just as they would investigate any service issue.

Reprocessing the accuracy data with some help from ChatGPT Code Interpreter, our filtering accuracy has been between 97-99% over the past 30 days. That is to say, 97-99% of the time, we reject the right messages. The huge challenge of spam filtering is to balance spam rejection against rejecting legitimate messages. Getting that formula right is essential to operate the kind of service MailChannels offers.

For the data nerds out there, much of the magic in the box at MailChannels is because of a very fast time-series reputation database that tracks per-sender reputation second by second. For instance, if you spool up a Cloudflare Worker and start sending spam, the first few messages might get delivered, but once we start seeing delivery failures and other signals associated with your Worker, automated rate limits will kick in. The total feedback loop happens in under one second and the reputation system supports tens of thousands of queries per second with queries taking much less than 1ms to resolve.

Fun fact: We tried replacing our time-series database with something off the shelf. We played with Clickhouse, Timescale, and a few others, but found that nothing met our needs better than the special-case software we built for this purpose many years ago. I’m sure giant internet companies and high-frequency trading outfits have much better things available, but for our specific purpose, we really need something special.

Regarding domain spoofing, we absolutely want to reduce or eliminate the abusive spoofing of emails on our platform. However, the nature of our service as a general email delivery endpoint for the hosting industry makes some straightforward-sounding domain verification controls hard to implement in practice. I’ll try to explain why this is the case.

One of the very common things people do with web hosting and email accounts is to forward emails to, say, a Gmail account. Some hosting providers do the right thing and wrap these emails using the Sender Rewriting Scheme (SRS), but many do not. As a result, many emails flow through MailChannels from major domains because it’s being legitimately forwarded. We use heuristics and machine learning to identify forwarded traffic, but a strong signal of forwarding is not always available.

Every major email provider, including Microsoft and Google, struggle to stop outgoing email abuse on their platform. A vast amount of spam and phishing originates from the big cloud platforms. Spam and phishing is also rampant on SendGrid and other transactional platforms. Even if you provide a robust domain verification requirement, when you operate at scale, you also have to have robust anti-abuse mechanisms in place to prevent email abuse

Even if there was a practical way for us to require tens of millions of users behind hundreds of web hosting companies to add Domain Lockdown records, forwarded traffic would still be blocked because people aren’t able to create records for domains they don’t own. Yet forwarding is quite a legitimate thing to do.

The proper solution here is for the email authentication standards to advance. The group developing the next version of the DMARC standard has been actively discussing what to do with SPF in particular, including the potential idea of removing SPF from the next version of DMARC.

As a final note, I want to say we appreciate criticism of our anti-abuse measures. It helps us improve our email filtering, which leads to our IP addresses having an even better reputation. Ultimately, that helps us deliver mail more reliably and to land more of our customers’ messages into inboxes.

3 Likes

What’s the plan for addressing the real issue here, where users can still spoof tonnes of your customers for the low price of $80?

We have no plan to force Domain Lockdown onto tens of millions of web hosting customers spread across hundreds of web hosting providers worldwide who send email through MailChannels, most of whom have no idea what “SPF” even means, let alone the technical knowhow to add a TXT record. However, we are working on some automations to help our customers roll Domain Lockdown out to their users.

As a first step, we are adding Domain Lockdown support to our cPanel plugin, which will make it easy for millions of web hosting users and their hosting providers to add a Domain Lockdown record with the push of a button. Because Domain Lockdown works in the DNS, our primary tool is to spread the knowledge of the capability to our customers, who can then automate Domain Lockdown roll-outs as they see fit.

We are also closely following the work of the people crafting the next version of DMARC. DMARC 1.0 was (in my opinion) lacking in its design to some degree. Specifically, if you have both an SPF record and a DKIM record, emails can pass DMARC verification if either of these mechanisms pass. In our opinion, DMARC 2.0 should either get rid of SPF entirely, or should provide a mechanism allowing domain owners to express that DKIM signing is required regardless of SPF status.

In practice, many email receivers examine the DKIM and SPF results and will tilt a filtering decision more heavily toward “reject” if they receive a message that is SPF-aligned but lacks a valid DKIM signature for a domain that typically signs its messages. These behind-the-scenes anti-abuse measures are often behind the black box of services like Gmail, but are very much discussed in industry forums such as M3AAWG.

In my opinion, SPF is trusted a bit too much, particularly given how DMARC works. For instance, Cloudflare’s SPF record includes the SPF records of Google, MailChimp, Zendesk, StatusPage, and Salesforce:

v=spf1 ip4:199.15.212.0/22 ip4:173.245.48.0/20 include:_spf.google.com include:spf1.mcsv.net include:spf.mandrillapp.com include:mail.zendesk.com include:stspg-customer.com include:_spf.salesforce.com -all

By including these records - as many companies do - Cloudflare is exposing itself to the risk that anyone using any IP address authorized by one of these services can send email from cloudflare.com. All it takes is one successful breach of the mechanisms used to vet domain ownership at one of these providers, and all bets are off. The messages will pass DMARC verification, which is doubtless not what the domain owner intended.

For this reason, I hope that the DMARC working group provides a way to remove SPF from DMARC alignment checks without forcing domain owners to delete their SPF record, which remains somewhat valuable in helping to detect abuse. Following their email discussions over the last few months, it seems likely that the next DMARC will enable this, among other improvements.

2 Likes

Looking forward to your BlackHat talk about this!

Until then, MailChannels was the only one with a vulnerability. Stop trying to hide it.

1 Like

These numbers are great, but none of the emails sent in testing this issue were ever blocked, including from high profile customers such as boston.gov.

Sure, that’s a great long-term goal. But in the short-term, you need to do what’s right and proactively restrict customers from sending from other’s domains. Basic API verification, unique SPF records, etc. are just some example ways competing services implement this.

I really hope you reconsider this approach. You need to be proactive here and educate them. It’s your responsibility to do this when your marketing talks so much about “secure”, and “trust”, and “improv[ing] your security”. What are you going to do when a large profile government domain starts being used for phishing or other acts? Wait for them report it and act afterwards? Maybe your anti-abuse mechanisms pick it up (unlikely from everything that’s been discussed/seen thus far)? At that point the damage might already be done.

This is not a great argument. It’s like saying, “someone could break into my house and steal my belongings, so why would I bother putting locks on my door”. It’s one step in a bigger security sandwich, sure, but dismissing this is really disappointing.

Sure, me too. We all hope for better security implementations of everything we use, but you can do better right now and implement the most basic of protections against this. Please stop deflecting with marketing and sales stuff. Please do better.

6 Likes

Looking forward to your BlackHat talk about this!
Until then, MailChannels was the only one with a vulnerability. Stop trying to hide it.

Respectfully, it’s not a vulnerability. The email RFC defines setting the envelope and header addresses (“return path”) as a feature, not a bug (emphasis is mine):

Efforts to make it more difficult for users to set envelope return path and header “From” fields to point to valid addresses other than their own are largely misguided: they frustrate legitimate applications in which mail is sent by one user on behalf of another, in which error (or normal) replies should be directed to a special address, or in which a single message is sent to multiple recipients on different hosts.

This specification does not further address the authentication issues associated with SMTP other than to advocate that useful functionality not be disabled in the hope of providing some small margin of protection against a user who is trying to fake mail.

A real vulnerability exists if the email recipient interprets a positive SPF result as a guarantee of domain authenticity. SPF is more properly understood as a mechanism that allows you to express a negative opinion of IPs that are not expressly authorized in your SPF record. In other words, SPF lets you say, “Here are the IPs where my email might originate; anything outside of these ranges is probably not me.”

Major email receivers do not interpret an SPF pass alone as an expression of domain authenticity. Receivers sometimes make this mistake anyway, such as when Google showed the UPS logo against messages that were expressly not from UPS. Such is the complexity of domain authentication that even a major player like Google can screw it up from time to time.

I really hope you reconsider this approach. You need to be proactive here and educate them. It’s your responsibility to do this when your marketing talks so much about “secure”, and “trust”, and “improv[ing] your security”. What are you going to do when a large profile government domain starts being used for phishing or other acts? Wait for them report it and act afterwards? Maybe your anti-abuse mechanisms pick it up (unlikely from everything that’s been discussed/seen thus far)? At that point the damage might already be done.

When we look at our traffic, we don’t see domain impersonation as a significant vector of abuse. Attackers have plenty of other ways to attack victims, such as using lookalike domains and making the content of messages impersonate the look and feel of a brand. In fact, the vast majority of attacks on major brands don’t even try to impersonate the brand’s domain. They send from any domain they can get their hands on - including Gmail accounts.

What matters is that the message content looks plausibly like it might have come from the brand. This is why we built a machine-learning model that recognizes brand impersonation based on visual appearance.

Indeed, Cloudflare Area 1 Security states right at the top of the product page that they block phishing campaigns “even when messages are from ‘trusted’ senders or domains.” The whole concept of “Zero Trust” now stretches to include email, because phishing attacks are increasingly originating from trusted, verified domains and accounts.

By rolling out Domain Lockdown as a feature of our cPanel plugin, we are taking a step toward helping eliminate the very narrow issue of domain impersonation, but we are not ignoring the rest of the stack, which is in reality far more important.

This is once again like saying, “in this neighbourhood crime rates are low, so you’re probably good to go without locks on your door”. That’s only (kinda) valid until someone gets broken into, people learn about the neighbourhood, and now everyone has an issue. Also forgive me if I don’t trust your own stats for “significant vector of abuse” measuring. It’s been shown by multiple people that these emails can be sent seemingly either without your knowledge, or worse if you do know, your care.

Okay, and I added cameras to my house to detect when people come in through my unlocked door so I can sound alarms. At that point, it might be too late, or maybe they wear a disguise, or sneak in through the window. All of these things you have are great IN ADDITION to basic API protections like sender verification, not instead of.


Please, stop deflecting and throwing around whataboutisms. Own this issue, alert your customers, and implement basic protections to prevent this. No one is asking you to solve the bigger problem of email spam; you’re simply being asked to protect your own customers in some of the most basic and trivial ways.

4 Likes

6888295605f63b49b30220ae703dfa47c1c89230b42628d655ee5a9e6a0d18e2

I can see why I’ve never bothered with MailChannels.

@ksimpson Can you please acknowledge publicly the severity of this issue?

And in addition, if and when you plan to roll out required sender domain verification for every outgoing email?

2 Likes

@cherryjimbo, while we believe our current anti-spam and anti-phishing measures are quite effective, as evidenced by our high delivery rates and low complaint rates, we always strive to do better. Requiring sender domain verification for all outgoing email is a complex undertaking given the nature of our service, but we are actively exploring ways to make progress here, starting with a new cPanel plugin that will allow millions of web hosting customers to start securing their domains with Domain Lockdown.

While I’m sure you already know this, I do want to highlight that new Cloudflare Workers users are already protected because we require a Domain Lockdown record to be in place before you can send mail through our API. And by the end of this week, even legacy Workers (those that sent emails before June 20th) will be forced to have a Lockdown record.

We appreciate the Cloudflare developer community pushing us on this issue. Rest assured, we are taking your feedback seriously. If you ever wish to reach out, I’m on Discord as @ttul and do my best to stay in touch with the Workers channels as well.

Thanks for the information. To make sure I understand, you DO intend to require sender domain verification for all outgoing email, there’s just not a timeline you can share yet?

And if so, until that timeline is hit, will you be endeavoring to educate your customers about this issue, and any steps they can take to mitigate it?

1 Like

Thank you for the follow-up. To clarify, we are actively exploring ways to expand sender domain verification more broadly across our service, but do not have a specific timeline we can commit to yet. This is a complex challenge and we want to ensure we develop the right solution.

In the meantime, we agree on the importance of educating customers. We are preparing an informational campaign that will guide customers on steps they can take to lock down their customer domains if they wish to do so, including using our new cPanel plugin as an option. We will also inform them of the potential risks of locking down their domains. For instance, if a domain is locked to a particular hosting service, then that domain will not be able to send replies on a mailing list hosted by a different hosting service.

While not foolproof, these proactive measures will help to prevent trivial spoofing of customer domains on our platform. We also plan to highlight that no single protection is sufficient and that we encourage the use of tools like DMARC and DKIM, which are well-accepted by the global email community. As a further note, we have been a supporting member of the Messaging, Malware, Mobile Anti-Abuse Working Group (M3AAWG) since 2007 and continue participating in this global organization’s efforts to coordinate the fight against spam, phishing, and malware.

I appreciate you keeping us focused on this important issue. We are committed to enhancing protections for our customers while balancing the impact that significant changes to email authentication practices may have. Please keep the feedback coming, and know we are working diligently behind the scenes to progress sender verification capabilities across our platform.

Good to hear. Please keep us updated on the progress of this and when you’ll be enforcing sender domain verification for all of your customers.

1 Like

I added the record and still getting the same error
any help would be appreciated.


I host this worker on custom url of my domain
/cf-mail.php

The error message gives you the cfid field that must be in your lockdown record. Make sure that the lockdown record is of the form

_mailchannels.yourdomain.com TXT “v=mc1 cfid=the_cfid_in_our_error_message”

1 Like

Any updates on rolling this out, and the educational campaign?

How long do you cache TXT records for?

Also, 500 Internal Server Error is an inappropriate status code, you should probably use 403.