Email routing 421 4.3.0 Upstream error to sender, but email delivered

Feedback Previews & Betas
1

Feedback

We have had an instance where an email was sent to a custom address in Cloudflare, however the email sent to the destination 4 times.
The email was sent more than 7 days ago (2nd December 2024), and I do not see it the activity logs.

The sender is using Exchange Online, so they could check message trace, and they could see the message was deferred with the following error for the message 3 times before finally showing as message sent.
“Reason: [{LED=421 4.3.0 Upstream error, please check Postmaster · Cloudflare Email Routing docs for possible reasons why.”
421 as I understand is a temporary or transient error, hence would be expected for sender to retry.
The times in the message trace showing as deferred (3 times) and the final sent (1 time) correctly corresponds with the time when the message arrived in the destination inbox (4 copies of the same email)

Is there anything I can do to check or troubleshoot and prevent this from happening again?
Or is this just a temporary problem and as a result of things beyond our control?

2

Where is the final destination of this message?

I will suggest checking the final destination, and as you mentioned you did from sender’s side, then see if you can find some sort of message trace, history or logging, at the final destination.

Without more information such as e.g. what happened (e.g. checking “Activity Logs” sooner), then it will be impossible to say for sure.

If the final destination responded with a 4XX code to Cloudflare, for a while, before it declared the acceptance of the message, which isn’t uncommon with most free mail providers these days, then you will need to research the issue together with the final destination (even if it’s a free mail), how you can deactivate that from their (the final destination’s) end.

There is also a chance that a transient / temporary network failure could possibly also have dropped the connection, e.g. before the final destination could respond properly to Cloudflare that it had accepted the message, or before Cloudflare (successfully) received a such response, and as such, due to the dropped connection, the result would be that Cloudflare would respond with “421 4.3.0” code to the actual sender, due to the network failure.

1 Like
3

Where is the final destination of this message?

Forwarded to Salesforce mailbox, where it gets processed into a customer lead/session.

I will suggest checking the final destination, and as you mentioned you did from sender’s side, then see if you can find some sort of message trace, history or logging, at the final destination.

We did. No errors found receiving. Just 4 copies of the same message, from about the same time as we found in Exchange

Attempt Exchange Time Salesforce time
1 10:57am 10:57am
2 11:12am 11:13am
3 11:30am 11:31am
4 12:32pm 12:32pm

Is there anything more to 421 4.3.0 Upstream error?
If I was able to check Activity Logs in time, what would it have likely shown?

There is also a chance that a transient / temporary network failure could possibly also have dropped the connection, e.g. before the final destination could respond properly to Cloudflare that it had accepted the message, or before Cloudflare (successfully) received a such response, and as such, due to the dropped connection, the result would be that Cloudflare would respond with “421 4.3.0 ” code to the actual sender, due to the network failure.

It looks like the email did get routed successfully. As there is nothing available to check right now, this would likely be a once off (I hope) as I checked other instance of this workflow and did not see the same problem.

4

Ok, I have been keeping an eye on the Cloudflare activity logs, and then I saw 2 instances where this failed again today.

Rejected reason:
Timeout while connecting/communicating to the upstream: network error: timed out

In Exchange, we see the message was deferred once and then delivered.

What is not clear yet if this was received by Salesforce once or twice. Something we are still checking.

Anyway, it is clear then, transient network issues might cause emails to be delivered multiple times.

5

Are you able to see the headers of these messages on the Salesforce side?

As in, … do they all have the exact same “From:”, “To”, “Message-ID” and “Date” headers?

Are the headers a (100%(?)) match all the way, or are there any kind of discrepancies?

“Upstream error” in this case, is simply Cloudflare’s way of telling that you should look at the at the final destination for further troubleshooting.

SMTP codes such as e.g. “421”, and the SMTP enhanced status code “4.3.0” have some meanings in the SMTP world, which can be found over here:

→ Simple Mail Transfer Protocol (SMTP) Enhanced Status Codes Registry

The “Upstream error” part is however “free-form”, and can be anything from an operator’s choice (e.g. choosen by Cloudflare), to the choice of the vendor of the mail server software (e.g. Exim, Postfix, Sendmail, et al), that is being used.

It is referring to a connectivity issue, which in this case is happening with the communication between Cloudflare and Salesforce.

But stuff like that was indeed what I would be looking for in the “Activity Log”.

One previous example I made, where I on the final destination was rejecting with a SMTP code “550”, and the extended code “5.7.1”, would be showing something like this:

I don’t know how long GraphQL would expose the data of the “Activity Log”, however, my understanding is that you can fetch (at least, some of all this) data from the “Activity Log”, through GraphQL.

Transient issues, regardless of when, how, and where they happen, can indeed cause a lot of strange results, including ones that (almost) no one would ever have thought about.

In your case, it does appear like the connectivity issue is happening just after Salesforce has received the message (“end of data” signal from Cloudflare), and before Cloudflare receiving the acknowledgement of of that signal, as I indicated above.

1 Like