SERVFAIL resolving s3-1-w.amazonaws.com

DNS & Network 1.1.1.1
#1

I’m frequently getting SERVFAIL resolving Amazon S3 hostnames, seems to be an issue on cloudflare’s side.

Connection info: https://cloudflare-dns.com/help/#eyJpc0NmIjoiWWVzIiwiaXNEb3QiOiJZZXMiLCJpc0RvaCI6Ik5vIiwicmVzb2x2ZXJJcC0xLjEuMS4xIjoiWWVzIiwicmVzb2x2ZXJJcC0xLjAuMC4xIjoiWWVzIiwicmVzb2x2ZXJJcC0yNjA2OjQ3MDA6NDcwMDo6MTExMSI6IlllcyIsInJlc29sdmVySXAtMjYwNjo0NzAwOjQ3MDA6OjEwMDEiOiJZZXMiLCJkYXRhY2VudGVyTG9jYXRpb24iOiJMSFIiLCJpc1dhcnAiOiJObyIsImlzcE5hbWUiOiJDbG91ZGZsYXJlIiwiaXNwQXNuIjoiMTMzMzUifQ==

My test results:

$ ./cf-dns-test.sh
cloudflare.com @ 1.1.1.1 - 0/1000 failed
s3-1-w.amazonaws.com @ 1.1.1.1 - 49/1000 failed
s3-1-w.amazonaws.com @ 8.8.8.8 - 0/1000 failed

And my test script:

#!/bin/sh
dns_test() {
    failed=0; try=1; dns=$1; hostname=$2
    for try in $(seq 1 1000); do [ -z "$(dig +short +tries=1 $hostname @$dns)" ] && failed=$((failed+1)); done
    echo $hostname @ $dns - $failed/$try failed
}

dns_test 1.1.1.1 cloudflare.com
dns_test 1.1.1.1 s3-1-w.amazonaws.com
dns_test 8.8.8.8 s3-1-w.amazonaws.com
1 Like
A long time has passed
#2

seeing this behavior as well, seems to have started yesterday at approx 17:20p PDT

only certain subdomains of amazonaws.com seem to be affected

1 Like
#3

Sorry about that, I’ll take a look.

1 Like
#4

Currently working for me from EWR using DNS over HTTPS. @noc9 Do you know which edge you’re hitting?

#5

the servers in question that are affected by this are in us-east-1 and hitting IAD

1 Like
#6

We are also seeing this issue when trying to resolve us-east-1 S3 bucket URLs (e.g. mydomain.s3.amazonaws.com).

#7

It seems like several bursts of traffic were ratelimited too aggressively which formed a feedback loop with increased retries in some places. It should be cleared up in IAD now.

#8

thanks for looking into and fixing this!

#9

Glad I found this thread. Thought I was losing my mind all day, checking DHCP server, switch, ISP, etc. Thanks for resolving the issue!

#10

Is this happening again? Can’t connect to 1.1.1.1 out of us-east-1.

#11

Contact Amazon or… Have problems with 1.1.1.1? *Read Me First*?