Cloudflare blocking minor ISP

Hi Cloudflare community,

I’m regret to report that Cloudflare is blocking legit access to several site from the CIDR blocks of a minor ISP in Italy.

This is the list of (verified) blocked site:

These are the CIDR blocks involved, all owned by the same minor ISP:

  • {redacted}/21AS60822
  • {redacted}/22AS60822

I have verified the blocking problems on three different connections from the same minor ISP; connections made with IP (and connectivity) owned by major ISP don’t have any problems.

As a technical note, all domain in the previous list are resolved with the same IP couple:

-{redacted}

  • {redacted}

So, what is the best course of action to make things work again for us minor ISP users?

1 Like

Cloudflare generally doesn’t hard-block an ISP.

What type of blocking error message are your users seeing? There should be an Error Code.

No error code, simple plain TCP/IP timeout.

This is an excerpt of today analysis (pfSense multi wan firewall):

[2.4.5-RELEASE][[email protected]<omissis>]/root: telnet -s 46.23.<minor_ISP_IP> 188.114.96.15 443
Trying 188.114.96.15...
^C
[2.4.5-RELEASE][[email protected]<omissis>]/root: telnet -s 82.190.<major_ISP_IP> 188.114.96.15 443
Trying 188.114.96.15...
Connected to 188.114.96.15.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

I have repeated same tests at the moment of writing this, same results.

I also checked if the minor ISP source IP was in some blacklists, but I haven’t found it in any.

Obviously, generic internet connections is working on minor ISP:

[2.4.5-RELEASE][[email protected]<omissis>]/root: telnet -s 46.23.<minor_ISP_IP> 8.8.8.8 443
Trying 8.8.8.8...
Connected to dns.google.
Escape character is '^]'.

Have you tried a traceroute?

Oh yes, I have forgot to say that there is ICMP reachability:

[2.4.5-RELEASE][[email protected]<omissis>]/root: ping -S 46.23.<minor_ISP_IP> 188.114.96.15
PING 188.114.96.15 (188.114.96.15) from 46.23.<minor_ISP_IP>: 56 data bytes
64 bytes from 188.114.96.15: icmp_seq=0 ttl=59 time=10.114 ms
64 bytes from 188.114.96.15: icmp_seq=1 ttl=59 time=9.719 ms
64 bytes from 188.114.96.15: icmp_seq=2 ttl=59 time=9.735 ms
64 bytes from 188.114.96.15: icmp_seq=3 ttl=59 time=9.922 ms
^C
--- 188.114.96.15 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.719/9.872/10.114/0.161 ms

I still think it’s something along the route that’s blocking, though I could be wrong.

Oddly enough, none of those hostnames resolve to the new 188 addresses for me. They’re the usual 172 and 104 subnets. Interestingly enough, they resolve to 188 in some EU countries, which is quite strange. @mvavrusa is a whiz with DNS and might know why they differ.

With one major exception. Stackblitz does not use Cloudflare. It’s actually on Amazon’s Cloudfront.

Can you try this:
curl -svo /dev/null https://githubbox.com --connect-to ::172.67.190.113

And see if that gets through?

2 Likes

Sorry for stackblitz, it gets through a wrong cut and paste.

Going back to HTTP host check on 172.67.190.113 you suggest, I think it works in full because it responds with a 302 to the github githubbox project page:

[2.4.5-RELEASE][[email protected]<omissis>]/root: curl -svo /dev/null https://githubbox.com --connect-to ::172.67.190.113
* Connecting to hostname: 172.67.190.113
* Trying 172.67.190.113:443...
* TCP_NODELAY set
* Connected to 172.67.190.113 (172.67.190.113) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /usr/local/share/certs/ca-root-nss.crt
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2331 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [148 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
* start date: Jul 7 00:00:00 2021 GMT
* expire date: Jul 6 23:59:59 2022 GMT
* subjectAltName: host "githubbox.com" matched cert's "githubbox.com"
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x803aa5800)
} [5 bytes data]
> GET / HTTP/2
> Host: githubbox.com
> user-agent: curl/7.67.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 302
< date: Thu, 03 Feb 2022 19:33:55 GMT
< content-length: 0
< location: https://github.com/dferber90/githubbox
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=0qfuhB%2FWUPkcyPsX3UW3lBlwrauFea2W4MpeysADEmWvMV9W4m9RXZczKsu2%2Be30ieqbMZMhV%2BdcNm%2B4OvlLMJGXExikyN9T2ppXsmHdxXnWOrPZYIVkNGcpNqWjXSuo"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6d7e17fc58e66841-SEA
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
{ [0 bytes data]
* Connection #0 to host 172.67.190.113 left intact

And I confirm the DNS resolution in the 188 octect here in Italy:

[2.4.5-RELEASE][[email protected]<omissis>]/root: dig githubbox.com +short @1.0.0.1
188.114.96.15
188.114.97.15
[2.4.5-RELEASE][[email protected]<omissis>]/root: dig download.wavebox.app +short @1.0.0.1
188.114.96.15
188.114.97.15

That might very well be the wrong IP address for the site. I’m going to bump this over to Support for them to check on the next time they’re here. Hopefully later today or tomorrow morning.

2 Likes

Any news? Today we lost phoronix.com

dig phoronix.com +short @1.1.1.1
188.114.96.15
188.114.97.15

Hello,

Thanks for flagging it. Can you please open a Support ticket and give more details of the ISP and IPs which are used to access. Also in the ticket please mention the results for `https://download.wavebox.app/cdn-cgi/trace so that we can know which data center the request hits.

Hi Anjana,

I’m afraid I cannot open any support ticket; I’m the customer of your customers (Wavebox, etc.) here, not your direct customer.

Nevertheless we, users of local ISP, are directly impacted by network disruption when we try to access sites protected by your services.

We can ring the bell of your severs (ICMP reachability) but they refuse to open the door (TCP timeout while attempting to connect HTTPS port), so I cannot even open the URL you propose.

CIDR blocks involved belong to ASN 60822, with that number you can get all the information you need to identity the ISP; maybe they and your NOC can tech talk to each other to resolve the matter, I have feedback that a contact request is made.

Email them: support AT cloudflare DOT com and then post the ticket # here.

2 Likes

I didn’t have time to open a support ticket in the past few days, but now I see that all connection issues are gone. I will keep an eye on the domains this week to see if the problem has been solved permanently.

1 Like

Hi, we are having the same problem with a lot of sites. Also with the above mentioned.
Examples www.defence-point.gr, emccgreece.gr, www.faysbook.gr, www.aek365.org.
We are also get internet from minor ISP (Greek National Infrastructures for Research and Technology S.A.)
All the above sites and a lot more, all resolve to 188.114.97.15 and 188.114.96.15.

The strange thing is that, it is happening only when we use googles DNS. If we use Cloudflare DNS, we can access these sites. However we have a huge network and can’t force all not to use googles DNS, and doesn’t seem a correct solution.

Thanks in advance.

After a day of troubleshooting, we came to some results. Cloudflare doesn’t block anything. We had some pirate movies sites blocked, which were resolved in the same ip with the above sites, by google’s dns (How is that possible??). So the problem is that google’s DNS resolves a lot of sites hosted by Cloudflare to the same ip. When we used Cloudflare’s DNS, every site resolved to its own ip and worked fine. I dont know why google’s DNS does that, we sure have more to investigate, but we found why this was happening.