Censored PoP & Orpheus not working around hostile outbound upstream in there

Here we go again. First of all, this is an issue with Cloudflare’s one particular outbound upstream (offering transit) in Istanbul into the Internet: AS6663 (namely Euroweb, Pantel, or Turk Telekom International. My home ISP is AS9121 for the records, which is a residential one and goes simply as Turk Telekomunikasyon AS)

Having heard about the announcement of Orpheus, I wanted to try bringing this to attention again. And no, this is not about circumventing access to blocked websites. Turk Telekom International is not even a Turkey-based company not unlike any other transit provider on-net in Turkey, and it has nothing to do with Turk Telekomunikasyon AS. To avoid confusion, I will henceforth refer to Turk Telekom International as AS6663.

Related post from months ago: Exit traffic from Istanbul PoP is likely being censored by the government

Orpheus promises to solve reachability issues between Cloudflare PoPs and origins by proxying traffic through other PoPs like with Smart Routing and Warp+. Announcement post depicts 522 (connection timed out) status as a trigger for a detour in actual route, but apparently they did not have this particular error in mind: 525.

I and several other people kept underlining, in Turkey, Internet traffic of CDN providers and enterprises (whether a local entity or not) are not to be scrubbed or filtered by a business offering transit service. Yet, breaking net neutrality and acting in a hostile way, AS6663 is clearly surveiling & sniffing SNI from TLS connections originating from Cloudflare to the origin (while marketing itself an international transit), and filtering traffic based on the value, causing the SSL handshake failure, thus 525.

Proof demonstrating AS6663 is filtering connections with SNI* value having pastebin.com**:
* Still, it can be IP-only, SNI/hostname-only, or a tuple of (IP, SNI/hostname). Not really sure.
** There are a lot of websites like Pastebin that were blocked years ago for no obvious and sane reasons. Taking Pastebin as an example here. This post is not about circumventing access to blocked websites.

home$ sudo wg show
interface: wg0
  public key: rl4TxJJPuPj2auu9S2ntAjBxJKzrNh34P35Pyh2Ui34=
  private key: (hidden)
  listening port: 49053
  fwmark: 0xca6c

peer: bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo=
  endpoint: 162.159.192.1:2408
  allowed ips: 0.0.0.0/0
  latest handshake: 1 minute, 54 seconds ago
  transfer: 4.51 MiB received, 322.72 KiB sent

home$ curl ipinfo.io/json
{
  "ip": "8.25.249.15",
  "city": "Istanbul",
  "region": "Istanbul",
  "country": "TR",
  "loc": "41.0138,28.9497",
  "org": "AS13335 Cloudflare, Inc.",
  "postal": "34096",
  "timezone": "Europe/Istanbul",
  "readme": "https://ipinfo.io/missingauth"
}⏎    

# I am on Warp, and using Cloudflare DoH for DNS.
home$ dig +short whoami.akamai.net
172.69.182.78

# Let us see the response by Cloudflare's reverse proxy first.
home$ curl -v https://pastebin.com
*   Trying 104.23.99.190:443...
* Connected to pastebin.com (104.23.99.190) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 17 00:00:00 2021 GMT
*  expire date: Jul 16 23:59:59 2022 GMT
*  subjectAltName: host "pastebin.com" matched cert's "pastebin.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e7c2f46580)
> GET / HTTP/2
> Host: pastebin.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 525 
< date: Thu, 23 Sep 2021 20:23:37 GMT
< content-type: text/html
< content-length: 207
< cache-control: no-store, no-cache
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 69367de929b85154-IST
< 
<html>
<head><title>525 Origin SSL Handshake Error</title></head>
<body bgcolor="white">
<center><h1>525 Origin SSL Handshake Error</h1></center>
<hr><center>cloudflare-nginx</center>
</body>
</html>
* Connection #0 to host pastebin.com left intact

##### Now directly to the Pastebin's origin (had to redact the hostname and IP).
### It is crystal clear now that AS6663 is filtering outbound traffic, and taking
### a non-net-neutral behavior while marketing itself as an international transit provider.
home$ curl -v https://ec0X.pastebin.com
*   Trying 49.12.154.19X:443...
* Connected to ec0X.pastebin.com (49.12.154.19X) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to ec0X.pastebin.com:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to ec0X.pastebin.com:443 

##### The same thing, but insecure HTTP.
home$ curl -v ec0X.pastebin.com
*   Trying 49.12.154.19X:80...
* Connected to ec0X.pastebin.com (49.12.154.19X) port 80 (#0)
> GET / HTTP/1.1
> Host: ec0X.pastebin.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

###### MTR lookup.
### See AS13335 (Cloudflare) handing the traffic to AS6663 for transit.
home$ mtr --report-wide --show-ips --aslookup ec0X.pastebin.com
Start: 2021-09-23T23:30:31+0300
HOST: home                                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    172.16.0.1                                                 0.0%    10   43.8  47.2  31.3  60.6   8.9
  2. AS13335  8.25.249.1                                                 0.0%    10   46.6  49.2  38.0  57.5   5.9
  3. AS6663   86.106.122.125                                             0.0%    10   46.5  57.3  38.3 126.1  25.1
  4. AS6663   31.210.9.42                                                0.0%    10   35.3  54.1  35.3  83.2  12.1
  5. AS6663   31.210.9.41                                                0.0%    10   45.9  47.9  34.8  65.1   8.6
  6. AS???    vix.invitel.net (193.203.0.160)                            0.0%    10   70.6  72.6  56.0  81.5   9.2
  7. AS???    193.203.0.199                                              0.0%    10   81.4  85.9  67.9 122.6  14.6
  8. AS24940  213-239-245-77.clients.your-server.de (213.239.245.77)     0.0%    10   90.3  97.4  79.3 107.7   9.4
  9. AS24940  vswitchgw.juniper2.dc1.nbg1.hetzner.com (213.239.245.62)   0.0%    10   81.4  76.8  67.5  91.1   9.1
 10. AS24940  ec0X.pastebin.com (49.12.154.19X)                         10.0%    10  180.7 173.8 161.6 186.9   8.3

Same tests from home without Warp:
See https://pastebin.com still does not work because of the outbound use of AS6663 in Cloudflare PoP, and direct connection to origin works because no AS6663 is involved. It is so funny to see my home ISP not doing the filtering when it is their responsibility to do so.

home$ curl ipinfo.io/json
{
  "ip": "94.235.125.113",
  "city": "Adana",
  "region": "Adana",
  "country": "TR",
  "loc": "36.9862,35.3253",
  "org": "AS20978 TT Mobil Iletisim Hizmetleri A.S",
  "postal": "01200",
  "timezone": "Europe/Istanbul",
  "readme": "https://ipinfo.io/missingauth"
}⏎     

# Using Cloudflare DoH for DNS.
home$ dig +short whoami.akamai.net
162.158.250.185

##### Let us see the response by Cloudflare's reverse proxy first.
### Still filtered by AS6663.
home$ curl -v https://pastebin.com
*   Trying 104.23.98.190:443...
* Connected to pastebin.com (104.23.98.190) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 17 00:00:00 2021 GMT
*  expire date: Jul 16 23:59:59 2022 GMT
*  subjectAltName: host "pastebin.com" matched cert's "pastebin.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x555692565580)
> GET / HTTP/2
> Host: pastebin.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 525 
< date: Thu, 23 Sep 2021 21:54:33 GMT
< content-type: text/html
< content-length: 207
< cache-control: no-store, no-cache
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 6937031dce61e17e-IST
< 
<html>
<head><title>525 Origin SSL Handshake Error</title></head>
<body bgcolor="white">
<center><h1>525 Origin SSL Handshake Error</h1></center>
<hr><center>cloudflare-nginx</center>
</body>
</html>
* Connection #0 to host pastebin.com left intact

##### Now directly to the Pastebin's origin (had to redact the hostname and IP).
### It works, because no AS6663 is involved, and no party is actively filtering any traffic.
### See also the offered server certificate serving a Cloudflare Origin Server Certificate.
### This is an indication that this hostname is meant to be put behind Cloudflare.
home$ curl -v https://ec0X.pastebin.com --insecure
*   Trying 49.12.154.19X:443...
* Connected to ec0X.pastebin.com (49.12.154.19X) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=CloudFlare, Inc.; OU=CloudFlare Origin CA; CN=CloudFlare Origin Certificate
*  start date: Sep 23 22:58:00 2020 GMT
*  expire date: Sep 23 22:58:00 2021 GMT
*  issuer: C=US; O=CloudFlare, Inc.; OU=CloudFlare Origin SSL Certificate Authority; L=San Francisco; ST=California
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5555c76c1580)
> GET / HTTP/2
> Host: ec0X.pastebin.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 502 
< server: nginx
< date: Thu, 23 Sep 2021 22:02:16 GMT
< content-type: text/html
< cache-control: no-cache
< 
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
* Connection #0 to host ec0X.pastebin.com left intact

##### The same thing, but insecure HTTP.
### It works, same as above.
home$ curl -v ec0X.pastebin.com
*   Trying 49.12.154.19X:80...
* Connected to ec0X.pastebin.com (49.12.154.19X) port 80 (#0)
> GET / HTTP/1.1
> Host: ec0X.pastebin.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Server: nginx
< Date: Thu, 23 Sep 2021 22:07:51 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< cache-control: no-cache
< 
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
* Connection #0 to host ec0X.pastebin.com left intact

###### MTR lookup.
### See that no AS6663 is involved, hence no filtering taking place.
home$ mtr --report-wide --show-ips --aslookup ec0X.pastebin.com
Start: 2021-09-24T01:09:58+0300
HOST: home                                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.20.59)                                                             0.0%    10    1.5   1.6   1.3   1.8   0.2
  2. AS???    ???                                                                                 100.0    10    0.0   0.0   0.0   0.0   0.0
  3. AS???    10.222.31.41                                                                         0.0%    10   14.4  18.8  14.4  24.9   3.4
  4. AS???    10.222.104.133                                                                       0.0%    10   23.8  19.2  13.2  28.4   5.1
  5. AS???    10.240.110.202                                                                       0.0%    10   13.8  17.4  13.8  22.8   2.5
  6. AS9121   195.175.63.1.static.turktelekom.com.tr (195.175.63.1)                                0.0%    10   40.4  19.9  14.8  40.4   7.6
  7. AS9121   01-adana-xrs-t2-2---01-avea-t3-1.statik.turktelekom.com.tr (81.212.24.210)           0.0%    10   20.1  17.8  14.4  22.3   2.3
  8. AS9121   06-ulus-xrs-t2-2---01-adana-xrs-t2-2.statik.turktelekom.com.tr (81.212.216.58)       0.0%    10   23.9  26.6  23.1  34.4   3.9
  9. AS9121   06-ebgp-ulus-sr12e-k---06-ulus-xrs-t2-2.statik.turktelekom.com.tr (81.212.217.12     0.0%    10   24.1  28.3  23.6  47.8   7.1
 10. AS9121   301-fra-col-2---06-ebgp-ulus-sr12e-k.statik.turktelekom.com.tr (212.156.101.40)      0.0%    10   67.3  70.4  67.3  76.9   3.2
 11. AS2914   ae-31.r01.frnkge13.de.bb.gin.ntt.net (213.198.52.89)                                 0.0%    10   78.6  77.9  74.7  82.8   2.4
 12. AS2914   213.198.82.130                                                                       0.0%    10   69.8  73.0  68.7  78.4   3.1
 13. AS24940  core12.nbg1.hetzner.com (213.239.252.26)                                             0.0%    10   68.9  70.9  65.8  83.3   5.0
 14. AS24940  vswitchgw.juniper2.dc1.nbg1.hetzner.com (213.239.245.62)                             0.0%    10   72.0  79.1  65.9 153.6  26.4
 15. AS24940  ec0X.pastebin.com (49.12.154.19X)                                                    0.0%    10  118.1 114.1 108.9 122.1   5.0

TLDR;

  • Cloudflare selects an outbound transit depending on the incoming one, as demonstrated in the related post (Exit traffic from Istanbul PoP is likely being censored by the government - #2 by ultimatedev) by @ultimatedev.
  • This particular outbound transit, AS6663, is actively sniffing, filtering, and dropping connections in a shady way from Cloudflare’s edge network to the origin, while marketing itself as an international transit provider which is supposed to be striving for net and content neutrality, and Cloudflare must do something about it.
  • Enforcing censorship decisions of a state is not the job of a transit provider, when local ISPs providing residential services are not even doing that as shown.
1 Like

I believe that Cloudflare team is already aware of this issue (obviously, high 5xx errors), but nothing they can do about it.

Does this affect all websites, or only several websites? Can you open Discord, etc (https://cloudflare-test.judge.sh/)

I wonder if it’s a routing issue from servers back to Turk Telecom, which is being filtered.

Any traffic sent out over AS6663 by Cloudflare to the origin can be affected. The destination just has to be in the blocklist of the Turkish government. This includes, but not limited to, particular domains and IP addresses such as Pastebin’s. We don’t have an open list of them.

Another outbound transit employed by Cloudflare at Istanbul PoP is TI Sparkle (Seabone) which works flawlessly when put into use. But Cloudflare decides on the transit to use depending on the incoming one, so Vodafone goes over Seabone without issues while Turk Telekom goes over AS6663 leading to the filtering of the traffic.

The point is that, AS6663 is an ISP selling itself an international transit provider, which means they have to keep net and content neutral. Think of Cogent, Telia, or Level 3 doing this for your international traffic leaving the borders of a country for the Internet, it is the same thing.

I believe Cloudflare is either not aware of this, or knowingly entered into a contract with the operators of AS6663 and accepted that their international traffic might be filtered as in compliance with Turkish government’s directives, compromising its promises to the community. Might be biased, but just my two cents.

Cloudflare has to ditch this transit.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.