Cloudflare-IPFS gateway corrupts (cuts-off) files

General
1

Hello,

There’s ongoing issue with Cloudflare IPFS gateway Cloudflare-ipfs.com, which affects only some Cloudflare IPFS PoPs (≈regions) and happens quite frequently.
The gateway does not serve the whole file and cuts it off in different places, for the whole file cache lifetime.

Here’s the example of such file:
https://cloudflare-ipfs.com/ipfs/Qmcsy6hcRS7mDA9ALA96XW4mst3iGEYmEvbj92yquKzyNF/proxy-ssl.js

It is 676614 bytes in size, but the real file is 950 kB.

Once the file is cached being cut-off, it is served in this state all the time in the same region, while the other regions are not affected, which encumbers debugging the issue.
Opening the file using other IPFS gateways, such as ipfs.io, returns full uncut file:
https://ipfs.io/ipfs/Qmcsy6hcRS7mDA9ALA96XW4mst3iGEYmEvbj92yquKzyNF/proxy-ssl.js

The full file has “return “DIRECT”;” string in the end of it.

Cut-off file served on Russian Cloudflare PoP:

Screenshot_20220111_085823-fs8
Screenshot_20220111_085823-fs8712×158 13.9 KB

I’ve tried to pre-download the file over multiple gateways before serving it on the website, but that doesn’t help much, it still eventually getting served broken in some regions but not in other.

This file has been broken in Russia since 26 December and is still served cut-off:
https://cloudflare-ipfs.com/ipfs/QmeRv6ei2jQM6H6VwrYaj8WZuKLd58EA9QaEVfRsYNRYNZ/proxy-ssl.js

Please check what could be wrong with the gateway.

1 Like
3

I’m seeing the same problem for https://cloudflare-ipfs.com/ipfs/QmTfXoED9voY5sT6Fmm1GhzJBriYrsXUw8Exd4Gzp3z9ue. Loading it in your browser you can see that only the first half of the image is returned. https://ipfs.io/ipfs/QmTfXoED9voY5sT6Fmm1GhzJBriYrsXUw8Exd4Gzp3z9ue returns the full image as expected.

4

Indeed, the link loads only half of the image for me. Cloudflare node in Sweden.
I’ve filled the ticket #2347312 to Cloudflare support. In fact, it was filled before my first forum post. I’ve updated the ticket with your URL.

5

Interesting. I am unlikely to be hitting the same edge as you so it looks like this issue is affecting multiple edge locations. Maybe they are doing tired caching or something or maybe it is a reproducible bug.

Example request:

HTTP/1.1 200 OK
Date: Sun, 23 Jan 2022 21:10:47 GMT
Content-Type: image/jpeg
Transfer-Encoding: chunked
Connection: keep-alive
CF-Ray: 6d2402c36cdc2a3c-ORD
Access-Control-Allow-Origin: *
Age: 11945
Cache-Control: max-age=86400
ETag: W/"QmTfXoED9voY5sT6Fmm1GhzJBriYrsXUw8Exd4Gzp3z9ue"
Last-Modified: Thu, 01 Jan 1970 00:00:01 GMT
Vary: X-Ipfs-Secure-Gateway, Service-Worker, Accept-Encoding
CF-Cache-Status: HIT
Access-Control-Allow-Headers: Content-Type
Access-Control-Allow-Headers: Range
Access-Control-Allow-Headers: User-Agent
Access-Control-Allow-Headers: X-Requested-With
Access-Control-Allow-Methods: GET
Access-Control-Expose-Headers: Content-Range
Access-Control-Expose-Headers: X-Chunked-Output
Access-Control-Expose-Headers: X-Stream-Output
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
X-Ipfs-Path: /ipfs/QmTfXoED9voY5sT6Fmm1GhzJBriYrsXUw8Exd4Gzp3z9ue
X-Ipfs-Root-Cid: QmTfXoED9voY5sT6Fmm1GhzJBriYrsXUw8Exd4Gzp3z9ue
Set-Cookie: __cf_bm=REDACTED; path=/; expires=Sun, 23-Jan-22 21:40:47 GMT; domain=.cloudflare-ipfs.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
6

Ticket:

I am able to replicate with the provided image and have raised this to our IPFS team to additional insight.

We’ll send an update once available but please let us know if you have any other questions or information around this issue in the meantime.

7

Ticket:

I just wanted to reach out and let you know this issue is still with our engineering team. We will continue to check back in as more information becomes available.

8

By the way, the image loads in full right now.

9

I saw this again with a CNAME site. This time the file was truncated all the way to 0 bytes!

% curl -i https://userscripts.kevincox.ca/gitlab-emoji.user.css
HTTP/2 200 
date: Sun, 06 Mar 2022 01:32:04 GMT
content-type: text/css; charset=utf-8
content-length: 0
cf-ray: 6e7755dcce848737-ORD
accept-ranges: bytes
access-control-allow-origin: *
age: 7122
cache-control: max-age=86400
etag: "QmX94j8AwTtNQ4nVmSgAzwBswAp24Zny1qSTzNukcwY4u5"
last-modified: Sat, 05 Mar 2022 23:33:21 GMT
vary: X-Ipfs-Secure-Gateway, Service-Worker, Accept-Encoding
cf-cache-status: HIT
access-control-allow-headers: Content-Type
access-control-allow-headers: Range
access-control-allow-headers: User-Agent
access-control-allow-headers: X-Requested-With
access-control-allow-methods: GET
access-control-expose-headers: Content-Range
access-control-expose-headers: X-Chunked-Output
access-control-expose-headers: X-Stream-Output
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
x-ipfs-path: /ipns/userscripts.kevincox.ca/gitlab-emoji.user.css
x-ipfs-root-cid: QmUjLJPH1SgC6R4NKPACh5TwdyxVbSSyWJF2ShAgsjPaTP
set-cookie: __cf_bm=REDACTED; path=/; expires=Sun, 06-Mar-22 02:02:04 GMT; domain=.userscripts.kevincox.ca; HttpOnly; Secure; SameSite=None
server: cloudflare
10

My data experienced cut-off two more times. I’ve sent everything to the ticket.

11

Hi!

I found this conversation because I have the same issue with this file:

Other gateways (fleek, ipfs.io, pinata) serve the file in its entirety.

One trouble is that some platforms like objkt.com uses Cloudflare ipfs gateway as the hardcoded way to display the original ipfs files if users want to see it (understandable, as most users don’t have ipfs desktop or a local node).

Is the bug considered as identified? Is there a tool to force recaching an ipfs file?

Thanks!

12

Chris Martinelli (Cloudflare)

Mar 6, 2022, 1:23 PM PST

Hello,

Thank you. Our IPFS team are working on this issue still. We will update you once this is resolved. I believe we have all the information we need for the time being, thank you.

We’ll send an update once available but please let us know if you have any other questions or information around this issue in the meantime.

Chris M.
Technical Support Engineer | Cloudflare

13

To add to this, the Cloudflare IPFS Gateway team have identified a bug and are working on a fix to be released shortly. We will provide more updates as they come!

2 Likes
14

A new version of our IPFS gateway has been released to address these issues. Please let us know if any errors are observed going forward.

3 Likes
15

Hi and thanks for the news Chris!

the file in question is now fine on my side / region.

I will check other files and report back if I have further issues

1 Like
16

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.