Serious CDN-introduced silent data corruption for byte range requests

[alessandro14]

What is the name of the domain?

cjrtnc.leaningtech.com

What is the issue you’re encountering

Data is silently corrupted when responding to byte range requests. We observed that Cloudflare (sensibly) downloads the whole resource from the origin server before responding to a range-based request, but we think random offsets are then, in some cases, added to the starting position of the data when the range response is created. We could verify that the response has the right length, but it begins at the wrong location in the original resource

What steps have you taken to resolve the issue?

We have extensively validated that the origin server is healthy and has valid data, by comparing against separately stored backups and checking the RAID array. Everything looks correct. There are also no error on the upstream server logs.

What are the steps to reproduce the issue?

The error is difficult to reproduce, but correlated with a cache MISS for a byte range request. The following URL is an example reported by a user, but it’s not the only one.

https://cjrtnc.leaningtech.com/3_20250225_603/8/jre/lib/resources.jar, range bytes 0-131071/1121910

More details as a base64 encoded dump of the correct and faulty data can be found on the Cloudflare discord, where I opened an issue first: Discord

[alessandro14]

I have found a oneliner that can be executed in Chrome’s console to reproduce the problem.

async function hack(count){for(var i=0;i<count;i++){var token=((Math.random()*100000)|0);try{var r = await fetch("https://cjrtnc.leaningtech.com/3.0/8/jre/lib/jce.jar?hack2="+token, {"headers": { "range": "bytes=0-16384", }, "method": "GET", });var data=new Uint8Array(await r.arrayBuffer()); console.log(i, String.fromCharCode(data[0],data[1]));}catch(e){console.log(token,"ERROR");}}}hack(50)

The count parameter can be freely altered. This should always print ‘PK’ on the console, since the file is JAR/ZIP file. See screenshot for an example of a failure.

[cscharff]

You should open a support issue, the community can’t resolve this issue.

[alessandro14]

The issue has been in the mean time reproduced and escalated and a ticket is opened already.