Hi guys …
I have seen in another post how to fetch remote file and send it in response
The code was like this :
addEventListener("fetch", event => {
event.respondWith(fetchAndStream(event.request))
});
async function fetchAndStream() {
// Fetch from origin server.
let response = await fetch('https://www.dropbox.com/s/9x3yzowigc0sswh/alfa.mp4?dl=1');
const responseInit = {
headers: {
'Content-Type': 'video/mp4',
'Content-Disposition' : 'attachment; filename="video.mp4"'
}
};
// ... and deliver our Response while that's running.
return new Response(response.body, responseInit)
}
I have similar use case when I want to send 20 to 50 Mb of Json files
So when fetching file remotely … It will load in worker memory ( as I think )
And this way I will hit memory CPU limit quickly ( after some concurrent request )
So how to use stream Json files to avoid that ?
Or is there better solution ?
My Understanding is that you have in total 128 MB per Worker availble which can be Global aka persistent non-volatile between Requests or Local Memory non persistent volatile.
Fetching the Resource and saving it in the local non persistent volatile Memory and not in the Global Memory as you do in the Variable it takes only 30 MB and
should be not a problem. Ofcourse sooner or later you will get i guess problems becouse
of the type of media aka video as you should use the stream service.
For Data it should work however with that size.
In total your Worker would use 30 MB from 128 MB per each request no matter how many parallel requests are done.
This would change if you need to do other operations like
cloning of the Response which then would double the Memory usage.
Also CPU Limit should be fine i think as you suspend the waiting for the Response
to the resource with the await command.
If you have a slow Connection that takes 1 Minute to deliver the Response to the fetch request
it should not end into canceling the request as it suspends it i think.
Maybe more details are needed here from the cloudflare team about what happens
when a fetch call to the origin excede the cpu limit for the worker.
Not sure if the command would be better here
event.waitUntil(promisePromise)void
Extend the lifetime of the event without blocking the response from being sent. Use this method to notify the runtime to wait for tasks (e.g. logging, analytics to third-party services, streaming and caching) that need to run longer than the usual time it takes to send a response.
My Understanding till yet was always that waiting for the fetch response does not
eat all the cpu limit even when doing 1000 Parallel request at the same time.
Would be great if this can be clarified better by the cloudflare team.
For your usecase video upload and display i think clearly you need
to activate stream.
Here you can find instructions how to upload Videos on Cloudflare.
what I understand from your reply is : every request come to my script will have a worker memory of 128 MB
so if I have for example 1 million request … each one of these request will have 128 memory
so I will never hit memory limit ( if my Json files didn’t exceed 60 MB )
is this correct ?
In total you have 128 MB per Worker.
It can be persisitent non volatile Global Memory.
If many Request allocate Global Memory and the limit is exceded you have problems.
And then there is the Local non persisitent volatile Memory
which is flushed / cleared ( garbage collection ) for each Request as it does not make sense
to keep this Memory after the response is sent.
B
ecouse of this You have for each Request 128MB Local Memory as it is freed / cleared per request
but for all Request only 128 MB Global Memory as it is not cleared per request.
it depedns how you allocate the Memory inside your Worker code.
As you store your Fetch Json Data in local memory
i dont see any problems as long your origin server can handle the load
and delivers a fast reply.
I would however ask myself why you need do this over Worker.
What you do actually is proxying your request.
In your case best would be make this request direct to the origin as the code does not make sense actually at all.
Client makes request to Cloudlfare then Cloudflare makes request to Origin Server
after this Origin Server response to Cloudflare and lastly Cloudflare response to Client.
Better would be make the JSON Request diret to the Origin Server.
This would keep your Cloudflare resouce usage low.
I would do it if I could but the problem from worker KV
this is my issue
I have thousands of json file and every one is from 20 to 60 mb
the total storage may exceed 2000 GB
if I save all files in Worker KV … I will have to pay 0.5 per 1 GB which is about 1000$ every month
so my solution is :
I will add all files to CDN bunny.net edge storage
which is cost 0.01 per 1 GB … so the cost is only about 20$
another thing is : I want only authenticated users to get these files
for that : I will authenticate user and then the script will fetch files from CDN bunny.net through public link then send it to userswithout showing them the file public link
in this way I will pay only about 25$ instead more than 1000$ per month
As said delivering this 60 MB Files
with the Code above should work.
But there is no need actually to use Worker for this and store the response in the cloudflare memory before you deliver it to the authenticated end user.
I would not recommend it doing it this way !.
The Solution for your Authentication Problem to deliver the Files
only to specific Users can be solved by
moving the Authetication to your Origin Server itself.
Instead of autheticate the User and then Proxy evertyhting over Cloudflare
You Authenticate the User on your Origin Server either
direct or over the cloudflare and then when all is good and the User is
authenticated you redirect him to the allgoodfordeliver script which streams the file to the user
without to be proxied over cloudflare itself which double triple the amount of traffic.
This fetch Code can be put also on the Origin Server dont need to be run on Cloudflare itself.
Your Way of solving this is not the best way and can be interrupted by cloudflare.
it not something that can be stable in my opinion.
For sure it will create a lot of Traffic World Wide which will cost Cloudflare a Lot of Money
and then sooner or later they will have to talk with you.
You need make sure that your use case is inline
with their policy.
Cloudflare is building a better Internet and it’s not our role to determine what content should or should not be allowed to be published. That said, if you’re using our network solely as a file locker, distributing malware or phishing, or otherwise causing per se harm then we will terminate use.
First : I don’t have origin server
I want to use these :
for FrontEnd: I will use Cloudflare pages
for BackEnd : I will use Cloudflare worker
for file storage : I will use bunny.net edge Storage
so the user will visit site hosted on Cloudflare Pages and make API requests to Cloudflare worker then the worker script will authenticate user and fetch file from bunny.net edge storage then send it back to user
Second : I didn’t know about abusing Policy but are you sure ?
I see here : Streaming large remote files - #5 by jiri
the Cloudflare stuff say this is intended use case
For Display Images and Videos on a Website
its allowed. Like for a Dentist Website.
But if you Build a File Locker Service which you do
with 2 Terra Byte of Data to be streamed distributed
in my opinion clearly not.
Best is for sure to talk direct with the Cloudflare Team.
Open a Ticket and explain them that you want stream
2 Terrabyte of Data World Wide to authenticated Users
and see what they Say.
From what i have read the last 12 Months here on the forum
is you will either have to go Enterprise
or your account will get deactivated.
You can send and receive JSON, but you can’t parse it in the worker, it is only possible to parse about 1.5MB of JSON. To avoid parsing the string/json, you cannot place it in a variable, it has to stream in chunks from the source (like KV) to the target (client) directly.
What is allowed, you can build apps or services that are web based (yes, even apps with large JSON), you cannot, however, create a file-hosting or image hosting service.
