A condition of exceptions to run the worker

workers

#1

hi sir.

can you create condition of exceptions to run the worker, example by user agent.

i know worker price:
$5.00 per domain.
free 10 million first request to worker.
After the first ten million, requests are billed at $0.50 per million.

that price is cheap but if more than 50% access is real human. in my problem is, i have cloud storage that contain more than 10 million images and my images is load on many more domain member. it’s no problem when worker run 10 million/day and if image access by real human.

But if image access by bot example google bot? it can run worker 100 million/day because googlebot crawl all domain member.

if you have custom menu to filter if user agent is googlebot or other bot on list can exclude worker.

please give me solution.
thank you.


#2

Hi @kputera410,

You can configure Workers to run only on certain URL patterns, under the “routes” tab in the Workers editor. If you specify that workers should be disabled at a particular route, then you will not be charged for requests to that route.

You cannot turn workers on/off by user agent. However, you can use the new “User Agent Blocking” feature (under “Firewall” in your Cloudflare settings) to block certain user agents. This blocking happens before Workers run, so if a request is blocked, you will not be charged for it. However, in this case you would have to completely block googlebot, which may not be what you want.


#3

hi sir. this is not solve my problem.

yes, i’m need to block google, but not block to crawl my image, just block to run worker.

please add menu to exclude worker by user agent.


#4

Hi @kputera410,

I’m sorry that this doesn’t solve the problem. :frowning:

Sorry, but we do not plan to add a way to filter on user agents. Otherwise, logically, we’d also need to create a way to filter on method, and a way to filter on Referer, and so on, and eventually we’d have to build a way to write filters in JavaScript… and we’d be back where we started. :confused:


#5

hi sir.

if yo can’t add that custom filter, maybe you can add custom plan example, $25 or $30 per month without limit worker running. because without googlebot filter , my worker can running more than 100 million per month.

please, i’m sure you can solve this problem.


#6

I’d love to hear more about your traffic. It’s surprising to me that Google crawls your images tens of millions of times a month. Do you have a User-Agent breakdown from your analytics?


#7

hi sir.

thats is because i’m create social gallery blog. user can signup and add domain to my server. all of user gallery is linking from one domain image source, but from one image domain i’m add screet key to check image linking from which domain. and use clodflare worker to block image hotlinking when domain referer not match with screet key.

one user can have many more blog and i’m not limit domain. i’m very happy with cloudflare worker, but after cloudflare release price type with pricing per 1 million request that is new problem for me.

i have more than 10 million images on storage, one user can add many more domain on blog cms with linking to my images storage.

example 1 user with 1 domain maybe linking to 50K images. 1 domain have images sitemap to add on google webmaster. google will be crawll all images at this day and tomorrow googlebot can crawl againt the same images, and tomorrow againt and againt.

that sample just 1 domain with linking 50K images, and as i’m said user can add many more domain, user can linking many more image on storage, and i have more than 1K user.


#8

hi sir.
this is statistic from cloudflare, before i’m change my images domain.

and that statistic is just with 15 member use image gallery, and at this time i have 1284 user.

and look this geography:

do you look, from 83 million, 78 millions is googlebot.

please give me solution


#9

Interesting! Have you considered asking Google to crawl it less frequently? https://support.google.com/webmasters/answer/48620?hl=en


#10

hi sir.

if i’m set google crawl rate to slow, image will not fast index. and user will be not use my service.

this example, if one month worker running 200 million i must pay $100 just for worker :scream: and my worker is just for block hotlinking :tired_face:

please help me.


#11

hi @zack and @KentonVarda

please help me.


#12

If you’ve got 200 million assets stored you might consider taking advantage of some of our additional Enterprise features and the flat rate pricing it provides.


#13

Hi @kputera410,

I’m sorry, but I don’t think we have a solution here. We do not have plans to implement additional triggers for selective worker execution. Sorry.


#14

hi sir.
for enterprise plan (Unlimited number of scripts supported.) this is not unlimited worker request right?


#15

hi @KentonVarda,

maybe you can set new price, example:
$5 include first 10 million request.
Additional requests billed at :
for next 50 million request is $0.5 / million request.
for next request is $0.3 / million request.

OR

create additional plans for worker, like page rules, example:
$60/month and can running worker up to 200 million request.
that price like $0.3/million request.


#16

Hi @kputera410,

Thanks for your feedback. At this time we don’t anticipate a change to the pricing, but we’ll keep this case in mind if and when we revisit it in the future.


#17

You’d need to contact the sales team for pricing discussions. Generally enterprise would include flat rate pricing based on estimated traffic tiers.


#18

@cscharff Since @kputera410 expressed that $100 was too high a price for Workers, I don’t think an enterprise plan is a viable solution here…


#19

hi @KentonVarda,

my problem is not about pricing, my problem is if im pay worker and 80% worker running is by googlebot.

i’m can use enterprise if you can make me believe that with an enterprise plan can run “workers” indefinitely.


#20

This topic was automatically closed after 14 days. New replies are no longer allowed.