Pretty interesting, it seems like Cloudflare will start to cache non-html files by default.
The account/org that received this email is on the enterprise plan, but it’s not exactly clear if this just applies to our enterprise zones or for all.
What mostly got my attention here is that they will even start to cache mp4 files, while the Cloudflare Workers’ ToS does not allow making use of video files and instead recommends to use Cloudflare’s Stream/video product.
I also noticed a bunch of media files in that list. Why the change? It’s not like they’ve suddenly acquired a bunch of extra cache storage, right? Maybe they’ve made adjustments to their algorithms for what’s worth caching. Though I do wonder if it really is limited to ENT only. That would make sense.
They would always cache those files with a Cache Everything rule, this change is just that they now cache these by default. They can always evict when needed!
It could easily be that by not caching these file types by default that Cloudflare are using excessive resources to continually fetch them from a slow origin.
I always assumed that a small video (background or promo reel) on a site was not really an issue. Running a MindGeek (NSFW) clone was a major problem.
The Free plan additionally supports APK, EXE, DMG, BIN, ISO, ZIP, RAR, ZST, TAR, BZ2, 7z, GZ, MP4, MKV, AVI, WEBM, MP3, OGG, and FLAC file extensions.
It still seems a bit weird to me that Cloudflare says they will cache video files by default for the free plan, while saying in their ToS that it’s not even allowed in the first place, unless I’m just misinterpreting it.
Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service or expressly allowed under our Supplemental Terms for a specific Service.
I think the ToS refer to serving video through the proxy, so it doesn’t make a difference whether it is cached or not. Having the cache level set to bypass does not allow serving any more video than having it cached, from my understanding.
Yeah exactly, which makes it more confusing, especially if they explicitly state in the developer documentation that the free plan can just cache/use those video files.
The documentation seems to contradict what the ToS says.
P.S
Seems like the ToS is in this case referring to both caching and serving, if I’m understanding it correctly
The Services are offered primarily as a platform to cache and serve web pages and websites.
I don’t think so. ToS doesn’t really care if it’s cached or not. There’s nothing wrong with the occasional binary file, but if they keep promoting Bandwidth Alliance for buckets, and this caching of media files, it’s going to be difficult to justify enforcement of ToS.
I’d second that.
Sure, you could also enforce caching and while the terms of service originally referred to caching they are now more generic about serving media content, but that’s actually not even the point.
Cloudflare is relatively adamant that its service is not for serving media files (and is actually relatively vague on that interpretation) but at the same time starts caching them by default. That is somewhat of a contradiction in my book.
Gen 11 AMD EPYC Milan servers related maybe ?
But yeah interesting for a lot of non-html files which would be subject to possible TOS infringements!
Maybe TOS will be changing soon? Hope they have some sort of meter/guage for non-html traffic/bandwidth as a ratio to HTML traffic/bandwidth to allow CF customers to gauge their own usage and they can have alert visual indicator to tell CF customers if they’re coming close to triggering a TOS violation. This would also invalidate any excuse from CF customers that they didn’t know or were unaware when you have an early warning indicator for such.
Then when folks query CF TOS for this, support can just say check the guage/ratio indicator to be sure you’re ok and if you get a warning, contact CF support to discuss.
The new servers have 4TB storage, and in video terms that would not last long. My business is a tadpole, and we are looking at putting 24 x 12.8TB NVMe drives in servers for video caching. Even if Cloudflare use all the storage in a PoP/Pod/Battery of servers, one Tube site would consume all the storage pretty quickly.
Ah yes, wonder what’s CF’s disk cache storage size breakdown like for file extensions 
wow 307TB of storage!
The price would probably get a bigger WOW! But the general thinking is that if you are delivering 100Gbps with a 99% cache hit rate, you still need 1Gbps of cache fill. Drop the cache hit rate to 90% and the cache fill rate starts to get interesting, especially if you are filling from spinning rust. Nothing in life is free, but having all the content on NVMe at the edge or mid-tier probably makes sense.
I am not even close to this storage size in total as if I put all my external (including internal) hard drives at home together 
Hopefully, it does not get some bad ransomware encoding all that (I think it would quit the work right after determining the total storage space needed to be encoded) 
At home, with actual disks, same. Online, on the other hand, I got way more than that…
At work I have many times that, just in NAS storage, but then you need to add VM storage, computers, etc.
I hope they have backups. I expect they do (which would be a great thing to know what a big tadpole business does in that regard, @Michael, if you can share).
Does anybody still use tape backup?
Many still do, for large storage arrays. The big cloud vendors even offer it in some capacity, Amazon does for their cheapest storage options (at least they say they do).
My personal storage is close to zero. I have two external backup drives at home, but everything else syncs to the cloud. There is no removable media in the house at all (except for some flash drives in my wife’s cameras), and when we moved house last weekend all the old DV tapes finally got degaussed and binned! We don’t even have a DVD/CD player, so no point having CDs or DVDs.
No easy answer, as it depends.
Most corporate backups (VMs, DBs, files etc) are made to disk, disk, tape, offsite tape and Cloud, with the retention policy in each being slightly different. Disk is mainly for rapid restore so retention is short, while Tape and Cloud can be used for longer term storage. Cloud is a last resort source of backups, due to cost. (More on that later). There are relatively few different data protection policies. Two years ago we refreshed the whole data protection environment, and it was easier and more reliable to have a simple one-size-fits-all policy. The way our backup solution works means that a system without much changes does not take much time or consume much space, so why not backup test systems the same as production systems?
Media is tricky, purely because of the sheer volume. We operate with media files that are between 50 and 200 Mbps. We generally dump two copies to tape on acquisition, and one of those tapes heads to an external facility for DR. When needed they get copied to disk. For production assets (those being worked on in post-production, transmission etc), we replicate the disks. Part of the logic being that restoring from anywhere would take too long, so unless a nuclear bomb hits, having the data on disk is worth the cost. We are currently nearing completion of a large object store using erasure coding for protection and replicating that data to the cloud, and tape will disappear totally from the media production environment.
For streaming, the assets are compressed to a set of assets with a maximum of 8Mbps. On the Origin streaming servers the disk is duplicated, but not backed up. If needed, we can regenerate the compressed files from the master, which is still pretty slow but taking backups would not really be worth while. (We had a fun weekend some years ago when a f**k-up deleted all the origin copies. We were able to regenerate all the origin copies before anything expired from the CDN cache, so nobody noticed. That would not be possible today, just because there is a constant volume of cache-misses hitting origin disk).
Is there good ransomware? The NVMe use case above is for cache, not origin storage, so ransomware is not really an issue. It is to stop content being read from disk where possible.
Unfortunately, yes. I’m sure there are some businesses where the cost of disk or cloud backup is a non-issue. I know a local health insurance company who only have about 150TB of total business data, so the cost of disk is essentially irrelevant. For us, the data egress cost of restoring all our data from Glacier type storage would still be significant, even if the data is not worth that much. I’m currently trying to see if our insurance company would cover the cost of restoring from cloud in the event that we lost most or all of our data. If they will, then tape will be done away with for backups.
Same, I do have old readers for those, but I can’t remember the last time I used them. New computers at home and the office have no optical drive. Some still use USB drives, but I hate them. If only the damn FTTH line with 500Mbps up were to come to my house…
Same! I just use two different directories, just to eliminate some retention in case of the test systems.
For production data I have an HA RAID storage array, which backups up locally pretty often to prevent accidental deletions and ransomware (it’s the main issue) and then daily backups to 2 cloud locations with basically unlimited retention (GDPR and such, but in those limits unlimited) for all other issues. All encrypted before upload, of course.
The other locations backs up its own data to the main office and the cloud, just in case.
I am luckily talking about a few TBs of data, plus VM backups. Not much to consider cost wise.
This is what I predict as well. The policy has historically been a means of preventing people from using Cloudflare at a disproportionate amount compared to other customers (usually when the only purpose of doing so is to get free bandwidth savings on media files), but with how many new network peers CF continues to add, the upstream transit costs might no longer be a real problem.
Or they can simply open up an extra tier of billing for non-html files via CF CDN edge servers instead. Maybe after a certain amount of traffic/bandwidth scaled quotas based on CF plans. So CF could determine that it would be ok for each CF free, pro, biz, enterprise plan to have say 1GB, 3GB, 5G, 10GB included quota for non-html files served via CF CDN edge and they offer up subscription paid extras to go over that quota?
For everyday regular sites that quota would be enough for the occasional non-html image to keep to TOS and have certainty that the customer isn’t breaching TOS.
Then in no uncertain terms, it conveys if you are serving a lot of non-html files, you will have to pay for it either way i.e. via Cloudflare Stream or via paid subscription upgraded quotas. In would be an additional revenue raising method.
