Workers isolation for Enterprise

I was wondering if the level of isolation for Workers is increased under an Enterprise plan? I.e. does an Enterprise plan get a dedicated JavaScript runtime with hypervisor level isolation from other processes?

The shared-process model of ordinary workers makes it difficult for us to justify using workers for anything security/auth related.

Cheers!

I thought the whole idea of workers was that it provided enterprise security and isolation for everyone :yum:

The workers are “isolated” there is no access between other workers. Don’t know how this will increase more isolation.

“ Everything supported by current Google Chrome stable release.
eval() and new Function() are disallowed for security reasons.”

2 Likes

To add:

For all Enterprise questions, you can get a faster/more definite answer by either emailing support (if you are currently an Enterprise customer) or contacting Enterprise sales via any contact form (eg. the “contact sales” button on this page).


To answer your question, There is security “by design” with the way CF workers are implemented. Instead of afterthought-type security with hypervisors or process isolation or VMs, it uses the V8 “Isolates” which are inherently secure as a big goal of V8/isolates was to prevent different JS contexts (different websites) from accessing data of other open tabs (such as the cookie data for google.com).

A vulnerability similar to what you’re describing (requiring a hypervisor) would mean someone has broken Chrome’s JS engine, which in itself is extremely rare. To further mitigate potential 0-day vulnerabilities, CF prevents the use of eval and new Function().

I heavily recommend watching this talk by @KentonVarda:

One interesting bit:

29:00

We don’t have the ability to spin up a process for every single tenant, but we can spin up a process, like one for enterprise users, one for established users who have been paying for a while, and one for free users when they come. We don’t currently have a free plan, but if we were to in the future. Then, we can put additional isolation around those, we can put those in a container or in a VM or whatever else we want. So that makes it pretty hard for an attacker to just sign up and get something good.


More resources:

https://youtu.be/nsQdRnmvCmI

3 Likes

A few things:

  1. I second @Judge’s recommendation to watch the video. Very interesting, gives some nice quick insight into how the Worker foundation is implemented.

  2. I do understand @squiggly’s concerns though. First of all, regardless of any possible issues with the implementation itself (see later), there still are issues like Meltdown which have the potential to circumvent any security measures in place. This is bad for your average Wordpress blog about baby diapers but is a serious issue for anything more critical, which Enterprise plans are likely to run. Secondly, Enterprise customers do pay dearly and should be able to expect more dedicated resources than us common folks.

  3. I am afraid, I do think however that Kenton dismissed the security issues too easily (sorry, @KentonVarda :slight_smile:). What I took from his talk (and I applaud him being open about it) is that Workers are - because of their V8 foundation - simply potentially insecure. I understand there is only so much Cloudflare can do (short of not offering Workers altogether) but they seem to be a very attractive attack vector for now. I guess even the Enterprise costs wont pay for a dedicated machine in every datacentre, but I still think (and hope this is already in place) these customers should not only run in separate processes but on separate machines altogether.

  4. @squiggly, I believe @Judge’s advice to contact sales is the best approach. They should be able to give you a quick and definitive answer to your concerns, better than all the guessing you will get here :slight_smile:

Thanks for the replies. I will consider the feedback and raise a question with Cloudflare sales directly.

There was a time when people argued that virtual machines were “insecure”. And it’s true: compared to dedicated physical machines, VMs are “less secure”. Sometimes hypervisors have security bugs that allow a guest to break out and take over the system. And many variants of Spectre let you read secrets out of a neighboring VM. (Most of the publicly-known variants have been solved in one way or another, but there are surely other variants not disclosed publicly that remain fully exploitable.)

But what is the answer to that? The public cloud relies on virtual machines. Do we throw the whole thing away because it is “insecure”? Of course not.

The thing is, security is not a boolean. There’s no such thing as a “secure” system or an “insecure” system. Instead, we need to talk about risk management. Every system has some level of security risk. Some systems have more risk than others. Often, there are additional measures you can take to reduce (or, if you aren’t careful, increase) overall risk involved in using some particular system. These risks need to be weighed against the advantages. Most people now agree that the advantages of the public cloud are profound and worth the additional risk.

Isolates bring some new risks – and also reduce some risks. On one hand, V8 is a complicated system with a large attack surface, and as a result sees a higher rate of bugs than hypervisors do. On the other hand, Workers are provided as JavaScript and Wasm rather than native code, and are recompiled by a trusted compiler; this helps defend against many kinds of possible attacks on the CPU architecture and operating system kernel. Also, rolling out security patches to V8 is cheap – we can push a patch worldwide within a couple hours. Hypervisor bugs are much more difficult to patch since rebooting (or live-migrating) guest VMs is disruptive/expensive.

So the relative risk is not completely clear; there are pros and cons. Meanwhile, Isolates enable us to deploy your code in every single Cloudflare location, quickly and cheaply. With virtual machines, this simply wouldn’t be possible – at least, not at a price point accessible to any but the largest enterprises.

To address your specific question, @squiggly, I can tell you that enterprise workers do not run in the same process and free workers. However, they do still share a process with other customers. (I don’t think I can go into more detail on how we divide up users at this time – the details are in flux anyway.)

Please see my talk for more technical details, including risks and risk reduction strategies. :slight_smile:

3 Likes

Judge already posted that very video and that was the video I commented on. No offence, Kenton, but you basically rehashed the points you made in the video.

Security might not be a boolean, but it should be pretty close to it and we should strive for it to be one. I also believe saying “nothing is secure” is making it too easy. It is true, there is always the risk of a system - believed to be secure - to actually contain a major security issue. But I’d argue there is a difference between using a system, which you assume to be secure until there is an actual issue discovered, and using a system which has security issues you are actually aware of. You addressed a V8 bug in your talk, which appeared to be quite serious. I cant tell, whether that issue has been fixed by now - or was actually already fixed at the time of your - talk, but if you run/manage/deploy systems which such an exploitable bug you open up yourself (and customers) to an unnecessary risk.

I’d also argue the argument “we have to risk it otherwise we couldnt provide that service” is a bit shaky. I understand the desire to offer the latest and shiniest cutting-edge technology but that should still not be at the risk of security.

Again, I am not saying the worker platform is necessarily insecure, but - as I mentioned before - I do believe it can be potentially an attractive attack vector and from this perspective I fully understand if people - particularly Enterprise customers - are asking for more or better isolation.

@sandro Whoa whoa… Of course that V8 bug has been fixed! We would never let a bug like that stay in production unpatched. In this case we had deployed the fix several weeks before the bug was publicly disclosed. When bugs like this happen, patching them is obviously priority #1. Sorry, but I thought that went without saying.

The “risk” I’m talking about is the risk of bugs that we don’t know about yet. Every system has that risk.

4 Likes

Apparently I communicated really poorly in my talk. :frowning: If it were actually the case that we were letting known security bugs fly in production, then you would be totally right to stay away from the product. But we absolutely wouldn’t. I’m sorry I wasn’t clearer about that.

Just want to be clear on this too: Workers is not vulnerable to Meltdown.

3 Likes

I don’t think that you where unclear at all, however, when it comes to security people want to confirm, not on assumptions, but in writing, that it’s as secure as it can get. Preferably more than one time too.

I would have assumed the bug was fixed, but it was not clear to me and there sometimes are issues which are difficult to patch right away → Meltdown, Spectre, etc.

Thanks for clarifying that, @KentonVarda.

Sure, of course. That is very difficult to impossible to rule out.

:+1:t2:

I just want to stress once more, my point was not trying to point out how secure or insecure workers might be, rather that I can understand someone - especially if they are on the highest plan - might want/require physical separation of their accounts, in order to minimise the risk of those very unknown future bugs.