-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(workflow): Fix multiple context leaks in reuseV8Context
executor
#1519
fix(workflow): Fix multiple context leaks in reuseV8Context
executor
#1519
Conversation
44f477c
to
04810fb
Compare
reuseV8Context
executorreuseV8Context
executor
I'm a bit confused about what is the ultimate goal and why this is needed. If we want to eliminate all side channels this is not going to work, e.g., you could have timing side-channels, and the v8 VM is not designed to eliminate side-channels. If the goal is that other untrusted code is running in the same reused v8 context and attacking other users by modifying the context, I'm not sure we are protecting all the primordial classes (a String, Number,...) either, and in any case, that's a very hard problem... Is this just to help prevent some mistakes, nothing to do with security? |
No, that's really not a security thing (that's simply impossible AFAIK with Node's The However, there's been some reports of users doing uncommon but still legitimate and reasonable things in their workflow code, that would result in context from one workflow execution leaking into another workflow execution. Depending on use cases, that may result in non-deterministic behaviors, memory leaks, or some other oddities. For example, one user recently reported that they were mocking functions in the
The problem here is that previously, when leaving a workflow's execution context, the Reusable VM's would not touch any global variable that existed right after start. In this case, that means that it would leave the Once you know it, it's quite easy to fix their code to avoid this issue (they did), but it's also an opportunity for us to make our engine more reliable. |
There's also one test, "Shared global state is frozen", that existed before this PR that was actually not testing what we thought it was testing. That is, the test was asserting attempting to add a custom property to some global object from Node's default vm context would be rejected. However, it was doing so by adding something to the This PR fixes that bug; the test is now asserting the proper behavior against the |
packages/test/src/test-isolation.ts
Outdated
(setTimeout as any).a = 1; | ||
(Array as any).a = 1; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you change this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That test was presumably testing that v8's built-in functions were not mutable, but the SDK redefines setTimeout
, so that test was actually useless. Turns out in fact that built-in functions were unexpectedly mutable.
packages/test/src/test-isolation.ts
Outdated
env.client.workflow.execute(sharedGlobalReassignment, { taskQueue, workflowId: randomUUID() }), | ||
env.client.workflow.execute(sharedGlobalReassignment, { taskQueue, workflowId: randomUUID() }), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This may be racy in CI and may not actually provide the test coverage you want. There's nothing guaranteeing that the workflows will run concurrently.
*/ | ||
readonly contextKeysToPreserve: Set<string>; | ||
private _context?: vm.Context; | ||
private pristine?: Map<string, PropertyDescriptor>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docstring.
const workflowModule: WorkflowModule = new Proxy( | ||
{}, | ||
{ | ||
get(_: any, fn: string) { | ||
return (...args: any[]) => { | ||
Object.assign(context, bag); | ||
for (const [pname, pdesc] of bag.entries()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: would have liked to prevent constructing the entries array for every time we call into the VM.
// Looks like Node/V8 is not properly syncing deletion of keys on the outter context | ||
// object to the inner globalThis object. Hence, we delete them from inside the context. | ||
context.__TEMPORAL_ARGS__ = keysToCleanup; | ||
vm.runInContext( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will add a bit of overhead for every time we enter the context, I would try to find an alternative way to do this.
04810fb
to
0bd389b
Compare
The code changed significantly since the first round of reviews. I'm closing this PR and opening a new one. |
What was changed
Fix multiple context leaks and bugs in the
reuseV8Context
executor:…by reassigning a new object to an existing shared global variable:
globalThis.console = { ...globalThis.console, wfId: workflowInfo().workflowId }
(fixes [Bug] Reusable VM allows context leak due to global variable reassignment #1476);…by modifying one of Node's built in global objects:
globalThis.Number.a = workflowInfo().workflowId
;…by deleting a previously set global variable:
globalThis.a = 1; await sleep(1) ; delete globalThis.a ; await sleep(1) ; globalThis.a = (globalThis.a || 0) + 1; /* globalThis.a is 2 rather than 1 */
…by defining global symbol properties:
const mySymbol = Symbol.for('...'); globalThis[mySymbol] = ...
(fixes [Bug] Reusable VM allows context leak due to global symbol properties #1592).