Error: callr subprocess failed: could not start R, exited with non-zero status

[ssobelvw]

I will need to spend time creating a reproducible example if required to help, but I am getting an error "Error: callr subprocess failed: could not start R, exited with non-zero status". Not sure if this is a bug with future.callr, or maybe more likely, with me or my machine, so I'm appealing for some help here... thanks in advance.

I am using an AMD Ryzen 9 5950 16-core, 32-thread machine, 32Gb RAM, and I started a process using future.callr using plan(callr, workers = 31). I would expect the whole job to run for about 9 hours, but after 1.5 hours I got the error above. I tried running a similar script on another less powerful computer, with a 2-year old Intel i5 chip with 4 cores/8 threads, that has 12Gb RAM, but using doParallel with makeCluster(7) and it ran successfully. Could my error be a RAM issue? (31 jobs divided into 32 GB RAM is a little more than 1GB RAM per callr R session, whereas 7 clusters into 12GB is 1.7 GB per cluster?)

Not sure if this is enough information to help, let me know what other info would help if not.

FYI, I tried running the same R script using parallel processing, but using doParallel, with makeCluster(31) on the more powerful machine above, and got an error "unserialize(socklist[[n]]):error reading from connection", so not sure if these two errors are related, and/or have to do with this particular machine (AMD Ryzen 9 5950), or available RAM, or what.

Any help would be greatly appreciated! I know this is a future.callr Github repo, but if anybody has any advice that would help fix my doParallel error I would welcome that too, either way.

[HenrikBengtsson]

Hi, I transferred this to the future discussion forum. Before anything else, what's your operating system and what version of R are you running?

[ssobel]

Hi Henrik, thanks for your prompt response and pardon my late response!

I am using Windows 10, my R version is 4.0.5 (2021-03-31) and RStudio version is 1.4.1106.

I was taking the past several days either working and being tired, or doing more testing. As a baseline I can run my script using no parallelization just fine, but I have tried the following parallization methods, and get these errors:

future.callr : "caller subprocess failed: could not start R, exited with non-zero status"
doParallel : "Error in unserialize(socklist[[n]]) : error reading from connection"
doSNOW : "Error in unserialize(socklist[[n]]) : error reading from connection"

The errors from using each of these methods occur after awhile of running, i.e. it doesn't crash right away, it crashes after completing many iterations, then crashes. I can't seem to figure out what iteration it crashes, but as I mentioned, not using parallelization runs just fine and finishes all the iterations perfectly. But the point of parallelization is to speed things up of course.

Would you have any idea of what might be causing this? If you give me an example function to iterate over I can run any tests you might think are helpful, or I can try to create a small reproducible example. Very, very, much appreciate your help or any insight you might have on this!

Thank you.

[HenrikBengtsson]

Hi. Yes, it's hard to say something specific without a reproducible example. If you could make your code available online, that might help.

So, both:

Error: callr subprocess failed: could not start R, exited with non-zero status

with plan(future.callr::callr) and

unserialize(socklist[[n]]):error reading from connection

when using doParallel::registerDoParallel(cl) with parallel::makeCluster(n) suggest that the R worker crashes. So, it could very well be for the same reason. OTH, the callr error indicates that it cannot even start successfully. Maybe you end up running out of memory.

FWIW, doSNOW and doParallel are basically the same parallel backend.

BTW, when you say you're using future.callr, are you using this with foreach, i.e. something like:

library(doFuture)
registerDoFuture()
plan(future.callr::callr, workers = n)
res <- foreach(...) %dopar% {
}

or do you use something else than foreach?

I would definitely try to troubleshoot this outside of the RStudio Console using a plain R session, i.e. by running R (aka Rterm) in the Windows Console. That would remove RStudio from the equation. It's not unheard of that people have problems running things from RStudio while it works in plain R.

[ssobelvw]

Hi Henrik, thanks for your email and continued patience helping me! I am currently running a long script that will run for several more days but then I can test it and provide a reproducible example. In the meantime let me see if I can answer your question… I am using the foreach construct with future.callr as you mentioned, like this: library(doFuture) registerDoFuture() plan(future.callr::callr, workers = n) res <- foreach(...) %dopar% { } Thanks for your tip… I will try to run it using plain R outside RStudio. Take care! From: Henrik Bengtsson ***@***.***> Sent: Wednesday, June 9, 2021 12:08 AM To: HenrikBengtsson/future ***@***.***> Cc: Scott Sobel ***@***.***>; Author ***@***.***> Subject: Re: [HenrikBengtsson/future] Error: callr subprocess failed: could not start R, exited with non-zero status (#493) Hi. Yes, it's hard to say something specific without a reproducible example. If you could make your code available online, that might help. So, both: Error: callr subprocess failed: could not start R, exited with non-zero status with plan(future.callr::callr) and unserialize(socklist[[n]]):error reading from connection when using doParallel::registerDoParallel(cl) with parallel::makeCluster(n) suggest that the R worker crashes. So, it could very well be for the same reason. OTH, the callr error indicates that it cannot even start successfully. Maybe you end up running out of memory. FWIW, doSNOW and doParallel are basically the same parallel backend. BTW, when you say you're using future.callr, are you using this with foreach, i.e. something like: library(doFuture) registerDoFuture() plan(future.callr::callr, workers = n) res <- foreach(...) %dopar% { } or do you use something else than foreach? I would definitely try to troubleshoot this outside of the RStudio Console using a plain R session, i.e. by running R (aka Rterm) in the Windows Console. That would remove RStudio from the equation. It's not unheard of that people have problems running things from RStudio while it works in plain R. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HenrikBengtsson_future_discussions_493-23discussioncomment-2D843283&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=7cY07-9VVQAidUevYmVxqc_cR7lRW_NCQOCUTK89yYM&m=LN4SXPWOtgf7e5FG0_MZgQCa_9uaWRjurhwGxG7ok6w&s=leaS3g6_sH7YCevLVB5CRDjcMkKrjGZnp15iNcvOl8I&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ATYACKE6O2XE6TVPAEVHIQLTR3SJPANCNFSM45OIOX6Q&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=7cY07-9VVQAidUevYmVxqc_cR7lRW_NCQOCUTK89yYM&m=LN4SXPWOtgf7e5FG0_MZgQCa_9uaWRjurhwGxG7ok6w&s=vExqZQEachayOQBr7REs-NjofvTphs7e9itN_9_UBl0&e=>.