How to solve connectivity problems when running parallel code in R, with the 'parallelly' package? #510
-
I did tests with the Parallel and Future packages, in which I run a function with different data, on my windows 10 machine. That is, this local cluster is made up of the cores of my processor, which is an i7. Specifically, what I did was, working with Parallel, the following:
That way I test parallelization and had no issues. Now, I want to perform those same tests with a cluster made up of a VPS and my windows 10 machine. In this case, the idea is that my windows 10 machine is the Master and that the VPS is the worker. Now come the complications because I am not sure what to do, to achieve the aforementioned. And I can't establish communication between the machines. I have a VPS, with Ubuntu 20.04 (64 bits), of which the provider gave me, on the one hand, the IP address, the user (root) and the password. Also, it has RStudio installed, therefore, I have the URL, username and password to start RStudio. By the way, with sys.info () I get the nodename and user, which I don't know where to use them or if I'm using them right. Here are the questions: 1.- How do I tell the cluster that my windows 10 machine is going to be the master and the VPS the worker ?. Will it be something like the following? (working with Parallelly using, for example, makeClusterPSOCK):
2.- How do I do so that once my script is executed on my windows machine, these two machines can “establish communication”? On the side of my windows machine I downloaded and installed puTTY, and I configured it, placing the IP address of the VPS, to access the VPS machine as a terminal. And with root and password I can use ssh commands remotely on the VPS. I understand, then, that I already have SSH installed on the VPS, right? (My knowledge is null in everything I am doing). And of course I already have ssh on my local machine. 3.- On the VPS side, what else do I have to do? I don't need to start Rstudio or the VPS terminal, right? 4.- Of the arguments that I have below, what things will be wrong?
OUTPUT IF I RUN THIS SCRIPT ON MY WINDOWS MACHINE: R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)
R es un software libre y viene sin GARANTIA ALGUNA.
Usted puede redistribuirlo bajo ciertas circunstancias.
Escriba 'license()' o 'licence()' para detalles de distribucion.
R es un proyecto colaborativo con muchos contribuyentes.
Escriba 'contributors()' para obtener más información y
'citation()' para saber cómo citar R o paquetes de R en publicaciones.
Escriba 'demo()' para demostraciones, 'help()' para el sistema on-line de ayuda,
o 'help.start()' para abrir el sistema de ayuda HTML con su navegador.
Escriba 'q()' para salir de R.
> library("parallelly")
> #options(parallelly.debug = TRUE)
> sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)
Matrix products: default
locale:
[1] LC_COLLATE=Spanish_Venezuela.1252 LC_CTYPE=Spanish_Venezuela.1252 LC_MONETARY=Spanish_Venezuela.1252
[4] LC_NUMERIC=C LC_TIME=Spanish_Venezuela.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] parallelly_1.25.0
loaded via a namespace (and not attached):
[1] compiler_4.1.0 parallel_4.1.0 tools_4.1.0
>
> Sys.info()
sysname release version nodename machine login
"Windows" "10 x64" "build 19041" "DESKTOP-HR15LKG" "x86-64" "jlmal"
user effective_user
"jlmal" "jlmal"
> ## NOTE: Drop 'dryrun = TRUE' below in order to actually connect. Add
> ## 'verbose = TRUE' if you run into problems and need to troubleshoot.
>
> #First, lets tell R what port we want to use:
> #define port
> Sys.setenv(R_PARALLEL_PORT = 11000)
>
> #check that it
> Sys.getenv("R_PARALLEL_PORT")
[1] "11000"
>
>
>
> #Now, we need to open the selected port in every computer of the network.
> #sudo ufw allow from 10.42.1.0/24 to any port 11000
>
> # Now we have to create an object defining the IPs of the computers in the network,
> # the number of cores to use from each computer, the user name, and the identity of the director.
> # This will be the spec argument required by parallel::makeCluster() to create the cluster
> # throughtout the machines in the network. It is a list of lists, with as many lists as nodes
> # are defined. Each sub-list has a slot named host with the IP of the computer where the given node is,
> # and user, with the name of the user in each computer.
>
> Sys.which("ssh")
ssh
"C:\\Windows\\System32\\OpenSSH\\ssh.exe"
> Sys.which("plink")
plink
"C:\\PROGRA~1\\PuTTY\\plink.exe"
>
> ## EXAMPLE: Three remote workers
> ## Setup of three R workers on two remote machines are set up
> workers <- c("localhost", "194.163.134.235")
> cl <- makeClusterPSOCK(workers, dryrun = TRUE, homogeneous = FALSE,autoStop=TRUE, tries=1, delay=10,
+ validate = TRUE, verbose = TRUE, master=Sys.info()[["nodename"]],
+ timeout = 10, rscript= "Rscript",rscript_args = c ("- e", shQuote ('setwd ("/ ruta / a")')),
+ # rscript_envs =Sys.setenv(),
+ rscript_libs= "*", methods= TRUE, useXDR = TRUE,
+ # outfile== "salida"
+ renice = 2, rshcmd = c("<rstudio-ssh>", "<putty-plink>", "<ssh>"),
+ rshopts = c("-i", "~/.ssh/my_private_key"), user="georges",
+ revtunnel = TRUE,
+ # rshlogfile= TRUE,
+ # rank=
+ manual = TRUE,
+ quiet = FALSE,
+ # setup_strategy="parallel",
+ # action =
+ )
[local output] Workers: [n = 2] ‘localhost’, ‘194.163.134.235’
[local output] Base port: 11000
[local output] Creating node 1 of 2 ...
[local output] - setting up node
[local output] - attempt #1 of 1
----------------------------------------------------------------------
Manually, start worker #1 on local machine ‘localhost’ with:
nice --adjustment=2 "Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods - e "setwd (\"/ ruta / a\")" -e ".libPaths(c(.libPaths()))" -e "workRSOCK <- tryCatch(parallel:::.slaveRSOCK, error=function(e) parallel:::.workRSOCK); workRSOCK()" MASTER=DESKTOP-HR15LKG PORT=11000 OUT=/dev/null TIMEOUT=10 XDR=TRUE
[local output] - collecting session information
[local output] Creating node 1 of 2 ... done
[local output] Creating node 2 of 2 ...
[local output] - setting up node
[local output] - attempt #1 of 1
[local output] Will search for specified 'rshcmd' types: ‘<rstudio-ssh>’, ‘<putty-plink>’, ‘<ssh>’
[local output] Found the following available 'rshcmd':
[local output] 1. ‘C:\PROGRA~1\RStudio\bin\MSYS-S~1\ssh.exe’ [type=‘rstudio-ssh’, version=‘OpenSSH_5.4p1, OpenSSL 1.0.0 29 Mar 2010’]
[local output] 2. ‘C:\PROGRA~1\PuTTY\plink.exe’, ‘-ssh’ [type=‘putty-plink’, version=‘plink: Release 0.75; Build platform: 64-bit x86 Windows; Compiler: clang 12.0.0 (https://github.com/llvm/llvm-project/ d28af7c654d8db0b68c175db5ce212d74fb5e9bc), emulating Visual Studio 2013 (12.0), _MSC_VER=1800; Source commit: c72200ff8851b0d95574b8a8a88a2780a243c66c’]
[local output] 3. ‘C:\Windows\System32\OpenSSH\ssh.exe’ [type=‘ssh’, version=‘OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2’]
[local output] Using 'rshcmd': ‘C:\PROGRA~1\RStudio\bin\MSYS-S~1\ssh.exe’ [type=‘rstudio-ssh’, version=‘OpenSSH_5.4p1, OpenSSL 1.0.0 29 Mar 2010’]
----------------------------------------------------------------------
Manually, (i) login into external machine ‘194.163.134.235’:
"C:\PROGRA~1\RStudio\bin\MSYS-S~1\ssh.exe" -R 11001:DESKTOP-HR15LKG:11000 -l georges -i ~/.ssh/my_private_key 194.163.134.235
and (ii) start worker #2 from there:
nice --adjustment=2 "Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods - e "setwd (\"/ ruta / a\")" -e ".libPaths(c(.libPaths()))" -e "workRSOCK <- tryCatch(parallel:::.slaveRSOCK, error=function(e) parallel:::.workRSOCK); workRSOCK()" MASTER=DESKTOP-HR15LKG PORT=11001 OUT=/dev/null TIMEOUT=10 XDR=TRUE
Alternatively, start worker #2 from the local machine by combining both step in a single call:
"C:\PROGRA~1\RStudio\bin\MSYS-S~1\ssh.exe" -R 11001:DESKTOP-HR15LKG:11000 -l georges -i ~/.ssh/my_private_key 194.163.134.235 "nice --adjustment=2 \"Rscript\" --default-packages=datasets,utils,grDevices,graphics,stats,methods - e \"setwd (\\"/ ruta / a\\")\" -e \".libPaths(c(.libPaths()))\" -e \"workRSOCK <- tryCatch(parallel:::.slaveRSOCK, error=function(e) parallel:::.workRSOCK); workRSOCK()\" MASTER=DESKTOP-HR15LKG PORT=11001 OUT=/dev/null TIMEOUT=10 XDR=TRUE"
[local output] - collecting session information
[local output] Creating node 2 of 2 ... done
>
> rscript_args = c("-e", shQuote("getRversion()"))
> rscript_args
[1] "-e" "\"getRversion()\""
> shQuote("getRversion()")
[1] "\"getRversion()\""
> OUTPUT IF I RUN THIS SCRIPT ON MY WINDOWS MACHINE, BUT WITH DRYRUN = FALSE:
OUTPUT IF I RUN THIS SCRIPT ON THE VPS WITH DRYRUN = FALSE:
5.- If I want the VPS is the master and my local machine is the worker. Do I only make this change?
Do I have to run the script from the Rstudio of the VPS? In that case, I don't have to run anything on my windows machine, do I? Just have it turned on ?. Well, as you may have noticed, I have a long way to go from a conceptual and also practical point of view to carry out the task I want to achieve. But I would really appreciate if you could give me some light with this, to see if I can establish automatic communication between the machines and be able to run parallel R code. In the following link there is information regarding the Parallelly package: https://parallelly.futureverse.org/reference/makeClusterPSOCK.html A big greeting |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 65 replies
-
Hi, I'll try to walk you through this. But first, you please tidy up the Markdown formatting of your post to use code-block formatting for all that output. Right now it's really hard to read because it's pasted in as regular text. |
Beta Was this translation helpful? Give feedback.
-
Thanks, I'm going to convert the text to markdown format.
El lun, 7 de jun. de 2021 a la(s) 21:11, Henrik Bengtsson (
***@***.***) escribió:
… Hi, I'll try to walk you through this. But first, you please tidy up the
Markdown formatting of your post to use code-block formatting for all that
output. Right now it's really hard to read because it's pasted in as
regular text.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#510 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANEXS6DOSUZ6F6DGSADHMLTTRUKUHANCNFSM46GSHSCA>
.
|
Beta Was this translation helpful? Give feedback.
-
Ready Henrik. I followed their instructions. I did exactly what you told
me. I already have the user named jose and the password that is also jose.
El mar, 8 de jun. de 2021 a la(s) 03:15, Henrik Bengtsson (
***@***.***) escribió:
… Before we even get starting with what needs to be done from R, let's
establish a few things first.
First, make sure you can ssh into your VPS machine using Windows 10's
built-in ssh client. This will allow you to not have to deal with PuTTY,
which will be much much easier for you. Please trust me on this one.
Assuming your VPS machine is 194.163.134.235:
- Run cmd to open a Windows Console
- Call ssh -l root 194.163.134.235
Second, and as any Linux expert will tell you, never ever log in as root
on a Linux system as part daily work (actually never). It's asking for
trouble because you can easily mess up the system with a few commands, and
more importantly, it's big potential security risk. So, your second task is
to create a new regular user, which you will use going forward instead of
root. If you don't know how to do this, ask the folks who gave you root
access to the VPS machine, or see for instance
https://www.makeuseof.com/linux-add-user/. You basically want to log into
the VPS machine as root and do something like:
# useradd jose# passwd jose
and the log out.
Third, verify that you can SSH to the VPN as that new user from your
Windows machine, e.g.
C:> ssh -l jose 194.163.134.235
by using the password of user jose that you entered for passwd.
After you've got this working, you should basically never log in as root
again.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#510 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANEXS6ES6CBRI35DC4ZKL73TRVVLJANCNFSM46GSHSCA>
.
|
Beta Was this translation helpful? Give feedback.
-
~~~
Good morning Henrik.
In the VPS, with
ssh -l jose 194.163.134.235 and then my password.
mkdir -p ~ / .ssh, I get the following message:mkdir: cannot create
directory '/ home / jose': Permission denied.
On my windows machine, placing C:> ssh-keygen. The key was generated correctly.
C:\>ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\jlmal/.ssh/id_rsa):
c:\Users\jlmal/.ssh/id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in c:\Users\jlmal/.ssh/id_rsa.
Your public key has been saved in c:\Users\jlmal/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:1VmMoMPrFECnqObvNosODibUywJZiM2SG2xixgZIV44 jlmal@DESKTOP-HR15LKG
The key's randomart image is:
+---[RSA 3072]----+
|+. ..oo . .. o. |
|*=. o. = . ..o. |
|BB+E... = . o |
|**.. = |
|+.o. S |
|oo. . o |
|+o.o . |
|=..oo |
| ooo+o |
+----[SHA256]-----+
C:> scp ~/.ssh/id_rsa.pub ***@***.***:.ssh/
Permission denied.
~~~
El mar, 8 de jun. de 2021 a la(s) 06:17, Henrik Bengtsson (
***@***.***) escribió:
… Good. Next step for you is to be able to log in to the VPS as jose
*without* having to type your password, e.g.
C:> ssh -l jose 194.163.134.235
...
{VPS}$
This can be done via a so called SSH public-key pair. First, make sure you
have a folder ~/.ssh in your jose account on your VPS machine, e.g.
C:\>ssh -l jose 194.163.134.235
Enter password: *******
{VPS}$ mkdir -p ~/.ssh
{VPS}$ chmod u=rwx,go= ~/.ssh
{VPS}$ exit
C:\>
Then, on your Windows 10 machine, create SSH keys:
C:> ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\alice/.ssh/id_rsa):
Created directory 'C:\Users\alice/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\alice/.ssh/id_rsa.
Your public key has been saved in C:\Users\alice/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:EVbS0inhfBC4CN91GJSbCbAb78aMiUsap6UMbzOx27I ***@***.***
The key's randomart image is:+---[RSA 3072]----+| .. oXX.. || . .o+*+= || ooo ++B. || o+o +o || . . S || .. * ||o =oo = ||oX*o . ||+oE*. |+----[SHA256]-----+
Third, copy the *public* part of the SSH key pair to the jose account on
the VPS machine:
C:> scp ~/.ssh/id_rsa.pub ***@***.***:.ssh/
If everything went well, you should now be able to log into the VPS
machine without entering the password. Verify with:
C:\>ssh -l jose 194.163.134.235
Let me know if it works. If you still have to enter the password, what
does:
{VPS}$ ls -la ~/.ssh
output (for the jose account)?
PS. Please comment via the GitHub web interface rather than reply to the
GitHub email. It looks like email replies mess up the comment thread.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#510 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANEXS6E7MDDPTBCAIYJFLD3TRWKWJANCNFSM46GSHSCA>
.
|
Beta Was this translation helpful? Give feedback.
-
Henrik,
I logged in to the VPS with the user georges1 and the password of that user.
And when logging in to the VPS again with ssh [email protected] it does not ask for the password.
And from there, my questions will begin |
Beta Was this translation helpful? Give feedback.
-
Henrik, |
Beta Was this translation helpful? Give feedback.
-
Hi Henrik,
Thank you for responding. Yes, the terminal. My R script would run fine in the console. I would then adjust the code to run in parallel:
flatten_chr(mlply(input, .fun = which_closest2, table=WWIS, .parallel = TRUE, .progress = "text")) (package “parallel” with function argument .parallel = TRUE)
and submit the job to the cluster sbatch BASH_Script.sh (10 cores and 24 GB RAM) and it would fail. It was only when I submitted the same job using MobaXterm that the job would run successfully.
Let me know if you can reproduce this same result on your end. Is it an authentication bug/issue from within RStudio’s terminal?
Regards,
Kevin
From: Henrik Bengtsson @.***
Sent: February 28, 2022 7:10 PM
To: HenrikBengtsson/future @.>
Cc: Kevin McDermott @.>; Comment @.***>
Subject: Re: [HenrikBengtsson/future] How to solve connectivity problems when running parallel code in R, with the 'parallelly' package? (Discussion #510)
Just to make sure, and to help future troubleshooters, when you wrote "RStudio's terminal", is that what you meant, or did you mean the RStudio Console?
|
Beta Was this translation helpful? Give feedback.
Henrik,
I'm ready to start working with parallel programming in R.
The procedure to establish communication without asking me for the password:
First, I deleted the users created in the VPS and created a new user called georges1.
Then, I went to the folder where the keys are on the windows machine:
I logged in to the VPS with the user georges1 and the password of that user.
And when logging in to the VPS again with ssh [email protected] it does not ask for the password.
I'm going to make a little script where…