From 887642bb55461d3f1b758dcb347637961888807a Mon Sep 17 00:00:00 2001 From: dwelsch-esi <116022979+dwelsch-esi@users.noreply.github.com> Date: Wed, 18 Jan 2023 18:57:02 -0800 Subject: [PATCH] Documentation: Self-Deployment (#1482) --- CHANGELOG.md | 9 +- doc/source/deployment/deploy_with_docker.rst | 66 ++++ doc/source/deployment/deploy_with_systemd.rst | 116 ++++++ doc/source/deployment/index.rst | 334 ++++++------------ doc/source/index.rst | 1 + 5 files changed, 293 insertions(+), 233 deletions(-) create mode 100644 doc/source/deployment/deploy_with_docker.rst create mode 100644 doc/source/deployment/deploy_with_systemd.rst diff --git a/CHANGELOG.md b/CHANGELOG.md index d33cca556..f8d643404 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Load plugins only when COVALENT_PLUGIN_LOAD environment variable has been set to a Truthy value. +### Docs +- Published Self-Deployment Guide + ## [0.213.0-rc.0] - 2023-01-18 ### Authors @@ -83,10 +86,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Co-authored-by: ArunPsiog - Co-authored-by: Alejandro Esquivel -### Fixed +### Fixed -- Optimization of logs on the GUI for large log file sizes. -- Fixed UI pagination not working for more than 11 pages +- Optimization of logs on the GUI for large log file sizes. +- Fixed UI pagination not working for more than 11 pages - Runtime field counting down for select running dispatches ## [0.211.0-rc.0] - 2023-01-10 diff --git a/doc/source/deployment/deploy_with_docker.rst b/doc/source/deployment/deploy_with_docker.rst new file mode 100644 index 000000000..1d7bbc7a2 --- /dev/null +++ b/doc/source/deployment/deploy_with_docker.rst @@ -0,0 +1,66 @@ +Deployment with Docker +###################### + +To run Covalent as a Docker container using public images, do the following. + +.. card:: 1. Get the latest Docker image for Covalent: + + .. code:: bash + + docker pull public.ecr.aws/covalent/covalent:latest + + .. note:: To get the current stable image of Covalent, use ``stable`` instead of ``latest``. + +.. card:: 2. Start Covalent: + + .. code:: bash + + docker container run -d --name covalent -p 48008:48008 public.ecr.aws/covalent/covalent:latest + + This starts the container in detached mode and forwards port ``48008`` to the host. + +.. card:: 3. To view the Covalent GUI, go to `http://localhost:48008 `_. + +.. card:: 4. Configure Covalent inside the container with environment variables. + +The following table lists the environment variables available to customize Covalent's execution environment at startup: + +.. list-table:: Covalent configuration environment variables + :widths: 20 80 + :header-rows: 1 + + * - Environment Variable + - Description + * - COVALENT_ROOT + - Root directory for the ``covalent`` process + * - COVALENT_CONFIG_DIR + - Directory that ``covalent`` searches for its configuration file, ``covalent.conf`` + * - COVALENT_PLUGINS_DIR + - Path where ``covalent`` looks to load any installed executor plugins + * - COVALENT_DATABASE + - Path to ``covalent``'s backend SQLite3 database + * - COVALENT_LOGDIR + - Path to ``covalent``'s log file + * - COVALENT_CACHE_DIR + - Directory used by ``covalent`` to store temporary objects during runtime + * - COVALENT_DATA_DIR + - Path to ``covalent``'s database directory + * - COVALENT_RESULTS_DIR + - Directory in which to store intermediate result objects + * - COVALENT_SVC_PORT + - TCP port on which ``covalent`` runs + * - COVALENT_SERVER_IFACE_ANY + - Boolean flag that causes ``covalent`` to listen on all network interfaces on the host + * - COVALENT_NUM_WORKERS + - Number of Dask workers in Covalent's default cluster + * - COVALENT_MEM_PER_WORKER + - Memory limit for each Dask worker + * - COVALENT_THREADS_PER_WORKER + - Number of threads with which to start each worker + + +.. card:: 5. For example, to start Covalent with two workers on port 8000: + + .. code:: bash + + docker container run --name covalent -p 8000:8000 -e COVALENT_NUM_WORKERS=2 -e COVALENT_SVC_PORT=8000 public.ecr.aws/covalent/covalent:latest diff --git a/doc/source/deployment/deploy_with_systemd.rst b/doc/source/deployment/deploy_with_systemd.rst new file mode 100644 index 000000000..09601ba51 --- /dev/null +++ b/doc/source/deployment/deploy_with_systemd.rst @@ -0,0 +1,116 @@ +################################ +Installing Covalent with Systemd +################################ + +.. note:: In these installation instructions, we assume ``Python3.8`` is available on the system and that all the commands are issued as ``root``. + +To install Covalent on a Linux physical or virtual host with ``systemd``, do the following: + +Prerequisites +------------- + +On Debian/Ubuntu based systems, install the *virtualenv* Python module at the system level: + +..code:: bash + + python3 -m pip install virtualenv + +Procedure +--------- + +.. card:: 1. Create the Python virtual environment in which to install Covalent: + + .. code:: bash + + python3 -m virtualenv /opt/virtualenvs/covalent + +.. card:: 2. Install Covalent in the virtual environment: + + .. code:: bash + + /opt/virtualenvs/covalent/bin/python -m pip install covalent + + This ensures that the latest release of Covalent along with all its dependencies are properly installed in the virtual environment. + +.. card:: 3. If you plan to use the AWS executor plugins with your Covalent deployment, install the ``covalent-aws-plugins``: + + .. code:: bash + + /opt/virtualenvs/covalent/bin/python -m pip install 'covalent-aws-plugins[all]' + +.. card:: 4. Create a ``systemd`` unit file for Covalent. + + Use the ``systemd`` ``Environment`` and ``EnvironmentFile`` directives to configure environment variables that determine Covalent's startup and runtime behavior. + + Customize the following sample ``covalent.service`` ``systemd`` unit file to your needs for hosting Covalent. On most Linux systems, this service file can be installed under ``/usr/lib/systemd/system``. For more information about the service file, see the ``systemd`` documentation `here `_. + + .. code:: bash + + [Unit] + Description=Covalent Dispatcher server + After=network.target + + [Service] + Type=forking + Environment=VIRTUAL_ENV=/opt/virtualenvs/covalent + Environment=PATH=/opt/virtualenvs/covalent/bin:$PATH + Environment=HOME=/var/lib/covalent + Environment=COVALENT_SERVER_IFACE_ANY=1 + EnvironmentFile=/etc/covalent/covalent.env + ExecStartPre=-/opt/virtualenvs/covalent/bin/covalent stop + ExecStart=/opt/virtualenvs/covalent/bin/covalent start + ExecStop=/opt/virtualenvs/covalent/bin/covalent stop + TimeoutStopSec=10 + + [Install] + WantedBy=multi-user.target + +.. card:: 5. Configure a ``service`` account on the server with only the privileges required to ensure proper Covalent functionality. + + Running Covalent as the root user is *not* recommended; this compromises security on the server. For one thing, the Covalent GUI's built-in terminal provides a login shell as the Covalent user – so if the Covalent server is running as root, users have access to a root shell on the server. + +.. card:: 6. To ensure that ``systemd`` invokes the Covalent server from within the virtual environment created earlier, set the ``VIRTUAL_ENV`` environment variable to the location of the virtual environment: + + .. code:: bash + + VIRTUAL_ENV=/opt/virtualenvs/covalent + + This ensures that the proper Python interpreter is used by Covalent at runtime. + +.. card:: 7. (Optional) Customize Covalent-specific environment variables: + + Create the file specified in the In the ``[Service]`` directive ``EnvironmentFile`` location (in the above example, ``/etc/covalent/covalent.env``). + + Populate the file with Covalent-specific environment variables such as ``COVALENT_CACHE_DIR``, ``COVALENT_DATABASE``, ``COVALENT_SVC_PORT`` and so on to customize Covalent's runtime environment. + +.. card:: 8. Once all the settings have been configured, start Covalent: + + .. code:: bash + + systemctl daemon-reload + systemclt start covalent.service + + .. note:: You only need to update ``systemd`` by executing the ``systemd daemon-reload`` command when a unit file is modified. + +.. card:: 9. Check the status of the service at any time with: + + .. code:: bash + + systemctl status covalent + +.. card:: 10. (Optional) Configure ``covalent.service`` to start on system bootup: + + .. code:: bash + + systemctl enable covalent.service + + +.. card:: 11. Once the service is running properly, connect to the Covalent GUI from a browser. + + Use the server hostname and port configured in the ``COVALENT_SVC_PORT`` environment variable. By default, Covalent start on port ``48008``. + +.. card:: 12. If you need to stop the server, use: + + .. code:: bash + + systemctl stop covalent.service diff --git a/doc/source/deployment/index.rst b/doc/source/deployment/index.rst index 7e57f5ef8..681c36643 100644 --- a/doc/source/deployment/index.rst +++ b/doc/source/deployment/index.rst @@ -2,333 +2,207 @@ Covalent Deployment Guide ************************* -Covalent supports both local and remote installations to suit different uses cases and compute requirements. For quick prototyping and testing running ``Covalent`` locally -might be sufficient but for dispatching large compute intensive workflows which may require lots of CPU cores and memory, deploying ``Covalent`` as a **remote** server (cloud/on-prem) would be a better alternative. This way -users can still develop their workflows locally and dispatch them to the remote Covalent server for execution. +Covalent supports both local and remote installation to suit different use cases. For quick prototyping and testing, running Covalent locally is sufficient. -.. image:: ./covalent-self-hosted.svg - :width: 800 - :alt: Covalent self hosted deployment - -========================= -Deployment with Docker -========================= - -Apart from installing ``Covalent`` locally within a Python virtual environment, users can also run Covalent as a docker container using the public images. The latest docker image for ``Covalent`` can be obtained as - -.. code:: bash - - docker pull public.ecr.aws/covalent/covalent:latest - - -.. note:: - - To obtain the stable image, the ``stable`` tag can be used instead of ``latest`` +For dispatching large compute-intensive workflows that require lots of CPU cores and memory, deploying Covalent as a *remote* server (cloud or on-premises) marshals resources more efficiently. Users can develop their workflows locally, then dispatch them to the remote Covalent server for execution. -Covalent can then be started by running the container as follows - -.. code:: bash - - docker container run -d --name covalent -p 48008:48008 public.ecr.aws/covalent/covalent:latest - -This will start the container in detached mode and map port ``48008`` back out to host. To view the UI, users can then go to `http://localhost:48008 `_. Users can still configure Covalent that's running inside the container via environment variables. -The following table lists out all the supported environment variables that users can specify to customize Covalent's execution environment at start up. - -.. list-table:: Covalent configuration environment variables - :widths: 20 80 - :header-rows: 1 - - * - Environment Variable - - Description - * - COVALENT_ROOT - - Root directory for the ``covalent`` process - * - COVALENT_CONFIG_DIR - - Directory that ``covalent`` will search for its configuration file, ``covalent.conf`` - * - COVALENT_PLUGINS_DIR - - Path where ``covalent`` will look to load any executor plugins installed - * - COVALENT_DATABASE - - Path to ``covalent``'s backend SQLite3 database - * - COVALENT_LOGDIR - - Path to ``covalent``'s log file - * - COVALENT_CACHE_DIR - - Directory to be used by ``covalent`` for storing temporary objects during runtime - * - COVALENT_DATA_DIR - - Path to ``covalent``'s database directory - * - COVALENT_RESULTS_DIR - - Directory on the filesystem to store the intermediate result objects - * - COVALENT_SVC_PORT - - TCP port on which ``covalent`` will start running - * - COVALENT_SERVER_IFACE_ANY - - Boolean value to allow ``covalent`` to listen on all network interfaces on the host - * - COVALENT_NUM_WORKERS - - Number of Dask workers to start as part of Covalent's default cluster - * - COVALENT_MEM_PER_WORKER - - Memory limit for each dask worker - * - COVALENT_THREADS_PER_WORKER - - Number of threads to start each worker with - - -As an example users can start Covalent with 2 workers on port 8000 as follows +.. image:: ./covalent-self-hosted.svg + :width: 800 + :alt: Covalent self hosted deployment -.. code:: bash - docker container run --name covalent -p 8000:8000 -e COVALENT_NUM_WORKERS=2 -e COVALENT_SVC_PORT=8000 public.ecr.aws/covalent/covalent:latest +On-Premise Deployment +##################### +Create a centralized deployment by installing the Covalent server on an on-prem server or virtual machine. -============================== -On-prem deployment -============================== +Deployment with Docker +---------------------- -The ``Covalent`` server can be installed and deployed on on-prem servers or virtual machines quite easily in order to centralize the deployment. This would enable users to host their Covalent servers on on-prem machines they may have access to or run them inside virtual machines. If the remote machines have `Docker `_ support enabled then the deployment is trivally simple and amounts to simply pulling and running the Covalent container from our public registries. The deployment can be customized by following the steps outlined in :ref:`Deployment with Docker ` section. +If the remote machine has `Docker `_ enabled then the deployment amounts to simply pulling and running the Covalent container from our public registries. The deployment can be customized by following the steps outlined in :doc:`./deploy_with_docker`. ------------------------ Deployment with Systemd ----------------------- +We recommend that you *not* install Covalent directly at the system level as its Python :doc:`version <../getting_started/compatibility>` and package dependencies can conflict with those of the system. -The Covalent server can also be installed and managed as a `systemd `_ service if desired. This can be a preferred approach if one would like to manage and administer the server via `systemd `_. There are several ways Covalent can be installed on a system and managed via systemd. For instance, users can directly install Covalent at the system level, install all the required plugins, create a ``covalent.service`` unit file and enable the service. - -.. warning:: - - Installing Covalent at the system level is **NOT** recommended as its Python package dependencies can potentially conflict with system packages. Moreover, the system Python version may not be compatible with Covalent. Refer to our compatibility matrix to see all the support Python versions - -The recommended approach for running Covalent under systemd is to create a Python virtual environment with Covalent installed and then run the systemd service. This approach ensures that the system level Python settings are not altered and any potential Python package dependency conflicts are averted. In this guide, for convenience we assume ``Python3.8`` is available on the system and all the commands are carried out as the **root** user. We first being by creating the Python virtual environment in which Covalent will be subsequently installed - -.. code:: bash - - python3 -m virtualenv /opt/virtualenvs/covalent - -.. note:: - - On Debian/Ubuntu based systems the **virtualenv** Python module can be installed at the system level via pip as follows ``python3 -m pip install virtualenv`` - -We can now install ``Covalent`` in this virtual environment as follows - -.. code:: bash - - /opt/virtualenvs/covalent/bin/python -m pip install covalent - - -.. note:: - - If users are looking to use the AWS executor plugins with their Covalent deployment the ``covalent-aws-plugins`` must be installed via ``/opt/virtualenvs/covalent/bin/python -m pip install 'covalent-aws-plugins[all]'`` - -This will ensure that the latest release of ``Covalent`` along with all its dependencies are properly installed in the virtual environment. We can now create a ``systemd`` unit file for Covalent and enable it to be managed by ``systemd``. -Systemd provides a convenient inferface to configure environment variables that will be exposed to the covalent server via the ``Environment`` and ``EnvironmentFile`` directives. We will leverage these interfaces to configure Covalent's startup and runtime behaviour. Users can use the following sample ``covalent.service`` systemd unit file and customize it for their needs when hosting Covalent themselves. On most linux systems, this service file can be installed under ``/usr/lib/systemd/system``. Users are encouraged to review the systemd documentation `here `_. - -.. code:: bash - - [Unit] - Description=Covalent Dispatcher server - After=network.target - - [Service] - Type=forking - Environment=VIRTUAL_ENV=/opt/virtualenvs/covalent - Environment=PATH=/opt/virtualenvs/covalent/bin:$PATH - Environment=HOME=/var/lib/covalent - Environment=COVALENT_SERVER_IFACE_ANY=1 - EnvironmentFile=/etc/covalent/covalent.env - ExecStartPre=-/opt/virtualenvs/covalent/bin/covalent stop - ExecStart=/opt/virtualenvs/covalent/bin/covalent start - ExecStop=/opt/virtualenvs/covalent/bin/covalent stop - TimeoutStopSec=10 +To run Covalent under ``systemd``, we recommend that you create a Python virtual environment with Covalent installed and manage Covalent with the ``systemd`` service. This approach ensures that the system-level Python settings are unchanged and averts Python package dependency conflicts. - [Install] - WantedBy=multi-user.target +To install and run Covalent under :code:`systemd`, use the instructions in :doc:`./deploy_with_systemd`. -To ensure that when systemd invokes the ``Covalent`` server, its from within the virtual environment created earlier, we need to the set ``VIRTUAL_ENV`` environment variable to its proper value - -.. code:: bash - - VIRTUAL_ENV=/opt/virtualenvs/covalent +Deployment on AWS +################# -Setting this variable to the location of the virtual environment is sufficient to ensure that the proper Python interpreter is used by Covalent at runtime. In the ``[Service]`` directive we set the ``EnvironmentFile`` location to ``/etc/covalent/covalent.env``. Users can optionally create this file and populate it with Covalent specific environment variables such as COVALENT_CACHE_DIR, COVALENT_DATABASE, COVALENT_SVC_PORT ... in order customize Covalent's runtime environment. +Deploy Covalent in an AWS account with any ``x86``-based EC2 instance. Deploying on AWS Cloud enables you to scale your deployments based on compute needs. -Once all the settings have been configured, Covalent can be started as follows +As with the Docker image, with each stable release we include a ready-to-use Amazon Machine Image (AMI) that is fully configured to start a Covalent server on instance boot. Query AWS Marketplace for the AMI ID directly from the console or via the ``aws cli`` command line tool. For example, the following CLI command queries details about the AMI released for version ``covalent==0.202.0``: .. code:: bash - systemctl daemon-reload - systemclt start covalent.service - + aws ec2 describe-images --owners Agnostiq --filter "Name=tag:Version,Values=0.202.0" -.. note:: - - The status of the service can be inspected by ``systemctl status covalent``. The systemd ``daemon-reload`` command must be executed each time a unit file has been modified to notify systemd about the changes - - -The ``covalent.service`` can also be enabled to start on boot via systemd as follows +Once you have the AMI ID, you can launch an EC2 instance in your account as follows: .. code:: bash - systemctl enable covalent.service - + aws ec2 run-instances --image-id --instance-type --subnet-id -security-group-ids --key-name -Once the service is running properly, users can connect to the Covalent's UI from their browser by via their remote machines hostname and the port they configured Covalent to run on via the ``COVALENT_SVC_PORT`` environment variable. By default, Covalent start on port ``48008``. The server can be stopped using systemd as follows +For more complicated deployments, infrastructure-as-code tools such as `AWS CloudFormation `_ and `Terraform `_ are available. -.. code:: bash - - systemctl stop covalent.service +Server-Based Covalent Best Practices +#################################### -.. warning:: +Although creating and dispatching workflows on a remote Covalent dispatcher is largely the same as with a local dispatcher, there are a few important differences. - Running Covalent as the root user is **NOT** recommended as it can have several security implications for the remote server. If possible, users must configure a ``service`` account on the system with just the right amount of privileges to ensure proper Covalent functionality. The Covalent UI has an in-built terminal for convenience and it present a login shell as the Covalent user i.e. if the Covalent server is running as root, then users will have access to a root shell on the server. This can potentially have major security implications, thus proper UNIX security polices and best practices must be followed when self-hosting Covalent on remote servers +Client Side Configuration +------------------------- -==================== -Deployment on AWS -==================== +When Covalent is hosted remotely there is no need to run the Covalent server on a user's local (client) machine, but you do have to pass the dispatcher address and port to the workflow. There are three ways to do this: -Users can deploy Covalent in their own AWS accounts with any ``x86`` based EC2 instance of their choice. Deploying on AWS cloud will allow users to vertically/horizontally scale up their deployments depending on their compute needs. +* In the client-side configuration file +* Using ``set_config`` +* In the ``dispatch`` and ``get_result`` methods -Similar to the docker image, with each stable release, a ready to use Amazon Machine Image (AMI) is also released that is fully configured to start a Covalent server on instance boot. Users can query AWS Marketplace for the AMI ID directly from the console or via the ``aws cli`` command line tool. - -.. code:: bash - aws ec2 describe-images --owners Agnostiq --filter "Name=tag:Version,Values=0.202.0" +Configuration File +~~~~~~~~~~~~~~~~~~ -The above CLI example illustrates one can query details about the AMI released for version ``covalent==0.202.0``. Once the AMI id is retrieved, users can launch on EC2 instance in their account as follows +On a client, when Covalent is imported it renders a `config` file that includes the dispatcher default address and port, ``localhost`` and ``48008``. Edit the ``dispatcher`` section of the client-side configuration, replacing the defaults with the remote values for the ``address`` and ``port``: .. code:: bash - aws ec2 run-instances --image-id --instance-type --subnet-id -security-group-ids --key-name - -For more complicated deployments infrastructure as code tools such as `AWS CloudFormation `_ or `Terraform `_ can be used. - - -=============== -Best Practices -=============== - + [dispatcher] + address = + port = + ... -Self-hosting Covalent on remote machines is an easy way to run compute intensive workflows on machines other than a user's local workstation. Although the experience of creating and dispatching workflows is largely the same, there a few subtleties to consider. +Using set_config +~~~~~~~~~~~~~~~~ +The dispatcher ``address`` and ``port`` can be set using the ``set_config`` method before dispatching any workflows: ---------------------------------- -Client/Server Side configuration ---------------------------------- +.. code:: python -When Covalent is deployed on remote machines Covalent parses all its configuration values from the configuration file it was deployed with i.e. **server side config**. The client side/local configuration file can be used by the client to set the dispatcher address and port information so that workflows can be dispatched to the remote server. + import covalent as ct -.. note:: + ct.set_config({"dispatcher.address": ""}) + ct.set_config({"dispatcher.port": ""}) - It is important to realize that when Covalent is hosted remotely there is no need for the Covalent server to be running on the user's local machine. Setting the server address and port in the user's local i.e. **client side** configuration file is enough for dispatching workflows + ... -On the client side, when Covalent is imported it renders a `config` file based on its default values. Users can edit the ``dispatcher`` section of the client side configuration with the new values for the ``address`` and ``port``. These values default to ``localhost`` and ``48008`` on client side. + dispatch_id = ct.dispatch(my_workflow)(*args, **kwargs) -.. code:: bash - [dispatcher] - address = - port = - ... +In the dispatch and get_result Functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The dispatcher ``address`` and ``port`` can also via the ``get_config`` method before dispatching any workflows +You can specify the dispatcher address and port directly in the ``ct.dispatch()`` and ``ct.get_result()`` functions: .. code:: python - import covalent as ct - - ct.set_config({"dispatcher.address": ""}) - ct.set_config({"dispatcher.port": ""}) - - ... - - dispatch_id = ct.dispatch(my_workflow)(*args, **kwargs) - - -Lastly, the dispatcher address can also be specified directly in the `ct.dispatch` and `ct.get_result` methods + import covalent as ct -.. code:: python + ... - import covalent as ct + dispatch_id = ct.dispatch(workflow, dispatcher_addr=":")(*args, **kwargs) + result = ct.get_result(dispatch_id, dispatcher_addr=":") - ... - dispatch_id = ct.dispatch(workflow, dispatcher_addr=":")(*args, **kwargs) - result = ct.get_result(dispatch_id, dispatcher_addr=":") +Executors +--------- +In the context of a hosted Covalent server, there are three ways to specify an executor for an electron: server-side, client-side, and partially defined. (Partially-defined is a variation on the client-side executor, but is explained separately). All three ways of specifying executors have pros and cons. ------------------- -Executors ------------------- +Server-Side Executors +~~~~~~~~~~~~~~~~~~~~~ -When Covalent is deployed remotely, it is important to understand how ``executors`` are handled by the server. For instance, in Covalent there are multiple ways users can specify an ``executor`` for an electron in their workflows and each of the cases has certain implications on how the executor information is parsed and handled by the remote server +In the server-side case, the client specifies only the short name of an executor on which to run an electron. The server constructs an instance of the named executor based on the configuration in its config file. The executor is constructed or recruited just in time for execution. -#. Using the executor short name +This is the way to define executors when the compute resources and executor specifications are managed centrally. .. code:: python - import covalent as ct + import covalent as ct - @ct.electron(executor="awsbatch") - def task(*args, **kwargs): + @ct.electron(executor="awsbatch") + def task(*args, **kwargs): ... return result -In this case, the server receives only the short name of the executor that ought to be used for executing the electron, thus the server will construct an instance of the specified executor using the configuration values specified in its config file i.e. **server side** during workflow execution just prior the the task being sent to the backend for execution. This is a very convenient way to choose executors in a workflow then the compute resources are being managed entirely by the remote server. +Pros: Executor configuration and creation is centralized in one location, on the server. Clients don't need to know the details of executor implementation. -.. warning:: +Cons: Clients are at the mercy of the server configuration. Executors have to be centrally managed and their names provided to clients. Executor configurations can be changed remotely through the Covalent GUI or by editing the configuration over SSH; this should be discouraged, if not prohibited, since the changes affect other clients' workflows without notifying them. - Users however should be cautious of any changes being made to the **server side** configurations from the UI or directly over a SSH connection to the remote server. +Client-Side Executors +~~~~~~~~~~~~~~~~~~~~~ -#. Passing an instance of the executor class with fully specified input arguments +In the client-side case, the client passes a fully specified instance of the executor class to the remote dispatcher. .. code:: python - import covalent as ct + import covalent as ct - awslambda = ct.executor.AWSLambdaExecutor(function_name="my-lambda-function", s3_bucket_name="my-s3-bucket-name") + awslambda = ct.executor.AWSLambdaExecutor(function_name="my-lambda-function", s3_bucket_name="my-s3-bucket-name") - @ct.electron(executor=awslambda) - def task(*args, **kwargs): + @ct.electron(executor=awslambda) + def task(*args, **kwargs): ... return result -When a fully specified instance of an executor is passed to the remote server then the client passed instance is pickled and transported to the remote server, which then uses that to execute the task on the user specified backend. In this case there is not ambiguity between the client and the server as to which values of the executor ought to be parsed from the **server side** configuration file since all the values are specified by the client at workflow dispatch time. - +When a client passes a fully specified instance of an executor, the instance is pickled (serialized) for transport. The server deserializes the instance, then uses it to execute the task on the client-specified backend. In this case there is no ambiguity between the client and the server as to the executor parameters since all the values are specified by the client at workflow dispatch time. -.. warning:: +Pros: There is no way the server can "surprise" the client by using a misdefined or redefined executor. - When providing executor information this way, users must ensure that the remote Covalent server has access to the executor backend. For instance, if the user is looking to use the ``AWSBatchExecutor`` in their workflows, then the remote Covalent server must have the proper IAM permissions and policies configured so that it can execute that task on the user's behalf using the AWS Batch service. +Cons: The submitter on the client side must ensure that the server has access to the executor resource. For example, if you require an ``AWSBatchExecutor`` in your workflows, then the remote Covalent server must have the proper IAM permissions and policies configured so that it can execute on your electron's behalf using the AWS Batch service. +Partially Defined Executors +~~~~~~~~~~~~~~~~~~~~~~~~~~~ -#. Passing an instance of an executor with partially specified input arguments +In this case, some parameter values are omitted from the executor's constructor. Omitted parameters are inferred from the client-side configuration during workflow construction, which occurs offline. The client does not interact with the dispatcher on the remote server until the workflow is submitted for execution. .. code:: python - import covalent as ct + import covalent as ct - awsbatch = ct.executor.AWSBatchExecutor(vcpus=2) + awsbatch = ct.executor.AWSBatchExecutor(vcpus=2) - @ct.electron(executor=awsbatch) - def task(*args, **kwargs): + @ct.electron(executor=awsbatch) + def task(*args, **kwargs): ... return result -In this case, all the parameter values that are omitted from the executor's constructor are inferred from the **client side** configuration/environment during workflow construction time. This occurs offline and the dispatcher/remote server is not interacted with until the workflow is submitted for execution. +From the server's perspective, this case is the same as the client-side executor: the executor is serialized for transport, and the server receives a fully specified instance. This case is broken out to emphasize that the client configuration can be exploited to "fill in" some of the executor parameters if they don't change for the particular client. + + +Environment Hygiene +------------------- + +By default, Covalent starts a local Dask cluster on which it executes those tasks for which no executor is specified. This cluster by default runs in the same environment as Covalent and shares all the environment's Python packages. + +Client-Side +~~~~~~~~~~~ +Especially when Covalent is running on a server, we recommend that you avoid using ``DepsPip`` calls in your workflows. The client-requested ``pip`` packages are installed in the same environment as Covalent, potentially leading to unexpected package conflicts and destabilizing the Covalent server. ------------------- -Environment Sanity ------------------- +Server-Side +~~~~~~~~~~~ -Covalent by default starts a local Dask cluster that it uses to execute tasks when executor metadata. This cluster by default runs in the same environment as Covalent and shares all the Python packages. In this case, users must be cautious of using any ``DepsPip`` call in their workflows as the user requested ``pip`` packages will be installed in the same environment as Covalent. This can potentially lead to unwarranted package conflicts and de-stabilize the Covalent server. +When hosting Covalent on a server, we recommend that you start a separate Dask cluster running either on an entirely different machine or in a separate virtual environment on the same machine. That way clients can share a Covalent virtual environment that is unmodified even if the workflows use frequent calls to ``DepsPip``. -As a best practice, it is **recommended** that users start a separate Dask cluster that runs either on an entirely different machine or in a separate virtual environment on the same machine. This way users can ensure that Covalent's virtual environment will remain unmodified even if the workflows use frequent calls to ``DepsPip``. +.. note:: When running a separate Dask cluster on server-hosted Covalent, you must modify Covalent's server side configuration file to reflect the location of the Dask cluster. -.. note:: - When running a separate Dask cluster, users must make Covalent aware of the cluster's scheduler address and port by modifying the **server side** configuration file so that Covalent can submit tasks to it as they appear in the workflow +LocalExecutor +------------- +We recommend that you avoid using the ``LocalExecutor`` except for debugging purposes. Especially on a server, ``LocalExecutor`` is non-performant and potentially unstable. ----------------------------- -LocalExecutor & I/O ----------------------------- +Large Inputs and Outputs +------------------------ -For performance and stability reasons, users must avoid using the ``LocalExecutor`` as much as possible and only use it for debugging purposes. Secondly, users must aim to avoid excessively large inputs and outputs for their electrons as they can consume a lot of system memory. +When submitting workflows to a hosted server, avoid constructing excessively large inputs and outputs for electrons. Remember that you're sharing a finite pool of memory with other clients. diff --git a/doc/source/index.rst b/doc/source/index.rst index 9792bc383..cf86642ef 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -161,6 +161,7 @@ Covalent is a quickly growing and vibrant community of enthusiasts, researchers, How-To Guide User Interface API Reference + Server Deployment Credentials .. toctree::