SFA: Unlock loading application from remote location, using fsspec

CHANGELOG.md

@@ -17,12 +17,23 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
   argument, enabling usage on single-file applications. Beforehand, only
   invocations of Python modules were possible.
   ```shell
+  # Install Responder with CLI extension.
+  pip install 'responder[cli]'
+
   # Start Responder application defined in Python module.
   responder run acme.app:api
 
   # Start Responder application defined in a single Python file.
   responder run examples/helloworld.py:api
   ```
+- CLI: `responder run` now also accepts URLs.
+  ```shell
+  # Install Responder with CLI extension (full).
+  pip install 'responder[cli-full]'
+
+  # Start Responder application defined in Python module at remote location.
+  responder run https://github.com/kennethreitz/responder/raw/refs/heads/main/examples/helloworld.py
+  ```
 
 ### Changed
 

docs/source/cli.rst

@@ -2,72 +2,149 @@ Responder CLI
 =============
 
 Responder installs a command line program ``responder``. Use it to launch
-a Responder application from a file or module.
+a Responder application from a file or module, either located on a local
+or remote filesystem, or object store.
 
-Launch application from file
-----------------------------
+Launch Module Entrypoint
+------------------------
+
+For loading a Responder application from a Python module, you will refer to
+its ``API()`` instance using a `Python entry point object reference`_ that
+points to a Python object. It is either in the form ``importable.module``,
+or ``importable.module:object.attr``.
+
+A basic invocation command to launch a Responder application:
+
+.. code-block:: shell
+
+    responder run acme.app
+
+The command above assumes a Python package ``acme`` including an ``app``
+module ``acme/app.py`` that includes an attribute ``api`` that refers
+to a ``responder.API`` instance, reflecting the typical layout of
+a standard Responder application.
+
+Loading a Responder application using an entrypoint specification will
+inherit the capacities of `Python's import system`_, as implemented by
+`importlib`_.
+
+Launch Local File
+-----------------
 
-Acquire minimal example application, `helloworld.py`_,
-implementing a basic echo handler, and launch the HTTP service.
+Acquire a minimal example single-file application, ``helloworld.py`` [1]_,
+to your local filesystem, giving you the chance to edit it, and launch the
+Responder HTTP service.
 
 .. code-block:: shell
 
     wget https://github.com/kennethreitz/responder/raw/refs/heads/main/examples/helloworld.py
     responder run helloworld.py
 
-In another terminal, invoke a HTTP request, for example using `HTTPie`_.
+.. note::
 
-.. code-block:: shell
+    To validate the example application, invoke a HTTP request, for example using
+    `curl`_, `HTTPie`_, or your favourite browser at hand.
+
+    .. code-block:: shell
 
-    http http://127.0.0.1:5042/hello
+        http http://127.0.0.1:5042/Hello
 
-The response is no surprise.
+    The response is no surprise.
 
-::
+    ::
 
-    HTTP/1.1 200 OK
-    content-length: 13
-    content-type: text/plain
-    date: Sat, 26 Oct 2024 13:16:55 GMT
-    encoding: utf-8
-    server: uvicorn
+        HTTP/1.1 200 OK
+        content-length: 13
+        content-type: text/plain
+        date: Sat, 26 Oct 2024 13:16:55 GMT
+        encoding: utf-8
+        server: uvicorn
 
-    hello, world!
+        Hello, world!
 
+.. [1] The Responder application `helloworld.py`_ implements a basic echo handler.
 
-Launch application from module
-------------------------------
+Launch Remote File
+------------------
 
-If your Responder application has been implemented as a Python module,
-launch it like this:
+You can also launch a single-file application where its Python file is stored
+on a remote location.
+
+Responder supports all filesystem adapters compatible with `fsspec`_, and
+installs the adapters for Azure Blob Storage (az), Google Cloud Storage (gs),
+GitHub, HTTP, and AWS S3 by default.
 
 .. code-block:: shell
 
-    responder run acme.app
+    # Works 1:1.
+    responder run https://github.com/kennethreitz/responder/raw/refs/heads/main/examples/helloworld.py
+    responder run github://kennethreitz:responder@/examples/helloworld.py
 
-That assumes a Python package ``acme`` including an ``app`` module
-``acme/app.py`` that includes an attribute ``api`` that refers
-to a ``responder.API`` instance, reflecting the typical layout of
-a standard Responder application.
+If you need access other kinds of remote targets, see the `list of
+fsspec-supported filesystems and protocols`_. The next section enumerates
+a few synthetic examples. The corresponding storage buckets do not even
+exist, so don't expect those commands to work.
 
-.. rubric:: Non-standard instance name
+.. code-block:: shell
+
+    # Azure Blob Storage, Google Cloud Storage, and AWS S3.
+    responder run az://kennethreitz-assets/responder/examples/helloworld.py
+    responder run gs://kennethreitz-assets/responder/examples/helloworld.py
+    responder run s3://kennethreitz-assets/responder/examples/helloworld.py
+
+    # Hadoop Distributed File System (hdfs), SSH File Transfer Protocol (sftp),
+    # Common Internet File System (smb), Web-based Distributed Authoring and
+    # Versioning (webdav).
+    responder run hdfs://kennethreitz-assets/responder/examples/helloworld.py
+    responder run sftp://user@host/kennethreitz/responder/examples/helloworld.py
+    responder run smb://workgroup;user:password@server:port/responder/examples/helloworld.py
+    responder run webdav+https://user:password@server:port/responder/examples/helloworld.py
+
+.. tip::
+
+    In order to install support for all filesystem types supported by fsspec, run:
+
+    .. code-block:: shell
+
+        uv pip install 'fsspec[full]'
+
+    When using ``uv``, this concludes within an acceptable time of approx.
+    25 seconds. If you need to be more selectively instead of using ``full``,
+    choose from one or multiple of the available `fsspec extras`_, which are:
+
+    abfs, arrow, dask, dropbox, fuse, gcs, git, github, hdfs, http, oci, s3,
+    sftp, smb, ssh.
 
-When your attribute that references the ``responder.API`` instance
-is called differently than ``api``, append it to the launch target
-address like this:
+Launch with Non-Standard Instance Name
+--------------------------------------
+
+By default, Responder will acquire an ``responder.API`` instance using the
+symbol name ``api`` from the specified Python module.
+
+If your main application file uses a different name than ``api``, please
+append the designated symbol name to the launch target address.
+
+It works like this for module entrypoints and local files:
 
 .. code-block:: shell
 
     responder run acme.app:service
+    responder run /path/to/acme/app.py:service
+
+It works like this for URLs:
 
-Within your ``app.py``, the instance would have been defined like this:
+.. code-block:: shell
+
+    responder run http://app.server.local/path/to/acme/app.py#service
+
+Within your ``app.py``, the instance would have been defined to use
+the ``service`` symbol name instead of ``api``, like this:
 
 .. code-block:: python
 
     service = responder.API()
 
-
-Build JavaScript application
+Build JavaScript Application
 ----------------------------
 
 The ``build`` subcommand invokes ``npm run build``, optionally accepting
@@ -78,13 +155,20 @@ where it expects a regular NPM ``package.json`` file.
 
     responder build
 
-When specifying a target directory, responder will change to that
+When specifying a target directory, Responder will change to that
 directory beforehand.
 
 .. code-block:: shell
 
     responder build /path/to/project
 
 
+.. _curl: https://curl.se/
+.. _fsspec: https://filesystem-spec.readthedocs.io/en/latest/
+.. _fsspec extras: https://github.com/fsspec/filesystem_spec/blob/1fe5695802679e8ce1ae242a44d4846195425509/pyproject.toml#L26-L68
 .. _helloworld.py: https://github.com/kennethreitz/responder/blob/main/examples/helloworld.py
 .. _HTTPie: https://httpie.io/docs/cli
+.. _importlib: https://docs.python.org/3/library/importlib.html
+.. _list of fsspec-supported filesystems and protocols: https://github.com/fsspec/universal_pathlib?tab=readme-ov-file#currently-supported-filesystems-and-protocols
+.. _Python entry point object reference: https://packaging.python.org/en/latest/specifications/entry-points/
+.. _Python's import system: https://docs.python.org/3/reference/import.html

responder/util/common.py

@@ -0,0 +1,9 @@
+from urllib.parse import urlparse
+
+
+def is_valid_url(url):
+    try:
+        result = urlparse(url)
+        return all([result.scheme, result.netloc])
+    except ValueError:
+        return False

responder/util/python.py

@@ -5,8 +5,13 @@
 import typing as t
 import uuid
 from pathlib import Path
+from tempfile import NamedTemporaryFile
 from types import ModuleType
 
+from upath import UPath
+
+from responder.util.common import is_valid_url
+
 logger = logging.getLogger(__name__)
 
 
@@ -31,12 +36,13 @@ def load_target(target: str, default_property: str = "api", method: str = "run")
         source to prevent security vulnerabilities.
 
     Args:
-        target: Module address (e.g., 'acme.app:foo') or file path (e.g., '/path/to/acme/app.py')
+        target: Module address (e.g., 'acme.app:foo'), file path (e.g., '/path/to/acme/app.py'),
+                or URL.
         default_property: Name of the property to load if not specified in target (default: "api")
-        method: Name of the method to verify on the loaded property (default: "run")
+        method: Name of the method to invoke on the API instance (default: "run")
 
     Returns:
-        The loaded property from the module
+        The API instance, loaded from the given property.
 
     Raises:
         ValueError: If target format is invalid
@@ -48,6 +54,22 @@ def load_target(target: str, default_property: str = "api", method: str = "run")
         >>> api.run()
     """  # noqa: E501
 
+    app_file = None
+    if is_valid_url(target):
+        upath = UPath(target)
+        frag = upath._url.fragment
+        suffix = upath.suffix
+        suffix = suffix.replace(f"#{frag}", "")
+        logger.info(f"Loading remote single-file application, source: {upath}")
+        name = "_".join([upath.parent.stem, upath.stem])
+        app_file = NamedTemporaryFile(prefix=f"{name}_", suffix=suffix, delete=False)
+        target = app_file.name
+        if frag:
+            target = f"{app_file.name}:{frag}"
+        logger.info(f"Writing remote single-file application, target: {target}")
+        app_file.write(upath.read_bytes())
+        app_file.flush()
+
     # Sanity checks, as suggested by @coderabbitai. Thanks.
     if not target or (":" in target and len(target.split(":")) != 2):
         raise InvalidTarget(f"Invalid target format: {target}")

setup.py

@@ -119,7 +119,11 @@ def run(self):
     setup_requires=[],
     install_requires=required,
     extras_require={
-        "cli": ["docopt-ng"],
+        "cli": [
+            "docopt-ng",
+            "fsspec[abfs,gcs,github,http,libarchive,s3]",
+            "universal-pathlib",
+        ],
         "develop": [
             "poethepoet",
             "pyproject-fmt; python_version>='3.7'",

tests/test_cli.py

@@ -150,21 +150,25 @@ def test_cli_build_invalid_package_json(
     assert any(item in stderr for item in to_list(expected_error))
 
 
+sfa_services_valid = [
+    str(Path("examples") / "helloworld.py"),
+    "https://github.com/kennethreitz/responder/raw/refs/heads/main/examples/helloworld.py",
+]
+
+
 # The test is marked as flaky due to potential race conditions in server startup
 # and port availability. Known error codes by platform:
 # - macOS:   [Errno 61] Connection refused (Failed to establish a new connection)
 # - Linux:   [Errno 111] Connection refused (Failed to establish a new connection)
 # - Windows: [WinError 10061] No connection could be made because target machine
 #            actively refused it
-@pytest.mark.flaky(reruns=5, reruns_delay=2)
-def test_cli_run(capfd):
+@pytest.mark.flaky(reruns=3, reruns_delay=2, only_rerun=["TimeoutError"])
+@pytest.mark.parametrize("target", sfa_services_valid, ids=sfa_services_valid)
+def test_cli_run(capfd, target):
     """
     Verify that `responder run` works as expected.
     """
 
-    # Invoke `responder run`.
-    target = Path("examples") / "helloworld.py"
-
     # Start a Responder service instance in the background, using its CLI.
     # Make it terminate itself after serving one HTTP request.
     server = ResponderServer(target=str(target), port=random_port(), limit_max_requests=1)