Normalize and make Content frontends and backends extensible

[bollwyvl]

Problem

We currently offer a union of files, backed by two storage backends, in the following order:

• pre-baked contents, as indexed in a file tree that roughly mirrors the Jupyter Contents REST API
• in-browser contents, managed by localForage

...with one storage frontend:

• a mock REST API, exposing Jupyter Contents

While this makes it possible to deliver the Lab file management UI, such as it is, it is not very ergonomic for:

• accessing from kernels
• providing alternate storage implementations e.g.
  • WebDAV (a la sharepoint, etc)
  • sqlite
  • git
  • fossil-scm
  • some rando REST API

Proposed Solution

• formalize and define an API for the existing backends
  • these would become two separate server extensions (Support federated serverlite extensions #104)
• separate the mock REST API frontend from the manager
• add an emscripten-compatible FS frontend for the overall contents

Additional context)

• some upstream thoughts about FS: Discuss desired features/syntax for persistence convenience js/py API pyodide/pyodide#1715 (comment)
• this may have impacts on add piplite for customizing pyolite packages, automate wheel management #310
  • packages are files, too!
• an example of an FS

[oeway]

FYI: I have been also looking into the storage options, and what I am using right now is using BrowserFS inside a ServiceWorker so we can manage files using varies backends (e.g. indexeddb), and the service worker can work as a in-browser file server. This allows all unified file access via HTTP, and for pyodide kernel, we can do XMLHTTPRequest in synchronous mode and create file-like object that works with large files.

Here is an demo shows how it work in a custom deployment of jupyterlite (available at https://jupyter.imjoy.io ):

Here is the notebook if you want to try it out: https://github.com/imjoy-team/jupyter.imjoy.io/blob/master/docs/files/elfinder-demo.ipynb

[jtpio]

Thanks for sharing this @oeway it looks cool!

[bollwyvl]

I've updated the issue description with a link to the emscripten filesystem built for the (maybe) sleeping starboard: of note, there is a link to the (even more dormant) BrowserFS example at the top, which has even better typing info, which is probably the most important to get started.

I'm thinking this is probably the route we should consider for emscripten-based (or maybe all?) kernels, and should be a major design consideration both here and on #463, as the experience would be basically seamless: touching a file in a kernel would make it appear in contents, a notebook would be able to (accurately) read its own state.

Of course, it's tempting for an initial implementation to just load up localforage directly in a worker and go to town... but sticking to the Jupyter Content API will give an implementer the most bang for their buck: one implementation against a custom REST/GraphQL/FileSystem/whatever, and they have custom notebook creation/storage as well as files available to kernels, which could include startup files, packages, or even entire pre-configured environments.

If the provider of this Contents API, in turn, was managed by a ServiceWorker, all the better, and indeed, having a single, offline-ready ServiceWorker that was servicing multiple open tabs is probably what is needed anyway, and would validate all of the hard paths we've chosen.

[bollwyvl]

Recently stumbled on the upcoming emscripten wasmfs, which sounds lovely, but maybe not something we can wait to trickle down into pyodide 😿.

[jtpio]

I think this was discussed somewhere else already, but maybe it could be interesting to move the existing IContents server plugin to an Contents.IDrive at some point. And try to leverage more the existing Contents.IDrive approach since there are a couple of existing extensions using it already (jupyterlab-github, jupyterlab-filesystem-access, and more).

This would have the benefit of moving the local storage logic to the frontend. And the plugin could then be reused in other lab apps if needed as a regular extension.

We would first need to make sure the DriveFS can be used from any drive so the kernels can still be aware of the contents.

It could also be interesting to split the localforage / IndexedDB storage and the pre-baked server contents in two separate drives so it's less confusing to know the origin of a file and whether it can be deleted or not. Although this would be less natural than having all the files in one place like it is right now.

Also related:

• Support for using a different IDrive jupyter/notebook#6665
• ** DO NOT MERGE** Experiment RTC and Drives jupyterlab/jupyterlab#13512