[DO NOT MERGE] Specify that the debug messages follow the debug adapter protocol

.gitignore

@@ -1,8 +1,6 @@
 MANIFEST
 build
 dist
-_build
-docs/gh-pages
 *.py[co]
 __pycache__
 *.egg-info
@@ -16,3 +14,16 @@ __pycache__
 .coverage
 .cache
 absolute.json
+htmlcov/
+
+# Sphinx documentation
+_build
+docs/_build/
+docs/gh-pages
+
+# PyBuilder
+target/
+
+# PyCharm
+.idea/
+*.iml

.travis.yml

@@ -1,19 +1,23 @@
+dist: xenial
 language: python
 python:
     - "nightly"
-    - '3.6-dev'
-    - 3.5
-    - 3.4
-    - 3.3
-    - 2.7
-sudo: false
+    - "3.8-dev"
+    - "3.7"
+    - "3.6"
+    - "3.5"
+    - "2.7"
 install:
   - pip install --upgrade setuptools pip
-  - pip install --upgrade --pre -e .[test] pytest-cov pytest-warnings codecov
+  - pip install --upgrade --upgrade-strategy eager --pre -e .[test] pytest-cov codecov 'coverage<5'
+  - pip freeze
 script:
   - py.test --cov jupyter_client jupyter_client
 after_success:
   - codecov
 matrix:
     allow_failures:
         - python: nightly
+branches:
+  except:
+    - /^auto-backport-of-pr-[0-9]+$/

MANIFEST.in

@@ -5,6 +5,7 @@ include README.md
 # Documentation
 graft docs
 exclude docs/\#*
+exclude docs/_*
 
 # Examples
 graft examples

docs/changelog.rst

@@ -4,6 +4,82 @@
 Changes in Jupyter Client
 =========================
 
+5.2.3
+=====
+
+`5.2.3 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2.3>`__
+
+- Fix hang on close in :class:`.ThreadedKernelClient` (used in QtConsole)
+  when using tornado with asyncio
+  (default behavior of tornado 5, see :ghpull:`352`).
+- Fix errors when using deprecated :attr:`.KernelManager.kernel_cmd`
+  (:ghpull:`343`, :ghpull:`344`).
+
+5.2.2
+=====
+
+`5.2.2 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2.2>`__
+
+- Fix :meth:`.KernelSpecManager.get_all_specs` method in subclasses
+  that only override :meth:`.KernelSpecManager.find_kernel_specs`
+  and :meth:`.KernelSpecManager.get_kernel_spec`.
+  See :ghissue:`338` and :ghpull:`339`.
+- Eliminate occasional error messages during process exit (:ghpull:`336`).
+- Improve error message when attempting to bind on invalid address (:ghpull:`330`).
+- Add missing direct dependency on tornado (:ghpull:`323`).
+
+
+5.2.1
+=====
+
+`5.2.1 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2.1>`__
+
+- Add parenthesis to conditional pytest requirement to work around a bug in the
+  ``wheel`` package, that generate a ``.whl`` which otherwise always depends on
+  ``pytest`` see :ghissue:`324` and :ghpull:`325`.
+
+5.2
+===
+
+`5.2 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2>`__
+
+- Define Jupyter protocol version 5.3:
+
+  - Kernels can now opt to be interrupted by a message sent on the control channel
+    instead of a system signal. See :ref:`kernelspecs` and :ref:`msging_interrupt`
+    (:ghpull:`294`).
+
+- New ``jupyter kernel`` command to launch an installed kernel by name
+  (:ghpull:`240`).
+- Kernelspecs where the command starts with e.g. ``python3`` or
+  ``python3.6``—matching the version ``jupyter_client`` is running on—are now
+  launched with the same Python executable as the launching process (:ghpull:`306`).
+  This extends the special handling of ``python`` added in 5.0.
+- Command line arguments specified by a kernelspec can now include
+  ``{resource_dir}``, which will be substituted with the kernelspec resource
+  directory path when the kernel is launched (:ghpull:`289`).
+- Kernelspecs now have an optional ``metadata`` field to hold arbitrary metadata
+  about kernels—see :ref:`kernelspecs` (:ghpull:`274`).
+- Make the ``KernelRestarter`` class used by a ``KernelManager`` configurable
+  (:ghpull:`290`).
+- When killing a kernel on Unix, kill its process group (:ghpull:`314`).
+- If a kernel dies soon after starting, reassign random ports before restarting
+  it, in case one of the previously chosen ports has been bound by another
+  process (:ghpull:`279`).
+- Avoid unnecessary filesystem operations when finding a kernelspec with
+  :meth:`.KernelSpecManager.get_kernel_spec` (:ghpull:`311`).
+- :meth:`.KernelSpecManager.get_all_specs` will no longer raise an exception on
+  encountering an invalid ``kernel.json`` file. It will raise a warning and
+  continue (:ghpull:`310`).
+- Check for non-contiguous buffers before trying to send them through ZMQ
+  (:ghpull:`258`).
+- Compatibility with upcoming Tornado version 5.0 (:ghpull:`304`).
+- Simplify setup code by always using setuptools (:ghpull:`284`).
+- Soften warnings when setting the sticky bit on runtime files fails
+  (:ghpull:`286`).
+- Various corrections and improvements to documentation.
+
+
 5.1
 ===
 

docs/conf.py

@@ -34,6 +34,7 @@
     'sphinx.ext.autodoc',
     'sphinx.ext.intersphinx',
     'sphinx.ext.napoleon',
+    'sphinxcontrib_github_alt',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -55,6 +56,8 @@
 copyright = '2015, Jupyter Development Team'
 author = 'Jupyter Development Team'
 
+github_project_url = "https://github.com/jupyter/jupyter_client"
+
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.

docs/environment.yml

@@ -3,8 +3,10 @@ channels:
   - conda-forge
 dependencies:
 - pyzmq
-- python==3.5
+- python==3.7
 - traitlets>=4.1
 - jupyter_core
 - sphinx>=1.3.6
 - sphinx_rtd_theme
+- pip:
+  - sphinxcontrib_github_alt

docs/index.rst

@@ -23,6 +23,7 @@ with Jupyter kernels.
 
    kernels
    wrapperkernels
+   kernel_providers
 
 .. toctree::
    :maxdepth: 2

docs/kernel_providers.rst

@@ -0,0 +1,156 @@
+================
+Kernel providers
+================
+
+.. note::
+   This is a new interface under development, and may still change.
+   Not all Jupyter applications use this yet.
+   See :ref:`kernelspecs` for the established way of discovering kernel types.
+
+Creating a kernel provider
+==========================
+
+By writing a kernel provider, you can extend how Jupyter applications discover
+and start kernels. For example, you could find kernels in an environment system
+like conda, or kernels on remote systems which you can access.
+
+To write a kernel provider, subclass
+:class:`jupyter_client.discovery.KernelProviderBase`, giving your provider an ID
+and overriding two methods.
+
+.. class:: MyKernelProvider
+
+   .. attribute:: id
+
+      A short string identifying this provider. Cannot contain forward slash
+      (``/``).
+
+   .. method:: find_kernels()
+
+      Get the available kernel types this provider knows about.
+      Return an iterable of 2-tuples: (name, attributes).
+      *name* is a short string identifying the kernel type.
+      *attributes* is a dictionary with information to allow selecting a kernel.
+
+   .. method:: make_manager(name)
+
+      Prepare and return a :class:`~jupyter_client.KernelManager` instance
+      ready to start a new kernel instance of the type identified by *name*.
+      The input will be one of the names given by :meth:`find_kernels`.
+
+For example, imagine we want to tell Jupyter about kernels for a new language
+called *oblong*::
+
+    # oblong_provider.py
+    from jupyter_client.discovery import KernelProviderBase
+    from jupyter_client import KernelManager
+    from shutil import which
+
+    class OblongKernelProvider(KernelProviderBase):
+        id = 'oblong'
+
+        def find_kernels(self):
+            if not which('oblong-kernel'):
+                return  # Check it's available
+
+            # Two variants - for a real kernel, these could be something like
+            # different conda environments.
+            yield 'standard', {
+                'display_name': 'Oblong (standard)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+            yield 'rounded', {
+                'display_name': 'Oblong (rounded)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+
+        def make_manager(self, name):
+            if name == 'standard':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '0'})
+            elif name == 'rounded':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '1'})
+            else:
+                raise ValueError("Unknown kernel %s" % name)
+
+You would then register this with an *entry point*. In your ``setup.py``, put
+something like this::
+
+    setup(...
+        entry_points = {
+        'jupyter_client.kernel_providers' : [
+            # The name before the '=' should match the id attribute
+            'oblong = oblong_provider:OblongKernelProvider',
+        ]
+    })
+
+Finding kernel types
+====================
+
+To find and start kernels in client code, use
+:class:`jupyter_client.discovery.KernelFinder`. This uses multiple kernel
+providers to find available kernels. Like a kernel provider, it has methods
+``find_kernels`` and ``make_manager``. The kernel names it works
+with have the provider ID as a prefix, e.g. ``oblong/rounded`` (from the example
+above).
+
+::
+
+    from jupyter_client.discovery import KernelFinder
+    kf = KernelFinder.from_entrypoints()
+
+    ## Find available kernel types
+    for name, attributes in kf.find_kernels():
+        print(name, ':', attributes['display_name'])
+    # oblong/standard : Oblong (standard)
+    # oblong/rounded : Oblong(rounded)
+    # ...
+
+    ## Start a kernel by name
+    manager = kf.make_manager('oblong/standard')
+    manager.start_kernel()
+
+.. module:: jupyter_client.discovery
+
+.. autoclass:: KernelFinder
+
+   .. automethod:: from_entrypoints
+
+   .. automethod:: find_kernels
+
+   .. automethod:: make_manager
+
+Kernel providers included in ``jupyter_client``
+===============================================
+
+``jupyter_client`` includes two kernel providers:
+
+.. autoclass:: KernelSpecProvider
+
+   .. seealso:: :ref:`kernelspecs`
+
+.. autoclass:: IPykernelProvider
+
+Glossary
+========
+
+Kernel instance
+  A running kernel, a process which can accept ZMQ connections from frontends.
+  Its state includes a namespace and an execution counter.
+
+Kernel type
+  The software to run a kernel instance, along with the context in which a
+  kernel starts. One kernel type allows starting multiple, initially similar
+  kernel instances. For instance, one kernel type may be associated with one
+  conda environment containing ``ipykernel``. The same kernel software in
+  another environment would be a different kernel type. Another software package
+  for a kernel, such as ``IRkernel``, would also be a different kernel type.
+
+Kernel provider
+  A Python class to discover kernel types and allow a client to start instances
+  of those kernel types. For instance, one kernel provider might find conda
+  environments containing ``ipykernel`` and allow starting kernel instances in
+  these environments.

docs/kernels.rst

@@ -6,19 +6,30 @@ Making kernels for Jupyter
 
 A 'kernel' is a program that runs and introspects the user's code. IPython
 includes a kernel for Python code, and people have written kernels for
-`several other languages <https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages>`_.
+`several other languages <https://github.com/jupyter/jupyter/wiki/Jupyter-kernels>`_.
 
-When Jupyter starts a kernel, it passes it a connection file. This specifies
+At kernel startup, Jupyter passes the kernel a connection file. This specifies
 how to set up communications with the frontend.
 
-There are two options for writing a kernel:
+There are three options for writing a kernel:
 
 1. You can reuse the IPython kernel machinery to handle the communications, and
    just describe how to execute your code. This is much simpler if the target
    language can be driven from Python. See :doc:`wrapperkernels` for details.
 2. You can implement the kernel machinery in your target language. This is more
    work initially, but the people using your kernel might be more likely to
    contribute to it if it's in the language they know.
+3. You can use the `xeus <https://github.com/QuantStack/xeus>`_ library that is
+   a C++ implementation of the Jupyter kernel protocol. Kernel authors only need to
+   implement the language-specific logic in their implementation
+   (execute code, auto-completion...). This is the simplest
+   solution if your target language can be driven from C or C++: e.g. if it has
+   a C-API like most scripting languages. Check out the
+   `xeus documentation <https://xeus.readthedocs.io/>`_ for more details.
+   Examples of kernels based on xeus include:
+     - `xeus-cling <https://github.com/QuantStack/xeus-cling>`_
+     - `xeus-python <https://github.com/QuantStack/xeus-python>`_
+     - `JuniperKernel <https://github.com/JuniperKernel/JuniperKernel>`_
 
 Connection files
 ================
@@ -132,11 +143,21 @@ JSON serialised dictionary containing the following keys and values:
   is found, a kernel with a matching `language` will be used.
   This allows a notebook written on any Python or Julia kernel to be properly associated
   with the user's Python or Julia kernel, even if they aren't listed under the same name as the author's.
+- **interrupt_mode** (optional): May be either ``signal`` or ``message`` and
+  specifies how a client is supposed to interrupt cell execution on this kernel,
+  either by sending an interrupt ``signal`` via the operating system's
+  signalling facilities (e.g. `SIGINT` on POSIX systems), or by sending an
+  ``interrupt_request`` message on the control channel (see
+  :ref:`msging_interrupt`). If this is not specified
+  the client will default to ``signal`` mode.
 - **env** (optional): A dictionary of environment variables to set for the kernel.
   These will be added to the current environment variables before the kernel is
-  started.
+  started.  Existing environment variables can be referenced using ``${<ENV_VAR>}`` and
+  will be substituted with the corresponding value.  Administrators should note that use
+  of ``${<ENV_VAR>}`` can expose sensitive variables and should use only in controlled
+  circumstances.
 - **metadata** (optional): A dictionary of additional attributes about this
-  kernel; used by clients to aid clients in kernel selection. Metadata added
+  kernel; used by clients to aid in kernel selection. Metadata added
   here should be namespaced for the tool reading and writing that metadata.
 
 For example, the kernel.json file for IPython looks like this::