Get best of the week

Visual Debugger for Jupyter

The progress of most software projects is based on small changes, which, before moving on, are carefully evaluated. Fast results of code execution and high speed of iterative development are some of the main reasons for the success of Jupyter. Especially in the field of scientific research.

Jupyter users love to experiment with notebooks; they use notebooks as an interactive means of communication. However, if we talk about the tasks of classical software development, for example, refactoring a large code base, then we can say that to solve such problems Jupyter is often changed to regular IDEs.


The JupyterLab environment The Jupyter

project has been working for several years to close the gap between notebooks and regular IDEs. These efforts are largely represented by the JupyterLab platform, which gives the developer a more advanced and convenient interface, which includes a file manager, text editors, consoles, notepads.

True, until recently, something was missing in JupyterLab. This is about what was the main reason why users were forced to switch to other environments. Jupyter users lacked a visual debugger. Users, especially those who are used to regular IDEs, have long asked for this feature.

Debugger for Jupyter


Today, after several months of development, we are pleased to present the first public release of the visual debugger for Jupyter!

This is only the first release, but the debugger already allows you to set breakpoints in notebook cells and in source files. It allows you to examine the contents of variables, view the call stack. And his capabilities are not limited to this.


Jupyter Visual Debugger in action

Testing the debugger on the binder project site



The binder project You can

try a new debugger in the case on the binder project website. Here is a demo notepad with which you can experiment.

Installation


The frontend of the debugger can be installed as a JupyterLab extension:

jupyter labextension install @jupyterlab/debugger

In future releases, the debugger frontend will be included in JupyterLab by default.

On the backend, for the debugger to work, there must be a kernel in which the Jupyter Debug Protocol is implemented (we will talk more about it below). So far, the only kernel that implements this protocol is xeus-python . It is the Jupyter core for the Python programming language. (The project development plans also include support for the debugger protocol in ipykernel.)

You can install the xeus-python kernel like this:

conda install xeus-python -c conda-forge

After xeus-python and the debugger extension are installed, everything should be ready to work with the Jupyter visual debugger.

Pay attention to the fact that there are also PyPI “wheels” for xeus-python, but they still have experimental status, and their availability depends on the platform.

Jupyter Debug Protocol Details


▍New message types for Control and IOPub channels


Jupyter kernels (the part of the infrastructure that executes user code) communicate with other parts of the system using a well-developed interprocess communication protocol .

There are several communication channels:

  • A channel Shellthat works according to the request / response scheme and is used, for example, for requests for code execution.
  • A channel IOPub, which is a one-way communication channel that runs from the core to the client. It is used, for example, to redirect standard output streams ( stdoutand stderr).
  • A channel Controlsimilar to a channel Shellbut operating on a different socket. Due to this, his messages are not queued together with requests for code execution and have a higher priority. The channel Controlhas already been used for queries Interruptand Shutdown. We decided that we would use it for commands sent to the debugger.

Two new message types have been added to the protocol:


▍ Debug Adapter Protocol


The key principle of the Jupyter architecture is the lack of binding to a certain programming language. Therefore, it is important that the debugger protocol supports adaptation to various kernel implementations.

A popular debugging standard is the Debug Adapter Protocol (DAP), developed by Microsoft. This is a JSON-based protocol that, for example, underlies the Visual Studio Code debugging subsystem. This protocol supports many backends designed for different languages.

As a result, it was completely natural for us to use DAP messages through recently added messages debug_[request/reply]and debug_event.

However, in the case of Jupyter this was not quite enough:

  • , , , Jupyter ( , , ). debug_request.
  • , , Jupyter, , , , , , .

The content of requests to the debugger and its answers, in addition to these two differences, corresponds to DAP.

All of these extensions to the Jupyter kernel protocol have been proposed as candidates for inclusion in the official specification. A related Jupyter development proposal (JEPyter Enhancement Proposal, JEP) can be found here .

Xeus-python is the first Jupyter kernel to support debugging


Xeus is a C ++ implementation of the Jupyter kernel protocol. This, in itself, is not the core. This is a library that helps develop kernels. This library is useful in developing kernels for languages ​​that have a C- or C ++ - API (like Python, Lua, or SQL). She takes care of the time-consuming tasks of implementing the Jupyter messaging protocol. This allows the author of the kernel to focus on tasks related to the interpretation of language constructs: on code execution, on its analysis, and so on.

Several cores were developed using xeus, including the popular xeus-cling core for C ++, based on the C ++ cling interpreter created in CERN. The xeus-python kernel is an ipykernel alternative based on xeus. Herematerial about the first release of the xeus-python kernel.

The xeus-python kernel is suitable for the first implementation of the debugger protocol for several reasons:

  • It supports competitive code execution, which allows you to perform work with the channel Controlin a separate thread.
  • It features a compact code base. Thanks to this, this kernel is a good "sandbox" in which it is convenient to implement something new. Implementing the first version of the debugger protocol in ipykernel would require more serious refactoring and more effort to coordinate changes in the early stages of development.

▍ Xeus-python kernel development plan


Among the short-term development goals of xeus-python are the following:

  • Adding support for IPython features. Now this, compared to ipykernel, is the main missing feature of xeus-python.
  • Improvement of PyPI- "wheels" of xeus-python.

▍About other cores


What has already been done in terms of support for debugging in the frontend can be used with any kernels that implement the extended kernel protocol.

This year we will work to provide debugging support in as many cores as possible.

Debugging support is coming soon for other xeus-based kernels that have much in common with the xeus-python kernel. For example, for such as xeus-cling.

Debugger Front End Architecture Details


The debugger extension for JupyterLab gives users the mechanisms that are usually expected from the IDE:

  • Sidebar with windows that implement the following features: monitoring variables, viewing breakpoints, viewing code, analyzing call stacks.
  • The ability to set breakpoints directly in the code, in particular in the code located in the notepad cells and in the consoles.
  • Visual markers indicating the position at which code execution stopped.

When working with Jupyter notebooks, information about the status of code execution is stored in the kernel. But the cell code can be executed, after which this cell can be deleted from the notebook. What should happen if the user, during debugging, wants to enter a remote block of code?

The extension supports this specific scenario and is able to display the code of a previously executed cell in read-only mode.


Switching to the code located in the remote cell You

can debug the code located in the consoles and in the files.


Debugging code from the console in JupyterLab


Debugging in JupyterLab the code in the file

Debugging can be enabled at the notepad level, which will allow users to debug the notepad and, at the same time, work with another notepad.


Debugging multiple notebooks at the same time

Variables can be viewed using a tree or table view.


Tools for

Exploring Variables The debugger extension for JupyterLab has been designed to work with any kernel that supports debugging.

Due to the fact that the extension relies on the DAP protocol, it abstracts the features of various languages ​​and gives the user a consistent debugging interface.

The following diagram shows the flow of messages sent during a debugging session between the user, the JupyterLab extension, and the kernel.


Using Debug Adapter Protocol in a debugger ( source )

Future plans


In 2020, we plan to seriously improve the debugger. Namely, we are talking about the following:

  • Support for outputting various MIME types in the variable explorer window.
  • Support for working with conditional breakpoints in the debugger interface.
  • Improving the usability of the debugger interface.
  • Implementing the ability to debug Voilà applications from the extension @jupyter-voila/jupyterlab-preview.

Dear readers! Have you tried the Jupyter visual debugger?

More articles:


All Articles