Published on
-
7 mins read

Google Summer of Code 2021 Work Report

Authors
gsoc-work-report

Project Abstract

MetaCall Jupyter Kernel is an open-source wrapper kernel that implements cross-language function calls through the MetaCall Core and the Polyglot REPL. MetaCall Core is an open-source library that brings the polyglot programming experience to Developers. With MetaCall, developers can embed different programming languages through an easy-to-use high-level API.

Leveraging IPython, this project aimed to create a kernel where the MetaCall Core can be loaded and launched through a Jupyter Notebook interface. With this Notebook, the users can try out writing and embedding code in different programming languages. It would also make it possible to use the MetaCall API and achieve the goal of bringing a unified interface for developers to try out and leverage MetaCall.

This year's project achieved the following:

  • Developed the initial minimal viable product centred around bringing a Jupyter kernel for MetaCall.
  • Built and integrated a Polyglot REPL using the MetaCall Core to the IPython through a subprocess.
  • Implemented and improved user-experience features centred around using the kernel with the Jupyter client.
  • Implemented CI/CD pipelines for the project with GitHub actions and set-up container-based deployment with Binder.
  • Created and delivered a full-fledged documentation website for the MetaCall Jupyter Kernel using Sphinx and ReadTheDocs.

Mentors: @giarve

Milestones

Setting up the Kernel

The first step of the project was to set up the wrapper kernel using IPython and expose it on a Jupyter client. Initially decided to go with Xeus, a C++ framework used to build Jupyter kernels, we moved forward with IPython. I created a wrapper kernel, which can be built and set up using Setuptools as the build system. The initial kernel forks a MetaCall exec subprocess and loads the code on a temporary file before passing it to the exec and retrieving the standard output.

The mechanism was not sufficient to move forward with defining extra languages (we provided initial support for NodeJS), hence we moved forward with automated language inference models with guesslang. Guesslang, provided a high-level API to automatically detect programming languages through the code. This affected the processing speed and we had to do away with the same later, in favour of user magics for faster inference.

Example usage of the MetaCall Jupyter wrapper kernel

Integrating Polyglot REPL

The second step of the project was to integrate the Polyglot REPL. For this milestone, I collaborated with @viferga and worked on a NodeJS-based REPL (Read-Evaluate-Print-Loop) which loads a Python and a Node REPL through the MetaCall's meta-object protocol. The REPL, which makes use of a high-level API, made it possible to extend its usage as per our understanding and needs and integrate it with any client interface.

The second part of this was integrating the REPL to the MetaCall Jupyter kernel. To make this possible, we made use of a subprocess that launches the REPL in the background and listens for standard inputs passed. This approach is commonly used by other wrapper kernels, hence we followed the same design principle. As each evaluation is passed to the kernel, the standard output from the REPL is fetched and displayed on the kernel in a clean logger output. This now allows a user to both, load, inter-change, inspect and evaluate the meta-object protocol from the kernel interface itself.

Example usage of the Polyglot REPL on the MetaCall Jupyter

Implementation of user-experience features

The third step of the project was to implement the user-experience features. These user-experience features defined the usability of the client interface and how the users can interact with the REPL. To achieve this, we decided to tweak the standard magics used on the Jupyter notebook interface, to match our requirements.

The user-experience centric features I added were:

  • ! : Run a Shell Command on the MetaCall Jupyter Kernel
  • $shutdown : Shutdown the MetaCall Jupyter Kernel
  • $inspect : Inspects the MetaCall to check all loaded functions
  • $loadfile: Loads a file onto the MetaCall which can be evaluated
  • $newfile: Creates a new file and appends the code mentioned below
  • %repl <tag>: Switch from different REPL (available tags: node, py)
  • >lang: Execute scripts using the MetaCall exec by saving them in a temporary file (available languages: python, javascript)
  • $loadcell <tag>: Loads a function onto the MetaCall to be evaluated
  • $help: Check all the commands and tags you can use while accessing the MetaCall Kernel
  • $available: Checks all the available REPLs on the Kernel

Implementation of CI/CD pipelines

For continuous testing, I built and managed the CI/CD pipelines for the project. The CI pipelines are built using GitHub Actions. Testing has been implemented using the pytest module, and the CI is responsible for building and validating the source and the documentation.

The primary CI file can be found on the .github/workflows/ci.yml and the codeql-analysis.yml respectively.

GitHub Actions in action with every Pull Request and push

Documentation website

For documentation, I built and published the MetaCall Kernel and the source documentation on Sphinx and ReadTheDocs. Sphinx is a documentation generator for Python and was the first choice of going ahead with documenting the project. The standard documentation ensures that the community support remains intact long after the coding period of Google Summer of Code is over. It would also ensure that future contributors can work on adding support for many languages as MetaCall Core extends the functionalities for more languages on the upstream.

Check out the documentation website on metacall-jupyter-kernel.readthedocs.io.

PyPi release

The kernel has been released on PyPi at pypi/metacall-jupyter. To get started, push on the terminal:

pip3 install metacall-jupyter

Pull Requests

  • #4: Add Jupyter kernel setup feat
  • #5: GitHub Action for CI Build CI
  • #6: Documentation website feat
  • #8: Linting with Flake8 and Black chore
  • #10: MetaCall launcher for Jupyter kernel feat
  • #12: Fixing Node execution with MetaCall fix
  • #14: Adding automated language check with guesslang and examples feat
  • #16: Custom log message display for unsupported languages feat
  • #18: Add magic support feat
  • #20: Add support for shell commands feat
  • #22: Fixing user experience features fix
  • #23: Adding examples + GH actions + fixing magics bug fix, CI
  • #37: Embeding MetaCall REPL on the kernel feat
  • #47: Adding red font for error output feat
  • #50: Adding support for $newfile magics feat
  • #51: Adding support for %inspect and %load magics feat
  • #54: Adding support for $help and $shutdown feat
  • #56: Adding support for $loadcell feat
  • #60: Adding support for $available, fixing Dockerfile and adding help links feat, fix
  • #62: Implement history access for the kernel feat
  • #64: Adding examples for the kernel usage chore
  • #68: Adding unit-tests for the kernel chore
  • #73: Adding documentation docs

Tags:

  • Feature : feat
  • Bug Fix : fix
  • Documentation: docs
  • Maintenance, testing and updates: chore

Statistics

  • Total PRs sent: 24
  • Total issues opened: 20
  • Total commits: 49
  • Total lines of code added: 4372
  • Total lines of code deleted: 1664
  • Total lines changed: 6036

Acknowledgements

It was a great experience working with the MetaCall team for GSoC'21. Thanks to my mentor Gil Arasa Verge and Vicente Eduardo Ferrer Garcia for giving me this wonderful opportunity and mentoring me. I look forward to continuing being a part of the MetaCall community and continue contributing to Jupyter kernel as well as other projects, like Core, along with helping new contributors find their way around the community.

Originally published here: https://medium.com/@metacall/google-summer-of-code-2021-development-of-jupyter-kernel-for-metacall-ddd552319ccf

Discuss on Twitter •