Creating a Chimera From 3 Languages and Having Fun Along the Way: Calling Rust From Python From Javascript

Have you ever considered how hard it can be to integrate codebases from multiple programming languages and ship the resulting chimera via the modern browser environment? Well, it was another rainy afternoon and I had nothing better to do. So I tried integrating Rust, Python, and JavaScript in a single application with a bit more complex data flow than the usual def add(a: int, b: int) -> int example. At first, it sounded quite complex and complicated, but thanks to the power of WebAssembly (and a bit of ugly code) it turned out to be quite doable and pleasant.

Let’s dive into the experience and possible lessons I drew from implementing this minimalistic multi-language demo. Hopefully, you will find it as interesting to read as it was for me to hack around and implement. And may whatever we create along the way look a bit better than the picture above - AI rendition of an anime-style chimera from crabs and snakes.

What are we going to do

I wanted to avoid a completely trivial use case for an application, but also I am not willing to spend more than a few hours on it. As a middle-ground, I’ve created a sorting visualization (you know, like those which show why bubble sort is slow). The logic will be split as follows:

The data flow is also depicted in the diagram below:

The plan for today is the following:

  1. Create a Rust library that implements one or several sorting algorithms and allows to peek into the data being sorted after each algorithm’s iteration
  2. Build the library in a way that allows it to be run from Python
  3. Write a Python module that would receive data to be sorted, pass it to the Rust library, and create a visualization of the process
  4. Re-build Rust in a WebAssembly-friendly way
  5. Integrate Rust & Python modules into a JavaScript & HTML widget
  6. Add visualization for the algorithm work into the HTML page
  7. Celebrate!

But before jumping to coding right away, let’s first take a look at our result. To see our application in action, please enter how many numbers are to be generated and sorted (I find numbers between 10 and 40 to produce the most readable pictures) and click “Start”.

When you do so our 3-layer application with bubble sort the number of random integers you’ve selected and displays its progress at every algorithm’s iteration.

Now that you have a taste of what we can do, let’s start coding.

Creating the baseline Rust library

The first step we need to cover is the “core logic” of our “complex application”. I’ll be using a Maturin toolchain (fka pyo3-pack). It allows building (and even publishing) crates with pyo3, rust-cpython, cffi, and uniffi bindings as well as Rust binaries as Python packages with minimal configuration.

There are of course other options for Rust <-> Python interoperability, but what sold me on Maturin is the fact that it can build wheel packages. Then these wheels can be installed and imported as you would do with any other Python-native module, abstracting away the Rust nature of the dependency.

First of all, let’s install the Maturin and create a new project:

python -m venv rusting-python
source ./rusting-python/bin/activate
pip install maturin
maturin new rusting-python-core
cd rusting-python-core

Inside the rusting-python-core directory, we find a typical Rust project structure. For now, let’s focus on the src/lib.rs and define the core structure for our application. We will import pyo3’s version of prelude and define the structure for the main logical block:

use pyo3::prelude::*;

#[pyclass]
struct Sorter {
    data: Vec<i32>,
    // we need these loop counters here
    // to allow sorting function to do 
    // one step at a time 
    i: i32,  
    j: i32,
    sorted: bool,
}

Here we already see the first difference from writing purely for Rust – we use a #[pyclass] macro to tell pyo3 that the Sorter should be available as a Python class when it is built.

And now let’s add the methods that we need for our app. All the methods that we want to export need to have a #[pymethods] macro to let pyo3 know what to wrap into the class we defined earlier:

#[pymethods]
impl Sorter {
    // methods described below go here
}

Firstly we need to define a new method that will act as a constructor when compiled into a Python module.

#[new]
fn new(data: Vec<i32>) -> Self {
    Sorter { data: data, i: 0, j: 0, sorted: false}
}

As you can see, we need to initialize data with the vector we want to sort. Additionally, we set the counters and sorting state to their initial values.

An interesting side note is that we need to pass the values in and out of the Rust module by value, so our Sorter and the new method deal with an actual Vec<i32> and not a reference to it.

Now to the most important (and ugly) part of the code – the sorting itself. For the initial approach, I have decided to go with an old (for sure) and trusty (not really) Bubble Sort for it is the most trivial algorithm I could code:

fn step(&mut self) -> Vec<i32> {
    if self.i < self.data.len() as i32 {
        if self.j < self.data.len() as i32 - self.i - 1 {
            if self.data[self.j as usize] > self.data[(self.j + 1) as usize] {
                self.data.swap(self.j as usize, (self.j + 1) as usize);
            }
            self.j += 1;
        } else {
            self.j = 0;
            self.i += 1;
        }
    } else {
        self.sorted = true;
    }
    self.data.clone()
}

For the most part, this code is as ugly as it is because we want to iterate one step at a time. In the case of Bubble Sort a step is each comparison. At the end of the method, we always return a clone of the vector being sorted. As before we need to return a value, not a reference.

Finally, we need to let the caller of this function know when the input is sorted. To allow for this I introduce a sorted flag – not the most elegant solution, but one that is simple and works.

And the final step of Rust coding for today is to expose the sorted flag. For this, we need our final piece of Rust code for now:

fn is_sorted(&self) -> bool {
    self.sorted
}

And now we can try and introduce our nice and safe Rust to the explicitness of Python.

Building Rust module for Python testing

Another nice feature of Maturin is that it allows for a very seamless developer experience. There are 2 main options to test our library in Python:

For iterative development and testing, I definitely prefer the first option. All you need to do is open the Rust library’s folder and run:

maturin develop

Under the hood, it will build the package & install it into your active Python interpreter.

Now, we can run Python and try the following code:

from rusting_python_core import Sorter

sorter = Sorter([5, 2, 1])

while not sorter.is_sorted():
    print(sorter.step())

And, ideally, Python will produce the following output:

[2, 5, 1]
[2, 1, 5]
[2, 1, 5]
[1, 2, 5]
[1, 2, 5]
[1, 2, 5]
[1, 2, 5]

Yes, bubble sort is not the most efficient way to sort stuff. But hey, it works! We have just got our Python script to call Rust module as if it’s just another Pythonic dependency.

Making sorting visual

It’s time to make the Python layer do something useful. Today “something useful” is plotting the sorting progress after each step. We will be using the matplotlib package because why not?

The code is fairly trivial:

import random
import pathlib
import matplotlib.pyplot as plt 

from rusting_python_core import Sorter

random_integers = [random.randint(0, 100) for _ in range(15)]
sorter = Sorter(random_integers)

file_path = pathlib.Path(__file__).parent.resolve()

image_counter = 1
while not sorter.is_sorted():
    plt.clf()
    plt.axis('off')

    data = sorter.step()
    xs = list(range(len(data)))
    plt.bar(xs, data)
    file_name = "%04d" % (image_counter,)
    plt.savefig(f"{file_path}/{file_name}.jpg", format='jpg')

    image_counter += 1

We loop over and over until is_sorter returns True, which in turn means that the algorithm finished running.

On each iteration, the script gets the current state of the data we are trying to sort, clears the previous render in matplotlib, and creates a new image, which is then pushed onto the disk.

When successfully run it will produce more than 100 images depicting the sorting process. Below is a gif that shows how our sorting works:

Bringing it all together

Now we finally get to the most interesting part: integrating all of the previous steps into the browser environment. There are a few things we need to take care of:

  1. Build our Rust library for the WebAssembly target
  2. Re-write the Python script to expose clearly defined methods to interact with it
  3. Create wrapper code with JavaScript to initialize & call our Python & Rust core

Building Rust for WebAssembly

Here we are lucky since Maturin & Rust combo makes crosscompilation extremely developer-friendly. First of all, we need to add a new compilation target via rustup:

rustup default nightly
rustup target add wasm32-unknown-emscripten

Then we will need to download & install emscripten-core linker:

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install 3.1.46
./emsdk activate 3.1.46
source ./emsdk_env.sh

You can notice that we are fixing the version of the emsdk. The reason for this is that later we will need it to match the version that was used to build our browser-based Python environment – Pyodide.

And finally, we can build a wheel package for our library:

maturin build --release --target wasm32-unknown-emscripten -i 3.11

Defining boundaries in the Python module

Now instead of running all scripts from Python we will expose 3 different methods from it:

For the initialization part, it’s quite straightforward: create a global variable and pass a list of numbers to sort:

sorter: Sorter | None = None    

def initialize(data: list[int]):
    global sorter
    sorter = Sorter(data)

The next part is a slightly modified image generation code we’ve already seen before:

def get_step_image() -> str:
    plt.clf()
    plt.axis('off')

    data = sorter.step()
    xs = list(range(len(data)))
    plt.bar(xs, data)

    ioBuffer = io.BytesIO()
    plt.savefig(ioBuffer, format='jpg')

    base64_image = base64.b64encode(ioBuffer.read()).decode()
    return base64_image

The main difference is the way we handle the “saving” of the newly generated image. Now instead of putting it into the file system, the function uses io.BytesIO() and base64 modules to encode the image as a Base64 string which is then returned.

The last function we’ll need is a small helper method to expose the state of the sorting process:

def is_sorted() -> bool:
    return sorter.is_sorted()

The final step is to “export” them for Pyodide by making the last expression of our script contain the functions we will use from JavaScript:

# all code is above
[initialize, get_step_image, is_sorted]

Creating JavaScript client

At last, there is some light at the end of the tunnel. Now we have both Rust & Python parts done and only need to glue them together with some JavaScript. As I have already briefly mentioned before, we’ll be using Pyodide to run the Python module from the JavaScript environment. It can be run both in browsers and from Node.js, but for today’s experiment we’ll go with the browser option.

Essentially, Pyodide provides an interpreter environment into which one can install libraries using MicroPip either from the pip repository or from the wheel package.

We will install the Rust library by simply providing a path on which it can be downloaded into the browser:

const pyodide = await loadPyodide();
await pyodide.loadPackage("micropip");
const micropip = pyodide.pyimport("micropip");
await micropip.install('./rusting_python_core-0.1.0-cp311-cp311-emscripten_3_1_46_wasm32.whl')

Next, we need to get callable (in JavaScript) objects from our Python code. To do so we can execute the Python module from the previous section:

const pythonScript = await (await fetch("./rust_sorting_visualizer.py")).text();

const [initialize, step, is_sorted] = pyodide.runPython(pythonScript);
document.rusting_python_core = {initialize, step, is_sorted};

And now let’s recreate the loop we already had in our Python implementation:

const randomIntegers = Array.from({length: 15}, () => Math.floor(Math.random() * 100));
document.rusting_python_core.initialize(randomIntegers);
while (!document.rusting_python_core.is_sorted()) {
    const res_py = document.rusting_python_core.step();
    const imgBase64 = res_py;
    document.querySelector("#blobImage").src = 'data:image/jpeg;base64,'+ imgBase64;
    await new Promise(r => setTimeout(r, 30));
}

The biggest difference is that now instead of saving each image to the disk we get it as a base64-encoded string and set it into the src attribute of our image element on the web page.

And that’s it! Congratulations for holding on throughout this experiment!

Lessons and outtakes

What does it mean in practice? Should we use this “architecture” whenever possible? Probably not so much. But on the other hand, it once again highlights the biggest strength of WebAssembly – it gives us insane portability for otherwise platform-dependent codebases. Be it a legacy C library that implements a parser for a long-forgotten language, or a cutting edge web3 framework, or anything in between – as WebAssembly is becoming more and more popular and supported we can be sure that interoperability between different technologies will get even better.

For me, one such example was one of the libraries I was using in my pet project. Namely, metar-taf-parser which I use to transform string METARs into JavaScript (or rather TypeScript) friendly objects. Recently I tried migrating one part of the project to Python only to find that available open-source METAR parsers fall short of what I’d ideally want. Not so long ago I’d had a choice of either suffering and using what I could find or trying to re-invent a wheel. But now after a few hours of playing around with dependencies and bundling, I was a happy user of the familiar tooling albeit in a different language.

To summarise, the next time when you are faced with lacking a needed (or desired) tooling or libraries for your language, try looking around the “neighboring yards” before giving up to despair and spending months re-inventing the wheel.

Additional reading