Hey Pd, meet Jade

In the previous article, I described a dataflow environment called “Pure-data” and mentioned that I’m working on something similar. It uses the same dataflow approach as Pd, but it’s (much) simpler so it can fit into a low-end ARM Cortex microcontroller. This design is called “Jade” and it’s written in C++.

Hello Jade …#

Jade currently works ok as proof-of-concept, but two aspects of it still need to be improved:

  1. The core dataflow mechanism uses messages which support nested data structures. I’m currently allocating these as small custom objects on the heap, which is inefficient when the bulk of the messages are expected to consist of one or two items, and most often just simple scalars.

  2. The construction of such “ad-hoc” messages needs to be very easy and fit naturally in C++, otherwise the task of generating these messages and sending them around will add too much friction. A string would be too inconvenient if all you want is to send a “tagged value” such as every 5 ms, where 5 and ms are values which are determined at run time.

In essence, what I’m after is a convenient (and low-overhead) way to construct small ad-hoc data structures in the same way as in the Python or Tcl scripting languages, but from C++. This quest is still on-going …

Terminology#

I’ve decided to use terms in Jade which differ somewhat from Pure-data’s conventions:

  • The basic building block in Jade is called a gadget (“object” in Pd).
  • Gadgets can be combined and wired together as a circuit (“patch” in Pd).
  • Note that each circuit is-a gadget, i.e. circuits can be nested.
  • The description of gadgets + wires to create a circuit is called a design (“abstraction” in Pd).

The reasons for this change in terminology are: “object” is confusing as term in C++ and “patch” is music-specific. Circuits are a better fit for embedded µCs with electronic devices attached to them.

Implementation#

Gadgets are defined as C++ classes derived from this struct (several details omitted for brevity):

struct Gadget {
    Gadget ();
    virtual ~Gadget ();

    virtual bool init (Msg msg);
    virtual void trigger (uint8_t in, Msg msg) =0;

    void post (Msg msg);
    void emit (uint8_t out, Msg msg);
};

Each gadget gets an init() call (once) to let it set up whatever internal state it wants to manage. After that, whenever a message is received, the trigger() method is called, identifying which (0-based) inlet got the message as well as the actual message. It is up to the gadget to pick apart the incoming message, handle it (or report an error via post()), and send out any number of messages via emit(). Gadgets do not know where messages come from, nor where they go to (if anywhere).

Circuit designs are defined as strings and parsed in Jade. Here is an example which passes messages through while also “printing” each one to some suitable output stream, e.g. a serial port:

c logger
g inlet
g print "log:"
g outlet
w 0 0 1 0
w 0 0 2 0

There are three gadgets in this “logger” circuit: #0 is an inlet, #1 is a pre-defined print gadget, and #2 is an outlet. The first wire connects the inlet gadget to the print gadget. The second wire passes all messages from the inlet to the outlet. Each incoming message is used twice: first printed and then passed on.

In this example, the print gadget has an optional extra argument, which is sent to its init() method on startup, allowing it to save that information and tag each messages it is going to process later on.

From the outside, the above “logger” example is simply a gadget with one inlet and one outlet which can be inserted in any other circuit to view all the messages coming through. Since circuits can be altered at runtime, this allows peeking inside a running application after the fact.

An application using Jade consist of three parts:

  1. The Jade “engine” which runs the whole system, pushing messages through active gadgets and circuits via their wiring. When there’s nothing to do, the system can enter low-power sleep mode.
  2. A set of gadgets such as inlet, print, and outlet. These are the “primitives” used in an application. They are implemented as classes derived from Gadget. Many more need to be defined, such as Pd’s metro and delay to support periodic tasks and time delays. Being C++, all gadgets must be compiled and uploaded as firmware before they can be used in a µC.
  3. A library with circuit designs, ready to use for assembling a complete application. A large collection of such designs could be pre-loaded in flash memory. For turn-key use, the entire application can be stored in flash, but during development they can just as easily be sent over a serial port, allowing for very quick development changes.

It’s too early to say how this will all work out, but that’s where I’m trying to take Jade and dataflow.

Status#

As mentioned, the above works right now, but I’m not satisfied with the way messages are constructed and passed around. I’m now exploring a compact byte-array design which avoids frequent allocations. It still supports nesting such as abc [123 "ha ha\n" 456] def but using a more efficient binary representation than these strings, and much easier to encode / decode in C++.