DroneNode Modules and Loosely Coupled Pub/Sub Messaging

 

Each boat (or other device) is based on a custom ESP32-based motherboard running our DroneNode firmware.  The firmware exposes a number of sensors, actuators, telemetry interfaces and control modules that can be configured for any given boat (device) through a simple config file.  To keep software modules loosely coupled, they communicate using a pub/sub messaging model.  This post describes both of these features in more detail.

See the DroneLink Architecture - "Boat Brains" article for more detail on the overall system including the central server and web management interface.

DroneNode Firmware Architecture

The DroneNode firmware architecture is grouped into 5 logical blocks, as illustrated in the following diagram:

1. Hardware - The physical interfaces exposed by the ESP32, including onboard Flash storage (LittleFS filesystem) and any peripheral devices connected to the ESP IO (digital/analog IO, I2C, SPI, Serial)
2. Done System - Low-level functionality responsible for:
1. Read a config file to create and configure module instances
2. Manage the hardware interfaces via an abstraction layer - including pin/port registration from software modules to detect IO conflicts and allows for future portable between different microcontrollers 
3. Low-level diagnostics exposed via a web mgmt interface hosted by the ESP32 local web server
4. Over the Air (OTA) software updates
3. Drone Modules - each encapsulating a block of functionality with a simple interface, representing either physical devices (sensors, actuators) or logical control blocks (e.g. Navigation)
4. Internal Communications (DroneLink) - the publisher-subscriber messaging service that allows modules to communicate with each other and also to communicate with other nodes on the network
5. External Communications (DroneMesh) - encapsulates DroneLink messages and routes them over the mesh network.  Includes:
1. Managing the various network interfaces for different physical transports
2. Traffic management using packet-level prioritisation
3. A remote interface for mesh diagnostics (congestion levels, trace route)

Drone Modules

It's the Drone Modules that actually provide the "useful" functionality for making an autonomous boat.   The firmware includes definitions for a number of module types, each can then be instanced one or more times and configured using a config file.  

Each module publishes a number of parameters that be consumed by other modules, or simply used for monitoring/diagnostic purposes.  They can also subscribe to a number of "inputs", that is to the values other modules publish.  As an example, the Nav module will subscribe to the lat/lon location parameter published by the GPS module, so that it knows where in the world the boat is.

The full configuration guide for the Nav module is shown below as an example:

This is the full list of available modules as of the time of writing (Jan 2023):

Avoid	Sail polar calibration module
CMPS12	Manages a CMPS12 I2C Compass
Cylon	Manage a strip of NEOPixels (WS2812B) - originally created for the Kit speedboat
Depth	Manages a simple Depth sensor (sonar transducer) controlled via Trigger and Echo pins
Diagnostic	Manages a Diagnostic 128x64 I2C display (1306 driver)
HMC5883L	Manages a HMC5883L I2C Compass
INA219	Manages an INA219 I2C power monitoring module
INA3221	Manages a 3-channel INA3221 I2C power monitoring module
LSM9DS1	Manages a LSM9DS1 I2C Compass
Management	Provides overall system management and a pub/sub interface to the DroneModuleManager
Motor	Manages a Motor via an H Bridge module using PWM
NMEA	Manages a serial NMEA GPS device
Nav	Manages navigation between waypoints accounting for crosswind drift
NetworkInterface	Base class for network interfaces
ODrive	Manages an ODrive module via serial
PanSerial	Generate serial commands for a slave pan controller module
PanTilt	Generate servo commands for a pan/tilt head to track a target (e.g. for camera or antenna trackers)
Polar	Sail polar calibration module
QMC5883L	Manages a QMC5883L I2C Compass
RFM69Telemetry	Manages DroneLink telemetry using an RFM69HW radio module
Receiver	Reads up to 4 PWM channels from an RC receiver
Sailor	Control a typical sailing boat
SerialTelemetry	Manages DroneLink telemetry over a serial port, optionally via a transparent telemetry module
Servo	Manages a PWM servo channel
SpeedControl	Convert a distance value (meters) into a speed control value (0..1)
Status	Monitors up to four parameters and publishes a corresponding status scene to a Neopixel module
TankSteer	Generate left and right motor speeds from target course, current heading and distance to target
Timer	Emits a configured message on a set interval (WIP)
TurnRate	Generate a turnRate command based on target vs current heading using a PID controller
UDPTelemetry	Manages DroneLink telemetry using UDP broadcast over WiFi
Waypoint	Load and manage a series of waypoints from a CSV file
WindFromWing	Estimate Wind Direction from Wing Compass and Tail position
Wind	Manages a Wind speed and direction sensor.

The web management interface includes a help system that has configuration guides for all of the available modules, along with complete configuration examples combining multiple modules for a typical application (e.g. a sail boat).

The full module-graph for one of the Paddle boats is shown below.  It shows the subscriptions between various modules as links to help visually debug the configuration.  There are also a number of modules that don't need any subscriptions to function correctly, for example. the network transport modules (RFM69, UDPT) and the general Management module.

Publish-Subscriber Messaging

See publish-subscriber model on Wikipedia for background theory.

Nodes, modules and their associated parameters (published or subscribed) are all accessed using a publish-subscriber model with a very simple addressing scheme made up of 3x 1-byte values:

    < node ID > < module ID > < parameter ID > 

Each node on the network is assigned a unique 1-byte node ID (0 and 255 are reserved).  Each module instance running on a node is assigned a unique 1-byte ID within the config file (0, 255 also reserved).  Finally, each parameter of each module is assigned a unique 6-bit ID, hard-coded into the firmware for that particular module type.

Every module includes a set of built-in parameters that allow for general management of the module over the network:

• Parameter 1: Status; 0=disabled, 1=enabled
• Parameter 2: Module name - to allow for discovery of configured modules over the network
• Parameter 3: Error state; 0=ok, 1=error
• Parameter 4: Reset Count: How many times has the modules own watchdog service triggered a reset
• Parameter 5: Module Type: A character string specifying the module type to allow for discovery over the network
• Parameter 6: Update Interval: How many milliseconds between updates in the main loop

Parameters can be up to 16-bytes in length using one of 4 basic data types: 

• 1-byte or 4-byte unsigned integer
• Float
• Character string  

For example, a GPS location will be encoded as a pair of float values for longitude and latitude.

When a published value for a module changes the new value is sent to a message queue, which will then forward the update to any modules (locally or externally) that subscribe to that value.  

If the update is destined for an external node, it is passed to the DroneMesh manager which will take care of traffic rate limiting (avoiding flooding the network with many fast-changing values) and routing. 

To minimise unwanted network traffic, module parameters have to be explicitly "published" in the configuration file.  For example, some parameters are not needed depending on the specific boat configuration and can be left hidden.

Address Syntax and Lazy Binding

When specifying parameter addresses in the configuration file, they can be specified in numeric form or using Node/Module/Parameter names in the following syntax:

[node ID, name or @] > [module ID or name] . [parameter ID or name]

The @ symbol for a node id is shorthand for the current node.  It helps keep configuration files readable and robust to changing the node ID.

Named modules or parameters only work for local modules, they can't be used for subscriptions to other nodes on the network.  The boot-up process uses a lazy binding strategy to establish subscriptions:

1. Instance and configure module parameters, add any subscriptions to a queue to be resolved later
2. Once all modules configured, then trigger module setup for each to initialise associated devices
3. Once all modules have completed setup, enter the main loop
4. During the main loop, periodically check to see if the queue of subscriptions can be resolved (i.e. names can be resolved) and register the subscriptions with the DroneLink messaging service.

Remote Management

As an extra layer of flexibility, some published parameters can be written to in order to change run-time behavior.  For example, to change the operating mode of the Nav mode from Idle to Active or to tune the PID values for a Servo controller.

DroneLink Message Structure

The full DroneLink message structure:

• < Source Node ID (8bit) > 
• < Target Node ID (8bit) > 
• < Module ID (8bit) > 
• < Packet Priority (2bit) > 
• < Parameter ID (6bit) > 
• < Parameter is Writable (1bit) > 
• < Payload Data Type (3bit) >
• < Payload Length (4bit) >
• < Payload (1-16 bytes) >

Packet priorities range from 0-3 (low, medium, high and critical), with 3 being the highest priority and reserved for mission-critical information like GPS location or navigation heading.  

Full list of data types:

• 0 = 1-byte unsigned integer
• 1 = Address (source node ID, target node ID, module ID, parameter ID)
• 2 = 4-byte unsigned integer
• 3 = Float (32bit)
• 4 = Character string
• 5 = Module name; stored as a character string
• 6 = Name query
• 7 = Value query

Discovery Process

To allow the management web interface to query devices on the network and enumerate the current configuration, all parameters support a remote query interface: 

• Name Queries: A name query is requested by sending a DroneLink message with data type 6 (empty payload) to the parameter of interest.  The module will then reply with a Module Name message (data type 5) containing the name it has been configured with.
• Value Queries: The current value of a parameter can be queried by sending a DroneLink message with data type 7 (empty payload) to the parameter of interest.  The module will reply with the current value using the regular data types (0-4) and the associated payload.

Nodes will slowly transmit all of their values (even if they haven't) changed to ensure a new server coming onto the network can get a full copy of the node's current state. The server can then fill in the missing module names by querying any missing info.  The discovery process can be disabled via a parameter in the Management module to save bandwidth if needed. 

Conclusion

The source code for the firmware is available on Github and can be compiled using Platform.io: 

https://github.com/Axford/DroneNode 

In a future article I will describe the other parts of the system in more detail (e.g. Mesh networking, Drone System).