Debug, visualize and test embedded C/C++ through instrumentation

• Software Development
• Test Equipment
• Debugging

A Short Introduction

Classical embedded debugging requires a debug probe that can access the internal memory bus and registers through a debug port, such as JTAG or SWD. For several reasons, the classical approach can sometime be painful. You may be locked to a vendor specific debug probe, or the probe may be costly, the software stack to use can be complex or awkward to use. Additionally, not all microcontroller supports non-intrusive memory accesses through the debug port, meaning that the CPU must be halted to access it, which is not always something we can afford; let’s consider a flight controller for example.

Another option exists: instrumenting our firmware to give it some debug capabilities. This method has some limitations but will also offers a series of advantages that can make it quite interesting. In this post, we will explore what instrumentation debugging means and how it can give you high level of debug and testing capabilities with a minimal software effort.

The general idea is to use an existing peripheral to access the memory instead of the debug fabric of your MCU. This require the firmware to pipe the data from that peripheral to a debug library that must be executed alongside your main application.

A software using this approach is sometime referred to as a "Runtime Debugger", a "Data Visualizer" or a "Calibration tool". The latter is a type of software that can read and write specific parameters in a device, generally identified by a numerical ID. A Runtime Debugger can do the same with any variables, making it a calibration tool on steroids.

Instrumentation based debugging is quite easy to enable and can work through the already available transceivers on your hardware, being a serial port, CAN bus or even some wireless transceiver.

Being able to monitor or edit in real time the internal variable of a software and getting samples that are synchronized with the firmware can replace most telemetry or configuration tools often designed in-house.

How To Proceed

First, we need a framework. The framework presented here is Scrutiny Debugger. It’s a free and open-source framework that I have written as a personal initiative. It comes in different parts

• An embedded library to be added to the MCU firmware
• A server that communicates with the firmware
• Clients that send read/write requests (and more) to the server

The first step will be to instrument the firmware. All we need to do is to invoke our debug library inside our main task, and optionally in the interesting scheduler tasks. Adding a hook in a scheduler task can let us access the memory from that task, effectively synchronizing our readings with it. This is very useful when working with some sort of control algorithm.

The job of the integrator here is to write the “instrumentation_process()” function. To enable Scrutiny more specifically, one must pipe the data from a transceiver to the library, invoke the library, then pipe back the data from the library to the transceiver.

Below is an example using a virtual serial bus using a CDC driver on a STM32F4 microcontroller

Let’s mention that this example is written in C. The Scrutiny embedded library is written in C++, but also offers a C Wrapper for C integration such as this one.

Once we have that enabled, all we need is to link against our instrumentation library and use a software that can speak its language. Scrutiny requires an additional post-build step that will permits some goodies, such as:

• Having the memory layout in a separate file from the .elf
• Having virtual variables (aliases)
• Inject a unique hash in the firmware to ensure the proper debug symbols are used.

The final build toolchain should look as follow.

We won’t cover the details of this integration here, it’s just a matter of invoking the correct command-line tools or invoke the proper CMake function if CMake is used. The documentation provides all the required details.

Scrutiny In Action

A quick look at the UI looks like this.

It’s a dashboard with different type of widgets:

1. A Watch widget: Gives a textual value in real time
2. A Continuous Graph widget to plot values received from the server
3. An Embedded Graph widget to configure a trigger-based datalog inside the firmware.

Widgets #1 and #2 gets their values from the continuous stream of values coming out of the server. They get real-time values that are updated at a rate throttled by the communication link we use to reach the device.

Widget #3 (Embedded graphs) is different. It requires the embedded library to start monitoring a series of value and wait for a trigger condition, like a scope, to happen. This is where having a hook in a scheduler task becomes useful as we can now make an acquisition at any rate enabled in the firmware, even high-speed control tasks. This feature is the best tool to find defects in a firmware.

Also, a very natural way of representing the variables in a C++ firmware is in a tree where each level represents a nesting level in the firmware. A file, a function, namespaces or classes are all logical grouping units that Scrutiny will use to organize the variables in a treelike structure.

Let’s suppose we have a file named main.cpp in which we have the code in the leftmost snippet. Scrutiny will display its content in the GUI using a tree as depicted in the rightmost image.

Each variable (or alias) has a file system-like path and they can be accessed with a Python SDK. This SDK enables Hardware-In-the-Loop (HIL) testing quite powerfully. Everything doable with the GUI can be done in a Python script – including getting triggered graphs.

Using the SDK, a Python script could reach a value as depicted below. The value of “var.value” is updated by a background thread and can also be written synchronously.

from scrutiny.sdk.client import Scrutinyclient
client=ScrutinyClient()
with client.connect('localhost', 8765):
    var = client.watch('/var/static/main.cpp/MyNamesoace/some_function/the_struct/member1')
    var.value=1234 # Synchronous write. Blocks until completion.
    while True:
        print(var.value)

Writing a variable synchronously means that the python script will block on a write operation and move to the next statement once the write has been confirmed by the device and acknowledged to the client. A synchronous script like this can be seen as a remote thread running in the embedded device, with slow memory accesses. 

Few Tricks

To make things a little more concrete, let me show you some of the most common tricks you'd want to do if you decide to enable your firmware with such framework

Exposing The IOs

When doing manual or HIL testing, one of the first things we need to do is mocking the inputs and outputs. A simple way of achieving this with Scrutiny would be to abstract the IO accesses behind a method that can alternate between a real IO or a mocked one.

// code is oversimplified to demonstrate a point.
struct MockedDigitalInput {
    bool use_mock;
    bool mocked_value;
};
volatile MockedDigitalInputs mocked_inputs[16]; //Assumes we have 16 channels
bool read_digital_input(int channel){
    if (mocked_inputs[channel].use_mock){
        return mocked_inputs[channel].mocked_value;
    }
    return HAL_READ_DIGITAL_INPUT(channel);
}

Exposing Registers

Another useful technique is to expose the register. For Scrutiny to know the existence of a readable/writable element, an entry in the debug symbols must exist. Those exist for variables that are defined by the compiler. Therefore, to expose the registers, we need to declare a variable of the right type and make sure it is located at the correct location.

//C++ source file
#include <cstdint>
union SomeRegister {
    uint32_t u32;
    struct {
        uint32_t some_value:13;
        uint32_t another_value:5;
        uint32_t flag1: 1;
    } bits;
};
volatile SomeRegister the_register __attribute__((section(".bss.register_section")));
// -----
// Linker Script
.registers 0x08000000 {
    *(.bss.register_section)
}

As simple as that. Scrutiny will now show /global/the_register/bits/some_value (and others), as a readable and writable unsigned integer.

Registers can also be exposed without involving the linker script by using a pointer, Scrutiny can dereference them — but this is a bit more prone to errors and memory must be allocated to the storage of the pointer.

Synchronizing a graph with a scope (or other hardware)

Scrutiny has the ability to capture a graph triggered on a software condition. It is possible to synchronize that graph with a hardware component simply by attaching a callback that toggles a pin to the embedded library configuration.

//main.cpp
#include <cstdint>
#include "scrutiny.hpp"
uint8_t scrutiny_rx_buffer[64];
uint8_t scrutiny_tx_buffer[128];
void my_custom_trigger_callback(){
    HAL_TOGGLE_IO(DEBUG_PIN); // Perfect to hook a scope!
}
void main(void){
    scrutiny::Config config;
    config.set_buffers(
        scrutiny_rx_buffer, sizeof(scrutiny_rx_buffer),     // Receive
        scrutiny_tx_buffer, sizeof(scrutiny_tx_buffer)      // Transmit 
        );
    // callback is tied here
    config.set_datalogging_trigger_callback(my_custom_trigger_callback);
    scrutiny::MainHandler scrutiny_main;
    scrutiny_main.init(&config);
    // ...
}

Piggy back the Scrutiny protocol

We already have a working communication between a host machine and an embedded firmware, why not leverage this to pass some custom data? It's possible since the embedded library has a protocol command called "User Command". This does nothing than passing the command data to a custom callback.

void my_user_command(
    uint8_t const subfunction,
    uint8_t const *request_data,
    uint16_t const request_data_length,
    uint8_t *response_data,
    uint16_t *response_data_length,
    uint16_t const response_max_data_length)
{
     printf("Received user command with subfunction %d and %d data bytes\n", (int) subfuction, (int)request_data_length); 
     if (response_max_data_length >= 13){  // Prevent overflows    
        *response_data_length=13;    
        strcpy(response_data, "Hello World!");    
     }
}
int main(){
    scrutiny::MainHandler scrutiny_handler;
    scrutiny::Config config;
    // ....
    config.set_user_command_callback(my_user_command);
    scrutiny_handler.init(&config);
    // ....
}

And this can be triggered from python like this

from scrutiny.sdk.client import ScrutinyClient
def main():
    client = ScrutinyClient()
    with client.connect('localhost', 8765):
        print(client.user_command(123, bytes([1,2,3,4,5])))

This will get the embedded application to print 

Received user command with subfunction 123 and 5 data bytes

and python to print 

Hello World!

Wrap up

This blog post was a concentrated introduction to Scrutiny. I tried to keep it as short and dense in information as possible. There are many subtleties and features that were not presented here. To know more, here are some resources:

• Scrutiny website
• GitHub repos (including a demo repo)
• A more detailed talk about the subject:

The main idea that you should remember from this article is that instrumentation-based debugging is easy to enable and quite powerful. If you happen to debug something with printf instead of a debug probe, you should take few hours to integrate Scrutiny instead and have much more debug power for no added cost and low efforts.

Sign in