Can Spotflow Solve Remote Debugging for Embedded Systems?

Anyone who has worked on a sufficiently complex embedded system has likely experienced some variation of the same problem: a device in the field stops behaving as expected, but there is very little information available to determine why.

In a simple system, debugging is usually straightforward. Connect a debugger, open a UART terminal, add a few print statements, and observe what happens. However, as systems scale, this approach quickly becomes impractical. Modern embedded products may contain dozens, hundreds, or even thousands of deployed devices spread across multiple locations. When one of those devices begins misbehaving, questions immediately arise:

• Which device failed?
• When did it fail?
• What was it doing at the time?
• Was the issue isolated to a single device or affecting an entire fleet?

Traditional debugging techniques such as UART logging, JTAG debugging, crash dumps, and custom telemetry can help answer these questions, but they often require significant engineering effort to deploy and maintain at scale.

Recently, I was introduced to Spotflow, an observability platform designed specifically for connected embedded devices. The goal of Spotflow is straightforward: provide visibility into deployed devices through centralized logging, telemetry collection, and device management capabilities.

At first glance, the concept is compelling. In the same way software engineers rely on observability platforms to monitor cloud applications and servers, Spotflow aims to bring similar capabilities to embedded systems.

In this series of blog posts, I will evaluate Spotflow from the perspective of an embedded engineer. I will walk through the actual setup process, explore the platform's capabilities, and document the challenges encountered along the way.

To begin, we will start with the official Spotflow Quick Start using The Zephyr Project on an NXP FRDM-MCXN947 development board.

Development Environment

For this demonstration, I used a Windows 11 environment.

Before building the Spotflow sample application, several development tools needed to be installed. Although the Spotflow Quick Start guide provides a streamlined setup process, it is useful to understand what each component contributes to the development environment.

Installing the Development Tools

The first step was installing the "MCUXpresso for VS Code" extension and associated NXP development tools. Using the MCUXpresso Installer, I selected the Zephyr Developer package.

This installation provided:

• Arm GNU Toolchain
• CMake (3.30.0)
• DeviceTree Compiler (1.6.1)
• Git (2.39.0)
• MCUXpresso for VS Code (26.5.49)
• Ninja (1.12.1)
• Visual Studio Code (latest)
• wget (1.21.4)
• Zephyr Dependencies (3.0.1)
• Zephyr Python Environment (4.4)
• Zephyr SDK (1.0.1)

These tools form the foundation of a Zephyr development environment.

The Arm GNU Toolchain provides the compiler used to generate executable firmware for the Cortex-M33 processor on the MCXN947. CMake and Ninja are used by Zephyr's build system to generate and execute build instructions. Git is required because Zephyr retrieves many of its components directly from source repositories.

The Zephyr SDK supplies architecture-specific tools and libraries used during compilation and linking, while the Zephyr Python Environment contains Python packages required by the Zephyr build system.

Creating the Spotflow Workspace

After installing the prerequisites, the Spotflow Quick Start guide provides a PowerShell script that automatically creates a West workspace:

Invoke-Expression "& { $(Invoke-RestMethod -Uri 'https://downloads.spotflow.io/spotflowup.ps1' -UseBasicParsing) } -zephyr -board frdm_mcxn947"

Rather than manually cloning repositories and configuring dependencies, this script creates a complete Zephyr workspace configured specifically for Spotflow. Upon success, you should see a screen that looks like this:

The generated workspace contained several important directories. Since I opted for the default name and location, for me, the folder is located in C:\spotflow-ws\. The Spotflow setup script automatically assembles these components into a ready-to-use workspace, eliminating much of the manual setup normally associated with Zephyr development.

This structure is standard for Zephyr-based projects. The workspace effectively acts as a self-contained development environment.

Initial Setup Challenges

Although the Spotflow Quick Start process is largely automated, I did encounter one significant issue during the initial setup.

At the time of writing, my system was running Python 3.14.5. Since this was the newest version available, I initially assumed it would be fully compatible with the Spotflow setup process. In practice, however, the environment behaved inconsistently. I encountered various dependency-related errors, including missing Python packages that should normally be provisioned automatically as part of the workspace setup.

However, after troubleshooting the environment, I ultimately replaced Python 3.14.5 with Python 3.12.10 and recreated the workspace.

Once I started using Python 3.12, the setup process behaved as expected. The workspace was created successfully, the required tooling was installed automatically, and the build process became significantly more predictable.

While I cannot definitively attribute every dependency issue encountered during setup to Python 3.14, switching to Python 3.12 resolved the environment problems and aligned with the versions commonly used by the Zephyr ecosystem.

The primary takeaway is simple: if the Spotflow Quick Start process behaves unexpectedly, verifying the installed Python version should be one of the first troubleshooting steps.

Building and Flashing the Sample Application

With the development environment configured, the next step was to build and deploy the Spotflow logging sample to the FRDM-MCXN947.

Required Software Configurations

For Spotflow to run successfully, the following configurations must be enabled in prj.conf in C:/spotflow-ws/modules/lib/spotflow/zephyr/samples/logs:

CONFIG_SPOTFLOW_DEVICE_ID="frdm-mcxn947-001"
CONFIG_SPOTFLOW_INGEST_KEY="<your_ingest_key>"

# Boards that support both Wi-Fi and Ethernet use Wi-Fi by default.
# If you want to use Ethernet instead, please use the following option:
CONFIG_SPOTFLOW_USE_ETH=y

# Enable logging to Spotflow
CONFIG_SPOTFLOW=y

For this evaluation, I configured the sample to use Ethernet by enabling CONFIG_SPOTFLOW_USE_ETH=y rather than the default of Wi-Fi.

The device ID uniquely identifies the board within Spotflow, while the ingest key authorizes the device to upload data to your Spotflow instance. Ingest keys can be generated through the Spotflow dashboard after creating an account.

To acquire it, first navigate to the screen labeled "Ingest Keys" on the menu:

You then have the option to either create an ingest key or copy the default. In this demonstration, I simply copied the default.

Building the Application

With the workspace prepared, the sample application can be built using:

west build --pristine --board frdm_mcxn947/mcxn947/cpu0

The output of a successful build looks like this:

After a successful build, Zephyr generates several output artifacts, including zephyr.elf and zephyr.bin. These files represent the executable firmware image that will be programmed onto the target board.

Flashing the Board

Programming the application onto the FRDM-MCXN947 is straightforward using Zephyr's built-in flashing support:

west flash

Once the flash operation completed, the board automatically rebooted and began executing the Spotflow sample application. A successful flash looks like this:

Verifying Application Execution

With the firmware successfully flashed onto the FRDM-MCXN947, the next step was to verify that the application was running as expected.

Monitoring UART Output

The FRDM-MCXN947 exposes a virtual serial port through its onboard MCU-Link debugger. On Windows, this appeared as MCU-Link VCom Port (COM11) in Device Manager.

Using PuTTY configured for COM11 at 115200 baud, I was able to monitor the application's UART output.

After opening the port, I was able to see the following output from Spotflow:

The output indicated that Zephyr had booted successfully and that the Spotflow logging backend had initialized correctly. At this point, the application appeared to be functioning normally.

The Device Never Appears in Spotflow

Although the UART output looked promising, one important detail quickly became apparent: the device never appeared in the Spotflow dashboard.

The sample application continued producing log messages:

[00:00:01.054,000] <inf> MAIN: main: Hello from Zephyr to Spotflow: 0
[00:00:03.054,000] <inf> MAIN: main: Hello from Zephyr to Spotflow: 1
[00:00:05.054,000] <inf> MAIN: main: Hello from Zephyr to Spotflow: 2
...

At this point, I expected the device to appear in the Spotflow dashboard. The UART output looked healthy, the application was running, and no obvious errors were being reported. However, the device never appeared online. Looking more closely at the UART output revealed the problem:

Link is down
Waiting for network...
Initializing Ethernet...

The Spotflow logging backend had initialized correctly, but the application was unable to establish a network connection, evident from the messages.

The Root Cause

Earlier in the setup process, the sample application had been configured for Ethernet using CONFIG_SPOTFLOW_USE_ETH=y in prj.conf.

This meant that before any communication with Spotflow could occur, the board first needed:

1. A physical Ethernet connection
2. An IP address via DHCP
3. DNS Resolution
4. A successful MQTT connection

Since the board was only connected to my laptop over USB, none of those requirements were being met. Put simply, the application was functioning correctly; it simply had no path to the Internet.

Connecting the Board to the Network

To provide network connectivity, I connected the FRDM-MCXN947 to a Google Nest Wi-Fi router using an Ethernet cable. Immediately after rebooting the board, the UART output changed significantly, as shown below.

The following messages began appearing:

Link speed 100 Mb, full duplex
Link is up
Interface is up -> starting DHCPv4
Network connectivity established and IP address assigned
TLS init
spotflow_mqtt_thread_entry: Spotflow registered TLS credentials
Resolving DNS
DNS resolved for mqtt.spotflow.io:8883
Using Spotflow device ID: frdm-mcxn947-001
Using secure transport
...
Using secure
MQTT client connected!
MQTT connected!

These messages confirmed that the board had successfully joined the network and established communication with Spotflow's cloud infrastructure.

Confirming Successful Device Registration

Shortly after the MQTT connection was established, the device appeared inside the Spotflow dashboard. The dashboard itself provides key information about the operating status of the device and the logs it is sending.

The board was now successfully transmitting data to Spotflow and could be monitored remotely through the dashboard. Hence, I was able to view the status of the board. In summary, however, for Spotflow to run, it needs a Wi-Fi or Ethernet connection to connect to the Spotflow servers to acquire the full benefits of the service, pictured below:

Conclusion

After resolving a Python version compatibility issue and providing Ethernet connectivity, the Spotflow Quick Start worked as advertised. The FRDM-MCXN947 successfully connected to Spotflow and began transmitting logs to the cloud dashboard.

With the device now successfully connected to Spotflow, we have verified the complete communication path: FRDM-MCXN947 → Ethernet → DHCP → DNS → MQTT/TLS → Spotflow Cloud

More importantly, this exercise verified the complete communication path between the device and Spotflow's infrastructure, providing a foundation for evaluating Spotflow in real embedded applications.

In the next article, we will move beyond the sample application and explore how Spotflow can be integrated into a custom Zephyr project to collect meaningful runtime data and telemetry.

Sign in