Chances are we can all agree that Embedded (Software) Engineers in general spend a significant amount of their time on debugging.

The goal with this discussion thread is to provide a place for you to share with others your favorite tools, strategies, tricks, best practices, personal stories, etc. Basically, anything related to #debugging embedded systems that you believe could be of potential benefit to others in the Embedded Engineers community and that maybe you have learned the hard way. This thread will be part of the new #FAQ section.

As always, thanks a lot for your time!

1) When ever possible, do NOT debug on embedded target.

Algorithmic code is best developed and debugged on a host,
then ported to the embedded target with embedded test cases.
For each C++ class I usually add a QwikTest() member that verifies
class functionality, built only in debug builds, and
run from a test command or even at startup.

2) Plan what you must, and how you are going to debug on target.

A few things best planned in advance on your PCB:

• PCB test pads or even headers, especially important with today's tiny component sizes, for:
  • power supplies
  • any SPI or I2C bus
  • any important signals you'll need to access
• I like to glue a header strip to the edge of the board for an easy connection to the scope/logic analyzer, and solder wires from the test pads to the header.
• LEDs for diagnostics. In case you need to blink SOS or a numeric status code, or otherwise make internal state visible.
• Test pad tied to GPIO for triggering scope or measuring duration.
• GPIO tied to an RC to a test pad. Use this for measuring percent of time spent in a critical routine (for example ISR); directly readable with a voltmeter.

And planned in advance for your software:

• output diagnostic stream, for example:
  • Simple UART,
  • Diagnostic messages encapsulated in your host communication protocol to a dedicated diagnostic console on the host
  • At minimum, most development IDEs support some form of output console (semi-hosting).
• Built in test data, for example test commands to simulate sensor inputs (and disable sensor reading). This allows you easily reproducible test cases, and testing when you don't have sensors and/or can't easily control sensor inputs during testing (for example using an evaluation board before you have final hardware).
• Time testing:
  • How are you going to check ISR latency and runtime?
  • other processing performance?
• Stack and other memory usage: How are you going to check this? I often include a diagnostic command to check and report stack usage for all tasks plus free storage status (especially given bugs in tools that purport to check this, grrr...)

3) Code Defensively

While this isn't specific to embedded, it is even more important because debugging is harder than on the host. For example, use plenty of assertions! You'll save a lot of time when you catch a logic error before it propagates.

4) Verify from the bottom upwards

Before trying to verify other functionality, check all your timing!
Some examples of where you can avoid driving yourself crazy with things that work intermittently:

• SPI and I2C speeds within permitted range for each peripheral?
• control pulses and timing likewise within range (did you wait long enough after the conversion trigger prior interrogating your ADC's results via SPI)?

You will have written a test plan for all of the above while awaiting your PCBs. During testing, I like to document the results of each test, for example here's the screen-shot showing SPI correctly clocked at 15MHz with proper phase relation to other signals:

5) Oh, and decent tools...

Poor tools can waste a lot of your time. Some of my favorite tools:

• Segger J-Link for ARM SWD debug interface. In past I have wasted way too much time with dysfunctional OCD and horrid buggy evaluation board built-in debuggers (I'm looking at you Freescale). Save yourself the aggravation and time! A decent SWD/JTAG adapter will download faster and work more reliably.
• A decent scope/logic analyzer is really important. I use a Keysight Technologies’ MSOX3054T with serial decode options to easily see SPI, RS-232, CAN, etc.
• Grid-Connect CAN-USB analyzer for CAN bus monitoring and stimulus

Hope you find this helpful!
Best Regards, Dave

Hello Dave,
Could you please elaborate on the first point?
Also what is QwikTest()?

Thank you!

Could you please elaborate on the first point?
Also what is QwikTest()?

Sure: In test-driven development, tests are coded as part of classes, including test data and assert if result is not as expected. I typically code this as a member function called QwikTest() to run the tests (defined/built only in debug builds). Start by getting a class to work on the host, and then move the class to the target with at minimum a smoke test verifying basic functionality...

Hope that's clear!
Best Regards, Dave

That did clear some doubts.

Thanks Dave

This really is where good embedded programming skills come to bear, bringing up a new board or debugging an existing one.  My first and best tool for the process pf debugging an embedded system is to read and understand the data sheets for the parts on the board. After confirming that the controller is functional, using a in-circuit programmer/debugger device, I start writing low-level functions, making sure that may understanding of the device is correct.  I like to get the interrupts functional first, verifying that the critical processes are functioning correctly (ADC/Timers/UARTs, etc.).  Once the data is being collected correctly, I like to work on the functionality or algorithmic process that using the data and drives the outputs.

As far as tools go, here are my favorites:

• Volt meter - I like to start with verifying the power sections of the design.  These need to be right or everything else is going to be effected.
• Oscilloscope - This is my go to piece of equipment.  This is where you really get to see if things are working.  With most embedded devices all of the interesting things are happening inside the part, if there are little to no external devices, you will need some help to see anything going on here, so I use unused pins to toggle on internal events so I have something to look at while trying to monitor the functions of the device.

As an example of this technique, here is an example of measure processor loading during interrupt processing:

The lower (cyan) trace is connected to an LED driver that I re-assigned for this measurement.  The signal is driven low upon entering the timer interrupt and returned high upon exiting.  The 'low' time represents the processing time of the interrupt, while the 'high' time represents the time available to the background task.  The upper (yellow) trace is a signal generated by the interrupt processing.  This signal is a channel select line that addresses one of the four MUX channels.  This signal is switched to the next channel every fourth interrupt. Using this technique, I was able to optimize each phase and balance the load between the different phases of the interrupt processes.

• In-circuit programmer/debugger - Being able to change and load code is fundamental to the the debugging process.  Having a good tool to modify code, read/modify EEPROM and check registers is very important.  Also a good debugger is helpful to set break-points, allowing you to trace the functioning of the code is helpful.
• Logic analyzer - Tracing multiple digital waveform is easy with a logic analyzer.  Very useful for viewing communications to/from the microprocessor to external devices. Complex triggering allow you to zoom in and find the information you need when there is a lot of data present.
• Signal generator and extra power supplies - With embedded systems that have external sensors and devices, being able to simulate the signals in a controlled manner is a good way to verify functionality and to test limits of the sensor ranges.
• Soldering tools and microscope - When things don't go right and problems are found, being able to fix them is important.  I find that being able to repair/replace/modify the board is very helpful.  

As far as best practices, I like to always try to reuse as much code as possible.  When designing a new board/product, I like to attempt to use the same microprocessor as in prior designs.  By being able to leverage an existing project as a starting place on a new project, I find that I can hit the road running. After doing a little (or in some cases a lot) of reworking the code to match the current project's desired functionality, I can truly focus on the changes and new code and assume that the untouched code if mostly functional.

I hope that this helps.

Favorite tools:

- Ethernet (TCP/IP) analyser - wireshark
- USB analyser - Ellisys and TotalPhase
- CAN TotalPhase Komodo
- VisualStudio for editing, developing, debugging
- Saleae oscilloscope/logic analyser
- J-Link and P&E Micro BDM/J-Tag/SWD debuggers for HW debugging
- IAR for debugging (it crashes occasionally but is still better, faster, more efficient and powerful that the clunky debugger uses in Eclipse based tools)
- Com0Com virtual com port loop back
- Git server for code repositories
- masses of development/evaluation boards!

Strategy used and refined since 2004 is to code, develop, test, debug and analyse complete development to 98% in simlulation environment. Test on HW for final proof and low level debugging/optimisation. When using new peripherals I "always" first create a simulation of them because this ensures deep understanding, allows the operation to be tested (in overall project) based on this understanding and later on to refined based on deviations observed on HW or less obvious effects.

Here are a couple of videos showing the use of simulation in VisualStudio and some usage cases (analysis/debugging):

HW debugging is restricted to very low level code whereby I always debug with fully optimised code (as will be delivered) making much use of the disassembly/memory/register views because this is where the action is. Higher level code debugging is less interesting on the HW since VisualStudio debugging capabilities are so much more powerful and this has generally been completed (including code coverage testing) before touching the HW.

Using this strategy I have developed many products without needing HW or only limited remote debugging on customer HW where needed.

Since I work as a consultant and trainer where I often help solving problems (eg. memory related faults, DMA errors) very low level debugging experience comes in handy. I am often surprised at the lack of knowledge of many engineers I help when it comes to the use of debuggers and debug strategies. I have the impression that this is an area that is often lacking in training and on-going development and causes inefficiency, especially when a more complicated case needs to be solved.

I am a real fan of the evaluation kits that most micro-controller manufacturers, or third party vendors, design.  They are usually less than $100 and it is great for starting your code before you get your prototype PCBs.  There are plenty of code examples, test points, and on-board peripherals, built in to the evaluation boards,  to give you confidence in the design and that the code will at least start up when you move on to the target system.

When I am bringing an embedded prototype up for the first time, I always try to get the RS-232 going first.  That is my favorite means of debugging, whether it is an on board peripheral or a complex algorithm, there is no substitute for running your code on the actual platform it is designed to run on.  In the past, I got code running on a simulator and then found that it didn't run the same when I ported it to my platform.  A few creative hooks with serial output has pulled my bacon from the fire more times than I can count.

Other than the trusty DVM, the o'scope is my only other tool for when I need to verify timing or waveforms.  This might sound basic, but I have found that simplicity + creativity = success.  By keeping it simple, you are forced to understand the nuts and bolts of everything in your design.  The best firmware designer also has an intimate understanding of the hardware, down to the register level, of every component in the design.  It is never a waste of time to spend hours going through data sheets before the schematic is even created.  It is the overlooked small details that will either kill the design or cause a lot of ugly backwires to fix an oversight. While windows programmers can afford to write code that is more abstract and layers above the driver level, embedded systems are all about the low level drivers and maximum efficiency, because embedded systems are real-time systems and in my case, I deal in hard-real-time systems.

I have worked with logic analyzers, monitors, simulators, evaluation kits and trial and error, but I believe the only way to do a thorough job of debugging is an emulator and their offspring, debuggers. There is no better way of knowing what is going on inside the the micro or stopping, monitoring and adjusting operation.

However there are some other techniques to help when you can't quite get the debugger handle on the problem. Bring out as many spare I/O pins as you can and even possibly have a few DIL switches or jumpers. As outputs these pins will allow you to measure execution times, whether parts of the code gets executed and when certain events happen and in what sequence. AS inputs they can help create conditions that may be difficult to achieve- for instance if you have an event that happens every 7 days, you can use a switch to accelerate the time to prove the code.

A special case of the additional outputs is the blinky LED.

I also debug a module at a time, as I write a code module, and once the module is tested, incrementally then test the system as a whole as built up to that point.

Debugging of embedded depend on a project, MCU used and on personal preferences of course..
Personally, last years, I am working mostly with Cortex-M/A ARM cores under Gcc/Eclipse environment, but in some cases I need to debug/develop some apps for small 8/16bits MCU (8051; MSP430; AVR; STM8). 

For Cortex-M/A ARM I am using GCC+Eclipse+OpenOcd+GDB. 

For small MCU - always proprietary versions of IDE - either Iar or Keil. 

A year or two ago I've discovered for myself a Segger IDE (was playing with nRF52xx using a trial version), and I'd like to say, that from proprietary IDE this one is one of the best I've seen on the market (can compare only to Keil/Iar/AtmelStudio/CoIDE).

When thinking about debugging, for me the most important is to understand exactly what is going on at any given time and the worst nightmare is a multi-threading application, where you don't know who is crashing a memory. You can see it as just a sudden crash of application or as one thread is not responding or a HardFault or similar... 

Back to a topic.

Universal debug technique, which I think, will last forever, is:

- breakpoints in the code;

- GPIO (use oscilloscope) - this is the fastest and closest to HW method, can be used for profiling;

- error handlers to catch critical/not critical errors;

Then, more sophisticated debug techniques - depend on MCU core:

- unbuffered or, better, buffered I/O (say, over uart);

- semihosting on Cortex-M based cores, but remember, semihosting is very, very slow;

- special utilityes to help debug a particular scenario (say RTOS);

Below are my personal preferences, if one will ask what is the best:

- use proprietary IDE if possible, as they have the best integration/debug capabilities to the MCU's. They have the best hardware debug tools integration. This is on your personal taste, say I hate Keil - just because they use thick frames in their GUI...

- OpenOcd + GDB: I am using this with Eclipse for many years, but in comparison to proprietary IDE, as embedded developer, I do not like it too much (actually this is because you need to know more to use this in right way). At least I hope, I will never need to use a command-line GDB debugging.

- For embedded Linux development I don't know other options except of GDB usage... Why not.

- My main embedded RTOS at the moment is FreeRtos, I've found the combination of Tracealyzer + JLink (with SWO) is the easiest for me and saves a lot of time when need to profile a project. Say for other RTOS development it was not such easy to see interaction between threads etc. Here you need a bit of free memory to use this tools.

Debugging in Embedded systems is one of the most important skills an embedded engineer should possess because most of the time you will be debugging code which is written by some else. There are many ways in which you can debug your code like through UART prints or memory dump. For programs which are on the bootloader and device driver side the best debugging tool is the JTAG and DSO or oscilloscope. In my company we used to debug the bootloader code with the help of a JTAG debugger like Lauterbach. With the help of a JTAG debugger you can see all the register values and you can step debug to see every instruction being executed one by one. IDE’s is also a very good tool to debug your program. In some cases, you can also use the LED’s in the development board to debug your software and check at which instruction your software crashed. The most common debugging techniques used in our company were JTAG and UART based debugging.

I believe this video by Jack Ganssle could be of value to some readers of this thread: