• RTOS
• Software Development
• Linux

Contents

• Introduction
• The Mistake
  • Causes
• The Strategy
• Marking Embedded Linux
  • Rootfs Overlay
  • Files To Mark
  • Marker Lines
  • Script
    • C/C++ Source Modifications
    • Linux Modifications
    • mark-build-myboard-linux.sh
  • Using The Script
    • Sourcing The Script
    • Setting Marks
    • Finding Marks
    • Running On The Target
    • Clearing Marks
• Hacking

We have many ways of tripping ourselves up during embedded system development, testing, deployment, support, and maintenance.

This is the first in a random series of posts on dumb mistakes I've made with embedded systems, and some strategies for dealing with them. I hope that by learning from my mistakes, you can avoid them.

That may save you minutes, hours, or days of wasted time and frustration, and keep you from banging your head against the wall.

The mistake here is to modify, build, load, and run some code, thinking that the whole process has succeeded and my modification is actually running on the target embedded device, when in fact something failed along the way, I didn't notice, and the device is unchanged. Then I start my testing or debugging or whatever, and it doesn't work the way I expect it.

There are variations, such as when someone else gives me their build to run, or I download a build from a build server or other system. Passing builds between people is common in support and debug situations; sometimes I send my builds to vendor customer support so they can try to recreate issues.

With all these builds flying around, it's easy to make a mistake.

Sometimes it's obvious that something isn't right. That's the best case scenario. But sometimes I may need to run it for a while, or go through some involved setup of other systems as part of it, and don't realize until I've wasted all that time and effort.

What are some of the causes of this mistake? Sometimes it's just simple inattention. In the rush to get things done, I missed a step or overlooked an error message.

The typical developer cycle is edit-build-load-run. Any of those steps can fail.

• During code modification, I pulled the wrong source from source control. Or I forgot to save the source file in my editor (yes, I've done that! DOHHH!).
• During building, I missed one of the steps of the multistep manual process. Or I didn't reconfigure the build properly. Or I didn't notice a failure message, and because I didn't start from a clean build (so I could save some time!), I still had enough build outputs lying around that the rest of the build ran ok, or at least appeared to.
• During loading (i.e. flashing the device, or whatever procedure deploys the code to it), I selected the wrong build image. Or I didn't stage it to the correct location that would cause it to be loaded automatically. Or I didn't notice the flash write failure.
• If the device allows multiple runnable copies (such as keeping a last-known-good backup copy, or alternate runtime directories), I ran the wrong image.

Sometimes these things can combine insidiously. For instance, I don't notice a flash write failure during loading, corrupting the main runnable image, but then when I run the device, the last-known-good copy kicks in and runs. So I think I'm running my code.

If I'm trying to run code that someone else has given me, they might have made one of those mistakes and not realized it.

Some of these are symptoms of poor build process. Manual build procedures are a real red flag. Anything requiring more than one step is an opportunity to make a mistake. Even a simple CMake, make combination can go wrong if I get interrupted while doing it.

Hence my constant advice to automate everything that can be automated. Automate, automate, automate!

Short-cutting a build process in order to save time is also a common cause. Here I might be using a good, well-automated procedure, but instead of cleaning all outputs and doing a full build, I do some type of partial or incremental build. I just want to change this one module, why should I take the time to rebuild everything from scratch?

That really can be a timesaver. Until it goes wrong. Maybe the automated procedure doesn't handle those cases well. Or maybe I misunderstood the process, and didn't do the partial build properly.

Some of these mistakes have caused me to develop obsessive-compulsive habits over the years:

• After editing a source file, I do a diff on it to verify the changes are there, and I didn't somehow mess them up during the edit and save (especially if I'm using an unfamiliar editor).
• After building, I check source, object, library, and image file timestamps and sizes. I compare MD5 sums of outputs to ensure the contents are actually different. I use the strings and grep commands to look for strings in them that I expect to have changed.
• After loading the build, I might be able to do similar things on the target device.

I want anything that can quickly give me evidence that I've accomplished the intended change, cross-correlated as much as possible.

The Strategy

The strategy I use for dealing with this mistake is to mark the build, that is, to "carve my name" into it. I want to mark it at critical points with information that identifies it as a custom build, built by whom, using what branch (or other source control identifier), at what date and time.

I like the marker points to show up quickly on system initialization, and at any distinct points which could be affected by customizations. The points are system-specific, such as on Linux or RTOS boot console output.

I call that "proof of life," proving that I've successfully modified the running system to include my changes.

My general strategy is to look for early startup log lines, grep through the source code to identify the modules and functions that logged them, then go modify those functions to add my own marker log lines.

I've been doing some form of this manually for years, but I decided it was time to write a script to automate the file modifications for embedded Linux. Then it's easy, fast, and repeatable, no matter how many files I need to mark. The script serves as a pattern that can be customized to different specific builds.

I particularly look for stable modules (i.e. modules that I'm otherwise not going to modify as part of development). That way I reserve any changes to those modules purely for marking purposes. That simplifies the part of the script that handles clearing marks: just revert those files.

This also falls in line with my general strategy for learning a new codebase or system, where I correlate log output with source modules and functions. That's very instructive to see how things actually fit together at runtime. Logging that includes source file references makes it easier, but a little creative grepping is usually sufficient either way.

The remainder of this post uses embedded Linux, built in the Ubuntu 18.04 development environment, as an example. However, all the concepts can be adapted to other embedded systems that log output, such as various RTOS's or bare metal, and other development environments, such as Windows.

What if a system doesn't have logging? That requires more creative marking and gets very system-specific. What about distinctive LED blink patterns or sounds, or GPIO output sequences you can capture with a logic analyzer (I've done that)? Do you have some way of examining memory contents? Can you look for strings embedded in the code with a debugger?

Marking provides a way of embedding evidence in the system. Then you retrieve that evidence by whatever means makes sense on that system.

An embedded Linux build that you customize with buildroot and that uses busybox has five distinct parts:

• Kernel: the linux kernel image that boots at power-up.
• Rootfs: the root filesystem that contains the standard filesystem hierarchy
• Busybox: the busybox shell, that contains the command shell and a number of built-in command line utilities.
• Rootfs overlay: your custom overlays to tailor the rootfs.
• Application: your application-specific code, both kernelspace and userspace.

Each of these has components that may log to the console on boot. The actual components that go into a specific build are customized with buildroot menuconfig and busybox menuconfig.

It's worth marking each part separately, even if you don't customize them, so that no matter what your build or loading process (including flashing different parts to different storage partitions), you can clearly identify each one as being part of the same build.

That helps identify strange situations such as a device ending up flashed with the new kernel, but the old rootfs. That's happened to me when the loading process encountered an error while preparing to program the rootfs partition. Since userspace applications go into rootfs, that means my application code didn't get updated.

The following shows early log output from the different parts, sanitized from a real system:

Kernel:  [    0.000000] Linux version 5.4.45 (sbranam@builder) (gcc version 8.2.1 20180802 (Linaro GCC 8.2-2018.08~dev)) #1 SMP PREEMPT Tue Feb 9 01:14:23 UTC 2021
Overlay: Setup eth0...
Rootfs:  [    1.479273] pps_core: LinuxPPS API ver. 1 registered

In this case, Busybox didn't log anything on its own, and there were no application loadable kernel modules (LKM's) or userspace daemons started up.

These are the files producing those outputs. Some of the details may vary with kernel version as well, so you'll need to refer to the specific version you're using. My links go to current master.

The overlay part is different from the others, because it's based on init.d scripts. There are different variants of init.d processing, with varying degrees of complexity.

On this particular system, it's simply a matter of placing bash scripts in the /etc/init.d directory, and the system executes all scripts whose names start with S in lexical order on startup. In this example, the S30eth0 script would be executed after S20something and before S37something.

The overlay files are local files that get combined by the build with a standard skeleton rootfs, so the final set of ordered scripts is a combination of the two, including override replacement of files in the skeleton.

Busybox is the thing that actually executes the init.d scripts on the target, because it has the bash shell.

How did I choose those particular files? As I mentioned, I want something that logs early in the system, and is stable code. These are the choices I made for the particular Linux system I'm using. The same thought process applies to any other kind of system.

For the kernel, init/main.c is a perfect candidate. It's the very first thing to log to the console, and I'm not going to touch it as part of my work. It's built into the kernel vmlinux image.

For rootfs, a suitable file is a bit harder to find. Some of the things that look like they might be suitable are actually built into the kernel. How do you know? Search the vmlinux file in your build output for their log strings using this command pipeline:

$ strings <pathToOutput>/vmlinux | grep <logString>

If you find them, that means they're built into the kernel.

I want something that's part of the standard Linux distribution, that's loaded from the rootfs during boot, not built into the kernel. So some kind of driver, an LKM (identifiable as a file with a .ko suffix). The PPS driver is a nice generic stable module in the Linux distro.

If you have your own custom LKM's, a stable one of those is a good candidate. Then you know your kernelspace application parts are getting in.

For busybox, the way it's distributed and built makes it more involved to customize. I decided for simplicity I didn't need to bother, since I have enough other evidence to show me it's my build. So I didn't investigate how to do it.

For the rootfs overlay, the simplest thing is to add a dummy init.d script that does nothing but log a marker, ordered early in the init sequence. Then clearing the mark is accomplished simply by deleting the file. Other init.d systems might require more involved modifications.

For userspace applications, if you have a daemon that's stable and gets started automatically, that would be a good candidate. Or if your application set includes a stable command-line program, you could modify that. That's what I chose to do here, modifying its usage output to add a marker line.

I want to add marker lines in this form, so that they get logged at the same time as the above log output:

[CUSTOMBUILD] part nickname branchname timestamp

Where part is the system part, nickname is some string that identifies the developer, branchname is some string that identifies the git branch (or some other source control identifier), and timestamp is the date and time at which the component was built (or marked, in the case of an init.d script).

The [CUSTOMBUILD] token provides a search target that I can grep in log files or search for in terminal output.

For example, say I have a project managed by Jira tickets with project name EMB, and I'm working on a branch to blink the blue LED, I want something like this:

There are of course many ways to automate this. You could use a Python script instead of bash. I find bash sufficient for things like this. I keep the script very simple and modular for easy adaptability, nothing fancy. Your scripting style may vary.

This particular script assumes that the Linux distro and the application code have all been placed in the same git repo, so clearing Linux and application marks is simply a single command to revert a list of files. Your setup may vary. Note that not all builds may have a rootfs overlay.

The script defines a set of functions that are very easy to use, as shown by mark-build-help. I can set the marks, clear them, reset them (updates timestamps in files that have a hard-coded timestamp), and find all the files that contain a mark in some form, whether text or binary.

It uses sed to modify files to add marker lines. Sed is very powerful. This is very basic usage to do in-place editing (the -i option) to search for target strings and then either append the lines after the target line or insert them ahead of the target.

For C/C++ source code modifications, the script takes advantage of compile-time string concatenation. This combines all adjacent string tokens into a single string.

That means that tokens that are preprocessor strings actually get evaluated and placed directly into the string at compile time.

As a result, the actual values of timestamp macros are stored as part of the strings in object, library, and executable files. Then searching for marker strings shows the whole marker, including the timestamp.

The exact source code lines that the script writes are very system dependent. Even within a system, different parts may need different styles of lines. I also like to include a comment warning not to merge the changes into the repo.

For general userspace C/C++ source, the script uses printf. It uses the standard predefined macros __DATE__ and __TIME__ to capture the build timestamp. The lines look like this:

// DO NOT MERGE, THIS FILE IS MODIFIED FOR MARKED BUILD.
printf( "[CUSTOMBUILD] application sbranam EMB-1066/blink-blue-led " __DATE__" " __TIME__"\n");

For kernel source, the script uses pr_notice, which is an alias for printk. However, it can't use __DATE__ and __TIME__, because they cause this error: error: macro "__DATE__" might prevent reproducible builds [-Werror=date-time].

Instead, the build generates a header file generated/compile.h that defines macro UTS_VERSION as the build timestamp, so the script includes the header and uses that timestamp. The lines look like this:

// DO NOT MERGE, THIS FILE IS MODIFIED FOR MARKED BUILD.
#include <generated/compile.h>
pr_notice( "[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led " UTS_VERSION"\n");

For creating shell scripts, the script uses echo, and hard-codes the date and time into the marker line. It also adds a shebang line. The lines look like this:

#!/bin/sh
# DO NOT MERGE, THIS FILE IS MODIFIED FOR MARKED BUILD.
echo "[CUSTOMBUILD] overlay sbranam EMB-1066/blink-blue-led" Mon Feb  8 21:19:45 EST 2021

Here's the script, mark-build-myboard-linux.sh. This is specific to Linux on my board, which doesn't currently start any custom LKM's or daemons. You can create a specific variant for each board and OS you work on, as well as for variations in build configurations.

You could also factor out the common part and create a family of scripts that reference that and tailor functions for each specific build, depending on how fancy you want to get. Here I'm going for simplicity.

#!/bin/sh
mark-build-usage(){
cat <<END
  Define bash command-line functions for marking source files to indicate this
  is a custom build.
  Usage: source mark-build <userNickname> <branchName> <linuxSrc> <appSrc> [<overlaySrc>]
  Example: source mark-build.sh sbranam EMB-1066/blink-blue-led Linux app overlays
  Parameters:
    <userNickname>  - Nickname string to identify user in markers.
    <branchName>    - Git branch name (or other source control identifier).
    <linuxSrc>      - Path to Linux source.
    <appSrc>        - Path to application source.
    <overlaySrc>    - [Optional] Path to buildroot overlay source.
END
}
MB_NICKNAME=$1
MB_BRANCHNAME=$2
MB_LINUX=$3
MB_APPLICATION=$4
MB_OVERLAY=$5
# Check for required parameters.
if [[ -z "$MB_NICKNAME" || "$1" == "-h" || "$1" == "--help" ]] ; then
    # No arguments provided.
    mark-build-usage
    return 1
elif [[ -z "$MB_APPLICATION" ]] ; then
    # Not all required arguments provided.
    echo
    echo "  ERROR: insufficient arguments."
    echo
    mark-build-usage
    return 1
else
    echo "Ready. Enter mark-build-help for description and usage of command-line functions."
fi
mark-build-help() {
cat <<END
Description:
  Shell functions to set and clear marks for a custom build.
  Marks build with log lines in the form:
    [CUSTOMBUILD] <part> <userNickname> <branchName> <timestamp>
  Where:
    <part>         is the build part (kernel, rootfs, overlay, application)
    <userNickname> identifies the user who performed the build
                   (current value $MB_NICKNAME)
    <branchName>   identifies the git branch or other source control identifier
                   (current value $MB_BRANCHNAME)
    <timestamp>    records the date time when built or marked.
  User shell functions (supports tab-completion):
    mark-build-set - Set the build markers for all parts.
    mark-build-reset - Update build markers.
    mark-build-clear - Clear build markers.
    mark-build-find - Find all marked source and binary files and show their
                      marker lines.
    mark-build-mark-kernel - Set the kernel build marker.
    mark-build-mark-rootfs - Set the rootfs build marker.
    mark-build-mark-overlay - Set the overlay build marker.
    mark-build-mark-application - Set the application build marker.
END
}
###############################################################################
# Main user functions.
###############################################################################
mark-build-set(){
    mark-build-mark-kernel
    mark-build-mark-rootfs
    mark-build-mark-overlay
    mark-build-mark-application
    # Add more mark-build-mark-<part> functions here.
}
mark-build-reset(){
    mark-build-clear
    mark-build-set
}
mark-build-clear(){
    git checkout -- ${MB_GIT_FILES[@]}
    rm ${MB_ADD_FILES[@]}
}
mark-build-find(){
    echo "Current timestamp: `date` (`date -u UTC)"
    echo
    # Find all source and binary files with marker.
    grep -lr CUSTOMBUILD * > /tmp/mark-build-find.txt
    readarray -t FILES < /tmp/mark-build-find.txt
    # Print the marker for each file found.
    for f in ${FILES[@]}; do
        echo $f:
        strings -20 $f | grep CUSTOMBUILD
        echo
    done
    # Cleanup.
    rm /tmp/mark-build-find.txt
}
# To add more mark-build-mark-<part> functions, follow the pattern in
# these functions. You can also add more files to a function, but
# it's usually sufficient to mark one file per part.
mark-build-mark-kernel(){
    mark-build-mark-linux-ksrc 0 kernel
}
mark-build-mark-rootfs(){
    mark-build-mark-linux-ksrc 1 rootfs
}
mark-build-mark-application(){
    mark-build-mark-linux-usrc 2 application
}
mark-build-mark-overlay(){
    mark-build-write-sh-script 0 overlay
}
###############################################################################
# Constants.
###############################################################################
MB_WARNING="DO NOT MERGE, THIS FILE IS MODIFIED FOR MARKED BUILD."
MB_C_WARNING="// $MB_WARNING"
MB_SH_WARNING="# $MB_WARNING"
MB_LNX_INCLUDE="#include <generated/compile.h>"
# Tailor the following lists to your system.
# Git repo files to mark.
MB_GIT_FILES=(
    $MB_LINUX/init/main.c
    $MB_LINUX/drivers/pps/pps.c
    $MB_APPLICATION/my_app/myapp.c
    # Add more git repo files here.
)
# Git repo file sed match targets, "a" for append, "i" for insert.
MB_GIT_SED_MATCHES=(
    "/linux_banner/a"
    "/LinuxPPS API/i"
    "/Usage:/i"
    # Add more repo file sed match targets here.
)
# Files to add.
MB_ADD_FILES=(
    $MB_OVERLAY/overlay/etc/init.d/S29custombuild
    # Add more files here.
)
###############################################################################
# Helper functions.
###############################################################################
# Format the source code line to mark a file.
mark-build-get-marker-line(){
    local start=$1
    local part=$2
    local separator=$3
    local timestamp=$4
    local end=$5
    local marker="[CUSTOMBUILD] $part $MB_NICKNAME $MB_BRANCHNAME$separator"
    MB_COMMAND="$start \"$marker\" $timestamp$end"
}
# Mark a Linux kernelspace source file.
mark-build-mark-linux-ksrc(){
    local file=${MB_GIT_FILES[$1]}
    local match=${MB_GIT_SED_MATCHES[$1]}
    local part=$2
    mark-build-get-marker-line "pr_notice(" $part " " UTS_VERSION "\"\\\\n\");"
    sed -i "$match $MB_C_WARNING\n$MB_LNX_INCLUDE\n$MB_COMMAND" $file
}
# Mark a Linux userspace source file.
mark-build-mark-linux-usrc(){
    local file=${MB_GIT_FILES[$1]}
    local match=${MB_GIT_SED_MATCHES[$1]}
    local part=$2
    # The extra space between date and time means it's easier just
    # to build the command in 2 steps.
    mark-build-get-marker-line "printf(" $part " " __DATE__
    MB_COMMAND=$MB_COMMAND"\" \" __TIME__\"\\\\n\");"
    sed -i "$match $MB_C_WARNING\n$MB_COMMAND" $file
}
# Add a marker shell script.
mark-build-write-sh-script(){
    file=${MB_ADD_FILES[$1]}
    part=$2
    mark-build-get-marker-line echo $part "" "`date`"
    cat > $file <<EOF
#!/bin/sh
$MB_SH_WARNING
$MB_COMMAND
EOF
    chmod +x $file
}

In the shell session where I'm working on the repo, I source the script to define the shell functions, specifying my username as the nickname, the git branch (or other source control reference with no intervening whitespace), and the relative paths to the Linux source, the application source, and the overlay directory (if there is one).

The paths are all relative to the repo. I don't need to be in the repo to source the script, but I do to use the functions. For example:

sbranam@ubuntu:~$ cd repos/emb-sys
sbranam@ubuntu:~/repos/emb-sys$ source ~/mark-build-myboard-linux.sh $USER EMB-1066/blink-blue-led linux-5.4 my-app buildroot/board/myboard/common
Ready. Enter mark-build-help for description and usage of command-line functions.

To set the marks, I use mark-build-set:

sbranam@ubuntu:~/repos/emb-sys$ mark-build-set

I can use git status and git diff to see the modifications. Now I do the build.

Once the build ends, to see all text and binary files that contain a mark, along with the actual mark, I use mark-build-find. It searches all directories starting from the current directory. Some times are reported in local time, and some in UTC, so the function first shows the current time in both forms.

I've sanitized some of the paths here, but it's interesting to see where various marked files end up in the build, between staging and output locations for sources, objects, libraries, executables, system images, and final packaged ELF file (what will be flashed onto the board).

Some of this depends on the way the board vendor has setup the packaging process, combined with their components and tools. So this also useful for learning about the build and packaging process.

Another interesting thing to note is the effect of squashfs. The vendor packages the final rootfs as a squashfs image in the ELF, which results in compression of the file contents. Therefore, most of the strings that are found in the pre-compressed rootfs get scrambled, so aren't found in the rootfs.squashfs file. But not all of them. I'm surprised to see that any marker information could be found in the squashfs output, but once again, that leaves helpful breadcrumbs for learning about the build process.

Then notice that both the kernel and partial rootfs marks are found in the final ELF fwprog/bst_bld_lnx_rfs.elf, providing some degree of confirmation that I have the desired contents before I flash it to the board. Seeing the marks with consistent timestamps in all the pre-ELF files gives me some confidence (though not a guarantee) that those components are in it as well. There's some variability in the timestamps due to the time taken to progress through compiling the various source files.

sbranam@ubuntu:~/repos/emb-sys$ mark-build-find
Current timestamp: Mon Feb  8 21:43:01 EST 2021 (Tue Feb  9 02:43:01 UTC)
buildroot/board/myboard/common/overlay/etc/init.d/S29custombuild:
echo "[CUSTOMBUILD] overlay sbranam EMB-1066/blink-blue-led" Mon Feb  8 21:19:45 EST 2021
fwprog/.temp/lnx.fw:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
fwprog/.temp/lnx.bin:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
fwprog/.temp/rfs.fw:
[CUSTOMBUILD] rootfs sbraTW
fwprog/rfs.elf:
[CUSTOMBUILD] rootfs sbraTW
fwprog/lnx.elf:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
fwprog/bst_bld_lnx_rfs.elf:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
[CUSTOMBUILD] rootfs sbraTW
fwprog/devfw/rfs.devfw:
[CUSTOMBUILD] rootfs sbraTW
fwprog/devfw/lnx_rfs_dtb.devfw:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
[CUSTOMBUILD] rootfs sbraTW
fwprog/devfw/lnx.devfw:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
linux-5.4/init/main.c:
pr_notice( "[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led " UTS_VERSION"\n");
linux-5.4/drivers/pps/pps.c:
pr_notice( "[CUSTOMBUILD] rootfs sbranam EMB-1066/blink-blue-led " UTS_VERSION"\n");
output.oem/myboard/build/my-app/myapp.o:
[CUSTOMBUILD] application sbranam EMB-1066/blink-blue-led Feb  9 2021 02:26:21
output.oem/myboard/build/my-app/myapp.c:
printf( "[CUSTOMBUILD] application sbranam EMB-1066/blink-blue-led " __DATE__" " __TIME__"\n");
output.oem/myboard/build/my-app/myapp:
[CUSTOMBUILD] application sbranam EMB-1066/blink-blue-led Feb  9 2021 02:26:21
output.oem/myboard/build/linux-custom/init/main.o:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/vmlinux.o:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/.tmp_vmlinux.kallsyms1:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/drivers/pps/pps_core.o:
5pps_core: [CUSTOMBUILD] rootfs sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/drivers/pps/pps.o:
5pps_core: [CUSTOMBUILD] rootfs sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/drivers/pps/pps_core.ko:
5pps_core: [CUSTOMBUILD] rootfs sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/vmlinux:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/.tmp_vmlinux.kallsyms2:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/build/linux-custom/arch/arm64/boot/Image:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/images/rootfs.squashfs:
[CUSTOMBUILD] rootfs sbraTW
output.oem/myboard/images/Image:
5[CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
output.oem/myboard/target/etc/init.d/S29custombuild:
echo "[CUSTOMBUILD] overlay sbranam EMB-1066/blink-blue-led" Mon Feb  8 21:19:45 EST 2021
output.oem/myboard/target/usr/bin/example:
[CUSTOMBUILD] application sbranam EMB-1066/blink-blue-led Feb  9 2021 02:26:21
output.oem/myboard/target/lib/modules/5.4.45/kernel/drivers/pps/pps_core.ko:
5pps_core: [CUSTOMBUILD] rootfs sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
my-app/myapp.c:
printf( "[CUSTOMBUILD] application sbranam EMB-1066/blink-blue-led " __DATE__" " __TIME__"\n");

On the target board, after I load this build and boot, I get the following logging:

[    0.000000] Linux version 5.4.45 (sbranam@builder) (gcc version 8.2.1 20180802 (Linaro GCC 8.2-2018.08~dev)) #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
[    0.000000] [CUSTOMBUILD] kernel sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
...
[CUSTOMBUILD] overlay sbranam EMB-1066/blink-blue-led Mon Feb 8 21:19:45 EST 2021
Setup eth0...
...
[    1.872227] pps_core: [CUSTOMBUILD] rootfs sbranam EMB-1066/blink-blue-led #1 SMP PREEMPT Tue Feb 9 02:31:01 UTC 2021
[    1.882901] pps_core: LinuxPPS API ver. 1 registered

When I look for the application, I find it in the rootfs. When I run it and trigger the usage output, I see the mark.

/ # which myapp
/usr/bin/myapp
/ # myapp -h
[CUSTOMBUILD] application sbranam EMB-1066/blink-blue-led Feb  9 2021 02:26:21
Usage: myapp [Options]
Options:
        -h: help
        -d: run as daemon mode
        -v: enable more verbose debug messages

Back in my repo, to clear all source code and overlay marks before I commit any real changes, I use mark-build-clear:

sbranam@ubuntu:~/repos/emb-sys$ mark-build-clear

Once again, I can use git status to verify the modifications no longer exist.

Because it's fast and easy, I can set and clear marks as often as I want.

Is this all just a bunch of hacking? Yes, definitely. But in a good way.

I think of this type of thing as "useful hacking," because it both serves a functional purpose for aiding in my job, and is very educational.

That's part of what I call "hacker mentality," which is focused on problem-solving through active exploration, learning, and applying what I've found in a tight feedback loop.

It combines the empirical approach of Thomas Edison ("Genius is one percent inspiration and ninety-nine percent perspiration.") with the analytical approach of Nikola Tesla ("Just a little theory and calculation would have saved him ninety percent of his labor.”).

In this case I learned a bunch of details about Linux and the build process that I had previously taken for granted. I can apply that same useful hacking to learning about other systems.

That knowledge makes me more versatile. I often find that it's serendipitously useful in subsequent work. Had I not gone through that process, I wouldn't have had that knowledge available to help out.

Sign in