Electronic component distibutor Digi-Key recently announced part tracing for surface-mount components purchased in cut-tape form. This is a big deal, and it’s a feature that is a good example of traceability. Some thing or process that has traceability basically just means that it’s possible to determine an object’s history or provenance: where it came from and what has happened to it since its creation. There are a whole bunch of ways you can apply traceability to embedded systems, both hardware and software, and it’s a really good habit to get into when you are involved in embedded hardware or software design.
To understand why Digi-Key’s part tracing is a big deal, you also have to understand what cut tape is.
Let’s say you want to buy some surface-mount parts… oh, I don’t know, let’s say some Murata GCJ188R71E105KA01D (why, that just rolls right off the tongue) which are 1μF 25V X7R 0603 capacitors. Great, that will be \$218.48 for a reel of 4000. What, you don’t need 4000 of them? You only need 50? OK, sure, then buy 50 of them for \$5.90 — Digi-Key will happily cut off part of a reel and send that to you instead. Problem solved. You get a strip of paper tape with capacitors in a series of regularly-spaced holes, kept in place by a thin plastic cover. Now you put together some prototypes and you use 36 of them. 14 are left over, and sit in a box on your desk, where they are nondescript parts in a nondescript package. You come back several years later, and look at it and scratch your head. What kind of capacitors are these again? Yeah, you can measure them as 1μF capacitors, but what voltage rating? which series? Not all 1μF capacitors are alike. Too bad, you’re out of luck.
But now Digi-Key is printing identifying metadata on the back of the tape — so as long as the fragment of tape is long enough and you trust that no one has swapped out the contents of the remaining components, you can get all sorts of information:
Digi-Key is the first distributor in the industry to offer this level of customization and enhancement and expects the offering to be available on 36,000 parts by June 2021. The following information is available to be printed on 8 mm-wide paper carrier tape products: sales order, invoice, line item (detail ID), date code, lot code, Digi-Key part number, manufacturer part number, manufacturer name, quantity, country of origin, catalog description, customer reference, and PO number. Orders must be a minimum length of 200 mm (about 8 inches), which equates to about 50 parts for 4 mm pitch tape, or 100 parts for 2 mm pitch tape. The part tracing information will be printed in English.
The part tracing page shows a bar code. It would be nice to have some kind of smartphone app so that you could spread out a bunch of these cut tape pieces on a table, back side up and side by side, take a picture of them, and get a list of the parts you have.
Provenance
Traceability and provenance are two closely related concepts. John Keogh disputes the interchangeability of these terms based on one definition in one dictionary of provenance, namely “the origin or source of something”, and along the way gets muddled between the ideas of safety versus source / origin. I disagree with this reasoning, as it is the other definition in Merriam-Webster — the history of ownership of a valued object or work of art or literature — which is the relevant one. In the art world, provenance is the documented history of a work of art. For example, it’s not just Kandisky’s Auf Weiss II that’s hanging in the Centre Pompidou in Paris, but Auf Weiss II, painted in 1923 and donated to the museum by his widow Nina Kandinsky in 1976.
Or, as another example, van Gogh’s The Starry Night — which is important to understand as an 1889 painting the artist sent to his brother Theo, who died a little more than a year later; Theo’s widow Jo sold it in 1900 and bought it back a few years later, then sold it again in 1906, when it was bought by a woman from Rotterdam who owned it until 1938, then by a New York art gallery, and then finally acquired by the Museum of Modern Art in New York City in 1941.
The historical records of an artwork’s sale help to vouch for the authenticity of a painting. Imagine an art dealer today, say one Sly P. Wolff, producing out of nowhere a painting for sale, claimed to be an unknown work by Vincent van Gogh, but with no previous record of it ever having existed, and no history of how Mr. Wolff had acquired it. There would be a great deal of skepticism, to say the least.
An object’s provenance consists of the records that can be produced because of its traceability.
Traceability in Hardware
I like traceability because it’s an antidote for the problem of Objects from Somewhere. A number of years ago, I stated in an article about CRCs
…
- You find source code from somewhere and use it.
- And everyone’s happy.
And then the next time you have to write communications software, you either find the code you wrote the last time, or you follow the last two steps.
Except that’s not enough for me. Because I don’t like finding source code from somewhere. Somewhere is a bad place to rely on. Somewhere has bugs and misinformation and it just floats around, like influenza, mostly causing mild havoc but occasionally fatal results. So I leave the code from somewhere to the amateurs. If I want to sleep well at night, I get my code from a reputable place. And it took me maybe the third time of going through the CRC merry-go-round to realize this.
Traceability in hardware prevents Objects from Somewhere, because now we can find out exactly where an object is from. If someone places an electronic thing in front of me, ideally I should be able to determine the following:
- what it is — circuit boards should have a board ID or part number, and a revision number, as part of the silk screen
- which unit it is (a serial number)
- how it has been changed from the original design — serial number and any ECO information should be written on a label or handwritten in a blank space on the board.
Ideally the same information should be written in on-board EEPROM so that it can be determined automatically by connecting the board to a computer via USB or UART.
Really good traceability also includes past history (where the board was and how it was used) along with traceability for its components and manufacturing. Medical devices are subject to this type of traceability through ISO 13485: an auditor should be able to walk up to any modern medical device, look up its part number and serial number, and obtain full records on the device including all of its components — so if, for example, the device has a circuit board with some 1μF capacitors on it, information about those specific capacitors should be available, including when they were made, which lot they came from (date/lot codes), and which distributor sold them. This is important in case a manufacturing problem is found with components in a certain batch or lot, or in cases where individual units on the production line were assembled or treated incorrectly during manufacturing. (For example: suppose some infusion pumps exhibit a defect, and auditors discover a pattern where almost all of these defective pumps are manufactured on Thursday afternoons. Later, the issue is traced to a certain employee who works the day shift and plays poker on Wednesday evenings so late into the night that he is tired, and sometimes makes a mistake in assembly. Without traceability of devices to their date and time of manufacture, it would be much harder to discover these sorts of patterns.)
Sometimes the auditor is an employee of a regulatory or certification agency, but sometimes it is just a customer or manager or employee who wants to double-check the history of a device. I should be able to walk up to a device and figure out exactly what it is, and what components were used to build it; if I cannot, then there is a flaw in the design or manufacturing process.
Microcontrollers often have a device ID that can be read electronically, using an in-circuit programmer or within firmware. For example, Microchip dsPIC microcontrollers have DEVID and DEVREV registers that can be used to determine which device and revision is present. For example, if DEVID == 0x7C64 then this is a dsPIC33CK128MP508, whereas 0x7C74 indicates a dsPIC33CK256MP508. The MPLAB X development environment will use this information and give a warning when an attempt is made to program a device with firmware meant for another. (This has saved me numerous times from incorrect programming attempts when I switch processors!)
Some of the more recent dsPICs have a 120-bit Unique Device ID (UDID) register that is programmed at the factory and can be read from an in-circuit programmer or from firmware. These registers don’t directly identify the board, but at least they cover the microcontroller and can be used to help provide some traceability information.
Traceability in Software / Firmware
Traceability in software (I’m going to use the term “software” for “firmware” interchangeably in this section) is similar, but also includes some other aspects.
First of all, the origin and history of the software is important:
- what application is it? what version is it? when was it released?
- what source code tree was used to produce it?
- what versions of tools and build flags were used by those tools? (compiler, linker, etc.)
- what changes were there from previous versions? (source control)
Determining the application and version is something that should be readable electronically through an ID field or by requesting a hash function of the contents of program memory.
Again, I should be able to walk up to a device or circuit board with software on it, and figure out exactly what software is running. If I can’t, then there’s a flaw in the software design. This is a pretty big pet peeve of mine, and it’s going to be the first question I ask when I help someone debug their embedded system. If the software is not exactly known, then I’m going to go away and do something else; it’s a waste of everybody’s time if we aren’t sure which source code was used to program the device.
Traceability in Design and Verification
Then there are aspects of traceability in the design/verification process itself:
-
fixing bugs and adding features should be traceable to requirements or to issue tracking numbers — this is a really good habit when entering commit messages for version control systems: include a key that references the issue being addressed, for example:
Fixed whosywhatsit iterator off-by-one error as documented in WHOSY-12345.Many code review tools will automatically parse these keys and automatically add hyperlinks to the issue tracking database. -
test result documentation should be traceable to the constituents of any verification test:
- the test procedure
- references to the exact version of software/hardware being tested (and serial number for the hardware)
- references to all test equipment being used
This last point about test equipment is critical. Don’t just say that you measured Vdd = 4.984 V; say what equipment you used to measure it: a FooBar FB123 Digital Multi-meter, serial #12345, last calibrated Feb 22 2021, due Feb 19 2022. Calibrated test equipment is itself traceable to NIST (or other test agency) metrology standards.
What, you aren’t using calibrated test equipment? Start doing this! You’re making design decisions on expensive engineering projects, so you’d better be sure your equipment meets its accuracy specifications. This is also another reason not to use really cheap test equipment — handheld DMMs cost about the same amount to calibrate, whether they are low-end or high-end multimeters. My expectation these days in the USA is that yearly DMM calibration is about \$40 - \$60 per handheld multimeter. (See for example custom-cal.com which advertises a standard calibration of \$55-\$60 for handheld Fluke DMMs and \$80 for benchtop Fluke DMMs.) So if you’re planning on having a calibrated multimeter in service for ten or fifteen years, it makes no sense to save pennies and purchase a cheap DMM for \$50, but then spend \$500 over the next ten years for calibration. Get something that has decent accuracy and resolution.
Rufus Xavier Sarsparilla Provides Us With a Status Update
There’s another critical aspect of documentation in design and verification that is important to note in its own right, namely the use of definite articles, pronouns, and antecedents.
Those of us of a certain age may remember Schoolhouse Rock, a series of educational animated musical videos that appeared during the commercial breaks of Saturday morning cartoons on the ABC television network. These videos covered multiplication, science, American history and civics (notably I’m Just a Bill), and grammar. The “Grammar Rock” videos were very catchy — Conjunction Junction, what’s your function… and Lolly’s, Lolly’s, Lolly’s, get your adverbs here… — but one in particular springs to mind on this topic. The song Rufus Xavier Sarsparilla, written by Bob Dorough and performed by Jack Sheldon, talks about pronouns:
Now, I have a friend named Rufus Xavier Sarsaparilla,
And I could say that Rufus found a kangaroo
That followed Rufus home
And now that kangaroo belongs
To Rufus Xavier Sarsaparilla.
Whew! I could say that, but I don’t have to,
‘Cause I got pronouns,
I can say, “HE found a kangaroo that followed HIM home and now IT is HIS”
The important lesson here, to the young child, is that You Too Can Use Pronouns to Save Unnecessary Effort!
You see, a pronoun was made to take the place of a noun,
‘Cause saying all those nouns over and over
Can really wear you down!
The important lesson to the engineer, however, should be to use pronouns with utmost caution. Not only the obvious ones like I/he/she/it, but also demonstrative pronouns (this/that/these/those). Pronouns are basically a hint that the reader/listener should retrieve some bit of previously-known information, called the antecedent, which has been replaced with a much shorter word (the pronoun) instead. Yes, you know that. You see, a pronoun was made to take the place of a noun.... But pronouns can be used sloppily and create ambiguous antecedents.
Compare these descriptions:
I ran the test again and it failed when that MOSFET blew up and let the magic smoke out. Ha ha, that stuff stinks! First I thought maybe it was the resistor, but then I replaced it and the MOSFET and tried it again, and this time it worked until it got to 20 A, but then it blew up again. I’m going to try that another time but then I think we may need to get the other board and check it with the DMM and try that instead. This isn’t a great design, is it?
and
I ran test WHOSYTEST-052 a second time on WHOSY board #002. MOSFET Q9 was destroyed and the test failed.
Possible causes may include resistor R23, the gate resistor for Q9.
After replacing R23 and Q9, I repeated WHOSYTEST-052 and MOSFET Q9 was destroyed again after current through Q9 reached approximately 20 A.
I will retry WHOSYTEST-052 one more time on WHOSY board #002, and if the board fails another time, I plan on trying the same test with WHOSY board #003, in case this is just a defect in board #002. Before testing, I will check board #003 with a calibrated DMM to verify all gate drive resistors have been correctly upgraded as per WHOSY-ECO-001 rework instructions.
The first is rather a casual narrative, which is opinionated at times, and has lots of sloppy pronouns. First I thought maybe it was the resistor, but then I replaced it and the MOSFET and tried it again, and this time it worked until it got to 20 A, but then it blew up again. That’s six uses of it, each presumably referring to different things: the cause of failure, the resistor, the test, the board, the current through the MOSFET, and the MOSFET itself.
The second is objective and precise. Be objective and precise.
Even such simple words as “the” can be prone to misinterpretation. The word “the” is a definite article and always refers to a specific thing… but sometimes it is not clear which thing. Legal documents avoid this ambiguity by defining terms, usually noted later by a capitalized noun, for example:
The Blobbery Software License Agreement (the “Agreement”) is between you and the Blobbery Unrealistic Expectations Corporation (“Blobbery”). Blobbery reserves the right to cancel the Agreement at any time. Your usage of Blobbery for online document research (“Research Usage”) is limited for each 30-day window (the “Window”). During each Window the Research Usage may be revoked by Blobbery without explanation and without recourse....
At any rate: when you describe the results of any test or investigation — whether it is in an informal email message or a formal test result — be as precise as you can. Don’t leave pronouns and definite articles up to misinterpretation.
Traceability in Documentation
Lastly, there should be traceability in documents themselves. Each document should have a unique identifier, complete with a revision ID. Furthermore, each page should list the document ID and the page number.
Semiconductor manufacturers are usually good at this, leaving you document IDs that you can cite in your own design documentation. For example:
- Microchip dsPIC33CK256MP508, datasheet DS70005349H
- Texas Instruments TLC555, datasheet SLFS043I
- Nexperia 74HC14, datasheet 74HC_HCT14
Datasheets from Nexperia and its former parent company NXP have a nice little feature where they stick in a unique ID in each figure, located discreetly in the corner, so taking a screenshot of a figure can be used to find the figure online later. For example, the mna208 in this timing diagram from the 74HC14 datasheet shows hysteresis levels:
This particular figure isn’t unique to one datasheet; it’s reused in the 74LVC1G58, but it does provide traceability back to one or more source documents.
Many of the Microchip microcontroller datatsheets have specifications with parameter IDs — for example, here are the ADC specifications from the dsPIC33CK256MP508 datasheet:
This makes it easy to refer to a particular parameter, for instance AD60 refers to typical sampling capacitance of dedicated ADC cores, whereas AD61 refers to typical sampling capacitance of the shared core. If I refer to sampling capacitance AD60 there is no confusion which of the two it is.
If you have important details in your design which are specified in manufacturers’ datasheets or application notes, and they provide IDs to help refer to documents or portions of documents, use those IDs.
If you are providing important details in your own documentation — well, of course you are, otherwise why would you bother writing those documents — make it easy to refer to them: tables, figures, and equations all deserve numbered labels. It is much easier for your readers to say “Equation 22” than to say “the second-to-last equation on page 34 which refers to \( V(x) \)”
It should go without saying that you should write down the calculations and rationale that drives design decisions. Design description documents are ideal places to collect this kind of information. There’s nothing worse than finding a schematic from five years ago where you picked a part, say a voltage regulator, with a certain rating, and you know you went through some calculations to justify that the part was used within its rated value, but you can’t find those calculations.
Help your readers find the information they need!
Wrapup
Today we looked at various techniques for traceability in embedded systems. What they all have in common is a way to be definite about specific details, by providing metadata in a way that can be readily accessed when needed.
The provenance, or history, of a specific device is important information in the medical and aerospace industries. It allows investigators to find correlations between defects in a device and the way it has been constructed and handled, or between device defects and components from a particular batch.
Software or firmware used to program devices should contain enough metadata (version numbers, build date/time, etc.) to assist in tracing and identifying the cause of problems.
Documentation should provide IDs to make it easy to refer to specific documents or sections of documents. Language used in documentation should be precise and avoid ambiguities caused by poor usage of pronouns and definite articles.
Remember, while you might be very familiar with the details of a design today, in several years you may have forgotten some of those details, and may need to pull up specific information related to a design or to particular product units built to that design. Don’t store these details in your head or in your desk or on your computer. Archive them in a secure place where they can be found easily from information available directly in the product. Someone — and it might be you! — may need these details to help answer all sorts of questions for quality control or legal reasons, or to apply your successes to the design of future products. Best practices in traceability will help provide those answers in the future.
© 2021 Jason M. Sachs, all rights reserved.
