Tenderfoot: How to Write a Great Bug Report
I am an odd sort of person. Why? Because I love a well written and descriptive bug report. I love a case that includes clear and easy to follow reproduction steps. I love a written bug report that includes all the necessary information on versions, configurations, connections and other system details. Why? Because I believe in efficiency. I believe that as an engineer I have a duty to generate value to my employer or customer. Great bug reports are one part of our collective never-ending goal of improving team efficiency and increasing the value we generate as engineers, quality assurance staff, or even as a customer.
What does a *bad* bug report look like? Let us pretend that Léon is an engineer developing embedded software for a contrived Ethernet enabled RF signal generator product, hereafter called “RFGen.” RFGen consists of a front panel control with various buttons, an Ethernet connection, a touch-screen LCD display, and a single RF output. RFGen outputs various signal patterns selected through either the front panel or through a proprietary TCP based device control protocol.
Léon arrives at work one morning and scans his incoming email. One email, from Günter in marketing, is titled “Pattern Delta does not work” and the body of the email reads:
“When I enable Pattern Delta, there is no signal on the RF output. Can this be fixed before next week’s trade show?”
Léon scratches his head a bit and wonders how this report could be correct. He then turns on his development unit, loads the latest embedded software, and attempts to reproduce the case. Twenty-one minutes later Léon writes the following response to Günter:
“I am sorry, but I was not able to reproduce the reported issue. Are there any additional details?”
Let us pause in our story and see where we are. Given the above, the following items are of immediate concern:
- The case report arrived informally via email. A case tracking system is preferred.
- The report does not include any version information.
- The report does not include any detailed steps to reproduce. For example, we do not know if Günter enabled the feature using only the front panel or if the TCP control protocol activated the feature.
- Léon responds politely, which is great, but the vague request for “additional details” is open ended and not clear. What additional details does Léon require?
There was one excellent element to Günter’s original report: “Can this be fixed before next week’s trade show?” This sentence in the report clearly communicates the urgency of the case.
Back to our story. Later, Günter arrives at the office and reads Léon’s email. Günter responds with:
“Well, I was playing with the recently delivered prototype, trying out all the various patterns and noticed that Pattern Delta did not work. Give me a few minutes to try again.”
Thankfully for all concerned, Günter then remembered a training session he attended on behalf of the marketing department, a session dedicated to how to correctly use the engineering department’s case tracking system. Günter searches for the training materials and the internal web site used to enter cases into the tracking system. An hour later, Léon receives the following email, automatically generated by the case tracking system.
Subject: Case 3981: Pattern Delta does not work after using Pattern Alpha
Steps to Reproduce:
1. Use the touch screen menu to enable Pattern Alpha
2. Confirm Pattern Alpha is present on the RF signal output
3. Use the menu again to enable Pattern Delta
“Pattern Delta” appears on the RF signal output.
No signal appears, the RF output is off.
100%, happens every time. But only if Pattern Alpha was the previous selection. Pattern Delta works correctly if activated after other patterns were activated.
Enable some other pattern, such as Pattern Theta, then enable Pattern Delta.
SW Version 0.2.3, FPGA version 56A1, prototype hardware rev 2. The device was not connected to Ethernet
Note: now that I see this only happens after using Pattern Alpha, I can avoid it during the upcoming trade show, but it would be nice to have a fix before the trade show if possible.
Much better, yes? Léon then reproduces the issue and within minutes finds a simple bit mask bug that was only revealed during the transition from Pattern Alpha to Pattern Delta. The case was fixed and new software delivered in time for the upcoming trade show.
Given our contrived example story, here are the extracted elements of an excellent bug report:
- As best as can be determined by the reporter, the report should describe only a single issue. Do not report multiple issues in a single case, as this complicates tracking the case to its final resolution.
- Clear steps to reproduce.
- Expected results, i.e. what the reporter believes should have happened.
- If a formal specification is available, reference the spec and appropriate section, paragraph, or page number.
- Actual results -- this is the bug.
- Any configuration information that might help in reproducing the issue, such as:
- hardware revisions
- software versions
- Data connections, environment, connected peripherals, etc.
- Reproducibility – does it happen every time?
- Work around or other method to avoid the bug, if any.
That is a lot of data to gather and type up, which certainly consumes additional time for the person reporting the case. However, the benefits outweigh the initial costs. Benefits include:
- Improved and faster communications
- Greatly reduced “back and forth” exchanges when trying to reproduce the case.
- The engineering team will spend less time extracting information from people and more time developing and fixing issues.
- Traceability – code changes can be directly associated to the case
- If many issues are reported in a single case, it becomes difficult to trace or track the associated software or other changes and their eventual releases.
- With clear steps, not only can the engineer assigned to fix the case quickly reproduce and resolve the case, but every other person exposed to the case can assess the case faster. For example, a project manager may scan the case and realize it is more important than originally filed, and raise the priority. Additionally, QA staff attempting to test future releases will be able to quickly confirm the case resolution by simply following the case’s already documented reproduction steps.
- If the project doesn’t have clear product specifications, the “Expected Results” section of any bug report might be extracted to add to existing specs or help create new specifications.
And for our reader's future usage, here is a template that could be used by your team for creating bug reports:
Steps to Reproduce:
1) Step 1
2) Step 2
-- Describe the expectations, i.e. what should have happened immediately after the last reproduction step.
-- What happened instead of the expected results? This is the bug.
-- Does this happen every time? 50% of the time? Rare?
-- Is there a method to avoid or work around this issue? This may be useful to the service department if the case was found in a fielded product.
-- Configuration might include: Software versions, hardware revisions, product revisions, signals connected, attachments in use, etc.
Note: when using a case tracking system, configuration items may often be defined on a per project basis as database fields, very useful for sorting cases and/or searching through active issues.
-- 1 – critical. 2 – Normal 3 – Low.
Note: priority is often defined or entered as a field/selection in the case tracking system’s database. Adjust priorities to fit your company’s requirements.
Easy right? With bugs, cases, issues, or however your team refers to them, a clear bug report is critical to any rapid development process. Please comment and contribute the “must have” details you would like to see in a bug report!
PS: If an additional follow-up post regarding “Elements of a Great Bug Tracking System” is of interest, please comment below.
Previous post by Matthew Eshleman:
Tenderfoot: Embedded Software and Firmware Specialties
Next post by Matthew Eshleman:
The Hardest Bug I Never Solved
From my own experience, I would add just two things:
has it worked before?
If so, what has changed since it worked last?
I've found this often yields useful / interesting information on what the user or customer has done to the system.
Great addition and comments! Thank you!
Yeah, right. Keep poking at them in a gentle way and they admit/remember something happened and they did XXX as a work-around, but that wasn't a change! - except they changed a config or run/script file, etc. But nothing was changed.....
"This is my g'gpas knife. we had to change the blade 2 time and the handle 5 times, but it's the same knife!"
I have had several bugs over the years where I have searched forums for a clue. I end up finding a post about the very same bug, but none about the solution. Some of the forum posts were several years old
one example: I was using a win XP VM, and could not get the USB connection to a programming pod to work - no connect to the pod. I finally found the workaround/solution - basically an errata. I tried it - success! bring out the fatted (fated) calf.
The next thing I did was goto the forum with the posting of the problem, and wrote up the solution with links, 8x10 color glossies with circles and arrows describing what each one was to be used in a court (oops.. slipped into Arlo mode there).
I also saved the salient parts of the forum original post and my write-up (with links to the forum trail), and put it into a nice little document && emailed it to myself. When I am looking for a gig, I have a writing example to send to the prospective client. They can see my (informal) writing style && that i give back. Every little bit helps.
To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.
Registering will allow you to participate to the forums on ALL the related sites and give you access to all pdf downloads.