Tenderfoot: How to Write a Great Bug Report

I am an odd sort of person. Why? Because I love a well written and descriptive bug report. I love a case that includes clear and easy to follow reproduction steps. I love a written bug report that includes all the necessary information on versions, configurations, connections and other system details. Why? Because I believe in efficiency. I believe that as an engineer I have a duty to generate value to my employer or customer. Great bug reports are one part of our collective never-ending goal of improving team efficiency and increasing the value we generate as engineers, quality assurance staff, or even as a customer.

What does a *bad* bug report look like? Let us pretend that Léon is an engineer developing embedded software for a contrived Ethernet enabled RF signal generator product, hereafter called “RFGen.” RFGen consists of a front panel control with various buttons, an Ethernet connection, a touch-screen LCD display, and a single RF output. RFGen outputs various signal patterns selected through either the front panel or through a proprietary TCP based device control protocol.

Léon arrives at work one morning and scans his incoming email. One email, from Günter in marketing, is titled “Pattern Delta does not work” and the body of the email reads:

“When I enable Pattern Delta, there is no signal on the RF output. Can this be fixed before next week’s trade show?”

Léon scratches his head a bit and wonders how this report could be correct. He then turns on his development unit, loads the latest embedded software, and attempts to reproduce the case. Twenty-one minutes later Léon writes the following response to Günter:

“I am sorry, but I was not able to reproduce the reported issue. Are there any additional details?”

Let us pause in our story and see where we are. Given the above, the following items are of immediate concern:

• The case report arrived informally via email. A case tracking system is preferred.
• The report does not include any version information.
• The report does not include any detailed steps to reproduce. For example, we do not know if Günter enabled the feature using only the front panel or if the TCP control protocol activated the feature.
• Léon responds politely, which is great, but the vague request for “additional details” is open ended and not clear. What additional details does Léon require?

There was one excellent element to Günter’s original report: “Can this be fixed before next week’s trade show?” This sentence in the report clearly communicates the urgency of the case.

Back to our story. Later, Günter arrives at the office and reads Léon’s email. Günter responds with:

“Well, I was playing with the recently delivered prototype, trying out all the various patterns and noticed that Pattern Delta did not work. Give me a few minutes to try again.”

Thankfully for all concerned, Günter then remembered a training session he attended on behalf of the marketing department, a session dedicated to how to correctly use the engineering department’s case tracking system. Günter searches for the training materials and the internal web site used to enter cases into the tracking system. An hour later, Léon receives the following email, automatically generated by the case tracking system.

From: Cases
To: Léon
Subject: Case 3981: Pattern Delta does not work after using Pattern Alpha

Steps to Reproduce: 
1. Use the touch screen menu to enable Pattern Alpha 
2. Confirm Pattern Alpha is present on the RF signal output 
3. Use the menu again to enable Pattern Delta

Expected Results: 
“Pattern Delta” appears on the RF signal output.

Actual Results: 
No signal appears, the RF output is off.

Reproducibility: 
100%, happens every time. But only if Pattern Alpha was the previous selection. Pattern Delta works correctly if activated after other patterns were activated.

Workaround: 
Enable some other pattern, such as Pattern Theta, then enable Pattern Delta.

Configuration: 
SW Version 0.2.3, FPGA version 56A1, prototype hardware rev 2. The device was not connected to Ethernet

Priority 3: 
Note: now that I see this only happens after using Pattern Alpha, I can avoid it during the upcoming trade show, but it would be nice to have a fix before the trade show if possible.

Much better, yes? Léon then reproduces the issue and within minutes finds a simple bit mask bug that was only revealed during the transition from Pattern Alpha to Pattern Delta. The case was fixed and new software delivered in time for the upcoming trade show.

Given our contrived example story, here are the extracted elements of an excellent bug report:

• As best as can be determined by the reporter, the report should describe only a single issue. Do not report multiple issues in a single case, as this complicates tracking the case to its final resolution.
• Clear steps to reproduce.
• Expected results, i.e. what the reporter believes should have happened. 
  • If a formal specification is available, reference the spec and appropriate section, paragraph, or page number.
• Actual results -- this is the bug.
• Any configuration information that might help in reproducing the issue, such as:
  • hardware revisions
  • software versions
  • Data connections, environment, connected peripherals, etc.
• Urgency/priority
• Reproducibility – does it happen every time?
• Work around or other method to avoid the bug, if any.

That is a lot of data to gather and type up, which certainly consumes additional time for the person reporting the case. However, the benefits outweigh the initial costs. Benefits include:

• Improved and faster communications
  • Greatly reduced “back and forth” exchanges when trying to reproduce the case.
• Efficiency
  • The engineering team will spend less time extracting information from people and more time developing and fixing issues.
• Traceability – code changes can be directly associated to the case
  • If many issues are reported in a single case, it becomes difficult to trace or track the associated software or other changes and their eventual releases.
• Reproducibility
  • With clear steps, not only can the engineer assigned to fix the case quickly reproduce and resolve the case, but every other person exposed to the case can assess the case faster. For example, a project manager may scan the case and realize it is more important than originally filed, and raise the priority. Additionally, QA staff attempting to test future releases will be able to quickly confirm the case resolution by simply following the case’s already documented reproduction steps.
• Documentation
  • If the project doesn’t have clear product specifications, the “Expected Results” section of any bug report might be extracted to add to existing specs or help create new specifications.

And for our reader's future usage, here is a template that could be used by your team for creating bug reports:

Steps to Reproduce:
1) Step 1
2) Step 2
3) ...

Expected Results:
-- Describe the expectations, i.e. what should have happened immediately after the last reproduction step.

Actual Results:
-- What happened instead of the expected results? This is the bug.

Reproducibility:
-- Does this happen every time? 50% of the time? Rare?

Workaround:
-- Is there a method to avoid or work around this issue? This may be useful to the service department if the case was found in a fielded product.

Configuration:
-- Configuration might include: Software versions, hardware revisions, product revisions, signals connected, attachments in use, etc.
Note: when using a case tracking system, configuration items may often be defined on a per project basis as database fields, very useful for sorting cases and/or searching through active issues.

Priority:
--  1 – critical. 2 – Normal 3 – Low.   
Note: priority is often defined or entered as a field/selection in the case tracking system’s database. Adjust priorities to fit your company’s requirements.

Easy right? With bugs, cases, issues, or however your team refers to them, a clear bug report is critical to any rapid development process. Please comment and contribute the “must have” details you would like to see in a bug report!

PS: If an additional follow-up post regarding “Elements of a Great Bug Tracking System” is of interest, please comment below.

Sign in