Software Metrics (cat flame > /dev/null)

Hi Dave,

On 7/15/2011 12:45 PM, Dave Nadler wrote:
> Many decades ago I worked for a well-known company
> that made testers. They had a nice bonus for the
> engineer that designed the board with the fewest
> field failures. The engineers regularly fought hard
> to design a memory board, which was a shoe-in to
> win the prize (compared to the tough analog front-
> ends exposed to regular customer abuse).

Presumably, this was for semiconductor memory (and not
core planes  :> )...

> They also didn't count labor hours in the metrics
> for board cost. That led them to take out of
> production a UART board using an *expensive*
> crystal and reintroduce its predecessor, which
> had hand-tweaked-and-soldered RC frequency
> generation. They also discontinued a subsystem
> using ribbon cables to reintroduce hand-soldered
> cable bundles because it was *clearly* less expensive.
> BTW, labor was even expensive in USA back then.

I don't understand.  Are there two different criteria at
play, here (cost and failure rate)?

> I could go on for hours...
>
> Most metrics used don't reflect what most of
> us would consider reality. Software metrics in
> use today lead to outcomes just as silly as
> those listed above...

But that, I think, is because the metrics are being used
for "business purposes" (cost accounting, etc.).

E.g., I just coded a "unified memory manager" to replace
the various different *types* of memory management
mechanisms used in many embedded systems.  Once I've
given it a thorough shake-down in an application, I
will go back and write comparable "traditional" tools
to provide the same functionality.  *Then*, I will
see what their "metrics" look like to help me evaluate
the utility (or disutility?) of this new approach.

I.e., if the new approach is *conceptually* more complicated
but "metrically" simpler/smaller/etc., then that speaks to
reliability, maintainability, etc. in a way more readily
defensible than some emotional "hand-waving".

> Hope this was entertaining and maybe even helpful,

Someday, someone will collect, catalog and publish all
these anecdotes so we can relive the chuckles in our
"declining years"  :>

On Friday, July 15, 2011 4:41:47 PM UTC-4, Don Y wrote:
> I don't understand.  Are there two different criteria at
> play, here (cost and failure rate)?

Bonus metric was simple failure rate.
Not core memory, I'm not that ancient.

Manufacturing metric cost excluded labor,
because "labor was same for all boards".

Em 15/7/2011 14:12, Mel escreveu:
> Cesar Rabak wrote:
> [ ... ]
>> If the teams feel 'rushed' is either because they don't have the
>> intellectual background to defend in robust way that the pace asked from
>> them is unattainable or because the 'work done' metrics takes only one
>> side of the equation and doesn't consider the others like already
>> mentioned elsewhere in the thread, like maintainability attributes,
>> testing results, quality, etc.
>
> It may not be intellectual background.  Alpha-dominance has a huge amount to
> do with management.
>
> 	Mel.
I think is something else. Please refer to my longer reply to Don's post.

-- 
Cesar Rabak
GNU/Linux User 52247.
Get counted: http://counter.li.org/

On Friday, July 15, 2011 3:41:12 PM UTC-4, Paul E. Bennett wrote:
> I think it might be better for them to read "Better Embedded Systems 
> Software" by Phil Koopman. Highly recommended for every developers desk and 
> all management conference tables (open at all chapters simultaneously by 
> preference).

On my desk as well, but in the few pages he devotes to
metrics the theme seems to be "this is a slippery slope"
and "you have to compare against yourself"...

Best Regards, Dave

Don Y wrote:

> This brings me back to my initial post:  *what* to track and
> *why* to track it (acknowledging how easily it is for "metrics for
> the sake of metrics" to lead one astray).

My process, through the four forms and register that tracks the development 
and forms the audit trail, provides numbers on the function points in the 
system, the number of errors or issues raised, the number of errors or 
issues corrected or dealt with, and the time taken for each one. I do not 
worry about counting LOC as it is not really that meaningful in Forth.

There is no real effort expended in collecting that data as it falls out of 
properly apply my process. A paper I gave at one of the Safety Systems 
Symposia has a description of my process (proceedings published by Springer-
Verlag). The core of the process is applicable at all levels and on all 
technologies involved in a project and meshes hierarchically throughout. The 
only other aspect of the process is knowing the documentation that needs to 
be produced.

-- 
********************************************************************
Paul E. Bennett...............<email://Paul_E.Bennett@topmail.co.uk>
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-510979
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************

Dave Nadler wrote:

> On Friday, July 15, 2011 3:41:12 PM UTC-4, Paul E. Bennett wrote:
>> I think it might be better for them to read "Better Embedded Systems
>> Software" by Phil Koopman. Highly recommended for every developers desk
>> and all management conference tables (open at all chapters simultaneously
>> by preference).
> 
> On my desk as well, but in the few pages he devotes to
> metrics the theme seems to be "this is a slippery slope"
> and "you have to compare against yourself"...
> 
> Best Regards, Dave

He does mean that it is most valid within the organisation and not across 
different organisations. It is a slippery slope if the wrong sort of metrics 
are gathered. However, keeping metrics on sensible facets of the development 
process will help in current and future development management, especially 
if you have a post-mortem at the end of each development to see what went 
right and what was wrong.

-- 
********************************************************************
Paul E. Bennett...............<email://Paul_E.Bennett@topmail.co.uk>
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-510979
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************

Em 15/7/2011 14:54, Don Y escreveu:
> Hi Cesar,
>
> [*much* elided as there is a lot of overlap with other posts]
>
> On 7/15/2011 9:33 AM, Cesar Rabak wrote:
>
>>> Yet the beans that are counted are often misleading, illusory, or flat
>>> out delusional. Which would you rather have: a product that costs $1M to
>>> develop, breaks in the field, alienates customers, and leads to lost
>>> sales for years, or a product that costs $5M to develop, works correctly
>>> and well straight out of the chute, and saves your marketing budget
>>> because your advertising becomes word-of-mouth?
>>
>> If you don't have clear ways to demonstrate the later, the risk of
>> expending 400% more in the project would make it very hard to be
>> approved.
>>
>> We have to break the vicious circle of the delusional measures and offer
>> the good ones that make sense in the business and technical realms.
>
> I think folks in 9-to-5's have little recourse, here. They are
> at the mercy of their managers (who are at the mercy of *their*
> managers, etc.). It doesn't matter how accurate your assessment
> of a project is if the higher-ups refuse to be bound by physical
> laws. :>
>
This way of thinking is akin to paralisis...

> Even working freelance (with a lot of lattitude as to what jobs I
> am willing to undertake), you are still pressured by having to pay
> the bills, etc. Clients don't like it when you say "No (it can't
> be done for that money/time/size/etc.)".

But you end up having to say it, isn't it?

>
> People *know* "where babies come from" -- so why are there *any*
> "unplanned pregnancies"? :>
>

This is non sequitur to our conversation. The answer is 'by the same 
reason' too many people drink and drive? Or do drugs? Or dare to do 
'stunt' like maneuvers in Youtube?

> The "Just Say No" type of thinking fails to acknowledge Reality.

If "Reality" is not ingrained in the framework of thinking of the 
person, the other side of the consequences of you point of view applies 
as well.

It is about this instill process on the gathering and *correct* use of 
metrics that we have to put to work and make these instruments part of 
the correct perception of the Reality.

>
> Having said all that, there is nothing that prevents you AS AN
> INDIVIDUAL from benefiting from tracking these sorts of metrics
> on your own (there are tools to do so for most of them) and using
> them to better understand *you* "process".

Yes. See my comment on this on another reply to another post of yours.


-- 
Cesar Rabak
GNU/Linux User 52247.
Get counted: http://counter.li.org/

Hi Cesar,

On 7/16/2011 10:13 AM, Cesar Rabak wrote:

>>>> Yet the beans that are counted are often misleading, illusory, or flat
>>>> out delusional. Which would you rather have: a product that costs
>>>> $1M to
>>>> develop, breaks in the field, alienates customers, and leads to lost
>>>> sales for years, or a product that costs $5M to develop, works
>>>> correctly
>>>> and well straight out of the chute, and saves your marketing budget
>>>> because your advertising becomes word-of-mouth?
>>>
>>> If you don't have clear ways to demonstrate the later, the risk of
>>> expending 400% more in the project would make it very hard to be
>>> approved.
>>>
>>> We have to break the vicious circle of the delusional measures and offer
>>> the good ones that make sense in the business and technical realms.
>>
>> I think folks in 9-to-5's have little recourse, here. They are
>> at the mercy of their managers (who are at the mercy of *their*
>> managers, etc.). It doesn't matter how accurate your assessment
>> of a project is if the higher-ups refuse to be bound by physical
>> laws. :>
>>
> This way of thinking is akin to paralisis...

But that (IMHO) is The Way It Is.  An "employee", when faced with
a PHB who refuses to face reality, has only one avenue of recourse:
to quit and find a new employer who (hopefully) isn't as delusional.

>> Even working freelance (with a lot of lattitude as to what jobs I
>> am willing to undertake), you are still pressured by having to pay
>> the bills, etc. Clients don't like it when you say "No (it can't
>> be done for that money/time/size/etc.)".
>
> But you end up having to say it, isn't it?

The difference, there, is that you *can* say "I told you so" when
Reality bears witness to your assertions.  A smart client will
learn from that experience.  A foolish client won't -- in which
case, you "move on".

>> People *know* "where babies come from" -- so why are there *any*
>> "unplanned pregnancies"? :>
>
> This is non sequitur to our conversation. The answer is 'by the same
> reason' too many people drink and drive? Or do drugs? Or dare to do
> 'stunt' like maneuvers in Youtube?

*Think* about the answers to the questions you posed (as well as
my analogy).  Despite overwhelming evidence (and, often, *explicit*
acknowledgement of the problems that a "worker" points out "ahead of
time") that "you guys are going to make a BIG mistake if you ignore
what Time and Experience are telling you", why *do* organizations
persist in this foolhardy behavior?

Do they think that *next* time Reality will be *different*?
How many times do you have to shoot yourself in the foot before
you realize you should MOVE YOUR FOOT??

>> The "Just Say No" type of thinking fails to acknowledge Reality.
>
> If "Reality" is not ingrained in the framework of thinking of the
> person, the other side of the consequences of you point of view applies
> as well.
>
> It is about this instill process on the gathering and *correct* use of
> metrics that we have to put to work and make these instruments part of
> the correct perception of the Reality.
>
>> Having said all that, there is nothing that prevents you AS AN
>> INDIVIDUAL from benefiting from tracking these sorts of metrics
>> on your own (there are tools to do so for most of them) and using
>> them to better understand *you* "process".
>
> Yes. See my comment on this on another reply to another post of yours.

Hi Paul,

On 7/16/2011 3:55 AM, Paul E. Bennett wrote:
> Don Y wrote:
>
>> This brings me back to my initial post:  *what* to track and
>> *why* to track it (acknowledging how easily it is for "metrics for
>> the sake of metrics" to lead one astray).
>
> My process, through the four forms and register that tracks the development
> and forms the audit trail, provides numbers on the function points in the
> system, the number of errors or issues raised, the number of errors or
> issues corrected or dealt with, and the time taken for each one. I do not
> worry about counting LOC as it is not really that meaningful in Forth.

I imagine computing the FP metric is relatively straight-forward?
"Words" = operators (and ins and outs are easily enumerated)?

> There is no real effort expended in collecting that data as it falls out of
> properly apply my process. A paper I gave at one of the Safety Systems
> Symposia has a description of my process (proceedings published by Springer-
> Verlag). The core of the process is applicable at all levels and on all
> technologies involved in a project and meshes hierarchically throughout. The
> only other aspect of the process is knowing the documentation that needs to
> be produced.

<frown>  I tried googling last night but nothing turned up.
(though I did happen across a photo of you? riding a unicycle
with a chimpanzee atop your shoulders beating a toy drum  :> )

Do you have a pointer to a {PDF,PS} -- or, a copy you can
email to me?

Thx,
--don

On Sun, 17 Jul 2011 09:45:33 -0700, Don Y <nowhere@here.com>
wrote:

>><snip>
>> There is no real effort expended in collecting that data as it falls out of
>> properly apply my process. A paper I gave at one of the Safety Systems
>> Symposia has a description of my process (proceedings published by Springer-
>> Verlag). The core of the process is applicable at all levels and on all
>> technologies involved in a project and meshes hierarchically throughout. The
>> only other aspect of the process is knowing the documentation that needs to
>> be produced.
>
><frown>  I tried googling last night but nothing turned up.
>(though I did happen across a photo of you? riding a unicycle
>with a chimpanzee atop your shoulders beating a toy drum  :> )
>
>Do you have a pointer to a {PDF,PS} -- or, a copy you can
>email to me?

I might ask for a copy, as well.  So a link would be nice.

Jon

P.S.
However, it is Springer-Verlag.  Unless draft rights were
retained or a slightly different version made, he may not
retain the rights to do more than send a copy on request.