A comparison of C implementations for the STM8

The STM8 is an 8-bit microcontroller architecture targeted by varioues C
implementations.
Du to the upcoming SDCC 3.6.0 release, I tried to do a quick comparison
of them:

http://www.colecovision.eu/stm8/compilers.shtml

Suggestions for improvements are welcome.

Philipp

On Wed, 25 May 2016 21:20:40 +0200, Philipp Klaus Krause wrote:

> The STM8 is an 8-bit microcontroller architecture targeted by varioues C
> implementations.
> Du to the upcoming SDCC 3.6.0 release, I tried to do a quick comparison
> of them:
> 
> http://www.colecovision.eu/stm8/compilers.shtml
> 
> Suggestions for improvements are welcome.

Thanks for the work.  It's interesting, disappointing, and good to know 
that none of them support double-precision floating point.  Supporting it 
shouldn't be a huge increment in the effort of writing the tools, and 
even on an 8-bitter there's sometimes reason to go ahead and use the 
processing time it takes to do double-precision math.

-- 
Tim Wescott
Control systems, embedded software and circuit design
I'm looking for work!  See my website if you're interested
http://www.wescottdesign.com

On 5/25/2016 12:20 PM, Philipp Klaus Krause wrote:
> The STM8 is an 8-bit microcontroller architecture targeted by varioues C
> implementations.
> Du to the upcoming SDCC 3.6.0 release, I tried to do a quick comparison
> of them:
>
> http://www.colecovision.eu/stm8/compilers.shtml
>
> Suggestions for improvements are welcome.

As only a *cursory* read (shopping day, today):
- any extensions (nonstandard) that a developer might find useful
- add some whitespace after the final table (e.g., a footer)
- opine as to why speed/size tradeoffs seem insignificant
- yellow on white is kinda brutal on the eyes (red, green, blue?)
- quality of libraries delivered with each
- "invent" some bit-oriented benchmarks (toggling an "I/O", etc.)
- library source availability (rebuild for different optimization)
- why *you* choose to use SDCC ("honestly")

I'll try to give it a more thorough review after my wallet has
been emptied...

On 5/25/2016 12:39 PM, Don Y wrote:
> On 5/25/2016 12:20 PM, Philipp Klaus Krause wrote:
>> The STM8 is an 8-bit microcontroller architecture targeted by varioues C
>> implementations.

>> http://www.colecovision.eu/stm8/compilers.shtml
>
> As only a *cursory* read (shopping day, today):
> - any extensions (nonstandard) that a developer might find useful
> - add some whitespace after the final table (e.g., a footer)
> - opine as to why speed/size tradeoffs seem insignificant
> - yellow on white is kinda brutal on the eyes (red, green, blue?)
> - quality of libraries delivered with each
> - "invent" some bit-oriented benchmarks (toggling an "I/O", etc.)
> - library source availability (rebuild for different optimization)
> - why *you* choose to use SDCC ("honestly")
>
> I'll try to give it a more thorough review after my wallet has
> been emptied...

Please don't take this as being overly critical; just pointing out
questions that an engaged reader could come away asking himself.
Keep or discard as much or as little as you choose...

First, I think you need to ask yourself WHY you are doing this as
that can color *how* you are doing it.  I.e., if you are evangelizing for
SDCC, then you might want to be disproportionately more critical (to
yourself) of things you say *for* SDCC (to ensure your bias isn't
coloring the questions you are asking).

[OTOH, there's nothing wrong with pointing out a product's strengths,
if they truly are!]

Your appraisal of standards compliance seems very subjective.  What
exactly does "far ahead of the competition" mean?  Number of features
correctly implemented?  How do you decide which of those features
are important vs. others that aren't supported?

Is it practical to tabulate the differences in UNSUPPORTED features
for each compiler?  (i.e., omit the stuff they *do* and highlight
the things they *don't* -- or do "nonconformingly")

    "Some compilers instead offer non-standard versions of these
    types with reduced precision..."

You've got a table, there.  Why not indicate WHICH compilers?
(i.e., does nonstandard appear as a "yes" or as a "no"?  How
do I differentiate between a "NONSTANDARD no" and a "MISSING no")

Presumably, enumerations and bitfields are supported by each?
Or, do those have to be preprocessor hacks?

[Speaking of "preprocessor", differences between their implementations?]

In your Code Quality assessment, you claim:
    "the floating-point parts, especially C library functions, dominate
    the benchmark"
But, given that you've just previously claimed this support is tenuous
among the different compilers, how do I determine what the consequences
of those implementations are in your assessment?    I.e., if it is
a "hacked" data type it could make the results look better *or* worse!

You note that:
    "for the others, they were modified as necessary to make them compile"
and, later:
    "Cosmic generates wrong code for Dhrystone"
I.e., was this a consequence of your "modifications"?  (perhaps incorrect)

You might elaborate on the sort of modifications that were necessary
given that these are all trying to be "standards compliant" (at least
for your comparison).  I.e., syntax errors?  data sizes?  etc.

Your statement:
     "Most software for embedded systems does not use floating-point arithmetic"
might be misleading.  Often, something isn't used simply because it isn't
PROVIDED in an efficient manner.  (where "efficient" is defined by the
application's constraints).  I.e., replacing a floating point operation
with one using fixed point on larger (wider) data types might be a
valid comparison in many implementations.

The Summary table is highly subjective.  And, you might want to move to
a different representation of "score":  "-" suggests incomplete/inadequate
whereas "one star" vs. "five stars" might be a better relative assessment.
(likewise the bias introduced by red == fail, green == pass)

You might also want to comment on tweeks that are offered in the libraries
available (e.g., trimmed down printf()'s, how/if the file oriented operations
are addressed -- in systems that don't have "files", support for reentrancy,
how clean the startup code is, etc.)

[BTW, I assume your benchmark's discount or normalize any crt0 costs
in each of the above metrics?]

Rock on!

(This reply is mainly for the OP - I am just replying to Don's post
because his post provides convenient spots for my comments to add to his
useful points.)

On 26/05/16 01:54, Don Y wrote:
> On 5/25/2016 12:39 PM, Don Y wrote:
>> On 5/25/2016 12:20 PM, Philipp Klaus Krause wrote:
>>> The STM8 is an 8-bit microcontroller architecture targeted by varioues C
>>> implementations.
> 
>>> http://www.colecovision.eu/stm8/compilers.shtml
>>
>> As only a *cursory* read (shopping day, today):
>> - any extensions (nonstandard) that a developer might find useful
>> - add some whitespace after the final table (e.g., a footer)
>> - opine as to why speed/size tradeoffs seem insignificant
>> - yellow on white is kinda brutal on the eyes (red, green, blue?)
>> - quality of libraries delivered with each
>> - "invent" some bit-oriented benchmarks (toggling an "I/O", etc.)
>> - library source availability (rebuild for different optimization)
>> - why *you* choose to use SDCC ("honestly")
>>
>> I'll try to give it a more thorough review after my wallet has
>> been emptied...
> 
> Please don't take this as being overly critical; just pointing out
> questions that an engaged reader could come away asking himself.
> Keep or discard as much or as little as you choose...
> 
> First, I think you need to ask yourself WHY you are doing this as
> that can color *how* you are doing it.  I.e., if you are evangelizing for
> SDCC, then you might want to be disproportionately more critical (to
> yourself) of things you say *for* SDCC (to ensure your bias isn't
> coloring the questions you are asking).

Good points.

Also consider how much effort you want to put into this.  How many
people are actually going to look at your page?  Many (such as me) are
just curious, but it is only really relevant for people who are picking
STM8 devices for a new project and want guidance in choosing tools -
that is not a big market these days.  If you were looking at ARM
compilers, or at least AVR or PIC tools, it would be different.  So
treat suggestions here with that in view - spend the effort if you are
enjoying the investigation, but feel entirely free to tell me or Don
that our ideas are nice but you don't want to spend the time on it.

> 
> [OTOH, there's nothing wrong with pointing out a product's strengths,
> if they truly are!]
> 
> Your appraisal of standards compliance seems very subjective.  What
> exactly does "far ahead of the competition" mean?  Number of features
> correctly implemented?  How do you decide which of those features
> are important vs. others that aren't supported?
> 
> Is it practical to tabulate the differences in UNSUPPORTED features
> for each compiler?  (i.e., omit the stuff they *do* and highlight
> the things they *don't* -- or do "nonconformingly")
> 

Standards compliance is important to some people, but for many small
embedded projects, code is written entirely for the one device and one
compiler.  It is a lot more useful to know that a compiler has an extra
feature for efficient handling of arrays of bits, than to know if it has
full C99 support (including, for example, wide character functions).

So a table with information based on features rather than just standards
would be useful (though more work, of course).  A rough figure (or set
of up to five stars as Don suggested) for how much of the C standards
are covered could be nice.  But it would also be useful with rows for
things like inline functions, mixed declarations and data,
recursion/re-entrant functions, true boolean types, data in flash,
interrupt support, inline assembly support, intrinsic functions, etc.

What about library features, device header files, code samples, etc.?

Are there any cases of incorrect code generation?


>    "Some compilers instead offer non-standard versions of these
>    types with reduced precision..."
> 
> You've got a table, there.  Why not indicate WHICH compilers?
> (i.e., does nonstandard appear as a "yes" or as a "no"?  How
> do I differentiate between a "NONSTANDARD no" and a "MISSING no")
> 
> Presumably, enumerations and bitfields are supported by each?
> Or, do those have to be preprocessor hacks?
> 
> [Speaking of "preprocessor", differences between their implementations?]
> 
> In your Code Quality assessment, you claim:
>    "the floating-point parts, especially C library functions, dominate
>    the benchmark"
> But, given that you've just previously claimed this support is tenuous
> among the different compilers, how do I determine what the consequences
> of those implementations are in your assessment?    I.e., if it is
> a "hacked" data type it could make the results look better *or* worse!
> 
> You note that:
>    "for the others, they were modified as necessary to make them compile"
> and, later:
>    "Cosmic generates wrong code for Dhrystone"
> I.e., was this a consequence of your "modifications"?  (perhaps incorrect)

As I understand it, Dhrystone itself has bugs and undefined behaviour.
(That's not uncommon - a SPEC benchmark was optimised to an empty loop
by a development version of gcc, due to its undefined behaviour.)

Generally speaking, Dhrystone and other such benchmarks written decades
ago for mainframes and minicomputers are completely useless as
comparisons in small microcontrollers.  The code is so different from
the sort of code that will be used on these devices, that it offers no
useful information.

I am not sure if there are easily available ready-made benchmark code
snippets that /are/ useful for such devices.  But I would be more
interested in microbenchmarks or even samples of generated code.  How
well does the compiler handle basic arithmetic?  What does it do for
samples like these:

uint8_t test1(uint8_t a, uint8_t b) {
	return (a * 3) + b;
}

uint32_t test2(uint16_t a, uint8_t b) {
	return ((uint32_t) a) * b;
}

Does it make good code for a switch?  If you write low-level  bit-banged
SPI routines, how fast can it run them (testing bit and I/O access,
loops, and perhaps inlining of pin on/off functions)?

> 
> You might elaborate on the sort of modifications that were necessary
> given that these are all trying to be "standards compliant" (at least
> for your comparison).  I.e., syntax errors?  data sizes?  etc.
> 
> Your statement:
>     "Most software for embedded systems does not use floating-point
> arithmetic"
> might be misleading.  Often, something isn't used simply because it isn't
> PROVIDED in an efficient manner.  (where "efficient" is defined by the
> application's constraints).  I.e., replacing a floating point operation
> with one using fixed point on larger (wider) data types might be a
> valid comparison in many implementations.
> 

It is certainly true that on most small embedded systems, floating point
is rare or non-existent, and its performance is a minor issue (because
you assume that it is ridiculously slow, and only use it when that is
okay).  But some benchmarks of a few basic operations can be a helpful
guide.

> The Summary table is highly subjective.  And, you might want to move to
> a different representation of "score":  "-" suggests incomplete/inadequate
> whereas "one star" vs. "five stars" might be a better relative assessment.
> (likewise the bias introduced by red == fail, green == pass)
> 
> You might also want to comment on tweeks that are offered in the libraries
> available (e.g., trimmed down printf()'s, how/if the file oriented
> operations
> are addressed -- in systems that don't have "files", support for
> reentrancy,
> how clean the startup code is, etc.)
> 
> [BTW, I assume your benchmark's discount or normalize any crt0 costs
> in each of the above metrics?]
> 
> Rock on!

On 25.05.2016 21:39, Don Y wrote:
> - yellow on white is kinda brutal on the eyes (red, green, blue?)

I used the colors that the compiler vendors use for themselves. IAR is
yellow. I'll try using a darker yellow first, but if that doesn't fix
the issue I might just choose whatever other color for the IAR bars.

Philipp

On 26.05.2016 01:54, Don Y wrote:
> […]
> Your appraisal of standards compliance seems very subjective.  What
> exactly does "far ahead of the competition" mean?  Number of features
> correctly implemented?  How do you decide which of those features
> are important vs. others that aren't supported?

I've now tried to clarify that part a bit. I consider SDCC and IAR far
ahead because neither Cosmic nor Raisonance make any effort to support
anything past C90 (well, theyhave //-comments but that's it).

> […]
> [Speaking of "preprocessor", differences between their implementations?]

I didn't notice any issues. But didn't really look into it either. I
didn't consider the preprocessor important enough to mention it in the
comparison (SDCC uses one derived from the gcc one but I never did any
advanced stuff with preprocessors).

> In your Code Quality assessment, you claim:
>    "the floating-point parts, especially C library functions, dominate
>    the benchmark"
> But, given that you've just previously claimed this support is tenuous
> among the different compilers, how do I determine what the consequences
> of those implementations are in your assessment?    I.e., if it is
> a "hacked" data type it could make the results look better *or* worse!

All four compilers replace the double in Whetstone by a 32-bit IEEE-like
floating point type that they also use for float. So this makes the
results incomparable to to other compilers, but they are still
comparable among the STM8-targeting compilers.

> You note that:
>    "for the others, they were modified as necessary to make them compile"
> and, later:
>    "Cosmic generates wrong code for Dhrystone"
> I.e., was this a consequence of your "modifications"?  (perhaps incorrect)
> 
> You might elaborate on the sort of modifications that were necessary
> given that these are all trying to be "standards compliant" (at least
> for your comparison).  I.e., syntax errors?  data sizes?  etc.

A few sentences have been added to clarify the nature of the changes and
that they shuldn't affect the correctness.

Philipp

Hi David,

On 5/26/2016 12:20 AM, David Brown wrote:
>> Please don't take this as being overly critical; just pointing out
>> questions that an engaged reader could come away asking himself.
>> Keep or discard as much or as little as you choose...
>>
>> First, I think you need to ask yourself WHY you are doing this as
>> that can color *how* you are doing it.  I.e., if you are evangelizing for
>> SDCC, then you might want to be disproportionately more critical (to
>> yourself) of things you say *for* SDCC (to ensure your bias isn't
>> coloring the questions you are asking).
>
> Good points.
>
> Also consider how much effort you want to put into this.  How many
> people are actually going to look at your page?  Many (such as me) are
> just curious, but it is only really relevant for people who are picking
> STM8 devices for a new project and want guidance in choosing tools -
> that is not a big market these days.

I look at it differently (motivation).  Assuming you're NOT doing it
"because it was an assignment" (i.e., rather, you are doing it because
you have some significant interest in the compiler/processor -- perhaps
because your shop uses it somewhat exclusively), then the effort
can be part of a learning experience.  And, that this can carry over to
developing analytical skills that could apply to similar activities
and evaluations.

For example, the fact that space/time optimizer tradeoffs didn't seem
significant would have me *wondering* why they weren't.  I.e., what was
it about the code that didn't lend itself to either optimization
strategy?  How could I avoid these "problems" in the way I craft *my*
code, going forward?

Admittedly, you can't just take some generic legacy code and determine how
it performs in *time* (as you might not have an appropriate test scaffold
in place or the code may not lend itself to that sort of profiling).  *But*,
you could run it through with the different optimizations enabled and see if
even the *space* changes -- and by how much.

If there are consequences to the use of a particular optimization strategy,
this could give you a metric to use in determining when/if you'd be willing
to avail yourself of that capability (e.g., if speed cause it to use
non-standard floats, this might be acceptable -- or UNacceptable!)

> If you were looking at ARM
> compilers, or at least AVR or PIC tools, it would be different.  So
> treat suggestions here with that in view - spend the effort if you are
> enjoying the investigation, but feel entirely free to tell me or Don
> that our ideas are nice but you don't want to spend the time on it.

Yes, hence my introductory paragraph, included above.

My comments, here, are a suggestion that you can also use this as a personal
learning experience to better understand the capabilities and limitations of
the tools at your disposal.  And, as the foundation for conducting similar
sorts of appraisals on other products going forward.  (as well as developing
a critical eye to be able to spot flaws in comparisons you might see others
performing to bias your opinion of <whatever>)

I.e., instead of as an aid for *others*, use it as an aid for oneself!

On 26.05.2016 09:20, David Brown wrote:
> Are there any cases of incorrect code generation?
> 

Apart from the Dhrystone Cosmic issue I didn't see any. I was surprised
by the non-standard bool of Raisonance, since casts to bool behave
differently than the standard. It might be a bug or intent, I didn't
find any documentation on it.
I didn't consider it feasible with limited time and resources to run a
big test suite through the compilers to get a comparison on code
correctness. We know there are some in SDCC (since it has a public bug
tracker), but I have no reason to assume that the other compilers are
any better or worse; they just don't publish the information.

Philipp

On 27.05.2016 10:59, Philipp Klaus Krause wrote:
> On 26.05.2016 09:20, David Brown wrote:
>> Are there any cases of incorrect code generation?
>>
> 
> Apart from the Dhrystone Cosmic issue I didn't see any.

It turned out that Cosmic does not do the promotion of arguments to int
on variable arguments that the standard requires. This resulted in the
Dhrystone self-test failing. I've updated the comparison accordingly.

Philipp