I'd like to compare C compilers targetting embedded systems - the kind
that only have a few KB of RAM, etc.

However there seems to be a lack of software I could use. Waht I'm
looking for should be free, written (mostly) in standard C, be not
totally different from typical embedded applications.

So far I see:
- dhrystone
- The Contiki OS
- the files I've been using to track sdcc over revisions at
https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list

Are there any other suitable benchmarks I could look at? They don't have
to be designed as benchmarks, other software, like the Contiki OS
mentioned above is OK, too. THey don't even have to "run", so libraries
would be OK, too.

Philipp

On 19/09/11 19:43, Philipp Klaus Krause wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions at
> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.
>
> Philipp

There are no good general benchmarks for small embedded systems - partly 
because it would be a great deal of work (standard C is not standard for 
many compilers - you need to use compiler-specific features to get the 
best out of them), and partly because it would be of very little use.

What you want to know is which tools are the best for /your/ code, not 
some general code written for completely different uses.  Most tool 
vendors provide free evaluation software - test them out on your own code.

On Sep 19, 10:43 pm, Philipp Klaus Krause <p...@spth.de> wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions athttps://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.
>
> Philipp

Hi Philipp,

Have you tried Coremark from EEMBC (www.coremark.org)?
It is available for free.
The other benchmarking suites from EEMBC require you to pay small
membership fee i guess.

Regards,
Abhishek

Am 20.09.2011 09:50, schrieb Abhishek:

> 
> Have you tried Coremark from EEMBC (www.coremark.org)?
> It is available for free.
> The other benchmarking suites from EEMBC require you to pay small
> membership fee i guess.

I now did, however the license is a problem:
1) The license. I can understand when benchmarking organizations don't
distribute older versions of their benchmarks and no longer accept
reults generated using an old version, like SPEC does. However the
Coremark license explicitly forbids you from just running or even
keeping a copy of older versions: You loose all rights to them each time
a new version is released.

I'll probably use it anyway and just hope they don't release a new
version anytime soon. Still, I'm looking for further benchmarks. The
Contiki OS seems perfect, but I'd like to find more of that kind.

Philipp

On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
> Am 20.09.2011 09:50, schrieb Abhishek:
>
>>
>> Have you tried Coremark from EEMBC (www.coremark.org)?
>> It is available for free.
>> The other benchmarking suites from EEMBC require you to pay small
>> membership fee i guess.
>
> I now did, however the license is a problem:
> 1) The license. I can understand when benchmarking organizations don't
> distribute older versions of their benchmarks and no longer accept
> reults generated using an old version, like SPEC does. However the
> Coremark license explicitly forbids you from just running or even
> keeping a copy of older versions: You loose all rights to them each time
> a new version is released.
>
> I'll probably use it anyway and just hope they don't release a new
> version anytime soon. Still, I'm looking for further benchmarks. The
> Contiki OS seems perfect, but I'd like to find more of that kind.

What about your own code?

-- 
Ian Collins

Am 21.09.2011 11:06, schrieb Ian Collins:
> On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
>> Am 20.09.2011 09:50, schrieb Abhishek:
>>
>>>
>>> Have you tried Coremark from EEMBC (www.coremark.org)?
>>> It is available for free.
>>> The other benchmarking suites from EEMBC require you to pay small
>>> membership fee i guess.
>>
>> I now did, however the license is a problem:
>> 1) The license. I can understand when benchmarking organizations don't
>> distribute older versions of their benchmarks and no longer accept
>> reults generated using an old version, like SPEC does. However the
>> Coremark license explicitly forbids you from just running or even
>> keeping a copy of older versions: You loose all rights to them each time
>> a new version is released.
>>
>> I'll probably use it anyway and just hope they don't release a new
>> version anytime soon. Still, I'm looking for further benchmarks. The
>> Contiki OS seems perfect, but I'd like to find more of that kind.
> 
> What about your own code?
> 

I'd like to get results that are useful to other people as well. I
intend to use some of my code as one of the benchmarks benchmark, but in
general I'd prefer to use more generally accepted stuff.

Philipp

Am 19.09.2011 22:10, schrieb David Brown:
> On 19/09/11 19:43, Philipp Klaus Krause wrote:
>> I'd like to compare C compilers targetting embedded systems - the kind
>> that only have a few KB of RAM, etc.
>>
>> However there seems to be a lack of software I could use. Waht I'm
>> looking for should be free, written (mostly) in standard C, be not
>> totally different from typical embedded applications.
>>
>> So far I see:
>> - dhrystone
>> - The Contiki OS
>> - the files I've been using to track sdcc over revisions at
>> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>>
>> Are there any other suitable benchmarks I could look at? They don't have
>> to be designed as benchmarks, other software, like the Contiki OS
>> mentioned above is OK, too. THey don't even have to "run", so libraries
>> would be OK, too.
>>
>> Philipp
> 
> There are no good general benchmarks for small embedded systems - partly
> because it would be a great deal of work (standard C is not standard for
> many compilers - you need to use compiler-specific features to get the
> best out of them), and partly because it would be of very little use.
> 
> What you want to know is which tools are the best for /your/ code, not
> some general code written for completely different uses.  Most tool
> vendors provide free evaluation software - test them out on your own code.
> 

I don't just want to know which tool is best for my code. I want to
know, which tool is best for typical code.
I am one of the developers of the free sdcc compiler and currently
working on register and stack allcoation. I see that what I implemented
results in better code when compiling my own programs, Contiki,
dhrystone, Coremark. But I'd like to know if there is any code for which
the new approach performs worse than the on epreviously used in sdcc.
And I'd like to know it know, instead of having to wait for bug reports
about code size regressions coming in after the next release.

Philipp

On 09/22/11 07:12 PM, Philipp Klaus Krause wrote:
> Am 21.09.2011 11:06, schrieb Ian Collins:
>> On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
>>> Am 20.09.2011 09:50, schrieb Abhishek:
>>>
>>>>
>>>> Have you tried Coremark from EEMBC (www.coremark.org)?
>>>> It is available for free.
>>>> The other benchmarking suites from EEMBC require you to pay small
>>>> membership fee i guess.
>>>
>>> I now did, however the license is a problem:
>>> 1) The license. I can understand when benchmarking organizations don't
>>> distribute older versions of their benchmarks and no longer accept
>>> reults generated using an old version, like SPEC does. However the
>>> Coremark license explicitly forbids you from just running or even
>>> keeping a copy of older versions: You loose all rights to them each time
>>> a new version is released.
>>>
>>> I'll probably use it anyway and just hope they don't release a new
>>> version anytime soon. Still, I'm looking for further benchmarks. The
>>> Contiki OS seems perfect, but I'd like to find more of that kind.
>>
>> What about your own code?
>>
>
> I'd like to get results that are useful to other people as well. I
> intend to use some of my code as one of the benchmarks benchmark, but in
> general I'd prefer to use more generally accepted stuff.

Every target and code base combination will behave differently, even on 
different members of the same processor family.  This makes general 
benchmarks useless for embedded systems.

-- 
Ian Collins

On 22/09/2011 09:16, Philipp Klaus Krause wrote:
> Am 19.09.2011 22:10, schrieb David Brown:
>> On 19/09/11 19:43, Philipp Klaus Krause wrote:
>>> I'd like to compare C compilers targetting embedded systems - the kind
>>> that only have a few KB of RAM, etc.
>>>
>>> However there seems to be a lack of software I could use. Waht I'm
>>> looking for should be free, written (mostly) in standard C, be not
>>> totally different from typical embedded applications.
>>>
>>> So far I see:
>>> - dhrystone
>>> - The Contiki OS
>>> - the files I've been using to track sdcc over revisions at
>>> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>>>
>>> Are there any other suitable benchmarks I could look at? They don't have
>>> to be designed as benchmarks, other software, like the Contiki OS
>>> mentioned above is OK, too. THey don't even have to "run", so libraries
>>> would be OK, too.
>>>
>>> Philipp
>>
>> There are no good general benchmarks for small embedded systems - partly
>> because it would be a great deal of work (standard C is not standard for
>> many compilers - you need to use compiler-specific features to get the
>> best out of them), and partly because it would be of very little use.
>>
>> What you want to know is which tools are the best for /your/ code, not
>> some general code written for completely different uses.  Most tool
>> vendors provide free evaluation software - test them out on your own code.
>>
>
> I don't just want to know which tool is best for my code. I want to
> know, which tool is best for typical code.
> I am one of the developers of the free sdcc compiler and currently
> working on register and stack allcoation. I see that what I implemented
> results in better code when compiling my own programs, Contiki,
> dhrystone, Coremark. But I'd like to know if there is any code for which
> the new approach performs worse than the on epreviously used in sdcc.
> And I'd like to know it know, instead of having to wait for bug reports
> about code size regressions coming in after the next release.
>

I understand what you are trying to achieve - I just don't think it is 
possible.  There is no such thing as "typcial code" for embedded 
systems, and often "benchmark code" is so artificial that it doesn't 
represent the real world (though it can be useful for identifying 
particular cases).

What you are doing now, testing with a sample of your own real-world 
programs and other real-world embedded code (Contiki), is the best you 
can do.  If you want to do more, then collect more samples in a similar 
vein.  If sdcc has a collection of example code, then that would be a 
good start - you may also find that some users will give you programs 
you can use.  The toughest test is real-world code that is written for 
portability - good examples there would be FreeRTOS and LWIP.  These are 
code bases that are written to work on a wide range of targets, and 
can't take advantage of any target or compiler-specific tricks and 
optimisations, and they are projects that are used by a great many people.

On 09/19/2011 07:43 PM, Philipp Klaus Krause wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions at
> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.

Elm's FAT filesystem implementation:

http://elm-chan.org/fsw/ff/00index_e.html