David Brown <david.brown@removethis.hesbynett.no> writes:

> On 17/07/12 19:13, John Devereux wrote:
>> David Brown<david.brown@removethis.hesbynett.no>  writes:
>>
>>> On 11/07/12 20:45, Walter Banks wrote:
>>>>
>>>>
>>>> Stefan Reuther wrote:
>>>>
>>>>> Walter Banks wrote:
>>>>>
>>>>>> Conformance testing is one way to define with confidence the
>>>>>> definition of a tool set. My surprise is that the FOSS community
>>>>>> has not developed conformance testing tools in part to the response
>>>>>> their cost. FOSS tools have been making the argument for free
>>>>>> tools for years. .
>>>>>
>>>>> gcc comes with an extensive test suite, why does this one not count?
>>>>>
>>>>> I am actually quite confident with it, simply because it has failed less
>>>>> often than "the others" on me.
>>>>>
>>>>
>>>> The GCC test suite is ad hoc, tests some things and misses others.
>>>> What it lacks is tests that can be directly linked back to specific
>>>> requirements. Tests that have names for example that go back to
>>>> standards reference numbers (section paragraph numbering).
>>>>
>>>> The GCC test suite is like a collection of regression tests that we
>>>> have built up over the years as part of our internal testing ,  from
>>>> customer support or just interesting code fragments that get
>>>> debated in news groups.
>>>>
>>>> Conformance tests should be written by someone that knows
>>>> the requirements but have no knowledge of the implementation
>>>>
>>>> Regards,
>>>>
>>>> Walter Banks
>>>>
>>>
>>> Different types of test suite are useful for different purposes.  And
>>> /no/ test suite is complete.  So if a commercial compiler developer
>>> says they use Plum Hall (or whatever) to test standards conformance,
>>> all you know is that they have run the test suite.  It doesn't tell
>>> you how much of the standards are really tested by Plum Hall (I'm sure
>>> they are good - but they cannot ever be perfect).  It might not tell
>>> you how much of Plum Hall the compiler passed or failed.  It won't
>>> tell you anything about the bugs in the compiler, or the quality of
>>> the generated code or the libraries.  It will tell you absolutely
>>> /nothing/ about the correctness of the compiler (or libraries, or code
>>> delivered with the toolchain) when you use different sets of compiler
>>> flags or options.
>>>
>>> Conformance tests tell you a bit about the toolchains standards
>>> conformance.  That can certainly be useful - but it is not a major
>>> part of being sure your tools generate correct object code from your
>>> source code.  And if you are particularly concerned about conformance
>>> tests, then you should probably invest in Plum Hall yourself and run
>>> them (or get a consultancy firm to do so) rather than just relying on
>>> the toolchain manufacturer.
>>>
>>> Of course, if you want conformance testing for gcc, and don't want to
>>> buy the tools yourself, you can always buy "commercial open source"
>>> toolchains such as provided by CodeSourcery / Mentor Graphics - they
>>> run their gcc builds through conformance testing tools, including Plum
>>> Hall.
>>
>> Hey David,
>>
>> Have you looked at ARMs distribution, perhaps as an alternative to
>> codesourcery now it has been taken over:
>>
>> <https://launchpad.net/gcc-arm-embedded>
>>
>> (Or is it the same thing?)
>>
>> John
>>
>
> That is an interesting link, and I'll look more into it later.  At the
> moment, it looks mostly like yet another source of pre-built gcc
> binaries - but having ARM behind it makes it stand out a bit.

That is what drew my attention too - interesting (and good) that they do
this considering they sell their own compiler too.

> There doesn't seem to be much activity there as yet.  And their
> testing appears to be just standard gcc testing, along with running
> some code on actual boards, in comparison to the additional testing
> (including conformance testing) done by Code Sourcery.  Still, it's
> good that ARM are doing this, and I know that ARM make a lot of
> contributions directly to gcc development.

They do seem to include some bug fixes on top of the FSF standard build
(as do CS I believe).

> There are several factors that make Code Sourcery different.  One
> obvious one is that they are a commercial company - they aim to get
> paid for their work, either directly (by subscriptions or consultancy
> fees) or indirectly (by making gcc better, more people will pay for
> subscriptions).  That means top-level professional support, and extra
> services such as extra testing.

> They also have a long history of working with gcc - if you look at the
> gcc mailing list archives or patch lists, you will see Code Sourcery
> email addresses all over the place stretching back for many years.  To
> the extent that gcc has such a thing, they have long been the
> "official" maintainers of a number of gcc ports including ARM, m68k,
> PPC and MIPS.

Absolutely, I have been aware of them in that role for a long time.

> Like many gcc and Code Sourcery fans, I was a bit concerned about how
> things would change once Mentor Graphics bought them.  Mentor is a
> much larger company, and one might fear that they would be trying to
> close off the tools, forcibly integrate them into large, expensive
> toolsets, and discourage Code Sourcery employees from doing "free"
> work such as contributing to gcc or helping people on mailing lists.
> However, I don't think there has been any of this - Code Sourcery have
> being doing as much or more of their "pure" gcc work, the zero-cost
> "lite" versions of the tools have clear place on the website along
> with the paid-for subscriptions, and there are new cheaper versions of
> the toolchains to reduce the entry price.  It looks to me like a very
> good match for everyone - Mentor provides a stability and "big" name,
> so that Code Sourcery can concentrate more on the technical stuff they
> are so good at, and all gcc users benefit.

Apart from "making" me sign up to their mailing list and receiving spam
for PCB CAD, I have seen no averse effects yet either.


-- 

John Devereux

On 17/07/12 19:13, John Devereux wrote:
> David Brown<david.brown@removethis.hesbynett.no>  writes:
>
>> On 11/07/12 20:45, Walter Banks wrote:
>>>
>>>
>>> Stefan Reuther wrote:
>>>
>>>> Walter Banks wrote:
>>>>
>>>>> Conformance testing is one way to define with confidence the
>>>>> definition of a tool set. My surprise is that the FOSS community
>>>>> has not developed conformance testing tools in part to the response
>>>>> their cost. FOSS tools have been making the argument for free
>>>>> tools for years. .
>>>>
>>>> gcc comes with an extensive test suite, why does this one not count?
>>>>
>>>> I am actually quite confident with it, simply because it has failed less
>>>> often than "the others" on me.
>>>>
>>>
>>> The GCC test suite is ad hoc, tests some things and misses others.
>>> What it lacks is tests that can be directly linked back to specific
>>> requirements. Tests that have names for example that go back to
>>> standards reference numbers (section paragraph numbering).
>>>
>>> The GCC test suite is like a collection of regression tests that we
>>> have built up over the years as part of our internal testing ,  from
>>> customer support or just interesting code fragments that get
>>> debated in news groups.
>>>
>>> Conformance tests should be written by someone that knows
>>> the requirements but have no knowledge of the implementation
>>>
>>> Regards,
>>>
>>> Walter Banks
>>>
>>
>> Different types of test suite are useful for different purposes.  And
>> /no/ test suite is complete.  So if a commercial compiler developer
>> says they use Plum Hall (or whatever) to test standards conformance,
>> all you know is that they have run the test suite.  It doesn't tell
>> you how much of the standards are really tested by Plum Hall (I'm sure
>> they are good - but they cannot ever be perfect).  It might not tell
>> you how much of Plum Hall the compiler passed or failed.  It won't
>> tell you anything about the bugs in the compiler, or the quality of
>> the generated code or the libraries.  It will tell you absolutely
>> /nothing/ about the correctness of the compiler (or libraries, or code
>> delivered with the toolchain) when you use different sets of compiler
>> flags or options.
>>
>> Conformance tests tell you a bit about the toolchains standards
>> conformance.  That can certainly be useful - but it is not a major
>> part of being sure your tools generate correct object code from your
>> source code.  And if you are particularly concerned about conformance
>> tests, then you should probably invest in Plum Hall yourself and run
>> them (or get a consultancy firm to do so) rather than just relying on
>> the toolchain manufacturer.
>>
>> Of course, if you want conformance testing for gcc, and don't want to
>> buy the tools yourself, you can always buy "commercial open source"
>> toolchains such as provided by CodeSourcery / Mentor Graphics - they
>> run their gcc builds through conformance testing tools, including Plum
>> Hall.
>
> Hey David,
>
> Have you looked at ARMs distribution, perhaps as an alternative to
> codesourcery now it has been taken over:
>
> <https://launchpad.net/gcc-arm-embedded>
>
> (Or is it the same thing?)
>
> John
>

That is an interesting link, and I'll look more into it later.  At the 
moment, it looks mostly like yet another source of pre-built gcc 
binaries - but having ARM behind it makes it stand out a bit.  There 
doesn't seem to be much activity there as yet.  And their testing 
appears to be just standard gcc testing, along with running some code on 
actual boards, in comparison to the additional testing (including 
conformance testing) done by Code Sourcery.  Still, it's good that ARM 
are doing this, and I know that ARM make a lot of contributions directly 
to gcc development.

There are several factors that make Code Sourcery different.  One 
obvious one is that they are a commercial company - they aim to get paid 
for their work, either directly (by subscriptions or consultancy fees) 
or indirectly (by making gcc better, more people will pay for 
subscriptions).  That means top-level professional support, and extra 
services such as extra testing.

They also have a long history of working with gcc - if you look at the 
gcc mailing list archives or patch lists, you will see Code Sourcery 
email addresses all over the place stretching back for many years.  To 
the extent that gcc has such a thing, they have long been the "official" 
maintainers of a number of gcc ports including ARM, m68k, PPC and MIPS.

Like many gcc and Code Sourcery fans, I was a bit concerned about how 
things would change once Mentor Graphics bought them.  Mentor is a much 
larger company, and one might fear that they would be trying to close 
off the tools, forcibly integrate them into large, expensive toolsets, 
and discourage Code Sourcery employees from doing "free" work such as 
contributing to gcc or helping people on mailing lists.  However, I 
don't think there has been any of this - Code Sourcery have being doing 
as much or more of their "pure" gcc work, the zero-cost "lite" versions 
of the tools have clear place on the website along with the paid-for 
subscriptions, and there are new cheaper versions of the toolchains to 
reduce the entry price.  It looks to me like a very good match for 
everyone - Mentor provides a stability and "big" name, so that Code 
Sourcery can concentrate more on the technical stuff they are so good 
at, and all gcc users benefit.

David Brown <david.brown@removethis.hesbynett.no> writes:

> On 11/07/12 20:45, Walter Banks wrote:
>>
>>
>> Stefan Reuther wrote:
>>
>>> Walter Banks wrote:
>>>
>>>> Conformance testing is one way to define with confidence the
>>>> definition of a tool set. My surprise is that the FOSS community
>>>> has not developed conformance testing tools in part to the response
>>>> their cost. FOSS tools have been making the argument for free
>>>> tools for years. .
>>>
>>> gcc comes with an extensive test suite, why does this one not count?
>>>
>>> I am actually quite confident with it, simply because it has failed less
>>> often than "the others" on me.
>>>
>>
>> The GCC test suite is ad hoc, tests some things and misses others.
>> What it lacks is tests that can be directly linked back to specific
>> requirements. Tests that have names for example that go back to
>> standards reference numbers (section paragraph numbering).
>>
>> The GCC test suite is like a collection of regression tests that we
>> have built up over the years as part of our internal testing ,  from
>> customer support or just interesting code fragments that get
>> debated in news groups.
>>
>> Conformance tests should be written by someone that knows
>> the requirements but have no knowledge of the implementation
>>
>> Regards,
>>
>> Walter Banks
>>
>
> Different types of test suite are useful for different purposes.  And
> /no/ test suite is complete.  So if a commercial compiler developer
> says they use Plum Hall (or whatever) to test standards conformance,
> all you know is that they have run the test suite.  It doesn't tell
> you how much of the standards are really tested by Plum Hall (I'm sure
> they are good - but they cannot ever be perfect).  It might not tell
> you how much of Plum Hall the compiler passed or failed.  It won't
> tell you anything about the bugs in the compiler, or the quality of
> the generated code or the libraries.  It will tell you absolutely
> /nothing/ about the correctness of the compiler (or libraries, or code
> delivered with the toolchain) when you use different sets of compiler
> flags or options.
>
> Conformance tests tell you a bit about the toolchains standards
> conformance.  That can certainly be useful - but it is not a major
> part of being sure your tools generate correct object code from your
> source code.  And if you are particularly concerned about conformance
> tests, then you should probably invest in Plum Hall yourself and run
> them (or get a consultancy firm to do so) rather than just relying on
> the toolchain manufacturer.
>
> Of course, if you want conformance testing for gcc, and don't want to
> buy the tools yourself, you can always buy "commercial open source"
> toolchains such as provided by CodeSourcery / Mentor Graphics - they
> run their gcc builds through conformance testing tools, including Plum
> Hall.

Hey David,

Have you looked at ARMs distribution, perhaps as an alternative to
codesourcery now it has been taken over:

<https://launchpad.net/gcc-arm-embedded>

(Or is it the same thing?)

John


-- 

John Devereux

On 11/07/12 20:45, Walter Banks wrote:
>
>
> Stefan Reuther wrote:
>
>> Walter Banks wrote:
>>
>>> Conformance testing is one way to define with confidence the
>>> definition of a tool set. My surprise is that the FOSS community
>>> has not developed conformance testing tools in part to the response
>>> their cost. FOSS tools have been making the argument for free
>>> tools for years. .
>>
>> gcc comes with an extensive test suite, why does this one not count?
>>
>> I am actually quite confident with it, simply because it has failed less
>> often than "the others" on me.
>>
>
> The GCC test suite is ad hoc, tests some things and misses others.
> What it lacks is tests that can be directly linked back to specific
> requirements. Tests that have names for example that go back to
> standards reference numbers (section paragraph numbering).
>
> The GCC test suite is like a collection of regression tests that we
> have built up over the years as part of our internal testing ,  from
> customer support or just interesting code fragments that get
> debated in news groups.
>
> Conformance tests should be written by someone that knows
> the requirements but have no knowledge of the implementation
>
> Regards,
>
> Walter Banks
>

Different types of test suite are useful for different purposes.  And 
/no/ test suite is complete.  So if a commercial compiler developer says 
they use Plum Hall (or whatever) to test standards conformance, all you 
know is that they have run the test suite.  It doesn't tell you how much 
of the standards are really tested by Plum Hall (I'm sure they are good 
- but they cannot ever be perfect).  It might not tell you how much of 
Plum Hall the compiler passed or failed.  It won't tell you anything 
about the bugs in the compiler, or the quality of the generated code or 
the libraries.  It will tell you absolutely /nothing/ about the 
correctness of the compiler (or libraries, or code delivered with the 
toolchain) when you use different sets of compiler flags or options.

Conformance tests tell you a bit about the toolchains standards 
conformance.  That can certainly be useful - but it is not a major part 
of being sure your tools generate correct object code from your source 
code.  And if you are particularly concerned about conformance tests, 
then you should probably invest in Plum Hall yourself and run them (or 
get a consultancy firm to do so) rather than just relying on the 
toolchain manufacturer.

Of course, if you want conformance testing for gcc, and don't want to 
buy the tools yourself, you can always buy "commercial open source" 
toolchains such as provided by CodeSourcery / Mentor Graphics - they run 
their gcc builds through conformance testing tools, including Plum Hall.

On 06/07/12 20:16, Rob Gaddi wrote:
> On Fri, 06 Jul 2012 20:03:17 +0200
> alb<alessandro.basili@cern.ch>  wrote:
>
>>
>> Can someone here explain what is the gain of an IDE?
>> I have to say that I hardly manage to find something rational or logical
>> behind the IDEs, starting from the concept of 'project' that nearly all
>> of them have but none of them describe formally.
>>
>> IMO the power you have with a Makefile goes way beyond the fancy icons
>> of an IDE, let alone that is far more portable and free from copyrights.
>>
>> My 2 cents.
>
> God I love having an IDE when I'm debugging.  Visual breakpoints,
> register display windows, expanded peripheral registers, all of that
> stuff is golden.
>
> And a good one also has good refactoring and code navigation tools that
> work across large numbers of files.  I think you can get that level of
> power from emacs or vim as well, but I haven't managed to find it in
> any sensible editor.
>
> But yes, their concept of a "project", and their firm belief that they
> know better than me how to organize one, is like dental surgery on a
> moving truck.
>

Eclipse works perfectly well with "makefile" projects.  That way you get 
the advantages of Eclipse (it's code navigation and highlighting tools, 
jumping from compiler error messages to the code, debugging tools, 
subversion support, etc., etc., - there are lots of features according 
to taste), while still having proper control of your project build 
system.  Eclipse's project management and build support is not bad 
compared to most IDE's, but you can always do better with your own 
makefiles.

Walter Banks wrote:
> Stefan Reuther wrote:
>>Walter Banks wrote:
>>>Conformance testing is one way to define with confidence the
>>>definition of a tool set. My surprise is that the FOSS community
>>>has not developed conformance testing tools in part to the response
>>>their cost. FOSS tools have been making the argument for free
>>>tools for years. .
>>
>>gcc comes with an extensive test suite, why does this one not count?
>>
>>I am actually quite confident with it, simply because it has failed less
>>often than "the others" on me.
> 
> The GCC test suite is ad hoc, tests some things and misses others.
> What it lacks is tests that can be directly linked back to specific
> requirements. Tests that have names for example that go back to
> standards reference numbers (section paragraph numbering).

The fact that commercial compilers have bugs indicates that commercial
conformance test suites also just test some things and miss others. So
what's the point?

Recent compiler bugs I've encountered are one tool claiming this
     char foo[] = { "bar" };
to not be valid C++. It is. But probably the test linked to �8.5.2(1)
did not catch it. Another one is a compiler crashing on
     class a { }; class b { };
     namespace { class c : public a, public b { }; }
Probably they've tested multiple inheritance, and anonymous namespaces,
but it seems they didn't test the combination of both.

A systematic test suite is no silver bullet. And a suite that tests for
previous problems is pretty useful. Whether it's more or less useful
remains to debate, I consider it good enough.


  Stefan

In article <jti0lk.1qo.1@stefan.msgid.phost.de>, stefan.news@arcor.de 
says...
> 
> Mark Borgerson wrote:
> > In article <jtfbgj.o8.1@stefan.msgid.phost.de>, stefan.news@arcor.de 
> >>>>Any sensible editor has support for ctags/etags, otherwise it doesn't
> >>>>deserve being called a sensible programmer's editor. This solves the
> >>>>navigation problem for me most of the time. Plus, my emacs has 1000
> >>>>files open anyway, so most of the time opening a file is C-x b
> >>>>name-of-file.c RET, no need to navigate through a file system. Try that
> >>>>with Eclipse.
> >>>
> >>>This raises a few questions:
> >>>
> >>>1.  Why does your emacs have 1000 files open?
> >>
> >>Why not?
> >>
> >>>2.  How do you pick the file you would like to edit among those 1000 
> >>>files?
> >>
> >>C-x b name-of-file.c RET
> > 
> > WOW!  you can keep track of 1000 file names in your head.  I have 
> > trouble remembering the last 50 files I've used.
> 
> I had a few years to learn them...
> 
> >>Or, actually, F10 (my shortcut for C-x b) nam<TAB>fi<TAB>c RET, using
> >>completion.
> >>
> >>>3.  What projects do you work on that have 1000 files?
> >>
> >>Automotive infotainment, from low level (boot loader) through the whole
> >>stack (drivers, file system, codec), to high level (HMI adaption).
> > 
> > That's a much larger system than any I've worked on.  Even the flight 
> > control system I worked on had only about 50-60 files---but we used 
> > pretty standard drivers for everything and depended on the OS (linux) to 
> > provide those, so we didn't have to recompile them.
> 
> What adds to my file count is that being a low-level guy, I get to debug
> the low-level problems the other developers have. Which means, over
> time, kernel and driver sources accumulate in my environment :-)

I feel your pain.  Luckily, I generally only have to solve my own 
problems.
> 
> >>>4.  How much system resources does it take to manage those 1000 files?
> >>
> >>The emacs process has a memory footprint of about 50-100 megabytes,
> >>containing all those files, and their undo data for the last half year.
> > 
> > Does the EMACs process have all the file structures, or does the OS have 
> > them and EMACS just keeps file handles?
> 
> It loads everything into RAM and then releases the file handles. It
> processes the files on first use (syntax coloring etc.).
That's the way it should be done.
> 
>
That mirrors a question I had  a few year ago.  I asked why  a data 
reduction program was reading bytes and words from a file instead of 
reading the whole file (a few hundred megabytes) into memory and 
processing the data using a pointer to the data.  A PC will give you 
hundreds of megabytes of data memory if you ask nicely----why should we 
be content to process data a few bytes at a time using a file handle?


Having gigabytes of RAM to handle tasks has certainly changed the 
paradigm from the time when we had tens of megabytes of disk storage and 
a memory usage was measuresed in blocks of  64KBytes.   Alas, many tools 
and operating systems still think of processing data via file handles 
and KBytes of memory.

Mark Borgerson

Stefan Reuther wrote:

> Walter Banks wrote:
>
> > Conformance testing is one way to define with confidence the
> > definition of a tool set. My surprise is that the FOSS community
> > has not developed conformance testing tools in part to the response
> > their cost. FOSS tools have been making the argument for free
> > tools for years. .
>
> gcc comes with an extensive test suite, why does this one not count?
>
> I am actually quite confident with it, simply because it has failed less
> often than "the others" on me.
>

The GCC test suite is ad hoc, tests some things and misses others.
What it lacks is tests that can be directly linked back to specific
requirements. Tests that have names for example that go back to
standards reference numbers (section paragraph numbering).

The GCC test suite is like a collection of regression tests that we
have built up over the years as part of our internal testing ,  from
customer support or just interesting code fragments that get
debated in news groups.

Conformance tests should be written by someone that knows
the requirements but have no knowledge of the implementation

Regards,

Walter Banks

Walter Banks wrote:
> Stefan Reuther wrote:
>>Walter Banks wrote:
>>>This is especially true of language implementation in FOSS tools
>>>and conformance to IEC / ISO standards. I have been shocked that
>>>of all the work that has been done on tools sets very little has been
>>>done in the FOSS community on conformance testing.
>>
>>For _formal_ conformance testing. This costs real money, and gives you a
>>sticker for glossy brochures. FOSS has/needs neither.
> 
> Conformance testing is one way to define with confidence the
> definition of a tool set. My surprise is that the FOSS community
> has not developed conformance testing tools in part to the response
> their cost. FOSS tools have been making the argument for free
> tools for years. .

gcc comes with an extensive test suite, why does this one not count?

I am actually quite confident with it, simply because it has failed less
often than "the others" on me.


  Stefan

Stefan Reuther wrote:

> Walter Banks wrote:
> > alb wrote:
> >>- standardization: the FOSS community has always paid great attention to
> >>the necessity of agreeing upon standards, from file formats to protocols
> >>and much more. A closed tool has not such an interest and from time to
> >>time even different versions of the same tool do not support file
> >>formats they originally created.
> >
> > The FOSS community has a lot of Ad Hoc standards but doesn't
> > participate in standards groups and generally don't support formal
> > standards.
>
> The Austin Group lists as members SuSE, RedHat, NetBSD, FreeBSD, Linux
> Standard Base. And if I look at C99, I find quite a number of gcc
> extensions in there.

 FOSS groups did not actively participate in C99 / C11. I would say
there are some GCC extensions and some extensions that GCC
adopted from other development work.

> > This is especially true of language implementation in FOSS tools
> > and conformance to IEC / ISO standards. I have been shocked that
> > of all the work that has been done on tools sets very little has been
> > done in the FOSS community on conformance testing.
>
> For _formal_ conformance testing. This costs real money, and gives you a
> sticker for glossy brochures. FOSS has/needs neither.

Conformance testing is one way to define with confidence the
definition of a tool set. My surprise is that the FOSS community
has not developed conformance testing tools in part to the response
their cost. FOSS tools have been making the argument for free
tools for years. .

w..