On 29/09/2022 18:03, Phil Hobbs wrote:
> pozz wrote:
>> Il 24/09/2022 17:43, David Brown ha scritto:
>>> On 24/09/2022 00:43, pozz wrote:
>>>> Many times I'd like to replace libc functions in embedded systems, 
>>>> because of some tests or because I need a different implementation.
>>>>
>>>> For example, sprintf implementation of newlib uses malloc and I 
>>>> can't use it many times.
>>>>
>>>> It sometimes happens that I can use malloc, with some restrictions. 
>>>> Just for test or optimization, I replace libc malloc with my own 
>>>> implementation.
>>>>
>>>> In these cases I use a very simple trick: I simply define a new 
>>>> sprintf()/malloc() function in one of my source code. During linking 
>>>> (GNU ld), the linker prefers my implementation and ignore the 
>>>> implementation in the libc. I don't know why this happens and if it 
>>>> is a standard way during linking, but it works for me.
>>>>
>>>> Recently I found a different behaviour, with an error from the 
>>>> linker: multiple definitions of sprintf.
>>>>
>>>> After some time, I understood it is caused by a piece of code that 
>>>> uses another libc function (ctime) that uses sprintf too, I suppose.
>>>> Maybe in this case, ctime needs sprintf from libc so the linker 
>>>> tries to add two sprintf and gives the error.
>>>>
>>>> First question: what exactly happens?
>>>>
>>>> Second question: is there a better method to override a standard 
>>>> libc function?
>>>> I know I can use preprocessor magic, but in this case I should write 
>>>> MYSPRINTF(...) that is much worse than sprintf(...).
>>>>
>>>>
>>>
>>> When a linker is looking for symbols used, it will first look in the 
>>> list of object files it is given, then move on to searching the 
>>> static libraries.&nbsp; So an override of a library function should have 
>>> priority.
>>>
>>> However, when it pulls in a symbol, it will pull in the whole object 
>>> file that it is in.&nbsp; (Section garbage collection might let it later 
>>> remove unneeded sections, but that is /after/ this stage.)&nbsp; A static 
>>> library file is a collection of object files, so when the link 
>>> requires pulling in a symbol from the library, it gets all the 
>>> symbols from the object file containing it.&nbsp; Usually, libraries are 
>>> build from lots of small files with a single function or a few highly 
>>> related functions in order to minimise this issue.&nbsp; (It reduces scope 
>>> for inter-procedural optimisations in the library, however.)
>>>
>>> In your case, it is not unlikely that the object library object file 
>>> that contains "sprintf" also contains "snprintf", "vsnprintf", and 
>>> other related functions.&nbsp; So although you have overriden "sprintf", 
>>> perhaps "ctime" uses "snprintf" and its object file also contains 
>>> "sprintf" - thus causing a conflict.
>>
>> ctime()[1] uses actime()[2] that uses siprintf()[3].
>> I overrided siprintf() too and this solved the linking process.
>>
>> However it's not completely clear to me. siprintf() is defined in a 
>> single file (siprintf.c), so I thought it was contained in a single 
>> object file (siprintf.o).

The process is recursive - perhaps siprintf.o brings in other files, 
ending in including something that also contains sprintf.  The map file 
cross-reference is your friend here.

>> However it seems there's an object file (lib_a-sprintf.o) that 
>> contains sprintf() and siprintf(). Maybe this multiple functions 
>> object file is generated during newlib build process.
> 
> An old-time hack to improve optimization with dumb linkers was to have a 
> .c file #include other .c files so that the compiler got to see more of 
> the code at a time.
> 
> Haven't seen that one in awhile.
> 

Unless you write code specifically targeting such a setup, you have a 
fair chance of breakage.  And any decent toolchain has better ways of 
handling this anyway - gcc has had "-fwhole-program" for a great many 
years, then link-time optimisation has become the modern replacement. 
clang/llvm has had LTO from the start, and many commercial embedded 
compilers have something similar.

pozz wrote:
> Il 24/09/2022 17:43, David Brown ha scritto:
>> On 24/09/2022 00:43, pozz wrote:
>>> Many times I'd like to replace libc functions in embedded systems, 
>>> because of some tests or because I need a different implementation.
>>>
>>> For example, sprintf implementation of newlib uses malloc and I can't 
>>> use it many times.
>>>
>>> It sometimes happens that I can use malloc, with some restrictions. 
>>> Just for test or optimization, I replace libc malloc with my own 
>>> implementation.
>>>
>>> In these cases I use a very simple trick: I simply define a new 
>>> sprintf()/malloc() function in one of my source code. During linking 
>>> (GNU ld), the linker prefers my implementation and ignore the 
>>> implementation in the libc. I don't know why this happens and if it 
>>> is a standard way during linking, but it works for me.
>>>
>>> Recently I found a different behaviour, with an error from the 
>>> linker: multiple definitions of sprintf.
>>>
>>> After some time, I understood it is caused by a piece of code that 
>>> uses another libc function (ctime) that uses sprintf too, I suppose.
>>> Maybe in this case, ctime needs sprintf from libc so the linker tries 
>>> to add two sprintf and gives the error.
>>>
>>> First question: what exactly happens?
>>>
>>> Second question: is there a better method to override a standard libc 
>>> function?
>>> I know I can use preprocessor magic, but in this case I should write 
>>> MYSPRINTF(...) that is much worse than sprintf(...).
>>>
>>>
>>
>> When a linker is looking for symbols used, it will first look in the 
>> list of object files it is given, then move on to searching the static 
>> libraries.&nbsp; So an override of a library function should have priority.
>>
>> However, when it pulls in a symbol, it will pull in the whole object 
>> file that it is in.&nbsp; (Section garbage collection might let it later 
>> remove unneeded sections, but that is /after/ this stage.)&nbsp; A static 
>> library file is a collection of object files, so when the link 
>> requires pulling in a symbol from the library, it gets all the symbols 
>> from the object file containing it.&nbsp; Usually, libraries are build from 
>> lots of small files with a single function or a few highly related 
>> functions in order to minimise this issue.&nbsp; (It reduces scope for 
>> inter-procedural optimisations in the library, however.)
>>
>> In your case, it is not unlikely that the object library object file 
>> that contains "sprintf" also contains "snprintf", "vsnprintf", and 
>> other related functions.&nbsp; So although you have overriden "sprintf", 
>> perhaps "ctime" uses "snprintf" and its object file also contains 
>> "sprintf" - thus causing a conflict.
> 
> ctime()[1] uses actime()[2] that uses siprintf()[3].
> I overrided siprintf() too and this solved the linking process.
> 
> However it's not completely clear to me. siprintf() is defined in a 
> single file (siprintf.c), so I thought it was contained in a single 
> object file (siprintf.o).
> However it seems there's an object file (lib_a-sprintf.o) that contains 
> sprintf() and siprintf(). Maybe this multiple functions object file is 
> generated during newlib build process.

An old-time hack to improve optimization with dumb linkers was to have a 
.c file #include other .c files so that the compiler got to see more of 
the code at a time.

Haven't seen that one in awhile.

Cheers

Phil Hobbs

-- 
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC / Hobbs ElectroOptics
Optics, Electro-optics, Photonics, Analog Electronics
Briarcliff Manor NY 10510

http://electrooptical.net
http://hobbs-eo.com

Il 24/09/2022 17:43, David Brown ha scritto:
> On 24/09/2022 00:43, pozz wrote:
>> Many times I'd like to replace libc functions in embedded systems, 
>> because of some tests or because I need a different implementation.
>>
>> For example, sprintf implementation of newlib uses malloc and I can't 
>> use it many times.
>>
>> It sometimes happens that I can use malloc, with some restrictions. 
>> Just for test or optimization, I replace libc malloc with my own 
>> implementation.
>>
>> In these cases I use a very simple trick: I simply define a new 
>> sprintf()/malloc() function in one of my source code. During linking 
>> (GNU ld), the linker prefers my implementation and ignore the 
>> implementation in the libc. I don't know why this happens and if it is 
>> a standard way during linking, but it works for me.
>>
>> Recently I found a different behaviour, with an error from the linker: 
>> multiple definitions of sprintf.
>>
>> After some time, I understood it is caused by a piece of code that 
>> uses another libc function (ctime) that uses sprintf too, I suppose.
>> Maybe in this case, ctime needs sprintf from libc so the linker tries 
>> to add two sprintf and gives the error.
>>
>> First question: what exactly happens?
>>
>> Second question: is there a better method to override a standard libc 
>> function?
>> I know I can use preprocessor magic, but in this case I should write 
>> MYSPRINTF(...) that is much worse than sprintf(...).
>>
>>
> 
> When a linker is looking for symbols used, it will first look in the 
> list of object files it is given, then move on to searching the static 
> libraries.&nbsp; So an override of a library function should have priority.
> 
> However, when it pulls in a symbol, it will pull in the whole object 
> file that it is in.&nbsp; (Section garbage collection might let it later 
> remove unneeded sections, but that is /after/ this stage.)&nbsp; A static 
> library file is a collection of object files, so when the link requires 
> pulling in a symbol from the library, it gets all the symbols from the 
> object file containing it.&nbsp; Usually, libraries are build from lots of 
> small files with a single function or a few highly related functions in 
> order to minimise this issue.&nbsp; (It reduces scope for inter-procedural 
> optimisations in the library, however.)
> 
> In your case, it is not unlikely that the object library object file 
> that contains "sprintf" also contains "snprintf", "vsnprintf", and other 
> related functions.&nbsp; So although you have overriden "sprintf", perhaps 
> "ctime" uses "snprintf" and its object file also contains "sprintf" - 
> thus causing a conflict.

ctime()[1] uses actime()[2] that uses siprintf()[3].
I overrided siprintf() too and this solved the linking process.

However it's not completely clear to me. siprintf() is defined in a 
single file (siprintf.c), so I thought it was contained in a single 
object file (siprintf.o).
However it seems there's an object file (lib_a-sprintf.o) that contains 
sprintf() and siprintf(). Maybe this multiple functions object file is 
generated during newlib build process.

On 24/09/2022 00:43, pozz wrote:
> Many times I'd like to replace libc functions in embedded systems, 
> because of some tests or because I need a different implementation.
> 
> For example, sprintf implementation of newlib uses malloc and I can't 
> use it many times.
> 
> It sometimes happens that I can use malloc, with some restrictions. Just 
> for test or optimization, I replace libc malloc with my own implementation.
> 
> In these cases I use a very simple trick: I simply define a new 
> sprintf()/malloc() function in one of my source code. During linking 
> (GNU ld), the linker prefers my implementation and ignore the 
> implementation in the libc. I don't know why this happens and if it is a 
> standard way during linking, but it works for me.
> 
> Recently I found a different behaviour, with an error from the linker: 
> multiple definitions of sprintf.
> 
> After some time, I understood it is caused by a piece of code that uses 
> another libc function (ctime) that uses sprintf too, I suppose.
> Maybe in this case, ctime needs sprintf from libc so the linker tries to 
> add two sprintf and gives the error.
> 
> First question: what exactly happens?
> 
> Second question: is there a better method to override a standard libc 
> function?
> I know I can use preprocessor magic, but in this case I should write 
> MYSPRINTF(...) that is much worse than sprintf(...).
> 
> 

I forgot to mention in my last post - if you override any standard 
library functions, make sure the function signature and the semantics 
are identical to or stronger than the standard requirements.  Compilers 
sometimes rely on the standard functions doing what they are supposed to 
do, regardless of whether you have overridden them or used the standard 
include files.  Thus you need to make sure your "sprintf" returns the 
value you'd get from standard "sprintf", because the compiler might use 
that value even if you don't explicitly collect it in your code using 
the function.

On 24/09/2022 00:43, pozz wrote:
> Many times I'd like to replace libc functions in embedded systems, 
> because of some tests or because I need a different implementation.
> 
> For example, sprintf implementation of newlib uses malloc and I can't 
> use it many times.
> 
> It sometimes happens that I can use malloc, with some restrictions. Just 
> for test or optimization, I replace libc malloc with my own implementation.
> 
> In these cases I use a very simple trick: I simply define a new 
> sprintf()/malloc() function in one of my source code. During linking 
> (GNU ld), the linker prefers my implementation and ignore the 
> implementation in the libc. I don't know why this happens and if it is a 
> standard way during linking, but it works for me.
> 
> Recently I found a different behaviour, with an error from the linker: 
> multiple definitions of sprintf.
> 
> After some time, I understood it is caused by a piece of code that uses 
> another libc function (ctime) that uses sprintf too, I suppose.
> Maybe in this case, ctime needs sprintf from libc so the linker tries to 
> add two sprintf and gives the error.
> 
> First question: what exactly happens?
> 
> Second question: is there a better method to override a standard libc 
> function?
> I know I can use preprocessor magic, but in this case I should write 
> MYSPRINTF(...) that is much worse than sprintf(...).
> 
> 

When a linker is looking for symbols used, it will first look in the 
list of object files it is given, then move on to searching the static 
libraries.  So an override of a library function should have priority.

However, when it pulls in a symbol, it will pull in the whole object 
file that it is in.  (Section garbage collection might let it later 
remove unneeded sections, but that is /after/ this stage.)  A static 
library file is a collection of object files, so when the link requires 
pulling in a symbol from the library, it gets all the symbols from the 
object file containing it.  Usually, libraries are build from lots of 
small files with a single function or a few highly related functions in 
order to minimise this issue.  (It reduces scope for inter-procedural 
optimisations in the library, however.)

In your case, it is not unlikely that the object library object file 
that contains "sprintf" also contains "snprintf", "vsnprintf", and other 
related functions.  So although you have overriden "sprintf", perhaps 
"ctime" uses "snprintf" and its object file also contains "sprintf" - 
thus causing a conflict.

You can find out more by looking at your map file (even from a failed 
link), especially with cross-references enabled in the map file.

Solutions then involve overriding the other library functions used by 
"ctime", using a different library (perhaps "newlib-nano" does not have 
the malloc issue in the first place), using something other than 
"ctime", accepting the library's "sprintf", etc.

(As an aside - you should be wary about using "sprintf".  "snprintf" is 
normally a better choice.)

pozz <pozzugno@gmail.com> wrote:
> After some time, I understood it is caused by a piece of code that uses 
> another libc function (ctime) that uses sprintf too, I suppose.
> Maybe in this case, ctime needs sprintf from libc so the linker tries to 
> add two sprintf and gives the error.
> 
> First question: what exactly happens?

It is possible to have a weak symbol, which provides a default linkage
unless the symbol is explicitly defined.  It may also just be that the
linker is satisfied with the local definition and never pulls in the newlib
symbol (eg if the function and its caller were in the same file, I think the
compiler would resolve it without asking the linker - eg if the function was
declared static) and the happenstance is the linker doesn't notice a clash.
(sometimes the ordering of objects on the linker command line matters)

> Second question: is there a better method to override a standard libc 
> function?
> I know I can use preprocessor magic, but in this case I should write 
> MYSPRINTF(...) that is much worse than sprintf(...).

Do you need the linker to know about your switch, or is preprocessor magic
alone ok?

You could do:
#define sprintf MYSPRINTF
in a header file that's included in all your source code.  As long as you
don't do that when compiling newlib that should be OK.  If you want to call
newlib's sprintf(), you can either see if newlib has an internal function
you can call (eg __sprintf), or simply implement your MYSPRINTF function in
a file that does not see the #define and is exposed to the default sprintf()
function.  Essentially at this point sprintf as a symbol only exists in
newlib, and the compiler sees you calling MYSPRINTF everywhere.

If you want to do more complicated things with the arguments, there are ways
to do varargs with macros if you need to.

Theo

On 9/23/22 6:43 PM, pozz wrote:
> Many times I'd like to replace libc functions in embedded systems, 
> because of some tests or because I need a different implementation.
> 
> For example, sprintf implementation of newlib uses malloc and I can't 
> use it many times.
> 
> It sometimes happens that I can use malloc, with some restrictions. Just 
> for test or optimization, I replace libc malloc with my own implementation.
> 
> In these cases I use a very simple trick: I simply define a new 
> sprintf()/malloc() function in one of my source code. During linking 
> (GNU ld), the linker prefers my implementation and ignore the 
> implementation in the libc. I don't know why this happens and if it is a 
> standard way during linking, but it works for me.
> 
> Recently I found a different behaviour, with an error from the linker: 
> multiple definitions of sprintf.
> 
> After some time, I understood it is caused by a piece of code that uses 
> another libc function (ctime) that uses sprintf too, I suppose.
> Maybe in this case, ctime needs sprintf from libc so the linker tries to 
> add two sprintf and gives the error.
> 
> First question: what exactly happens?
> 
> Second question: is there a better method to override a standard libc 
> function?
> I know I can use preprocessor magic, but in this case I should write 
> MYSPRINTF(...) that is much worse than sprintf(...).
> 
> 
My experiance is that a common cause is that if multiple functions are 
defined in a file, it is normally all or nothing for including that 
module, so you need to find out what all is defined in that file and 
make a definition for ALL the functions that might be used by pieces of 
the library that you use.

Many times I'd like to replace libc functions in embedded systems, 
because of some tests or because I need a different implementation.

For example, sprintf implementation of newlib uses malloc and I can't 
use it many times.

It sometimes happens that I can use malloc, with some restrictions. Just 
for test or optimization, I replace libc malloc with my own implementation.

In these cases I use a very simple trick: I simply define a new 
sprintf()/malloc() function in one of my source code. During linking 
(GNU ld), the linker prefers my implementation and ignore the 
implementation in the libc. I don't know why this happens and if it is a 
standard way during linking, but it works for me.

Recently I found a different behaviour, with an error from the linker: 
multiple definitions of sprintf.

After some time, I understood it is caused by a piece of code that uses 
another libc function (ctime) that uses sprintf too, I suppose.
Maybe in this case, ctime needs sprintf from libc so the linker tries to 
add two sprintf and gives the error.

First question: what exactly happens?

Second question: is there a better method to override a standard libc 
function?
I know I can use preprocessor magic, but in this case I should write 
MYSPRINTF(...) that is much worse than sprintf(...).