Forums

newlib, FreeRTOS, reentrancy, heap and related questions

Started by pozz February 6, 2020
Usually arm gcc compiler uses newlib (or newlib-nano) for standard C 
libraries (memset, malloc, printf, time and so on).

I sometimes replace newlib functions, because I don't like them. First 
of all, I replace snprintf because newlib implementation uses malloc and 
I don't like to use malloc, mostly if it can be avoided.
And for printf-like functions, there are a few implementations that 
don't use malloc.

When newlib is used with FreeRTOS, there are two heaps: one used by 
FreeRTOS and one used by newlib. Dave[1] suggests to replace FreeRTOS 
heap with newlib heap. Why don't do the contrary? newlib malloc is not 
reentrant, you should implement malloc_lock/unlock. If we're able to 
force newlib to use FreeRTOS heap, I think it would be better, because 
FreeRTOS malloc is natively multitasking safe.

Many times I need to have a date and time. time() and gettimeofday() 
from newlib are good, but I need to implement a _gettimeofday() function 
for my platform. So the value added by newlib is very little (I could 
implement a time() or gettimeofday() myself).

So the final question. If I remove printf/malloc/time/... from newlib, 
is newlib(-nano) needed yet? Is it possible to avoid using newlib at all?
I know I could need memset, strchr, strtok, but those functions can be 
implemented if needed (I think there are many implementations available, 
even in newlib project).

Of course, I don't need stdio features (open, close, read, write, exit, 
...).

I know newlib is for resource constrained embedded devices, so it is 
already small, but I have the feeling that it's bigger than I need, 
mostly with FreeRTOS.


[1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html
On 2/6/20 7:26 AM, pozz wrote:
> Usually arm gcc compiler uses newlib (or newlib-nano) for standard C > libraries (memset, malloc, printf, time and so on). > > I sometimes replace newlib functions, because I don't like them. First > of all, I replace snprintf because newlib implementation uses malloc and > I don't like to use malloc, mostly if it can be avoided. > And for printf-like functions, there are a few implementations that > don't use malloc. > > When newlib is used with FreeRTOS, there are two heaps: one used by > FreeRTOS and one used by newlib. Dave[1] suggests to replace FreeRTOS > heap with newlib heap. Why don't do the contrary? newlib malloc is not > reentrant, you should implement malloc_lock/unlock. If we're able to > force newlib to use FreeRTOS heap, I think it would be better, because > FreeRTOS malloc is natively multitasking safe. >
The FreeRTOS heap functions implement a smaller set of fuctionality than do the standard C library as implemented in newlib, in particular the realloc function, which is used (I am pretty sure) in some of the library. It is fairly trivial to write a 'heapc.c' memory allocator file that has FreeRTOS use the native malloc/free with also provides teh needed malloc_lock/unlock to make them thread safe.
> Many times I need to have a date and time. time() and gettimeofday() > from newlib are good, but I need to implement a _gettimeofday() function > for my platform. So the value added by newlib is very little (I could > implement a time() or gettimeofday() myself). > > So the final question. If I remove printf/malloc/time/... from newlib, > is newlib(-nano) needed yet? Is it possible to avoid using newlib at all? > I know I could need memset, strchr, strtok, but those functions can be > implemented if needed (I think there are many implementations available, > even in newlib project). > > Of course, I don't need stdio features (open, close, read, write, exit, > ...). > > I know newlib is for resource constrained embedded devices, so it is > already small, but I have the feeling that it's bigger than I need, > mostly with FreeRTOS. > > > [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html
Yes, it is quite possible to totally implement you own 'standard library' yourself and totally replace newlib-nano. The question comes does doing so buy you enough to be worth the time.
Il 06/02/2020 13:37, Richard Damon ha scritto:
> On 2/6/20 7:26 AM, pozz wrote: >> Usually arm gcc compiler uses newlib (or newlib-nano) for standard C >> libraries (memset, malloc, printf, time and so on). >> >> I sometimes replace newlib functions, because I don't like them. First >> of all, I replace snprintf because newlib implementation uses malloc >> and I don't like to use malloc, mostly if it can be avoided. >> And for printf-like functions, there are a few implementations that >> don't use malloc. >> >> When newlib is used with FreeRTOS, there are two heaps: one used by >> FreeRTOS and one used by newlib. Dave[1] suggests to replace FreeRTOS >> heap with newlib heap. Why don't do the contrary? newlib malloc is not >> reentrant, you should implement malloc_lock/unlock. If we're able to >> force newlib to use FreeRTOS heap, I think it would be better, because >> FreeRTOS malloc is natively multitasking safe. >> > The FreeRTOS heap functions implement a smaller set of fuctionality than > do the standard C library as implemented in newlib, in particular the > realloc function, which is used (I am pretty sure) in some of the library.
Do you mean that realloc(), that is used by some newlib functions, isn't implemented by FreeRTOS heap? I have the sensation that I don't use those newlib functions that use realloc(), because I avoid to use functions that use heap at all. In this case, I think using FreeRTOS heap only instead of newlib heap has some sense.
> It is fairly trivial to write a 'heapc.c' memory allocator file that has > FreeRTOS use the native malloc/free with also provides teh needed > malloc_lock/unlock to make them thread safe.
Why not the contrary? Why don't use FreeRTOS heap management only? Is it impossible for some reason to completely avoid newlib heap functions?
>> Many times I need to have a date and time. time() and gettimeofday() >> from newlib are good, but I need to implement a _gettimeofday() >> function for my platform. So the value added by newlib is very little >> (I could implement a time() or gettimeofday() myself). >> >> So the final question. If I remove printf/malloc/time/... from newlib, >> is newlib(-nano) needed yet? Is it possible to avoid using newlib at all? >> I know I could need memset, strchr, strtok, but those functions can be >> implemented if needed (I think there are many implementations >> available, even in newlib project). >> >> Of course, I don't need stdio features (open, close, read, write, >> exit, ...). >> >> I know newlib is for resource constrained embedded devices, so it is >> already small, but I have the feeling that it's bigger than I need, >> mostly with FreeRTOS. >> >> >> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html > > Yes, it is quite possible to totally implement you own 'standard > library' yourself and totally replace newlib-nano. The question comes > does doing so buy you enough to be worth the time.
I'm wondering how many standard functions I need to implement without newlib. 1, 10 or 100? How complex would be to implement them? Of course, only me can answer.
On 06/02/2020 13:57, pozz wrote:
> Il 06/02/2020 13:37, Richard Damon ha scritto: >> On 2/6/20 7:26 AM, pozz wrote:
>>> So the final question. If I remove printf/malloc/time/... from >>> newlib, is newlib(-nano) needed yet? Is it possible to avoid using >>> newlib at all? >>> I know I could need memset, strchr, strtok, but those functions can >>> be implemented if needed (I think there are many implementations >>> available, even in newlib project). >>> >>> Of course, I don't need stdio features (open, close, read, write, >>> exit, ...). >>> >>> I know newlib is for resource constrained embedded devices, so it is >>> already small, but I have the feeling that it's bigger than I need, >>> mostly with FreeRTOS. >>> >>> >>> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html >> >> Yes, it is quite possible to totally implement you own 'standard >> library' yourself and totally replace newlib-nano. The question comes >> does doing so buy you enough to be worth the time. > > I'm wondering how many standard functions I need to implement without > newlib. 1, 10 or 100? How complex would be to implement them? > Of course, only me can answer.
It depends a lot on what you need to do. A good many of the standard C functions are rarely useful in small embedded systems (hands up those who need the locale functions or wide character handling). Most systems can also do without the maths functions - perhaps also implementing their own for a more appropriate balance between speed and accuracy. There are a few oddities to watch out for. Technically, you can't actually implement malloc() in standard C, nor can you implement memcpy or memmove, due to the type aliasing and effective type rules. If you want to be sure of problem-free code that is safe regardless of optimisation, link-time optimisation, new generations of compilers, etc., then you'll be quite careful and make good use of gcc attributes. Another possibility here is to move to C++. It has better support for memory allocation functions, memory pools, etc. And you can avoid one of the silliest aspects of malloc/free - the fact that the memory allocator implementation has to store the size of the allocated block somewhere. Usually when you are freeing memory, you know the size already - storing it in the block just makes things less efficient. C++ also has functions for "washing" pointers so that you can keep the compiler informed about aliasing and re-use of memory, for better efficiency and safety. (I haven't tried much of this myself as yet, but I expect to do so before long.)
Il 06/02/2020 14:58, David Brown ha scritto:
> On 06/02/2020 13:57, pozz wrote: >> Il 06/02/2020 13:37, Richard Damon ha scritto: >>> On 2/6/20 7:26 AM, pozz wrote: > >>>> So the final question. If I remove printf/malloc/time/... from >>>> newlib, is newlib(-nano) needed yet? Is it possible to avoid using >>>> newlib at all? >>>> I know I could need memset, strchr, strtok, but those functions can >>>> be implemented if needed (I think there are many implementations >>>> available, even in newlib project). >>>> >>>> Of course, I don't need stdio features (open, close, read, write, >>>> exit, ...). >>>> >>>> I know newlib is for resource constrained embedded devices, so it is >>>> already small, but I have the feeling that it's bigger than I need, >>>> mostly with FreeRTOS. >>>> >>>> >>>> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html >>> >>> Yes, it is quite possible to totally implement you own 'standard >>> library' yourself and totally replace newlib-nano. The question comes >>> does doing so buy you enough to be worth the time. >> >> I'm wondering how many standard functions I need to implement without >> newlib. 1, 10 or 100? How complex would be to implement them? >> Of course, only me can answer. > > It depends a lot on what you need to do. A good many of the standard C > functions are rarely useful in small embedded systems (hands up those > who need the locale functions or wide character handling). Most systems > can also do without the maths functions - perhaps also implementing > their own for a more appropriate balance between speed and accuracy.
Yes, indeed those are my assumptions.
> There are a few oddities to watch out for. Technically, you can't > actually implement malloc() in standard C,
If I really need heap, I would use FreeRTOS implementation that is thread-safe and already available.
> nor can you implement memcpy > or memmove, due to the type aliasing and effective type rules. If you > want to be sure of problem-free code that is safe regardless of > optimisation, link-time optimisation, new generations of compilers, > etc., then you'll be quite careful and make good use of gcc attributes.
What about copying byte by byte? Here[1] you can see newlib memcpy implementation. If PREFER_SIZE_OVER_SPEED or __OPTIMIZE_SIZE__ is defined, the implementation is really copying byte by byte. I don't really know how newlib used by my compiler (CubeIDE from ST) was compiled, maybe I'm using dumb version of memcpy already. This is an extract from a listing: 08025850 <memcpy>: 8025850: b510 push {r4, lr} 8025852: 1e43 subs r3, r0, #1 8025854: 440a add r2, r1 8025856: 4291 cmp r1, r2 8025858: d100 bne.n 802585c <memcpy+0xc> 802585a: bd10 pop {r4, pc} 802585c: f811 4b01 ldrb.w r4, [r1], #1 8025860: f803 4f01 strb.w r4, [r3, #1]! 8025864: e7f7 b.n 8025856 <memcpy+0x6> I'm not an expert of assembly, but it seems to me it is implemented in the simple and not optimized way.
> Another possibility here is to move to C++. It has better support for > memory allocation functions, memory pools, etc. And you can avoid one > of the silliest aspects of malloc/free - the fact that the memory > allocator implementation has to store the size of the allocated block > somewhere. Usually when you are freeing memory, you know the size > already - storing it in the block just makes things less efficient. C++ > also has functions for "washing" pointers so that you can keep the > compiler informed about aliasing and re-use of memory, for better > efficiency and safety. (I haven't tried much of this myself as yet, but > I expect to do so before long.) >
[1] https://github.com/bminor/newlib/blob/master/newlib/libc/string/memcpy.c
On Thursday, February 6, 2020 at 7:57:57 AM UTC-5, pozz wrote:
> Do you mean that realloc(), that is used by some newlib functions, isn't > implemented by FreeRTOS heap?
realloc() use within newlib is documented on my web page you referenced.
> >> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html
Whatever you do, check the map and make sure you aren't accidentally using memory management functions you haven't covered. Hope that helps, Best Regards, Dave
Il 06/02/2020 16:21, Dave Nadler ha scritto:
> On Thursday, February 6, 2020 at 7:57:57 AM UTC-5, pozz wrote: >> Do you mean that realloc(), that is used by some newlib functions, isn't >> implemented by FreeRTOS heap? > > realloc() use within newlib is documented on my web page you referenced.
You're right, I didn't notice: > Another option is wrap newlib&rsquo;s malloc-family to use FreeRTOS free > storage (ie heap_4.c), and specify newlib support for FreeRTOS. Tell > the linker to wrap all newlib's malloc-family functions (using > -Xlinker --wrap=malloc etc.), and provide a wrapper function that > calls the FreeRTOS functions. I tried that, but newlib's printf family > uses realloc, which is not supported in FreeRTOS heap implementations. So my sensation was correct, replacing printf-like functions with others that don't use malloc is sufficient to avoid using newlib heap at all. Most probably there are other newlib functions that need realloc(), but I think they can be replaced as well as printf. With those assumptions, are there any other drawbacks in using FreeRTOS heap only?
> >>>> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html > > Whatever you do, check the map and make sure you aren't accidentally > using memory management functions you haven't covered.
Yes, good suggestion.
On 06/02/2020 16:06, pozz wrote:
> Il 06/02/2020 14:58, David Brown ha scritto:
> >> nor can you implement memcpy >> or memmove, due to the type aliasing and effective type rules.&#2013266080; If you >> want to be sure of problem-free code that is safe regardless of >> optimisation, link-time optimisation, new generations of compilers, >> etc., then you'll be quite careful and make good use of gcc attributes. > > What about copying byte by byte? > Here[1] you can see newlib memcpy implementation. If > PREFER_SIZE_OVER_SPEED or __OPTIMIZE_SIZE__ is defined, the > implementation is really copying byte by byte. > > I don't really know how newlib used by my compiler (CubeIDE from ST) was > compiled, maybe I'm using dumb version of memcpy already. > > This is an extract from a listing: > > 08025850 <memcpy>: > &#2013266080;8025850:&#2013266080;&#2013266080;&#2013266080; b510&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; push&#2013266080;&#2013266080;&#2013266080; {r4, lr} > &#2013266080;8025852:&#2013266080;&#2013266080;&#2013266080; 1e43&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; subs&#2013266080;&#2013266080;&#2013266080; r3, r0, #1 > &#2013266080;8025854:&#2013266080;&#2013266080;&#2013266080; 440a&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; add&#2013266080;&#2013266080;&#2013266080; r2, r1 > &#2013266080;8025856:&#2013266080;&#2013266080;&#2013266080; 4291&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; cmp&#2013266080;&#2013266080;&#2013266080; r1, r2 > &#2013266080;8025858:&#2013266080;&#2013266080;&#2013266080; d100&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; bne.n&#2013266080;&#2013266080;&#2013266080; 802585c <memcpy+0xc> > &#2013266080;802585a:&#2013266080;&#2013266080;&#2013266080; bd10&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; pop&#2013266080;&#2013266080;&#2013266080; {r4, pc} > &#2013266080;802585c:&#2013266080;&#2013266080;&#2013266080; f811 4b01&#2013266080;&#2013266080;&#2013266080;&#2013266080; ldrb.w&#2013266080;&#2013266080;&#2013266080; r4, [r1], #1 > &#2013266080;8025860:&#2013266080;&#2013266080;&#2013266080; f803 4f01&#2013266080;&#2013266080;&#2013266080;&#2013266080; strb.w&#2013266080;&#2013266080;&#2013266080; r4, [r3, #1]! > &#2013266080;8025864:&#2013266080;&#2013266080;&#2013266080; e7f7&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; b.n&#2013266080;&#2013266080;&#2013266080; 8025856 <memcpy+0x6> > > I'm not an expert of assembly, but it seems to me it is implemented in > the simple and not optimized way. >
It is not the actual copying that is the problem - copying by char is simple and safe (though often inefficient). The issue is that the C standards say memcpy also copies the effective type in certain circumstances - there is no way to specify that in C, and it is therefore a special feature of the library memcpy. A homemade memcpy does not have that same feature. (In a similar vain, there is no way to get memory in standard C that has "no declared type" except via the library malloc and friends - a homemade malloc won't do.) I am not sure what the best solution is here. Anyway, for memcpy make sure the compiler can use the builtin versions where possible (avoid -ffreestanding, or use -fbuiltin) as this will give far better code.
Il 06/02/2020 21:41, David Brown ha scritto:
> On 06/02/2020 16:06, pozz wrote: >> Il 06/02/2020 14:58, David Brown ha scritto: > >> >>> nor can you implement memcpy >>> or memmove, due to the type aliasing and effective type rules.&#2013266080; If you >>> want to be sure of problem-free code that is safe regardless of >>> optimisation, link-time optimisation, new generations of compilers, >>> etc., then you'll be quite careful and make good use of gcc attributes. >> >> What about copying byte by byte? >> Here[1] you can see newlib memcpy implementation. If >> PREFER_SIZE_OVER_SPEED or __OPTIMIZE_SIZE__ is defined, the >> implementation is really copying byte by byte. >> >> I don't really know how newlib used by my compiler (CubeIDE from ST) >> was compiled, maybe I'm using dumb version of memcpy already. >> >> This is an extract from a listing: >> >> 08025850 <memcpy>: >> &#2013266080;&#2013266080;8025850:&#2013266080;&#2013266080;&#2013266080; b510&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; push&#2013266080;&#2013266080;&#2013266080; {r4, lr} >> &#2013266080;&#2013266080;8025852:&#2013266080;&#2013266080;&#2013266080; 1e43&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; subs&#2013266080;&#2013266080;&#2013266080; r3, r0, #1 >> &#2013266080;&#2013266080;8025854:&#2013266080;&#2013266080;&#2013266080; 440a&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; add&#2013266080;&#2013266080;&#2013266080; r2, r1 >> &#2013266080;&#2013266080;8025856:&#2013266080;&#2013266080;&#2013266080; 4291&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; cmp&#2013266080;&#2013266080;&#2013266080; r1, r2 >> &#2013266080;&#2013266080;8025858:&#2013266080;&#2013266080;&#2013266080; d100&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; bne.n&#2013266080;&#2013266080;&#2013266080; 802585c <memcpy+0xc> >> &#2013266080;&#2013266080;802585a:&#2013266080;&#2013266080;&#2013266080; bd10&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; pop&#2013266080;&#2013266080;&#2013266080; {r4, pc} >> &#2013266080;&#2013266080;802585c:&#2013266080;&#2013266080;&#2013266080; f811 4b01&#2013266080;&#2013266080;&#2013266080;&#2013266080; ldrb.w&#2013266080;&#2013266080;&#2013266080; r4, [r1], #1 >> &#2013266080;&#2013266080;8025860:&#2013266080;&#2013266080;&#2013266080; f803 4f01&#2013266080;&#2013266080;&#2013266080;&#2013266080; strb.w&#2013266080;&#2013266080;&#2013266080; r4, [r3, #1]! >> &#2013266080;&#2013266080;8025864:&#2013266080;&#2013266080;&#2013266080; e7f7&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080;&#2013266080; b.n&#2013266080;&#2013266080;&#2013266080; 8025856 <memcpy+0x6> >> >> I'm not an expert of assembly, but it seems to me it is implemented in >> the simple and not optimized way. >> > > It is not the actual copying that is the problem - copying by char is > simple and safe (though often inefficient).&#2013266080; The issue is that the C > standards say memcpy also copies the effective type in certain > circumstances - there is no way to specify that in C, and it is > therefore a special feature of the library memcpy.
Could you make an example? I didn't understand. Anyway as you can see, newlib just implements memcpy in pure C language when compiled without optimizations. Are you saying it's bugged?
> A homemade memcpy > does not have that same feature.&#2013266080; (In a similar vain, there is no way to > get memory in standard C that has "no declared type" except via the > library malloc and friends - a homemade malloc won't do.)&#2013266080; I am not sure > what the best solution is here. > > Anyway, for memcpy make sure the compiler can use the builtin versions > where possible (avoid -ffreestanding, or use -fbuiltin) as this will > give far better code. >
I don't use -ffreestanding, but I don't know if I'm using -fbuiltin. Anyway you are suggesting to use builtin functions that are functions built *in* the compiler and not in the newlib. Another reason to consider useless newlib.
Dave, in your page you suggest to use -Wrap compiler option to replace 
newlib malloc with FreeRTOS malloc.

I think it's not necessary. If you define a malloc() function, the 
linker should use it instead of newlib malloc, without emitting any 
warning or errors about duplicate definition.

I use this to replace printf-like functions.

Linker searches for a function in the object files that you're linking 
and only if it isn't able to find it will try with libraries.