Regarding calculation of free memory

In comp.arch David Gay <dgay@beryl.cs.berkeley.edu> wrote:
> 
> Well, frankly, for the Linux case, you can go and look at the source code
> to check if there's any extra magic entry points or other dependencies. For
> the others, it definitely pays to check linker output files to make sure
> you didn't, e.g., accidentally end up with two copies of malloc. Yes,
> you'll have to repeat such checks if you change libc version - that's a
> price you have to pay. And if you're going to say "but I've run into
> problems", I'd like to get a specific description, rather than some
> nebulous "It happened to me somewhere, sometime, somehow".
> 

So you can in fact in the Solaris case too these days, though that 
hasn't really been needed so far AFAICT. 

-- 
	Sander

+++ Out of cheese error +++

In comp.arch Nick Maclaren <nmm1@cus.cam.ac.uk> wrote:
> 
> As far as Solaris etc. goes, you first have to find out which version
> (or versions) of malloc the application is using, INCLUDING those
> called by dynamic libraries.  That is unusually hard under Solaris,
> because ldd doesn't do what you apparently thinks that it does, and
> a fair amount of reverse engineering of the behaviour of the linker
> and loader is needed.  Then you have to extract the relevant symbols,
> see which library is using what, and decide if there is a clash.

Ahem. Sorry, Nick, but at the very least in case of Solaris, it is very 
easy to find out what exactly it is that the dynamic linker does and 
comapre the results of different runs. This is certainly not advanced 
hacker level. 

> Regards,
> Nick Maclaren.

-- 
	Sander

+++ Out of cheese error +++

"Michael N. Moran" <mike@mnmoran.org> wrote in message 
news:4fk0f.278$Pf5.148@bignews6.bellsouth.net...
> Hank Oredson wrote:
>> "Nick Maclaren" <nmm1@cus.cam.ac.uk> wrote in message 
>> news:dhqq8e$hn8$1@gemini.csx.cam.ac.uk...
>>
>>>In article <fmY%e.6314$oc.4151@newsread2.news.pas.earthlink.net>,
>>>Hank Oredson <horedson@earthlink.net> wrote:
>>>
>>>>>Problem one:  in C, there is no memory management beyond stack scoped
>>>>>except by using malloc.  Any algorithm that requires more than that
>>>>>can't be done if you don't use malloc.
>>>>
>>>>sbrk()?
>>>
>>>Not in C - in fact, I can't think of any standard that it IS in,
>>>because it makes such extremely horrible assumptions about the
>>>implementation.
>>
>> How strange. I use it. In C.

> Nick's point is that it is not a part of the C standard.

Who cares? When programming, one uses the tools
available. Nitpicking "standard" vs. "POSIX" vs. etc.
is just a waste of time. If we need better standards,
write them. But in any case I was not thinking of
C at all, and particularly not of Unix / Linux in my
original comments. I use the "C hacker" way of talking
about things because there are not a lot of folks who
will understand when I say "Just use ER MCOR$, it
will work the same in FOR and FTN." The point is that
the application can manage memory better than the OS
can, and there is always some way for the application
to get hold of that memory to manage.

> My Linux manpage for sbrk() says to #include <unistd.h>
> and under the "CONFORMING TO" heading says this:
>
>        brk  and  sbrk  are  not defined in the C Standard
>        and are deliberately excluded from  the  POSIX.1
>        standard  (see  paragraphs  B.1.1.1.3  and B.8.3.3).

So what?

> Of course you *can* call it from C, but you can't expect
> it to be there on all systems, especially "bare metal"
> embedded systems.

The programmer must know her tools. If she is writing a
banking backroom system, she will not be interested in
porting it to a Blackberry, or an access point.

-- 

  ... Hank

http://home.earthlink.net/~horedson
http://home.earthlink.net/~w0rli

In article <1128388888.617990@haldjas.folklore.ee>,
Sander Vesik  <sander@haldjas.folklore.ee> wrote:
>In comp.arch Nick Maclaren <nmm1@cus.cam.ac.uk> wrote:
>> 
>> As far as Solaris etc. goes, you first have to find out which version
>> (or versions) of malloc the application is using, INCLUDING those
>> called by dynamic libraries.  That is unusually hard under Solaris,
>> because ldd doesn't do what you apparently thinks that it does, and
>> a fair amount of reverse engineering of the behaviour of the linker
>> and loader is needed.  Then you have to extract the relevant symbols,
>> see which library is using what, and decide if there is a clash.
>
>Ahem. Sorry, Nick, but at the very least in case of Solaris, it is very 
>easy to find out what exactly it is that the dynamic linker does and 
>comapre the results of different runs. This is certainly not advanced 
>hacker level. 

Oh, really?  It is (relatively) easy to find out what is loaded,
though even that needs fairly advanced knowledge.  But finding out
which library and which symbol in that library is used to satisfy
another symbol in another library is not so easy.  I had a problem
recently that came down to this.  Nasty.

Solaris is not as extreme as Tru64, but you can still get multiple
different objects of the same name in the same executable (or, at
least, could until recently), and finding out whether you have and
which one is used for what is definitely hard.  In particular, there
is no PRECISE specification of any Unix linker that I know of, let
alone any on the loader :-(

I still have a program somewhere that I wrote to investigate this
area, and I was flabberghasted by what I found.  I was expecting
oddities, but not on the scale that I found them.


Regards,
Nick Maclaren.

In article <4fk0f.278$Pf5.148@bignews6.bellsouth.net>,
Michael N. Moran <mike@mnmoran.org> wrote:
>
>Nick's point is that it is not a part of the C standard.
>My Linux manpage for sbrk() says to #include <unistd.h>
>and under the "CONFORMING TO" heading says this:
>
>        brk  and  sbrk  are  not defined in the C Standard
>        and are deliberately excluded from  the  POSIX.1
>        standard  (see  paragraphs  B.1.1.1.3  and B.8.3.3).
>
>Of course you *can* call it from C, but you can't expect
>it to be there on all systems, especially "bare metal"
>embedded systems.

Yes.  And you can't expect it to work, and be used, the same way,
either, which is a more serious problem.  But you and I are talking
about programs where people care about portability and RAS, both
between systems and over time - and the happy hackers don't even
understand the concept.  Some of them will learn, as new versions
of even their favourite system causes their code to break; others
will not learn from experience.


Regards,
Nick Maclaren.

David Gay <dgay@beryl.CS.Berkeley.EDU> writes:


>nmm1@cus.cam.ac.uk (Nick Maclaren) writes:
>> In article <1128335888.327425@haldjas.folklore.ee>,
>> Sander Vesik <sander@haldjas.folklore.ee> writes:
>> |> In comp.arch Nick Maclaren <nmm1@cus.cam.ac.uk> wrote:
>> |> > 
>> |> > No.  Replacing malloc is not supported in C, and often breaks
>> |> > the nastier vendor and third-party libraries.  The reason is that
>> |> > the suite of malloc-related functions is not independent, and they
>> |> > often use non-standard interfaces.
>> |> 
>> |> There are still systems where you can reasonably do so - shipping 
>> |> more than one possible malloc to use is a good hint for example. 
>> 
>> It's a hint that the VENDOR can do it - but not that the application
>> programmer can.  Yes, an experienced hacker can, which was the basis
>> of my remark that I can, but I can't advise users to.

>Ahhh, now I know why I've never had any problems replacing malloc on Linux,
>Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)

You can certainly do so in Solaris (where we do ship far too many:
libc malloc, BSD malloc, SysV -lmalloc, mmap -lmapmalloc, -lmtmalloc,
watchmalloc (for debugging) and libumem).

And you can substitute your own as long as you make sure you
implement *all* routines our malloc libraries commonly export
(mallocs often define some of their functions in terms of the others;
but some implement foo() in terms of bar() and others vice versa and
this tend to lead to nasty problems if you don't roll a complete implementation)

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

nmm1@cus.cam.ac.uk (Nick Maclaren) writes:

>I still have a program somewhere that I wrote to investigate this
>area, and I was flabberghasted by what I found.  I was expecting
>oddities, but not on the scale that I found them.

The Solaris runtime linker allows you to do a complete trace of all
symbol references and which particular symbol definition they are
bound to.

Prior to Solaris 10 where some of the system libraries existed in
static form, the problem was intractable as malloc implementations
may have found their way into the exectubale in static form; mixing
malloc implementations in perilous, at best.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

In article <434247e3$0$11079$e4fe514c@news.xs4all.nl>, Casper H.S. Dik <Casper.Dik@Sun.COM> writes:
|> nmm1@cus.cam.ac.uk (Nick Maclaren) writes:
|> 
|> >I still have a program somewhere that I wrote to investigate this
|> >area, and I was flabberghasted by what I found.  I was expecting
|> >oddities, but not on the scale that I found them.
|> 
|> The Solaris runtime linker allows you to do a complete trace of all
|> symbol references and which particular symbol definition they are
|> bound to.

That wasn't the issue I was referring to.  It was worst on Tru64,
but I can't remember if it was serious on Solaris or it still
exists.  It was the situation where reference A (= fred) could not
be matched by object B (= fred) unless object C (= fred) was included
first (perhaps by a reference to object D (= joe)).  Seriously.

The most likely cause of this was incompatible references, which
could be 'mated' if there were a Fortran COMMON block of the same
name.

|> Prior to Solaris 10 where some of the system libraries existed in
|> static form, the problem was intractable as malloc implementations
|> may have found their way into the exectubale in static form; mixing
|> malloc implementations in perilous, at best.

Exactly.


Regards,
Nick Maclaren.

In article <434243de$0$11074$e4fe514c@news.xs4all.nl>,
Casper H.S. Dik <Casper.Dik@Sun.COM> writes:
|> 
|> >Ahhh, now I know why I've never had any problems replacing malloc on Linux,
|> >Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)
|> 
|> You can certainly do so in Solaris (where we do ship far too many:
|> libc malloc, BSD malloc, SysV -lmalloc, mmap -lmapmalloc, -lmtmalloc,
|> watchmalloc (for debugging) and libumem).
|> 
|> And you can substitute your own as long as you make sure you
|> implement *all* routines our malloc libraries commonly export
|> (mallocs often define some of their functions in terms of the others;
|> but some implement foo() in terms of bar() and others vice versa and
|> this tend to lead to nasty problems if you don't roll a complete implementation)

cd /usr/lib
for i in *alloc*.so
do
echo +++ $i +++
nm $i | grep GLOB | grep -v UNDEF | sed 's/^.*|//' | grep -v '^_' | \
sort -u | xargs echo
done

+++ libbsdmalloc.so +++
SUNW_1.1 free malloc realloc
+++ libmalloc.so +++
SUNW_1.1 calloc free malloc realloc
+++ libmapmalloc.so +++
SUNW_0.7 SUNW_1.1 SUNWprivate_1.1 calloc cfree free mallinfo malloc
mallopt memalign realloc valloc
+++ libmtmalloc.so +++
SUNW_1.1 SUNW_1.1.1 SUNWprivate_1.1 calloc free malloc mallocctl
memalign realloc valloc

I can't remember which system it was that had an undocumented call
that I had to reverse engineer from the pseudo-assembler.


Regards,
Nick Maclaren.

Nick Maclaren wrote:

>  As I said, this is (usually) within the ability of an experienced hacker.

I understand your point about that.

>  As far as Solaris etc. goes, you first have to find out which version
>  (or versions) of malloc the application is using, INCLUDING those
>  called by dynamic libraries.  That is unusually hard under Solaris,
>  because ldd doesn't do what you apparently thinks that it does, and
>  a fair amount of reverse engineering of the behaviour of the linker
>  and loader is needed.  Then you have to extract the relevant symbols,
>  see which library is using what, and decide if there is a clash.

However, I'm confused with the mention of "reverse engineering" in the
above paragraph. For Solaris, at least, things one needs to do are fairly
well documented[1]. It's also possible to automate the process, so one
doesn't have to do it again and again by hand.

Could you give me an example of something that required reverse
engineering?

[1] Occasionaly you can find bugs in linker software, but they are being
    promptly fixed, at least in my experience.

-- 
 .-.   .-.    Yes, I am an agent of Satan, but my duties are largely
(_  \ /  _)   ceremonial.
     |
     |        dave@fly.srk.fer.hr