I am using electric fence 2.1.13 to try to find a memory allocation problem that occurs after my application runs for about 3 hours. When I link to the electric fence library, I get "ElectricFence Exiting: mprotect() failed: Cannot allocate memory" during initialization. Could this be the source of the error that takes 3 hours to occur? I wonder because all I see at this point is a 12 byte malloc. According to a comment in efence.c, "On some systems it will be necessary to increase the amount of swap space in order to debug large programs that perform lots of allocation, because of the per-buffer overhead." How does one increase the amount of swap space? I am running Linux 2.6.26 on an MPC8248.
ElectricFence Exiting: mprotect() failed: Cannot allocate memory
Started by ●October 13, 2008
Reply by ●October 14, 20082008-10-14
On Oct 13, 7:32=A0pm, Bill <jobhunt...@aol.com> wrote:> I am using electric fence 2.1.13 to try to find a memory allocation > problem that occurs after my application runs for about 3 hours. =A0When > I link to the electric fence library, I get "ElectricFence Exiting: > mprotect() failed: Cannot allocate memory" during initialization. > Could this be the source of the error that takes 3 hours to occur? =A0I > wonder because all I see at this point is a 12 byte malloc.I doubt that's the source of the error that takes 3 hours to occur.> According to a comment in efence.c, "On some systems it will be > necessary to increase the amount of swap space in order to debug large > programs that perform lots of allocation, because of the per-buffer > overhead." =A0How does one increase the amount of swap space? =A0I am > running Linux 2.6.26 on an MPC8248.I would recommend doing invasive debugging on a test system with significant additional memory. It's hard to help you without knowing more about your hardware. Do you have a hard drive? Do you have USB ports? How much memory do you have? DS
Reply by ●October 14, 20082008-10-14
Bill <jobhunts02@aol.com> writes:> I am using electric fence 2.1.13 to try to find a memory allocation > problem that occurs after my application runs for about 3 hours.What kind of problem? Efence is good at finding memory corruption problems, not memory allocation problems.> When > I link to the electric fence library, I get "ElectricFence Exiting: > mprotect() failed: Cannot allocate memory" during initialization. > Could this be the source of the error that takes 3 hours to occur?Unlikely.> I wonder because all I see at this point is a 12 byte malloc.A *single* 12-byte malloc was performed by that point? If so, your copy of efence is misconfigured, miscompiled, or busted in some other way.> According to a comment in efence.c, "On some systems it will be > necessary to increase the amount of swap space in order to debug large > programs that perform lots of allocation, because of the per-buffer > overhead."Efence adds 1 page guard to every malloc. It is very rarely helpful in debugging non-toy applications. You may have better luck with Valgrind.> How does one increase the amount of swap space? I am > running Linux 2.6.26 on an MPC8248.Try "man swapon". Cheers, -- In order to understand recursion you must first understand recursion. Remove /-nsp/ for email.
Reply by ●October 14, 20082008-10-14
After about 3 hours, the program seg faults when trying to do a malloc 65K bytes. At the time, according to top, there is plenty of memory available I tried using valgrind but it slowed down my application so much that it was useless.. On Oct 13, 9:19=A0pm, Paul Pluzhnikov <ppluzhnikov-...@gmail.com> wrote:> Bill <jobhunt...@aol.com> writes: > > I am using electric fence 2.1.13 to try to find a memory allocation > > problem that occurs after my application runs for about 3 hours. > > What kind of problem? > Efence is good at finding memory corruption problems, not memory > allocation problems. > > > When > > I link to the electric fence library, I get "ElectricFence Exiting: > > mprotect() failed: Cannot allocate memory" during initialization. > > Could this be the source of the error that takes 3 hours to occur? > > Unlikely. > > > I wonder because all I see at this point is a 12 byte malloc. > > A *single* 12-byte malloc was performed by that point? > If so, your copy of efence is misconfigured, miscompiled, or busted > in some other way. > > > According to a comment in efence.c, "On some systems it will be > > necessary to increase the amount of swap space in order to debug large > > programs that perform lots of allocation, because of the per-buffer > > overhead." > > Efence adds 1 page guard to every malloc. > It is very rarely helpful in debugging non-toy applications. > You may have better luck with Valgrind. > > > How does one increase the amount of swap space? =A0I am > > running Linux 2.6.26 on an MPC8248. > > Try "man swapon". > > Cheers, > -- > In order to understand recursion you must first understand recursion. > Remove /-nsp/ for email.
Reply by ●October 14, 20082008-10-14
I have a total of 128 MB of flash on my target board. No USB ports. Monitoring top, it does not appear that memory is being leaked, but it is behaving as if running out of memory. Is there a better way than top to monitor memory? On Oct 13, 8:15=A0pm, David Schwartz <dav...@webmaster.com> wrote:> On Oct 13, 7:32=A0pm, Bill <jobhunt...@aol.com> wrote: > > > I am using electric fence 2.1.13 to try to find a memory allocation > > problem that occurs after my application runs for about 3 hours. =A0Whe=n> > I link to the electric fence library, I get "ElectricFence Exiting: > > mprotect() failed: Cannot allocate memory" during initialization. > > Could this be the source of the error that takes 3 hours to occur? =A0I > > wonder because all I see at this point is a 12 byte malloc. > > I doubt that's the source of the error that takes 3 hours to occur. > > > According to a comment in efence.c, "On some systems it will be > > necessary to increase the amount of swap space in order to debug large > > programs that perform lots of allocation, because of the per-buffer > > overhead." =A0How does one increase the amount of swap space? =A0I am > > running Linux 2.6.26 on an MPC8248. > > I would recommend doing invasive debugging on a test system with > significant additional memory. It's hard to help you without knowing > more about your hardware. Do you have a hard drive? Do you have USB > ports? How much memory do you have? > > DS
Reply by ●October 14, 20082008-10-14
On Mon, 13 Oct 2008 22:01:08 -0700 (PDT), Bill <jobhunts02@aol.com> wrote:>After about 3 hours, the program seg faults when trying to do a malloc >65K bytes. At the time, according to top, there is plenty of memory >availableApparently you do not have even 64 KiB of _contiguous_ virtual memory available, but only a huge number of smaller fragments all over the memory. I guess that the system would run a few hours longer, if the largest allocation was 8 KiB :-). Sounds like a typical dynamic memory fragmentation problem. The other alternative, if the stack and dynamic memory occupy the same memory area (one growing upwards and the other downwards) is that he stack size is constantly increasing due to a programming error, finally inhibiting the growth of the heap. Paul
Reply by ●October 14, 20082008-10-14
Bill <jobhunts02@aol.com> writes:> I have a total of 128 MB of flash on my target board. No USB ports. > Monitoring top, it does not appear that memory is being leaked, but it > is behaving as if running out of memory.Still assuming your description is correct, it is behaving as if the malloc-code made an invalid memory access because of a corrupted pointer inside the heap. But you can easily verify if the allocation should have succeeded, ie if there was a continuous area of at least 64K of 'unused VM' available: 1. Modify the segfault handler in the kernel to send a SIGSTOP instead of a SIGSEGV. 2. Use pmap to inspect the address space layout of the affected process after it has been stopped by the signal.
Reply by ●October 14, 20082008-10-14
> ... a memory allocation > problem that occurs after my application runs for about 3 hours.Under glibc, setting the shell environment variable "export MALLOC_CHECK_=2" [note the trailing underscore] performs additional internal consistency checks that are relatively inexpensive. Run "info libc" then search for MALLOC_CHECK_. man swapon # how to increase swap space. /proc/<pid>/maps reveals summary information for one process. /proc/<pid>/smaps reveals more details for one process. /proc/meminfo reports a system-wide summary. --
Reply by ●October 14, 20082008-10-14
John Reiser <jreiser@BitWagon.com> writes:>> ... a memory allocation >> problem that occurs after my application runs for about 3 hours. > > Under glibc, setting the shell environment variable "export MALLOC_CHECK_=2" > [note the trailing underscore] performs additional internal consistency checks > that are relatively inexpensive. Run "info libc" then search for MALLOC_CHECK_. > > man swapon # how to increase swap space.The OP is using a PPC-based SoC. I doubt that he has any swap space on board.
Reply by ●October 14, 20082008-10-14
Below is what pmap -x gives for the process (snmpd) upon failing at a call to malloc for 65536 bytes. Does anything here would indicate a possible problem trying to malloc 65536 bytes? It should be noted that a call to pmap -x before the failure while snmpd was still running gave identical results. Therefore, I wonder if the cause of the problem can be seen here? Address Kbytes RSS Anon Locked Mode Mapping 0f8b8000 64 - - - r-x-- libresolv-2.6.so 0f8c8000 252 - - - ----- libresolv-2.6.so 0f907000 4 - - - r---- libresolv-2.6.so 0f908000 4 - - - rwx-- libresolv-2.6.so 0f909000 8 - - - rwx-- [ anon ] 0f91b000 16 - - - r-x-- libnss_dns-2.6.so 0f91f000 252 - - - ----- libnss_dns-2.6.so 0f95e000 4 - - - r---- libnss_dns-2.6.so 0f95f000 4 - - - rwx-- libnss_dns-2.6.so 0f970000 40 - - - r-x-- libnss_files-2.6.so 0f97a000 252 - - - ----- libnss_files-2.6.so 0f9b9000 4 - - - r---- libnss_files-2.6.so 0f9ba000 4 - - - rwx-- libnss_files-2.6.so 0f9cb000 28 - - - r-x-- librt-2.6.so 0f9d2000 252 - - - ----- librt-2.6.so 0fa11000 4 - - - r---- librt-2.6.so 0fa12000 4 - - - rwx-- librt-2.6.so 0fa23000 1264 - - - r-x-- libc-2.6.so 0fb5f000 252 - - - ----- libc-2.6.so 0fb9e000 8 - - - r---- libc-2.6.so 0fba0000 12 - - - rwx-- libc-2.6.so 0fba3000 12 - - - rwx-- [ anon ] 0fbb6000 12 - - - r-x-- libEclipseHms.so 0fbb9000 252 - - - ----- libEclipseHms.so 0fbf8000 4 - - - rwx-- libEclipseHms.so 0fc09000 16 - - - r-x-- libEclipseVer.so 0fc0d000 252 - - - ----- libEclipseVer.so 0fc4c000 4 - - - rwx-- libEclipseVer.so 0fc5d000 12 - - - r-x-- libEclipsePai.so 0fc60000 252 - - - ----- libEclipsePai.so 0fc9f000 4 - - - rwx-- libEclipsePai.so 0fcb0000 8 - - - r-x-- libEclipseCil.so 0fcb2000 256 - - - ----- libEclipseCil.so 0fcf2000 4 - - - rwx-- libEclipseCil.so 0fd03000 12 - - - r-x-- libEclipseConf.so 0fd06000 256 - - - ----- libEclipseConf.so 0fd46000 4 - - - rwx-- libEclipseConf.so 0fd57000 16 - - - r-x-- libEclipseSem.so 0fd5b000 252 - - - ----- libEclipseSem.so 0fd9a000 4 - - - rwx-- libEclipseSem.so 0fdab000 12 - - - r-x-- libEclipseLog.so 0fdae000 256 - - - ----- libEclipseLog.so 0fdee000 4 - - - rwx-- libEclipseLog.so 0fdff000 8 - - - r-x-- libEclipseLst.so 0fe01000 252 - - - ----- libEclipseLst.so 0fe40000 4 - - - rwx-- libEclipseLst.so 0fe51000 80 - - - r-x-- libpthread-2.6.so 0fe65000 256 - - - ----- libpthread-2.6.so 0fea5000 4 - - - r---- libpthread-2.6.so 0fea6000 4 - - - rwx-- libpthread-2.6.so 0fea7000 8 - - - rwx-- [ anon ] 0feb9000 640 - - - r-x-- libm-2.6.so 0ff59000 252 - - - ----- libm-2.6.so 0ff98000 4 - - - r---- libm-2.6.so 0ff99000 12 - - - rwx-- libm-2.6.so 0ffac000 12 - - - r-x-- libdl-2.6.so 0ffaf000 252 - - - ----- libdl-2.6.so 0ffee000 4 - - - r---- libdl-2.6.so 0ffef000 4 - - - rwx-- libdl-2.6.so 10000000 1192 - - - r-x-- snmpd 10169000 32 - - - rwx-- snmpd 10171000 552 - - - rwx-- [ anon ] 30000000 116 - - - r-x-- ld-2.6.so 3001d000 24 - - - rw--- [ anon ] 30023000 4 - - - r--s- [ shmid=3D0x0 ] 30024000 4 - - - rw--- [ anon ] 30025000 4 - - - r--s- [ shmid=3D0x0 ] 3005c000 4 - - - r---- ld-2.6.so 3005d000 4 - - - rwx-- ld-2.6.so 3005e000 4 - - - ----- [ anon ] 3005f000 8188 - - - rw--- [ anon ] 3085e000 4 - - - ----- [ anon ] 3085f000 8188 - - - rw--- [ anon ] 7ff61000 332 - - - rw--- [ stack ] -------- ------- ------- ------- ------- total kB 25084 - - - On Oct 14, 1:43=A0am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:> Bill <jobhunt...@aol.com> writes: > > I have a total of 128 MB of flash on my target board. =A0No USB ports. > > Monitoring top, it does not appear that memory is being leaked, but it > > is behaving as if running out of memory. > > Still assuming your description is correct, it is behaving as if the > malloc-code made an invalid memory access because of a corrupted > pointer inside the heap. But you can easily verify if the allocation > should have succeeded, ie if there was a continuous area of at least > 64K of 'unused VM' available: > > =A0 =A0 1. Modify the segfault handler in the kernel to send a SIGSTOP > =A0 =A0 =A0 =A0instead of a SIGSEGV. > > =A0 =A0 2. Use pmap to inspect the address space layout of the affected > =A0 =A0 =A0 =A0process after it has been stopped by the signal.
