Linux Kernel speculation

We have a number of systems based on the ppc core in a Virtex 4 FX12.  The
system has 4MB flash RAM for program storage and 32MB RAM and uses one of
the EDK-supplied Ethernet cores for comms (temac).

Some time ago I ported a 2.6.15 kernel onto it and we've been running with
that ever since.  The system boots with a small bootloader which loads uboot
which in turn loads the Linux kernel.  The kernel has an initrd and boots
into a ram filesystem that's loaded off a compressed image on the flash.

The total boot time is of the order of 35 seconds - that's from a cold start
to the application running with tcp sockets open to receive stuff from the 
LAN.  I'd like to explore ways of making that faster.

It seems that the biggest pauses in the boot process are to do with the
initrd loading, then the flash RAM being decompressed into RAM for the
ramdisk, and then there's a long pause while the Etherent hardware 
initialises.  There's also about 5 seconds lost at the start while the 
system looks for keypresses to activate the two boot loaders' flash-ram
loading tools, and I'm going to replace both of those to look for a 
jumper setting on the board.

What's the most effective route to further reducing the boot time? For
example, is there anything I can do to get rid of the initrd, or to
speed up loading a flash disk into RAM?  Is there any benefit to going
for a later kernel - later desktop kernels seems to bood a lot faster
than older ones on the same hardware, but I'm unsure how much of that
is to do with more efficient driver and other module loading and
how muchg is to do with the basic kernel itself.  I've already removed
everything from the kernel that's not needed and everything that it needs
is compiled in, I believe.

I'm really looking for ideas for where to look next!  Well, that and to
get some feeling for if I'm going to be able to get a worthwile speed
return - like all these things there's a tradeoff between effort and worth.

Thanks in advance,
Nobby

Nobby Anderson wrote:

> The total boot time is of the order of 35 seconds - that's from a cold start
> to the application running with tcp sockets open to receive stuff from the 
> LAN.  I'd like to explore ways of making that faster.

35 seconds is long. You can reduce it to below 2 seconds:

http://www.embeddedarm.com/software/arm-linux-fastboot-ts7300.php

Looks like they use a proprietary boot loader, by maybe their modified
busybox helps you a bit:

http://www.embeddedarm.com/software/arm-linux-fastboot-ts7400.php

There are lots of other projects when searching with Google for faster
Linux booting. Some time ago I've read about an init.d replacements named
Upstart:

http://www.linux.com/archive/articles/57213

I don't know if it is good, but the description looks good.

If you don't need a full init system, you can write your own shell script
and let it execute as the init process, which starts only the required
daemons, initializes your network etc., maybe all in background, and then
your applications. I've done this for a product, much simpler than a full
init.d system for small embedded systems, which don't need different run
levels.

-- 
Frank Buss, fb@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de

On Thu, 21 May 2009 12:36:38 -0500, Nobby Anderson <nobby@invalid.invalid> wrote:
>We have a number of systems based on the ppc core in a Virtex 4 FX12.  The
>system has 4MB flash RAM for program storage and 32MB RAM and uses one of
>the EDK-supplied Ethernet cores for comms (temac).

>Some time ago I ported a 2.6.15 kernel onto it and we've been running with
>that ever since.  The system boots with a small bootloader which loads uboot
>which in turn loads the Linux kernel.  The kernel has an initrd and boots
>into a ram filesystem that's loaded off a compressed image on the flash.

>The total boot time is of the order of 35 seconds - that's from a cold start
>to the application running with tcp sockets open to receive stuff from the 
>LAN.  I'd like to explore ways of making that faster.

>It seems that the biggest pauses in the boot process are to do with the
>initrd loading, then the flash RAM being decompressed into RAM for the
>ramdisk, and then there's a long pause while the Etherent hardware 
>initialises.  There's also about 5 seconds lost at the start while the 
>system looks for keypresses to activate the two boot loaders' flash-ram
>loading tools, and I'm going to replace both of those to look for a 
>jumper setting on the board.

>What's the most effective route to further reducing the boot time? For
>example, is there anything I can do to get rid of the initrd, or to
>speed up loading a flash disk into RAM?  Is there any benefit to going
>for a later kernel - later desktop kernels seems to bood a lot faster
>than older ones on the same hardware, but I'm unsure how much of that
>is to do with more efficient driver and other module loading and
>how muchg is to do with the basic kernel itself.  I've already removed
>everything from the kernel that's not needed and everything that it needs
>is compiled in, I believe.

Do you have unnecessary devices compiled in your kernel or as modules
in your initrd? 

I have a diskless multimedia computer that boots off a LAN and it
takes <25 seconds to boot to a gui application.  Without the gui, it
would be <20 seconds, for example the ssh server is available in 15
seconds from bios start.

Frank Buss <fb@frank-buss.de> wrote:
> Nobby Anderson wrote:
>
>> The total boot time is of the order of 35 seconds - that's from a cold start
>> to the application running with tcp sockets open to receive stuff from the 
>> LAN.  I'd like to explore ways of making that faster.
>
> 35 seconds is long. You can reduce it to below 2 seconds:
>
> http://www.embeddedarm.com/software/arm-linux-fastboot-ts7300.php
>
> Looks like they use a proprietary boot loader, by maybe their modified
> busybox helps you a bit:
>
> http://www.embeddedarm.com/software/arm-linux-fastboot-ts7400.php
>
> There are lots of other projects when searching with Google for faster
> Linux booting. Some time ago I've read about an init.d replacements named
> Upstart:
>
> http://www.linux.com/archive/articles/57213
>
> I don't know if it is good, but the description looks good.
>
> If you don't need a full init system, you can write your own shell script
> and let it execute as the init process, which starts only the required
> daemons, initializes your network etc., maybe all in background, and then
> your applications. I've done this for a product, much simpler than a full
> init.d system for small embedded systems, which don't need different run
> levels.
>
Thanks for the info, I'd love a 2 second boot time!

None of the existing time is taken in init, the init script here just
configures the Etherent and starts the application.  It's all in the
kernel/ramdisk loading.  I'm not sure how I could speed up the loading 
process - I suspect the ARM stuff in the links above doesn't use
compressed disk and kernel images for a start, which would help, but I only
have 4MB flash so I need to do that.

Nobby 

AZ Nomad <aznomad.3@PremoveOBthisOX.COM> wrote:
> Do you have unnecessary devices compiled in your kernel or as modules
> in your initrd? 
>
> I have a diskless multimedia computer that boots off a LAN and it
> takes <25 seconds to boot to a gui application.  Without the gui, it
> would be <20 seconds, for example the ssh server is available in 15
> seconds from bios start.

Yes, everything is compiled in.  The two main pauses are loading the 
initrd and then uncompressing the system ram disk.  Sadly seeing as
I only have 4MB flahs I have to compress everything including the kernel
image.

Nobby 

Nobby Anderson wrote:

> Yes, everything is compiled in.  The two main pauses are loading the 
> initrd and then uncompressing the system ram disk.  Sadly seeing as
> I only have 4MB flahs I have to compress everything including the kernel
> image.

Depends on your CPU, but loading the ramdisk and decompressing should be
fast, only some seconds. Maybe you can put some serial port debug outputs
in the kernel with timestamps, to see where most of the time is wasted?

-- 
Frank Buss, fb@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de

On May 21, 10:41=A0pm, Nobby Anderson <no...@invalid.invalid> wrote:
> Frank Buss <f...@frank-buss.de> wrote:
> > Nobby Anderson wrote:
>
> >> The total boot time is of the order of 35 seconds - that's from a cold=
 start
> >> to the application running with tcp sockets open to receive stuff from=
 the
> >> LAN. =A0I'd like to explore ways of making that faster.
>
> > 35 seconds is long. You can reduce it to below 2 seconds:
>
> >http://www.embeddedarm.com/software/arm-linux-fastboot-ts7300.php
>
> > Looks like they use a proprietary boot loader, by maybe their modified
> > busybox helps you a bit:
>
> >http://www.embeddedarm.com/software/arm-linux-fastboot-ts7400.php
>
> > There are lots of other projects when searching with Google for faster
> > Linux booting. Some time ago I've read about an init.d replacements nam=
ed
> > Upstart:
>
> >http://www.linux.com/archive/articles/57213
>
> > I don't know if it is good, but the description looks good.
>
> > If you don't need a full init system, you can write your own shell scri=
pt
> > and let it execute as the init process, which starts only the required
> > daemons, initializes your network etc., maybe all in background, and th=
en
> > your applications. I've done this for a product, much simpler than a fu=
ll
> > init.d system for small embedded systems, which don't need different ru=
n
> > levels.
>
> Thanks for the info, I'd love a 2 second boot time!
>
> None of the existing time is taken in init, the init script here just
> configures the Etherent and starts the application. =A0It's all in the
> kernel/ramdisk loading. =A0I'm not sure how I could speed up the loading
> process - I suspect the ARM stuff in the links above doesn't use
> compressed disk and kernel images for a start, which would help, but I on=
ly
> have 4MB flash so I need to do that.
>
> Nobby

Maybe use threaded code as a form of compression. Work out how long a
subroutine call return takes and then search for any code sequence
used more than n times where length makes for a compression
recursively, with longest common code first. Any flashy register
renaming in the compiled code will make for less compression of the
code this way, and grater cache usage if any cache is present.

cheers jacko

On Thu, 21 May 2009 16:43:43 -0500, Nobby Anderson wrote:

>> Do you have unnecessary devices compiled in your kernel or as modules
>> in your initrd? 
>>
>> I have a diskless multimedia computer that boots off a LAN and it
>> takes <25 seconds to boot to a gui application.  Without the gui, it
>> would be <20 seconds, for example the ssh server is available in 15
>> seconds from bios start.
> 
> Yes, everything is compiled in.

Then you shouldn't need an initrd. The rationale behind initrd is so that
you can load modules which are needed for mounting the root filesystem
(e.g. IDE/SCSI/RAID/USB drivers, filesystem modules).

For an embedded system, you can just build a kernel with all of the
relevant drivers built in.

Also, filesystems such as cramfs and squashfs decompress data on-demand,
rather than decompressing the entire image into RAM. This reduces the
start-up time and memory usage, but results in slower access.

Nobody <nobody@nowhere.com> wrote:
> On Thu, 21 May 2009 16:43:43 -0500, Nobby Anderson wrote:
>
>>> Do you have unnecessary devices compiled in your kernel or as modules
>>> in your initrd? 
>>>
>>> I have a diskless multimedia computer that boots off a LAN and it
>>> takes <25 seconds to boot to a gui application.  Without the gui, it
>>> would be <20 seconds, for example the ssh server is available in 15
>>> seconds from bios start.
>> 
>> Yes, everything is compiled in.
>
> Then you shouldn't need an initrd. The rationale behind initrd is so that
> you can load modules which are needed for mounting the root filesystem
> (e.g. IDE/SCSI/RAID/USB drivers, filesystem modules).

Sorry, I had confused myself.  There isn't an inird, that was in earlier
versions and it had not registered that it was no longer there because
I still get a kernal boot message "Freeing initrd memory".  However there
is no initrd which is what I expect seeing as everything is compiled
in.

> For an embedded system, you can just build a kernel with all of the
> relevant drivers built in.
>
> Also, filesystems such as cramfs and squashfs decompress data on-demand,
> rather than decompressing the entire image into RAM. This reduces the
> start-up time and memory usage, but results in slower access.

OK, thanks, they're worth looking at.  I re-timed the boot, and it's
just under 30 seconds.  Six are in the initial countdowns for the low
level bootloader and uboot, both of which I can get rid of, about six
are taken uncompressing the initrd and six goes initialising the 
temac ethernet adapter. Uboot also checks the compressed ram image
which takes a couple of seconds and I could dump that - the system will
either work or not, if the ramdisk image is corrupt there is no benefit
in knowing early on, not a lot I cna do about it.  If I can get rid of 
the ramdisk decompression time that would be worthwhile, so I'll definitely
look at the other filesystems you mentioned.  Once the application is
running it uses no disk at all.

Thanks,
Nobby

>Frank Buss <fb@frank-buss.de> wrote:
>> Nobby Anderson wrote:
>>
>>> The total boot time is of the order of 35 seconds - that's from a cold
start
>>> to the application running with tcp sockets open to receive stuff from
the 
>>> LAN.  I'd like to explore ways of making that faster.
>>
>> 35 seconds is long. You can reduce it to below 2 seconds:
>>
>> http://www.embeddedarm.com/software/arm-linux-fastboot-ts7300.php
>>
>> Looks like they use a proprietary boot loader, by maybe their modified
>> busybox helps you a bit:
>>
>> http://www.embeddedarm.com/software/arm-linux-fastboot-ts7400.php
>>
>> There are lots of other projects when searching with Google for faster
>> Linux booting. Some time ago I've read about an init.d replacements
named
>> Upstart:
>>
>> http://www.linux.com/archive/articles/57213
>>
>> I don't know if it is good, but the description looks good.
>>
>> If you don't need a full init system, you can write your own shell
script
>> and let it execute as the init process, which starts only the required
>> daemons, initializes your network etc., maybe all in background, and
then
>> your applications. I've done this for a product, much simpler than a
full
>> init.d system for small embedded systems, which don't need different
run
>> levels.
>>
>Thanks for the info, I'd love a 2 second boot time!
>
>None of the existing time is taken in init, the init script here just
>configures the Etherent and starts the application.  It's all in the
>kernel/ramdisk loading.  I'm not sure how I could speed up the loading 
>process - I suspect the ARM stuff in the links above doesn't use
>compressed disk and kernel images for a start, which would help, but I
only
>have 4MB flash so I need to do that.

Get rid of U-Boot. Have your bootrom load your kernel in RAM and jump
right into it. Then disable the interface for modifying the kernel
arguments. Finally boot to an initial ramdisk, drop the user to the shell
while you do other things in the background. This should get you to under a
5 second boot time.

Regards,
Eddie