Linux boot drive issue

On 20/04/14 19:25, edward.ming.lee@gmail.com wrote:
>
>> There is good reason why these are all labelled "sd" these days -
>> they are all part of the same subsystem and are thus treated the
>> same, with all the advantages that stem from that.
>
> PATA and SATA drivers are separated from USB MSD.  There is no reason
> for them to be named the same.

No, they are not separated - they make use of the same SCSI subsystem. 
Originally there was more separation, with PATA totally separate from 
SCSI, but they were rationalised and collected together.  Since the SCSI 
subsystem was more advanced, PATA and SATA were moved into it and then 
the USB MSD devices joined in.

Obviously they devices use separate transport layers (USB, SATA, etc.). 
  But other than that, they are handled very much the same.

There are many advantages in this.  The most obvious example is when you 
have a USB-connected harddisk - it can be treated like an ordinary disk, 
with the same kind of mounting, the same filesystems, the same 
flexibility for swap partitions, use in RAID setups or LVM arrays, etc.

In Linux, it is an important part of the philosophy that you have 
distinct layers, and within each layer different devices are treated the 
same when possible.  So block devices are treated the same, whether they 
are hard disks, network disk blocks, file loopback devices, RAID arrays, 
etc.  And disk storage is treated the same, whether it is a hard disk, a 
USB stick, a network storage, etc.

>
>> Hard disks on SATA get enumerated first -
>
> I have instances where the SATA drive got initialized after the UMSD.
> UMSD is already initalized because it is the boot drive.
>

So what?  No one ever gets these mixed up.  If you don't understand the 
low-level details, just trust the installer - it will not get it wrong. 
  (I encourage you to learn about the underlying implementation, if you 
are able, because it is interesting to know what goes on behind the 
scenes.  But it is certainly not essential.)

>> installation media such as CD, DVD or USB come later.  And your
>> installer knows fine which device it is running from.
>
> CD/DVD are named /dev/cdrom, not /dev/sd*
>

No, CD/DVD devices are named /dev/sr* - i.e., SCSI Readonly (rather than 
SCSI Disk).  Names such as /dev/cdrom, /dev/cdrw, /dev/dvd, etc., are 
all symbolic links for user convenience.

> There are many advantages in this.  The most obvious example is when you 
> have a USB-connected harddisk - it can be treated like an ordinary disk, 
> with the same kind of mounting, the same filesystems, the same 
> flexibility for swap partitions, use in RAID setups or LVM arrays, etc.

No argument about the common layer.  But the entry layer is different.

> 
> So what?  No one ever gets these mixed up.  

Imagine a removable boot disk.  There could be zero, one or two disk ahead of it.  So, should it be sda, sdb or sdc?

> If you don't understand the 
> low-level details, just trust the installer

I am trying to build the installer.

> >> installation media such as CD, DVD or USB come later.  And your
> >> installer knows fine which device it is running from.
> > CD/DVD are named /dev/cdrom, not /dev/sd*
 
> No, CD/DVD devices are named /dev/sr* - i.e., SCSI Readonly (rather than 
> SCSI Disk).  Names such as /dev/cdrom, /dev/cdrw, /dev/dvd, etc., are 
> all symbolic links for user convenience.

OK, my mistake.  It was dedicated device before.  By your logic, CD/DVD uses the same common layer, but with a different name than fixed drive.  Why can't we have a different name for removal drive?

On 21/04/14 16:35, edward.ming.lee@gmail.com wrote:
>
>> There are many advantages in this.  The most obvious example is
>> when you have a USB-connected harddisk - it can be treated like an
>> ordinary disk, with the same kind of mounting, the same
>> filesystems, the same flexibility for swap partitions, use in RAID
>> setups or LVM arrays, etc.
>
> No argument about the common layer.  But the entry layer is
> different.
>
>>
>> So what?  No one ever gets these mixed up.
>
> Imagine a removable boot disk.  There could be zero, one or two disk
> ahead of it.  So, should it be sda, sdb or sdc?
>
>> If you don't understand the low-level details, just trust the
>> installer
>
> I am trying to build the installer.
>
>>>> installation media such as CD, DVD or USB come later.  And
>>>> your installer knows fine which device it is running from.
>>> CD/DVD are named /dev/cdrom, not /dev/sd*
>
>> No, CD/DVD devices are named /dev/sr* - i.e., SCSI Readonly (rather
>> than SCSI Disk).  Names such as /dev/cdrom, /dev/cdrw, /dev/dvd,
>> etc., are all symbolic links for user convenience.
>
> OK, my mistake.  It was dedicated device before.  By your logic,
> CD/DVD uses the same common layer, but with a different name than
> fixed drive.  Why can't we have a different name for removal drive?
>

If you are building your own installer, you can arrange for it to set up 
whatever naming rules you want.  Typically in a modern Linux 
distribution, you use udev rules for this.  If you want, you can give 
different types of devices different names - you don't /have/ to follow 
the standard ones used by everyone else.  And if you want, you can give 
additional symbolic link names to devices - just as is typically done 
with cdrom drives.

If you don't already understand this, you are not ready to build your 
own installer.

So I recommend doing some reading about device drivers, device naming, 
udev, and other base parts of a modern Linux system.  I recommend 
finding an appropriate Usenet group or other source of information and 
advice.  Get a VirtualBox installation, so that you can practice your 
installation.  Study existing installers such as the ones used by Redhat 
Fedora, Debian, Mint, Ubuntu, Gentoo, and Arch Linux.  Think long and 
hard about whether the world really needs a new Linux installer - or if 
you would be better off working with one of the existing ones and 
contributing useful changes there.

I don't want to discourage anyone from trying to learn by doing, or any 
other sort of learning, but it sounds to me that you are out of your 
depth.  Using the inevitable car analogy, you have come to this cycle 
club to complain about the state of car lighting systems, when you 
apparently want to build a rally car but have not yet learned to drive.

On Sat, 19 Apr 2014 20:06:34 -0700, edward.ming.lee wrote:

> On Saturday, April 19, 2014 3:43:47 PM UTC-7, Tim Wescott wrote:
>> On Sat, 19 Apr 2014 07:55:06 -0700, edward.ming.lee wrote:
>> 
>> > I am trying to make a bootable USB disk to install on a laptop.  In
>> > the old days, it was simple and can probably do it with less than 1G
>> > flash drive.  I could build on sda1 and install on hda1 (PATA).
>> > But, in the name of progress, things are more difficult.  It would
>> > take around 5G of flash.  Booting 5M kernel and 30M ram disk takes
>> > several minutes.  Building the kernel takes hours.  But the biggest
>> > problem is SATA.
>> 
>> > With a poorly made decision, IMHO, they named the SATA "sd" as well.
>> > So,
>> > now i have to make sdb1 and install on sda1, if i am lucky.  With
>> > both SATA and USB MSD fighting for the first "sd" name, results are
>> > unpredictable.  I get either sda1 UMSD and sdb1 SATA, or sdb1 UMSD
>> > and sda1 SATA.
>> 
>> > I am very tempted to, rip out the kernel and rename:
>> > 1. "hd" for SATA and "sd" for UMSD 2. "sd" for SATA and "ud" for UMSD
>> > 3.
>> > "hd" for SATA and "ud" for UMSD
>> > 
>> > What do you guys think?b
>> 
>> And yet Ubuntu manages to do it on a 700MB CD, still, or a 1GB USB
>> stick.
> 
> LinuxMint/Ubuntu kernel is slightly older (3.8) and not very useful by
> itself after installation.  After pulling in all the necessary packages,
> the working set is around 5G.  I have 8G SD to store all the files, so
> no need to do all the apt-get for every machine.
> 
> Latest kernel (3.13) is several times bigger, with lots of junks
> included.

My understanding of linux kernel compiles is that you are free to leave 
out much of the stuff you don't need, and thus can often make a custom 
kernel that is smaller (and faster) than one that includes everything up 
to the kitchen sink so it can work on any 'puter.

Are you talking about the image or all the source you need to compile the 
image?  I just checked, my biggest initrd.img file on this machine is 
14MB and change.  That's MB, _not_ GB.  And that's the initrd.img that 
ships with Ubuntu, not anything that's been slimmed-down.

>> Perhaps see if you can see what Cannonical does to make it work, then
>> shamelessly copy that?
> 
> I need to patch something in the kernel, so need to compile rather than
> copy.

Well, perhaps you should re-read what I said, then.  I didn't say to copy 
their _code_, I said to figure out their _method_ and learn from it.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

> 
> My understanding of linux kernel compiles is that you are free to leave 
> out much of the stuff you don't need, and thus can often make a custom 
> kernel that is smaller (and faster) than one that includes everything up 
> to the kitchen sink so it can work on any 'puter.

Yes, but the newest release is to compile everything as module.
I have to move files away, build the ramdisk, then move them
back to /lib/modules.

> Are you talking about the image or all the source you need to compile the 
> image?  I just checked, my biggest initrd.img file on this machine is 
> 14MB and change.  That's MB, _not_ GB.  And that's the initrd.img that 
> ships with Ubuntu, not anything that's been slimmed-down.

You probably have an older kernel.  
The released LinuxMint/Ubuntu is 3.8.0 with 30MB initrd.

Linux 3.13 compiles 1G bytes of modules, compressed to 200M.
It takes 20 minutes to boot on a modern laptop.  Most of the
modules are not used and they are immediately replaced by the
root file system anyway.  I managed to stripe it to 20M and
4 minutes to boot.  But it's a headache to compile and move
files back and forth.

Perhaps there should be a "b" option that build the boot modules,
in addition to the "m" option.

All these work because i am trying to replace one line in the kernel.
"Root drive is not ready, press 'S' to skip".
The USB root drive will be ready, just need a small delay.

Pressing a key to skip is no big deal, except when there is nobody
to press it in an embedded system.

On 21/04/14 23:46, edward.ming.lee@gmail.com wrote:
>
>>
>> My understanding of linux kernel compiles is that you are free to leave
>> out much of the stuff you don't need, and thus can often make a custom
>> kernel that is smaller (and faster) than one that includes everything up
>> to the kitchen sink so it can work on any 'puter.
>
> Yes, but the newest release is to compile everything as module.
> I have to move files away, build the ramdisk, then move them
> back to /lib/modules.
>
>> Are you talking about the image or all the source you need to compile the
>> image?  I just checked, my biggest initrd.img file on this machine is
>> 14MB and change.  That's MB, _not_ GB.  And that's the initrd.img that
>> ships with Ubuntu, not anything that's been slimmed-down.
>
> You probably have an older kernel.
> The released LinuxMint/Ubuntu is 3.8.0 with 30MB initrd.
>
> Linux 3.13 compiles 1G bytes of modules, compressed to 200M.
> It takes 20 minutes to boot on a modern laptop.  Most of the
> modules are not used and they are immediately replaced by the
> root file system anyway.  I managed to stripe it to 20M and
> 4 minutes to boot.  But it's a headache to compile and move
> files back and forth.

You are imagining things.

First, almost no one compiles /all/ the possible modules - even general 
purpose kernel compiles like Ubuntu will only compile a small fraction 
of the drivers and modules supported.  If you choose to compile 
/everything/, then you get /everything/.

Secondly, modules take no time at boot up - they are only loaded and 
used if they are needed.  That's the main idea of making them modules 
rather than statically linking them into the kernel.

>
> Perhaps there should be a "b" option that build the boot modules,
> in addition to the "m" option.
>
> All these work because i am trying to replace one line in the kernel.
> "Root drive is not ready, press 'S' to skip".
> The USB root drive will be ready, just need a small delay.
>
> Pressing a key to skip is no big deal, except when there is nobody
> to press it in an embedded system.
>

I recently built a 3.14 kernel for use in an embedded system.  I put 
quite a bit into the kernel itself, and it is 4.5 MB.  There are 2.3 MB 
of modules.  It boots in a few seconds on a 800 MHz ARM, and that is 
with the root filesystem on NFS.

On 21/04/14 22:46, edward.ming.lee@gmail.com wrote:
>
>>
>> My understanding of linux kernel compiles is that you are free to leave
>> out much of the stuff you don't need, and thus can often make a custom
>> kernel that is smaller (and faster) than one that includes everything up
>> to the kitchen sink so it can work on any 'puter.
>
> Yes, but the newest release is to compile everything as module.
> I have to move files away, build the ramdisk, then move them
> back to /lib/modules.
>
>> Are you talking about the image or all the source you need to compile the
>> image?  I just checked, my biggest initrd.img file on this machine is
>> 14MB and change.  That's MB, _not_ GB.  And that's the initrd.img that
>> ships with Ubuntu, not anything that's been slimmed-down.
>
> You probably have an older kernel.
> The released LinuxMint/Ubuntu is 3.8.0 with 30MB initrd.
>
> Linux 3.13 compiles 1G bytes of modules, compressed to 200M.
> It takes 20 minutes to boot on a modern laptop.

On a 3 year old dual processor machine, my xubuntu
3.2.0-60-generic x86_64 4.9MB kernel goes from o/s selection
screen to graphical login screen in 12s

I fail to see how your system is 100 times slower and
250 times larger.

> All these work because i am trying to replace one line in the kernel.
> "Root drive is not ready, press 'S' to skip".
> The USB root drive will be ready, just need a small delay.
>
> Pressing a key to skip is no big deal, except when there is nobody
> to press it in an embedded system.

Hence I presume you are interested neither in the /installation/
time nor the kernel /compilation/ time of an embedded system,
but are only interested in the boot time of an installed
system.

> You are imagining things.

Easy enough to check it.  Just download it from kernel.org and see for yourself.

> 
> First, almost no one compiles /all/ the possible modules - even general 
> purpose kernel compiles like Ubuntu will only compile a small fraction 
> of the drivers and modules supported.  If you choose to compile 
> /everything/, then you get /everything/.

I am just saying that the default .config compile almost everything.

> 
> Secondly, modules take no time at boot up

They goes into initrdisk, as part of building the kernel.

> - they are only loaded and 
> used if they are needed.  That's the main idea of making them modules 
> rather than statically linking them into the kernel.

This is only true for the main root file system.

 
> > Perhaps there should be a "b" option that build the boot modules,
> > in addition to the "m" option.
> 
> > All these work because i am trying to replace one line in the kernel.
> > "Root drive is not ready, press 'S' to skip".
> > The USB root drive will be ready, just need a small delay.
> > Pressing a key to skip is no big deal, except when there is nobody
> > to press it in an embedded system.
 
> I recently built a 3.14 kernel for use in an embedded system.  I put 
> quite a bit into the kernel itself, and it is 4.5 MB.  There are 2.3 MB 
> of modules.  It boots in a few seconds on a 800 MHz ARM, and that is 
> with the root filesystem on NFS.

Yes, your network would be a lot faster than USB.

> On a 3 year old dual processor machine, my xubuntu
> 3.2.0-60-generic x86_64 4.9MB kernel goes from o/s selection
> screen to graphical login screen in 12s

Mine is just as fast booting from SATA.

> I fail to see how your system is 100 times slower and
> 250 times larger.

Slow when booting from USB.

> Hence I presume you are interested neither in the /installation/

I am interested in building an installable USB flash drive, which would install a copy into the SATA drive.  All the settings should be established in advance.  No options/keys during installation.

> time nor the kernel /compilation/ time of an embedded system,

But i don't want to wait hours to compile.

> but are only interested in the boot time of an installed
> system.

Slow installation boot time is not critical, but still annoying.

On 22/04/14 23:15, edward.ming.lee@gmail.com wrote:
>
>> You are imagining things.
>
> Easy enough to check it.  Just download it from kernel.org and see for yourself.
>
>>
>> First, almost no one compiles /all/ the possible modules - even general
>> purpose kernel compiles like Ubuntu will only compile a small fraction
>> of the drivers and modules supported.  If you choose to compile
>> /everything/, then you get /everything/.
>
> I am just saying that the default .config compile almost everything.

No, the "default" .config does not compile almost everything - you have 
a choice of defaults, and these are just a starting point.  /Choose/ 
what you want to build.  /You/ are the one planning on writing your own 
installer - /you/ choose what modules to include.

It's like going to the supermarket.  There is a lot of choice, but you 
don't /have/ to buy one of every product in the shop!


>
>>
>> Secondly, modules take no time at boot up
>
> They goes into initrdisk, as part of building the kernel.

That's only true if you are building an initramfs, and even then you 
only include the modules you /need/ to have in the initramfs.  Your 
initramfs contains only those modules needed to get the main root 
filesystem mounted (such as drivers for the filesystems, and perhaps 
RAID, LVM, crypto, etc.), along with any user-space programs needed to 
get your real root mounted.

>
>> - they are only loaded and
>> used if they are needed.  That's the main idea of making them modules
>> rather than statically linking them into the kernel.
>
> This is only true for the main root file system.

Yes, which is why you put your extra modules in your main filesystem. 
You only put the pre-root essentials into your initramfs.

>
>
>>> Perhaps there should be a "b" option that build the boot modules,
>>> in addition to the "m" option.
>>
>>> All these work because i am trying to replace one line in the kernel.
>>> "Root drive is not ready, press 'S' to skip".
>>> The USB root drive will be ready, just need a small delay.
>>> Pressing a key to skip is no big deal, except when there is nobody
>>> to press it in an embedded system.
>
>> I recently built a 3.14 kernel for use in an embedded system.  I put
>> quite a bit into the kernel itself, and it is 4.5 MB.  There are 2.3 MB
>> of modules.  It boots in a few seconds on a 800 MHz ARM, and that is
>> with the root filesystem on NFS.
>
> Yes, your network would be a lot faster than USB.
>

No, a reasonable USB stick will do at least 10 MB/s, which is the same 
as you get with 100 Mbit Ethernet (at peak).  And even if you have the 
world's worst USB stick, the speed difference would not explain a boot 
time difference in the range of 100 between your claimed 20 minutes and 
my few seconds.