Verifying SD Cards

On 5/4/2016 1:41 PM, Theo Markettos wrote:
> In comp.sys.raspberry-pi rickman <gnuarm@gmail.com> wrote:
>> Seems worse than that, the Raspberry Pi is not SDXC/EXFAT compatible.
>> SDXC means sizes larger than 32 GB.  Those devices are formatted as
>> exFAT and so won't boot from that partition.  But they can be booted by
>> creating more than one partition with the first being FAT32.  I think
>> the other partition can be exFAT.
>
> SDXC is a card type.  You have a different initialisation procedure for
> SD or SDHC or SDXC.  The Pi supports SDXC.
>
> exFAT is a disc format.  If you're using Debian you need to install
> exfat-fuse.  Or you can reformat to something more sensible.
>
> The Pi boot partition has to be FAT.  exFAT is not FAT.  Supporting exFAT
> would require a) a more complex format to be implemented in the relatively
> small mask ROM on the SoC, b) making that implementation bomb-proof in the
> face of all the strange cards out there and c) paying a patent licence to
> Microsoft for every Pi shipped.  The simpler and cheaper option is not to
> use exFAT.

Hmmm... you've said the Pi boot partition has to be "FAT" which is what 
I said.  You've said twice to not use exFAT, but you don't say what 
*can* be used.  If FAT32 were to be used for all partitions on an SDXC 
card, it would require four partitions for a 128 GB card.  That pretty 
well sucks.  Even needing two partitions sucks.  Would NTFS work?


>> I don't see anything about a speed limitation.  In fact, UHS-1 has a
>> speed of 13 MB/s and speeds higher than this have already been measured
>> on the rPi.  So what about the rPi would limit UHS-1 performance?
>>
>> "UHS-I has a bus interface speed of up to 104 MB/s."
>
> You can set the emmc_clock in config.txt - it's usually 100MHz.  I think
> that will be the upper limit on speed, but I don't remember all the modes
> that are listed in the spec (there is a 208 MHz mode in there I think, which
> is where 104MB/s comes from).  Since I haven't seen the Arasan IP core
> documentation I don't know what the hardware actually supports.

I misread the quote I provided.  I thought it was bits rather than 
bytes.  So that's not a limitation I think.


>> I don't get why they keep increasing the size capabilities of the SD
>> cards so incrementally.  SDHC only covered 4 GB to 32 GB expanding the
>> size 16 fold.  SDXC expands the range another 64 fold.  Why so little?
>> Memory capacity will continue to ramp up doubling every two years, so
>> why not give the standard some legs?  SDXC will need to be replaced in
>> another 8 to 10 years.
>
> This irritates me too.  Lack of support can reduce the longevity of existing
> devices (it's now getting trickier to buy 2GB cards to work on old non-SDHC
> devices, for instance).
>
> The initialisation process is 'just software' so it's possible to upgrade an
> existing device to support SDHC/SDXC/etc, but in many things (cameras, USB
> readers etc) the software doesn't get upgraded.
>
> Theo
> (who has a vendor-supplied SD IP core which does initialisation in hardware,
> which makes supporting SDHC much more painful)

I assume the SDHC cards topped out at 32 GB because they used FAT32 
which also tops out at 32 GB.  Rather than switch the format used, they 
just worked within that limit and postponed the bigger change for SDXC 
cards until later.

-- 

Rick C

On 5/4/2016 2:15 PM, rickman wrote:
> On 5/4/2016 1:41 PM, Theo Markettos wrote:
>> In comp.sys.raspberry-pi rickman <gnuarm@gmail.com> wrote:
>>> Seems worse than that, the Raspberry Pi is not SDXC/EXFAT compatible.
>>> SDXC means sizes larger than 32 GB.  Those devices are formatted as
>>> exFAT and so won't boot from that partition.  But they can be booted by
>>> creating more than one partition with the first being FAT32.  I think
>>> the other partition can be exFAT.
>>
>> SDXC is a card type.  You have a different initialisation procedure for
>> SD or SDHC or SDXC.  The Pi supports SDXC.
>>
>> exFAT is a disc format.  If you're using Debian you need to install
>> exfat-fuse.  Or you can reformat to something more sensible.
>>
>> The Pi boot partition has to be FAT.  exFAT is not FAT.  Supporting exFAT
>> would require a) a more complex format to be implemented in the
>> relatively
>> small mask ROM on the SoC, b) making that implementation bomb-proof in
>> the
>> face of all the strange cards out there and c) paying a patent licence to
>> Microsoft for every Pi shipped.  The simpler and cheaper option is not to
>> use exFAT.
>
> Hmmm... you've said the Pi boot partition has to be "FAT" which is what
> I said.  You've said twice to not use exFAT, but you don't say what
> *can* be used.  If FAT32 were to be used for all partitions on an SDXC
> card, it would require four partitions for a 128 GB card.  That pretty
> well sucks.  Even needing two partitions sucks.  Would NTFS work?
>
>
>>> I don't see anything about a speed limitation.  In fact, UHS-1 has a
>>> speed of 13 MB/s and speeds higher than this have already been measured
>>> on the rPi.  So what about the rPi would limit UHS-1 performance?
>>>
>>> "UHS-I has a bus interface speed of up to 104 MB/s."
>>
>> You can set the emmc_clock in config.txt - it's usually 100MHz.  I think
>> that will be the upper limit on speed, but I don't remember all the modes
>> that are listed in the spec (there is a 208 MHz mode in there I think,
>> which
>> is where 104MB/s comes from).  Since I haven't seen the Arasan IP core
>> documentation I don't know what the hardware actually supports.
>
> I misread the quote I provided.  I thought it was bits rather than
> bytes.  So that's not a limitation I think.
>
>
>>> I don't get why they keep increasing the size capabilities of the SD
>>> cards so incrementally.  SDHC only covered 4 GB to 32 GB expanding the
>>> size 16 fold.  SDXC expands the range another 64 fold.  Why so little?
>>> Memory capacity will continue to ramp up doubling every two years, so
>>> why not give the standard some legs?  SDXC will need to be replaced in
>>> another 8 to 10 years.
>>
>> This irritates me too.  Lack of support can reduce the longevity of
>> existing
>> devices (it's now getting trickier to buy 2GB cards to work on old
>> non-SDHC
>> devices, for instance).
>>
>> The initialisation process is 'just software' so it's possible to
>> upgrade an
>> existing device to support SDHC/SDXC/etc, but in many things (cameras,
>> USB
>> readers etc) the software doesn't get upgraded.
>>
>> Theo
>> (who has a vendor-supplied SD IP core which does initialisation in
>> hardware,
>> which makes supporting SDHC much more painful)
>
> I assume the SDHC cards topped out at 32 GB because they used FAT32
> which also tops out at 32 GB.  Rather than switch the format used, they
> just worked within that limit and postponed the bigger change for SDXC
> cards until later.

I'm a bit confused.  I just looked it up and FAT32 has an upper limit of 
2 TB!  So why can't FAT32 be used for one partition holding all of a 64 
or 128 GB SD card?

-- 

Rick C

On 5/4/2016 2:21 PM, rickman wrote:
> On 5/4/2016 2:15 PM, rickman wrote:
>>
>> I assume the SDHC cards topped out at 32 GB because they used FAT32
>> which also tops out at 32 GB.  Rather than switch the format used, they
>> just worked within that limit and postponed the bigger change for SDXC
>> cards until later.
>
> I'm a bit confused.  I just looked it up and FAT32 has an upper limit of
> 2 TB!  So why can't FAT32 be used for one partition holding all of a 64
> or 128 GB SD card?

I did a little digging and there is some sort of functional limit to 
FAT32 at 32 GB but it seems to mostly be Microsoft's laziness by not 
supporting the creation of larger partitions with their built in tools, 
or maybe just some of them.  I found a utility that let me format the 64 
GB SDXC card to FAT32.

So will I be able to boot a raspberry Pi from this SDXC card this way?

http://arius.com/photos/FAT32_64GB.png

-- 

Rick C

On 5/4/2016 11:46 AM, rickman wrote:
> On 5/4/2016 2:21 PM, rickman wrote:
>> On 5/4/2016 2:15 PM, rickman wrote:
>>>
>>> I assume the SDHC cards topped out at 32 GB because they used FAT32
>>> which also tops out at 32 GB.  Rather than switch the format used, they
>>> just worked within that limit and postponed the bigger change for SDXC
>>> cards until later.
>>
>> I'm a bit confused.  I just looked it up and FAT32 has an upper limit of
>> 2 TB!  So why can't FAT32 be used for one partition holding all of a 64
>> or 128 GB SD card?
>
> I did a little digging and there is some sort of functional limit to
> FAT32 at 32 GB but it seems to mostly be Microsoft's laziness by not
> supporting the creation of larger partitions with their built in tools,
> or maybe just some of them.  I found a utility that let me format the 64
> GB SDXC card to FAT32.
>
> So will I be able to boot a raspberry Pi from this SDXC card this way?
>
> http://arius.com/photos/FAT32_64GB.png
>
Isn't the FAT32 limit due to address space limitations?
If you want bigger partitions, you need to have bigger allocation units.
Wastes a lot of space if you have many small files.

In comp.sys.raspberry-pi rickman <gnuarm@gmail.com> wrote:
> Hmmm... you've said the Pi boot partition has to be "FAT" which is what 
> I said.  You've said twice to not use exFAT, but you don't say what 
> *can* be used.  If FAT32 were to be used for all partitions on an SDXC 
> card, it would require four partitions for a 128 GB card.  That pretty 
> well sucks.  Even needing two partitions sucks.  Would NTFS work?

You need to format the boot partition with FAT (16 or 32).  You can make     
additional partition(s) in whatever format you like.
  
The standard Raspbian install has a small boot partition in FAT and the rest
of the SD card is ext3 or ext4.  This model is fairly common in PC Linux
installs - a basic ext2/ext3 partition for booting, and then whatever crazy
RAID/LVM/ZFS/encrypted partition setup you want for the rest of the disc. 
It means you don't have to teach the bootloader how to understand your
encrypted RAID and prompt for your smartcard or whatever it might need, you
can do that once the kernel is running.
  
If you want to make additional data, rather than OS, partitions, NTFS or
exFAT would be fine.  You'd have to install the necessary packages to read  
them, though (ntfs-3g and exfat-fuse).

> I assume the SDHC cards topped out at 32 GB because they used FAT32 
> which also tops out at 32 GB.  Rather than switch the format used, they 
> just worked within that limit and postponed the bigger change for SDXC 
> cards until later.

I think I may have got it slightly wrong.

Between SD and SDHC is a command-level difference.  It isn't just a question
of formatting, the firmware responds differently.  This is the flowchart:
http://www.chlazza.net/imgs/SDcardInitFlowchart_3.01.png
- the left branch is for SD, the right branch for SDHC (where you find out
capacity at the end)
 
Between SDHC and SDXC I can't see any protocol differences (not that I've
actually interacted with an SDXC at the command level, hence my confusion).
So I think it's marketing that out of the box SDHC is formatted FAT32 and
SDXC is formatted exFAT.  Underneath they speak the same protocol and
there's nothing to stop you reformatting as one or the other.

(One slight wrinkle is that the wear levelling may be optimised for the
FAT32 or exFAT format - but that goes out the window when you're trying to
use the card for an OS anyway).

> I assume the SDHC cards topped out at 32 GB because they used FAT32 
> which also tops out at 32 GB.  Rather than switch the format used, they 
> just worked within that limit and postponed the bigger change for SDXC 
> cards until later.

You could be right there.  Windows not supporting formatting FAT32 > 32GB
might be the kind of marketing-inspired reason for the name change, rather
than any difference in substance of the actual hardware.

Theo

On Wed, 4 May 2016 14:21:52 -0400, rickman <gnuarm@gmail.com> wrote:

>On 5/4/2016 2:15 PM, rickman wrote:
>> On 5/4/2016 1:41 PM, Theo Markettos wrote:
>>> In comp.sys.raspberry-pi rickman <gnuarm@gmail.com> wrote:
>>>> Seems worse than that, the Raspberry Pi is not SDXC/EXFAT compatible.
>>>> SDXC means sizes larger than 32 GB.  Those devices are formatted as
>>>> exFAT and so won't boot from that partition.  But they can be booted by
>>>> creating more than one partition with the first being FAT32.  I think
>>>> the other partition can be exFAT.
>>>
>>> SDXC is a card type.  You have a different initialisation procedure for
>>> SD or SDHC or SDXC.  The Pi supports SDXC.
>>>
>>> exFAT is a disc format.  If you're using Debian you need to install
>>> exfat-fuse.  Or you can reformat to something more sensible.
>>>
>>> The Pi boot partition has to be FAT.  exFAT is not FAT.  Supporting exFAT
>>> would require a) a more complex format to be implemented in the
>>> relatively
>>> small mask ROM on the SoC, b) making that implementation bomb-proof in
>>> the
>>> face of all the strange cards out there and c) paying a patent licence to
>>> Microsoft for every Pi shipped.  The simpler and cheaper option is not to
>>> use exFAT.
>>
>> Hmmm... you've said the Pi boot partition has to be "FAT" which is what
>> I said.  You've said twice to not use exFAT, but you don't say what
>> *can* be used.  If FAT32 were to be used for all partitions on an SDXC
>> card, it would require four partitions for a 128 GB card.  That pretty
>> well sucks.  Even needing two partitions sucks.  Would NTFS work?
>>
>>
>>>> I don't see anything about a speed limitation.  In fact, UHS-1 has a
>>>> speed of 13 MB/s and speeds higher than this have already been measured
>>>> on the rPi.  So what about the rPi would limit UHS-1 performance?
>>>>
>>>> "UHS-I has a bus interface speed of up to 104 MB/s."
>>>
>>> You can set the emmc_clock in config.txt - it's usually 100MHz.  I think
>>> that will be the upper limit on speed, but I don't remember all the modes
>>> that are listed in the spec (there is a 208 MHz mode in there I think,
>>> which
>>> is where 104MB/s comes from).  Since I haven't seen the Arasan IP core
>>> documentation I don't know what the hardware actually supports.
>>
>> I misread the quote I provided.  I thought it was bits rather than
>> bytes.  So that's not a limitation I think.
>>
>>
>>>> I don't get why they keep increasing the size capabilities of the SD
>>>> cards so incrementally.  SDHC only covered 4 GB to 32 GB expanding the
>>>> size 16 fold.  SDXC expands the range another 64 fold.  Why so little?
>>>> Memory capacity will continue to ramp up doubling every two years, so
>>>> why not give the standard some legs?  SDXC will need to be replaced in
>>>> another 8 to 10 years.
>>>
>>> This irritates me too.  Lack of support can reduce the longevity of
>>> existing
>>> devices (it's now getting trickier to buy 2GB cards to work on old
>>> non-SDHC
>>> devices, for instance).
>>>
>>> The initialisation process is 'just software' so it's possible to
>>> upgrade an
>>> existing device to support SDHC/SDXC/etc, but in many things (cameras,
>>> USB
>>> readers etc) the software doesn't get upgraded.
>>>
>>> Theo
>>> (who has a vendor-supplied SD IP core which does initialisation in
>>> hardware,
>>> which makes supporting SDHC much more painful)
>>
>> I assume the SDHC cards topped out at 32 GB because they used FAT32
>> which also tops out at 32 GB.  Rather than switch the format used, they
>> just worked within that limit and postponed the bigger change for SDXC
>> cards until later.
>
>I'm a bit confused.  I just looked it up and FAT32 has an upper limit of 
>2 TB!  So why can't FAT32 be used for one partition holding all of a 64 
>or 128 GB SD card?

Use the Windows 98 format utility.  XP did not support more than 32G
on FAT32.
<https://technet.microsoft.com/en-us/library/cc938432.aspx>

On Wed, 04 May 2016 12:24:16 -0700, mike <ham789@netzero.net> wrote:

>On 5/4/2016 11:46 AM, rickman wrote:
>> On 5/4/2016 2:21 PM, rickman wrote:
>>> On 5/4/2016 2:15 PM, rickman wrote:
>>>>
>>>> I assume the SDHC cards topped out at 32 GB because they used FAT32
>>>> which also tops out at 32 GB.  Rather than switch the format used, they
>>>> just worked within that limit and postponed the bigger change for SDXC
>>>> cards until later.
>>>
>>> I'm a bit confused.  I just looked it up and FAT32 has an upper limit of
>>> 2 TB!  So why can't FAT32 be used for one partition holding all of a 64
>>> or 128 GB SD card?
>>
>> I did a little digging and there is some sort of functional limit to
>> FAT32 at 32 GB but it seems to mostly be Microsoft's laziness by not
>> supporting the creation of larger partitions with their built in tools,
>> or maybe just some of them.  I found a utility that let me format the 64
>> GB SDXC card to FAT32.
>>
>> So will I be able to boot a raspberry Pi from this SDXC card this way?
>>
>> http://arius.com/photos/FAT32_64GB.png
>>
>Isn't the FAT32 limit due to address space limitations?
>If you want bigger partitions, you need to have bigger allocation units.
>Wastes a lot of space if you have many small files.


FAT32 has a 2**32 sector limitation, and a 2**28 cluster limitation.
At 2TB, you'd need 8KB clusters.

The 2**32 sector limitation is trivial, and exists in a single 32-bit
field in the boot sector (and it's backup), should someone patch*
around that, you could then have 2**28 clusters of 32 or 64KB (most
stuff will support the former, a lot of stuff will break with the
latter), or 8/16 TB.

The 28-bit cluster count field is fairly ingrained in FAT32, but
probably would not be hard to tweak to about 30** bits (the reserved
bits are not well used).  Combined with the sector count fix, you
could have 32 or 64TB volumes.

Of course, either change would require everyone to agree to use it if
the portability of FAT32 volumes is actually of interest.  The other
question is whether you'd actually want to.  FAT is a pretty bad
format for modern sized storage devices - random accesses are painful,
you're limited to 4GB files, the directories are a mess if you use
long filenames, and it's not particularly robust.  Its one saving
grace is universal support.


*Presumably you could use the same technique that's use to support the
*32* bit sector count - if the 16-bit sector count field is zero, use
the (newer) 32-bit field.  If that's zero, use a yet bigger field.

**More is possible, 31 should also be fairly easy, going to (most) of
a 32-bit number should be possible too, but would require a more
serious change to FAT management logic.

On 5/4/2016 3:24 PM, mike wrote:
> On 5/4/2016 11:46 AM, rickman wrote:
>> On 5/4/2016 2:21 PM, rickman wrote:
>>> On 5/4/2016 2:15 PM, rickman wrote:
>>>>
>>>> I assume the SDHC cards topped out at 32 GB because they used FAT32
>>>> which also tops out at 32 GB.  Rather than switch the format used, they
>>>> just worked within that limit and postponed the bigger change for SDXC
>>>> cards until later.
>>>
>>> I'm a bit confused.  I just looked it up and FAT32 has an upper limit of
>>> 2 TB!  So why can't FAT32 be used for one partition holding all of a 64
>>> or 128 GB SD card?
>>
>> I did a little digging and there is some sort of functional limit to
>> FAT32 at 32 GB but it seems to mostly be Microsoft's laziness by not
>> supporting the creation of larger partitions with their built in tools,
>> or maybe just some of them.  I found a utility that let me format the 64
>> GB SDXC card to FAT32.
>>
>> So will I be able to boot a raspberry Pi from this SDXC card this way?
>>
>> http://arius.com/photos/FAT32_64GB.png
>>
> Isn't the FAT32 limit due to address space limitations?
> If you want bigger partitions, you need to have bigger allocation units.
> Wastes a lot of space if you have many small files.

That's not a limit.  That's a tradeoff.  I suppose if you had millions 
of small files it would waste a lot of space.  Up to that point it's a 
macht nichts or mox nix, your choice.

The point is there is nothing to prevent the use of a single bootable 
partition on SDXC cards.

I really don't like partitioning mass storage into multiple smaller 
partitions.

-- 

Rick C

Hi rickman,

> I'm a bit confused.  I just looked it up and FAT32 has an upper limit of
> 2 TB!  So why can't FAT32 be used for one partition holding all of a 64
> or 128 GB SD card?

My BENQ GH700 camera works well with a FAT32 formated (faked) 64 GB SDXC 
(branded fom save, but I#m very sure it isn't true, because it is tooo 
slow but holds 64 GB)
The only drawback, The camera can make videos with 1080p FHD with 60 
fps, but not the SD-card. Next limit: FAT32 only supports limitrd 
filesize. So I have to split the video sequences. Sometimes it is very 
ugly because I miss a few seconds every few minutes :-(

Regards

Marte

Hi Theo,

 > The standard Raspbian install has a small boot partition in FAT and 
the rest
> of the SD card is ext3 or ext4.

Thats the same I did with my fairphone. Officially they only support 32 
GM SD-cards. I use a 64 GB wit a first partition with 20 GB and tge 
second with the rest using ext4. Using Apps2SD or link2SD you can use 
the ext4 partition pretty like internal flash.

Marte