Mark Borgerson expounded in
news:MPG.270c0da2b301cc94989c72@news.eternal-september.org: 

>> From your "16MBytes of FAT" -
>> 
>> 16MB / 512 => 32 sectors.
> 
> How do you divide  (16 x 1024 x 1024) by 512  and get 32?
> 
> When I do that division I get 32768 sectors.

My bad. I was thinking 32k.

In any case, formatting a FAT is at least two orders
of magnitude less than formatting the whole 
drive/partition. Look at the ratios involved
(assuming FAT32):

128 cluster entries in one FAT sector,

 vs 

8 sectors per cluster (for example).

This is 1 sector vs 128 * 8 sectors, i.e.
a ratio of 1 : 1024 !

It's actually worse than that because if you do the 
whole drive you do FAT + clusters:

1 : 1024 + 1

So formatting a FAT takes 1/1025th the time it takes
to do the whole drive. Very significant, that.

Even if you dumbed it down to 1 sector per cluster
which nobody does since FAT12 floppies, you're looking
at a ratio of 1 : 128+1.

>> The only caveat would be if you tested for and found 
>> bad sectors. Then you'd also need to mark those clusters 
>> containing bad sectors as bad (unavailable) - FFF7 (FAT16).
> 
> I don't think  you even use FAT16 on a 32GB volume with a single 
> partition.
> 
> Mark Borgerson

The procedure is necessary and the same for both FAT16 and FAT32.

http://support.microsoft.com/kb/310561

"Windows XP supports the creation of primary partitions and logical 
drives of up to 4 gigabytes (GB) using the FAT16 file system. 
The maximum cluster size is 64K."

So you can see that the 32K limit was extended to 64K in XP. And
yes, FAT16 has its limits, but so what? Use FAT32. MS says
"FAT32 supports drives up to 2 terabytes in size.". FAT32 only
changes a few things (from FAT16) and is a simple upgrade to 
support (it was trivial for my own software to support both).

As far as long file names go, you don't have to support them in
your AVR code, unless you really really want to. I don't bother.

Warren

In article <Xns9E018EE612954WarrensBlatherings@85.214.73.210>, 
ve3wwg@gmail.com says...
> Mark Borgerson expounded in
> news:MPG.270bb366405ce964989c70@news.eternal-september.org: 
> 
> > In article <Xns9E016F015CB2EWarrensBlatherings@85.214.73.210>, 
> > ve3wwg@gmail.com says...
> >> Writing 32 sectors isn't huge (32k/512 bytes each).  Comparing
> >> it to a file write, if you use 8 sectors per cluster, it is only 
> >> two clusters worth of writing.
> > 
> > I don't understand this.
> > 
> > IIRC,  a cluster can be no more than 32KB for Windows and FAT32.
> 
> See:
> 
> http://www.maverick-os.dk/FileSystemFormats/FAT16
> _FileSystem.html#SectorsPerCluster
> 
> "Sectors Per Cluster: This is the number of sectors per cluster. The 
> allowed values are: 1, 2, 4, 8, 16, 32 or 128. But de-facto is that most 
> combinations of 'BytesPerCluster' * 'SectorsPerCluster' which gives a 
> total value over 32 Kb per cluster, are not supported on many system[s]."
> 
> Note: in the above, I believe the author meant 'BytesPerSector',
> rather than 'BytesPerCluster'. Otherwise his math is meaningless.

I think the author is also mixing up bits and bytes.  32KB  != 32Kb.
> 
> Given his 1999 copyright date:
> 
> <META NAME="copyright" CONTENT="(C) 1999  Lasse Krogh Thygesen">
> 
> his "level of support statement" may not be accurate 
> anymore (no mention of any O/S later than WinNT or Win98 
> is mentioned).
> 
> > A 32GB volume then needs about 1,024,000 entries of four bytes each.
> > or 4MBytes for the minimum size FAT.   I can't get from there to
> > your  32 sectors.
> 
> From your "16MBytes of FAT" -
> 
> 16MB / 512 => 32 sectors.

How do you divide  (16 x 1024 x 1024) by 512  and get 32?

When I do that division I get 32768 sectors.
> 
> >> > I was thinking that, if the empty disk had all sectors filled
> >> > with 0x00, you might not have to write all the FAT sectors,
> >> > since 0x00000000 is the FAT marker for an unused cluster.
> >> 
> >> That doesn't work on a couple of levels. The first FAT entry
> >> is supposed to include the media type byte, though I doubt
> >> many implementations pay any attention to it there. 
> > 
> > So, could you just write the first FAT entry and leave the
> > rest of the sectors as 0x00s?
> 
> Yes (assuming you meant that the remaining sectors are zeroed
> _and_ written out).
> 
> The only caveat would be if you tested for and found 
> bad sectors. Then you'd also need to mark those clusters 
> containing bad sectors as bad (unavailable) - FFF7 (FAT16).

I don't think  you even use FAT16 on a 32GB volume with a single 
partition.


Mark Borgerson

Warren expounded in
news:Xns9E018EE612954WarrensBlatherings@85.214.73.210: 

> Mark Borgerson expounded in
> news:MPG.270bb366405ce964989c70@news.eternal-september.org: 
> 
>> In article <Xns9E016F015CB2EWarrensBlatherings@85.214.73.210>, 
>> ve3wwg@gmail.com says...
>>> Writing 32 sectors isn't huge (32k/512 bytes each).  Comparing
>>> it to a file write, if you use 8 sectors per cluster, it is only 
>>> two clusters worth of writing.
>> 
>> I don't understand this.
>> 
>> IIRC,  a cluster can be no more than 32KB for Windows and FAT32.
> 
> See:
> 
> http://www.maverick-os.dk/FileSystemFormats/FAT16
> _FileSystem.html#SectorsPerCluster
> 
> "Sectors Per Cluster: This is the number of sectors per cluster. The 
> allowed values are: 1, 2, 4, 8, 16, 32 or 128. But de-facto is that
> most combinations of 'BytesPerCluster' * 'SectorsPerCluster' which
> gives a total value over 32 Kb per cluster, are not supported on many
> system[s]." 
> 
> Note: in the above, I believe the author meant 'BytesPerSector',
> rather than 'BytesPerCluster'. Otherwise his math is meaningless.
> 
> Given his 1999 copyright date:
> 
> <META NAME="copyright" CONTENT="(C) 1999  Lasse Krogh Thygesen">
> 
> his "level of support statement" may not be accurate 
> anymore (no mention of any O/S later than WinNT or Win98 
> is mentioned).
> 
>> A 32GB volume then needs about 1,024,000 entries of four bytes each.
>> or 4MBytes for the minimum size FAT.   I can't get from there to
>> your  32 sectors.
> 
> From your "16MBytes of FAT" -
> 
> 16MB / 512 => 32 sectors.
> 
>>> > I was thinking that, if the empty disk had all sectors filled
>>> > with 0x00, you might not have to write all the FAT sectors,
>>> > since 0x00000000 is the FAT marker for an unused cluster.
>>> 
>>> That doesn't work on a couple of levels. The first FAT entry
>>> is supposed to include the media type byte, though I doubt
>>> many implementations pay any attention to it there. 
>> 
>> So, could you just write the first FAT entry and leave the
>> rest of the sectors as 0x00s?
> 
> Yes (assuming you meant that the remaining sectors are zeroed
> _and_ written out).
> 
> The only caveat would be if you tested for and found 
> bad sectors. Then you'd also need to mark those clusters 
> containing bad sectors as bad (unavailable) - FFF7 (FAT16).

Actually, there was one more thing I forgot about:

The last part of the FAT will likely need FAT entries
indicating that the last n clusters are unavailable. This
happens when your FAT table has more entries in it than,
available clusters to manage.

Otherwise, instead of running out of space, you'll get
I/O errors when the FS attempts to use non-existing
clusters (unusable space will be allocated from the FAT).

Warren

Mark Borgerson expounded in
news:MPG.270bb366405ce964989c70@news.eternal-september.org: 

> In article <Xns9E016F015CB2EWarrensBlatherings@85.214.73.210>, 
> ve3wwg@gmail.com says...
>> Writing 32 sectors isn't huge (32k/512 bytes each).  Comparing
>> it to a file write, if you use 8 sectors per cluster, it is only 
>> two clusters worth of writing.
> 
> I don't understand this.
> 
> IIRC,  a cluster can be no more than 32KB for Windows and FAT32.

See:

http://www.maverick-os.dk/FileSystemFormats/FAT16
_FileSystem.html#SectorsPerCluster

"Sectors Per Cluster: This is the number of sectors per cluster. The 
allowed values are: 1, 2, 4, 8, 16, 32 or 128. But de-facto is that most 
combinations of 'BytesPerCluster' * 'SectorsPerCluster' which gives a 
total value over 32 Kb per cluster, are not supported on many system[s]."

Note: in the above, I believe the author meant 'BytesPerSector',
rather than 'BytesPerCluster'. Otherwise his math is meaningless.

Given his 1999 copyright date:

<META NAME="copyright" CONTENT="(C) 1999  Lasse Krogh Thygesen">

his "level of support statement" may not be accurate 
anymore (no mention of any O/S later than WinNT or Win98 
is mentioned).

> A 32GB volume then needs about 1,024,000 entries of four bytes each.
> or 4MBytes for the minimum size FAT.   I can't get from there to
> your  32 sectors.

From your "16MBytes of FAT" -

16MB / 512 => 32 sectors.

>> > I was thinking that, if the empty disk had all sectors filled
>> > with 0x00, you might not have to write all the FAT sectors,
>> > since 0x00000000 is the FAT marker for an unused cluster.
>> 
>> That doesn't work on a couple of levels. The first FAT entry
>> is supposed to include the media type byte, though I doubt
>> many implementations pay any attention to it there. 
> 
> So, could you just write the first FAT entry and leave the
> rest of the sectors as 0x00s?

Yes (assuming you meant that the remaining sectors are zeroed
_and_ written out).

The only caveat would be if you tested for and found 
bad sectors. Then you'd also need to mark those clusters 
containing bad sectors as bad (unavailable) - FFF7 (FAT16).

Warren

In article <Xns9E016F015CB2EWarrensBlatherings@85.214.73.210>, 
ve3wwg@gmail.com says...
> Mark Borgerson expounded in news:MPG.270a85515a5d8ab5989c6a@news.eternal-
> september.org:
> 
> > I think I misinterpreted your first statement.  I missed the fact
> > that you did say that you had to write _one_ FAT table when  you
> > format the card.   However, on a large card, writing 16MBytes
> > of FAT could take quite a while.
> 
> Writing 32 sectors isn't huge (32k/512 bytes each).  Comparing
> it to a file write, if you use 8 sectors per cluster, it is only 
> two clusters worth of writing.

I don't understand this.

IIRC,  a cluster can be no more than 32KB for Windows and FAT32.

A 32GB volume then needs about 1,024,000 entries of four bytes each.
or 4MBytes for the minimum size FAT.   I can't get from there to
your  32 sectors.
> 
> > I was thinking that, if the empty disk had all sectors filled
> > with 0x00, you might not have to write all the FAT sectors,
> > since 0x00000000 is the FAT marker for an unused cluster.
> 
> That doesn't work on a couple of levels. The first FAT entry
> is supposed to include the media type byte, though I doubt
> many implementations pay any attention to it there. 

So, could you just write the first FAT entry and leave the
rest of the sectors as 0x00s?
> 
> Also if you use FAT32, your root directory is located in
> a cluster. So at least one cluster has to be pre-allocated.
> 
> But those things could be localized to the first FAT sector.
> 
Mark Borgerson

Mark Borgerson expounded in news:MPG.270a85515a5d8ab5989c6a@news.eternal-
september.org:

> I think I misinterpreted your first statement.  I missed the fact
> that you did say that you had to write _one_ FAT table when  you
> format the card.   However, on a large card, writing 16MBytes
> of FAT could take quite a while.

Writing 32 sectors isn't huge (32k/512 bytes each).  Comparing
it to a file write, if you use 8 sectors per cluster, it is only 
two clusters worth of writing.

> I was thinking that, if the empty disk had all sectors filled
> with 0x00, you might not have to write all the FAT sectors,
> since 0x00000000 is the FAT marker for an unused cluster.

That doesn't work on a couple of levels. The first FAT entry
is supposed to include the media type byte, though I doubt
many implementations pay any attention to it there. 

Also if you use FAT32, your root directory is located in
a cluster. So at least one cluster has to be pre-allocated.

But those things could be localized to the first FAT sector.

Warren

In article <Xns9E005C2987DF7WarrensBlatherings@85.214.73.210>, 
ve3wwg@gmail.com says...
> Mark Borgerson expounded in
> news:MPG.2706a0cd11868d9989c58@news.eternal-september.org: 
> 
> > In article <Xns9DFD9C76F438SnarkCharmedFerSure@188.40.43.230>, 
> > snark@cogeco.ca says...
> 
> >> But but.. on a memory card, unlike a floppy, you don't need
> >> to write _every_ sector.  You need to write the boot sector,
> >> root dir sector(s) (FAT16), and _one_ FAT table. If
> >> you're formatting FAT32, then you also need to allocate one
> >> cluster to hold the start of the root directory and fill 
> >> that cluster with zeros. All remaining clusters can
> >> be left in "as is" condition.
> > 
> > Are you assuming that the card has never been used and
> > is starting with 0xFF in all sectors?  I suppose that would
> > be fine if the system was never reused or tested before
> > being deployed.
> ...
> > Mark Borgerson
> 
> It doesn't matter. Why would it? 
> 
> When you allocate a new cluster to a root/sub-directory, you 
> zero it then (zero is the 'end of directory' marker).  When you 
> allocate a cluster to a file (being written and extended), you 
> just make the cluster available - the application just overwrites 
> the garbage in those sectors.

I think I misinterpreted your first statement.  I missed the fact
that you did say that you had to write _one_ FAT table when  you
format the card.   However, on a large card, writing 16MBytes
of FAT could take quite a while.

I was thinking that, if the empty disk had all sectors filled
with 0x00, you might not have to write all the FAT sectors,
since 0x00000000 is the FAT marker for an unused cluster.


> 
> The flash controller takes care of erasing the "pages" of
> memory and "rewrites" the sector as needed. To the end user, 
> it just looks like a disk.
> 
Mark Borgerson

Mark Borgerson expounded in
news:MPG.2706a0cd11868d9989c58@news.eternal-september.org: 

> In article <Xns9DFD9C76F438SnarkCharmedFerSure@188.40.43.230>, 
> snark@cogeco.ca says...

>> But but.. on a memory card, unlike a floppy, you don't need
>> to write _every_ sector.  You need to write the boot sector,
>> root dir sector(s) (FAT16), and _one_ FAT table. If
>> you're formatting FAT32, then you also need to allocate one
>> cluster to hold the start of the root directory and fill 
>> that cluster with zeros. All remaining clusters can
>> be left in "as is" condition.
> 
> Are you assuming that the card has never been used and
> is starting with 0xFF in all sectors?  I suppose that would
> be fine if the system was never reused or tested before
> being deployed.
...
> Mark Borgerson

It doesn't matter. Why would it? 

When you allocate a new cluster to a root/sub-directory, you 
zero it then (zero is the 'end of directory' marker).  When you 
allocate a cluster to a file (being written and extended), you 
just make the cluster available - the application just overwrites 
the garbage in those sectors.

The flash controller takes care of erasing the "pages" of
memory and "rewrites" the sector as needed. To the end user, 
it just looks like a disk.

Warren

In article <Xns9DFD9C76F438SnarkCharmedFerSure@188.40.43.230>, 
snark@cogeco.ca says...
> Mark Borgerson expounded in news:MPG.27067f7f10afd8ec989c57@news.eternal-
> september.org:
> 
> > In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, 
> > ve3wwg@gmail.com says...
> 
> >> The next attempt to allocate a cluster will see the next 
> >> available cluster number as zero (none). This indicates
> >> the need to start at the beginning of the FAT. If the FAT
> >> search (from the beginning) still comes up empty, then you 
> >> know you're out of space. But in this case you'll get the 
> >> lowest freed cluster from the last file delete.
> > 
> > If the deleted file happens to be somewhere in the middle
> > of the SD card, you may have to scan a lot of FAT to
> > find that free cluster.
> 
> Simple optimizations don't guarantee that you always save
> a long search. ;-)
> 
> You can of course, make a slight improvement by noting 
> that if the saved available cluster is currently
> zero and you're deleting clusters, make note of one of them.
> Then at least you skip some of that overhead, in the
> next allocate cluster call.
> 
> >> > I've considered that option.  On a 16MHZ MPU with SPI SD
> >> > access, it can take quite a long time to pre-allocate
> >> > all the clusters on a 32GB SDHC card.   
> >> 
> >> Surely this is a small startup time, compared to months of
> >> data capture. You can do this in less time than it 
> >> takes to format the card.
> > 
> > It could be small in relation to the months of logging, but
> > large in relation to the other tasks needed to prepare
> > the system..    On a FAT32 system, the FAT can be as large
> > as about 16MB.   That's about 32768 512-byte sectors.
> > On a slow system, you might only be able to write 15K or
> > 30 sectors per second.   That means that formatting the
> > SD card might take 15 minutes.   
> 
> But but.. on a memory card, unlike a floppy, you don't need
> to write _every_ sector.  You need to write the boot sector,
> root dir sector(s) (FAT16), and _one_ FAT table. If
> you're formatting FAT32, then you also need to allocate one
> cluster to hold the start of the root directory and fill 
> that cluster with zeros. All remaining clusters can
> be left in "as is" condition.

Are you assuming that the card has never been used and
is starting with 0xFF in all sectors?  I suppose that would
be fine if the system was never reused or tested before
being deployed.
> 
> So if you choose your sectors/cluster paramter wisely and
> choose 1 FAT table, then you only need to write a very small
> fraction of the whole FS image.
> 
> For floppies, the format was also a phsical sector operation 
> in addition to setting up the FS. But a memory card doesn't
> need this.  
> 
> The only reason I can think of for doing a full format on
> a memory card, would be to discover any bad sectors. But 
> this is something that the FS software might be able to 
> do on the fly, if it is designed for it.
If you're using an SD card, doesn't the internal controller
in the card handle that (as well as wear leveling)?
> 
> I don't have any evidence on how often memory cards have
> bad sectors.  But an up front screening process could 
> eliminate unsuitable cards.  It really comes down to how
> critical your data capture application is.  If it _is_
> indeed critical, then you don't have any choice in this
> matter anyway- a full format will be required.
> 
> >> I don't see it being any more trouble than a format operation.
> >> If you leave it to your MCU to do (at startup), then it need not 
> >> be part of the human procedure.
> > 
> > That's not a bad idea.  The formatting of the disk could be
> > part of a burn-in procedure at initial build.
> > 
> > The only other downside is that you could lose 16MB (or 32 if
> > there are dual FATS) of storage space.   That's not a large
> > factor, though.
> 
> You only need 1 FAT - so format it that way. Problem solved.
> 
> Multiple FATs were designed for flakey floppy disks.  Hardly
> necessary on a memory card, unless you intend to use it close
> to the flash rewrite limits.  But that wouldn't be prudent
> for data logging purposes.
> 
> >> > That sounds like quite a complex scheme for an MSP430 running
> >> > at 16MHz and collecting 2KBytes of data per second for months.
> >> > Those low-priority tasks  might also have significant power 
> >> > consumption.
> >> > 
> >> > Mark Borgerson
> >> 
> >> It is complex, but an option. Given that you have months
> >> of time to chain together a file, I don't think you're time
> >> challenged. Of course it will contend with the SD/mmc device
> >> at some level and this may introduce unacceptable latency,
> >> which must be considered. 
> > 
> > That might be the largest problem.  The MSP430 doesn't have
> > a lot of spare RAM to buffer data if some other process is
> > blocking access to the SD card.
> 
> True, and just running another "task" requires allocating a
> separate stack etc.
> 
> >> Pre-allocating is definitely simpler and could be done after 
> >> a RESET, after opening the FS on the SD/mmc card.
> > 
> > Given that it might take 15 to 60 minutes,  It probably can't
> > be done on each power up or reset.  That's particularly true
> > if the FS has to retain calibration data over multiple logging
> > sessions.
> > 
> > Mark Borgerson
> 
> I assumed continuous operation, once started. Obviously, if it
> must be restartable, then perhaps it could be done only as part 
> of some special startup handshake, like a button press.
> 
Mark Borgerson

Mark Borgerson expounded in news:MPG.27067f7f10afd8ec989c57@news.eternal-
september.org:

> In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, 
> ve3wwg@gmail.com says...

>> The next attempt to allocate a cluster will see the next 
>> available cluster number as zero (none). This indicates
>> the need to start at the beginning of the FAT. If the FAT
>> search (from the beginning) still comes up empty, then you 
>> know you're out of space. But in this case you'll get the 
>> lowest freed cluster from the last file delete.
> 
> If the deleted file happens to be somewhere in the middle
> of the SD card, you may have to scan a lot of FAT to
> find that free cluster.

Simple optimizations don't guarantee that you always save
a long search. ;-)

You can of course, make a slight improvement by noting 
that if the saved available cluster is currently
zero and you're deleting clusters, make note of one of them.
Then at least you skip some of that overhead, in the
next allocate cluster call.

>> > I've considered that option.  On a 16MHZ MPU with SPI SD
>> > access, it can take quite a long time to pre-allocate
>> > all the clusters on a 32GB SDHC card.   
>> 
>> Surely this is a small startup time, compared to months of
>> data capture. You can do this in less time than it 
>> takes to format the card.
> 
> It could be small in relation to the months of logging, but
> large in relation to the other tasks needed to prepare
> the system..    On a FAT32 system, the FAT can be as large
> as about 16MB.   That's about 32768 512-byte sectors.
> On a slow system, you might only be able to write 15K or
> 30 sectors per second.   That means that formatting the
> SD card might take 15 minutes.   

But but.. on a memory card, unlike a floppy, you don't need
to write _every_ sector.  You need to write the boot sector,
root dir sector(s) (FAT16), and _one_ FAT table. If
you're formatting FAT32, then you also need to allocate one
cluster to hold the start of the root directory and fill 
that cluster with zeros. All remaining clusters can
be left in "as is" condition.

So if you choose your sectors/cluster paramter wisely and
choose 1 FAT table, then you only need to write a very small
fraction of the whole FS image.

For floppies, the format was also a phsical sector operation 
in addition to setting up the FS. But a memory card doesn't
need this.  

The only reason I can think of for doing a full format on
a memory card, would be to discover any bad sectors. But 
this is something that the FS software might be able to 
do on the fly, if it is designed for it.

I don't have any evidence on how often memory cards have
bad sectors.  But an up front screening process could 
eliminate unsuitable cards.  It really comes down to how
critical your data capture application is.  If it _is_
indeed critical, then you don't have any choice in this
matter anyway- a full format will be required.

>> I don't see it being any more trouble than a format operation.
>> If you leave it to your MCU to do (at startup), then it need not 
>> be part of the human procedure.
> 
> That's not a bad idea.  The formatting of the disk could be
> part of a burn-in procedure at initial build.
> 
> The only other downside is that you could lose 16MB (or 32 if
> there are dual FATS) of storage space.   That's not a large
> factor, though.

You only need 1 FAT - so format it that way. Problem solved.

Multiple FATs were designed for flakey floppy disks.  Hardly
necessary on a memory card, unless you intend to use it close
to the flash rewrite limits.  But that wouldn't be prudent
for data logging purposes.

>> > That sounds like quite a complex scheme for an MSP430 running
>> > at 16MHz and collecting 2KBytes of data per second for months.
>> > Those low-priority tasks  might also have significant power 
>> > consumption.
>> > 
>> > Mark Borgerson
>> 
>> It is complex, but an option. Given that you have months
>> of time to chain together a file, I don't think you're time
>> challenged. Of course it will contend with the SD/mmc device
>> at some level and this may introduce unacceptable latency,
>> which must be considered. 
> 
> That might be the largest problem.  The MSP430 doesn't have
> a lot of spare RAM to buffer data if some other process is
> blocking access to the SD card.

True, and just running another "task" requires allocating a
separate stack etc.

>> Pre-allocating is definitely simpler and could be done after 
>> a RESET, after opening the FS on the SD/mmc card.
> 
> Given that it might take 15 to 60 minutes,  It probably can't
> be done on each power up or reset.  That's particularly true
> if the FS has to retain calibration data over multiple logging
> sessions.
> 
> Mark Borgerson

I assumed continuous operation, once started. Obviously, if it
must be restartable, then perhaps it could be done only as part 
of some special startup handshake, like a button press.

Warren