Embedded RTOS - features poll

In article <24idnVMNn619dwXRnZ2dnUVZ_tydnZ2d@giganews.com>, 
nospam@nowhere.com says...
> 
> 
> Mark Borgerson wrote:
> 
> > In article <Xns9DFA61F529F79SnarkCharmedFerSure@188.40.43.230>, 
> > snark@cogeco.ca says...
> > 
> >>Vladimir Vassilevsky expounded in 
> >>news:oeidnTWnfd1fMgrRnZ2dnUVZ_sidnZ2d@giganews.com:
> >>
> >>
> >>>Warren wrote:
> >>>
> >>>
> >>>>Can you implement a file system with 1K of SRAM?  Its 
> >>>>difficult, but I've done if for FAT16/32.
> >>>
> >>>One guy showed me his very minimal FAT implementation which took less 
> >>>then 512 bytes of RAM. There is no need to buffer a full sector. This, 
> >>>of course, implied tremendous overhead and inconvenience.
> >>
> >>Indeed!
> >>
> >>
> >>>> You have to 
> >>>>play some shell games with sector buffers to make it 
> >>>>possible.  But this emphasizes the point that a file
> >>>>system implementation for 8-bit is going to be a lot 
> >>>>different than a 32-bit platform. Resource limitations
> >>>>enforce some practical realities.
> >>>
> >>>The real point is that it is plain stupid to mount things like FAT or 
> >>>USB host or TCP/IP on 8-bitter with 1K of RAM. It is not wothy, those 
> >>>are be crippled implementations with lots of limitations.
> >>>
> >>
> >>Of course it is limited, but hardly stupid. You don't need
> >>a linux kernel on a 32/64 bit platform doing data logging.
> >>
> >>You can of course accomplish this with a little 8-bitter, 
> >>logging to a FATFS on a stick. It simply takes a little 
> >>planning.
> > 
> > 
> > If you mean logging to a USB memory stick, that will require
> > your 8-bitter to implement a USB Host interface.
> 
> At least a part of USB host to support for the basic read/write 
> operations on the mass storage device.
> 
> > My approach has been to use SD or SD micro cards with an
> > in-house sequential file system.  It doesn't need an
> > RTOS,  just an interrupt handler to collect and queue
> > the data and a main loop to pull from the queue and 
> > write to SD.  That works up to about 400 16-bit samples
> > per second  on an average current around 10mA using
> > an MSP430.   I don't think you can run linux effectively
> > on 10mA at 3.3V.
> 
> We used simple sequential file system also, despite of the obvious 
> limitations. Later we switched to FAT with POSIX API, and it was so much 
> better. There is no need for Linux; a standalone full featured 
> multithreaded FAT takes ~50kb + buffers. The power consumption is 
> determined by the number of transactions per second rather then the 
> memory size.
> 
I agree.  If you are writing to flash or SD cards,  the current
drain during a sector write may be 10X that used just to acquire
the data.

One problem I've run across with very sensitive analog systems is
the EMI and power supply transients that occur when the SD
card writes a sector.


Mark Borgerson

In article <Xns9DFAA391E9C36SnarkCharmedFerSure@81.169.183.62>, 
snark@cogeco.ca says...
> Mark Borgerson expounded in
> news:MPG.27028556ec82117d989c4d@news.eternal-september.org: 
> 
> > In article <Xns9DFA61F529F79SnarkCharmedFerSure@188.40.43.230>, 
> > snark@cogeco.ca says...
> >> Vladimir Vassilevsky expounded in 
> >> news:oeidnTWnfd1fMgrRnZ2dnUVZ_sidnZ2d@giganews.com:
> >> 
> >> > Warren wrote:
> >> > 
> >> >> Can you implement a file system with 1K of SRAM?  Its 
> >> >> difficult, but I've done if for FAT16/32.
> >> > 
> >> > One guy showed me his very minimal FAT implementation which took
> >> > less then 512 bytes of RAM. There is no need to buffer a full
> >> > sector. This, of course, implied tremendous overhead and
> >> > inconvenience. 
> >> 
> >> Indeed!
> >> 
> >> >>  You have to 
> >> >> play some shell games with sector buffers to make it 
> >> >> possible.  But this emphasizes the point that a file
> >> >> system implementation for 8-bit is going to be a lot 
> >> >> different than a 32-bit platform. Resource limitations
> >> >> enforce some practical realities.
> >> > 
> >> > The real point is that it is plain stupid to mount things like FAT
> >> > or USB host or TCP/IP on 8-bitter with 1K of RAM. It is not wothy,
> >> > those are be crippled implementations with lots of limitations.
> >> > 
> >> > VLV
> >> 
> >> Of course it is limited, but hardly stupid. You don't need
> >> a linux kernel on a 32/64 bit platform doing data logging.
> >> 
> >> You can of course accomplish this with a little 8-bitter, 
> >> logging to a FATFS on a stick. It simply takes a little 
> >> planning.
> > 
> > If you mean logging to a USB memory stick, that will require
> > your 8-bitter to implement a USB Host interface.
> 
> Nope. Why bring USB into it?  Just use a SD memory card.
> The interface for it is trivial.
> 
> > My approach has been to use SD or SD micro cards with an
> > in-house sequential file system.  It doesn't need an
> > RTOS,  just an interrupt handler to collect and queue
> > the data and a main loop to pull from the queue and 
> > write to SD.  ...
> > Mark Borgerson
> 
> Fine, but that is not user friendly to the user receiving
> the SD data. Now he/she needs a special app to pull it off
> the stick.
> 
> OTOH, you can use minimal FATFS software to create and
> write log file(s).

The problem with a truly minimal FAT16 or FAT32 FS is that
you either have to cache the FAT and directory entry or
you get two or more sector writes for each file write operation.
That may be OK if you have very large RAM buffers and only
write once every ten seconds.  It doesn't work well with
2K or so of RAM.

For FAT32,  another problem with simple implementations is that
writing a new cluster may require traversing the fat to
allocate the next open cluster.   Avoiding a long sequence
of sector reads can require either a modification to the
simple FS, or a large RAM buffer.   That problem  killed FAT32
for when my SDHC files got up to about 200MB.

This particular logger has to sit on an oceanographic
mooring and collect 15 channels of data at 100Hz for
about 4 to 6 months with internal batteries.  Even with
lithium primary cells, the power budget only allows about
20mA for the logger and sensors.
> 
> Anyone can just write sectors to the SD without a FS, but 
> it is terribly inelegant.

True.  That's why I wrote a simple sequential file system.
The logger comes with a host program that knows how
to upload smaller files via USB and can handle the FS when
the SD card is plugged into a PC.   The files are in
binary format, so some processing program in inevitable.

Mark Borgerson

you can certify a bowl of spaghetti as just the same as you can
certify sushi -- it does not mean they are both automatically suitable


On Sep 21, 3:51=A0pm, rickman <gnu...@gmail.com> wrote:
> On Sep 20, 10:53=A0am, Chris H <ch...@phaedsys.org> wrote:
> I wouldn't try to knock his work too much. =A0FreeRTOS is very widely
> used and is looking a bit like a defacto standard. =A0The SafeRTOS is
> his paycheck and I am sure it is a bit harder to use "safely", so it
> is much more likely that others will need his services to help with
> their application. =A0I can't find any real fault with what he offers or
> how he offers it. =A0I only wish I had something of this utility to
> offer the world (and make an income from).
>
> Rick

Mark Borgerson expounded in news:MPG.2702c86b7c64fd25989c4f@news.eternal-
september.org:

>> OTOH, you can use minimal FATFS software to create and
>> write log file(s).
> 
> The problem with a truly minimal FAT16 or FAT32 FS is that
> you either have to cache the FAT and directory entry or
> you get two or more sector writes for each file write operation.

Not exactly.  You only search and update the FAT when
you're adding clusters. Yes, you need one at the beginning
of a new file, but thereafter, it depends upon your
sectors/cluster parameter. Only if this value was 1, would
every [sector] file write require diddling the FAT again to
add a new cluster.  Sectors per cluster are usually 8 or
higher (especially higher if the FS is large).

> For FAT32,  another problem with simple implementations is that
> [adding] a new cluster may require traversing the fat to
> allocate the next open cluster.   Avoiding a long sequence
> of sector reads can require either a modification to the
> simple FS, or a large RAM buffer.   That problem  killed FAT32
> for when my SDHC files got up to about 200MB.

This is a problem indeed, if you always start from the
beginning of the FAT table.  However, an uncomplicated
way to improve this is to make note of the lowest available
cluster number and store _that_ in a 32-bit variable for
the FS.  Then, the _next_ time you go hunting for a free
FAT entry, you only need to start from the FAT sector
containing the free cluster entry (which is easily computed).

>> Anyone can just write sectors to the SD without a FS, but 
>> it is terribly inelegant.
> 
> True.  That's why I wrote a simple sequential file system.
> The logger comes with a host program that knows how
> to upload smaller files via USB and can handle the FS when
> the SD card is plugged into a PC.   The files are in
> binary format, so some processing program in inevitable.
> 
> Mark Borgerson

If performance in writing is needed and you wanted to retain
the advantage of FAT, I think there are possibilities. For
example, you could preallocate all your clusters for your
log file at the beginning, since you know the sample rate 
and duration.  Your FS software would simply need to provide
a "rewrite" file sector function as samples are written out.

Other alternatives could include the building of a list of free
clusters up front.  As long as no other FS cluster allocating
operations are going on at the same time you could simply
log your data to those clusters directly. Then another task,
could link them together into a file, on a lower priority
basis, ending with and perhaps periodically updating the
directory entry for the involved log file.

Warren

In article <Xns9DFCA067825DWarrensBlatherings@81.169.183.62>, 
ve3wwg@gmail.com says...
> Mark Borgerson expounded in news:MPG.2702c86b7c64fd25989c4f@news.eternal-
> september.org:
> 
> >> OTOH, you can use minimal FATFS software to create and
> >> write log file(s).
> > 
> > The problem with a truly minimal FAT16 or FAT32 FS is that
> > you either have to cache the FAT and directory entry or
> > you get two or more sector writes for each file write operation.
> 
> Not exactly.  You only search and update the FAT when
> you're adding clusters. Yes, you need one at the beginning
> of a new file, but thereafter, it depends upon your
> sectors/cluster parameter. Only if this value was 1, would
> every [sector] file write require diddling the FAT again to
> add a new cluster.  Sectors per cluster are usually 8 or
> higher (especially higher if the FS is large).


> 
> > For FAT32,  another problem with simple implementations is that
> > [adding] a new cluster may require traversing the fat to
> > allocate the next open cluster.   Avoiding a long sequence
> > of sector reads can require either a modification to the
> > simple FS, or a large RAM buffer.   That problem  killed FAT32
> > for when my SDHC files got up to about 200MB.
> 
> This is a problem indeed, if you always start from the
> beginning of the FAT table.  However, an uncomplicated
> way to improve this is to make note of the lowest available
> cluster number and store _that_ in a 32-bit variable for
> the FS.  Then, the _next_ time you go hunting for a free
> FAT entry, you only need to start from the FAT sector
> containing the free cluster entry (which is easily computed).
> 
That is probably the best way to handle it if there is no
capability to  delete files.  If you can delete files, you
can end up with the free FAT entries scattered all through
the table.  The next available FAT entry might be at a
lower entry than the last used entry.  When that happens,
you're back to traversing the FAT.
> >> Anyone can just write sectors to the SD without a FS, but 
> >> it is terribly inelegant.
> > 
> > True.  That's why I wrote a simple sequential file system.
> > The logger comes with a host program that knows how
> > to upload smaller files via USB and can handle the FS when
> > the SD card is plugged into a PC.   The files are in
> > binary format, so some processing program in inevitable.
> > 
> > Mark Borgerson
> 
> If performance in writing is needed and you wanted to retain
> the advantage of FAT, I think there are possibilities. For
> example, you could preallocate all your clusters for your
> log file at the beginning, since you know the sample rate 
> and duration.  Your FS software would simply need to provide
> a "rewrite" file sector function as samples are written out.

I've considered that option.  On a 16MHZ MPU with SPI SD
access, it can take quite a long time to pre-allocate
all the clusters on a 32GB SDHC card.   

I actually wrote a PC application to "clone" such a pre-allocated
SD card using  direct device writes to "format" the SD card.
It turned out to be more trouble than it was worth.

If you have all of the 32GB card appearing as one large file,
you would then need an app to break out the  data into smaller
chunks for analysis.
> 
> Other alternatives could include the building of a list of free
> clusters up front.  As long as no other FS cluster allocating
> operations are going on at the same time you could simply
> log your data to those clusters directly. Then another task,
> could link them together into a file, on a lower priority
> basis, ending with and perhaps periodically updating the
> directory entry for the involved log file.
> 

That sounds like quite a complex scheme for an MSP430 running
at 16MHz and collecting 2KBytes of data per second for months.
Those low-priority tasks  might also have significant power 
consumption.


Mark Borgerson

Mark Borgerson expounded in
news:MPG.27057fc7879fd102989c53@news.eternal-september.org: 

> In article <Xns9DFCA067825DWarrensBlatherings@81.169.183.62>, 
> ve3wwg@gmail.com says...

>> > For FAT32,  another problem with simple implementations is that
>> > [adding] a new cluster may require traversing the fat to
>> > allocate the next open cluster.   Avoiding a long sequence
>> > of sector reads can require either a modification to the
>> > simple FS, or a large RAM buffer.   That problem  killed FAT32
>> > for when my SDHC files got up to about 200MB.
>> 
>> This is a problem indeed, if you always start from the
>> beginning of the FAT table.  However, an uncomplicated
>> way to improve this is to make note of the lowest available
>> cluster number and store _that_ in a 32-bit variable for
>> the FS.  Then, the _next_ time you go hunting for a free
>> FAT entry, you only need to start from the FAT sector
>> containing the free cluster entry (which is easily computed).
>> 
> That is probably the best way to handle it if there is no
> capability to  delete files.  If you can delete files, you
> can end up with the free FAT entries scattered all through
> the table.  The next available FAT entry might be at a
> lower entry than the last used entry.  When that happens,
> you're back to traversing the FAT.

Deletes don't present a problem. The only time the effect of
a delete becomes critical is when you're nearly out of space 
on the volume. But the algorithm I described still works 
without complications. 

Let's say you've allocated the last cluster near the end 
of the FAT. Now your remembered lowest available cluster number 
is zero because the FAT search didn't encounter a next 
free cluster. Cluster numbers start at 2, so zero is a good 
way to indicate 'no cluster'. The current cluster allocation
has been satisfied, but the next "might" fail, but..

Then a file gets deleted and say it frees up 2 more clusters.

The next attempt to allocate a cluster will see the next 
available cluster number as zero (none). This indicates
the need to start at the beginning of the FAT. If the FAT
search (from the beginning) still comes up empty, then you 
know you're out of space. But in this case you'll get the 
lowest freed cluster from the last file delete.

So deletes don't complicate this. There is no rule that 
says you have to allocate the lowest numbered cluster.

If you wanted to get fancy, you'd actually prefer to keep 
clusters close together for a given file (if it were a 
physical disk). But as you know, this is hardly necessary 
in embedded applications and unnecessary for a
memory based FS.

>> If performance in writing is needed and you wanted to retain
>> the advantage of FAT, I think there are possibilities. For
>> example, you could preallocate all your clusters for your
>> log file at the beginning, since you know the sample rate 
>> and duration.  Your FS software would simply need to provide
>> a "rewrite" file sector function as samples are written out.
> 
> I've considered that option.  On a 16MHZ MPU with SPI SD
> access, it can take quite a long time to pre-allocate
> all the clusters on a 32GB SDHC card.   

Surely this is a small startup time, compared to months of
data capture. You can do this in less time than it 
takes to format the card.

> I actually wrote a PC application to "clone" such a pre-allocated
> SD card using  direct device writes to "format" the SD card.
> It turned out to be more trouble than it was worth.

I don't see it being any more trouble than a format operation.
If you leave it to your MCU to do (at startup), then it need not 
be part of the human procedure.

> If you have all of the 32GB card appearing as one large file,
> you would then need an app to break out the  data into smaller
> chunks for analysis.

That is your choice in application design.  You have the
same issue whether the data is physically stored or in
a file system's "file".

>> Other alternatives could include the building of a list of free
>> clusters up front.  As long as no other FS cluster allocating
>> operations are going on at the same time you could simply
>> log your data to those clusters directly. Then another task,
>> could link them together into a file, on a lower priority
>> basis, ending with and perhaps periodically updating the
>> directory entry for the involved log file.
> 
> That sounds like quite a complex scheme for an MSP430 running
> at 16MHz and collecting 2KBytes of data per second for months.
> Those low-priority tasks  might also have significant power 
> consumption.
> 
> Mark Borgerson

It is complex, but an option. Given that you have months
of time to chain together a file, I don't think you're time
challenged. Of course it will contend with the SD/mmc device
at some level and this may introduce unacceptable latency,
which must be considered. 

Pre-allocating is definitely simpler and could be done after 
a RESET, after opening the FS on the SD/mmc card.

Warren

In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, 
ve3wwg@gmail.com says...
> Mark Borgerson expounded in
> news:MPG.27057fc7879fd102989c53@news.eternal-september.org: 
> 
> > In article <Xns9DFCA067825DWarrensBlatherings@81.169.183.62>, 
> > ve3wwg@gmail.com says...
> 
> >> > For FAT32,  another problem with simple implementations is that
> >> > [adding] a new cluster may require traversing the fat to
> >> > allocate the next open cluster.   Avoiding a long sequence
> >> > of sector reads can require either a modification to the
> >> > simple FS, or a large RAM buffer.   That problem  killed FAT32
> >> > for when my SDHC files got up to about 200MB.
> >> 
> >> This is a problem indeed, if you always start from the
> >> beginning of the FAT table.  However, an uncomplicated
> >> way to improve this is to make note of the lowest available
> >> cluster number and store _that_ in a 32-bit variable for
> >> the FS.  Then, the _next_ time you go hunting for a free
> >> FAT entry, you only need to start from the FAT sector
> >> containing the free cluster entry (which is easily computed).
> >> 
> > That is probably the best way to handle it if there is no
> > capability to  delete files.  If you can delete files, you
> > can end up with the free FAT entries scattered all through
> > the table.  The next available FAT entry might be at a
> > lower entry than the last used entry.  When that happens,
> > you're back to traversing the FAT.
> 
> Deletes don't present a problem. The only time the effect of
> a delete becomes critical is when you're nearly out of space 
> on the volume. But the algorithm I described still works 
> without complications. 
> 
> Let's say you've allocated the last cluster near the end 
> of the FAT. Now your remembered lowest available cluster number 
> is zero because the FAT search didn't encounter a next 
> free cluster. Cluster numbers start at 2, so zero is a good 
> way to indicate 'no cluster'. The current cluster allocation
> has been satisfied, but the next "might" fail, but..
> 
> Then a file gets deleted and say it frees up 2 more clusters.
> 
> The next attempt to allocate a cluster will see the next 
> available cluster number as zero (none). This indicates
> the need to start at the beginning of the FAT. If the FAT
> search (from the beginning) still comes up empty, then you 
> know you're out of space. But in this case you'll get the 
> lowest freed cluster from the last file delete.

If the deleted file happens to be somewhere in the middle
of the SD card, you may have to scan a lot of FAT to
find that free cluster.
> 
> So deletes don't complicate this. There is no rule that 
> says you have to allocate the lowest numbered cluster.

That's correct. However, that is the way it was done in
the simple FAT system I started with.
> 
> If you wanted to get fancy, you'd actually prefer to keep 
> clusters close together for a given file (if it were a 
> physical disk). But as you know, this is hardly necessary 
> in embedded applications and unnecessary for a
> memory based FS.
> 
> >> If performance in writing is needed and you wanted to retain
> >> the advantage of FAT, I think there are possibilities. For
> >> example, you could preallocate all your clusters for your
> >> log file at the beginning, since you know the sample rate 
> >> and duration.  Your FS software would simply need to provide
> >> a "rewrite" file sector function as samples are written out.
> > 
> > I've considered that option.  On a 16MHZ MPU with SPI SD
> > access, it can take quite a long time to pre-allocate
> > all the clusters on a 32GB SDHC card.   
> 
> Surely this is a small startup time, compared to months of
> data capture. You can do this in less time than it 
> takes to format the card.

It could be small in relation to the months of logging, but
large in relation to the other tasks needed to prepare
the system..    On a FAT32 system, the FAT can be as large
as about 16MB.   That's about 32768 512-byte sectors.
On a slow system, you might only be able to write 15K or
30 sectors per second.   That means that formatting the
SD card might take 15 minutes.   Multiply that by 4 cards,
and you have about an hour to format the storage on the
logging system.
> 
> > I actually wrote a PC application to "clone" such a pre-allocated
> > SD card using  direct device writes to "format" the SD card.
> > It turned out to be more trouble than it was worth.
> 
> I don't see it being any more trouble than a format operation.
> If you leave it to your MCU to do (at startup), then it need not 
> be part of the human procedure.

That's not a bad idea.  The formatting of the disk could be
part of a burn-in procedure at initial build.

The only other downside is that you could lose 16MB (or 32 if
there are dual FATS) of storage space.   That's not a large
factor, though.
> 
> > If you have all of the 32GB card appearing as one large file,
> > you would then need an app to break out the  data into smaller
> > chunks for analysis.
> 
> That is your choice in application design.  You have the
> same issue whether the data is physically stored or in
> a file system's "file".
> 
> >> Other alternatives could include the building of a list of free
> >> clusters up front.  As long as no other FS cluster allocating
> >> operations are going on at the same time you could simply
> >> log your data to those clusters directly. Then another task,
> >> could link them together into a file, on a lower priority
> >> basis, ending with and perhaps periodically updating the
> >> directory entry for the involved log file.
> > 
> > That sounds like quite a complex scheme for an MSP430 running
> > at 16MHz and collecting 2KBytes of data per second for months.
> > Those low-priority tasks  might also have significant power 
> > consumption.
> > 
> > Mark Borgerson
> 
> It is complex, but an option. Given that you have months
> of time to chain together a file, I don't think you're time
> challenged. Of course it will contend with the SD/mmc device
> at some level and this may introduce unacceptable latency,
> which must be considered. 

That might be the largest problem.  The MSP430 doesn't have
a lot of spare RAM to buffer data if some other process is
blocking access to the SD card.
> 
> Pre-allocating is definitely simpler and could be done after 
> a RESET, after opening the FS on the SD/mmc card.

Given that it might take 15 to 60 minutes,  It probably can't
be done on each power up or reset.  That's particularly true
if the FS has to retain calibration data over multiple logging
sessions.


Mark Borgerson

Mark Borgerson expounded in news:MPG.27067f7f10afd8ec989c57@news.eternal-
september.org:

> In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, 
> ve3wwg@gmail.com says...

>> The next attempt to allocate a cluster will see the next 
>> available cluster number as zero (none). This indicates
>> the need to start at the beginning of the FAT. If the FAT
>> search (from the beginning) still comes up empty, then you 
>> know you're out of space. But in this case you'll get the 
>> lowest freed cluster from the last file delete.
> 
> If the deleted file happens to be somewhere in the middle
> of the SD card, you may have to scan a lot of FAT to
> find that free cluster.

Simple optimizations don't guarantee that you always save
a long search. ;-)

You can of course, make a slight improvement by noting 
that if the saved available cluster is currently
zero and you're deleting clusters, make note of one of them.
Then at least you skip some of that overhead, in the
next allocate cluster call.

>> > I've considered that option.  On a 16MHZ MPU with SPI SD
>> > access, it can take quite a long time to pre-allocate
>> > all the clusters on a 32GB SDHC card.   
>> 
>> Surely this is a small startup time, compared to months of
>> data capture. You can do this in less time than it 
>> takes to format the card.
> 
> It could be small in relation to the months of logging, but
> large in relation to the other tasks needed to prepare
> the system..    On a FAT32 system, the FAT can be as large
> as about 16MB.   That's about 32768 512-byte sectors.
> On a slow system, you might only be able to write 15K or
> 30 sectors per second.   That means that formatting the
> SD card might take 15 minutes.   

But but.. on a memory card, unlike a floppy, you don't need
to write _every_ sector.  You need to write the boot sector,
root dir sector(s) (FAT16), and _one_ FAT table. If
you're formatting FAT32, then you also need to allocate one
cluster to hold the start of the root directory and fill 
that cluster with zeros. All remaining clusters can
be left in "as is" condition.

So if you choose your sectors/cluster paramter wisely and
choose 1 FAT table, then you only need to write a very small
fraction of the whole FS image.

For floppies, the format was also a phsical sector operation 
in addition to setting up the FS. But a memory card doesn't
need this.  

The only reason I can think of for doing a full format on
a memory card, would be to discover any bad sectors. But 
this is something that the FS software might be able to 
do on the fly, if it is designed for it.

I don't have any evidence on how often memory cards have
bad sectors.  But an up front screening process could 
eliminate unsuitable cards.  It really comes down to how
critical your data capture application is.  If it _is_
indeed critical, then you don't have any choice in this
matter anyway- a full format will be required.

>> I don't see it being any more trouble than a format operation.
>> If you leave it to your MCU to do (at startup), then it need not 
>> be part of the human procedure.
> 
> That's not a bad idea.  The formatting of the disk could be
> part of a burn-in procedure at initial build.
> 
> The only other downside is that you could lose 16MB (or 32 if
> there are dual FATS) of storage space.   That's not a large
> factor, though.

You only need 1 FAT - so format it that way. Problem solved.

Multiple FATs were designed for flakey floppy disks.  Hardly
necessary on a memory card, unless you intend to use it close
to the flash rewrite limits.  But that wouldn't be prudent
for data logging purposes.

>> > That sounds like quite a complex scheme for an MSP430 running
>> > at 16MHz and collecting 2KBytes of data per second for months.
>> > Those low-priority tasks  might also have significant power 
>> > consumption.
>> > 
>> > Mark Borgerson
>> 
>> It is complex, but an option. Given that you have months
>> of time to chain together a file, I don't think you're time
>> challenged. Of course it will contend with the SD/mmc device
>> at some level and this may introduce unacceptable latency,
>> which must be considered. 
> 
> That might be the largest problem.  The MSP430 doesn't have
> a lot of spare RAM to buffer data if some other process is
> blocking access to the SD card.

True, and just running another "task" requires allocating a
separate stack etc.

>> Pre-allocating is definitely simpler and could be done after 
>> a RESET, after opening the FS on the SD/mmc card.
> 
> Given that it might take 15 to 60 minutes,  It probably can't
> be done on each power up or reset.  That's particularly true
> if the FS has to retain calibration data over multiple logging
> sessions.
> 
> Mark Borgerson

I assumed continuous operation, once started. Obviously, if it
must be restartable, then perhaps it could be done only as part 
of some special startup handshake, like a button press.

Warren

In article <Xns9DFD9C76F438SnarkCharmedFerSure@188.40.43.230>, 
snark@cogeco.ca says...
> Mark Borgerson expounded in news:MPG.27067f7f10afd8ec989c57@news.eternal-
> september.org:
> 
> > In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, 
> > ve3wwg@gmail.com says...
> 
> >> The next attempt to allocate a cluster will see the next 
> >> available cluster number as zero (none). This indicates
> >> the need to start at the beginning of the FAT. If the FAT
> >> search (from the beginning) still comes up empty, then you 
> >> know you're out of space. But in this case you'll get the 
> >> lowest freed cluster from the last file delete.
> > 
> > If the deleted file happens to be somewhere in the middle
> > of the SD card, you may have to scan a lot of FAT to
> > find that free cluster.
> 
> Simple optimizations don't guarantee that you always save
> a long search. ;-)
> 
> You can of course, make a slight improvement by noting 
> that if the saved available cluster is currently
> zero and you're deleting clusters, make note of one of them.
> Then at least you skip some of that overhead, in the
> next allocate cluster call.
> 
> >> > I've considered that option.  On a 16MHZ MPU with SPI SD
> >> > access, it can take quite a long time to pre-allocate
> >> > all the clusters on a 32GB SDHC card.   
> >> 
> >> Surely this is a small startup time, compared to months of
> >> data capture. You can do this in less time than it 
> >> takes to format the card.
> > 
> > It could be small in relation to the months of logging, but
> > large in relation to the other tasks needed to prepare
> > the system..    On a FAT32 system, the FAT can be as large
> > as about 16MB.   That's about 32768 512-byte sectors.
> > On a slow system, you might only be able to write 15K or
> > 30 sectors per second.   That means that formatting the
> > SD card might take 15 minutes.   
> 
> But but.. on a memory card, unlike a floppy, you don't need
> to write _every_ sector.  You need to write the boot sector,
> root dir sector(s) (FAT16), and _one_ FAT table. If
> you're formatting FAT32, then you also need to allocate one
> cluster to hold the start of the root directory and fill 
> that cluster with zeros. All remaining clusters can
> be left in "as is" condition.

Are you assuming that the card has never been used and
is starting with 0xFF in all sectors?  I suppose that would
be fine if the system was never reused or tested before
being deployed.
> 
> So if you choose your sectors/cluster paramter wisely and
> choose 1 FAT table, then you only need to write a very small
> fraction of the whole FS image.
> 
> For floppies, the format was also a phsical sector operation 
> in addition to setting up the FS. But a memory card doesn't
> need this.  
> 
> The only reason I can think of for doing a full format on
> a memory card, would be to discover any bad sectors. But 
> this is something that the FS software might be able to 
> do on the fly, if it is designed for it.
If you're using an SD card, doesn't the internal controller
in the card handle that (as well as wear leveling)?
> 
> I don't have any evidence on how often memory cards have
> bad sectors.  But an up front screening process could 
> eliminate unsuitable cards.  It really comes down to how
> critical your data capture application is.  If it _is_
> indeed critical, then you don't have any choice in this
> matter anyway- a full format will be required.
> 
> >> I don't see it being any more trouble than a format operation.
> >> If you leave it to your MCU to do (at startup), then it need not 
> >> be part of the human procedure.
> > 
> > That's not a bad idea.  The formatting of the disk could be
> > part of a burn-in procedure at initial build.
> > 
> > The only other downside is that you could lose 16MB (or 32 if
> > there are dual FATS) of storage space.   That's not a large
> > factor, though.
> 
> You only need 1 FAT - so format it that way. Problem solved.
> 
> Multiple FATs were designed for flakey floppy disks.  Hardly
> necessary on a memory card, unless you intend to use it close
> to the flash rewrite limits.  But that wouldn't be prudent
> for data logging purposes.
> 
> >> > That sounds like quite a complex scheme for an MSP430 running
> >> > at 16MHz and collecting 2KBytes of data per second for months.
> >> > Those low-priority tasks  might also have significant power 
> >> > consumption.
> >> > 
> >> > Mark Borgerson
> >> 
> >> It is complex, but an option. Given that you have months
> >> of time to chain together a file, I don't think you're time
> >> challenged. Of course it will contend with the SD/mmc device
> >> at some level and this may introduce unacceptable latency,
> >> which must be considered. 
> > 
> > That might be the largest problem.  The MSP430 doesn't have
> > a lot of spare RAM to buffer data if some other process is
> > blocking access to the SD card.
> 
> True, and just running another "task" requires allocating a
> separate stack etc.
> 
> >> Pre-allocating is definitely simpler and could be done after 
> >> a RESET, after opening the FS on the SD/mmc card.
> > 
> > Given that it might take 15 to 60 minutes,  It probably can't
> > be done on each power up or reset.  That's particularly true
> > if the FS has to retain calibration data over multiple logging
> > sessions.
> > 
> > Mark Borgerson
> 
> I assumed continuous operation, once started. Obviously, if it
> must be restartable, then perhaps it could be done only as part 
> of some special startup handshake, like a button press.
> 
Mark Borgerson

Mark Borgerson expounded in
news:MPG.2706a0cd11868d9989c58@news.eternal-september.org: 

> In article <Xns9DFD9C76F438SnarkCharmedFerSure@188.40.43.230>, 
> snark@cogeco.ca says...

>> But but.. on a memory card, unlike a floppy, you don't need
>> to write _every_ sector.  You need to write the boot sector,
>> root dir sector(s) (FAT16), and _one_ FAT table. If
>> you're formatting FAT32, then you also need to allocate one
>> cluster to hold the start of the root directory and fill 
>> that cluster with zeros. All remaining clusters can
>> be left in "as is" condition.
> 
> Are you assuming that the card has never been used and
> is starting with 0xFF in all sectors?  I suppose that would
> be fine if the system was never reused or tested before
> being deployed.
...
> Mark Borgerson

It doesn't matter. Why would it? 

When you allocate a new cluster to a root/sub-directory, you 
zero it then (zero is the 'end of directory' marker).  When you 
allocate a cluster to a file (being written and extended), you 
just make the cluster available - the application just overwrites 
the garbage in those sectors.

The flash controller takes care of erasing the "pages" of
memory and "rewrites" the sector as needed. To the end user, 
it just looks like a disk.

Warren