Forums

Embedded RTOS - features poll

Started by Stargazer September 19, 2010
In article <24idnVMNn619dwXRnZ2dnUVZ_tydnZ2d@giganews.com>, 
nospam@nowhere.com says...
> > > Mark Borgerson wrote: > > > In article <Xns9DFA61F529F79SnarkCharmedFerSure@188.40.43.230>, > > snark@cogeco.ca says... > > > >>Vladimir Vassilevsky expounded in > >>news:oeidnTWnfd1fMgrRnZ2dnUVZ_sidnZ2d@giganews.com: > >> > >> > >>>Warren wrote: > >>> > >>> > >>>>Can you implement a file system with 1K of SRAM? Its > >>>>difficult, but I've done if for FAT16/32. > >>> > >>>One guy showed me his very minimal FAT implementation which took less > >>>then 512 bytes of RAM. There is no need to buffer a full sector. This, > >>>of course, implied tremendous overhead and inconvenience. > >> > >>Indeed! > >> > >> > >>>> You have to > >>>>play some shell games with sector buffers to make it > >>>>possible. But this emphasizes the point that a file > >>>>system implementation for 8-bit is going to be a lot > >>>>different than a 32-bit platform. Resource limitations > >>>>enforce some practical realities. > >>> > >>>The real point is that it is plain stupid to mount things like FAT or > >>>USB host or TCP/IP on 8-bitter with 1K of RAM. It is not wothy, those > >>>are be crippled implementations with lots of limitations. > >>> > >> > >>Of course it is limited, but hardly stupid. You don't need > >>a linux kernel on a 32/64 bit platform doing data logging. > >> > >>You can of course accomplish this with a little 8-bitter, > >>logging to a FATFS on a stick. It simply takes a little > >>planning. > > > > > > If you mean logging to a USB memory stick, that will require > > your 8-bitter to implement a USB Host interface. > > At least a part of USB host to support for the basic read/write > operations on the mass storage device. > > > My approach has been to use SD or SD micro cards with an > > in-house sequential file system. It doesn't need an > > RTOS, just an interrupt handler to collect and queue > > the data and a main loop to pull from the queue and > > write to SD. That works up to about 400 16-bit samples > > per second on an average current around 10mA using > > an MSP430. I don't think you can run linux effectively > > on 10mA at 3.3V. > > We used simple sequential file system also, despite of the obvious > limitations. Later we switched to FAT with POSIX API, and it was so much > better. There is no need for Linux; a standalone full featured > multithreaded FAT takes ~50kb + buffers. The power consumption is > determined by the number of transactions per second rather then the > memory size. >
I agree. If you are writing to flash or SD cards, the current drain during a sector write may be 10X that used just to acquire the data. One problem I've run across with very sensitive analog systems is the EMI and power supply transients that occur when the SD card writes a sector. Mark Borgerson
In article <Xns9DFAA391E9C36SnarkCharmedFerSure@81.169.183.62>, 
snark@cogeco.ca says...
> Mark Borgerson expounded in > news:MPG.27028556ec82117d989c4d@news.eternal-september.org: > > > In article <Xns9DFA61F529F79SnarkCharmedFerSure@188.40.43.230>, > > snark@cogeco.ca says... > >> Vladimir Vassilevsky expounded in > >> news:oeidnTWnfd1fMgrRnZ2dnUVZ_sidnZ2d@giganews.com: > >> > >> > Warren wrote: > >> > > >> >> Can you implement a file system with 1K of SRAM? Its > >> >> difficult, but I've done if for FAT16/32. > >> > > >> > One guy showed me his very minimal FAT implementation which took > >> > less then 512 bytes of RAM. There is no need to buffer a full > >> > sector. This, of course, implied tremendous overhead and > >> > inconvenience. > >> > >> Indeed! > >> > >> >> You have to > >> >> play some shell games with sector buffers to make it > >> >> possible. But this emphasizes the point that a file > >> >> system implementation for 8-bit is going to be a lot > >> >> different than a 32-bit platform. Resource limitations > >> >> enforce some practical realities. > >> > > >> > The real point is that it is plain stupid to mount things like FAT > >> > or USB host or TCP/IP on 8-bitter with 1K of RAM. It is not wothy, > >> > those are be crippled implementations with lots of limitations. > >> > > >> > VLV > >> > >> Of course it is limited, but hardly stupid. You don't need > >> a linux kernel on a 32/64 bit platform doing data logging. > >> > >> You can of course accomplish this with a little 8-bitter, > >> logging to a FATFS on a stick. It simply takes a little > >> planning. > > > > If you mean logging to a USB memory stick, that will require > > your 8-bitter to implement a USB Host interface. > > Nope. Why bring USB into it? Just use a SD memory card. > The interface for it is trivial. > > > My approach has been to use SD or SD micro cards with an > > in-house sequential file system. It doesn't need an > > RTOS, just an interrupt handler to collect and queue > > the data and a main loop to pull from the queue and > > write to SD. ... > > Mark Borgerson > > Fine, but that is not user friendly to the user receiving > the SD data. Now he/she needs a special app to pull it off > the stick. > > OTOH, you can use minimal FATFS software to create and > write log file(s).
The problem with a truly minimal FAT16 or FAT32 FS is that you either have to cache the FAT and directory entry or you get two or more sector writes for each file write operation. That may be OK if you have very large RAM buffers and only write once every ten seconds. It doesn't work well with 2K or so of RAM. For FAT32, another problem with simple implementations is that writing a new cluster may require traversing the fat to allocate the next open cluster. Avoiding a long sequence of sector reads can require either a modification to the simple FS, or a large RAM buffer. That problem killed FAT32 for when my SDHC files got up to about 200MB. This particular logger has to sit on an oceanographic mooring and collect 15 channels of data at 100Hz for about 4 to 6 months with internal batteries. Even with lithium primary cells, the power budget only allows about 20mA for the logger and sensors.
> > Anyone can just write sectors to the SD without a FS, but > it is terribly inelegant.
True. That's why I wrote a simple sequential file system. The logger comes with a host program that knows how to upload smaller files via USB and can handle the FS when the SD card is plugged into a PC. The files are in binary format, so some processing program in inevitable. Mark Borgerson
you can certify a bowl of spaghetti as just the same as you can
certify sushi -- it does not mean they are both automatically suitable


On Sep 21, 3:51=A0pm, rickman <gnu...@gmail.com> wrote:
> On Sep 20, 10:53=A0am, Chris H <ch...@phaedsys.org> wrote: > I wouldn't try to knock his work too much. =A0FreeRTOS is very widely > used and is looking a bit like a defacto standard. =A0The SafeRTOS is > his paycheck and I am sure it is a bit harder to use "safely", so it > is much more likely that others will need his services to help with > their application. =A0I can't find any real fault with what he offers or > how he offers it. =A0I only wish I had something of this utility to > offer the world (and make an income from). > > Rick
Mark Borgerson expounded in news:MPG.2702c86b7c64fd25989c4f@news.eternal-
september.org:

>> OTOH, you can use minimal FATFS software to create and >> write log file(s). > > The problem with a truly minimal FAT16 or FAT32 FS is that > you either have to cache the FAT and directory entry or > you get two or more sector writes for each file write operation.
Not exactly. You only search and update the FAT when you're adding clusters. Yes, you need one at the beginning of a new file, but thereafter, it depends upon your sectors/cluster parameter. Only if this value was 1, would every [sector] file write require diddling the FAT again to add a new cluster. Sectors per cluster are usually 8 or higher (especially higher if the FS is large).
> For FAT32, another problem with simple implementations is that > [adding] a new cluster may require traversing the fat to > allocate the next open cluster. Avoiding a long sequence > of sector reads can require either a modification to the > simple FS, or a large RAM buffer. That problem killed FAT32 > for when my SDHC files got up to about 200MB.
This is a problem indeed, if you always start from the beginning of the FAT table. However, an uncomplicated way to improve this is to make note of the lowest available cluster number and store _that_ in a 32-bit variable for the FS. Then, the _next_ time you go hunting for a free FAT entry, you only need to start from the FAT sector containing the free cluster entry (which is easily computed).
>> Anyone can just write sectors to the SD without a FS, but >> it is terribly inelegant. > > True. That's why I wrote a simple sequential file system. > The logger comes with a host program that knows how > to upload smaller files via USB and can handle the FS when > the SD card is plugged into a PC. The files are in > binary format, so some processing program in inevitable. > > Mark Borgerson
If performance in writing is needed and you wanted to retain the advantage of FAT, I think there are possibilities. For example, you could preallocate all your clusters for your log file at the beginning, since you know the sample rate and duration. Your FS software would simply need to provide a "rewrite" file sector function as samples are written out. Other alternatives could include the building of a list of free clusters up front. As long as no other FS cluster allocating operations are going on at the same time you could simply log your data to those clusters directly. Then another task, could link them together into a file, on a lower priority basis, ending with and perhaps periodically updating the directory entry for the involved log file. Warren
In article <Xns9DFCA067825DWarrensBlatherings@81.169.183.62>, 
ve3wwg@gmail.com says...
> Mark Borgerson expounded in news:MPG.2702c86b7c64fd25989c4f@news.eternal- > september.org: > > >> OTOH, you can use minimal FATFS software to create and > >> write log file(s). > > > > The problem with a truly minimal FAT16 or FAT32 FS is that > > you either have to cache the FAT and directory entry or > > you get two or more sector writes for each file write operation. > > Not exactly. You only search and update the FAT when > you're adding clusters. Yes, you need one at the beginning > of a new file, but thereafter, it depends upon your > sectors/cluster parameter. Only if this value was 1, would > every [sector] file write require diddling the FAT again to > add a new cluster. Sectors per cluster are usually 8 or > higher (especially higher if the FS is large).
> > > For FAT32, another problem with simple implementations is that > > [adding] a new cluster may require traversing the fat to > > allocate the next open cluster. Avoiding a long sequence > > of sector reads can require either a modification to the > > simple FS, or a large RAM buffer. That problem killed FAT32 > > for when my SDHC files got up to about 200MB. > > This is a problem indeed, if you always start from the > beginning of the FAT table. However, an uncomplicated > way to improve this is to make note of the lowest available > cluster number and store _that_ in a 32-bit variable for > the FS. Then, the _next_ time you go hunting for a free > FAT entry, you only need to start from the FAT sector > containing the free cluster entry (which is easily computed). >
That is probably the best way to handle it if there is no capability to delete files. If you can delete files, you can end up with the free FAT entries scattered all through the table. The next available FAT entry might be at a lower entry than the last used entry. When that happens, you're back to traversing the FAT.
> >> Anyone can just write sectors to the SD without a FS, but > >> it is terribly inelegant. > > > > True. That's why I wrote a simple sequential file system. > > The logger comes with a host program that knows how > > to upload smaller files via USB and can handle the FS when > > the SD card is plugged into a PC. The files are in > > binary format, so some processing program in inevitable. > > > > Mark Borgerson > > If performance in writing is needed and you wanted to retain > the advantage of FAT, I think there are possibilities. For > example, you could preallocate all your clusters for your > log file at the beginning, since you know the sample rate > and duration. Your FS software would simply need to provide > a "rewrite" file sector function as samples are written out.
I've considered that option. On a 16MHZ MPU with SPI SD access, it can take quite a long time to pre-allocate all the clusters on a 32GB SDHC card. I actually wrote a PC application to "clone" such a pre-allocated SD card using direct device writes to "format" the SD card. It turned out to be more trouble than it was worth. If you have all of the 32GB card appearing as one large file, you would then need an app to break out the data into smaller chunks for analysis.
> > Other alternatives could include the building of a list of free > clusters up front. As long as no other FS cluster allocating > operations are going on at the same time you could simply > log your data to those clusters directly. Then another task, > could link them together into a file, on a lower priority > basis, ending with and perhaps periodically updating the > directory entry for the involved log file. >
That sounds like quite a complex scheme for an MSP430 running at 16MHz and collecting 2KBytes of data per second for months. Those low-priority tasks might also have significant power consumption. Mark Borgerson
Mark Borgerson expounded in
news:MPG.27057fc7879fd102989c53@news.eternal-september.org: 

> In article <Xns9DFCA067825DWarrensBlatherings@81.169.183.62>, > ve3wwg@gmail.com says...
>> > For FAT32, another problem with simple implementations is that >> > [adding] a new cluster may require traversing the fat to >> > allocate the next open cluster. Avoiding a long sequence >> > of sector reads can require either a modification to the >> > simple FS, or a large RAM buffer. That problem killed FAT32 >> > for when my SDHC files got up to about 200MB. >> >> This is a problem indeed, if you always start from the >> beginning of the FAT table. However, an uncomplicated >> way to improve this is to make note of the lowest available >> cluster number and store _that_ in a 32-bit variable for >> the FS. Then, the _next_ time you go hunting for a free >> FAT entry, you only need to start from the FAT sector >> containing the free cluster entry (which is easily computed). >> > That is probably the best way to handle it if there is no > capability to delete files. If you can delete files, you > can end up with the free FAT entries scattered all through > the table. The next available FAT entry might be at a > lower entry than the last used entry. When that happens, > you're back to traversing the FAT.
Deletes don't present a problem. The only time the effect of a delete becomes critical is when you're nearly out of space on the volume. But the algorithm I described still works without complications. Let's say you've allocated the last cluster near the end of the FAT. Now your remembered lowest available cluster number is zero because the FAT search didn't encounter a next free cluster. Cluster numbers start at 2, so zero is a good way to indicate 'no cluster'. The current cluster allocation has been satisfied, but the next "might" fail, but.. Then a file gets deleted and say it frees up 2 more clusters. The next attempt to allocate a cluster will see the next available cluster number as zero (none). This indicates the need to start at the beginning of the FAT. If the FAT search (from the beginning) still comes up empty, then you know you're out of space. But in this case you'll get the lowest freed cluster from the last file delete. So deletes don't complicate this. There is no rule that says you have to allocate the lowest numbered cluster. If you wanted to get fancy, you'd actually prefer to keep clusters close together for a given file (if it were a physical disk). But as you know, this is hardly necessary in embedded applications and unnecessary for a memory based FS.
>> If performance in writing is needed and you wanted to retain >> the advantage of FAT, I think there are possibilities. For >> example, you could preallocate all your clusters for your >> log file at the beginning, since you know the sample rate >> and duration. Your FS software would simply need to provide >> a "rewrite" file sector function as samples are written out. > > I've considered that option. On a 16MHZ MPU with SPI SD > access, it can take quite a long time to pre-allocate > all the clusters on a 32GB SDHC card.
Surely this is a small startup time, compared to months of data capture. You can do this in less time than it takes to format the card.
> I actually wrote a PC application to "clone" such a pre-allocated > SD card using direct device writes to "format" the SD card. > It turned out to be more trouble than it was worth.
I don't see it being any more trouble than a format operation. If you leave it to your MCU to do (at startup), then it need not be part of the human procedure.
> If you have all of the 32GB card appearing as one large file, > you would then need an app to break out the data into smaller > chunks for analysis.
That is your choice in application design. You have the same issue whether the data is physically stored or in a file system's "file".
>> Other alternatives could include the building of a list of free >> clusters up front. As long as no other FS cluster allocating >> operations are going on at the same time you could simply >> log your data to those clusters directly. Then another task, >> could link them together into a file, on a lower priority >> basis, ending with and perhaps periodically updating the >> directory entry for the involved log file. > > That sounds like quite a complex scheme for an MSP430 running > at 16MHz and collecting 2KBytes of data per second for months. > Those low-priority tasks might also have significant power > consumption. > > Mark Borgerson
It is complex, but an option. Given that you have months of time to chain together a file, I don't think you're time challenged. Of course it will contend with the SD/mmc device at some level and this may introduce unacceptable latency, which must be considered. Pre-allocating is definitely simpler and could be done after a RESET, after opening the FS on the SD/mmc card. Warren
In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, 
ve3wwg@gmail.com says...
> Mark Borgerson expounded in > news:MPG.27057fc7879fd102989c53@news.eternal-september.org: > > > In article <Xns9DFCA067825DWarrensBlatherings@81.169.183.62>, > > ve3wwg@gmail.com says... > > >> > For FAT32, another problem with simple implementations is that > >> > [adding] a new cluster may require traversing the fat to > >> > allocate the next open cluster. Avoiding a long sequence > >> > of sector reads can require either a modification to the > >> > simple FS, or a large RAM buffer. That problem killed FAT32 > >> > for when my SDHC files got up to about 200MB. > >> > >> This is a problem indeed, if you always start from the > >> beginning of the FAT table. However, an uncomplicated > >> way to improve this is to make note of the lowest available > >> cluster number and store _that_ in a 32-bit variable for > >> the FS. Then, the _next_ time you go hunting for a free > >> FAT entry, you only need to start from the FAT sector > >> containing the free cluster entry (which is easily computed). > >> > > That is probably the best way to handle it if there is no > > capability to delete files. If you can delete files, you > > can end up with the free FAT entries scattered all through > > the table. The next available FAT entry might be at a > > lower entry than the last used entry. When that happens, > > you're back to traversing the FAT. > > Deletes don't present a problem. The only time the effect of > a delete becomes critical is when you're nearly out of space > on the volume. But the algorithm I described still works > without complications. > > Let's say you've allocated the last cluster near the end > of the FAT. Now your remembered lowest available cluster number > is zero because the FAT search didn't encounter a next > free cluster. Cluster numbers start at 2, so zero is a good > way to indicate 'no cluster'. The current cluster allocation > has been satisfied, but the next "might" fail, but.. > > Then a file gets deleted and say it frees up 2 more clusters. > > The next attempt to allocate a cluster will see the next > available cluster number as zero (none). This indicates > the need to start at the beginning of the FAT. If the FAT > search (from the beginning) still comes up empty, then you > know you're out of space. But in this case you'll get the > lowest freed cluster from the last file delete.
If the deleted file happens to be somewhere in the middle of the SD card, you may have to scan a lot of FAT to find that free cluster.
> > So deletes don't complicate this. There is no rule that > says you have to allocate the lowest numbered cluster.
That's correct. However, that is the way it was done in the simple FAT system I started with.
> > If you wanted to get fancy, you'd actually prefer to keep > clusters close together for a given file (if it were a > physical disk). But as you know, this is hardly necessary > in embedded applications and unnecessary for a > memory based FS. > > >> If performance in writing is needed and you wanted to retain > >> the advantage of FAT, I think there are possibilities. For > >> example, you could preallocate all your clusters for your > >> log file at the beginning, since you know the sample rate > >> and duration. Your FS software would simply need to provide > >> a "rewrite" file sector function as samples are written out. > > > > I've considered that option. On a 16MHZ MPU with SPI SD > > access, it can take quite a long time to pre-allocate > > all the clusters on a 32GB SDHC card. > > Surely this is a small startup time, compared to months of > data capture. You can do this in less time than it > takes to format the card.
It could be small in relation to the months of logging, but large in relation to the other tasks needed to prepare the system.. On a FAT32 system, the FAT can be as large as about 16MB. That's about 32768 512-byte sectors. On a slow system, you might only be able to write 15K or 30 sectors per second. That means that formatting the SD card might take 15 minutes. Multiply that by 4 cards, and you have about an hour to format the storage on the logging system.
> > > I actually wrote a PC application to "clone" such a pre-allocated > > SD card using direct device writes to "format" the SD card. > > It turned out to be more trouble than it was worth. > > I don't see it being any more trouble than a format operation. > If you leave it to your MCU to do (at startup), then it need not > be part of the human procedure.
That's not a bad idea. The formatting of the disk could be part of a burn-in procedure at initial build. The only other downside is that you could lose 16MB (or 32 if there are dual FATS) of storage space. That's not a large factor, though.
> > > If you have all of the 32GB card appearing as one large file, > > you would then need an app to break out the data into smaller > > chunks for analysis. > > That is your choice in application design. You have the > same issue whether the data is physically stored or in > a file system's "file". > > >> Other alternatives could include the building of a list of free > >> clusters up front. As long as no other FS cluster allocating > >> operations are going on at the same time you could simply > >> log your data to those clusters directly. Then another task, > >> could link them together into a file, on a lower priority > >> basis, ending with and perhaps periodically updating the > >> directory entry for the involved log file. > > > > That sounds like quite a complex scheme for an MSP430 running > > at 16MHz and collecting 2KBytes of data per second for months. > > Those low-priority tasks might also have significant power > > consumption. > > > > Mark Borgerson > > It is complex, but an option. Given that you have months > of time to chain together a file, I don't think you're time > challenged. Of course it will contend with the SD/mmc device > at some level and this may introduce unacceptable latency, > which must be considered.
That might be the largest problem. The MSP430 doesn't have a lot of spare RAM to buffer data if some other process is blocking access to the SD card.
> > Pre-allocating is definitely simpler and could be done after > a RESET, after opening the FS on the SD/mmc card.
Given that it might take 15 to 60 minutes, It probably can't be done on each power up or reset. That's particularly true if the FS has to retain calibration data over multiple logging sessions. Mark Borgerson
Mark Borgerson expounded in news:MPG.27067f7f10afd8ec989c57@news.eternal-
september.org:

> In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, > ve3wwg@gmail.com says...
>> The next attempt to allocate a cluster will see the next >> available cluster number as zero (none). This indicates >> the need to start at the beginning of the FAT. If the FAT >> search (from the beginning) still comes up empty, then you >> know you're out of space. But in this case you'll get the >> lowest freed cluster from the last file delete. > > If the deleted file happens to be somewhere in the middle > of the SD card, you may have to scan a lot of FAT to > find that free cluster.
Simple optimizations don't guarantee that you always save a long search. ;-) You can of course, make a slight improvement by noting that if the saved available cluster is currently zero and you're deleting clusters, make note of one of them. Then at least you skip some of that overhead, in the next allocate cluster call.
>> > I've considered that option. On a 16MHZ MPU with SPI SD >> > access, it can take quite a long time to pre-allocate >> > all the clusters on a 32GB SDHC card. >> >> Surely this is a small startup time, compared to months of >> data capture. You can do this in less time than it >> takes to format the card. > > It could be small in relation to the months of logging, but > large in relation to the other tasks needed to prepare > the system.. On a FAT32 system, the FAT can be as large > as about 16MB. That's about 32768 512-byte sectors. > On a slow system, you might only be able to write 15K or > 30 sectors per second. That means that formatting the > SD card might take 15 minutes.
But but.. on a memory card, unlike a floppy, you don't need to write _every_ sector. You need to write the boot sector, root dir sector(s) (FAT16), and _one_ FAT table. If you're formatting FAT32, then you also need to allocate one cluster to hold the start of the root directory and fill that cluster with zeros. All remaining clusters can be left in "as is" condition. So if you choose your sectors/cluster paramter wisely and choose 1 FAT table, then you only need to write a very small fraction of the whole FS image. For floppies, the format was also a phsical sector operation in addition to setting up the FS. But a memory card doesn't need this. The only reason I can think of for doing a full format on a memory card, would be to discover any bad sectors. But this is something that the FS software might be able to do on the fly, if it is designed for it. I don't have any evidence on how often memory cards have bad sectors. But an up front screening process could eliminate unsuitable cards. It really comes down to how critical your data capture application is. If it _is_ indeed critical, then you don't have any choice in this matter anyway- a full format will be required.
>> I don't see it being any more trouble than a format operation. >> If you leave it to your MCU to do (at startup), then it need not >> be part of the human procedure. > > That's not a bad idea. The formatting of the disk could be > part of a burn-in procedure at initial build. > > The only other downside is that you could lose 16MB (or 32 if > there are dual FATS) of storage space. That's not a large > factor, though.
You only need 1 FAT - so format it that way. Problem solved. Multiple FATs were designed for flakey floppy disks. Hardly necessary on a memory card, unless you intend to use it close to the flash rewrite limits. But that wouldn't be prudent for data logging purposes.
>> > That sounds like quite a complex scheme for an MSP430 running >> > at 16MHz and collecting 2KBytes of data per second for months. >> > Those low-priority tasks might also have significant power >> > consumption. >> > >> > Mark Borgerson >> >> It is complex, but an option. Given that you have months >> of time to chain together a file, I don't think you're time >> challenged. Of course it will contend with the SD/mmc device >> at some level and this may introduce unacceptable latency, >> which must be considered. > > That might be the largest problem. The MSP430 doesn't have > a lot of spare RAM to buffer data if some other process is > blocking access to the SD card.
True, and just running another "task" requires allocating a separate stack etc.
>> Pre-allocating is definitely simpler and could be done after >> a RESET, after opening the FS on the SD/mmc card. > > Given that it might take 15 to 60 minutes, It probably can't > be done on each power up or reset. That's particularly true > if the FS has to retain calibration data over multiple logging > sessions. > > Mark Borgerson
I assumed continuous operation, once started. Obviously, if it must be restartable, then perhaps it could be done only as part of some special startup handshake, like a button press. Warren
In article <Xns9DFD9C76F438SnarkCharmedFerSure@188.40.43.230>, 
snark@cogeco.ca says...
> Mark Borgerson expounded in news:MPG.27067f7f10afd8ec989c57@news.eternal- > september.org: > > > In article <Xns9DFD5DB3E1435WarrensBlatherings@188.40.43.230>, > > ve3wwg@gmail.com says... > > >> The next attempt to allocate a cluster will see the next > >> available cluster number as zero (none). This indicates > >> the need to start at the beginning of the FAT. If the FAT > >> search (from the beginning) still comes up empty, then you > >> know you're out of space. But in this case you'll get the > >> lowest freed cluster from the last file delete. > > > > If the deleted file happens to be somewhere in the middle > > of the SD card, you may have to scan a lot of FAT to > > find that free cluster. > > Simple optimizations don't guarantee that you always save > a long search. ;-) > > You can of course, make a slight improvement by noting > that if the saved available cluster is currently > zero and you're deleting clusters, make note of one of them. > Then at least you skip some of that overhead, in the > next allocate cluster call. > > >> > I've considered that option. On a 16MHZ MPU with SPI SD > >> > access, it can take quite a long time to pre-allocate > >> > all the clusters on a 32GB SDHC card. > >> > >> Surely this is a small startup time, compared to months of > >> data capture. You can do this in less time than it > >> takes to format the card. > > > > It could be small in relation to the months of logging, but > > large in relation to the other tasks needed to prepare > > the system.. On a FAT32 system, the FAT can be as large > > as about 16MB. That's about 32768 512-byte sectors. > > On a slow system, you might only be able to write 15K or > > 30 sectors per second. That means that formatting the > > SD card might take 15 minutes. > > But but.. on a memory card, unlike a floppy, you don't need > to write _every_ sector. You need to write the boot sector, > root dir sector(s) (FAT16), and _one_ FAT table. If > you're formatting FAT32, then you also need to allocate one > cluster to hold the start of the root directory and fill > that cluster with zeros. All remaining clusters can > be left in "as is" condition.
Are you assuming that the card has never been used and is starting with 0xFF in all sectors? I suppose that would be fine if the system was never reused or tested before being deployed.
> > So if you choose your sectors/cluster paramter wisely and > choose 1 FAT table, then you only need to write a very small > fraction of the whole FS image. > > For floppies, the format was also a phsical sector operation > in addition to setting up the FS. But a memory card doesn't > need this. > > The only reason I can think of for doing a full format on > a memory card, would be to discover any bad sectors. But > this is something that the FS software might be able to > do on the fly, if it is designed for it.
If you're using an SD card, doesn't the internal controller in the card handle that (as well as wear leveling)?
> > I don't have any evidence on how often memory cards have > bad sectors. But an up front screening process could > eliminate unsuitable cards. It really comes down to how > critical your data capture application is. If it _is_ > indeed critical, then you don't have any choice in this > matter anyway- a full format will be required. > > >> I don't see it being any more trouble than a format operation. > >> If you leave it to your MCU to do (at startup), then it need not > >> be part of the human procedure. > > > > That's not a bad idea. The formatting of the disk could be > > part of a burn-in procedure at initial build. > > > > The only other downside is that you could lose 16MB (or 32 if > > there are dual FATS) of storage space. That's not a large > > factor, though. > > You only need 1 FAT - so format it that way. Problem solved. > > Multiple FATs were designed for flakey floppy disks. Hardly > necessary on a memory card, unless you intend to use it close > to the flash rewrite limits. But that wouldn't be prudent > for data logging purposes. > > >> > That sounds like quite a complex scheme for an MSP430 running > >> > at 16MHz and collecting 2KBytes of data per second for months. > >> > Those low-priority tasks might also have significant power > >> > consumption. > >> > > >> > Mark Borgerson > >> > >> It is complex, but an option. Given that you have months > >> of time to chain together a file, I don't think you're time > >> challenged. Of course it will contend with the SD/mmc device > >> at some level and this may introduce unacceptable latency, > >> which must be considered. > > > > That might be the largest problem. The MSP430 doesn't have > > a lot of spare RAM to buffer data if some other process is > > blocking access to the SD card. > > True, and just running another "task" requires allocating a > separate stack etc. > > >> Pre-allocating is definitely simpler and could be done after > >> a RESET, after opening the FS on the SD/mmc card. > > > > Given that it might take 15 to 60 minutes, It probably can't > > be done on each power up or reset. That's particularly true > > if the FS has to retain calibration data over multiple logging > > sessions. > > > > Mark Borgerson > > I assumed continuous operation, once started. Obviously, if it > must be restartable, then perhaps it could be done only as part > of some special startup handshake, like a button press. >
Mark Borgerson
Mark Borgerson expounded in
news:MPG.2706a0cd11868d9989c58@news.eternal-september.org: 

> In article <Xns9DFD9C76F438SnarkCharmedFerSure@188.40.43.230>, > snark@cogeco.ca says...
>> But but.. on a memory card, unlike a floppy, you don't need >> to write _every_ sector. You need to write the boot sector, >> root dir sector(s) (FAT16), and _one_ FAT table. If >> you're formatting FAT32, then you also need to allocate one >> cluster to hold the start of the root directory and fill >> that cluster with zeros. All remaining clusters can >> be left in "as is" condition. > > Are you assuming that the card has never been used and > is starting with 0xFF in all sectors? I suppose that would > be fine if the system was never reused or tested before > being deployed.
...
> Mark Borgerson
It doesn't matter. Why would it? When you allocate a new cluster to a root/sub-directory, you zero it then (zero is the 'end of directory' marker). When you allocate a cluster to a file (being written and extended), you just make the cluster available - the application just overwrites the garbage in those sectors. The flash controller takes care of erasing the "pages" of memory and "rewrites" the sector as needed. To the end user, it just looks like a disk. Warren