Hi All, I'm looking for a good and small flash file system, i.e. I'm looking for a way to store data in flash without having to store all the data at specific locations. So suppose you receive a data stream and need to store this for a while, and than a next stream get in, the old stream needs to be preserved until it is old, and the new one needs to be stored as well. I've read about Tiny File System (TFS) in embedded systems Firmware demystified, does anybody have any experience with this or such a system?? thanks, Martijn
Tiny Flash File System
Started by ●July 21, 2003
Reply by ●July 21, 20032003-07-21
I use the flash in the '149 as a long term data logging store. I use
segment A to store fixed, or very rarely changed parameters. I use
segment B to store the last programmed address. This works fine when you
know the record length. Essentially when I get to the end of the
allocated memory I start erasing segments from the beginning, and write
a flash cycle counter back to segment A. This information tells me that
I last wrote to abcd, that I have written the flash data base nn times,
and that, because abcd is in the third allocated flash segment, I am
overwriting from the start again, and need to erase the next segment
once I get close to the end of this one.
In segment B, because the last programmed address will never be 0xFFFF,
I use the entire segment, At power on I look for the first address in
the block that is -1 (0xFFFF). This is my next free address to write. If
I reach the end of segment I re-write the last entry to the first
address in the segment after erasing it. This lets me store 63
addresses, or programming events per loop through memory. I could
allocate 1 or more 512 byte segments for larger re-writes, but this is a
simple scheme that works well for fixed length records.
For variable length records I would probably use either a larger number
of segments to store the9ir individual addresses, or use an EOF marker.
At the end of the day it is a lot easier though to fix the record length
if at all possible.
Al
Martijn Broens wrote:
> Hi All,
>
> I'm looking for a good and small flash file system, i.e. I'm
looking for
> a way to store data in flash without having to store all the data at
> specific locations.
> So suppose you receive a data stream and need to store this for a while,
> and than a next stream get in, the old stream needs to be preserved
> until it is old, and the new one needs to be stored as well. I've read
> about Tiny File System (TFS) in embedded systems Firmware demystified,
> does anybody have any experience with this or such a system??
>
> thanks,
> Martijn
>
>
>
>
>
>
>
> .
>
>
>
> ">http://docs.yahoo.com/info/terms/
>
>
>
Reply by ●July 21, 20032003-07-21
Hi Al, Thanks for the reply, would this also count for data streams of about 314Chars ascii data? I thought of using it like a ringbuffer or some thing like that, only now you would need to keep track of where a stream starts, and ends. Or is this a wrong perspective? thanks, Martijn -----Oorspronkelijk bericht----- Van: onestone [mailto:onestone@ones...] Verzonden: maandag 21 juli 2003 9:07 Aan: msp430@msp4... Onderwerp: Re: [msp430] Tiny Flash File System I use the flash in the '149 as a long term data logging store. I use segment A to store fixed, or very rarely changed parameters. I use segment B to store the last programmed address. This works fine when you know the record length. Essentially when I get to the end of the allocated memory I start erasing segments from the beginning, and write a flash cycle counter back to segment A. This information tells me that I last wrote to abcd, that I have written the flash data base nn times, and that, because abcd is in the third allocated flash segment, I am overwriting from the start again, and need to erase the next segment once I get close to the end of this one. In segment B, because the last programmed address will never be 0xFFFF, I use the entire segment, At power on I look for the first address in the block that is -1 (0xFFFF). This is my next free address to write. If I reach the end of segment I re-write the last entry to the first address in the segment after erasing it. This lets me store 63 addresses, or programming events per loop through memory. I could allocate 1 or more 512 byte segments for larger re-writes, but this is a simple scheme that works well for fixed length records. For variable length records I would probably use either a larger number of segments to store the9ir individual addresses, or use an EOF marker. At the end of the day it is a lot easier though to fix the record length if at all possible. Al Martijn Broens wrote: > Hi All, > > I'm looking for a good and small flash file system, i.e. I'm looking for > a way to store data in flash without having to store all the data at > specific locations. > So suppose you receive a data stream and need to store this for a while, > and than a next stream get in, the old stream needs to be preserved > until it is old, and the new one needs to be stored as well. I've read > about Tiny File System (TFS) in embedded systems Firmware demystified, > does anybody have any experience with this or such a system?? > > thanks, > Martijn > > > > > > > > . > > > > ">http://docs.yahoo.com/info/terms/ > > > <http://rd.yahoo.com/M4081.3551198.4824677.1261774/D=egroupweb/S05 005378:HM/A63535/R=0/SIGps6rfef/*http:/www.ediets.com/start.cfm?co de0504&media=atkins> click here <http://us.adserver.yahoo.com/l?M4081.3551198.4824677.1261774/D=egrou pmail/S=:HM/A63535/rand5545669> . ">http://docs.yahoo.com/info/terms/> Terms of Service.
Reply by ●July 21, 20032003-07-21
Al,
Do you rewrite Segment A or Segment B each time you write a record to
flash? Won't you wear our the info segments well before the main flash?
Perhaps this isn't an issue if you have a known maximum number of writes
to the same sector for the lifetime of the device.
Having written a flash filing system for POS terminal devices, where
data retention is paramount, we did exactly what most FFS do: wear
leveling (so we don't wear out sectors too soon) bad sector remapping
(so that when we do, we can detect it) and transactioning (so on a
powerfail the FS wasn't fried). It's almost impossible to test this
on
a real device, so we implemented a simulation of it to introduce random
bit errors in the flash and power failure.
The idea was not to use a single "master" sector to store important
data
as if it gets rewritten each time a true data sector is written, the
master sector gets worn out ("burnt") first.
-- Paul.
> -----Original Message-----
> From: onestone [mailto:onestone@ones...]
> Sent: 21 July 2003 08:07
> To: msp430@msp4...
> Subject: Re: [msp430] Tiny Flash File System
>
>
> I use the flash in the '149 as a long term data logging store. I use
> segment A to store fixed, or very rarely changed parameters. I use
> segment B to store the last programmed address. This works
> fine when you
> know the record length. Essentially when I get to the end of the
> allocated memory I start erasing segments from the beginning,
> and write
> a flash cycle counter back to segment A. This information
> tells me that
> I last wrote to abcd, that I have written the flash data base
> nn times,
> and that, because abcd is in the third allocated flash segment, I am
> overwriting from the start again, and need to erase the next segment
> once I get close to the end of this one.
>
> In segment B, because the last programmed address will never
> be 0xFFFF,
> I use the entire segment, At power on I look for the first address in
> the block that is -1 (0xFFFF). This is my next free address
> to write. If
> I reach the end of segment I re-write the last entry to the first
> address in the segment after erasing it. This lets me store 63
> addresses, or programming events per loop through memory. I could
> allocate 1 or more 512 byte segments for larger re-writes,
> but this is a
> simple scheme that works well for fixed length records.
>
> For variable length records I would probably use either a
> larger number
> of segments to store the9ir individual addresses, or use an
> EOF marker.
> At the end of the day it is a lot easier though to fix the
> record length
> if at all possible.
>
> Al
>
> Martijn Broens wrote:
>
> > Hi All,
> >
> > I'm looking for a good and small flash file system, i.e.
> I'm looking
> > for a way to store data in flash without having to store
> all the data
> > at specific locations. So suppose you receive a data stream
> and need
> > to store this for a while, and than a next stream get in, the old
> > stream needs to be preserved until it is old, and the new
> one needs to
> > be stored as well. I've read about Tiny File System (TFS)
> in embedded
> > systems Firmware demystified, does anybody have any experience with
> > this or such a system??
> >
> > thanks,
> > Martijn
> >
> >
> >
> >
> >
> >
> >
> > .
> >
> >
> >
> > ">http://docs.yahoo.com/info/terms/
> >
> >
> >
>
>
> ------------------------ Yahoo! Groups Sponsor
> ---------------------~--> Control Hunger EZ with fast acting
> oral EZ Appetite Suppressant Spray from VitaminBoost.com. 1
> Month Supply - 2oz., 80 sprays: $19.97.
> http://www.challengerone.com/t/l.asp?cid(66&lp=ezappetite3.h
tml
http://us.click.yahoo.com/aSJFwB/XWXGAA/ySSFAA/CFFolB/TM
---------------------------------~->
.
">http://docs.yahoo.com/info/terms/
Reply by ●July 21, 20032003-07-21
Hi Paul,
Could you tell me where i can find more info on how these ffs systems
work?
thanks,
Martijn
-----Oorspronkelijk bericht-----
Van: Paul Curtis [mailto:plc@plc@...]
Verzonden: maandag 21 juli 2003 11:23
Aan: msp430@msp4...
Onderwerp: RE: [msp430] Tiny Flash File System
Al,
Do you rewrite Segment A or Segment B each time you write a record to
flash? Won't you wear our the info segments well before the main flash?
Perhaps this isn't an issue if you have a known maximum number of writes
to the same sector for the lifetime of the device.
Having written a flash filing system for POS terminal devices, where
data retention is paramount, we did exactly what most FFS do: wear
leveling (so we don't wear out sectors too soon) bad sector remapping
(so that when we do, we can detect it) and transactioning (so on a
powerfail the FS wasn't fried). It's almost impossible to test this
on
a real device, so we implemented a simulation of it to introduce random
bit errors in the flash and power failure.
The idea was not to use a single "master" sector to store important
data
as if it gets rewritten each time a true data sector is written, the
master sector gets worn out ("burnt") first.
-- Paul.
> -----Original Message-----
> From: onestone [mailto:onestone@ones...]
> Sent: 21 July 2003 08:07
> To: msp430@msp4...
> Subject: Re: [msp430] Tiny Flash File System
>
>
> I use the flash in the '149 as a long term data logging store. I use
> segment A to store fixed, or very rarely changed parameters. I use
> segment B to store the last programmed address. This works
> fine when you
> know the record length. Essentially when I get to the end of the
> allocated memory I start erasing segments from the beginning,
> and write
> a flash cycle counter back to segment A. This information
> tells me that
> I last wrote to abcd, that I have written the flash data base
> nn times,
> and that, because abcd is in the third allocated flash segment, I am
> overwriting from the start again, and need to erase the next segment
> once I get close to the end of this one.
>
> In segment B, because the last programmed address will never
> be 0xFFFF,
> I use the entire segment, At power on I look for the first address in
> the block that is -1 (0xFFFF). This is my next free address
> to write. If
> I reach the end of segment I re-write the last entry to the first
> address in the segment after erasing it. This lets me store 63
> addresses, or programming events per loop through memory. I could
> allocate 1 or more 512 byte segments for larger re-writes,
> but this is a
> simple scheme that works well for fixed length records.
>
> For variable length records I would probably use either a
> larger number
> of segments to store the9ir individual addresses, or use an
> EOF marker.
> At the end of the day it is a lot easier though to fix the
> record length
> if at all possible.
>
> Al
>
> Martijn Broens wrote:
>
> > Hi All,
> >
> > I'm looking for a good and small flash file system, i.e.
> I'm looking
> > for a way to store data in flash without having to store
> all the data
> > at specific locations. So suppose you receive a data stream
> and need
> > to store this for a while, and than a next stream get in, the old
> > stream needs to be preserved until it is old, and the new
> one needs to
> > be stored as well. I've read about Tiny File System (TFS)
> in embedded
> > systems Firmware demystified, does anybody have any experience with
> > this or such a system??
> >
> > thanks,
> > Martijn
> >
> >
> >
> >
> >
> >
> >
> > .
> >
> >
> >
> > ">http://docs.yahoo.com/info/terms/
> >
> >
> >
>
>
> ------------------------
<http://rd.yahoo.com/M%1812.3170658.4537139.1261774/D=egroupweb/S05
005378:HM/A52964/R=0/SIGt2ts2ch/*http:/www.netflix.com/Default?mqs
o`178276&partid170658> click here
<http://us.adserver.yahoo.com/l?M%1812.3170658.4537139.1261774/D=egrou
pmail/S=:HM/A52964/randg0778609>
.
">http://docs.yahoo.com/info/terms/> Terms of Service.
Reply by ●July 21, 20032003-07-21
Paul Curtis wrote: > Al, > > Do you rewrite Segment A or Segment B each time you write a record to > flash? Won't you wear our the info segments well before the main flash? > Perhaps this isn't an issue if you have a known maximum number of writes > to the same sector for the lifetime of the device. No. Segment A is used for stuff that might get re-written once in every few months typically. For example one particular system has up to 32 'personalities', This includes different tables for data output, different logo screens for different clients, and different levels of security. This data might only ever change when the system is re-allocated to another client. In addition this segment is used to track bad addresses. since the system data only occupies 16 bytes I have the ability to track up to 56 failed memory words. I haven't yet had to use one, but it's there. Segment B keeps a running count, hence once it is erased the next pointer is written sequentially, hence with segment B I can write 64 entries before I need to erase the segment and start again. Typically the machine might write 4 entries a day, so segment B gets re-written every 16 days. I figure that 330 erase/program cycles in 10 years should be well within reach of modern day flash, and have the bad memory tracker to compensate for failures. If this gets filled It will notify the user. I didn't think it worth while allowing it to auto allocate a new segment for this. I thought of allocating an entire segment for failure tracking but didn't want to waste the memory. Oh, forgot, I take advantage of the fact, in this design, that I only use memory above 8000H for data store, so bit 15 is always set. A memory failure that does not have bit 15 set indicates an entire segment is faulty , or that all 56 word failures reside in one segment, hence that segment is taken out. I designed this to be extremely simple, yet to be able to make the very most of the available resources. > > Having written a flash filing system for POS terminal devices, where > data retention is paramount, we did exactly what most FFS do: wear > leveling (so we don't wear out sectors too soon) bad sector remapping > (so that when we do, we can detect it) and transactioning (so on a > powerfail the FS wasn't fried). It's almost impossible to test this on > a real device, so we implemented a simulation of it to introduce random > bit errors in the flash and power failure. > > The idea was not to use a single "master" sector to store important data > as if it gets rewritten each time a true data sector is written, the > master sector gets worn out ("burnt") first. My write frequencies suggest that the device will be replaced long before I'm likely to wear out the flash. The failed memory tracker can also track failures in the so called data segments. The device does not have a hardware on/off switch. It is all software controlled, hence the user would have to open the device up to be able to disconnect the battery and cause a power fail durign flash write (that's their problem, and voids any warranty. I have tamper detectors in the box as the units are franchised). I also use a 3V3 switcher for power regulation, and monitor the supply voltage so that no attempt is made to write flash when the battery is too low. This includes temperature monitoring, so that battery performance at different temperatures is accounted for. I don't think I missed anything, but would love to know if there are any flaws you guys can see with this. Al
Reply by ●July 21, 20032003-07-21
Al,
[ big snip ]
> I don't think I missed anything, but would
love to know if
> there are any
> flaws you guys can see with this.
Given the write frequency, there's not much that can go wrong with your
implementation, I think.
My case had a high load on the flash. There are a lot of "database"
entries that get updated in a POS terminal during the day and night,
such as the hot card file and transaction logs (consider use in a
supermarket checkout). These things are used every day, day in day out,
and so flash wear *is* a problem. Hence, I coded the FFS to cater for
imperfect flashing, power failure, and high usage. Typically a POS
terminal has 512K of FLASH to manage.
-- Paul.
Reply by ●July 21, 20032003-07-21
Martijn,
> Could you tell me where i can find more info on
how these ffs
> systems work?
The FFS implementation depends upon the flash technology you're managing
(NAND, NOR). A Google around for TrueFFS and Flash Filing System will
get you somewhere. There is Linux code, by Aleph One I believe, that
manages a NAND-technology FLASH disk. Anything that manages flash on an
SD/MMC/SmartMedia card or Compact Flash card will require wear leveling
and requisite protection, so start looking around for that. Intel used
to have their Small Block Manager software available for download if you
were using an Intel flash, but I think that is now only available
through a license agreement. Similarly, Samsung had management code if
you used the Smart Media cards (I used these for one project, but didn't
use their code to implement the filing system).
For myself, I implemented the flash filing code from scratch for
customer use--twice, for different customers. It didn't need to be FAT
compatible, so a simple file system on top of a robust block manager was
all that was required of these two embedded systems.
-- Paul.
Reply by ●July 21, 20032003-07-21
Martijn Broens wrote: > Hi Al, > > Thanks for the reply, would this also count for data streams of about > 314Chars ascii data? I thought of using it like a ringbuffer or some > thing like that, only now you would need to keep track of where a stream > starts, and ends. Or is this a wrong perspective? Two simple ways to approach this that immediately spring to mind. I'll assume that your data base starts at 8000H, and ends at FDFFH, ie 512 bytes short of the last segment. This saves getting messy with the interrupt vectors. This method allows you to save 107 files at 314 characters long, without messing around with 7 bit compression. If your streams are all the same length thien the only thing you need to do is track the next free memory space. All records will start at 8000H, and each record will have an absolute position in memory. This simplifies tracking as well, you only need to use one byte in the Segment B index, thus you could track all 107 entries by clearing there relevant entry byte to 0 if used. In fact you could even allocate one bit per record to indicate its use, and clear the bits to 0, but this seems unnecessarily fiddly. Alternatively you could get even simpler and allocate one segment per record, thus you could deal with any record length up to 512 bytes. and buffer up to 63 records. This is wasteful of memory though and I hate waste. If your records are of different length you have a couple of options open to you. If I can assume by ASCII you mean characters from 0-127 then you can use BIT 7 to indicate End Of File (EOF). Thus you can again get away with tracking the very last file position only, for writing, and use EOF markers to extract individual records. memory scanning is very fast. If you want to track each record and you can guaraantee them being under 512 bytes I would be tempted to use Segment B again, or segments A and B if necessary and just store the length of each record in words. This makes recovery a bit easier, but when you wrap around memory you will have to make sure you erase all entries that are affected by each erased segment. Failing that you can track the head and tail of each entry, but will need to use more memory to do this. Al > > thanks, > Martijn > -----Oorspronkelijk bericht----- > Van: onestone [mailto:onestone@ones...] > Verzonden: maandag 21 juli 2003 9:07 > Aan: msp430@msp4... > Onderwerp: Re: [msp430] Tiny Flash File System > > I use the flash in the '149 as a long term data logging store. I use > segment A to store fixed, or very rarely changed parameters. I use > segment B to store the last programmed address. This works fine when you > > know the record length. Essentially when I get to the end of the > allocated memory I start erasing segments from the beginning, and write > a flash cycle counter back to segment A. This information tells me that > I last wrote to abcd, that I have written the flash data base nn times, > and that, because abcd is in the third allocated flash segment, I am > overwriting from the start again, and need to erase the next segment > once I get close to the end of this one. > > In segment B, because the last programmed address will never be 0xFFFF, > I use the entire segment, At power on I look for the first address in > the block that is -1 (0xFFFF). This is my next free address to write. If > > I reach the end of segment I re-write the last entry to the first > address in the segment after erasing it. This lets me store 63 > addresses, or programming events per loop through memory. I could > allocate 1 or more 512 byte segments for larger re-writes, but this is a > > simple scheme that works well for fixed length records. > > For variable length records I would probably use either a larger number > of segments to store the9ir individual addresses, or use an EOF marker. > At the end of the day it is a lot easier though to fix the record length > > if at all possible. > > Al > > Martijn Broens wrote: > > >>Hi All, >> >>I'm looking for a good and small flash file system, i.e. I'm looking > > for > >>a way to store data in flash without having to store all the data at >>specific locations. >>So suppose you receive a data stream and need to store this for a > > while, > >>and than a next stream get in, the old stream needs to be preserved >>until it is old, and the new one needs to be stored as well. I've read >>about Tiny File System (TFS) in embedded systems Firmware demystified, >>does anybody have any experience with this or such a system?? >> >>thanks, >>Martijn >> >> >> >> >> >> >> >>. >> >> >> >>">http://docs.yahoo.com/info/terms/ > >> >> > > > > > > > > <http://rd.yahoo.com/M4081.3551198.4824677.1261774/D=egroupweb/S05 > 005378:HM/A63535/R=0/SIGps6rfef/*http:/www.ediets.com/start.cfm?co > de0504&media=atkins> click here > > > <http://us.adserver.yahoo.com/l?M4081.3551198.4824677.1261774/D=egrou > pmail/S=:HM/A63535/rand5545669> > > . > > > > ">http://docs.yahoo.com/info/terms/> Terms of Service. > > > > > > > . > > > > ">http://docs.yahoo.com/info/terms/ > > >
Reply by ●July 21, 20032003-07-21
Paul Curtis wrote:
> Al,
>
> [ big snip ]
>
>
>>I don't think I missed anything, but would love to know if
>>there are any
>>flaws you guys can see with this.
>
>
> Given the write frequency, there's not much that can go wrong with
your
> implementation, I think.
>
> My case had a high load on the flash. There are a lot of
"database"
> entries that get updated in a POS terminal during the day and night,
> such as the hot card file and transaction logs (consider use in a
> supermarket checkout). These things are used every day, day in day out,
> and so flash wear *is* a problem. Hence, I coded the FFS to cater for
> imperfect flashing, power failure, and high usage. Typically a POS
> terminal has 512K of FLASH to manage.
In my long ago misspent youth I worked for ICL, I got out as they
started moving from mainframes to POS. Given adequate memory, and I
assume that 512k is more than enough to hold a days transactions, I
would have thought that most modern flash parts could handle the <4,000
write cycles used in 10 years. Of course, even with my simple
implementation I've used flash erase/write verification, and flash
failure. Power fail isn't an issue as I explained. In a system with
on/off hardwiring I 'd be very tempted to add a supercap and power fail
sensor to make sure I can get the final write completed, and in mission
critical stuff I'd FEC the whole lot. Fortunately my needs were simple
and predictable, and, within the limits imposed by the MSP430 give me
excellent results.
Al
