Hi Everybody,
I was reading through several posts in this group and in the ZW BB
looking for information on the various FAT implementations on Serial
Flash devices.
I was looking information specially on FAT implementations supporting
SF1000, SD/MMC or Atmel DataFlashes. Im aware (and its a design
goal for me) that only SD/MMC formats provide interoperability with
PC platforms.
Im concerned, however, about the wear leveling issue and how it
could be implemented on FAT Flash devices. As long as I know, FAT
uses some critical structures (BPB, MBR, Root Dir) and assumes they
are located as fixed and non-moving physical locations on the device.
At first look, that's incompatible with any wear leveling algorithm,
that are generally based on the possibility of balancing the
write/erase cycles of physical sectors, using always the younger free
ones on every logical rewrite instead of rewriting the same physical
sector.
Although Ive seen some articles on the net discussing some wear-
leveling algorithms, they assume there's a HAL (Hardware Abstraction
Layer) located somewhere the FAT Filesystem and the physical
sectors/blocks in the device). The fact is that if that HAL is not
implemented at device-level (nor Atmel ATDBxxxx, nor MMC devices do
it, I guess) implementing it at filesystem level makes that device
incompatible with PC readers.
In the other hand, implementing the FAT format as is in Flash
memories, makes them mostly WORM (Write Once, Read Multiple) devices,
due to the fact that any simple file create/write/append involves
several updates of the FAT (File Allocation Table) and/or Root Dir
sectors.
Its not hard to imagine that the sectors containing the FAT (File
Allocation Table) will quickly consume its write/erase cycles, making
the entire flash unusable.
Have any of you considered this issue ? Have any of you tested the
way the ZW or other (SHDesigns, Progressive Resources and others) FAT
implementations take the wear leveling issue into account ?
In the light of the popularity of Flash memories and the amount of
people starting to use this kind of devices for mass storage, I found
this issue worth to discuss a little.
Dont you agree ?
Regards,
Alberto Lopez
FAT / Flash Wear Leveling
Started by ●September 16, 2005
Reply by ●September 16, 20052005-09-16
At 02:55 PM 9/16/2005, you wrote:
>In the other hand, implementing the FAT format as is in Flash
>memories, makes them mostly WORM (Write Once, Read Multiple) devices,
>due to the fact that any simple file create/write/append involves
>several updates of the FAT (File Allocation Table) and/or Root Dir
>sectors.
The FAT library caches writes. This reduces wear on the the FAT.
>Its not hard to imagine that the sectors containing the FAT (File
>Allocation Table) will quickly consume its write/erase cycles, making
>the entire flash unusable.
Yes, FAT is stupid. I believe FAT32 has some
hooks for using alternate sectors for the FAT
table. IDE drives used to guarantee the first n
cylinders would always be good to get around the problem.
>Have any of you considered this issue ? Have any of you tested the
>way the ZW or other (SHDesigns, Progressive Resources and others) FAT
>implementations take the wear leveling issue into account ?
Yes, in my FAT (And ZW version), the first thing
I did was implement caching on the FAT to reduce
writes. In my implementation, the FAT is only
updated on a file close. ZW delays theirs also.
Just caching the FAT on a 100k file
create/write/close cut the rewrites by a factor
of about 100 (512-byte allocation size).
>In the light of the popularity of Flash memories and the amount of
>people starting to use this kind of devices for mass storage, I found
>this issue worth to discuss a little.
>
>Dont you agree ?
Yes, that I why I like MMC cards. They have
built-in wear-leveling and alternate sector
mapping in hardware. I would still want writes minimized.
ZW uses NAND flash on RCM33xx boards and needs a
lot of code (buggy code also) to handle recovery,
ECC and sector mapping. This is a mess and I
avoid it like the plague. Tons of complicated
code, very slow performance and apps never making
it to main() bbram is lost make it difficult to
use. Add that they used xD that can't be read on
a PC and it is completely useless IMHO.
I prefer to do it with the MMC hardware. Much
less code and recovery is better. Also, the
rabbit is the main limitation on speed. Moving
the problem to the MMC card's built-in fast NAND
controller makes it faster and much more
reliable. Still the rabbit can't use the card at
full speed with ZW FAT. :( The MMC is about 4x
the speed of the Atmel flash, but ZW FAT only
gives about 10% more speed. Seems a shame to have
a device that does 670kB/sec and only get
220kB/sec. But still a respectable speed.
For built-in atmel flash, I use my own
filesystem. There is no FAT on the device. It is
simlilar to the old CP/M filesystem except it has
64-char file names. The FAT is basically
distributed across the directory entries.
Directory/block allocation is circular.
That evens out writes. Add caching and it works well.
<Scott ------------
| Scott G. Henion | shenion@shen... |
| Consultant | Stone Mountain, GA |
| SHDesigns | PGP Key 0xE98DDC48 |
|Rabbit Libs: http://www.shdesigns.org/rabbit/ |
------------ --
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.10.24/101 - Release Date: 9/13/2005
>In the other hand, implementing the FAT format as is in Flash
>memories, makes them mostly WORM (Write Once, Read Multiple) devices,
>due to the fact that any simple file create/write/append involves
>several updates of the FAT (File Allocation Table) and/or Root Dir
>sectors.
The FAT library caches writes. This reduces wear on the the FAT.
>Its not hard to imagine that the sectors containing the FAT (File
>Allocation Table) will quickly consume its write/erase cycles, making
>the entire flash unusable.
Yes, FAT is stupid. I believe FAT32 has some
hooks for using alternate sectors for the FAT
table. IDE drives used to guarantee the first n
cylinders would always be good to get around the problem.
>Have any of you considered this issue ? Have any of you tested the
>way the ZW or other (SHDesigns, Progressive Resources and others) FAT
>implementations take the wear leveling issue into account ?
Yes, in my FAT (And ZW version), the first thing
I did was implement caching on the FAT to reduce
writes. In my implementation, the FAT is only
updated on a file close. ZW delays theirs also.
Just caching the FAT on a 100k file
create/write/close cut the rewrites by a factor
of about 100 (512-byte allocation size).
>In the light of the popularity of Flash memories and the amount of
>people starting to use this kind of devices for mass storage, I found
>this issue worth to discuss a little.
>
>Dont you agree ?
Yes, that I why I like MMC cards. They have
built-in wear-leveling and alternate sector
mapping in hardware. I would still want writes minimized.
ZW uses NAND flash on RCM33xx boards and needs a
lot of code (buggy code also) to handle recovery,
ECC and sector mapping. This is a mess and I
avoid it like the plague. Tons of complicated
code, very slow performance and apps never making
it to main() bbram is lost make it difficult to
use. Add that they used xD that can't be read on
a PC and it is completely useless IMHO.
I prefer to do it with the MMC hardware. Much
less code and recovery is better. Also, the
rabbit is the main limitation on speed. Moving
the problem to the MMC card's built-in fast NAND
controller makes it faster and much more
reliable. Still the rabbit can't use the card at
full speed with ZW FAT. :( The MMC is about 4x
the speed of the Atmel flash, but ZW FAT only
gives about 10% more speed. Seems a shame to have
a device that does 670kB/sec and only get
220kB/sec. But still a respectable speed.
For built-in atmel flash, I use my own
filesystem. There is no FAT on the device. It is
simlilar to the old CP/M filesystem except it has
64-char file names. The FAT is basically
distributed across the directory entries.
Directory/block allocation is circular.
That evens out writes. Add caching and it works well.
<Scott ------------
| Scott G. Henion | shenion@shen... |
| Consultant | Stone Mountain, GA |
| SHDesigns | PGP Key 0xE98DDC48 |
|Rabbit Libs: http://www.shdesigns.org/rabbit/ |
------------ --
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.10.24/101 - Release Date: 9/13/2005
Reply by ●September 19, 20052005-09-19
--- In rabbit-semi@rabb..., Scott Henion <shenion@s...> wrote:
> At 02:55 PM 9/16/2005, you wrote:
>
> I prefer to do it with the MMC hardware. Much
> less code and recovery is better...
I agree. But my concern is that MMC is in its final days... SD is
quickly replacing it, and may be hard to obtain MMC memories in the
near future. Are SD devices backward-compatible ? i.e Can they be
read as MMC ones ?
> rabbit is the main limitation on speed. Moving
> the problem to the MMC card's built-in fast NAND
> controller makes it faster and much more
> reliable. Still the rabbit can't use the card at
> full speed with ZW FAT. :( The MMC is about 4x
> the speed of the Atmel flash, but ZW FAT only
> gives about 10% more speed. Seems a shame to have
> a device that does 670kB/sec and only get
> 220kB/sec. But still a respectable speed.
Is this a SPI bus speed limit or a CPU speed limit ? Does it
improve on 44 Mhz devices ? > For built-in atmel flash, I use my own
> filesystem. There is no FAT on the device. It is
> simlilar to the old CP/M filesystem except it has
> 64-char file names. The FAT is basically
> distributed across the directory entries.
> Directory/block allocation is circular.
> That evens out writes. Add caching and it works well.
Is that "CP/M alike" library available for other people (i.e Me) ?
Im already one of your happy customers ;-) (for the Ethernet
downloader) Thanks for your comments Scott !!
Alberto Lopez
> At 02:55 PM 9/16/2005, you wrote:
>
> I prefer to do it with the MMC hardware. Much
> less code and recovery is better...
I agree. But my concern is that MMC is in its final days... SD is
quickly replacing it, and may be hard to obtain MMC memories in the
near future. Are SD devices backward-compatible ? i.e Can they be
read as MMC ones ?
> rabbit is the main limitation on speed. Moving
> the problem to the MMC card's built-in fast NAND
> controller makes it faster and much more
> reliable. Still the rabbit can't use the card at
> full speed with ZW FAT. :( The MMC is about 4x
> the speed of the Atmel flash, but ZW FAT only
> gives about 10% more speed. Seems a shame to have
> a device that does 670kB/sec and only get
> 220kB/sec. But still a respectable speed.
Is this a SPI bus speed limit or a CPU speed limit ? Does it
improve on 44 Mhz devices ? > For built-in atmel flash, I use my own
> filesystem. There is no FAT on the device. It is
> simlilar to the old CP/M filesystem except it has
> 64-char file names. The FAT is basically
> distributed across the directory entries.
> Directory/block allocation is circular.
> That evens out writes. Add caching and it works well.
Is that "CP/M alike" library available for other people (i.e Me) ?
Im already one of your happy customers ;-) (for the Ethernet
downloader) Thanks for your comments Scott !!
Alberto Lopez
Reply by ●September 19, 20052005-09-19
> I agree. But my concern is that MMC is in its final days... SD is
> quickly replacing it, and may be hard to obtain MMC memories in the
> near future. Are SD devices backward-compatible ? i.e Can they be read
> as MMC ones ?
>
Yes, SD is sort of MMC extension. Looking through the SPI
protocol, they are just the same.
> quickly replacing it, and may be hard to obtain MMC memories in the
> near future. Are SD devices backward-compatible ? i.e Can they be read
> as MMC ones ?
>
Yes, SD is sort of MMC extension. Looking through the SPI
protocol, they are just the same.
Reply by ●September 19, 20052005-09-19
--- In rabbit-semi@rabb..., "Jaroslav Nemec" <nemec@v...>
wrote:
> > I agree. But my concern is that MMC is in its final days... SD is
> > quickly replacing it, and may be hard to obtain MMC memories in the
> > near future. Are SD devices backward-compatible ? i.e Can they be
read
> > as MMC ones ?
> >
>
> Yes, SD is sort of MMC extension. Looking through the SPI
> protocol, they are just the same.
And, what about hardware/mechanical compatibility ? Do I have to allow
for an SD socket in my designs ? Or SD memories can be inserted into
MMC sockets ?
Thanks Jaroslav !
Alberto Lopez
> > I agree. But my concern is that MMC is in its final days... SD is
> > quickly replacing it, and may be hard to obtain MMC memories in the
> > near future. Are SD devices backward-compatible ? i.e Can they be
read
> > as MMC ones ?
> >
>
> Yes, SD is sort of MMC extension. Looking through the SPI
> protocol, they are just the same.
And, what about hardware/mechanical compatibility ? Do I have to allow
for an SD socket in my designs ? Or SD memories can be inserted into
MMC sockets ?
Thanks Jaroslav !
Alberto Lopez
Reply by ●September 19, 20052005-09-19
> And, what about hardware/mechanical compatibility ? Do I have to
allow
> for an SD socket in my designs ? Or SD memories can be inserted into
> MMC sockets ?
>
Well, MMC fits into SD connector. But not the other way around, i.e.
you cannot put a SD card into MMC connector.
Just design with the SD connector and you are just fine. Those
signals identical for both cards share same pins, of course. You can
see that most devices that use SD cards (cameras, mpeg players,
...) allows also using MMC in the same way.
Good luck!
J.
> for an SD socket in my designs ? Or SD memories can be inserted into
> MMC sockets ?
>
Well, MMC fits into SD connector. But not the other way around, i.e.
you cannot put a SD card into MMC connector.
Just design with the SD connector and you are just fine. Those
signals identical for both cards share same pins, of course. You can
see that most devices that use SD cards (cameras, mpeg players,
...) allows also using MMC in the same way.
Good luck!
J.
Reply by ●September 19, 20052005-09-19
At 03:31 PM 9/19/2005, you wrote:
>--- In rabbit-semi@rabb..., Scott Henion <shenion@s...> wrote:
> > At 02:55 PM 9/16/2005, you wrote:
> >
> > I prefer to do it with the MMC hardware. Much
> > less code and recovery is better...
>
> I agree. But my concern is that MMC is in its final days... SD is
>quickly replacing it, and may be hard to obtain MMC memories in the
>near future. Are SD devices backward-compatible ? i.e Can they be
>read as MMC ones ?
SD uses the same protocol and physical size. The
only difference is SD supports a write protect
switch. As long as you use a socket that supports both it should work.
I'll pick up a SD card, it should work. The
original lib said it worked with SD cards.
I don't know about "LLC in its final days". I see
far mare cards available. I would bet you will
see more devices support SD and are MMC-backward compatible.
> Is this a SPI bus speed limit or a CPU speed limit ? Does it
>improve on 44 Mhz devices ?
The CPU instruction time is the limit to speed. I
get 900kBytes/sec raw read speed (44mHz) with my
external SPI controller. The internal SPI is
limited to CPU/4 (11mHz). I can get it up
considerably if I hardcode timings in the libs.
But my libs are generic; I also need to support the <400khz speed during init.
For FAT, the ZW lib is the biggest limitation.
Using the 900kB/sec raw speed (driver is ASM and
the same for both DC and ST), I get 550kB/sec
reads with my ST FAT lib. Only 220kB/sec using ZW
lib. I could tweak the MMC lib to get about
1.4mB/sec, but it would hardly change the speed
due to the overhead of ZW FAT. The Atmel
DataFlash driver is about 1/4 the speed
physically, and it gets 200kB/sec reads. So a 4:1
speedup only gave a 10% change. So obviously the
FAT lib is the limiting factor. The FAT lib uses
XMEM buffers and the DC compiler is very slow accessing xmem.
> > For built-in atmel flash, I use my own
> > filesystem. There is no FAT on the device. It is
> > simlilar to the old CP/M filesystem except it has
> > 64-char file names. The FAT is basically
> > distributed across the directory entries.
> > Directory/block allocation is circular.
> > That evens out writes. Add caching and it works well.
>
> Is that "CP/M alike" library available for other people (i.e Me) ?
Sure, just as long as you use it with the
Softools Compiler. It is on my web site. I have
no plans to port it to DC. The lib uses far
pointers and xmem for about everything (uses very
little root memory.) Rewriting things like:
struct FILE far * file;
file->dir_entry.block_list[curblock]=bnum;
to:
unsigned long file;
xsetint(file+offsetof(FILE,direntry)+offsetof(DIR_ENTRY,blocklist)+(curblock*sizeof(unsiged),bnum);
would take too much work and the end result is a
real pain to work with. Then I'd have two very
different sources to maintain. Just porting the
MMC driver to DC took about 5X as long as It took to port it to ST.
>Im already one of your happy customers ;-) (for the Ethernet
>downloader)
Good to hear.
<Scott>
------------
| Scott G. Henion | shenion@shen... |
| Consultant | Stone Mountain, GA |
| SHDesigns | PGP Key 0xE98DDC48 |
|Rabbit Libs: http://www.shdesigns.org/rabbit/ |
------------ --
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.11.1/104 - Release Date: 9/16/2005
>--- In rabbit-semi@rabb..., Scott Henion <shenion@s...> wrote:
> > At 02:55 PM 9/16/2005, you wrote:
> >
> > I prefer to do it with the MMC hardware. Much
> > less code and recovery is better...
>
> I agree. But my concern is that MMC is in its final days... SD is
>quickly replacing it, and may be hard to obtain MMC memories in the
>near future. Are SD devices backward-compatible ? i.e Can they be
>read as MMC ones ?
SD uses the same protocol and physical size. The
only difference is SD supports a write protect
switch. As long as you use a socket that supports both it should work.
I'll pick up a SD card, it should work. The
original lib said it worked with SD cards.
I don't know about "LLC in its final days". I see
far mare cards available. I would bet you will
see more devices support SD and are MMC-backward compatible.
> Is this a SPI bus speed limit or a CPU speed limit ? Does it
>improve on 44 Mhz devices ?
The CPU instruction time is the limit to speed. I
get 900kBytes/sec raw read speed (44mHz) with my
external SPI controller. The internal SPI is
limited to CPU/4 (11mHz). I can get it up
considerably if I hardcode timings in the libs.
But my libs are generic; I also need to support the <400khz speed during init.
For FAT, the ZW lib is the biggest limitation.
Using the 900kB/sec raw speed (driver is ASM and
the same for both DC and ST), I get 550kB/sec
reads with my ST FAT lib. Only 220kB/sec using ZW
lib. I could tweak the MMC lib to get about
1.4mB/sec, but it would hardly change the speed
due to the overhead of ZW FAT. The Atmel
DataFlash driver is about 1/4 the speed
physically, and it gets 200kB/sec reads. So a 4:1
speedup only gave a 10% change. So obviously the
FAT lib is the limiting factor. The FAT lib uses
XMEM buffers and the DC compiler is very slow accessing xmem.
> > For built-in atmel flash, I use my own
> > filesystem. There is no FAT on the device. It is
> > simlilar to the old CP/M filesystem except it has
> > 64-char file names. The FAT is basically
> > distributed across the directory entries.
> > Directory/block allocation is circular.
> > That evens out writes. Add caching and it works well.
>
> Is that "CP/M alike" library available for other people (i.e Me) ?
Sure, just as long as you use it with the
Softools Compiler. It is on my web site. I have
no plans to port it to DC. The lib uses far
pointers and xmem for about everything (uses very
little root memory.) Rewriting things like:
struct FILE far * file;
file->dir_entry.block_list[curblock]=bnum;
to:
unsigned long file;
xsetint(file+offsetof(FILE,direntry)+offsetof(DIR_ENTRY,blocklist)+(curblock*sizeof(unsiged),bnum);
would take too much work and the end result is a
real pain to work with. Then I'd have two very
different sources to maintain. Just porting the
MMC driver to DC took about 5X as long as It took to port it to ST.
>Im already one of your happy customers ;-) (for the Ethernet
>downloader)
Good to hear.
<Scott>
------------
| Scott G. Henion | shenion@shen... |
| Consultant | Stone Mountain, GA |
| SHDesigns | PGP Key 0xE98DDC48 |
|Rabbit Libs: http://www.shdesigns.org/rabbit/ |
------------ --
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.11.1/104 - Release Date: 9/16/2005
Reply by ●September 20, 20052005-09-20
Hi Scott,
Thanks for the clarification on SD/MMC issues and for sharing the
results of your speed limit tests.
On the speed topic, I guess I have something to give back.
Browsing through SFLASH.LIB Ive seen that it uses basically 2
commands to read (plus other 2 to write) an Atmel serial flash. One
for transfering a page to a RAM bank(0x53/0x55) and the other to read
the RAM (through SPI)(0xD4/0xD6)
Atmel devices support another command (0xD2) to access any byte at
any adrress and perform a direct transfer of any size. That, at
first, reduces to half the number of sent SPI command bytes and saves
the page to RAM delay (700 uS in Atmel datasheet) It also simplifies
the 24-bit address calculation saving some CPU cycles too.
Based on this, I wrote an sf_readPageDirect() function that
reduces flash access time in about 35% for page-based reads (As in
FAT)
My current work needs to implement a binary search for an 8 byte
key in an ordered 50,000 records table. Doing a direct read of the
strictly required 8 bytes instead of a full 1024/1056 bytes block for
every binary search reduced the final search time for a factor of
10 !!! (98 mSeg against 10 mSeg !!!)
The function also supports direct root memory usage instead of the
mandatory xmem buffers used by sf_readDeviceRAM() in SFLASH.LIB
saving some extra CPU cycles.
Atmel flashes also support a direct flash write, but Im not sure
it would be usefull because it would only work on already erased
memory space.
Id be glad to share the source code with anyone interested on. > Libs for the Softools compiler ...
Not in my plans for the near future. I have a lot of code already
written for several designs and the ST compiler doesnt support all
the DC features Im using on them. Anyway, I think it would be worth
to take a look at them, are they free ?
Regards,
Alberto Lopez
Thanks for the clarification on SD/MMC issues and for sharing the
results of your speed limit tests.
On the speed topic, I guess I have something to give back.
Browsing through SFLASH.LIB Ive seen that it uses basically 2
commands to read (plus other 2 to write) an Atmel serial flash. One
for transfering a page to a RAM bank(0x53/0x55) and the other to read
the RAM (through SPI)(0xD4/0xD6)
Atmel devices support another command (0xD2) to access any byte at
any adrress and perform a direct transfer of any size. That, at
first, reduces to half the number of sent SPI command bytes and saves
the page to RAM delay (700 uS in Atmel datasheet) It also simplifies
the 24-bit address calculation saving some CPU cycles too.
Based on this, I wrote an sf_readPageDirect() function that
reduces flash access time in about 35% for page-based reads (As in
FAT)
My current work needs to implement a binary search for an 8 byte
key in an ordered 50,000 records table. Doing a direct read of the
strictly required 8 bytes instead of a full 1024/1056 bytes block for
every binary search reduced the final search time for a factor of
10 !!! (98 mSeg against 10 mSeg !!!)
The function also supports direct root memory usage instead of the
mandatory xmem buffers used by sf_readDeviceRAM() in SFLASH.LIB
saving some extra CPU cycles.
Atmel flashes also support a direct flash write, but Im not sure
it would be usefull because it would only work on already erased
memory space.
Id be glad to share the source code with anyone interested on. > Libs for the Softools compiler ...
Not in my plans for the near future. I have a lot of code already
written for several designs and the ST compiler doesnt support all
the DC features Im using on them. Anyway, I think it would be worth
to take a look at them, are they free ?
Regards,
Alberto Lopez
