Flash File System for Dataflash

Hi all

I wrote a flash file system for an 8051 project using a 2MB dataflash over
the last 3 years. I tried to stay as close to stdio from dos (fopen, etc)
as possible. I had to shape and change it a few times as hind sight makes
us cleverer. I needed some special functionality like shrinking the file
from the front like a FIFO that caused me to veer of a bit (programmer’s
creative license).  It also has the ability to handle binary files. I
stored and retrieved bitmaps successfully.
The version I call final has been in the field for a while now and seem
stable. Here and there a piece of the fat disappears with disastrous
effect. I wrote recovery code but of course I never tested it, after stress
test I was happy. All in all over 3K pieces of hardware run it now so the 2
or3 problems I had don’t worry me much.
I would like to put this code somewhere where we can all use it and maybe
contribute to it. I’ll need some help with a repository and copyright
stuff. I would like it to be as free as possible and see if any fs experts
out there would like to comment or improve on it.
If you have experience with this kind of stuff and can help please do, if
you don’t, I would like to request that we keep the thread as clean as
possible, DONT CLUTTER IT WITH NONSENS! Let’s see how efficiently we can
get this out there and usable for everyone.

Regards

Hein de Kock	   
					
---------------------------------------		
This message was sent using the comp.arch.embedded web interface on
http://www.EmbeddedRelated.com

heindekock wrote:

> The version I call final has been in the field for a while now and seem
> stable. Here and there a piece of the fat disappears with disastrous
> effect. I wrote recovery code but of course I never tested it, after stress
> test I was happy. 

Why didn't you test it? Maybe it has even more disastrous effects when it
runs?

Can you say some details of your implementation? There are some nice
implementations which implements algorithm to distribute the write
acitivies over the whole flash to avoid erasing and writing one block too
many times. And there are sophisticated error recovering algorithms, so
that one block can fail completly without loss of data. See e.g. jffs2, the
flash file system for Linux.

> All in all over 3K pieces of hardware run it now so the 2
> or3 problems I had don’t worry me much.
> I would like to put this code somewhere where we can all use it and maybe
> contribute to it. I’ll need some help with a repository and copyright
> stuff. I would like it to be as free as possible and see if any fs experts
> out there would like to comment or improve on it.

You could use Sourceforge or Google code. Sourceforge was getting a bit
fat, with slow webpages, advertises etc., so one of my projects moved to
Google code ( http://code.google.com/p/lispbuilder/ ) . Google code has a
nice and clean interface and is fast. I guess they use their football field
sized computer centers for hosting it, in parallel to their search engine.

Regarding license: If you want that most people can use it, just use the
BSD license:

http://www.opensource.org/licenses/bsd-license.php

But it depends on your requirements. Some people think GPL is nice, but for
me it is too restrictive for most projects, because I can't use it in
closed source projects.

-- 
Frank Buss, fb@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de

heindekock wrote:
> Hi all
>=20
> I wrote a flash file system for an 8051 project using a 2MB dataflash o=
ver
> the last 3 years.

We developed the compatible FAT/FAT32 file system for any project using=20
any media which provides sector read/write functions. The flash specific =

functionality such as wear leveling and the replacement of the bad=20
blocks is the responsibility of the lower level subsystem.

> I tried to stay as close to stdio from dos (fopen, etc)
> as possible.

We provide POSIX/SUSV3 compliant interface. The multitask access and the =

file sharing is supported.

> I had to shape and change it a few times as hind sight makes
> us cleverer. I needed some special functionality like shrinking the fil=
e
> from the front like a FIFO that caused me to veer of a bit (programmer=F2=
=C0=D9s
> creative license).  It also has the ability to handle binary files. I
> stored and retrieved bitmaps successfully.
> The version I call final has been in the field for a while now and seem=

> stable. Here and there a piece of the fat disappears with disastrous
> effect. I wrote recovery code but of course I never tested it, after st=
ress
> test I was happy. All in all over 3K pieces of hardware run it now so t=
he 2
> or3 problems I had don=F2=C0=D9t worry me much.

It took us a while to iron out the bugs. We had to develop the special=20
test suite to verify the operation in the variety of possible scenarios.

> I would like to put this code somewhere where we can all use it and may=
be
> contribute to it. I=F2=C0=D9ll need some help with a repository and cop=
yright
> stuff. I would like it to be as free as possible and see if any fs expe=
rts
> out there would like to comment or improve on it.

I am the firm unbeliever in the mickey mouse stuff. Leaving aside the=20
copyright/opensource issues, everyone gets what he pays for.

> If you have experience with this kind of stuff and can help please do, =
if
> you don=F2=C0=D9t, I would like to request that we keep the thread as c=
lean as
> possible, DONT CLUTTER IT WITH NONSENS! Let=F2=C0=D9s see how efficient=
ly we can
> get this out there and usable for everyone.

There is a zillion of the short weight realizations of FAT for the small =

microcontrollers already. Why another one?


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

>We developed the compatible FAT/FAT32 file system for any project
using=20
>any media which provides sector read/write functions. The flash specific
=
>
>functionality such as wear leveling and the replacement of the bad=20
>blocks is the responsibility of the lower level subsystem.
>

Good for you - not everyone need FAT12/16/32 compatibility and wear
leveling on Flash goes without saying. The main reason for this design was
to let the FAT float around and grow wherever it finds open space. Rapidly
changing data can destroy a fixed fat quickly, I worn out Fat12/16 systems
and a DiskOnChip with fast changing data over a just a few months. The FFS
calls read write and erase of a page, oviously you can change the hardware
layer to implement that on any flash.

>We provide POSIX/SUSV3 compliant interface. The multitask access and the
=
>
>file sharing is supported.
>

Good for you again!

>It took us a while to iron out the bugs. We had to develop the
special=20
>test suite to verify the operation in the variety of possible scenarios.
>

Good!

>I am the firm unbeliever in the mickey mouse stuff. Leaving aside the=20
>copyright/opensource issues, everyone gets what he pays for.
>


>
>There is a zillion of the short weight realizations of FAT for the small
=
>
>microcontrollers already. Why another one?
>
>

I though I asked as nicely to not clutter the thread. I understand you
want to advertise your far superior product and skills, but maybe this is
not the place for it.
Now please if you have nothing constructive to add, please, please do not
comment. There may just be something useful here for someone, even if that
someone is not you.
	   
					
---------------------------------------		
This message was sent using the comp.arch.embedded web interface on
http://www.EmbeddedRelated.com

Vladimir Vassilevsky wrote:
> heindekock wrote:
>> I wrote a flash file system for an 8051 project using a 2MB dataflash
>> over the last 3 years.
> 
> We developed the compatible FAT/FAT32 file system for any project using
> any media which provides sector read/write functions. The flash specific
> functionality such as wear leveling and the replacement of the bad
> blocks is the responsibility of the lower level subsystem.

I haven't seen heindekock's code to relate it, but FAT on a block
translation layer looks to me like the worst of both worlds: the
inherent unreliability of FAT combined with the expensiveness of a block
translation layer. Hence my flash filesystem works on raw flash.

>> The version I call final has been in the field for a while now and seem
>> stable. Here and there a piece of the fat disappears with disastrous
>> effect. I wrote recovery code but of course I never tested it, after
>> stress test I was happy. All in all over 3K pieces of hardware run it
>> now so the 2 or3 problems I had don't worry me much.
> 
> It took us a while to iron out the bugs. We had to develop the special
> test suite to verify the operation in the variety of possible scenarios.

Same here. My current "run after major modifications before checkin"
test suite runs for half a day and needs about 100 GB temp space.

>> I would like to put this code somewhere where we can all use it and maybe
>> contribute to it. I'll need some help with a repository and copyright
>> stuff. I would like it to be as free as possible and see if any fs
>> experts out there would like to comment or improve on it.
> 
> I am the firm unbeliever in the mickey mouse stuff. Leaving aside the
> copyright/opensource issues, everyone gets what he pays for.

Did you finish your FAT implementation without taking a peep at Linux
source? And does it get upper/lowercase right on filesystems written by
Windows NT? Free source just cannot be useless.


  Stefan

Hi Stefan

Before this get totally out of hand, let me just give you a quick
rundown.

The basic Idea is not to tie a File Allocation table to a specific aria
where it will wear out the flash. Rather to consider the file object as a
piece of memory like you would in a linked list. This means you just have
to find the first (indexed) peace and then you can access all the files
header info’s sequentially.  But let’s stick the name FAT for clarity.

The System starts by initializing the Flash, determining how big the Flash
Page sizes are and then setting the appropriate offset addresses for wear
level info, File info and data.

Once the flash type is determined and initialized it will allocate
(malloc) ram for a “FAT” page, A “FATBAK” page and the amount of
open files you want, let’s say 4. Then it scans the flash from start page
(configurable) to end page, looking for a fat page and a fatback page. If
not found, it’s clearly not formatted so it will be formatted with
callbacks for the systems that has output and input. If found (the pages
are indexed), the start page can be trace from there without the need to
continue stepping through the flash.

Now that we have a fat start page and a backup start page we can load that
page into RAM, read out file headers happily and find the file start
addresses with an offset in the media.

To open a file with name “FILE”, the fat entries will be stepped
through until such a file is found, the Object will be loaded with start
page, file number etc. the handle (pointer to object) will be returned.

So with a 2MB flash that generally uses 512 to 528 byte pages you roughly
need 3K for the 2 fats and the 4 files you may want open at the same time.
The SPI code for hardware, the wear leveling code, sector disturb/refresh
code,  the file systems and a multitude common file routines for text file
manipulation all fit in 32K block on an 8051.

I have stored bitmaps and text files of up to 500K – ¼ of the specific
flash’s size without effort on this very small system. It is not intended
for the larger Linux or ARM environment where as you mentioned, there are
many available systems for.

Now I maintain my original point, this may be useful for some and it
certainly works on the 8 different applications I rolled it out on of which
there is about 3000 units in the field. If someone has a need to store and
retrieve data on flash memory, this may work for them and for that reason I
would like to put it out there, I am sure yours is much better and I’m
really not trying to compete with you or anyone for that matter. So if you
are not in favor of such, I take note, I apologize, hell ill even stand
corrected, but it may just be suitable for one of the millions of people
who were not going to buy you product anyway.

Please – this intended to help, not for financial gain of any sort and
I’m sure it will not affect your business in any way.

Still, I thank you for your input but I take exception to terms like
“Mickey mouse stuff” let part while things are still reasonably civil.	
  
					
---------------------------------------		
This message was sent using the comp.arch.embedded web interface on
http://www.EmbeddedRelated.com

Stefan Reuther wrote:

> Vladimir Vassilevsky wrote:
> 
>>heindekock wrote:
>>
>>>I wrote a flash file system for an 8051 project using a 2MB dataflash
>>>over the last 3 years.
>>
>>We developed the compatible FAT/FAT32 file system for any project using
>>any media which provides sector read/write functions. The flash specific
>>functionality such as wear leveling and the replacement of the bad
>>blocks is the responsibility of the lower level subsystem.
> 
> 
> I haven't seen heindekock's code to relate it, but FAT on a block
> translation layer looks to me like the worst of both worlds: the
> inherent unreliability of FAT combined with the expensiveness of a block
> translation layer. Hence my flash filesystem works on raw flash.

Yes, however:

1) FAT format is recognized by all systems.
2) FAT is suitable for any media, removable or fixed.
3) Working with the raw flash ties many things to the parameters of this 
particular flash. It is good to have the filesystem abstracted from media.


>>I am the firm unbeliever in the mickey mouse stuff. Leaving aside the
>>copyright/opensource issues, everyone gets what he pays for. 
> 
> Did you finish your FAT implementation without taking a peep at Linux
> source? 

I glanced over JFFS and YAFFS.

> And does it get upper/lowercase right on filesystems written by
> Windows NT? Free source just cannot be useless.

FAT was written to MS specification; I also looked what Windows actually 
does.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

>The version I call final has been in the field for a while now and seem
>stable. Here and there a piece of the fat disappears with disastrous
>effect. I wrote recovery code but of course I never tested it, after stres=
s
>test I was happy. All in all over 3K pieces of hardware run it now so the =
2
>or3 problems I had don=92t worry me much.

hahaha, DO NOT WANT !

On Sep 8, 10:21=A0am, bigbrownbeastie
<bigbrownbeastiebigbrownf...@googlemail.com> wrote:
> >The version I call final has been in the field for a while now and seem
> >stable. Here and there a piece of the fat disappears with disastrous
> >effect. I wrote recovery code but of course I never tested it, after str=
ess
> >test I was happy. All in all over 3K pieces of hardware run it now so th=
e 2
> >or3 problems I had don=92t worry me much.
>
> hahaha, DO NOT WANT !

0.1% chance of failure, is that suitable for IEC61508 SIL -3 ?

Vladimir Vassilevsky wrote:
> Stefan Reuther wrote:
>> Vladimir Vassilevsky wrote:
>>> We developed the compatible FAT/FAT32 file system for any project using
>>> any media which provides sector read/write functions. The flash specific
>>> functionality such as wear leveling and the replacement of the bad
>>> blocks is the responsibility of the lower level subsystem.
>>
>> I haven't seen heindekock's code to relate it, but FAT on a block
>> translation layer looks to me like the worst of both worlds: the
>> inherent unreliability of FAT combined with the expensiveness of a block
>> translation layer. Hence my flash filesystem works on raw flash.
>
> Yes, however:
> 
> 1) FAT format is recognized by all systems.

This is irrelevant if you talk to flash. At least, I rarely solder out
the flash from my device and put it into the PC's floppy drive.

> 2) FAT is suitable for any media, removable or fixed.

Or "unsuitable" due to its unreliability and inefficiency. It doesn't do
journaling, so if you give it into the hands of consumers, they can and
will break it.

> 3) Working with the raw flash ties many things to the parameters of this
> particular flash. It is good to have the filesystem abstracted from media.

The same goes for FAT. My flash file system is tied to the flash's block
size in much the same way as FAT's or UDF's block size is tied to the
underlying device's block size. Nothing more. You can even increase the
capacity on the fly by adding more flash.

>> And does it get upper/lowercase right on filesystems written by
>> Windows NT? Free source just cannot be useless.
> 
> FAT was written to MS specification; I also looked what Windows actually
> does.

MS specifications do not tell you how Windows NT stores upper/lower case
in 8.3 names. Probably that's why all our direct competitors get it
wrong :-)


  Stefan