SD Card FAT support issues (dosfs, fatfs,fatlib)

OK, firstly I should say that I'm aware there has been recent discussion of
embedded FAT here. I have looked at Lewin Edwards' DOSFS 1.01, FatFS R0.02a
and Sham's Fatlib v2.0.

I have an ARM7 based instrument that already is in production but would like
to a support for external memory cards to store measurement data and
configuration. The system only has 128K RAM and 4 MByte Flash internally
(plus 32K RAM in the CPU). Much of the RAM is used but can probably find a
few K, maybe 6K. There is already a file system in part of the Flash that
can store 'files' - about 2.5 MB, the rest of the flash is the code image
that runs directly. The file system is a simple one of my own design.

Now, it would be nice if could optionally use a SD card to store this data,
or at least be able to backup and restore from it. To do this though I would
need to map the SD Card FAT system to match my API or at least pull the two
toward each other. I suspect that I use too much functionality though. I can
open, read, write and seek in files. I can create, delete and rename files &
directories (including moving in the tree), also inumerate directories.
There is a current working directory. File names are up to 31 characters,
including spaces and upper and lower case - but not Unicode.

First problem. None of the three FAT modules above support long file names.
So, the first question is how easy might this be to add, or is there code
out there code available that can do this.

Next, Fatlib seems to be far to limited. The API allows opening files,
reading and writing. No enumeration, delete, rename etc. So I have discarded
this for now.

Next, DOSFS has much more - enumerate directories, seek, delete,
multi-volume, but no rename (though I suspect that should be to hard to add
at least for in situ). However, the first test I tried was to build the
emulation that uses a disk image but this produced an error. I created the
directory MYDIR1 required for the test on the real card before taking the
image. Before the test the directory has entries for . and .. but aftwards
it had entries for . and WRTEST.TXT - it had killed the .. directory!

Finally, I had a look at FatFS. The big problem here is that it does not
support a current working directory. All path names are treated as full. Nor
does it have rename, but perhaps I can live with that. I guess I could fix
the current directory issue at the textual level by a layer prefixing text
to the file names.

But still, none of these has long file name support.

Has anyone out there used any of these FAT modules in earnest? Has anyone
got a better solution or patches that give long file name support? If not
then I'll either try to add long file names to FatFS or find some
work-around in the instrument to avoid the need.

Many thanks,
Peter

Peter Dickerson wrote:

> First problem. None of the three FAT modules above support long file names.
> So, the first question is how easy might this be to add, or is there code
> out there code available that can do this.

There is a group that is contemplating adding LFNs to dosfs.

Observe that there are two known bugs in the current version of dosfs:

1. Error crossing cluster boundaries on FAT32 volumes.
2. Cannot open file in root directory. (All the applications where I
use the code work inside a subdirectory to avoid root entry limitations
on FAT12/FAT16 - that's how this bug escaped).

I have patches for both of these bugs but haven't finished testing them
yet.

> Finally, I had a look at FatFS. The big problem here is that it does not
> support a current working directory. All path names are treated as full. Nor

I don't know about their decisions, but this was an *explicitly* stated
design decision in dosfs. A "working directory" is an operating system
concept, because it belongs to a process. Observe that dosfs was
designed to be friendly to a multitasking operating system; it uses no
global variables, and a consequence of this is that the caller needs to
keep track of a "working directory", if such a concept exists.

"larwe" <zwsdotcom@gmail.com> wrote in message
news:1157454725.133973.206380@b28g2000cwb.googlegroups.com...
>
> Peter Dickerson wrote:
>
> > First problem. None of the three FAT modules above support long file
names.
> > So, the first question is how easy might this be to add, or is there
code
> > out there code available that can do this.
>
> There is a group that is contemplating adding LFNs to dosfs.

Timescales? Links? I've done the proof of concept stage for the device and
can talk SPI to the Card. So I'm looking to move on quickly.

> Observe that there are two known bugs in the current version of dosfs:
>
> 1. Error crossing cluster boundaries on FAT32 volumes.

I was using a FAT16 image 512 MB, 32 sectors/cluster. So I did think I was
bit by this.

> 2. Cannot open file in root directory. (All the applications where I
> use the code work inside a subdirectory to avoid root entry limitations
> on FAT12/FAT16 - that's how this bug escaped).

I don't think this was my issue either because the file was in a
subdirectory (MYDIR1) but there may have been files in the root at the time.
I will redo the tests with a clean file system.

> I have patches for both of these bugs but haven't finished testing them
> yet.

OK, great.

> > Finally, I had a look at FatFS. The big problem here is that it does not
> > support a current working directory. All path names are treated as full.
Nor
>
> I don't know about their decisions, but this was an *explicitly* stated
> design decision in dosfs. A "working directory" is an operating system
> concept, because it belongs to a process. Observe that dosfs was
> designed to be friendly to a multitasking operating system; it uses no
> global variables, and a consequence of this is that the caller needs to
> keep track of a "working directory", if such a concept exists.

Yes, I understand this. I think in a multitasking enviroment there is more
to be done than this though. A certain amount of synchrization is required.
What if two processes each create a file in the same directory? They need to
avoid private copies of directory sectors, fat sectors etc. Not that this is
a problem for me because I am single threaded. The reason I used a working
directory was because it was easy to implement if the FS code but worse in
the application. My (limited) understanding of FAT is that a current working
directory would amount to a cluster number for the start of the directory
data and so could be used to avoid scanning the directory tree every time.
If thats so then a cluster number could be used as a directory handle passed
in addition to the file path. In that case DFS_OpenFile(...) would take an
addition DIRHANDLE parameter (say), plus a new function to obtain such a
handle.

Thanks very much for your prompt help.

Peter

Peter Dickerson wrote:
> 
... snip ...
> 
> First problem. None of the three FAT modules above support long
> file names. So, the first question is how easy might this be to
> add, or is there code out there code available that can do this.

Nor will they.  There are patent problems involved.

-- 
"I'm the commander--see, I don't have to explain -- I don't need
 to explain why I say things. That's the interesting thing about
 being the President. Maybe somebody needs to explain to me why
 they say something, but I don't feel like I owe anybody an
 explanation." -                     George W. Bush, 2002-11-19

Peter Dickerson wrote:

> > There is a group that is contemplating adding LFNs to dosfs.
>
> Timescales? Links? I've done the proof of concept stage for the device and
> can talk SPI to the Card. So I'm looking to move on quickly.

I've no idea of timescales. The project is an open-source bootloader,
the name eludes me at the moment, but I will look it up when I get
home. I've had a few technical/bureaucratic email exchanges with two
people involved with the project.

I will not be offering LFN support in dosfs - I made it clear to this
other group that while they're welcome to do whatever they like with my
code, they will be forking permanently away from my code and I won't
help them maintain it, nor will I roll their changes into the dosfs I
am offering to the public.

LFNs are not going to become part of dosfs for the following reasons:

1. Not part of the design goals. dosfs is designed for minimal embedded
systems.

2. _AFAIK_ (and I could be wrong) dosfs is currently patent-pure.
Adding LFN support definitely infringes on a MS patent related to
storing multiple filenames in a single filesystem.

3. Not required for any application where I use it. dosfs is used as a
drop-in black box in several paid projects I still maintain. I'm not
making API changes that aren't necessary for those applications.

> > 1. Error crossing cluster boundaries on FAT32 volumes.
>
> I was using a FAT16 image 512 MB, 32 sectors/cluster. So I did think I was
> bit by this.

Hmm. There are no known bugs with FAT16 except the "can't open root
file" bug. If you can zip up your image file and send it to me, I might
be able to interpret what is happening.

> to be done than this though. A certain amount of synchrization is required.
> What if two processes each create a file in the same directory? They need to

Sure, there's additional work required in the OS layer. I'm just saying
that dosfs can't enforce working directories because it doesn't know
who owns it - anyone who calls dosfs is expected to provide, with every
single API call, a pointer to user-supplied data structure(s) that
define(s) _all_ information required to execute the API function.

> the application. My (limited) understanding of FAT is that a current working
> directory would amount to a cluster number for the start of the directory

Hmm. That's an interesting idea. Could also be implemented in the FS
layer - or just above it - using a MRU list of directories vs. start
cluster.

"CBFalconer" <cbfalconer@yahoo.com> wrote in message
news:44FD69D4.BD4E318D@yahoo.com...
> Peter Dickerson wrote:
> >
> ... snip ...
> >
> > First problem. None of the three FAT modules above support long
> > file names. So, the first question is how easy might this be to
> > add, or is there code out there code available that can do this.
>
> Nor will they.  There are patent problems involved.

I understand that there are patent problems although it I thought that this
was still an open question. If this is now resolved in MS's favour then I
won't go down that route.

So, what is the current status?

Peter

"larwe" <zwsdotcom@gmail.com> wrote in message
news:1157461492.145717.299800@i3g2000cwc.googlegroups.com...
>
> Peter Dickerson wrote:
>
> > > There is a group that is contemplating adding LFNs to dosfs.
> >
> > Timescales? Links? I've done the proof of concept stage for the device
and
> > can talk SPI to the Card. So I'm looking to move on quickly.
>
> I've no idea of timescales. The project is an open-source bootloader,
> the name eludes me at the moment, but I will look it up when I get
> home. I've had a few technical/bureaucratic email exchanges with two
> people involved with the project.

:-( I'll take that as too late for my project.

> I will not be offering LFN support in dosfs - I made it clear to this
> other group that while they're welcome to do whatever they like with my
> code, they will be forking permanently away from my code and I won't
> help them maintain it, nor will I roll their changes into the dosfs I
> am offering to the public.
>
> LFNs are not going to become part of dosfs for the following reasons:
>
> 1. Not part of the design goals. dosfs is designed for minimal embedded
> systems.
>
> 2. _AFAIK_ (and I could be wrong) dosfs is currently patent-pure.
> Adding LFN support definitely infringes on a MS patent related to
> storing multiple filenames in a single filesystem.
>
> 3. Not required for any application where I use it. dosfs is used as a
> drop-in black box in several paid projects I still maintain. I'm not
> making API changes that aren't necessary for those applications.

Fair enough. No need to entangle your project with patent disputes. If this
patent is confirmed then I won't use LFN either. I'll have to find some
other way such as force long names to short but not include the long name
records.

> > > 1. Error crossing cluster boundaries on FAT32 volumes.
> >
> > I was using a FAT16 image 512 MB, 32 sectors/cluster. So I did think I
was
> > bit by this.

that should be didn't.

> Hmm. There are no known bugs with FAT16 except the "can't open root
> file" bug. If you can zip up your image file and send it to me, I might
> be able to interpret what is happening.

I will make sure that I can reproduct the problem from clean, and if there
is still an issue I'll take you up on your offer to look.

> > to be done than this though. A certain amount of synchrization is
required.
> > What if two processes each create a file in the same directory? They
need to
>
> Sure, there's additional work required in the OS layer. I'm just saying
> that dosfs can't enforce working directories because it doesn't know
> who owns it - anyone who calls dosfs is expected to provide, with every
> single API call, a pointer to user-supplied data structure(s) that
> define(s) _all_ information required to execute the API function.

OK. So the problem is you don't know other processes' working directories
only this one's (if the working dir was in the data structure). I think that
problem is at a similar level to the problem of multiple processes modifying
the same sector. Global info is needed to co-ordinate modifications, such as
a locked sector table (or cache).

> > the application. My (limited) understanding of FAT is that a current
working
> > directory would amount to a cluster number for the start of the
directory
>
> Hmm. That's an interesting idea. Could also be implemented in the FS
> layer - or just above it - using a MRU list of directories vs. start
> cluster.

A directory start clucter cache that would help even without a working
directory. I do something like this in my Flash file system. In my case,
because I am single threaded I can do stuff like flush the cache on
directory modifications

Peter

Peter Dickerson wrote:

> > > First problem. None of the three FAT modules above support long
> > > file names. So, the first question is how easy might this be to
> >
> > Nor will they.  There are patent problems involved.
>
> I understand that there are patent problems although it I thought that this
> was still an open question. If this is now resolved in MS's favour then I
> won't go down that route.

I'm not aware that there has ever been any litigation about this. I'm
not even sure if MS actively enforce it. However it isn't a gray area:
the way LFNs are implemented on FAT volumes is patented - and hence I'm
not going to implement them (never mind that I have no use for them
either).

Have you noticed that almost no embedded devices that support FAT
support LFNs?

Support for FAT filesystems per se is supposedly covered by some MS
patents also, but the area is much murkier here, and I certainly feel
it's an acceptable risk to use FAT as a filesystem in a fielded
appliance.

The patents are clearly not rigorously enforced, but it's simply a
headache I don't want to deal with. If I had a compelling reason to use
LFNs, I might reconsider that - until such a time, I won't get my hands
dirty on the issue. (Supporting LFNs makes life very difficult in
constrained systems. Filenames no longer have a fixed length, and paths
can become obscenely long. Some extension-parsing code - not mine - may
break because the filename can legitimately contain periods other than
that separating the extension from the filename. The extension might
not be three letters long, and tables for identifying extensions have
to be enlarged. Etc, etc.).

"Peter Dickerson" <first{dot}surname@tesco.net> wrote in message
news:sAbLg.69$s4.18@newsfe3-win.ntli.net...
> OK, firstly I should say that I'm aware there has been recent discussion
of
> embedded FAT here. I have looked at Lewin Edwards' DOSFS 1.01, FatFS
R0.02a
> and Sham's Fatlib v2.0.
>
> I have an ARM7 based instrument that already is in production but would
like
> to a support for external memory cards to store measurement data and
> configuration. The system only has 128K RAM and 4 MByte Flash internally
> (plus 32K RAM in the CPU). Much of the RAM is used but can probably find a
> few K, maybe 6K. There is already a file system in part of the Flash that
> can store 'files' - about 2.5 MB, the rest of the flash is the code image
> that runs directly. The file system is a simple one of my own design.
>
> Now, it would be nice if could optionally use a SD card to store this
data,
> or at least be able to backup and restore from it. To do this though I
would
> need to map the SD Card FAT system to match my API or at least pull the
two
> toward each other. I suspect that I use too much functionality though. I
can
> open, read, write and seek in files. I can create, delete and rename files
&
> directories (including moving in the tree), also inumerate directories.
> There is a current working directory. File names are up to 31 characters,
> including spaces and upper and lower case - but not Unicode.
>
> First problem. None of the three FAT modules above support long file
names.
> So, the first question is how easy might this be to add, or is there code
> out there code available that can do this.
>
> Next, Fatlib seems to be far to limited. The API allows opening files,
> reading and writing. No enumeration, delete, rename etc. So I have
discarded
> this for now.
>
> Next, DOSFS has much more - enumerate directories, seek, delete,
> multi-volume, but no rename (though I suspect that should be to hard to
add
> at least for in situ). However, the first test I tried was to build the
> emulation that uses a disk image but this produced an error. I created the
> directory MYDIR1 required for the test on the real card before taking the
> image. Before the test the directory has entries for . and .. but aftwards
> it had entries for . and WRTEST.TXT - it had killed the .. directory!
>
> Finally, I had a look at FatFS. The big problem here is that it does not
> support a current working directory. All path names are treated as full.
Nor
> does it have rename, but perhaps I can live with that. I guess I could fix
> the current directory issue at the textual level by a layer prefixing text
> to the file names.
>
> But still, none of these has long file name support.
>
> Has anyone out there used any of these FAT modules in earnest? Has anyone
> got a better solution or patches that give long file name support? If not
> then I'll either try to add long file names to FatFS or find some
> work-around in the instrument to avoid the need.
>
> Many thanks,
> Peter
>
>

You may also want to take a look at EFSL, which is an open source file
system
library that has a port for ARM7.  It can be found
at: http://sourceforge.net/projects/efsl/  Make sure you use the stable
version.

Mike

larwe wrote:

>>First problem. None of the three FAT modules above support long file names.
>>So, the first question is how easy might this be to add, or is there code
>>out there code available that can do this.
> 
> 
> There is a group that is contemplating adding LFNs to dosfs.
> 
> Observe that there are two known bugs in the current version of dosfs:
> 
> 1. Error crossing cluster boundaries on FAT32 volumes.
> 2. Cannot open file in root directory. (All the applications where I
> use the code work inside a subdirectory to avoid root entry limitations
> on FAT12/FAT16 - that's how this bug escaped).
> 
> I have patches for both of these bugs but haven't finished testing them
> yet.

Back in February, I had sent you some patches; I don't think you may
have received them ;) They fix a few bugs and add mixed-case filename
support to dosfs.  I have put the diffs (sorry, they aren't unified)
and a text file at:

http://www.cybertheque.org/sw/ports/ATT_SysV/2.1/dosfs

The patched version should work on any svr4 compliant *nix o/s.

Regards,

Michael Grigoni
Cybertheque Museum