>> Function pointers cause a bit of an issue in overlay linker/locators. The
>> linker attempts to overlay code and memory because there is no MMU or heap.

> Function pointers addresses are calculated at compile time and stored
> in flash.

Calculation and storage are not the issue.  _Calling_ functions via 
those pointers is.

The problem is that some architectures (most notably the 8051) just 
don't have enough stack space (nor pointer registers) to afford putting 
anything besides the return address in there.  It forbids itself to put 
automatic variables or function arguments on these systems' stacks.  So 
C compilers for such architectures allocate even automatic variables 
statically.  To reduce the impact of that on RAM consumption, they 
analyze the call tree in order to find out which functions' automatic 
data are never needed simultaneously, so they can statically be 
allocated to the same location.  That optimization is called data 
overlaying.

Function pointers make that analysis hard --- potentially impossible, even.

> I use them on HCS08 microcontrollers and never had a problem.

Consider yourself having been spared a worthwile, educating experience.

> If you statically link the architecture dependent code, you force a
> potential user to modify your code.

We're talking embedded systems here (and potentially small ones at that) 
--- users don't get to decide whether they want to modify the code or 
not.  More often than not, they won't even know that there is code 
running in that thing they just bought.

Hi,

> Function pointers cause a bit of an issue in overlay linker/locators. The
> linker attempts to overlay code and memory because there is no MMU or hea=
p.
> With function pointer it can=92t trace the call tree so it can=92t do any
> overlay. I learnt this the hard way, had to go back in redo allot of fanc=
y
> code with normal functions.

forgive pal, but this is the very first time I hear that.

Function pointers addresses are calculated at compile time and stored
in flash.

I use them on HCS08 microcontrollers and never had a problem.

If you statically link the architecture dependent code, you force a
potential user to modify your code.

> The device I discovered a problem on also uses UIP as TCPIP stack and PPP
> for a GPRS connection and the modem is quite busy. I saw the modem/serial
> code go haywire and over running a buffer in a bad comms scenario. I thin=
k
> this is what damaged the FAT.

Ok.

You might want to have PC-lint check your code for buffers overrun.

> The version I put on Google is just a FS, I will put it on a JTAG debugge=
r
> and put break points in all the error and recovery code arias, then run t=
he
> test routine in an endless loop for a few days to see what it takes to we=
ar
> out the flash and how the code reacts when the flash starts getting
> unreliable.

Ok

Enrico Migliore

>Hi,
>
>> In the error I experienced, it looked liked some other process messed
up
>> my FAT buffer.
>
>there might be an error in the code...
>
>Compile your code with Visual C after activating the LEVEL 4 option
>which is the most severe (the Visual C Express edition is now free
>from Microsoft).
>Make sure the compiler issues absolutely no warnings.
>
>Use function pointers to call all the functions that are architecture
>dependent.
>Don't be scared by function pointers: even 8 bit microcontrollers with
>1k flash can handle them correctly.
>In this way, your C code will compile cleanly on ANY architecture.
>
>See the following example: the client of your code, will have to make
>2 function calls, before using the filesystem:
>
>#include <fs.h>
>
>void main (void)
>{
>   fs_config_t *conf;
>
>   fs_init ();
>
>   conf->disk_sector_read = dataflash_read;
>   conf->disk_sector_write = dataflash_write;
>
>   fs_config (&conf);
>
>}
>
>
>> I was quite impressed with the recovery routine that picked
>> up all the loose pages and reconstructed the FAT.
>
>Nice feature.
>
>Good luck pal :-)
>
>ciao,
>Enrico Migliore
>

Function pointers cause a bit of an issue in overlay linker/locators. The
linker attempts to overlay code and memory because there is no MMU or heap.
With function pointer it can&rsquo;t trace the call tree so it can&rsquo;t do any
overlay. I learnt this the hard way, had to go back in redo allot of fancy
code with normal functions.

The device I discovered a problem on also uses UIP as TCPIP stack and PPP
for a GPRS connection and the modem is quite busy. I saw the modem/serial
code go haywire and over running a buffer in a bad comms scenario. I think
this is what damaged the FAT.

The version I put on Google is just a FS, I will put it on a JTAG debugger
and put break points in all the error and recovery code arias, then run the
test routine in an endless loop for a few days to see what it takes to wear
out the flash and how the code reacts when the flash starts getting
unreliable.

Could be fun!

Cheers

Hein de Kock	   
					
---------------------------------------		
This message was sent using the comp.arch.embedded web interface on
http://www.EmbeddedRelated.com

Hi,

> In the error I experienced, it looked liked some other process messed up
> my FAT buffer.

there might be an error in the code...

Compile your code with Visual C after activating the LEVEL 4 option
which is the most severe (the Visual C Express edition is now free
from Microsoft).
Make sure the compiler issues absolutely no warnings.

Use function pointers to call all the functions that are architecture
dependent.
Don't be scared by function pointers: even 8 bit microcontrollers with
1k flash can handle them correctly.
In this way, your C code will compile cleanly on ANY architecture.

See the following example: the client of your code, will have to make
2 function calls, before using the filesystem:

#include <fs.h>

void main (void)
{
   fs_config_t *conf;

   fs_init ();

   conf->disk_sector_read = dataflash_read;
   conf->disk_sector_write = dataflash_write;

   fs_config (&conf);

}


> I was quite impressed with the recovery routine that picked
> up all the loose pages and reconstructed the FAT.

Nice feature.

Good luck pal :-)

ciao,
Enrico Migliore

On Sep 9, 3:02 am, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
> karthikbalaguru wrote:
> > Yes, wear levelling & power failure scenarios are not well handled
> > with FAT16 and it is well handled in the JFS. Further, there is no
> > much necessity to create files in runtime in embedded.
>
> Huh?
>
> If there is no need to create files at runtime, then there is no need
> for the file system. You can include the data as the constant arrays.
>
> BTW, there is a whole big area of the embedded systems for data
> acquisition, monitoring and logging. All they do is creating and writing
> files.
>
In most of the scenarios constant arrays are preferred in embedded.
But Yes, if you go in for lot of logging then it is required.

Karthik Balaguru

karthikbalaguru wrote:

> Yes, wear levelling & power failure scenarios are not well handled
> with FAT16 and it is well handled in the JFS. Further, there is no
> much necessity to create files in runtime in embedded.

Huh?

If there is no need to create files at runtime, then there is no need 
for the file system. You can include the data as the constant arrays.

BTW, there is a whole big area of the embedded systems for data 
acquisition, monitoring and logging. All they do is creating and writing 
files.

Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

On Sep 8, 11:50=A0pm, Enrico <zig...@libero.it> wrote:
> Hi,
>
> > I wrote a flash file system for an 8051 project using a 2MB dataflash o=
ver
> > the last 3 years.
>
> Ok
>
> > I tried to stay as close to stdio from dos (fopen, etc)
> > as possible.
>
> fopen() and his brothers belong to the C standard I/O library
> (<stdio.h>) and not to the filesystem.
>
> Filesystems export only: open (), close (), read (), write (), lseek
> (), et cetera
>
> Filesystems compile and run perfectly without the standard library
> code.
>
>
>
>
>
> >I had to shape and change it a few times as hind sight makes
> > us cleverer. I needed some special functionality like shrinking the fil=
e
> > from the front like a FIFO that caused me to veer of a bit (programmer=
=92s
> > creative license). =A0It also has the ability to handle binary files. I
> > stored and retrieved bitmaps successfully.
> > The version I call final has been in the field for a while now and seem
> > stable. Here and there a piece of the fat disappears with disastrous
> > effect. I wrote recovery code but of course I never tested it, after st=
ress
> > test I was happy. All in all over 3K pieces of hardware run it now so t=
he 2
> > or3 problems I had don=92t worry me much.
> > I would like to put this code somewhere where we can all use it and may=
be
> > contribute to it. I=92ll need some help with a repository and copyright
> > stuff. I would like it to be as free as possible and see if any fs expe=
rts
> > out there would like to comment or improve on it.
> > If you have experience with this kind of stuff and can help please do, =
if
> > you don=92t, I would like to request that we keep the thread as clean a=
s
> > possible, DONT CLUTTER IT WITH NONSENS! Let=92s see how efficiently we =
can
> > get this out there and usable for everyone.
>
> Years ago, I extracted the FAT16 filesystem from Pat Villani's FreeDOS
> project and made it architecture independent.
> The only function architecture dependent were:
>
> media_init()
> media_read () =A0/* read a sector */
> media_write () =A0/* write a sector */
>
> I also extracted the MINIX standard I/O library and compiled it over
> the FAT16 filesystem.
>
> The code was debugged on Visual C 6.0.
> As a media, I used a usb disk plugged in the PC.
>
> The code is in production on hundreds of embedded devices (M16C+
> dataflash). All works nice. Never had problems.
>
> It was a very important C exercise... That I won't ever repeat again
> in my life.
>
> FAT filesystems are risky on embedded systems for the following
> reasons:
>
> 1) Unless you purchase a commercial product, the second copy of the
> FAT is never used to back up if the first fails
> 2) FAT16 don't do WEAR LEVELLING, that means that after 10.000/50.000
> of write sessions your Dataflash may fail
> 3) Speaking of embedded systems: in many many cases, there's
> absolutely no need to create new files at run time.
> 4) Power fail failure during write sessions may corrupt FAT and files
>
> Back then I was very excited running the FreeDOS filesystem and the
> MINIX stdio library on a 16 bit M16C microcontroller...
> Production volumes now are too big and I don't wanna risk...I save my
> data at fixed flash addresses, use 32 CRC, and keep 2 copies of the
> data.
>
> Suggestions for you :-)
>
> Dataflash Write
> ------------------
> 1) A write must be followed by a read, to check if you wrote the
> sector correctly
> 2) Poll the status register of the dataflash, and wait for the write
> to be completed. Don't assume the
> 3) Use the extra 8 bytes of each sector to store the CRC (16 bit is
> fine)
> 4) You may want to use ECC code correction. I remember correctly, 8
> bytes may correct up to 8 bit on a buffer of 256.
> 5) Use wear levelling if the application is supposed to write
> frequently onto the Dataflash.
> 6) Don't go 20 MHz... Slow down to 4 or 8 MHz
>
> Dataflash Read
> ------------------
> 1) Each time you read a sector onto the Dataflash, do it two times on
> 2 different buffers and then compare their content.
>
> The best filesystem for embedded sytems is a journaling fileystem.
> Each time you make a write, you add a record to the log.
> In case of Power Failure, you lose only the very last write and the
> filesystem is safe.
>
> Hope this'll help
> Enrico Migliore

Yes, wear levelling & power failure scenarios are not well handled
with FAT16 and it is well handled in the JFS. Further, there is no
much
necessity to create files in runtime in embedded. Agreed w.r.t.
embedded systems .

Karthik Balaguru

>
>FAT filesystems are risky on embedded systems for the following
>reasons:
>
>1) Unless you purchase a commercial product, the second copy of the
>FAT is never used to back up if the first fails
>2) FAT16 don't do WEAR LEVELLING, that means that after 10.000/50.000
>of write sessions your Dataflash may fail
>3) Speaking of embedded systems: in many many cases, there's
>absolutely no need to create new files at run time.
>4) Power fail failure during write sessions may corrupt FAT and files
>
>Back then I was very excited running the FreeDOS filesystem and the
>MINIX stdio library on a 16 bit M16C microcontroller...
>Production volumes now are too big and I don't wanna risk...I save my
>data at fixed flash addresses, use 32 CRC, and keep 2 copies of the
>data.
>
>
>Suggestions for you :-)
>
>
>Dataflash Write
>------------------
>1) A write must be followed by a read, to check if you wrote the
>sector correctly
>2) Poll the status register of the dataflash, and wait for the write
>to be completed. Don't assume the
>3) Use the extra 8 bytes of each sector to store the CRC (16 bit is
>fine)
>4) You may want to use ECC code correction. I remember correctly, 8
>bytes may correct up to 8 bit on a buffer of 256.
>5) Use wear levelling if the application is supposed to write
>frequently onto the Dataflash.
>6) Don't go 20 MHz... Slow down to 4 or 8 MHz
>
>Dataflash Read
>------------------
>1) Each time you read a sector onto the Dataflash, do it two times on
>2 different buffers and then compare their content.
>
>The best filesystem for embedded sytems is a journaling fileystem.
>Each time you make a write, you add a record to the log.
>In case of Power Failure, you lose only the very last write and the
>filesystem is safe.
>
>Hope this'll help
>Enrico Migliore
>

Thank you for the input

The code is now available at
http://code.google.com/p/small-ffs/downloads/list

This little system changed a lot from its inception. Speed was of course
an issue, especially since it originally rolled out on a standard 8051 12
clock cycle CPU with a 36 Meg crystal. This only runs at 3 MIPS so the data
flash speed was not reached as I did not have a spi integrated in the 8051,
all IOs clocked from code making effective speed very slow. The above code
was ported onto the DK3300 dev kit from ST so its slightly better but still
not fast and I didn&rsquo;t bother to use spi port again.

What you mentioned is mostly done with the exception of a CRC check it
just became to taxing on the system. I used the extra bytes but just to get
a few more bytes in a page, it bit my bad when one production run came with
the D type flash in binary address mode. This is an irreversible setting so
I had to quickly build in a check for that setting, luckily the ram buffers
are allocated dynamically after the page size is determined to handle a
variety of data flash sizes (I mostly use 16Mbit/2Mbyte with 512/528 bytes
per page).

The application that first ran the system had about 40 files, hardly ever
bigger than 20K and exclusively text, I mixed the use of fixed location
memory with normal fat like structure. A few setup type files never
changed, and the rest constantly changed. In the traditional fat like
scheme every time I wanted to remove data from the front of the file, I
would open the file read out the spent data, write the remaining data to a
new file, remove original and rename the new to the old. Slow, risky and a
huge source of wear. It irritated me so much that I designed this system
with the ability to just shift the file start page/position in the fat so
the file can shrink from the front without rewriting all the content. I
also tagged the pages with file number, prev page, next page and an index.
This allows an integrity check on the chain of pages which I do at startup
or at user request to look for any lost/unlinked pages.

In the demo above I write 100 files of 10K bytes (1Meg total used), then
rewrite them, then append another 5K (1.5Meg total used) then clip, as I
call it, 1k from the front (1.4K total used) then rename them all and
lastly delete them all.

I&rsquo;m comfortable with the current level of performance for my needs.
However, the very rapidly changing data like usage meters I store raw in
the flash for both speed and wear purposes.

In the error I experienced, it looked liked some other process messed up
my FAT buffer. I was quite impressed with the recovery routine that picked
up all the loose pages and reconstructed the FAT. The fat backup becomes
the fat in this case and any loose pages are removed. But to test that
properly I will have to step into that section of code and disrupt the fat
update process to simulate a failure. I just haven&rsquo;t felt compelled to do
that yet. Hopefully I get some energetic contributors that feel like
digging into that part of the code or maybe simulate the flash with another
small micro or even a PC, then we can drop chars and pages as we please to
see how robust the code is.

thanks again for the input.	   
					
---------------------------------------		
This message was sent using the comp.arch.embedded web interface on
http://www.EmbeddedRelated.com

Hi,

> I wrote a flash file system for an 8051 project using a 2MB dataflash ove=
r
> the last 3 years.

Ok

> I tried to stay as close to stdio from dos (fopen, etc)
> as possible.

fopen() and his brothers belong to the C standard I/O library
(<stdio.h>) and not to the filesystem.

Filesystems export only: open (), close (), read (), write (), lseek
(), et cetera

Filesystems compile and run perfectly without the standard library
code.


>I had to shape and change it a few times as hind sight makes
> us cleverer. I needed some special functionality like shrinking the file
> from the front like a FIFO that caused me to veer of a bit (programmer=92=
s
> creative license). =A0It also has the ability to handle binary files. I
> stored and retrieved bitmaps successfully.
> The version I call final has been in the field for a while now and seem
> stable. Here and there a piece of the fat disappears with disastrous
> effect. I wrote recovery code but of course I never tested it, after stre=
ss
> test I was happy. All in all over 3K pieces of hardware run it now so the=
 2
> or3 problems I had don=92t worry me much.
> I would like to put this code somewhere where we can all use it and maybe
> contribute to it. I=92ll need some help with a repository and copyright
> stuff. I would like it to be as free as possible and see if any fs expert=
s
> out there would like to comment or improve on it.
> If you have experience with this kind of stuff and can help please do, if
> you don=92t, I would like to request that we keep the thread as clean as
> possible, DONT CLUTTER IT WITH NONSENS! Let=92s see how efficiently we ca=
n
> get this out there and usable for everyone.

Years ago, I extracted the FAT16 filesystem from Pat Villani's FreeDOS
project and made it architecture independent.
The only function architecture dependent were:

media_init()
media_read ()  /* read a sector */
media_write ()  /* write a sector */

I also extracted the MINIX standard I/O library and compiled it over
the FAT16 filesystem.

The code was debugged on Visual C 6.0.
As a media, I used a usb disk plugged in the PC.

The code is in production on hundreds of embedded devices (M16C+
dataflash). All works nice. Never had problems.

It was a very important C exercise... That I won't ever repeat again
in my life.

FAT filesystems are risky on embedded systems for the following
reasons:

1) Unless you purchase a commercial product, the second copy of the
FAT is never used to back up if the first fails
2) FAT16 don't do WEAR LEVELLING, that means that after 10.000/50.000
of write sessions your Dataflash may fail
3) Speaking of embedded systems: in many many cases, there's
absolutely no need to create new files at run time.
4) Power fail failure during write sessions may corrupt FAT and files

Back then I was very excited running the FreeDOS filesystem and the
MINIX stdio library on a 16 bit M16C microcontroller...
Production volumes now are too big and I don't wanna risk...I save my
data at fixed flash addresses, use 32 CRC, and keep 2 copies of the
data.


Suggestions for you :-)


Dataflash Write
------------------
1) A write must be followed by a read, to check if you wrote the
sector correctly
2) Poll the status register of the dataflash, and wait for the write
to be completed. Don't assume the
3) Use the extra 8 bytes of each sector to store the CRC (16 bit is
fine)
4) You may want to use ECC code correction. I remember correctly, 8
bytes may correct up to 8 bit on a buffer of 256.
5) Use wear levelling if the application is supposed to write
frequently onto the Dataflash.
6) Don't go 20 MHz... Slow down to 4 or 8 MHz

Dataflash Read
------------------
1) Each time you read a sector onto the Dataflash, do it two times on
2 different buffers and then compare their content.

The best filesystem for embedded sytems is a journaling fileystem.
Each time you make a write, you add a record to the log.
In case of Power Failure, you lose only the very last write and the
filesystem is safe.

Hope this'll help
Enrico Migliore