Compact flash transfer rate (w/microcontroller)

Hi all,
in a previous thread we discussed about MMC timings and write speed.
These memory do not seem to guarantee a minimum sustained write bandwidth.

I was thinking of migrating to CompactFlash (more pins, but seem 
faster). Producers claim a very high sustained write speed (>1MB/sec), 
but I'm not so confident to obtain such values.

Can someone, according to its experience, comment on the write 
performance that can be obtained when a microcontroller controls a 
compact flash card for data logging?
(i.e. how much time do I have to wait for the flash to complete the 
sector write after I send all the 512 bytes?)

In my application I need to store about 120KB/sec, so if the whole 
operations lasts some ms, I should be just fine!


As usual any comment, suggestion or discussion is welcome!

Thank you!Aldo

> in a previous thread we discussed about MMC timings and write speed.
> These memory do not seem to guarantee a minimum sustained write
> bandwidth.
>
> I was thinking of migrating to CompactFlash (more pins, but seem
> faster). Producers claim a very high sustained write speed (>1MB/sec),
> but I'm not so confident to obtain such values.
>
> Can someone, according to its experience, comment on the write
> performance that can be obtained when a microcontroller controls a
> compact flash card for data logging?
> (i.e. how much time do I have to wait for the flash to complete the
> sector write after I send all the 512 bytes?)
>
> In my application I need to store about 120KB/sec, so if the whole
> operations lasts some ms, I should be just fine!

I have written an image capture system that writes to compact flash. The
system uses FAT formatted CF cards. I found that the write speed is partly
determined by the compact flash card itself: on one card I wrote at
approximately 120 kB/s whereas another card was 30kB/s. You can get CF cards
with different speeds: 4x, 8x, etc. This write time was on a fairly slow
processor.

    Andrew

So, do you think that it would be enough to buy a fast compact-flash? (i 
heard about 66x!)
Which brand/model/size allowed you to achieve the 120KB/sec?

Tnx 4 your answer!
Aldo


Andrew Jackson wrote:
>>in a previous thread we discussed about MMC timings and write speed.
>>These memory do not seem to guarantee a minimum sustained write
>>bandwidth.
>>
>>I was thinking of migrating to CompactFlash (more pins, but seem
>>faster). Producers claim a very high sustained write speed (>1MB/sec),
>>but I'm not so confident to obtain such values.
>>
>>Can someone, according to its experience, comment on the write
>>performance that can be obtained when a microcontroller controls a
>>compact flash card for data logging?
>>(i.e. how much time do I have to wait for the flash to complete the
>>sector write after I send all the 512 bytes?)
>>
>>In my application I need to store about 120KB/sec, so if the whole
>>operations lasts some ms, I should be just fine!
> 
> 
> I have written an image capture system that writes to compact flash. The
> system uses FAT formatted CF cards. I found that the write speed is partly
> determined by the compact flash card itself: on one card I wrote at
> approximately 120 kB/s whereas another card was 30kB/s. You can get CF cards
> with different speeds: 4x, 8x, etc. This write time was on a fairly slow
> processor.
> 
>     Andrew
> 
> 

> So, do you think that it would be enough to buy a fast compact-flash?
> (i heard about 66x!)
> Which brand/model/size allowed you to achieve the 120KB/sec?
>

The "fast" CF card was made by Sandisk. You can improve the speed by writing
multiple sectors and (perhaps) not updating the FAT every time.

    Andrew

How many X's did it have? I mean 8x,12x,66x?

Yes, your hints could improve performance, I'm quite confident about this.

But how do you detect the end of write of a sector?
Do you just wait an appropriate number of msecs or poll the RDY/-BSY signal?




Andrew Jackson wrote:
>>So, do you think that it would be enough to buy a fast compact-flash?
>>(i heard about 66x!)
>>Which brand/model/size allowed you to achieve the 120KB/sec?
>>
> 
> 
> The "fast" CF card was made by Sandisk. You can improve the speed by writing
> multiple sectors and (perhaps) not updating the FAT every time.
> 
>     Andrew
> 
> 

On Mon, 27 Jun 2005 14:45:04 +0200, Haldo <haldo@somewhere.net> wrote:
>In my application I need to store about 120KB/sec, so if the whole 
>operations lasts some ms, I should be just fine!
>
>
>As usual any comment, suggestion or discussion is welcome!
>
>Thank you!Aldo


Aldo,

With those CF cards that you can buy these days 120KB/sec ist
absolutely no problem - not even with a slow 8-bitter.

I.e. on a Rabbit 2000 I achived ~450KB/sec "netto" write speed towards
a FAT 16 filesystem.

The key issue is the sector IO routine. Here it's well worth the
effort to use assembly if your controller is slow. I.e. formentioned
R2000 only did ~100KB/sec useing 'C' code and the Zworld compiler.

HTH

Markus

> I.e. on a Rabbit 2000 I achived ~450KB/sec "netto" write speed towards
> a FAT 16 filesystem.

Wow.. that's great! Considering that you also update the fat, the 
"gross" (raw sectors) write rate is probably even higher!
Did you perform  "single" sector writes or "multiple"?

> 
> The key issue is the sector IO routine. Here it's well worth the
> effort to use assembly if your controller is slow. I.e. formentioned
> R2000 only did ~100KB/sec useing 'C' code and the Zworld compiler.
> 

Do you think that assembly or C could make the difference?
And what about the impact of:
- memory buffers
- single/multiple sectors write
- mode of CF: ATA mode, Memory Mode, etc.


Did you use some publicly available code, or did you just write your own 
routines?


Really helpful! Ciao,
Aldo

Haldo wrote:
> So, do you think that it would be enough to buy a fast compact-flash? (i
> heard about 66x!)

Lexar makes a 80x CF (~1.2MB/s).  However, the CF interface probably
peak out at 1.5MB/s.  We can read 1.5MB/s from IDE-CF on an AMD 2500
processor.

>> I.e. on a Rabbit 2000 I achived ~450KB/sec "netto" write speed towards
>> a FAT 16 filesystem.
>
>Wow.. that's great! Considering that you also update the fat, the 
>"gross" (raw sectors) write rate is probably even higher!
>Did you perform  "single" sector writes or "multiple"?

Single sector writes are just fine. Thats especially true if you
consider the fact that memory is costly on small systems. Todays CF
cards will anyways be magnitudes faster than your embedded controller
(asuming your are not playing with a 400Mhz XScale or such :-) ) so
multiple sector writes would only place more burden on the part of the
host system.

>> 
>> The key issue is the sector IO routine. Here it's well worth the
>> effort to use assembly if your controller is slow. I.e. formentioned
>> R2000 only did ~100KB/sec useing 'C' code and the Zworld compiler.
>> 
>
>Do you think that assembly or C could make the difference?

Ohh yesss definately. Again, especially the code writing and reading a
sector is VERY important. I got the best results by also using A10
that said the so called but totally missleading "memory mapped mode".
There you can use special block move operations your controller might
support (as it's the case with the R2000 i.e.). Note thought that the
difference is not all that big. 

The C versus assembly story is of course different when it comes to
implementing FAT. I never ever would consider implementing FAT in
assembly these days. That's an area where 'C' is much more apropriate.

>And what about the impact of:
>- memory buffers

You must buffer wisely. My FAT library uses two FAT cache sectors. A
"regular" one which is used to follow cluster chains during IO
whereever it ocures, and - most important for the write speed - one
that "hovers" over the area from where new clusters are allocated.
like this allocating a new cluster does not require seeking all of the
FAT everytime you need a new one. Then there are also two general
purpose IO buffers used and one per open file. This is however only
the case because my implementation must and does support multiple
"processes" in paralell which must be able to modify a single file
concurently.

Another trick is to only buffer sector data where needed. I.e. small
writes will go into the cache secotr where bigger amonts got into the
cache up until it's full and flushed, but the the remaing part of the
user buffer space is directly written up to the reminder which then
goes again into the cache etc. With tricks like this you can increase
write speed for "bigger" amounts of data siginficantly - where bigger
is everything > 512 KB. Needless to say that you also can have lot's
of influence on the application layer. Chooseing filelayouts etc. so
as the file IO wher it have to be fast is sector alligned can have a
significant impact etc. etc.

>- single/multiple sectors write

As mentioned, single. IMHO most CF cards not even really support
multiple in that the number of sectors returned is always 1. I seem to
remember that the specs even mention this somewhere but might be that
this has changed over time or that I remember incorrectly.

>- mode of CF: ATA mode, Memory Mode, etc.

The mode does not matter that much except for the trick with the block
move operations which if memory serves only is available in memory
mapped mode. Note though that while there is a difference in the
achvieable speed it is not THAT big (i.e. probably 10% gain or such).

>Did you use some publicly available code, or did you just write your own 
>routines?

I'm one of those crazy people who are faster in writing their own
library than understanding someone elses code :-) Seriousely, I had to
write all of it on my own for various reasons. 

If you need complete FAT compliance this is going to use a week or two
of your time if you can limit yourself to FAT 16 and no long name
suport (the latter would be a real nightmare btw.)

However, if you only are after logging data, you can prealocate the
file in a first simple step, then simply write consecutive sectors
during the real logging. If you do this cleverly, you really don't
have to have much really FAT specific code hence you might save lots
of time. Don't expect to be able to port existing say Linux FAT Code
to an 8 bitter though. 

Code that deals with sector IO should be readily available. If you
really are in the dare need of this, I can help you out. 

HTH

Markus

"Haldo" <haldo@somewhere.net> wrote in message
news:d9p1ql$k00$1@newsfeed.cineca.it...
> How many X's did it have? I mean 8x,12x,66x?
>
> Yes, your hints could improve performance, I'm quite confident about this.
>
> But how do you detect the end of write of a sector?
> Do you just wait an appropriate number of msecs or poll the RDY/-BSY
signal?
>

Read the status register.

Ross