Vladimir Vassilevsky wrote:
> "David Brown" <d...@westcontrol.removethisbit.com> wrote in message
>>CF cards and other earlier flash devices are not that great at wear
>>levelling and bad block handling (that's one of the reasons for
>>flash-specific file systems like YAFS and JFFS2).
> 
> The actual erase block size in NAND flash is something like 32/64/128/256KB,
> being bigger for the higher capacity devices. What it implies: any write
> operation through the IDE interface is actually read - copy - erase -
> modify - write at the controller level.

Not if the implementor of the controller firmware did his homework. If
that's the case, a 512-byte block write ends up as a 512-byte block
write at the flash device, at a new address, and the old address is just
marked as unusable. No read/erase/modify/write for every operation.
Garbage compaction happens a some time in the background.

> Since this kitchen is hidden behind IDE, there is no point in
> using YAFS or JFS or such.

Exactly. Although I heard some controller firmware used in consumer
flash disks optimizes based on the assumption that the file system at
the other end of the IDE or USB interface is FAT.

> A disk cache with the blocks of 32k makes a lot of sense though.

Not more or less for a flash disk than any other disk.


  Stefan

Stefan Reuther wrote:

> Vladimir Vassilevsky wrote:
> 
>>"David Brown" <d...@westcontrol.removethisbit.com> wrote in message

>>The actual erase block size in NAND flash is something like 32/64/128/256KB,
>>being bigger for the higher capacity devices. What it implies: any write
>>operation through the IDE interface is actually read - copy - erase -
>>modify - write at the controller level.
> 
> Not if the implementor of the controller firmware did his homework.

This is what described in the appnotes from Samsung and Sandisk.

> If
> that's the case, a 512-byte block write ends up as a 512-byte block
> write at the flash device, at a new address, and the old address is just
> marked as unusable. No read/erase/modify/write for every operation.
> Garbage compaction happens a some time in the background.

So something like FAT or MFT has to be maintained internally, and it has 
to be done in the background instead of by every transaction. This 
scheme doesn't seem to be very applicable for the removable media.

>>Since this kitchen is hidden behind IDE, there is no point in
>>using YAFS or JFS or such.
> 
> Exactly. Although I heard some controller firmware used in consumer
> flash disks optimizes based on the assumption that the file system at
> the other end of the IDE or USB interface is FAT.
> 
> 
>>A disk cache with the blocks of 32k makes a lot of sense though.
> Not more or less for a flash disk than any other disk.

There is a very significant penalty in the speed of the flash write 
operation  at the IDE level if the short blocks are used. The difference 
can be as high as 10 times or so. According to the appnotes, this 
happens due to the read - modify - write thing.

> 
> 
>   Stefan

VLV

David Brown schreef:
> Vladimir Vassilevsky wrote:
>>
>> Guess how many bad blocks are typical for NAND flash of several GB 
>> capacity? As many as 2 percent! There could be the whole areas of 
>> hundreds of megabytes of the contiguous bad cells, as well as the 
>> random scatter.
>>
>> It is possible to do the extensive read/write test to find the most of 
>> the unreliable blocks; but it takes many hours.
>>
>> I didn't encounter this problem until we started to use the high 
>> capacity CF cards. The bad blocks were very rare for the cards of 1GB 
>> and below. Since the flash iself is hidden behind the IDE interface 
>> and a compatible file system, and the read/write performance is 
>> critical, it is generally impossible to apply an error correction scheme.
>>
>> I was under impression that flash is more reliable then HDD; now I see 
>> that it is not so. Do you know how reliable are the IDE flash drives?
>>
> 
> NAND flash always has defects in manufacturing - the devices are 
> designed to cope with a certain level of faults to make manufacturing 
> cheaper (the same applies to many other types of chips, and hard disks). 
>  Each sector in NAND has extra space for error correction and detection 
> (IIRC, 512 byte sectors are actually 528 bytes in size).  Bad blocks can 
> be detected and marked during manufacture and testing, and blocks that 
> go bad (due to wearing out) are detected in use and the data moved to 
> different blocks.
> 
> The same thing is done with hard disks - the controller detects bad 
> blocks, and re-maps them.  There are a few differences, however - on 
> hard disks, you get bad blocks in manufacturing but it is rare that a 
> good block goes bad in use.  With flash, the controller can almost 
> always spot a bad block and recover the data (since it's normally a 
> single bit failure, the ECC will fix it), while on a hard disk you lose 
> data.  

Modern hard disks rely (heavily) on ECC too, likewise a hard disk can 
spot blocks becoming bad before they become unrecoverable. In other 
words: the situation for traditional hard disks is not that different.

David Brown schreef:
> Didi wrote:
>> Dombo wrote:
>>> Didi schreef:
>>>> Vladimir Vassilevsky wrote:
>>>>> ...
>>>>>
>>>>> I was under impression that flash is more reliable then HDD; now I see
>>>>> that it is not so. Do you know how reliable are the IDE flash drives?
>>>>>
>>>> Vladimir,
>>>> if flash were a viable and reliable replacement for HDDs this would
>>>> have
>>>> happened for years by now, the costs would have gone down. They are
>>>> not, and given their limited number of write cycles they are bound to
>>>> stay out of the way of normal disks (which have achieved an amazing
>>>> level of performance).
>>> When the platter densities of (mechanical) HDD's went up at a certain
>>> point manufactures of HDD's had to resort to error correction schemes to
>>> obtain reliable operation. A modern HDD would be unusable without ECC.
>>> It appears that high density flash is going the same direction.
>>
>> True, but HDDs don't wear out with writing - and flash does.
>> This is a major advantage flash does not promise to catch up
>> with - at least for the time being.
> 
> HDDs wear out through use.  The lifetime is roughly dependant on the 
> time the hard disk has been powered up, and how much the head is moved. 
>  It's thus fairly independent of the size. 

Also the number of spin-ups of ordinary HDD's is only guaranteed to 
about 50.000 cycles. This can be a limiting factor when one needs to 
employ aggressive power saving strategies.

Vladimir Vassilevsky wrote:
> Stefan Reuther wrote:
>> Vladimir Vassilevsky wrote:
>>> "David Brown" <d...@westcontrol.removethisbit.com> wrote in message
>>> The actual erase block size in NAND flash is something like
>>> 32/64/128/256KB,
>>> being bigger for the higher capacity devices. What it implies: any write
>>> operation through the IDE interface is actually read - copy - erase -
>>> modify - write at the controller level.
>>
>> Not if the implementor of the controller firmware did his homework.
> 
> This is what described in the appnotes from Samsung and Sandisk.

Interesting. Failure-prone for removable media, and reduces the part's
life time :->

>> If that's the case, a 512-byte block write ends up as a 512-byte block
>> write at the flash device, at a new address, and the old address is just
>> marked as unusable. No read/erase/modify/write for every operation.
>> Garbage compaction happens a some time in the background.
> 
> So something like FAT or MFT has to be maintained internally, and it has
> to be done in the background instead of by every transaction. This
> scheme doesn't seem to be very applicable for the removable media.

The remapping table can be reconstructed at any time from the data on
the media. The scheme is called log-structured file system. The
implementations I've seen so far (Green Hills and YAFFS, although I've
only started looking closer at the latter) work this way.


  Stefan

On Jun 30, 2:13 pm, Stefan Reuther <stefan.n...@arcor.de> wrote:

> Interesting. Failure-prone for removable media, and reduces the part's
> life time :->

Not really.  When you think about it, writes and read-modify-writes
are the embedded controller's only easy opportunity for moving
frequently modified data to a less used block - this is the easy
change to accomplish wear leveling.

I would also assume that enough energy can be stored in an on-card
capacitor to flush a full block write from cache ram to the NAND, at
least provided that the destination is already erased, which you'd
probably do premptively since the destination is going to be a
different (less worn) location anyway.

Le Sun, 29 Jun 2008 09:53:28 -0500, Vladimir Vassilevsky a écrit:

> "Habib Bouaziz-Viallet" <h...@rigel.systems> wrote in message
> news:48672e24$0$21859$4...@news.free.fr...
>> Le Fri, 27 Jun 2008 11:10:29 -0500, Vladimir Vassilevsky a écrit:
>>
>> > Since the flash iself is hidden behind the IDE interface and
>> > a compatible file system, and the read/write performance is critical, it
>> > is generally impossible to apply an error correction scheme.
>> "... generally impossible to apply an error correction scheme ..."
>> Are you really a serious guy or you comes here on subjects that you do not
>> master?
> 
> You blockheads should bow to the opportunity of receiving a lesson of
> wisdom.
And i bow in front of you Vlad or should i say in front of your wisdom.
> 
>> I will try to make the kind of "Vlad response" :
>> Is it a hobbyist project ? I think this is. mmmhh, Let see ... who cares
>> about the safety of hobbyists projects ?
> 
> Good comment, 
Really ?
> Would you suggest a solution which:
> 
> 1) Small, low power, robust and swapable.
> 2) Sustain the read/write speeds of 5MB/s at the least
> 3) Compatible with standard flash card readers regardess of OS
> 
Robust ???? Compatible ????
To meet these conditions i suggest to avoid using NAND Flash
based technology.

If I had the ultimate solution for data storage I would probably be on a
beach in the bahamas with charming creatures around me :-)


> Vladimir Vassilevsky
> DSP and Mixed Signal Design Consultant
> www.abvolt.com

-- 
HBV

Habib Bouaziz-Viallet wrote:

> Le Sun, 29 Jun 2008 09:53:28 -0500, Vladimir Vassilevsky a =C3=A9crit:
>>"Habib Bouaziz-Viallet" <h...@rigel.systems> wrote in message
>>news:48672e24$0$21859$4...@news.free.fr...
>>>Le Fri, 27 Jun 2008 11:10:29 -0500, Vladimir Vassilevsky a =C3=A9crit:=

>>>
>>>
>>>>Since the flash iself is hidden behind the IDE interface and
>>>>a compatible file system, and the read/write performance is critical,=
 it
>>>>is generally impossible to apply an error correction scheme.
>>>
>>>"... generally impossible to apply an error correction scheme ..."
>>>Are you really a serious guy or you comes here on subjects that you do=
 not
>>>master?


>>You blockheads should bow to the opportunity of receiving a lesson of
>>wisdom.
>=20
> And i bow in front of you Vlad or should i say in front of your wisdom.=


Eh? Wasn't it said to bow to the _opportunity_ ?

>>>I will try to make the kind of "Vlad response" :
>>>Is it a hobbyist project ? I think this is. mmmhh, Let see ... who car=
es
>>>about the safety of hobbyists projects ?
>>
>>Good comment,=20
>=20
> Really ?

Yes. But don't get a swelled head.

>=20
>>Would you suggest a solution which:
>>
>>1) Small, low power, robust and swapable.
>>2) Sustain the read/write speeds of 5MB/s at the least
>>3) Compatible with standard flash card readers regardess of OS
>>
> Robust ???? Compatible ????
> To meet these conditions i suggest to avoid using NAND Flash
> based technology.
> If I had the ultimate solution for data storage I would probably be on =
a
> beach in the bahamas with charming creatures around me :-)

If your goal is beach with charming creatures, you are perhaps in the=20
wrong kind of business for that. Even the ultimate storage solution=20
won't help.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

David Brown wrote:
<snip>

> Good manufacturers quote MTBF numbers 
> that are orders of magnitude higher than for hard disks, and wear is no 
> longer a practical issue for larger flash disks (I've seen flash disks 
> spec'ed for *continuous* 20 MB/s writes for years).  See also 
> <http://wiki.eeeuser.com/ssd_write_limit>; - a 4 GB Eee PC disk should be 
> fine for a normal user for 25 years.  And since wear is levelled across 
> a disk, a 128 GB disk will survive 32 times as much use for the same time.

I don't follow this logic; are you saying that for statistically average
usage in a system that doesn't thrash, wear-leveling on a larger device
has more room to work and thus has a longer life-cycle?  Doesn't this
argument fall apart with modern VM o/s usage and thrashing?

Michael

msg wrote:
> David Brown wrote:
> <snip>
> 
>> Good manufacturers quote MTBF numbers that are orders of magnitude 
>> higher than for hard disks, and wear is no longer a practical issue 
>> for larger flash disks (I've seen flash disks spec'ed for *continuous* 
>> 20 MB/s writes for years).  See also 
>> <http://wiki.eeeuser.com/ssd_write_limit>; - a 4 GB Eee PC disk should 
>> be fine for a normal user for 25 years.  And since wear is levelled 
>> across a disk, a 128 GB disk will survive 32 times as much use for the 
>> same time.
> 
> I don't follow this logic; are you saying that for statistically average
> usage in a system that doesn't thrash, wear-leveling on a larger device
> has more room to work and thus has a longer life-cycle?  Doesn't this
> argument fall apart with modern VM o/s usage and thrashing?
> 

Modern OS's (even windows) do not thrash unless you try to run too much 
with too little ram.

And yes, if you have a larger disk to spread the writes, then there will 
be fewer erase/writes per block for the same amount of write usage - 
that's fairly obvious maths.

And even if you do occasionally thrash the swap file or swap partition, 
the argument still applies - a larger disk will last longer.