"David Brown" <d...@westcontrol.removethisbit.com> wrote in message
news:486611d3$0$14988$8...@news.wineasy.se...
> Vladimir Vassilevsky wrote:
> >
> > Guess how many bad blocks are typical for NAND flash of several GB
> > capacity? As many as 2 percent! There could be the whole areas of
> > hundreds of megabytes of the contiguous bad cells, as well as the random
> > scatter.
> >
> > It is possible to do the extensive read/write test to find the most of
> > the unreliable blocks; but it takes many hours.
> >
> > I didn't encounter this problem until we started to use the high
> > capacity CF cards. The bad blocks were very rare for the cards of 1GB
> > and below. Since the flash iself is hidden behind the IDE interface and
> > a compatible file system, and the read/write performance is critical, it
> > is generally impossible to apply an error correction scheme.
> >
> > I was under impression that flash is more reliable then HDD; now I see
> > that it is not so. Do you know how reliable are the IDE flash drives?
> >
>
> NAND flash always has defects in manufacturing - the devices are
> designed to cope with a certain level of faults to make manufacturing
> cheaper (the same applies to many other types of chips, and hard disks).
>   Each sector in NAND has extra space for error correction and detection
> (IIRC, 512 byte sectors are actually 528 bytes in size).  Bad blocks can
> be detected and marked during manufacture and testing, and blocks that
> go bad (due to wearing out) are detected in use and the data moved to
> different blocks.

The utterly bad blocks are detected at manufacturing; however there is a
bunch of the unreliable blocks which takes hours of testing to discover. If
the bad block is detected when in use, it means that the data is lost
already. It is too late to hide it by remaping.

> CF cards and other earlier flash devices are not that great at wear
> levelling and bad block handling (that's one of the reasons for
> flash-specific file systems like YAFS and JFFS2).

The actual erase block size in NAND flash is something like 32/64/128/256KB,
being bigger for the higher capacity devices. What it implies: any write
operation through the IDE interface is actually read - copy - erase -
modify - write at the controller level. The other surcumstance of that is
the speed penalty for the writes misaligned to the erase block size. Since
this kitchen is hidden behind IDE, there is no point in using YAFS or JFS or
such. A disk cache with the blocks of 32k makes a lot of sense though.


Vladimir Vassilevsky
DSP and Mixed Signal Consultant
www.abvolt.com

Vladimir Vassilevsky wrote:
> "David Brown" <d...@westcontrol.removethisbit.com> wrote in message
> news:486611d3$0$14988$8...@news.wineasy.se...
>> Vladimir Vassilevsky wrote:
>>> Guess how many bad blocks are typical for NAND flash of several GB
>>> capacity? As many as 2 percent! There could be the whole areas of
>>> hundreds of megabytes of the contiguous bad cells, as well as the random
>>> scatter.
>>>
>>> It is possible to do the extensive read/write test to find the most of
>>> the unreliable blocks; but it takes many hours.
>>>
>>> I didn't encounter this problem until we started to use the high
>>> capacity CF cards. The bad blocks were very rare for the cards of 1GB
>>> and below. Since the flash iself is hidden behind the IDE interface and
>>> a compatible file system, and the read/write performance is critical, it
>>> is generally impossible to apply an error correction scheme.
>>>
>>> I was under impression that flash is more reliable then HDD; now I see
>>> that it is not so. Do you know how reliable are the IDE flash drives?
>>>
>> NAND flash always has defects in manufacturing - the devices are
>> designed to cope with a certain level of faults to make manufacturing
>> cheaper (the same applies to many other types of chips, and hard disks).
>>   Each sector in NAND has extra space for error correction and detection
>> (IIRC, 512 byte sectors are actually 528 bytes in size).  Bad blocks can
>> be detected and marked during manufacture and testing, and blocks that
>> go bad (due to wearing out) are detected in use and the data moved to
>> different blocks.
> 
> The utterly bad blocks are detected at manufacturing; however there is a
> bunch of the unreliable blocks which takes hours of testing to discover. If
> the bad block is detected when in use, it means that the data is lost
> already. It is too late to hide it by remaping.
> 

The point of ECC - Error checking and *correcting* - is that slightly 
bad blocks do not lead to lost data.  The most common problem in flash 
blocks (excluding any totally failed blocks found in manufacturing) are 
single-bit errors - bits that can't erase or program properly.  These 
single-bit errors do not lead to data loss, and the flash controller can 
easily detect and correct them.  It is even possible that the controller 
will continue using a block with bad bits, and will not disable the 
block until a certain number of bad bits have been found (I don't know 
what error rates are used here in practice).

>> CF cards and other earlier flash devices are not that great at wear
>> levelling and bad block handling (that's one of the reasons for
>> flash-specific file systems like YAFS and JFFS2).
> 
> The actual erase block size in NAND flash is something like 32/64/128/256KB,
> being bigger for the higher capacity devices. What it implies: any write
> operation through the IDE interface is actually read - copy - erase -
> modify - write at the controller level. The other surcumstance of that is
> the speed penalty for the writes misaligned to the erase block size. Since
> this kitchen is hidden behind IDE, there is no point in using YAFS or JFS or
> such. A disk cache with the blocks of 32k makes a lot of sense though.
> 

YAFS and JFFS are designed for when you have direct access to the flash 
- when you are using a controller that handles the wear levelling and 
block placement (such as for CF cards and IDE/SAS/SATA controllers), you 
should not use flash-specific file systems of that sort.

Dombo wrote:
> Didi schreef:
> > Vladimir Vassilevsky wrote:
> >> ...
> >>
> >> I was under impression that flash is more reliable then HDD; now I see
> >> that it is not so. Do you know how reliable are the IDE flash drives?
> >>
> >
> > Vladimir,
> > if flash were a viable and reliable replacement for HDDs this would
> > have
> > happened for years by now, the costs would have gone down. They are
> > not, and given their limited number of write cycles they are bound to
> > stay out of the way of normal disks (which have achieved an amazing
> > level of performance).
>
> When the platter densities of (mechanical) HDD's went up at a certain
> point manufactures of HDD's had to resort to error correction schemes to
> obtain reliable operation. A modern HDD would be unusable without ECC.
> It appears that high density flash is going the same direction.

True, but HDDs don't wear out with writing - and flash does.
This is a major advantage flash does not promise to catch up
with - at least for the time being.

Didi

------------------------------------------------------
Dimiter Popoff               Transgalactic Instruments

http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/sets/72157600228621276/

Le Fri, 27 Jun 2008 11:10:29 -0500, Vladimir Vassilevsky a écrit:

> Since the flash iself is hidden behind the IDE interface and 
> a compatible file system, and the read/write performance is critical, it 
> is generally impossible to apply an error correction scheme.
"... generally impossible to apply an error correction scheme ..."
Are you really a serious guy or you comes here on subjects that you do not
master?

I will try to make the kind of "Vlad response" :
Is it a hobbyist project ? I think this is. mmmhh, Let see ... who cares
about the safety of hobbyists projects ?  
> 
> Vladimir Vassilevsky
> DSP and Mixed Signal Design Consultant http://www.abvolt.com

-- 
HBV

Didi wrote:
> Dombo wrote:
>> Didi schreef:
>>> Vladimir Vassilevsky wrote:
>>>> ...
>>>>
>>>> I was under impression that flash is more reliable then HDD; now I see
>>>> that it is not so. Do you know how reliable are the IDE flash drives?
>>>>
>>> Vladimir,
>>> if flash were a viable and reliable replacement for HDDs this would
>>> have
>>> happened for years by now, the costs would have gone down. They are
>>> not, and given their limited number of write cycles they are bound to
>>> stay out of the way of normal disks (which have achieved an amazing
>>> level of performance).
>> When the platter densities of (mechanical) HDD's went up at a certain
>> point manufactures of HDD's had to resort to error correction schemes to
>> obtain reliable operation. A modern HDD would be unusable without ECC.
>> It appears that high density flash is going the same direction.
> 
> True, but HDDs don't wear out with writing - and flash does.
> This is a major advantage flash does not promise to catch up
> with - at least for the time being.
> 

HDDs wear out through use.  The lifetime is roughly dependant on the 
time the hard disk has been powered up, and how much the head is moved. 
  It's thus fairly independent of the size.  Flash wears out through 
erase-write cycles on blocks.  So the more blocks you have to spread the 
wear, the longer the lifetime, and the more you read rather than write, 
the longer the lifetime.  So as flash drives get bigger, they are 
surpassing HDDs for reliability and lifetime.  For common desktop usage, 
a 32 GB flash disk will probably far outlast a typical hard disk - with 
256 GB and bigger flash disks, even high quality hard disks are no 
longer competitive on reliability and lifetime for real applications.

The big issue is the cost per GB - hard disks are still much much 
cheaper.  A second issue is speed - standard hard disks are 
significantly faster than standard flash disks.  But that will change - 
flash speeds are easily scaled (just use several devices in parallel) 
once the price is right.

"David Brown" <d...@westcontrol.removethisbit.com> wrote in message
news:48676bc8$0$14992$8...@news.wineasy.se...

> HDDs wear out through use.  The lifetime is roughly dependant on the
> time the hard disk has been powered up, and how much the head is moved.

A HDD has one big  problem: the lifetime tends to go to the end when it
falls on the floor.
Yes there are the special HDDs, accelerometers, suspensions and such;
however it makes HDD much less attractive then a flash alternatives.
Besides, a flash card is easy to swap.

>   It's thus fairly independent of the size.

But the small and light HDD is more likely to survive the contact with the
floor :)

>  Flash wears out through
> erase-write cycles on blocks.  So the more blocks you have to spread the
> wear, the longer the lifetime, and the more you read rather than write,
> the longer the lifetime.

The flash write endurance (as stated by the manufacturers) is at the order
of millions of cycles. This is more then enough for many applications; there
is really no point to bother about the wear leveling. The problem I am
facing is the infant mortality of the unreliable cells.

>  So as flash drives get bigger, they are
> surpassing HDDs for reliability and lifetime.  For common desktop usage,
> a 32 GB flash disk will probably far outlast a typical hard disk - with
> 256 GB and bigger flash disks, even high quality hard disks are no
> longer competitive on reliability and lifetime for real applications.
>
> The big issue is the cost per GB - hard disks are still much much
> cheaper.  A second issue is speed - standard hard disks are
> significantly faster than standard flash disks.

A common x300 8GB Lexar CF card from WallMart sustains the read/write speeds
at the order of 25...30MB/s. Those are the numbers that I measured; the
upper limit is because of the hardware, not the flash.

>  But that will change -
> flash speeds are easily scaled (just use several devices in parallel)
> once the price is right.

The falling on the floor, weight/size,  power consumption, instant readyness
and the heat dissipation are the problems of HDDs which flash doesn't have.

Vladimir Vassilevsky
DSP and Mixed Signal Consultant
www.abvolt.com

"Habib Bouaziz-Viallet" <h...@rigel.systems> wrote in message
news:48672e24$0$21859$4...@news.free.fr...
> Le Fri, 27 Jun 2008 11:10:29 -0500, Vladimir Vassilevsky a écrit:
>
> > Since the flash iself is hidden behind the IDE interface and
> > a compatible file system, and the read/write performance is critical, it
> > is generally impossible to apply an error correction scheme.
> "... generally impossible to apply an error correction scheme ..."
> Are you really a serious guy or you comes here on subjects that you do not
> master?

You blockheads should bow to the opportunity of receiving a lesson of
wisdom.

> I will try to make the kind of "Vlad response" :
> Is it a hobbyist project ? I think this is. mmmhh, Let see ... who cares
> about the safety of hobbyists projects ?

Good comment, but where is meat?
Would you suggest a solution which:

1) Small, low power, robust and swapable.
2) Sustain the read/write speeds of 5MB/s at the least
3) Compatible with standard flash card readers regardess of OS


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
www.abvolt.com

Un bel giorno Vladimir Vassilevsky digitò:

> Guess how many bad blocks are typical for NAND flash of several GB 
> capacity? As many as 2 percent!

If with "typical" you mean "the grade used for low-cost consumer
electronics", you are right. But NAND normally come at least in two or
three different grade options: no bad blocks, 2% initial bad blocks without
dynamic bad blocks (until the minimum number of erase cycles), 2% bad block
with dynamic bad blocks, etc.

> I was under impression that flash is more reliable then HDD; now I see 
> that it is not so. Do you know how reliable are the IDE flash drives?

I believed that solid-state drives would have taken over a lot easier, too.
Instead the good old rotating junk continues to resist very well, except in
very hot or very vibrational or very space-restrained applications.

-- 
emboliaschizoide.splinder.com

Vladimir Vassilevsky wrote:
> "David Brown" <d...@westcontrol.removethisbit.com> wrote in message
> news:48676bc8$0$14992$8...@news.wineasy.se...
> 
>> HDDs wear out through use.  The lifetime is roughly dependant on the
>> time the hard disk has been powered up, and how much the head is moved.
> 
> A HDD has one big  problem: the lifetime tends to go to the end when it
> falls on the floor.
> Yes there are the special HDDs, accelerometers, suspensions and such;
> however it makes HDD much less attractive then a flash alternatives.
> Besides, a flash card is easy to swap.
> 
>>   It's thus fairly independent of the size.
> 
> But the small and light HDD is more likely to survive the contact with the
> floor :)
> 

If you are planning on dropping your drive, flash certainly has the 
advantage!

>>  Flash wears out through
>> erase-write cycles on blocks.  So the more blocks you have to spread the
>> wear, the longer the lifetime, and the more you read rather than write,
>> the longer the lifetime.
> 
> The flash write endurance (as stated by the manufacturers) is at the order
> of millions of cycles. This is more then enough for many applications; there
> is really no point to bother about the wear leveling. The problem I am
> facing is the infant mortality of the unreliable cells.
> 

Write endurance is not normally in the millions for NAND flash - perhaps 
you are thinking of NOR flash, which is inherently more reliable (but 
costs more per bit)?  With wear levelling, write endurance is not a big 
issue for modern flash disks - without it, you would wear out parts of 
your disk (such as the FAT on FAT formatted drives).

>>  So as flash drives get bigger, they are
>> surpassing HDDs for reliability and lifetime.  For common desktop usage,
>> a 32 GB flash disk will probably far outlast a typical hard disk - with
>> 256 GB and bigger flash disks, even high quality hard disks are no
>> longer competitive on reliability and lifetime for real applications.
>>
>> The big issue is the cost per GB - hard disks are still much much
>> cheaper.  A second issue is speed - standard hard disks are
>> significantly faster than standard flash disks.
> 
> A common x300 8GB Lexar CF card from WallMart sustains the read/write speeds
> at the order of 25...30MB/s. Those are the numbers that I measured; the
> upper limit is because of the hardware, not the flash.
> 
>>  But that will change -
>> flash speeds are easily scaled (just use several devices in parallel)
>> once the price is right.
> 
> The falling on the floor, weight/size,  power consumption, instant readyness
> and the heat dissipation are the problems of HDDs which flash doesn't have.
> 
> Vladimir Vassilevsky
> DSP and Mixed Signal Consultant
> www.abvolt.com
> 
> 

David Brown wrote:

> Vladimir Vassilevsky wrote:
> 
>> The flash write endurance (as stated by the manufacturers) is at the 
>> order of millions of cycles. This is more then enough for many applications; 
>> there is really no point to bother about the wear leveling. The problem I am
>> facing is the infant mortality of the unreliable cells.
>>
> 
> Write endurance is not normally in the millions for NAND flash - perhaps 
> you are thinking of NOR flash, which is inherently more reliable (but 
> costs more per bit)?

Millions of cycles is what stated in the datasheets for the finished 
devices like flash cards or jump drives. We can only guess what methods 
are actually employed inside. Perhaps the wear leveling is taken care 
off. However the reliability appears to be surprisingly low, especially 
for the higher capacity devices.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com