Shutdown behavior and flash storage devices

Hi All - I'm wondering what other folks do about this issue...

Consumer flash storage devices (USB memory stick, SD card, etc)
have a nice internal wear-leveling controller. When one does a write 
operation, lots of sectors may be internally rejiggered to provide 
uniform wear (so things that are never rewritten from the application 
point of view are actually moved around and rewritten). This activity is 
invisible from normal application's point of view, but takes some time. 
If the device is powered off during such activity, data corruption... 
So, its necessary to provide the device some time after the last write 
before it is powered off. Plenty of embedded stuff I've seen too often 
corrupts memory at power off, and of course the device manufacturers 
blame the memory manufacturers...

So, how do you avoid this kind of corruption in your designs?

Thanks,
Best Regards, Dave

Dave Nadler <drn@nadler.com> wrote:
> Hi All - I'm wondering what other folks do about this issue...
> 
> Consumer flash storage devices (USB memory stick, SD card, etc)
> have a nice internal wear-leveling controller. When one does a write 
> operation, lots of sectors may be internally rejiggered to provide 
> uniform wear (so things that are never rewritten from the application 
> point of view are actually moved around and rewritten). This activity is 
> invisible from normal application's point of view, but takes some time. 
> If the device is powered off during such activity, data corruption... 
> So, its necessary to provide the device some time after the last write 
> before it is powered off. Plenty of embedded stuff I've seen too often 
> corrupts memory at power off, and of course the device manufacturers 
> blame the memory manufacturers...
> 
> So, how do you avoid this kind of corruption in your designs?

If it's a USB stick, SCSI has an UNLOAD command which is what you do when
you eject removable media like a CD.  I would hope that a USB stick (USB
mass storage = SCSI protocol) would use it as a hint to flush its write
cache and get ready to be removed, but I don't actually know if it does.

Maybe they can also detect the USB signalling going away, which is what
happens when you pull the stick, a few ms before the power disconnects, due
to the layout of the USB A connector.

Enterprise SSDs have power loss protection so you can pull the power at any
time and they'll tidy up nicely using stored energy in a bank of capacitors.

Theo

On 7/24/2021 8:19 AM, Dave Nadler wrote:
> Hi All - I'm wondering what other folks do about this issue...
> 
> Consumer flash storage devices (USB memory stick, SD card, etc)
> have a nice internal wear-leveling controller. When one does a write operation, 
> lots of sectors may be internally rejiggered to provide uniform wear (so things 
> that are never rewritten from the application point of view are actually moved 
> around and rewritten). This activity is invisible from normal application's 
> point of view, but takes some time. If the device is powered off during such 
> activity, data corruption... So, its necessary to provide the device some time 
> after the last write before it is powered off. Plenty of embedded stuff I've 
> seen too often corrupts memory at power off, and of course the device 
> manufacturers blame the memory manufacturers...
> 
> So, how do you avoid this kind of corruption in your designs?

Two (transient) copies of each "chunk" of data.  Update the second
copy (on or before impending power fail).  When complete, toggle
a pointer to reference it as the most recent (assuming you don't
have metadata that automatically provides that sort of information).

The "first" copy can now be removed and its space reallocated to
some other "second copy"

Then, move on to the next "chunk".  Design your algorithms so that
you can tolerate some chunks being "stale" (not updated in time)
wrt the others.

On 7/25/2021 4:10, Don Y wrote:
> On 7/24/2021 8:19 AM, Dave Nadler wrote:
>> Hi All - I'm wondering what other folks do about this issue...
>>
>> Consumer flash storage devices (USB memory stick, SD card, etc)
>> have a nice internal wear-leveling controller. When one does a write 
>> operation, lots of sectors may be internally rejiggered to provide 
>> uniform wear (so things that are never rewritten from the application 
>> point of view are actually moved around and rewritten). This activity 
>> is invisible from normal application's point of view, but takes some 
>> time. If the device is powered off during such activity, data 
>> corruption... So, its necessary to provide the device some time after 
>> the last write before it is powered off. Plenty of embedded stuff I've 
>> seen too often corrupts memory at power off, and of course the device 
>> manufacturers blame the memory manufacturers...
>>
>> So, how do you avoid this kind of corruption in your designs?
> 
> Two (transient) copies of each "chunk" of data.  Update the second
> copy (on or before impending power fail).  When complete, toggle
> a pointer to reference it as the most recent (assuming you don't
> have metadata that automatically provides that sort of information).
> 
> The "first" copy can now be removed and its space reallocated to
> some other "second copy"
> 
> Then, move on to the next "chunk".  Design your algorithms so that
> you can tolerate some chunks being "stale" (not updated in time)
> wrt the others.

Don, this strategy is obviously good (standard?) for say disk writes
etc., but I don't know if it will be OK for flash as it involves erase
then write, AFAIK "erase" is the most killing part. Perhaps they do
a lot of "refresh", i.e. read while they can then rewrite the same
locations? I don't really know, just wondering how they do it.

On 7/24/2021 9:10 PM, Don Y wrote:
>> So, how do you avoid this kind of corruption in your designs?
> 
> Two (transient) copies of each "chunk" of data.  Update the second
> copy (on or before impending power fail).  When complete, toggle
> a pointer to reference it as the most recent (assuming you don't
> have metadata that automatically provides that sort of information).
> 
> The "first" copy can now be removed and its space reallocated to
> some other "second copy"
> 
> Then, move on to the next "chunk".  Design your algorithms so that
> you can tolerate some chunks being "stale" (not updated in time)
> wrt the others.

Don, I don't think that works. The entire contents of the volume can
be corrupted if power is removed during wear-leveling operations that 
are invisible to the host OS. And I guess you are assuming no file 
system is in use (because file system internals would still get corrupted)?

On 7/25/2021 4:18 AM, Dimiter_Popoff wrote:
> On 7/25/2021 4:10, Don Y wrote:
>> On 7/24/2021 8:19 AM, Dave Nadler wrote:
>>> Hi All - I'm wondering what other folks do about this issue...
>>>
>>> Consumer flash storage devices (USB memory stick, SD card, etc)
>>> have a nice internal wear-leveling controller. When one does a write 
>>> operation, lots of sectors may be internally rejiggered to provide uniform 
>>> wear (so things that are never rewritten from the application point of view 
>>> are actually moved around and rewritten). This activity is invisible from 
>>> normal application's point of view, but takes some time. If the device is 
>>> powered off during such activity, data corruption... So, its necessary to 
>>> provide the device some time after the last write before it is powered off. 
>>> Plenty of embedded stuff I've seen too often corrupts memory at power off, 
>>> and of course the device manufacturers blame the memory manufacturers...
>>>
>>> So, how do you avoid this kind of corruption in your designs?
>>
>> Two (transient) copies of each "chunk" of data.  Update the second
>> copy (on or before impending power fail).  When complete, toggle
>> a pointer to reference it as the most recent (assuming you don't
>> have metadata that automatically provides that sort of information).
>>
>> The "first" copy can now be removed and its space reallocated to
>> some other "second copy"
>>
>> Then, move on to the next "chunk".  Design your algorithms so that
>> you can tolerate some chunks being "stale" (not updated in time)
>> wrt the others.
> 
> Don, this strategy is obviously good (standard?) for say disk writes
> etc., but I don't know if it will be OK for flash as it involves erase
> then write, AFAIK "erase" is the most killing part. Perhaps they do
> a lot of "refresh", i.e. read while they can then rewrite the same
> locations? I don't really know, just wondering how they do it.

Keep a spare block (or two) that are already erased, on hand
(this cuts into the capacity of the device but not in a meaningful way).

As you need to write data to the device, select a page from the
erased block to accept your data.  When the number of "clean pages"
falls below a threshold, schedule another (free!) block to be erased.

If the erasure is interrupted, then you see the "free" block as
free-but-not-erased.

If the page write is interrupted, then you see the page as unclean
*and* the data it WANTED to contain as still residing in the
"previous" location.

There is always going to be a window in which you can get screwed
by unfortunate power cycling.  But, you can shrink this window
(at some manageable cost).

If, instead, you only perform the program/erase/write cycle
on the "parameter block" when you need to update the "parameters",
then you have to move the existing parameters into volatile storage.
Then, program/erase the block.  Then write the parameters back.
Then, verify the write.

This leaves the parameters (in volatile memory) "exposed" for a long
time -- seconds per block.  If you "set up" the write operation ahead
of time, you can trim this to a millisecond, or so.

On 7/25/2021 5:49 AM, Dave Nadler wrote:
> On 7/24/2021 9:10 PM, Don Y wrote:
>>> So, how do you avoid this kind of corruption in your designs?
>>
>> Two (transient) copies of each "chunk" of data.  Update the second
>> copy (on or before impending power fail).  When complete, toggle
>> a pointer to reference it as the most recent (assuming you don't
>> have metadata that automatically provides that sort of information).
>>
>> The "first" copy can now be removed and its space reallocated to
>> some other "second copy"
>>
>> Then, move on to the next "chunk".  Design your algorithms so that
>> you can tolerate some chunks being "stale" (not updated in time)
>> wrt the others.
> 
> Don, I don't think that works. The entire contents of the volume can
> be corrupted if power is removed during wear-leveling operations that are 
> invisible to the host OS. And I guess you are assuming no file system is in use 
> (because file system internals would still get corrupted)?

[I don't use filesystems; do you use a filesystem to manage your *RAM*?  :> ]

I use "raw" devices and have the FTL (and wear-leveling) in my firmware.
So, I know what the hardware is being asked to do when I talk to it;
there's no "black magic" sitting in the middle that I can't quantify/qualify.

If you're going to use a "smart" device, then you have to come up with a
strategy that lets you recover from arbitrary levels of corruption
(because you don't know what that device is going to be doing when the
fit hits the shan).

I suspect you will see your losses limited to a single block (or so) as
it would require more resources to transfer a block's contents into
RAM while another block is nominated to receive them.

There might be different strategies employed by different device vendors.
E.g., one may defer the program/erase of that block until a later time;
another may opt to do it "now".

I don't see how you can expose enough of the internals of a "smart" device
in a manufacturer/model-independent manner.  I.e., even storing a hash with
each "file", there's no guarantee as to WHERE that will reside on the device.
And, when it might "move" (be exposed to loss).

[You also have to be prepared for "good data" being lost in such a reshuffling;
in my case, I can ensure "what I have, I keep (but may lose something NEW)"]

If your needs are for things like configuration parameters (presumably few),
can you consider an EEPROM?  Use FLASH for bigger things -- that tend to
change less frequently (and in more user-visible ways)?

In article <sdhb29$ibh$1@gioia.aioe.org>, Dave Nadler  <drn@nadler.com> wrote:
>Hi All - I'm wondering what other folks do about this issue...
>
>Consumer flash storage devices (USB memory stick, SD card, etc)
>have a nice internal wear-leveling controller. When one does a write 
>operation, lots of sectors may be internally rejiggered to provide 
>uniform wear (so things that are never rewritten from the application 
>point of view are actually moved around and rewritten). This activity is 
>invisible from normal application's point of view, but takes some time. 
>If the device is powered off during such activity, data corruption... 
>So, its necessary to provide the device some time after the last write 
>before it is powered off. Plenty of embedded stuff I've seen too often 
>corrupts memory at power off, and of course the device manufacturers 
>blame the memory manufacturers...
>
>So, how do you avoid this kind of corruption in your designs?
>
>Thanks,
>Best Regards, Dave

It is not necessary to do this.  Here's a very simple way to recover with
no data loss ever.  It is simply "log-only" or "journal-only" storage.

The device keeps 10% capacity reserved (pre-erased) (or less, this is a
performance number).  The driver has enough RAM to track where every logical
block is (this isn't much RAM).  And then writing is always done as a
logfile--write whatever the user data is starting at block 0, and move up.
After writing user data, write a log entry (this can be just right after the
user data).  Writes never go to the block the user indicates, they always go
only to the log pointer.  Reading looks up where that block's latest copy is,
and reads that.

When you're at less than 10% space left (near the end of the device for
first-pass writes), you simply read from +10% from the current log write
position, and write it to the log (compacting it to avoid re-writing any
blocks which are now obsolete) until you free up space.  Since many blocks are
rewritten, when we compact them we free space.  Once copied, then erase the
blocks which were compacted.  Then just keep growing the "log" through the
newly erased blocks.

Upon power up, scan entire device for the log info, and rebuild the RAM index.

Power can be removed at any time, and fully recovered, although obviously the
most recent data may be lost, but we can roll back to the last consistent
state just like journaled file systems (the log entries need a checksum so
we can validate them).  Wear leveling is achieved by design.

There are lots of small ways to improve the above, so do that.

I have no idea what actual devices do, I suspect it's something much more
complex and less robust.

Kent

On 7/25/2021 15:49, Dave Nadler wrote:
> On 7/24/2021 9:10 PM, Don Y wrote:
>>> So, how do you avoid this kind of corruption in your designs?
>>
>> Two (transient) copies of each "chunk" of data.  Update the second
>> copy (on or before impending power fail).  When complete, toggle
>> a pointer to reference it as the most recent (assuming you don't
>> have metadata that automatically provides that sort of information).
>>
>> The "first" copy can now be removed and its space reallocated to
>> some other "second copy"
>>
>> Then, move on to the next "chunk".  Design your algorithms so that
>> you can tolerate some chunks being "stale" (not updated in time)
>> wrt the others.
> 
> Don, I don't think that works. The entire contents of the volume can
> be corrupted if power is removed during wear-leveling operations that 
> are invisible to the host OS. And I guess you are assuming no file 
> system is in use (because file system internals would still get corrupted)?
> 

If you have no knowledge of how the flash medium is written - which
would normally be the case with USB sticks or SD cards etc. - the most
efficient way to mitigate data loss probability is to do as much
write cacheing as you can afford at system level. Of course again
you will be prone to data loss on unexpected power loss but updating
the write cache will be done at (much) longer intervals and in
relatively brief bursts, thus the probability to be hit gets lower.
The rest can only be got rid of by having large enough caps on
the power supply and an early warning that power has been lost
(usually at the input of the stepdowns and you are running on fumes.

Of course you can employ various strategies to mimick flash drive
behaviour at a higher level, Don suggested one, Kent also did,
but these mean having different drivers for different media and
all the consequences of that. I prefer to talk to standard SCSI
or ATA behaving things and leave it to them to handle their
unique specifics. [I remember how I *had to* introduce write
cacheing at system level, in the mid 90-s I started to do backups
under DPS to magneto-optical devices; early days DPS would
simply update its CAT (Cluster Allocation Table) every time
it got modifier and these sectors got corrupted before the backup
would finish.... :-) ].

Dimiter

======================================================
Dimiter Popoff, TGI             http://www.tgi-sci.com
======================================================
http://www.flickr.com/photos/didi_tgi/

On 7/25/2021 11:53 AM, Kent Dickey wrote:
> In article <sdhb29$ibh$1@gioia.aioe.org>, Dave Nadler  <drn@nadler.com> wrote:
>> Hi All - I'm wondering what other folks do about this issue...
>>
>> Consumer flash storage devices (USB memory stick, SD card, etc)
>> have a nice internal wear-leveling controller. When one does a write
>> operation, lots of sectors may be internally rejiggered to provide
>> uniform wear (so things that are never rewritten from the application
>> point of view are actually moved around and rewritten). This activity is
>> invisible from normal application's point of view, but takes some time.
>> If the device is powered off during such activity, data corruption...
>> So, its necessary to provide the device some time after the last write
>> before it is powered off. Plenty of embedded stuff I've seen too often
>> corrupts memory at power off, and of course the device manufacturers
>> blame the memory manufacturers...
>>
>> So, how do you avoid this kind of corruption in your designs?
>>
>> Thanks,
>> Best Regards, Dave
> 
> It is not necessary to do this.  Here's a very simple way to recover with
> no data loss ever.  It is simply "log-only" or "journal-only" storage.
> 
> The device keeps 10% capacity reserved (pre-erased) (or less, this is a
> performance number).  The driver has enough RAM to track where every logical
> block is (this isn't much RAM).  And then writing is always done as a
> logfile--write whatever the user data is starting at block 0, and move up.
> After writing user data, write a log entry (this can be just right after the
> user data).  Writes never go to the block the user indicates, they always go
> only to the log pointer.  Reading looks up where that block's latest copy is,
> and reads that.
> 
> When you're at less than 10% space left (near the end of the device for
> first-pass writes), you simply read from +10% from the current log write
> position, and write it to the log (compacting it to avoid re-writing any
> blocks which are now obsolete) until you free up space.  Since many blocks are
> rewritten, when we compact them we free space.  Once copied, then erase the
> blocks which were compacted.  Then just keep growing the "log" through the
> newly erased blocks.
> 
> Upon power up, scan entire device for the log info, and rebuild the RAM index.
> 
> Power can be removed at any time, and fully recovered, although obviously the
> most recent data may be lost, but we can roll back to the last consistent
> state just like journaled file systems (the log entries need a checksum so
> we can validate them).  Wear leveling is achieved by design.
> 
> There are lots of small ways to improve the above, so do that.
> 
> I have no idea what actual devices do, I suspect it's something much more
> complex and less robust.

You're missing the point/role of the internal controller in the mix.

Imagine 8 blocks of memory in the device.  Blocks 1, 2 and 3 (no special
significance) contain highly static code/data.  They get written *once*
when the device is manufactured.  So, each has seen *1* program/erase/write
cycle.

Meanwhile, blocks 4-8 are constantly hammered on.  The code is constantly
updating the "files" that occupy those blocks (the files don't TOUCH the
first 3 blocks, by definition).

After 1,000 updates, blocks 4-8 have seen 1,000 program/erase/write cycles.

The underlying FLASH device will have a MAXCYCLES specified, based on the
technology used (SLC, MLC, TLC, QLC, etc.), process geometry, etc.
For consumer devices, this number tends to be lower -- because consumers
tend to want to purchase "big" instead of "durable" (so, MLC/TLC/etc.
technology).

Assume the MAXCYCLES is 1,002 (actual numbers are unimportant -- if Dave feels
he will be taxing the medium over the life of his design).

So, after two more updates, the device is *broken*.  Because blocks 4-8 will
each have hit their 1,002 cycle limit and stopped working (yeah, I know it's
not a brick wall; but there is *some* number at which ECC errors prove
unmanageable and the controller marks those blocks (4-8) as bad.

Meanwhile, 1,2 and 3 each have 1,001 cycles of wear available -- that they
aren't (and WON'T!) be using.  Had those 3,003 cycles been "shared" among
all of the blocks, then the files being stored in 4-8 would still be
writable (even though the next write might be into blocks 1, 3, 5, 6 and 8).

So, the controller has to deliberately *move* a copy of the data in blocks
1, 2 and/or 3 into 4-8 -- consuming one cycle of 4-8.  But, freeing up
1001 cycles in 1, 2, 3!

You, on the outside, can't tell how it is doing this, when it is doing this
or even *if* it is doing this!  So, you can't predict what parts of the flash
will be corrupted as power goes away.  Maybe the VTOC gets hosed (so you can't
even FIND the files).  Or, some bookkeeping metadata used by the controller.

The logging idea works for magnetic media where what's there, stays there,
and all you have to worry about is whether or not the *new* stuff made it in
"under the wire", or not.

As an analogy, next time your RAID array is rebuilding, cycle power.
See how easily it sorts out WHERE it was, in the process (and think
about the resources that it spent to make that possible -- all in a $5
thumb drive??)