USB memory sticks for root file system - experiences

I am not sure whether folks in this group would count a device such as
the pogoplug
as an embedded device, but it applies also to many development
boards.

I am experimenting for some time with two pogoplugs running linux,
booting arch linux from an USB memory stick.
It seems that particular the cheap supermarket memory sticks do not
last long.
In one case, within a week the USB memory stick got unusable, it
cannot even be formatted anymore.  The last thing I did to it - after
it showed problems with ext2 - was formatting it back to vfat and
running a capacity checker under windows.

Another stick showed frequently file system corruptions.
Now a linux box thinks the file system is clean, but uBoot does not
recognize the partition table.

The two failing USB stickes were emtec and disk2go.

A colleague recommended Verbatim sticks, I tried one on a development
board and that worked so far.
I will test my pogoplug with a new Verbatim stick.

Any other recommendations?

Andreas

On 11/14/2011 11:33 AM, acd wrote:
> I am not sure whether folks in this group would count a device such as
> the pogoplug
> as an embedded device, but it applies also to many development
> boards.
>
> I am experimenting for some time with two pogoplugs running linux,
> booting arch linux from an USB memory stick.
> It seems that particular the cheap supermarket memory sticks do not
> last long.
> In one case, within a week the USB memory stick got unusable, it
> cannot even be formatted anymore.  The last thing I did to it - after
> it showed problems with ext2 - was formatting it back to vfat and
> running a capacity checker under windows.
>

Memory devices (both USB, SD card and other types) have limitations on 
the number of times blocks can be erased. The problem is that the 
minimum erase size can be pretty large, so that making a small 
modification to a file, or to an inode, can result in a large block of 
flash that needs to be erased.

These problems get worse if you modify structures that sit on the 
boundary of two erase blocks, because it requires erasing both of them.

To make these devices last longer, manufacturers typically have a number 
of blocks at the beginning of the device that have been optimized for 
frequent access, so they are well suited to FAT tables, and such.

The position of those special blocks is not consistent among vendors, 
but since these devices come pre-formatted, it is best to leave that 
formatting alone, assuming that the manufacturer already matched the 
filesystem to the card, as best as possible. Also, the FAT filesystem 
itself is a good match for the card, since the FAT updates are 
concentrated in local tables, and not spread out all over the disk like 
ext2 inodes. So, it's best if you write your application so it can use 
FAT filesystem. If you must reformat the card with ext2, at least use 
the original partition table (even if it wastes 4M at the beginning like 
some cards do).

Also, to reduce writes, make sure you set the 'noatime' option in your 
mount command, otherwise the inode is rewritten even if you just read a 
file.

Arlet Ottens  wrote:

>... Also, the FAT filesystem 
>itself is a good match for the card, since the FAT updates are 
>concentrated in local tables, and not spread out all over the disk like 
>ext2 inodes.

I would expect the device's built-in wear leveling algorithm to
actually spread write operations all over the "disk", even if the same
logical blocks are rewritten.

>Also, to reduce writes, make sure you set the 'noatime' option in your 
>mount command, otherwise the inode is rewritten even if you just read a 
>file.

For the same reason, relocate /tmp and /var/log to a RAM disk.

--
Roberto Waltman

[ Please reply to the group,
  return address is invalid ]

acd wrote:

> I am not sure whether folks in this group would count a device such as
> the pogoplug
> as an embedded device, but it applies also to many development
> boards.
> 
> I am experimenting for some time with two pogoplugs running linux,
> booting arch linux from an USB memory stick.
> It seems that particular the cheap supermarket memory sticks do not
> last long.

Make sure you don't have accidental power outages, especially when 
writing to device. That could result in all kinds of problems from file 
system corruption to flash hardware failures.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

Roberto Waltman wrote:


> I would expect the device's built-in wear leveling algorithm to
> actually spread write operations all over the "disk", even if the same
> logical blocks are rewritten.

Yes, and those algorithms are beyond your control. If something breaks 
there, the flash device is dead and there is no way to recover.

VLV

On 11/14/2011 03:24 PM, Roberto Waltman wrote:
> Arlet Ottens  wrote:
>
>> ... Also, the FAT filesystem
>> itself is a good match for the card, since the FAT updates are
>> concentrated in local tables, and not spread out all over the disk like
>> ext2 inodes.
>
> I would expect the device's built-in wear leveling algorithm to
> actually spread write operations all over the "disk", even if the same
> logical blocks are rewritten.

Yes, it will do that. But if you have a small inode in the middle of a 4 
MB erase block, and you update that inode (containing file size, 
allocation blocks, and access/update times), it will still have to erase 
an entire 4MB block. Even with wear levelling, it can add up quickly, 
especially because the inodes are spread over the disk, instead of 
combined in the same erase block.

The advantage of the FAT filesystem is that all FAT entries are grouped 
together, and that the device firmware 'knows' where the FAT is, and it 
has been optimized to handle frequent changes in that particular area.

On 14 Nov., 15:46, Vladimir Vassilevsky <nos...@nowhere.com> wrote:
> Make sure you don't have accidental power outages, especially when
> writing to device. That could result in all kinds of problems from file
> system corruption to flash hardware failures.
>

The problem is that I could sometimes not reach the device (in the
case of the Pogoplug) through network anymore, so I have no choice but
just pull the power plug.
I will check all the other tips (I used the linux installation
described on
http://archlinuxarm.org), but I still think that ext2/ext3 is better
than vfat considering the problems I see. Not to mention that uBoot
seems not to support vfat.

Andreas

On 11/14/2011 04:33 PM, acd wrote:
> On 14 Nov., 15:46, Vladimir Vassilevsky<nos...@nowhere.com>  wrote:
>> Make sure you don't have accidental power outages, especially when
>> writing to device. That could result in all kinds of problems from file
>> system corruption to flash hardware failures.
>>
>
> The problem is that I could sometimes not reach the device (in the
> case of the Pogoplug) through network anymore, so I have no choice but
> just pull the power plug.
> I will check all the other tips (I used the linux installation
> described on
> http://archlinuxarm.org), but I still think that ext2/ext3 is better
> than vfat considering the problems I see. Not to mention that uBoot
> seems not to support vfat.

You may be able to use other options, such as using two partitions. One 
VFAT, in the same place as the old FAT partition, with the same 
properties, and another one ext2/ext3. If you use the ext2 mostly for 
read-only stuff, it won't wear the flash.

Hi Andreas,

On 11/14/2011 3:33 AM, acd wrote:

> I am experimenting for some time with two pogoplugs running linux,
> booting arch linux from an USB memory stick.
> It seems that particular the cheap supermarket memory sticks do not
> last long.
> In one case, within a week the USB memory stick got unusable, it
> cannot even be formatted anymore.  The last thing I did to it - after
> it showed problems with ext2 - was formatting it back to vfat and
> running a capacity checker under windows.
>
> Another stick showed frequently file system corruptions.
> Now a linux box thinks the file system is clean, but uBoot does not
> recognize the partition table.
>
> The two failing USB stickes were emtec and disk2go.
>
> A colleague recommended Verbatim sticks, I tried one on a development
> board and that worked so far.
> I will test my pogoplug with a new Verbatim stick.
>
> Any other recommendations?

Does your system write to any part of the root filesystem?
(e.g., /var/log et al.).  Short test: mount root a RO
and see *what* is writing to that filesystem (and where!)
and either stop the writes (or reduce their frequency by
orders of magnitude.... e.g., if you are logging at .info
level you're just  being wasteful) *or* redirect them to
some other "volatile" (e.g., memfs) file system that can
tolerate the abuse.

(This will probably also give you some small performance
gains)

(Hint:  make the root file system only large enough to
cover the "static" (non-changing) portion)

Hi Andreas,

On 11/14/2011 8:33 AM, acd wrote:
> On 14 Nov., 15:46, Vladimir Vassilevsky<nos...@nowhere.com>  wrote:
>> Make sure you don't have accidental power outages, especially when
>> writing to device. That could result in all kinds of problems from file
>> system corruption to flash hardware failures.
>
> The problem is that I could sometimes not reach the device (in the
> case of the Pogoplug) through network anymore, so I have no choice but
> just pull the power plug.

As it is possible (likely?) that the memory failure may be causing
the "inaccessibility" problem, consider instrumenting your write(2)
so that it does a write-verify instead.  Then, be sure all of your
write(2c) invocations check the result code (and do something
appropriate in the case of the *failed* write!).

This might let you discover the problem sooner (while some
functionality still remains)

> I will check all the other tips (I used the linux installation
> described on
> http://archlinuxarm.org), but I still think that ext2/ext3 is better
> than vfat considering the problems I see. Not to mention that uBoot
> seems not to support vfat.