Checking large numbers of files

On 10/28/2016 3:50 AM, kalvin.news@gmail.com wrote:
> For a very crude check you can check that the file size hasn't changed in
> addition to file time stamp.

"File size" just tells me the *stored* metadata hasn't "been altered"
for the file.  There is still the possibility that the actual size
*has*, effectively, changed (think: access errors, sparse file support,
etc.).  Knowing the object's size, with certainty, still requires its
more thorough examination.

I am leary of using timestamps as they are too easily altered.  E.g.,
I've seen build scripts that deliberately, unconditionally, touch(1)
some "marker file" just to effectively record their actions -- despite
the contents of the file being unaltered.  (e.g., consider a file
named "MostRecentBuildTime" -- the timestamp of which actually
contains the information the file's presence is intended to convey).

As I am looking for information that can help me INTELLIGENTLY decide
where more careful scrutiny should be INITIALLY focused, I'd ideally
like things that can't be as easily altered *without* indicating real
potential changes to the object's content.  E.g., altering the file's
size can't be accomplished without actually altering the file's content!

[However, if I am only using the timestamp to tell if *this* file has
changed from *its* previous state (i.e., not using it to tell if this
file is DIFFERENT/changed from the original from which it was copied),
it potentially offers *some* information.]

> Then, you could possibly compute a fast hash
> for each file and use it as the first line of verification instead of using
> more expensive hash algorithms. If you just need to check if the file has
> been altered a simple 64-bit CRC might be usable.

I've benchmarked a variety of different hashes to get a feel for how
expensive they are.  There is a HUGE range of performance -- like
2 orders of magnitude (neglecting hardware accelerators)!

But, the real pissers are the access and transport delays for the raw file
data; getting it off the media and *to* the "process" that is actually
examining it!  In many cases, the volume is an external drive (which can
mean the interface is the bottleneck), a "consumer" drive (intended to be
inexpensive, but not very "performant"), uses an old interface (e.g.,
USB2) or is inherently slow (e.g., optical media).

So, looking at the file contents is the *last* thing I'd want to do
to get a feel for the file's integrity -- it is the FINAL assessment!
Ideally, I want indicators of "bad data" that are far less expensive
(e.g., if a file SHOULD be 16KB and it stat(2)'s as 30GB, I sure as
hell don't want to waste time hashing 30 billion bytes before deciding
that the file (or its metadata) has been corrupted!

> You should of course
> evaluate the implications of using hash algorithms that can produce
> collisions and that are not considered safe and robust.

Yes.  But, the hash isn't trying to protect against an active adversary
attempting to deliberately manipulate the files contents in a selective
manner to preserve the original hash.  Instead, I just have to evaluate
likely media failure modes that could potentially alter the data in ways
that would NOT be detected by the hash.

> Some file systems
> provide hooks that can monitor file alterations. Creating an observer which
> will monitor any file changes, you will be notified if any file should be
> altered.

This requires the filesystem/media to be visible to such an observer at all
times.  That would force all the media into a live repository, of sorts
(regardless of whether it is centralized or distributed).

And, it doesn't address the potential for bitrot.

I'm looking for something where the media can be offline and/or potentially
manipulated "unsupervised" (and, while not *maliciously*, perhaps prone
to missteps).  I also have to tolerate R/O media as well as a prohibition
against adding "breadcrumbs" to the monitored media (i.e., no "control files"
littering the monitored media).

[E.g., to take mass-produced optical media and verify *it* is intact
as well as R/W media for which the client does not have write/create
privileges -- or that may have write-protect tabs]

perjantai 28. lokakuuta 2016 20.50.03 UTC+3 Don Y kirjoitti:
> > Some file systems
> > provide hooks that can monitor file alterations. Creating an observer which
> > will monitor any file changes, you will be notified if any file should be
> > altered.
> 
> This requires the filesystem/media to be visible to such an observer at all
> times.  That would force all the media into a live repository, of sorts
> (regardless of whether it is centralized or distributed).
> 
> And, it doesn't address the potential for bitrot.
> 
> I'm looking for something where the media can be offline and/or potentially
> manipulated "unsupervised" (and, while not *maliciously*, perhaps prone
> to missteps).  I also have to tolerate R/O media as well as a prohibition
> against adding "breadcrumbs" to the monitored media (i.e., no "control files"
> littering the monitored media).
> 
> [E.g., to take mass-produced optical media and verify *it* is intact
> as well as R/W media for which the client does not have write/create
> privileges -- or that may have write-protect tabs]

Disclaimer: I am not an expert on the data storage, but here are my thoughts:

- Use a RAID system which has enough redundancy to detect and correct defects in the media. Depending of the RAID level used, the capability to data recovery varies.

- Duplicate or mirror the data across different physical media.

- Use a file system which provides integrity check and error correction. The ZFS is one example of such file system https://en.wikipedia.org/wiki/ZFS The ZFS provides also a snapshot feature.

On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
> Disclaimer: I am not an expert on the data storage, but here are my
> thoughts:
>
> - Use a RAID system which has enough redundancy to detect and correct
> defects in the media. Depending of the RAID level used, the capability to
> data recovery varies.

I started out using RAID5 "hardware" arrays (and other, more sophisticated
configurations).  This requires you to have multiple spindles running
concurrently (else the array can't *detect* discrepancies).  Compact
arrays also tend to suffer from cooling problems.

It also ties you to an often proprietary (or, at least, poorly documented)
implementation ("volume format").  So, a failure of the hardware can leave
you with possibly intact volume(s) and no straightforward means of
accessing their contents.

Moving volumes between arrays is seldom supported (unless you have identical
arrays -- hardware/software -- AND a well considered means of doing this.
This effectively precludes having volumes off-line.  Introducing a "foreign"
volume to an array will often result in the array prompting you to
"initialize" it (wipe it clean!).

> - Duplicate or mirror the data across different physical media.

This is RAID1.  Many of the above issues still apply.  In any case,
the mirrors must be spinning concurrently for the array to function.
(It has no way of knowing what the "other" copy of the "file"
looks like unless it can *see* it, now, in its entirety).

This begs two crucial questions:
- are BOTH copies of the file examined when you try to access it?
   Or, is a "primary" copy accessed and the "secondary" copy only accessed
   if the primary throws a read error?
- If a discrepancy is found between the TWO instances, how is the
   "winner" selected?

Plus, the more subtle question seldom considered:
- are files that are not "manually" accessed EVER verified?  Or, do you
   have to wait until you need/want a file to discover if it is intact??
   (Do *you* periodically check the contents of the batteries in your
   flashlight to verify they are functioning BEFORE you need them?
   Does the *flashlight* take on the responsibility of performing this
   test on your behalf??)

Consider the above when you have many terabytes of data at stake.
Do you want ALL of it "spinning" whenever you need to access *any*
of it?  How long will it take for a proactive array to examine/verify
ALL of the data just to reassure you?  (will you have the ENTIRE
array "up" for that length of time?  Or, will you power it up just
long enough to access the file(s) of interest, then power it back down,
again?  Or, will you power up ONLY the portion of the array known to
contain the file(s) you seek?)

As I said, I've been down this route.  Ever hear what twenty to
fifty 10-15K drives sound like spinning at the same time?
<http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>. 
  Even a 12-14 drive "shelf" is loud:
<http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>. 
  Remember,
they're all throwing off heat in a relatively dense package so you
have lots of (redundant!) fans in place to keep them from melting...

The "problem" with these solutions is they are designed for high performance.
Serving files continuously, typically, to an "organization".  And, with a
high degree of reliability/availability.

For an *archive*, you tend to need lots of capacity but comparably little
bandwidth!  (how *often* will you access its contents?  how quickly will
you need the file "delivered" to you?)  And, you can trade reliability and
availability for bandwidth and cost.  Do you keep your 2013 tax returns
in your top left desk drawer (where they are easily accessible)?  Or, have
you moth-balled them to a more capacious site that's, perhaps, a bit more
difficult to access??

[E.g., the six spindles in my P4 repository are each considerably more
expensive, reliable and performant than any *one* "consumer" spindle
in my "archive".  OTOH, I can replace one of the consumer drives
*today* -- with <whatever> the sweet spot in that market happens to
be at this moment; replacing one of the 15K SAS drives is going to require
me to have it "on hand" if I want to replace it, today!]

> - Use a file system which provides integrity check and error correction. The
> ZFS is one example of such file system https://en.wikipedia.org/wiki/ZFS The
> ZFS provides also a snapshot feature.

ZFS brings other baggage in much the same way as RAID.  The volumes in a
particular pool tend to need to remain in that pool (without explicit
maintenance).  They are inaccessible OUTSIDE that context.

By contrast, extending the archive more along the lines of JBOD allows
each volume to be treated and accessed individually and independantly.
You can move a drive from one JBOD (which, in the degenerate case,
could just be a drive bay in a PC!) to another without having to
do anything magical in the volume manager.  (e.g., plugging or unplugging
an external USB/Firewire drive!)

But, "something" has to know how to keep track of "what is where", *now*.
A physical location (bay 3 on shelf 2) doesn't mean anything.  Even the
associated logical location (/dev/c3t1d0s2) is ambiguous as any physical
volume may happen to occupy it, currently.

[I.e., you need to be able to uniquely identify *volumes* and map them
to physical and logical locations, AS THEY APPEAR]

My "distributed" approach lets me use whatever filesystem(s) are
appropriate to the needs as *I* perceive them.  E.g., I can have a
RAID50 "volume" whose contents I track, verify and scrub alongside a
single FAT32 volume.  Or, a CD9660!  Or (several) ZFS pools!  For
example, I've been systematically moving files from NTFS media
(cuz it was easy to DIRECTLY acccess from Wintel boxes) to FFS
volumes (which are better suited to my diskless NAS appliances).
The rest of my "system" doesn't care about the change.

The volumes may be sitting idle, in active use or offline in a storage
closet (try powering down a single drive in a RAID array/ZFS pool!)

It decouples the implementation from the performance AND reliability
aspects.  E.g., one volume may reside on a 15K drive while a "backup"
of that volume may reside on an optical medium stored in a Tyvek envelope
in a desk drawer!  There may be a dozen instances of one particular file
on one or more physical volumes/media types; and a *single* instance of
some other file in my entire collection!  (i.e., I can tell if that
single instance gets corrupted -- but can do nothing to recreate it
if that occurs).

This approach is ideal for my use.  And, easily extensible to handle
the addition of other media (e.g., I can add the contents of an SMB
share to the dataset so that those portions of its contents that are
of interest to me are automatically accessible and verifiable whenever
the system "notices" that SMB share being "on-line" -- even if it,
itself, is an external/relocatable device that was last seen on
some *other* mountpoint/share!)

The problem is quickly finding *potential* problems -- without
exposing the underbelly of each filesystem, raw device, media
type, etc. (e.g., I don't care if the files reside on a WORM
drive, hard disk, ORB, etc. -- just the names of the objects
and their "containers")

[I could have included support for tape volumes but have been
aggressively discarding tape archives -- copy them to disk
and archive them there.  A pair of 4-5TB drives takes up far
less space (and $$) than a stack of LTO or DLT tapes! (well,
an LTO-7 would win on size -- but just barely]

On 30/10/16 07:15, Don Y wrote:
> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>> Disclaimer: I am not an expert on the data storage, but here are my
>> thoughts:
>> - Use a RAID system which has enough redundancy to detect and correct
> I started out using RAID5 "hardware" arrays (and other, more sophisticated
> configurations).  This requires you to have multiple spindles running
> concurrently (else the array can't *detect* discrepancies).  Compact
> arrays also tend to suffer from cooling problems.
>
> It also ties you to an often proprietary (or, at least, poorly documented)
> implementation ("volume format").  So, a failure of the hardware can leave
> you with possibly intact volume(s) and no straightforward means of
> accessing their contents.

No. Almost all commercial RAID systems (Synology, Qnap, etc)
use Linux RAID support. If the hardware fails, you can mount
the drives on any Linux system that has enough SATA ports.

> Moving volumes between arrays is seldom supported (unless you have
> identical arrays -- hardware/software

No, not true, and for the same reason. E.g. if you have a
Synology DS1815+ with two four-drive RAID5 volumes in it, you
can pull the four drives of a volume (without even powering off)
and plug them into a 4-bay Qnap system. You need to configure
the volume so it gets mounted, but because it's just a Linux
RAID volume, it just works.

<snip lots more un-educated speculation>

Clifford Heath.

On 10/29/2016 3:47 PM, Clifford Heath wrote:
> On 30/10/16 07:15, Don Y wrote:
>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>> Disclaimer: I am not an expert on the data storage, but here are my
>>> thoughts:
>>> - Use a RAID system which has enough redundancy to detect and correct
>> I started out using RAID5 "hardware" arrays (and other, more sophisticated
>> configurations).  This requires you to have multiple spindles running
>> concurrently (else the array can't *detect* discrepancies).  Compact
>> arrays also tend to suffer from cooling problems.
>>
>> It also ties you to an often proprietary (or, at least, poorly documented)
>> implementation ("volume format").  So, a failure of the hardware can leave
>> you with possibly intact volume(s) and no straightforward means of
>> accessing their contents.
>
> No. Almost all commercial RAID systems (Synology, Qnap, etc)
> use Linux RAID support. If the hardware fails, you can mount
> the drives on any Linux system that has enough SATA ports.

Really?  So my Sun RAID systems run Linux??

>> Moving volumes between arrays is seldom supported (unless you have
>> identical arrays -- hardware/software
>
> No, not true, and for the same reason. E.g. if you have a
> Synology DS1815+ with two four-drive RAID5 volumes in it, you
> can pull the four drives of a volume (without even powering off)
> and plug them into a 4-bay Qnap system. You need to configure
> the volume so it gets mounted, but because it's just a Linux
> RAID volume, it just works.

Yeah, and when I take that drive out of my Sun RAID array and
put it in that Synology box it's going to work just fine!

> <snip lots more un-educated speculation>

Seems like *you're* the one speaking from limited experience!

But, then again, you've probably never owned any "big iron".
When was the last time you had more than 4 (or 12) spindles
running on the same box?  24?  48?

On 10/29/2016 4:19 PM, Don Y wrote:
> On 10/29/2016 3:47 PM, Clifford Heath wrote:

>> <snip lots more un-educated speculation>
>
> Seems like *you're* the one speaking from limited experience!
>
> But, then again, you've probably never owned any "big iron".
> When was the last time you had more than 4 (or 12) spindles
> running on the same box?  24?  48?

Ah, forgive me.  Obviously, you expected me to NOT use any of these
technologies until they were available in FOSS forms (for folks with
deep pockets and short arms!).

Instead, I should have kept all of this data on scraps of paper
in a series of large shoeboxes -- until I could use some "free"
medium/technology to store it.

[Hint:  I had a pair of D1000's arrays stuffed with 4G drives in
the late 90's -- ~96GB.  Then, upgraded the drives to 9G, 18G
and 36G before, eventually, abandoning the "large number of hot
spindles approach".  Where was your Synology box, then?
    "Synology Inc. was founded in January 2000, ..."
No problem!  We'll just look at Qnap:
    "Quality Network Appliance Provider was founded in 2004..."
Hmmm, I guess I'd still have to keep all those little slips of
paper lying around in shoeboxes waiting for more affordable means
of archiving the data!]

[I wonder when Linux support for "software RAID" was viable??]

(sigh)  Amusing how SHALLOW the experiences of many folk!
And, how *deep* and authoritative they THINK them to be!

Makes you wonder what sort of "education" they're hiding behind
(that they think others of us are LACKING!)

Do your homework before spouting off your pseudo-authoritative
drivel.

On 30/10/16 10:19, Don Y wrote:
> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>> On 30/10/16 07:15, Don Y wrote:
>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>> thoughts:
>>>> - Use a RAID system which has enough redundancy to detect and correct
>>> I started out using RAID5 "hardware" arrays (and other, more
>>> sophisticated
>>> configurations).  This requires you to have multiple spindles running
>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>> arrays also tend to suffer from cooling problems.
>>>
>>> It also ties you to an often proprietary (or, at least, poorly
>>> documented)
>>> implementation ("volume format").  So, a failure of the hardware can
>>> leave
>>> you with possibly intact volume(s) and no straightforward means of
>>> accessing their contents.
>>
>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>> use Linux RAID support. If the hardware fails, you can mount
>> the drives on any Linux system that has enough SATA ports.
>
> Really?  So my Sun RAID systems run Linux??

Ahh, my apologies. I didn't realize you were running a computer antiques 
museum. I thought you were trying to ensure the availability of some data.

On 30.10.2016 г. 03:00, Clifford Heath wrote:
> On 30/10/16 10:19, Don Y wrote:
>> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>>> On 30/10/16 07:15, Don Y wrote:
>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>>> thoughts:
>>>>> - Use a RAID system which has enough redundancy to detect and correct
>>>> I started out using RAID5 "hardware" arrays (and other, more
>>>> sophisticated
>>>> configurations).  This requires you to have multiple spindles running
>>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>>> arrays also tend to suffer from cooling problems.
>>>>
>>>> It also ties you to an often proprietary (or, at least, poorly
>>>> documented)
>>>> implementation ("volume format").  So, a failure of the hardware can
>>>> leave
>>>> you with possibly intact volume(s) and no straightforward means of
>>>> accessing their contents.
>>>
>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>> use Linux RAID support. If the hardware fails, you can mount
>>> the drives on any Linux system that has enough SATA ports.
>>
>> Really?  So my Sun RAID systems run Linux??
>
> Ahh, my apologies. I didn't realize you were running a computer antiques
> museum. I thought you were trying to ensure the availability of some data.

So what is he supposed to do with data which are useless outside of
a particular environment. Apparently he has plenty of past projects
spanning a few decades back to maintain - how do you suggest he will
separate movable from non-movable data. Then the sheer amount of data
he is talking is just huge.
Linux/foss may be a huge part of life but it is restricted to the needs
of the general public which feeds it directly. There are other aspects
of tech life which are simply beyond the scope of foss/linux, beyond
its capabilities as they are.
I strongly suspect Don is not an office secretary running an office 
archive and I also strongly suspect he knows what he is doing.
Well I also suspect he knows he will find not what he is looking
for - an easier way to navigate his huge archives - but there is nothing
wrong wanting to talk or just moan about that, this is what groups are
for really.

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/

On 30/10/16 13:05, Dimiter_Popoff wrote:
> On 30.10.2016 г. 03:00, Clifford Heath wrote:
>> On 30/10/16 10:19, Don Y wrote:
>>> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>>>> On 30/10/16 07:15, Don Y wrote:
>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>>>> thoughts:
>>>>>> - Use a RAID system which has enough redundancy to detect and correct
>>>>> I started out using RAID5 "hardware" arrays (and other, more
>>>>> sophisticated
>>>>> configurations).  This requires you to have multiple spindles running
>>>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>>>> arrays also tend to suffer from cooling problems.
>>>>>
>>>>> It also ties you to an often proprietary (or, at least, poorly
>>>>> documented)
>>>>> implementation ("volume format").  So, a failure of the hardware can
>>>>> leave
>>>>> you with possibly intact volume(s) and no straightforward means of
>>>>> accessing their contents.
>>>>
>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>>> use Linux RAID support. If the hardware fails, you can mount
>>>> the drives on any Linux system that has enough SATA ports.
>>>
>>> Really?  So my Sun RAID systems run Linux??
>>
>> Ahh, my apologies. I didn't realize you were running a computer antiques
>> museum. I thought you were trying to ensure the availability of some
>> data.
>
> So what is he supposed to do with data which are useless outside of
> a particular environment. Apparently he has plenty of past projects
> spanning a few decades back to maintain - how do you suggest he will
> separate movable from non-movable data. Then the sheer amount of data
> he is talking is just huge.

No, his amount of data is trivial compared to the petabytes
being accumulated daily and managed by internet-scale companies,
who almost all use *commodity* hardware running *open source*
software.

> I strongly suspect Don is not an office secretary running an office
> archive and I also strongly suspect he knows what he is doing.

Don is a "special flower". He faces problems that no-one else
has ever faced, and he analyses them with an insight that no-one
else has ever had, requiring solutions that no-one else has ever
found.

None of these descriptions are true, of course.

For what it's worth, data management is a global profession with
a hundred major specialties and a thousand vendors, who all have
experts with more experience than Don has in his toenails. I've
just returned from a paid global speaking trip where I addressed
some of them. "Availability" isn't my specialty, I just rely on
commodity equipment to manage my measly few dozen terabytes. But
what do I know, I don't work with special problems like Don's.

On 30.10.2016 г. 05:06, Clifford Heath wrote:
> On 30/10/16 13:05, Dimiter_Popoff wrote:
>> On 30.10.2016 г. 03:00, Clifford Heath wrote:
>>> On 30/10/16 10:19, Don Y wrote:
>>>> On 10/29/2016 3:47 PM, Clifford Heath wrote:
>>>>> On 30/10/16 07:15, Don Y wrote:
>>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>>>>>>> Disclaimer: I am not an expert on the data storage, but here are my
>>>>>>> thoughts:
>>>>>>> - Use a RAID system which has enough redundancy to detect and
>>>>>>> correct
>>>>>> I started out using RAID5 "hardware" arrays (and other, more
>>>>>> sophisticated
>>>>>> configurations).  This requires you to have multiple spindles running
>>>>>> concurrently (else the array can't *detect* discrepancies).  Compact
>>>>>> arrays also tend to suffer from cooling problems.
>>>>>>
>>>>>> It also ties you to an often proprietary (or, at least, poorly
>>>>>> documented)
>>>>>> implementation ("volume format").  So, a failure of the hardware can
>>>>>> leave
>>>>>> you with possibly intact volume(s) and no straightforward means of
>>>>>> accessing their contents.
>>>>>
>>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>>>> use Linux RAID support. If the hardware fails, you can mount
>>>>> the drives on any Linux system that has enough SATA ports.
>>>>
>>>> Really?  So my Sun RAID systems run Linux??
>>>
>>> Ahh, my apologies. I didn't realize you were running a computer antiques
>>> museum. I thought you were trying to ensure the availability of some
>>> data.
>>
>> So what is he supposed to do with data which are useless outside of
>> a particular environment. Apparently he has plenty of past projects
>> spanning a few decades back to maintain - how do you suggest he will
>> separate movable from non-movable data. Then the sheer amount of data
>> he is talking is just huge.
>
> No, his amount of data is trivial compared to the petabytes
> being accumulated daily and managed by internet-scale companies,
> who almost all use *commodity* hardware running *open source*
> software.
>
>> I strongly suspect Don is not an office secretary running an office
>> archive and I also strongly suspect he knows what he is doing.
>
> Don is a "special flower". He faces problems that no-one else
> has ever faced, and he analyses them with an insight that no-one
> else has ever had, requiring solutions that no-one else has ever
> found.
>
> None of these descriptions are true, of course.
>
> For what it's worth, data management is a global profession with
> a hundred major specialties and a thousand vendors, who all have
> experts with more experience than Don has in his toenails. I've
> just returned from a paid global speaking trip where I addressed
> some of them. "Availability" isn't my specialty, I just rely on
> commodity equipment to manage my measly few dozen terabytes. But
> what do I know, I don't work with special problems like Don's.


Well,

 >> Linux/foss may be a huge part of life but it is restricted to the
 >> needs of the general public which feeds it directly. There are
 >> other aspects of tech life which are simply beyond the scope of
 >> foss/linux, beyond its capabilities as they are.

But I know many of the foss/linux crowd think otherwise - which is
OK, everyone sees the walls of their own room. No need to be grumpy
about other people wanting to talk about other things though.

Dimiter