Checking large numbers of files

>>>>>>> It also ties you to an often proprietary (or, at least, poorly
>>>>>>> documented)
>>>>>>> implementation ("volume format").  So, a failure of the hardware can
>>>>>>> leave
>>>>>>> you with possibly intact volume(s) and no straightforward means of
>>>>>>> accessing their contents.
>>>>>>
>>>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>>>>> use Linux RAID support. If the hardware fails, you can mount
>>>>>> the drives on any Linux system that has enough SATA ports.
>>>>>
>>>>> Really?  So my Sun RAID systems run Linux??
>>>>
>>>> Ahh, my apologies. I didn't realize you were running a computer antiques
>>>> museum. I thought you were trying to ensure the availability of some
>>>> data.
>>>
>>> So what is he supposed to do with data which are useless outside of
>>> a particular environment. Apparently he has plenty of past projects
>>> spanning a few decades back to maintain - how do you suggest he will
>>> separate movable from non-movable data. Then the sheer amount of data
>>> he is talking is just huge.
>>
>> No, its much simpler than that:  I am supposed to foresee the future.
>
> No, you're supposed to move with the times, and move your data to
> a system that's replaceable, not one from a company that died almost
> a decade ago. Especially if you are actually bothered with maintaining
> the integrity and availability of said data... as you claim to be.

And that's EXACTLY what I have done!  Where was your Qnap NAS when
I moved away from DEC and Sun's RAID?  Linus hadn't even started on
his opus -- let alone the Qnap founders!  Was I supposed to ANTICIPATE
their availability at some future date and find some OTHER way of
handling the data until they came along with the great and glorious
solution?  Why not wait longer for ZFS to come along?  And, simply
not generate any data in the intervening time?  What do I do with the
Sun and DEC volumes in the meantime?

Why should I have kept things on RAID5 instead of RAID1?   Why RAID1
instead of "manually mirroring" drive contents between media?  Its
an ARCHIVE, not a live file store!  The only "performance" criteria
is how well it REMEMBERS, not how "real-time accessible" it is!

What should I pick to safeguard against NEXT year's obsolescence?

I started out using (9T) tape as tertiary storage; to extend my
disk capacity beyond the tiny ~50MB drives of the time.  I could
put more than that on a reel of tape and just load whatever
applications I needed onto the (smallish) disk.  Buy a box of Blackwatch
and I've got a dozen or more "disks" of (admittedly slow) storage
for far less $$ than buying another disk or two!

This works great in the Land of DOS, SysV and even early Windows (3.1).
Things tend not to be too heavily intertwined.  And, if your work ethic
doesn't have you frittering around from one set of applications to a
completely different set DAILY, its like moving to a dedicated
workstation based on the CURRENT needs.  (You've probably got to design
the hardware before you layout the board; and, do both of those before
you're writing and debugging code ON that hardware)

OTOH, really frustrating if you've just wiped the CAD tools in favor
of the software development tools -- only to discover you've one more
"little thing" to attend to with those CAD tools!  :<

[Can you spell "discipline"?]

Eventually, I added SCSI HBA's to my machines and one of these:
<http://www.ebay.com/itm/DEC-Compaq-DS-SWXRA-HC-StorageWorks-7000-Disk-Array-/370183381790>
stuffed with 4G SE SCSI drives (or, maybe it was 9G drives; I get
this box and the next one confused in my mind as I can't recall
where I was living with each of them).  I could access half of the
box from one computer (two HBA's per machine) and the other half
from the other computer (identical computers).  So, one set of
applications on one machine, another set on the other machine.
No "conflict" in the SCSI array.  No dicking around wiping off disks
and reloading "new" applications from tape each time I changed foxus!

Then (or, perhaps, before), I inherited a pair of Sun A1000's:
<http://computermuseum.wiwi.hu-berlin.de/small/server2_02.jpg>
Much smaller and less capacity.  Different type of drives:  SCA instead
of SE50.  And only 12 of them per.

But, RAID5 was a PITA to deal with.  If the lights flickered (common)
or I was careless and powered down the wrong computer at the wrong time
("Let's see, is this box connected to THIS computer or THAT one?")
I'd end up having to wait for a lengthy verification phase.  I'm not
getting work done when I'm waiting ANXIOUSLY for a RAID array to rebuild!

[Tiny drives, but also SLOWER drives -- by today's standards -- and
controllers]

And, with drives at $1K each, it was annoying to be "throwing away"
~1/3 of those drives.  (for a while, every drive was in the ~$1K
ballpark; move to 9G drives and the price was the same as 4G had been
previously, etc.)

[I may have these remembered in the wrong chronological order]

So, I went to RAID1 and picked up some capacity without spending
any extra money.  (these would be termed "hardware RAID" as it
was implemented *in* the disk box, not in the host(s))

The DEC box is huge.  And, *both* boxes are unsightly and power hungry
(no need for supplemental heat in the winter!).  And, they took
different types of drives!  So, everything purchased for the DEC
box would ONLY work with the DEC box; similarly for the Sun box.

[This effectively made half of the drives obsolescent, moving forward;
choose:  which interface technology is most future-safe?]

Eventually, I discarded all but one of the DEC "shelf"s (7 drives)
and just kept the (plastic) "disk modules" in a supply closet,
each with its disk inside.  I could then plug/unplug these
modules at will; label each appropriately and I could add and
subtract capabilities as needed:  "Lets see, today I'll need
AutoCAD and Ventura..."

So, the "shelf" just became the equivalent of a modern USB (bus).
My FreeBSD and NetBSD boxes were tolerant of powering down the
shelf and replacing drives without rebooting so it was excellent!
DOS/Windows was a bit more finicky -- but, not exceedingly so!

I did the same thing with ONE of the A1000's.  While smaller than
the DEC box, two of them still ate up a fair bit of space and
power (and, I don't need to be able to access all of those
drives at the same time!  That's the first FLAW in a RAID approach
in my environment -- I'm only using a very small subset of the
TOTAL storage I have available, at any given instant!)

Over time, I moved from RAID1 configuration to JBOD's.  More data
online with fewer disks!  Yet, still more than I "absolutely needed".
And, I could swap drives at will without worrying about keeping
RAID1 pairs side-by-side!

But, now I had to take deliberate action to ensure the "backup"
of each disk was "in sync" with the "original" copy.  Logistically
pretty easy (pull one of the "other" drives and install the
backup drive; compare/rsync one mountpoint with that of the other;
remove backup drive module and replace on closet shelf).

This sort of "manual mirroring" is how I continued thereafter.
Disks kept getting bigger, machines kept changing, but the
strategy kept working -- as long as I could install a SCSI HBA in
the new machine, the old data remained accessible!

I played with several different "PERC"s in an attempt to bring
drives "inside" the PC.  And, RAID controllers from American
Megatrends, Adaptec, QLogic, IBM, etc.  Almost all were annoying
and difficult to use.  You never were sure what the "safe" answer
was to any error condition reported!  (they were always eager to
reinitialize volumes as the prefered remedy).  And, as I was
running several different OS's on the same hosts, lack of support
for a particular HBA on one OS would effectively render it
unusable on ANY of those OS's ("Gee, I can't access that media
from this OS -- despite the fact that the OS can recognize that
filesystem!  The entire medium is absent!").  IIRC, QLogic was
one of the most troublesome when it came to the FOSS OS's I
was running, back then.

Of course, USB made this even easier!  Now, I could deal with
individual drives instead of whole shelf's.  And, 100-200G on a
single drive!  For peanuts!  (10K and 15K SCSI drives are
STILL expensive!  Ditto for SAS, etc.)

Then I started building file servers with external SCSI disks
cuz the PC's didn't have room for that many spindles internally.
Nor power supplies with 100-200W to spare above the needs of the
(power hungry) PC!

[fond memories of a Sun LX in that role -- small, quiet].

And, now the contents were accessible to ALL machines over
the network -- instead of just the machine with the SCSI HBA!
As I wasn't looking to push huge amounts of data in short
periods of time (10Base2), a small machine could easily handle
the task.  Megabytes of RAM instead of the gigabytes of today.

Then NAS's.  LinkStation, SnapServer 1100/2200/4500, BANAS 440,
Linksys, Netgear <mumble>, etc.  Why use a "real computer" with
provisions for a keyboard, display, etc. if all you need is an
appliance?

Then, 1U servers repurposed as 4 drive NAS's (having the advantage
of allowing me to add SCSI HBA's for external tape and disk
support).

Then, back to dedicated appliances (e.g., a dedicated DBMS server,
another box to handle the "simple services" for the network:
DNS/SMTP/FTP/TFTP/xfs/etc.), etc.

And, each change/addition meant more things to keep track of.
More procedures to remember (if this device doesn't boot, drag out
a monitor and keyboard and do the following;  if this device doesn't
come online, tip(1) to it and use this for a login/password; if this
array needs to be rebuilt then follow THESE steps; etc.) and more
things to go wrong as well as distract me from the work I that I
*want* to do!  (If I wanted to be an IT professional, I'd have
set out to *become* an IT professional!)

Eventually, I rescued a dozen E140 thin clients (1GHz/1GB).
<http://www.instant-axess.co.uk/prod_images_blowup/Neoware_e1402.jpg>
Quiet.  No fan.  Relatively low power (compared to a full PC) yet
capable of supporting a single PCI card!  Stuff a SCSI HBA in
one and it is now a "SCSI device server".  Tether an external
SCSI enclosure to it and you can now access that drive as if
you had a genuine NAS (run NetBSD on the E140).  Tack on a 12
drive array (e.g., Sun 711, A1000/D1000) and you've got a hefty
chunk of disk available!  And, the e140 is the bottleneck, not
the drive array!

Or, replace the PCI card with a quad USB2 card and 4 external
USB disks.  I.e. all my existing media is still accessible!
Cash outlays have slowed, etc.  Move to cheap consumer grade
drives cuz I can now spin down any drives that aren't needed
without having to disassemble a shelf!  Move all the original
CD/DVD images onto bigger (1/2/3/4 TB drives) and hide the original
media under the bed (as the ultimate "backups" -- annoyingly tedious
to resort to using them but silly to discard them!)

Fast forward to rescue several FX160's (2GHz/4GB) -- smaller,
sexier and run even cooler!  Serve up a custom image that
gives them each the functionality they need -- without
requiring any of them to have a "system disk"  (with 4G of
RAM, its easy to ensure they never need swap; they are appliances,
not computers!  What they do today they will do tomorrow!)

But, still adopting the "manual mirror" philosophy.  Point
rsync at two drives and let it do its thing.  Or, use comparable
GUI tools under Windows.  Have to make labels for each physical
drive so I know which drive has each type of content:  "Where are
the user manuals?  I need to lookup the pinout of this industrial
scale to which I'm interfacing..."

Then, manually walk through the file system looking for a filename
that hopefully jogs my memory.  If not found, come up with a second
choice:  "Maybe its in with the research papers?"

Feed the hundreds of clipart/font/multimedia disks into the machine
to catalog their contents -- so I don't have to physically examine
each medium each time I want to see what it contains!  Copy the CD's
onto yet another big disk -- so I can just mount an ISO image instead
of having to dig through spindles of original CD's.  (but, save the
CD's under the bed as backups, like before!)

Do I do this for EVERY medium that I have?  How else will I be
able to browse through them without physically loading each,
individually?  Sure would be nice if there was some sort of INDEX
of "what was where".  And, if that index didn't require the
files to be on writeable media (silly for it to do so!).

Ah, but then certain media exist individually in some places
(e.g., *this* CD) and as parts of collections in others (e.g.,
\\CLIPART\cd's).  How to track which is which?  How to verify
each instance is true to its origins?  How to LOCATE the
original instance of a particular object found to be defective
(or damaged out of carelessness -- hard to overwrite a CD/DVD;
much easier to overwrite a copy of that object on magnetic media!)

And, the primary role of each of these appliances is to serve
files across the network.  But, other devices (hosts) also serve
files.  Why not catalog and verify THEIR exported files (subject
to the criteria that they specify in much the same way as these
other appliances) whenever they are "seen"?  As the appliances
(or SOME host!) know how to do this, there is no need for these
exporting hosts to run that same code.  So, supporting a Solaris
host, *BSD host, Windows host or even a Linux host would not
require any additional software development effort.  They
each appear through a normalized interface (NFS/CIFS).

Thus, I make the best choices that I can, based on my past experiences
with technologies and the time and money I want to throw at a problem.
I keep track of the problems I encounter and factor those into my future
plans.  E.g., unfortunate to initially choose SE50 SCSI drives; better
move would have been SCA from the start!  It would be as unfortunate
as investing in PATA drives NOW.

But, it's MY experience that I evaluate, not urban legends or
folks with different budgets/priorities!  Many folks have had
multiple drive failures; I've had two in 40 years (and one may
have been my fault -- a laptop drive in an unorthodox orientation
in an unvented enclosure running 24/7/365).  I've been "summoned"
at all hours of day and night to "rescue" colleagues who've
experienced a "catastrophic" failure:  "Sorry, there aren't any
stores open at this hour!  Oh, you have a dog and pony scheduled
for the early morning hours?  Hmmm... that's going to be a bit
of a problem!  Do you have your original install media and a
recent backup of your work?  I can bring over a drive and we
can spend the next few hours trying to get you back up and
running.  Will the wife grumble if I ring the doorbell at this
hour of the night?  Or, do you want to meet me at the door??"

If I listened to other folks' experiences, I wouldn't trust
disk drives.

I also wouldn't trust TAPE drives -- yet have had great experiences
with them, as well!

I burn a lot of optical media and haven't had a "laser" burn out (of
course, I have lots of drives on which to burn media so that load isn't
as concentrated as it might be for other folks).

Maybe I just buy better quality peripherals than folks who treat them
as "check off" items ("Whats the cheapest large disk you have available?")

On the advice of others, I'd take my RCS repositories and convert
them all to CVS (with a few man-years of effort on the "fixups").
Then SVN, then Hg, Git, etc.  Next week, I'll have to see what's
/en vogue/.

*Or*, I'll keep a working version of RCS available and RELY ON IT
to support those sources.  Ditto CVS, SVN, etc.  And, pmake, bmake,
nmake, etc.  There are 24 hours in a day and I sure don't want to
spend them "digging (boring) ditches".

Etc.

I can't tell The Boss to hire someone to tackle all this busywork
so I can be free to "design".  I can't even ask him to pick up some
paper for my printer ("it's YOUR printer; why do you think *I* should
be maintaining it?).  And, many of the folks I've worked for don't
have staff to take on those jobs themselves (in many cases, *I* am
the Engineering Department).  *I* am The Boss!  So, *I* have to find
the necessary labor and resources for each "requirement"!

>> Perhaps Clifford has the luxury of an IT department to fall back on.
>
> Never have had, probably never will. My clients have, but not me.

So *they* are carrying the cost of maintaining your tools in the long
haul.  Do they ever call and ask if you've got a copy of a particular
tool squirreled away -- cuz they misplaced the "original" that they
purchased for you project?  Or, maybe a copy of your sources??

[Do you tell them "No"?  Do you tell them "Yes" but bill them for
the time to fetch that and arrange for its delivery?  Or, do you
just give them a cockeyed smile and an exasperated sigh as you deliver
what they need?  You *do* realize I've been down this path!]

>> Or, maybe he's got an eidetic memory and can remember and recreate how
>> the schematics, artwork, specifications, contracts, software, test suites
>> and other materials that he used to create Project X for Client Y
>> in Year Z.
>
> Pretty much, yes. See below.

Ah, I'm jealous!  I rely on having records of everything I've done.
I only commit names/faces/birthdates/bereavement dates/etc to memory.
(I also memorize car ownership -- so I have a good idea of who
is waving when I see a silhouetted behind tinted glass!)  Having to
resort to a "cheat sheet" for those things is, IMO, "tacky".

For projects, I can remember what and why, but rarely detail.  And,
with good recordkeeping, see no need to take on that effort.  Esp
as I rarely have to revisit some project for a "fix" but, rather, to
refresh my memory as to how/why I made some particular decision
(the *decision* being the thing I remember) so I can leverage that
thought process for some other project (likely for a different
client)

E.g., I know I've a few lines of (ASM) code that will test for
BCD divisible by four (leap year).  And, a formal specification
of the leap year algorithm complete with Gregorian calendar
adjustments.  And, which project is likely to have had that
*ASM* code in it.

But, as I'm HIGHLY unlikely to implement any of those things in
ASM at this point in my career, I won't bother to track them down
*or* re-derive them (they are uninteresting problems).

OTOH, I know I have some robust algorithms for bidirectional
linear barcode decoding, keypad debouncing with N-key rollover,
etc.  And, I know how those were coded (not in ASM, not in a HLL)
so I can easily glean the nature of the algorithms if I want to
port them to another implementation.

I have to label things to know what's contained within.  The walls
of our garage are lined, floor to ceiling, with 10x6x18 inch boxes
each bearing a label:  hammers, screwdrivers, SCSI2 cables, CAT5
patch cords, speech synthesizers, access points, DDR memory, etc.

In the office, every external disk drive has a self-adhesive label
(thank you, p-touch) to provide a clue as to its contents:  system images,
FOSS OS's, research projects, MP3's, etc.  The "second copy" of each
bearing a '2' after the name.  A disk inside a machine has the machine
name to vouch for its LIKELY content.

[FWIW, there are two laminated cheat sheets in the driver side seat
back on SWMBO's vehicle enumerating the various CURRENT settings for
its electronics package.  Why should I waste time remembering these
after I've determined their APPROPRIATE settings?]

Since my first CS class, I've been of the mind of letting machines
do what they do best:  remember stuff and repetitive calculations.
I've little/no desire to clutter my meatware with trivia that
a machine can remember more reliably than me.  I'll do the things
that the machine can't easily do (e.g., physically mount a removable
medium that it wants to examine).  But, I'll let *it* do the sh*t
work!

>> I, OTOH, take a keen interest in every project I undertake.
>
> Good for you. I respect that.
>
>> The Linux camp tends to be full of zealots.  Whatever the "problem",
>> Linux (or some other FOSS product) is ALWAYS the answer!
>
> I agree, that's a problem. The FOSS world is almost full of
> software that isn't worth the price of the media that stores it.
> But when something exists that solves your problem... why
> re-invent the wheel?

I don't see anything that "exists that solves the problem".
I have no desire to reinvent ANY wheel.  OTOH, I'm not keen
on WALKING when I *could* "ride" -- but for the lack of wheels!

I see (and have used) lots of things that solve pieces of
the problem -- both FOSS and commercial offerings.  But,
like the products of most "consultants", they expect YOU to
change your way of doing things to coincide with THEIR
expectations:
    "Why would you want to do THAT?"
    "Why did you want to do it YOUR way?  How is that BETTER??"
    (why do you put onions on your hotdog??)

I approach problems, instead, as "what is the LEAST imposition I
can make on the user to facilitate his doing what he would LIKE
to do?  How can I *infer* what he wants/needs by observing what
he is doing NATURALLY?"

How can I concoct a "solution" that imposes the least on *me*
by exploiting my own knowledge of HOW *I* work and my actual
work environment?  (I have no interest in making my solution
available to "YOU"; so, no concern over how YOU work or
YOUR environment!)

>> To do that -- to be ABLE to do that -- takes a significant investment.
>> Call Apple/MS/Clifford's_Employer and ask about a product they
>> released 5 years ago...
>
> I am my own employer, yet I still occasionally get paid to assist
> with code I wrote fifteen years ago. I can still tell you which
> file and function to look (in three million lines of code)
> for a given feature - from memory.

I can't.  I only remember algorithms -- i.e., I can tell you whether
a cubic bezier will have a cusp, inflection point, be a straight
line or a simple curve... just from an examination of the control
points.  If I wanted to know *how* all of those were determined,
I'd sit down with the knowledge that they CAN be determined and
then recreate the derivation.  I wouldn't clutter my mind with
it.  Just like I can't recall the taper angle of a cabinet tip
screwdriver (despite knowing it intimately, at one time!)

Or, drag out a document that I'd prepared that describes the
process and rationale in detail.

The latter helps OTHERS benefit from my efforts.  Particularly
important if I'm *expecting* others to maintain code that I've
developed (and seldom have interest in -- for that client -- thereafter)

Controlling the rudder on a boat to keep it on a particular course
(to a specific destination) despite varying cross currents and
winds is essentially the same as controlling the moisture content
in a manufacturing process or the speed of a motor under varying loads.
Yet, there is very little "common code" shared among those three
applications (all projects I've done).  Knowing the statements that
were executed in one application -- or in which "function/module" to
find them -- would help little with either of the others!

What's important is the approach to the problem and a grasp of
the application domain; how varied your past experiences and your
ability to extrapolate from them to new problems domains.

Reuse *designs*, not code.

>> The problem I outlined, here, was coming up with quick algorithms
>> to highlight likely discrepancies WITHOUT exhaustively examining
>> the contents of all of the files in a particular SET of files.
>
> There is no such algorithm. All the file metadata in the world
> will not tell you that the file has a few flipped bits not detected
> by error-correcting codes - you need to actually read the file.
> And even after you've done that, those bits could flip five minutes
> later.

Of course there is!  If the file's size has changed from that which had
been recorded, then either the recorded datum is in error (my assumptions
are that the RDBMS is *never* wrong!), the dirent is munged *or* the
file's contents have actually been changed (and/or truncated/extended).

    "The problem this has is examine-file() is expensive -- for large
    numbers of large files!

    "I'd like to SHORT-CIRCUIT this to provide an EARLY INDICATION of
    POSSIBLE PROBLEMS.  E.g., stat(2) each "theFile" to identify any
    that are "Removed" without incurring the cost of examining the
    file in its entirety.

What you can't know is whether APPARENTLY good metadata indicates
unaltered file contents.  (reread my above statement:  "highlight likely
discrepancies", not "likely integrity")

Other metadata (like timestamps for create/modify/access) can be
too easily bodged WITHOUT the file's contents being altered.  This
leads to a forced "detailed verification" step to determine
the meaning of that "bodge":  is it superfluous?  or, is it indicative
of a change that is hidden by an unchanged file size??

I.e., if I quickly examine all of the sizes for a tracked portion of
the dataset, I can be reasonably CERTAIN to note that something HAS
changed.  The actual nature of the change may be immaterial at this
point in time.  If they are all intact, I know nothing about whether or
not they are, in fact, INTACT.  An altered timestamp just tells me to
check in greater detail -- and that effort might be entirely wasted.

I have an application that INSISTS any file that I open "has been
altered" and offers to save the altered file FOR me.  Yet, I *know*
the file has not been altered.  If I allow it to save the file,
I will end up with the exact same file on my disk -- but with a
different timestamp!  If I've been looking at the file for a while
and "absorbed" in that activity, I may forget whether or not I've
made any actual changes!  When I go to exit and am prompted to save the
file, I'll be forced to err on the safe side -- even though the
content might not have been changed!  The "updated" file will be
IDENTICAL to the file as it existed before I opened it!

Later, I will see that files that depend on that file will have to
be updated -- needlessly -- just to bring THEIR timestamps into
compliance with the original file!

Thus, timestamps are too easily bodged to be effective indicators.

[I can recall naively touch(1)-ing the contents of the entire filesystem
on my first SysV box.  Made all the timestamps look nice and consistent!
Of course, there, the timestamps actually *were* important!]

As for "flipped bits", the only thing I can do to guard against that
"five minutes later" is to perform the check, again -- after > 5 minutes
have elapsed!  And, if the medium isn't spinning 5 minutes hence,
I'll have to wait until the next time I have such an opportunity!
(by remembering that it is "due" for another check)

On 10/30/2016 3:45 PM, Ali wrote:
> On Thursday, October 27, 2016 at 11:23:58 PM UTC+1, Don Y wrote:

>> The problem this has is examine-file() is expensive -- for large numbers
>> of large files!
>>
>> I'd like to short-circuit this to provide an early indication of possible
>> problems.  E.g., stat(2) each "theFile" to identify any that are
>> "Removed" without incurring the cost of examining the file in its
>> entirety.
>>
>> [Of course, all files MUST be eventually examined as you can't verify
>> accessibility (BadRead), actual size (BadSize) or correct signature
>> (BadHash) without looking at every byte!  But, you can get the simple
>> checks out of the way expeditiously to alert you to "obvious" problems!]
>>
>> I can get an approximate indication of "actual size" just by stat(2)-ing
>> the file, as well -- relying on examine_file() to later confirm that
>> figure.
>>
>> The hash, of course, can't be guesstimated without trodding through the
>> file in its entirety.
>>
>> The problem with *this* approach is I'd have to lock the entire object to
>> ensure the integrity of the "early results" agrees with that of the
>> "later results"  (the original approach only requires taking a lock on the
>> file being examined *while* it is being examined).
>>
>> Are there any "tricks" I can use, here?  E.g., the equivalent of CoW
>> would ensure the integrity of my data (for the *original* instance) in
>> the presence of such a global lock.  But, I think only the Bullet Server
>> would give me any behavior approaching this...?
>>
>> Also, how wary would one be relying on the files' timestamps as a (crude)
>> indication of "alteration"?  (It seems far too easy to alter a timestamp
>> without altering content/size so I've been leary of relying on that as
>> having any "worth").
>
> I would suggest to compartmentalize this function. There are unnecessary
> conditional operators stacked to achieve a ‘something’ common.

Its just pseudo-code -- intended to show what has to happen, overall.

I.e., if a particular file in the list presented is NOT found on
the media "as expected", then it should be considered as having been
REMOVED (deleted) from the medium -- naming it in the list of files
presupposes that it SHOULD be there.  If it *is* there but can;t
be read/examined, then it is INACCESSIBLE -- naming it in the list
of files presupposes that it WILL be accessible (i.e., no read/seek
errors involved in examining its entire content).

Having verified the file is present and accessible, a record for it
should be available in the database -- yet another presupposition.
When a record IS available, the size recorded should coincide with
the size of the actual file encountered on the medium.  And, having
the correct size should also lead to the contents having hashed
to the correct "signature" recorded.

Failing any of these conditions is an indication that the file presented
has not been encountered on the media in the form expected.

You can promote the "not found in database" test to precede the
examination of the file's contents.  But, that test will rarely fail;
the list of files to be checked comes FROM the database!  The test
just ensures the database hasn't changed in that regard wrt THIS
particular file in the interval between when the list of files
was prepared and when this particular file happened to be examined
(checking a file can take MINUTES or more; plenty of time for the
database to encounter changes -- unless I take a lock on it for
the duration of the testing process.  Or, interleave the file list
creation with the actual verification)

> How about breaking apart file handler and DB results? That should provide
> two clear paths to work with. //late to party!

You can't compare the "actual" OBSERVATIONS of the file to the
stored EXPECTATIONS until you have examined the file.

The question is how much "examine_file()" can be decomposed and
the consequences of that decomposition.

E.g., if implemented, instead, as (I'm being lazy in my rewrite and trying to
maintain a resemblance to the original code so this is a quick cut-n-paste):

verify(files: list of string ): list of (string, reason)
{
     failures := list of (string, reason);

     while (files != nil) {
        theFile := hd files;
        do {
           (result) := check_file_exists(theFile);
           if (NotFound == result)
              {Result = Removed;      break;}

           (Result, Size, Signature) := query_DBMS(theFile);
           if (NotFound == Result)
              {Result = NoRecord;     break;}

           // assume file must remain in the database, now that
           // it has been spotted there;  likewise, assume file
           // must persist on the medium as it has been seen there
           // previously!  Otherwise, rely on a lock, transaction
           // or some explicit "recovery" algorithm
           (size) := check_file_size(theFile);
           if (Size != size)
              {Result = BadSize;      break;}

           // same assumptions as above!
           (result) = verify_file_accesible(theFile);
           // of course, this would actually be folded into the
           // operation below; you'd not need to make two passes
           // over the file to verify it can be read in its
           // entirety and THEN compute the hash in a separate pass
           if (BadRead == result)
              {Result = Inaccessible; break;}

           // same assumptions as above!
           (signature) := check_file_signature(theFile);
           if (Signature != signature)
              {Result = BadHash;      break;}
        } while (ONCE);

        if (Success != Result) {
           failures = (theFile, Result) :: failures;
        }

        files = tl files;
     }
     return( failures );
}

The problem with this approach is it gives you several windows where the
state of the file, the filesystem (the metadata accessed by check_file_size
and check_file_exists) itself and the DBMS can get out of sync.  You
have to take a lock on the database and the filesystem!

[A "transactional filesystem" has been discussed in this context; so,
like the DBMS, you can rollback operations on the filesystem to ensure
it always remains in sync with the data you've stored regarding its
state (you have two copies of that data and no other way to ensure they
REMAIN in sync while you perform these lengthy tests]

The bigger problem lies in structuring the loop to process each file
in the list in its entirety -- before advancing to the next file
in the list.

Imagine, instead:

verify(files: list of string ): list of (string, reason)
{
     failures := list of (string, reason);

     non_existent := verify_existence(files)
     failures = non_existent :: failures;    // unsupported syntax

     // FAIL:  should have removed non_existent from files!
     not_recorded := verify_record(files);
     failures = not_recorded :: failures;    // unsupported syntax

     // FAIL:  should have removed not_recorded from files!
     wrong_sized := verify_sizes(files);
     failures = wrong_sized :: failures;    // unsupported syntax

     // FAIL:  should have removed wrong_sized from files!
     inaccessible := verify_read(files);
     failures = not_readable :: failures;    // unsupported syntax

     // FAIL:  should have removed inaccessible from files!
     counterfeit := verify_content(files);
     failures = counterfeit :: failures;    // unsupported syntax

     return( failures )
}

verify_existence(files: list of string ): list of (string, reason)
{
     failures := list of (string, reason);

     while (files != nil) {
        theFile := hd files;

        (result) := check_file_exists(theFile);
        if (NotFound == result)
           failures = (theFile, Removed) :: failures;

        files = tl files;
     }
     return( failures );
}

verify_record(files: list of string ): list of (string, reason)
{
     failures := list of (string, reason);

     while (files != nil) {
        theFile := hd files;

        (Result, Size, Signature) := query_DBMS(theFile);
        if (NotFound == Result)
           failures = (theFile, NoRecord) :: failures;

        files = tl files;
     }
     return( failures );
}

verify_sizes(files: list of string ): list of (string, reason)
{
     failures := list of (string, reason);

     while (files != nil) {
        theFile := hd files;

        (Result, Size, Signature) := query_DBMS(theFile);
        (size) := check_file_size(theFile);
        if (Size != size)
           failures = (theFile, BadSize) :: failures;

        files = tl files;
     }
     return( failures );
}

etc.  (hopefully you can see the pattern emerging, here...)

There's a subtle but important difference.  The END RESULT is
POTENTIALLY the same list of failures.  There can be subtle
differences in the list as different aspects of each file
(and the database entries) will be processed at different
relative wall-clock times.  E.g, the signature of the first
file won't be checked at the same time relative to the
invocation of the main function.  Indeed, it may not be
checked AT ALL -- based on which preceding tests might fail!
(compared to my initial implementation which examined ALL
aspects of a file before advancing to the next)

Another subtlety is that at each point where "failures" is
updated, the presence of ANY "theFile" in that list is a
positive indication that the file in question is *NOT*
intact.  And, by extension, that the list of files is not
completely intact!

The caller only has to wait for the *final* assessment to
be "assured" that the ENTIRE list is INTACT (in so far as
these tests can vouch).  Other information can be leaked,
sooner (as shown)

So, if you have a medium whose integrity is suspect, you
get an early indication that it *is*, in fact, corrupted!
You aren't forced to wait for a THOROUGH examination of
all files preceding the KNOWN OFFENDER (i.e., the file
that has a size of 3 bytes instead of 283922; or the file
that has "gone missing"; or the dirent that can't be
accessed; etc.) before that determination can be made!
   "Crap!  This DVD+R is showing signs of bitrot!"
   "Crap!  I must have accidentally deleted part of this medium!"

The remedial actions you take when armed with this information
sooner (vs. later) can be very different.

E.g., suspicious of a failing medium, I might not want to wait
hours for its entire contents to be ROUTINELY VERIFIED before
taking action to preserve those objects that might ONLY exist
on that medium.  Or, the objects that are of "true value".
ESPECIALLY if the device is throwing read errors and incurring
lengthy timeout/retries (e.g., an optical medium).

If, as is probably the MOST COMMON CASE, a set of objects had been
accidentally DELETED, then waiting until they were encountered in
the "list of files" could potentially be after many (thousand, millions?)
of files have been THOROUGHLY examined -- and found to be intact
(i.e., The Norm for any reliable medium!)  Checking for the existence
of ALL files before checking the content of EACH file would make
that information known sooner rather than later.  This affords
you the opportunity to restore those files as the removed copies
may have been your sole "backup"; the originals are at risk until
you can recreate those backup copies!

Of course, the "hazards" and races that this approach presents differ
significantly from the previous -- and original! -- approaches.
So, while the results may be the same if done during "regularly scheduled
system maintenance" (like a commercial IT department would do), they
won't be the same if the database and the filesystems are being actively
used when the tests are performed.

[Indeed, when the filesystems are NOT being used, the media are
typically POWERED DOWN in my case!  No dedicated period for
"routine system maintenance"; that has to happen alongside my
normal usage -- and ADAPT to those usage patterns!]

On 29/10/16 22:15, Don Y wrote:
> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:

Don, let me try to understand the basis of your problem here.  Correct
me at any point that I am wrong (I am writing in many small paragraphs
to make that easier).  I have also added a few comments inline below my
main post.

1. You have lots of files, held on computers, and you want to make sure
they don't get changed or corrupted.

2. You don't trust "metadata" methods such as sizes and timestamps,
because files could be changed or corrupted without affecting these.

3. You don't want to read through the entire file, because they are big.

4. You want some sort of algorithm for detecting file change or
corruption, given 2 and 3.

If that is the case, then you will be disappointed - no such algorithm
could exist.


So now we have to consider what would give you a convenient, economical
and reliable way to store your files safely.

First, I assume that your old existing systems are irrelevant except as
a source to copy from.  Spending time and effort on maintaining outdated
hardware is a waste - I expect you to set up a new system.

Second, I expect you to have off-site backups.  This could be done by
duplicating the main system, or by simply having a number of off-site
copies on DVD, archived hard disks, cloud storage, etc.

Third, I expect you to leave behind your prejudices and knowledge of
limitations of old systems, and to look forward.


Files change, get lost, or corrupted due to four main causes:

1. A user changes them in some way (intentionally or unintentionally).

2. Malware or hackers change them.

3. Bit rot or sector wear causes some data to be lost on a disk.

4. Serious hardware failure, such as a disk head crash losing the whole
disk.

I expect you to handle 1 and 2 mainly through good security and regular
backups.  I strongly recommend using a filesystem or setup with cheap
snapshots - then you can take them regularly.

There is a myth that hard disk sectors can get corrupted and return
incorrect data without an error indication - the so-called "undetected
read error".  In theory, this /can/ occur - but unless you are using the
data quantities of CERN or Google, you will never see it in your
lifetime (that's why I call it a myth).  You /might/ come across
firmware errors in disks that give such undetected read errors, though
it is highly unlikely.  "Unrecoverable read errors", on the other hand,
are not as uncommon as one would like - sometimes a disk is simply
unable to return the data you ask for, but it can at least inform you of
the failure.


The general method to protect against failures is redundancy combined
with systems to spot errors.  For storage, that is almost invariably
done using RAID.

So your basic setup here is to have two drives with a RAID-1 mirror.
Anything that is written to the mirror gets written to both drives.  If
a sector read fails (despite the drive's extensive error checking and
correcting codes), the RAID controller (hardware or software) will read
the second copy from the other disk.  It will then re-write this correct
data to the first disk, which the drive will map to a spare sector.  If
a drive fails entirely, the controller will read from the other drive.

"Bit rot", or slow bit corruption on disk sectors, is prevented by
scrubbing.  You regularly read through the entire array, for both disks.
 If a sector has had corruption on some bits, the drives ECC coding will
usually let it correct the error automatically, and it will re-write the
sector (with re-mapping if necessary) to refresh the data.  If the
corruption is too great, the RAID controller will re-write the sector
based on the second copy.

Thus RAID keeps your data reliable, at least until a catastrophic
failure (such as the whole server dying) - that's why you have backups.


For more features, I would recommend using BTRFS or ZFS to handle your
RAID mirror.  These filesystems give you additional checksumming (for
protection against drive firmware errors, or extremely rare undetected
read errors), as well as very cheap snapshots and other goodies.

On the other hand, if you want the simplest "will work on anything, even
old Linux systems or Windows systems" solution, then use Linux md RAID
with metadata format 1.0.  This puts the RAID information at the end of
the disk - all the rest of the disk appears /exactly/ as a normal
non-raid disk, and can be read from anything else.


>> Disclaimer: I am not an expert on the data storage, but here are my
>> thoughts:
>>
>> - Use a RAID system which has enough redundancy to detect and correct
>> defects in the media. Depending of the RAID level used, the capability to
>> data recovery varies.
> 
> I started out using RAID5 "hardware" arrays (and other, more sophisticated
> configurations).  This requires you to have multiple spindles running
> concurrently (else the array can't *detect* discrepancies).  Compact
> arrays also tend to suffer from cooling problems.

You will need two spindles (or SSDs) concurrently - that's the only way
to get two copies of your data on two disks.

Don't bother with RAID-5.  Either use RAID-1 if your data fits
comfortably on a single disk, or use RAID-6.  Modern disks have less
cooling requirements than old ones, if you go for RAID-6.

> 
> It also ties you to an often proprietary (or, at least, poorly documented)
> implementation ("volume format").  So, a failure of the hardware can leave
> you with possibly intact volume(s) and no straightforward means of
> accessing their contents.

Use software RAID - either Linux, or BSD if you prefer.  Avoid
proprietary hardware solutions.

> 
> Moving volumes between arrays is seldom supported (unless you have
> identical
> arrays -- hardware/software -- AND a well considered means of doing this.
> This effectively precludes having volumes off-line.  Introducing a
> "foreign"
> volume to an array will often result in the array prompting you to
> "initialize" it (wipe it clean!).
> 
>> - Duplicate or mirror the data across different physical media.
> 
> This is RAID1.  Many of the above issues still apply.  In any case,
> the mirrors must be spinning concurrently for the array to function.
> (It has no way of knowing what the "other" copy of the "file"
> looks like unless it can *see* it, now, in its entirety).

If you are really desperate to avoid having two disks spinning at the
same time, it is possible with Linux md raid to designate one drive as
"write mostly".  It will then only be used when writing (obviously),
when scrubbing (obviously), or when reading if the first drive fails to
return the correct data.  I cannot think of any real benefit of this,
but it is possible.

> 
> This begs two crucial questions:

No, it /raises/ two questions.

> - are BOTH copies of the file examined when you try to access it?

No.  That would be highly inefficient, and does not contribute to
improved reliability.

>   Or, is a "primary" copy accessed and the "secondary" copy only accessed
>   if the primary throws a read error?

Yes.

> - If a discrepancy is found between the TWO instances, how is the
>   "winner" selected?

Discrepancies don't occur with non-negligible probability unless you
have had a massive hardware failure, or power-failure during writing.
See <http://neil.brown.name/blog/20100211050355> for a discussion on the
topic.

> 
> Plus, the more subtle question seldom considered:
> - are files that are not "manually" accessed EVER verified?  Or, do you
>   have to wait until you need/want a file to discover if it is intact??

That's what "scrubbing" is for.  Hardware raid controllers should do
this automatically, with software raid it is usually done with a cronjob.

>   (Do *you* periodically check the contents of the batteries in your
>   flashlight to verify they are functioning BEFORE you need them?
>   Does the *flashlight* take on the responsibility of performing this
>   test on your behalf??)

A torch is not a raid device.

> 
> Consider the above when you have many terabytes of data at stake.

RAID like this is /exactly/ what people use when they have TB of data at
stake.

When they have more than that, they use redundant servers, or split the
two halves of their software RAID-1 pairs across different boxes.

> Do you want ALL of it "spinning" whenever you need to access *any*
> of it?  

Why is that a problem?  Most hard disks have no problem spinning all the
time.

> How long will it take for a proactive array to examine/verify
> ALL of the data just to reassure you?  (will you have the ENTIRE
> array "up" for that length of time?  Or, will you power it up just
> long enough to access the file(s) of interest, then power it back down,
> again?  Or, will you power up ONLY the portion of the array known to
> contain the file(s) you seek?)

Again, what is the problem?

If this is an array you rarely access, you can spin down all the disks
when it is not in use.

> 
> As I said, I've been down this route.  Ever hear what twenty to
> fifty 10-15K drives sound like spinning at the same time?
> <http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>.
>  Even a 12-14 drive "shelf" is loud:
> <http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>.
>  Remember,
> they're all throwing off heat in a relatively dense package so you
> have lots of (redundant!) fans in place to keep them from melting...
> 

Why are you considering a decade old high-end datacenter storage system?
 These are designed for installations where noise doesn't matter, and
you have massive cooling systems.

Just get a simple little server box with space for a few disks.  I
really like the HP Microservers - Gen 8 devices are dirt cheap,
reliable, and have four neat 3.5" drive bays behind a door in the front.
 They are perfect for file servers unless you need really high-end
performance.

> The "problem" with these solutions is they are designed for high
> performance.
> Serving files continuously, typically, to an "organization".  And, with a
> high degree of reliability/availability.
> 

So don't get one.

> For an *archive*, you tend to need lots of capacity but comparably little
> bandwidth!  (how *often* will you access its contents?  how quickly will
> you need the file "delivered" to you?)  And, you can trade reliability and
> availability for bandwidth and cost.  Do you keep your 2013 tax returns
> in your top left desk drawer (where they are easily accessible)?  Or, have
> you moth-balled them to a more capacious site that's, perhaps, a bit more
> difficult to access??

What capacity are you looking for here?

On 10/29/2016 11:00 PM, Clifford Heath wrote:

[attrs elided]

>>>>>>>>>> Disclaimer: I am not an expert on the data storage, but here
>>>>>>>>>> are my
>>>>>>>>>> thoughts:
>>>>>>>>>> - Use a RAID system which has enough redundancy to detect and
>>>>>>>>>> correct
>>>>>>>>> I started out using RAID5 "hardware" arrays (and other, more
>>>>>>>>> sophisticated
>>>>>>>>> configurations).  This requires you to have multiple spindles
>>>>>>>>> running
>>>>>>>>> concurrently (else the array can't *detect* discrepancies).
>>>>>>>>> Compact
>>>>>>>>> arrays also tend to suffer from cooling problems.
>>>>>>>>>
>>>>>>>>> It also ties you to an often proprietary (or, at least, poorly
>>>>>>>>> documented)
>>>>>>>>> implementation ("volume format").  So, a failure of the hardware
>>>>>>>>> can
>>>>>>>>> leave
>>>>>>>>> you with possibly intact volume(s) and no straightforward means of
>>>>>>>>> accessing their contents.
>>>>>>>>
>>>>>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc)
>>>>>>>> use Linux RAID support. If the hardware fails, you can mount
>>>>>>>> the drives on any Linux system that has enough SATA ports.
>>>>>>>
>>>>>>> Really?  So my Sun RAID systems run Linux??
>>>>>>
>>>>>> Ahh, my apologies. I didn't realize you were running a computer
>>>>>> antiques
>>>>>> museum. I thought you were trying to ensure the availability of some
>>>>>> data.
>>>>>
>>>>> So what is he supposed to do with data which are useless outside of
>>>>> a particular environment. Apparently he has plenty of past projects
>>>>> spanning a few decades back to maintain - how do you suggest he will
>>>>> separate movable from non-movable data. Then the sheer amount of data
>>>>> he is talking is just huge.
>>>>
>>>> No, his amount of data is trivial compared to the petabytes
>>>> being accumulated daily and managed by internet-scale companies,
>>>> who almost all use *commodity* hardware running *open source*
>>>> software.
>>>>
>>>>> I strongly suspect Don is not an office secretary running an office
>>>>> archive and I also strongly suspect he knows what he is doing.
>>>>
>>>> Don is a "special flower". He faces problems that no-one else
>>>> has ever faced, and he analyses them with an insight that no-one
>>>> else has ever had, requiring solutions that no-one else has ever
>>>> found.
>>>>
>>>> None of these descriptions are true, of course.
>>>>
>>>> For what it's worth, data management is a global profession with
>>>> a hundred major specialties and a thousand vendors, who all have
>>>> experts with more experience than Don has in his toenails. I've
>>>> just returned from a paid global speaking trip where I addressed
>>>> some of them. "Availability" isn't my specialty, I just rely on
>>>> commodity equipment to manage my measly few dozen terabytes. But
>>>> what do I know, I don't work with special problems like Don's.
>>>
>>> Well,
>>>
>>>  >> Linux/foss may be a huge part of life but it is restricted to the
>>>  >> needs of the general public which feeds it directly. There are
>>>  >> other aspects of tech life which are simply beyond the scope of
>>>  >> foss/linux, beyond its capabilities as they are.
>>>
>>> But I know many of the foss/linux crowd think otherwise - which is
>>> OK, everyone sees the walls of their own room. No need to be grumpy
>>> about other people wanting to talk about other things though.
>>
>> Clifford wants his answer to be "right".
>
> No. I want you to consider that your problem might not be unique,
> and that you should *first* look closely at existing solutions.

Here's my "IT environment" to better gauge how "common" it is (keep in
mind it's just a one-man shop!):

+ 19 "Free standing" computers (and associated hardwired peripherals as I'm
   SURE folks will grumble "why so many?" and can thus see how hard it would
   be to attach ALL the various I/O's to a smaller number of hosts as well
   as the issues keeping a wider variety of applications "cooperating"
   within a host!)
- email/WWW machine (laser printer)
- TV media tank (TV, wireless UI's)
- SWMBO's workstation (external disk for storing her photo archive)
- CAD/EDA workstation (digitizing tablet, DSO, logic analyzer, motion
   controller, "Nano Board", Unisite, SCSI support, DLT70)
- Graphic Design (color calibrated B size scanner, calibrated oversize film
   scanner, calibrated monitors, calibrated "photo printer", NuLooq controller,
   SCSI support, DLT70)
- Software Development (A size scanner w/ADF, ICEs/"targets")
- Document Preparation (A size scanner w/ADF, 36 inch Wide Format printer)
- Multimedia Author (video digitizers, head tracking camera, DLP, whiteboard
   controller)
- VM server (dual wide SCSI support, HVD SCSI support, 4T+system disk spinning)
- Network Services (HEADLESS, small UPS guaranteed to work!  :> )
- Live Repository (SCSI support, 5T+system disk spinning)
- 2 1U servers that routinely get repurposed (4 removable drives, each;
   presently doing duty as a DBMS server and a "removable drive" NAS)
- Solaris 8 host (Chimera II, external 711 case)
- Solaris 10 host (external 711 case, DLT70?)
- ShuttleX "shoebox" (continually being repurposed for short term uses -- like
   evaluating new distros, applications, etc. without "contaminating" other
   machines; tonight, it's busy building a DVD image for one of SWMBO's
   "art demos")
- Docked Tablet PC (my "dead tree library" in electronic form so I can
   read on a decent size screen and "write" annotations; also a convenient
   location from which to converse with the "network services" box as well
   as explore external USB disks)
- Sun Voyager (nostalgia piece)
- Compaq Portable 386 (legacy ISA support, Opus PM)
[there are two other boxes that are on their way out the door; SWMBO's old
machine and a box I use to check IDE drives for their contents before
discarding them as well]

+ 7 laptops (different OS's, different toolkits, different I/O capabilities;
   one configured for troublshooting certain types of kit in the field, another
   for general purpose use while traveling; another small one for "just email",
   etc.)

+ 10 diskless appliances
- SCSI peripheral server
- FW peripheral server
- 2 X Terminals
- 6 "new NAS system" (4 external drives, each)

+ 59 External disks (may have missed a few; haven't double-counted SWMBO's)
- 3 FW
- 25 SCSI (12+12+1)
- 29 USB
- 2 USB "fingerprint protected"
- ~200GB of "persistent" thumb drives

+ 7 NAS's (1, 2 and 4 drive models)

+ 12 "bare" drives (in a box) containing images of each system, "as built"

+ 30+ oddball drives lying around the contents of which I have to sort out
   (i.e., "do these files exist elsewhere among my machines/archive?  If
   so, these instances can probably be erased; if not, I need to get them
   under some formal set of controls")

+ 3 networked printers (one of which has an internal disk)

+ 6 spindles of "burned" optical media

+ 4 spindles of "published" COTS optical media (applications)

+ ?? "published" COTS music CD's

Here's MY list of requirements (much of this cobbled from my design notes so
they are conversational in presentation):
- support any media for which I have hardware and the hosting OS has drivers
   (note the "application" need not run under said hosting OS!)  E.g., the
   mundane options here include SCSI, FC-AL, FW, USB, PATA/SATA drives,
   optical media, any other "storage" media that can be accessed as a
   "mass storage device", etc.).  I'll concede that the host OS must export
   a filesystem interface to the device in order to be considered
   (e.g., don't worry about raw tape)
- use individual drives and portions thereof (slices, etc.) as "data stores"
- use individual portions of filesystems similarly (e.g., mount points)
- use *any* filesystem for which the "media hosting" OS has support
- be unconcerned re: the "mount point" of any such data store
   (e.g., a CD on U: or V: is treated NO DIFFERENTLY than if it had been
   mounted under /user/home/MyFiles _ON A DIFFERENT HOST_)
- "notice" when data stores come "on-line" as SMB/NFS shares (so a network
   client can do this processing instead of having to install a LOCAL client
   on each box that could potentially host such a share)
- support R/O media (i.e., no "breadcumbs" on the monitored media; you've got
   a database available -- or, MAKE one available on a host that can be
   accessed from any client; web/telnet/ssh user interface)
- don't require the raw (character) device to be exposed
- allow fine-grained specification of objects to track (i.e., /foo/file1
   while avoiding /foo/file2)
- allow quick specification of large swaths of any datastore to be
   monitored (e.g., /folder; *.jpg; etc.)
- let me associate an action with an object (set of objects).  E.g.,
   "If this object differs from your notion of what it *should* be,
   update your notion of what it should be vs. alert me that it has
   deviated from expectation".  This can then be coupled with ACTIONS:
   "If your notion of an object has been updated, update (schedule for
   update) all instances that are reflective of that OLD notion"
   (e.g., this gives you an "autobackup" capability)
- periodically (my choice of frequency on a per object basis) "verify" the
   accessibility & integrity of each object on all data stores currently
   available (note also the issues raised in this original post)  I'd
   obviously not be keen on verifying hundreds of optical media on a WEEKLY
   basis!  Or, even MONTHLY!  OTOH, feeding one or two into whichever
   workstation I happen to be using on a given day wouldn't be burdensome and
   would manage to trod through the entire set in a reasonable amount of time
- asynchronously alert when media needs to be physically mounted so that it
   can be accessed for its periodic tasks (i.e., if it needs to COMPARE
   mounted_medium/foo to not_yet_mounted_medium/bar; or, if it needs to
   do a periodic check of unmounted_medium/folder)
- schedule those actions so I'm not a slave to its mindless whims:
     "Please load 'Windows XP Reinstall Disc' so I can verify setup.exe."
     "Please load 'Adobe CC 2015 Install Disc' so I can verify autorun.ini"
     "Please load 'Windows XP Reinstall Disc' so I can verify autorun.ini."
     "Please load 'Adobe CC 2015 Install Disc' so I can verify..."
   (substitute other verbs for "verify" and you can see the pattern to be
   avoided).  Note that I may not acquiesce to that request the instant it
   is issued.  And, by the time I am willing to do so, the system's
   priorities may have shifted:  "No, do THIS, instead!"
- notify when it sees "changes of significance" on any of the monitored media
   and data stores -- which objects have disappeared or been altered; media
   that are "having problems", etc.  I.e., "You should consider creating a new
   copy of the 'ProjectXYZBackup#2' as it is throwing lots of read errors!
   Soon, you may NOT be able to recover its contents to create such a backup
   (I *know* this because I know everything "important" that resides on that
   data store and know that most of those objects ONLY exist in that ONE PLACE.
   You don't have to bother exploring that issue manually as I anticipated you
   would like to know that sort of thing to determine your plan of action!)"
   A human user should be able to catch that notification as it happens *or*
   at some future time (as the user might not be in attendance when emitted)
- identify likely file duplicates to assist me in DEFINING those fixed
   relationships (i.e., "\\MYSHARE\path\to\Y seems like it MIGHT be the same
   file as \\SOMEWHEREELSE\another\path\to\X baed on its size and hash; next
   time I *see* both of them at the same time, I'll check their contents
   in a byte-for-byte comparison.  *If* they coincide, I will SUGGEST they
   MIGHT be the same object -- though that may be purely coincidental!  *You*
   will have to let me know if they, in fact, are.  Thereafter, I will use
   knowledge of this relationship to suggest each as a "backup" for the
   other, if/when the need arises.  Or, I can alert you to this and, when
   you've got time, you can make both of these available for me to examine
   sooner instead of later!")
- let me change those definitions/relationships as the data's needs change
   E.g., /project1/Copyright and /project2/COPYRIGHT are identical and
   will be treated as potential backups of each other -- UNTIL such a time
   that I decide the terms of Project2 warrant a change to ITS copyright.
   Thereafter, the "discrepancy" noted between these two objects should
   NOT prompt an alert/suggest that one had been "corrupted" and in need
   of a "restore" but, rather, should treat them as totally independant
   objects with their own characteristics and criteria.
- support a variety of arbitrary containers (e.g., a tarball is a container,
   NOT a file!  Ditto RAR, ZIP, 7z, ISO, SIT, VHD, etc.  The order of the
   files *within* the container is not controlled -- which WOULD be the case
   if it was treated as a file!  So, removing /path/filename from foo.iso
   WILL change the date, size and signature of foo.iso.  But, if the remaining
   TRACKED contents of foo.iso are not "corrupted" in the process, no remedial
   action should be signalled.   Just like rearranging the physical order
   of files in a subdirectory shouldn't cause those files to be "repaired";
   the container hasn't been altered!)
- treat the contents of all such containers on a par with similar objects
   in other/different containers (i.e., \\MYSHARE\another\path\to\Y,
   foo.zip(/path/to/X) and <someDVDinadeskdrawer>\boo -- where X, Y and boo
   are actually identical in content!)
- in EXACTLY the same way that I can add files to a folder/directory (e.g.,
   to replace files that have "gone missing") the contents of each of these
   other "containers" should be alterable.  And, as the TRACKED objects IN
   the container may not be affected by these operations, said action
   shouldn't result in remedial actions/warnings!  So, I should be able to
   add "MyPersonalNotes" to a tgz archive without the archive itself being
   considered altered.
- as above, if a container is accidentally deleted (folder, archive, ISO,
   etc.), the TRACKED contents of that folder (and subfolders within)
   should be recreatable to the extent that "backups" of those components
   are available elsewhere in the system -- even if not presently on-line
   (cuz the database KNOWS what was contained in said container and where
   the backups of those objects exist)
- spoofing the above, let me introduce a new medium and force ALL of some
   TRACKED medium's contents to be "restored" to it (i.e., mount something
   as /my/files and expect everything recorded as being contained on that
   to be ADDED to this new medium from backup instances.  Objects which can't
   be recovered have to be reported as TANGIBLE LOSSES.
- further, let me *migrate* tracked portions of the archive to new media
   (e.g., replace some files from medium A and some from medium B onto a new
   medium C -- while leaving the balance in place or possibly migrating
   them to other existing media).  I.e., allow media to evolve without
   burdening the user with lots of DBMS busywork redistributing the records
   associated with the existing tracked content
- STILL work when I'm running Windows 47, Solaris 18 and NetBSD 23.6 and
   when the new holographic drives are affordable and commonplace.  As long
   as some one or more of those OS's have reliable drivers to access the
   associated media and support the network protocol interfaces required.
- won't require me to be perpetually upgrading *it* to keep it
   running with the changes in technology, creeping featurism, etc.
   I.e., the hosting OS for each data store bears that cost (along with
   continuing support for the file sharing protocols used)
- fix existing bugs without requiring me to sign on to a whole new
   SET of (potential) bugs (i.e., upgrade) -- "Gee, we have version
   ThisYearThisMonthThisDayNow available!  Update soon!!"
- won't cost significantly more than the consumer drives that I
   can buy dirt cheap from the local Costco on an hour's notice
   or require me to arbitrarily replace the processors and OS's
   hosting those data stores (e.g., "New and improved!  Now rewritten
   entirely in Java.  Hardware accelerator not included...")

How am I going to USE these capabilities (again cobbled from design notes)?

+ When I build a new machine, I'll export its complete filesystem and
   let this gizmo take a snapshot of its contents (name, size, hash).
   At any time, I will then be able to ask "what has changed".
+ For some systems, I will identify key components that I want to be
   able to easily restore as a result of the above examination.  When/if
   they are found altered, the gizmo will be able to tell me -- as well
   as tell me where to find *a* copy of the original object (which I
   will then be responsible for installing)
+ When I want to REbuild a machine, I should be able to mount
   a formatted medium and "trick" the gizmo into thinking it was
   a previous instance of that "tracked" set of files so it can
   locate suitable "backup copies" and tell me which media I need
   to mount for the gizmo to be able to restore those files
   (I do this probably once a week, on average.  Sometimes several
   times a day!  But, typically using Clonezilla images as I don't,
   otherwise, have a filesystem image of each machine in a tangible
   form.  I shouldn't need a second tool -- CZ -- to do this!)
+ Likewise, when I repair a machine for a friend/colleague, I
   should be able to just mount the media and tell the gizmo to
   "track" its contents; then, ask it to make a copy of them and
   verify it (so I can restore it after I'm done mucking about)
   A side effect of the above requirements is I can more definitively
   identify "what broke" in said machine -- instead of just shrugging
   my shoulders and saying, "It works, now..."
+ For the filesystems and boot images served to diskless appliances and
   workstations, this will allow those objects to be replaced on the
   *server* to restore any impacted operations
+ I can drag a bunch of files onto a laptop before taking a trip, tell
   the gizmo to note those contents.  When I return, I can determine
   *if* I had altered any while I was using the device.  Or, if NONE
   appear altered (to a lesser degree of certainty than an exhaustive
   examination of their entire contents; chances are, a files' size
   will be changed if I've made appreciable changes -- not JUST its
   signature!).  The source of the originals copied onto the laptop
   may be varied (e.g., on my last trip, I dragged several research
   papers from my "Technical Papers" archive; several speech synthesizer
   implementations from my software archive; the specifications for
   my synthesizers from my specification archive; and my "current sources"
   from the live repository -- so I could continue work on that project
   while away from home).  So, the checks/compares will need to be
   able to access each of these volumes (none of which are typically
   on-line).  If I'm distracted by other responsibilities at the
   end of my trip, it is likely that the laptop will just get returned
   to a closet shelf.  I may forget to "resync" its contents with
   my archived "originals".  Some time later, I will rediscover the
   laptop (perhaps taking it to the library for some research) and
   notice the files in its "playpen":  "Can I safely delete these?
   Or, are there some changes here that have never been reflected
   back into the main archive?"
+ Of course, much of this behaviour mimics how a VCS works.  So,
   can be used to merge changes from multiple sources
+ I can move my font collection from \Fonts on <somevolume> to
   \DTP\fonts on some other volume.  The database needs to know
   that this has transpired -- so it doesn't complain that
   <somevolume>\Fonts has been accidentally deleted!  (I did
   exactly this, last night as I needed to make room for some
   other, "more related" files on a particular PAIR of volumes;
   the <somevolume1>\Fonts and <somevolume2>\Fonts being easiest
   to move "intact".
+ In my routine housekeeping (disk-keeping?), I will OFTEN stumble
   on a file that "belongs" (based on some arbitrary personal
   criteria) in some other "container".  Or, will realize the
   need to add a notation to such a container, etc.  I should be
   able to move that file (or "install" that file) to the appropriate
   container and mark it (if not already marked) as "significant".
   I've recently been "injecting" files called "SerialNumber.txt"
   into the ISO images of most of my purchased software.  As the
   ISO is "just a container", doing this should be EXACTLY the
   same as moving a file to a FOLDER that held the ISO's contents!
+ I can, of course, quickly check any medium to see if bits are
   missing.  And, with a tiny bit of work, see if things have been
   *added* that aren't yet being "tracked".  E.g., each time I add
   an MP3 to the USB sticks that hold my music compilations for
   SWMBO's vehicle, I can verify if the music ARCHIVE has all of
   the same contents preserved ("Hmmm.... looks like I forgot to
   archive a copy/copies of this album!")  Ditto for videos
   (Flash, etc.)
+ When I sit down at a workstation and make changes to files that
   are being tracked (and now visible to network clients via an
   exported share), alert me if I've changed something I *shouldn't*
   have; or, silently update any "backups" (or, make a note that
   those backups need to be performed and request my assistance,
   eventually, in mounting the appropriate media).  I.e., *pull*
   updates off the "originating host" instead of requiring me to
   *push* them to backups.  Note that this can be combined with
   "watching" the file to see if it is ever (accidentally?) removed.
   So, all hosts magically get an UNDO capability regardless of the
   native OS's intrinsic capabilities in that regard.  No need for
   a "recycle" bin or an "undo utility" (that may or may not be able
   to recover a deleted file on a busy filesystem!)
+ Update a particular tool/service/utility.  Wonder if it will be
   well behaved -- or, if it will place past work at risk.  Snapshot
   the objects with which it is likely to interact.  Install new
   tool.  Play with it in a manner that should NOT, intuitively,
   alter any existing objects.  Run gizmo to see if any of those
   objects have been changed -- why?  If unhappy with this, remove
   the faulty objects and the "update", tell gizmo to restore the
   "now missing" objects.  I'll be doing this, soon, with my P4
   repository -- no desire to but many TB "at risk" on the assumption
   that there are no "problems" with the update!
+ Because all this is already in a DBMS, its a lead-pipe cinch to
   extend the schema to create other tables that allow you to group
   objects based on need/usage.  E.g., "these are the archive objects
   that are pertinent to doing work on the speech synthesizer, off site.
   Fetch them for me!"
+ I "find" a medium.  E.g., an unlabeled CD/DVD.  Or, all these disk
   drives scattered around the workbench and floor.  What's on it?  Is
   it important?  What was I doing with it at that time?  Was it a pull
   from a machine I retired?  Or, upgraded?  Did I remember to
   distribute ALL of its contents onto the appropriate workstations
   (I am continually subdividing responsibilities to distribute them
   on a greater number of more "specialized" machines)?  If I examine
   its contents in detail, will I instinctively remember where I
   LIKELY would have placed each object thereon?  Or, do I have to
   go hunting to see if I can find the object?  What if it is a
   file with a nonsense name (e.g., sal_20.pdf) that I've likely
   renamed to something more meaningful (e.g., "Finite Precision of
   Linear-Phase FIR Filters.pdf")?  Where will I have filed that:
   under "\Papers\DSP"?  or, maybe, "\Projects\Vocal Tract Modeling"?
   (perhaps you can see why there are so many disks just "scaterred
   about", here -- not serving any ACTIVE function!)

Wanna know the IT professionals approach to all this?  Install
a giant filestore that is up 24/7.  Use a redundant technology
(RAID, ZFS, etc.)  Copy EVERY medium onto it.  Put everything
under a VCS.  Add scripts to automate scrubbing and the notifications
outlined above.  Install VCS clients on all hosts.  Ensure all
users are disciplined to check things in and out of the repository
as required and WHEN required.  Limit which users can access each
object and perform actions on the repository.  Put in place procedures
whereby staff can request additional capabilities for specific users.
Appoint someone to be responsible for each portion of the repository.
Prohibit the use of laptops or other systems that can "move out of
the view of said system" for indeterminate periods of time.  Treat
any data outside of the system as "nonexistent"/unsupported.  etc.

Budget a healthy amount to replace drives that are kept spinning
despite only be accessed "rarely".  Increase budget for electricity
to keep them all spinning -- along with additional cooling, etc.
Make sure the entire system is on a UPS that is aggressively maintained.
Create multiple backups if disinclined to resort to alternative
backup media.  Put a copy off-site.  Ensure adequate fire protection.

Yada-yada-yada.

Then, when Joe Employee calls looking for a file that he had
LAST WEEK, shrug their shoulders and say they can't find it!
[I've heard MANY folks make that same complaint re: their IT
departments:  "They do backups EVERY night -- scheduled maintenance.
We have to leave our computers on so they can do this, remotely!
So, why can't they ever retrieve something that I 'lost'?  All
they seem to ever do is REINSTALL WINDOWS!"]

SWMBO would come home with similar horror stories from her
colleagues almost weekly:  "They -- IT -- have got a frigging ARMY
wandering around the place.  What the hell do they do if they can't
even address this BASIC need?"  She quickly learned to keep her own
backups archived locally to safeguard against their folly!

Yeah, I surely need a "professional IT department" to simplify
my life -- NOT!  :>

Wanna know how *I've* done it, before now?  A shoestring budget
in time and capital!  As I'm not looking for performance out of
the bulk of the archive, I can afford inexpensive consumer drives
(1/2/3/4 TB).  My oldest are more than 5 years old and still no
problems.  (Of course, their total power-on-hours are REALLY low
as they only need to spin when I'm interested in their contents;
I'm willing to wait 60 seconds for them to spin up when/if I need
them!  And, disciplined enough to spin them down as soon as I'm
done "needing them".  Disturbing that warranty periods have shrunk
noticeably on these items, over the years!).

OTOH, things like the VM server and live repository tend to need
much higher performance drives.  The VM server is actually running
code off those drives (VM's) so I'm not keen on reducing the apparent
performance of those VM's just to save a few dollars!  Likewise, when
the live repository is up, its because I'm pulling an old release or
pushing a new release onto/into it.  Chances are, it won't be a
simple, one-shot event (like adding a file to the consumer disk archive
or retrieving an ISO from said archive).

Likewise, the 711 enclosures hold "live data" for the two Sun boxen.
For the same reason you wouldn't run your workstation off a
sluggish-but-inexpensive consumer-grade disk, it would be silly
to run those off disks of that type!

Some external media can straddle the performance criteria.  E.g, my
"Microsoft offline updates" drive is only used when I build a new
machine.  So, I can afford to wait a bit instead of requiring a
speed demon (in a well cooled enclosure!).  OTOH, I don't want to
wait forever!  Damn update process takes a long time as it is without
adding a sluggish disk to the mix!

Ensuring duplicate copies of the archive remain in sync requires
self-discipline -- make each change (plus or minus) in every place it is
applicable!  To make keeping track of those places easier, mirror
entire volumes:  so Archive:/5 and Archive:/6 always have the exact
same *contents* (at the filesystem level, not at the raw device level,
as RAID would promote!)

Can't remember if you updated both copies?  Too bad, so sad... you'll
have to sync the entire volumes (instead of just making the incremental
changes that you KNOW are needed WHEN you made them to the first instance)

Unsure if those DVD+-R's are still intact?  Mount them individually and
tar cpf - * > /dev/null 2> /errors.  Hope!  When it starts throwing
errors, try reading it in a different drive.  Capture as many files
as you can from each drive that you try.  Hope you end up with a
complete set.  Curse yourself for not checking the medium two weeks
ago (when it might have been less corrupted!).

Or, create another magnetic disk copy of them and discard the optical
media entirely!

When I start working on a particular workstation, I remember to fire
up appropriate archive volumes (on another host) and "compare" the
local copies to the remote copies to see if something has changed that
I need to be aware of.  And, to fetch additional resources as the need
arises (e.g., maybe an application that I'd like to install locally that
I'd not installed previously -- I did that with ghostscript, last
week, on a box).  I rely on my own notion of what I need in lieu of
treating EVERY object as if under version control -- just in case
I want to inquire as to whether a new version/release is available,
somewhere:  "Ah, I haven't installed the most recent update to Firefox
on this machine.  Why don't I do this while THAT archive volume is on-line?"

[Note my hosts don't talk to the outside world.  So, they can't fetch their
own updates based on the whims of the application developers -- I don't
want changes to my work environment that I am not aware of and don't approve.
Will I now update Firefox on the OTHER workstations?  Why?  Will that
make my access of HTML documents in my local internet any better?  Will
the web interfaces to these peripherals perform better with an updated
browser?]

WHEN you discover something out of sync?  Start perspiring.  Be extremely
cautious with all of your actions going forward (don't want to accidentally
delete your sole copy of something because you got "Archive/5" and "Archive/6"
confused in your mind ("Now, where are the GOOD files?  And, where do I want
to copy them to??")  And, you don't want to update the NEW copy with the
contents of the OLD file!

[This is the equivalent scenario to a RAID failure -- except in that case,
your role is passive and with little control over the outcome.  The
array decides how it is going to recover -- regardless of the value
you may place on individual objects *in* that array!  ("Please save
my tax records, first!  I can always recreate the source code I've
been writing for the past month!")]

When the likely place for a backup doesn't yield the expected results,
resort to the "backups of last resort":  MO, tape, etc. and HOPE you
can find what you want on one of them.  Keep lists of their contents
in text files that you can grep(1) so you don't have to mount every
medium just to see if it *might* have what you need.

Need to bytewise compare two media?  Start them working just
before heading off to bed -- you are guaranteed NOT to interfere
with their actions by YOUR actions!

Damn near ALL of this is just tedium.  There is little that requires
much conscious thought.  Or, little more than a simple algorithm could
MORE EFFECTIVELY and MORE RELIABLY implement!

This new approach just mechanizes and automates things I've already been
doing -- while removing some of the artificial constraints imposed by
the limitations of my meatware:  it would be much easier to just label
drives "1", "2", "3", "4"... and put stuff wherever there is space,
KNOWING <something> will remember where I put it, for me!

Instead, I have to label drives:  "Commercial OS's 1", "Commercial OS's 2";
"MP3's 1", "MP3's 2"; "Research Papers 1", "Research Papers 2"; "COTS Apps 1",
"COTS Apps 2"; etc.

And, when one of these "groups" gets too big, having to create some
ARTIFICIAL partitioning of the content so I can move "half of it" to
another medium (and another medium 2)!

[The IT professional avoids these issues by just throwing everything in
one big, integrated store!  Depending on the FS support, even blurring
the physical boundaries between media!!  Just spend more money to
grow the solution.]

So, given that my needs are NOT UNIQUE (?), what's the off-the-shelf
(commercial or free) solution that addresses ALL of them?  Or, will
you resort to the consultants' deflection of any challenge:  *I* am
doing something "wrong" -- because I'm not doing it the way THEY
envisioned?  ("Fill in your name BEFORE you fill in your address.
And, do all of that before I will tell you what your order total
will be.  Because that's how I approached the problem in my mind!
Clearly, your desire to do otherwise is incorrect...")

On 31/10/16 12:43, Don Y wrote:
> 
> Wanna know the IT professionals approach to all this?  Install
> a giant filestore that is up 24/7.  Use a redundant technology
> (RAID, ZFS, etc.)  Copy EVERY medium onto it.  

OK so far - anything else is madness.

> Put everything
> under a VCS.

Everything that can sensible go under a version control system (all
development files, in particular), should be in a VCS system.  That is
not for redundancy, availability, etc. - it is so that you have control
of your development process, you can see who did what, when, and why,
and you can roll back changes as needed.

You may also want more than one VCS - different systems can have
different strengths and weaknesses, and you can't assume that one size
fits all.  Of course, you don't want more different systems than necessary.

>  Add scripts to automate scrubbing and the notifications
> outlined above.  

Such scripts will be part of a normal RAID installation.

> Install VCS clients on all hosts.  

For anything using the VCS, yes.

> Ensure all
> users are disciplined to check things in and out of the repository
> as required and WHEN required.

For anything using the VCS, yes.

Again, that is independent of your archiving or storage system - it is
basic good practice for development work.

>  Limit which users can access each
> object and perform actions on the repository.  Put in place procedures
> whereby staff can request additional capabilities for specific users.

No, give everyone root access to each machine, and let them use whatever
devices they want for whatever purpose.  Tell them it's fine to lend
company laptops to their kids.

Of /course/ you need procedures in place for giving the right people
access to the right systems!  You are trying to run a business, not a
stag party.


> Appoint someone to be responsible for each portion of the repository.
> Prohibit the use of laptops or other systems that can "move out of
> the view of said system" for indeterminate periods of time.  Treat
> any data outside of the system as "nonexistent"/unsupported.  etc.

Checking out something from a repository does not mean removing it from
the main system...

> 
> Budget a healthy amount to replace drives that are kept spinning
> despite only be accessed "rarely".  Increase budget for electricity
> to keep them all spinning -- along with additional cooling, etc.

A spinning disk takes a couple of watts.  If the electricity for that
shows up in your budget, you have /big/ problems.

And the lifetime of hard disks bears little relationship to the amount
of time they are kept spinning - each spin-up / spin-down cycle is worth
many hours or even days of running time for MTBF, depending on the
characteristics of the drive.

> Make sure the entire system is on a UPS that is aggressively maintained.

A UPS can be maintained quite peacefully.  The aggression I see here is
in your posts.

> Create multiple backups if disinclined to resort to alternative
> backup media.  Put a copy off-site.  Ensure adequate fire protection.

Seems reasonable to me.

> 
> Then, when Joe Employee calls looking for a file that he had
> LAST WEEK, shrug their shoulders and say they can't find it!
> [I've heard MANY folks make that same complaint re: their IT
> departments:  "They do backups EVERY night -- scheduled maintenance.
> We have to leave our computers on so they can do this, remotely!
> So, why can't they ever retrieve something that I 'lost'?  All
> they seem to ever do is REINSTALL WINDOWS!"]

Then try getting competent IT folk.  Snapshot backups are for recovering
lost files - user error is far and away the most common use for backups.

> 
> SWMBO would come home with similar horror stories from her
> colleagues almost weekly:  "They -- IT -- have got a frigging ARMY
> wandering around the place.  What the hell do they do if they can't
> even address this BASIC need?"  She quickly learned to keep her own
> backups archived locally to safeguard against their folly!

I can't quite figure out what you are running here.  Is it a home, or a
company, or an antique computer museum and graveyard?  Is your wife an
employee?  If not, why is she relevant to your company IT setup?

> 
> Yeah, I surely need a "professional IT department" to simplify
> my life -- NOT!  :>

From how you describe it, no profession IT folk would go near your
setup.  Scrap the lot of it is the only professional answer.  So that
leaves you as an amateur to play with your own systems in whatever way
you want, and to shout and rant at anyone who is willing to try and
offer useful suggestions.

Hi David,

On 10/31/2016 4:19 AM, David Brown wrote:
> On 29/10/16 22:15, Don Y wrote:
>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote:
>
> Don, let me try to understand the basis of your problem here.  Correct
> me at any point that I am wrong (I am writing in many small paragraphs
> to make that easier).  I have also added a few comments inline below my
> main post.
>
> 1. You have lots of files, held on computers, and you want to make sure
> they don't get changed or corrupted.

I want to ensure they "remain accessible".  I'm guarding against data loss.
Both for "old" data and "freshly minted" data.  I.e., wanting to ensure
the work I do TODAY is preserved without my having to take deliberate
steps to ensure it ends up "where it belongs".

> 2. You don't trust "metadata" methods such as sizes and timestamps,
> because files could be changed or corrupted without affecting these.

No, I don't trust *certain* metadata (notably timestamps).  I am
willing to rely on a dirent's "size" to be reflective of the files
actual size.  Whether I can *access* that many bytes is a different
issue -- something I can only determine by trying to access those
bytes.

As a medium can "fail" in such a way that the dirent is munged -- but
the data may still be intact (just not accessible via the normal
mechanisms), if the dirent says the file name is now "boogie"
instead of "myfile", then I have no choice but to assume "myfile"
is gone -- I'm not trying to implement a "data recovery service".

> 3. You don't want to read through the entire file, because they are big.

Correct.  And, data rates may be slow.  Imagine trying to "verify"
CD's at a few megabytes per second!

And, if I have information that suggests the file is NOT "as it should be",
then why undertake the effort to PROVE that it isn't?  "Gee, this file
is supposed to be 32GB but I stat(2) it at just 5KB!  Either the dirent
is munged -- the file really IS 32GB but the dirent is in error -- OR,
the file has been altered/replaced/truncated/etc."

> 4. You want some sort of algorithm for detecting file change or
> corruption, given 2 and 3.

Also, some way of verifying NON change/corruption.  There are many
ways to ask the question and different approaches to getting an
answer.  "Has ANYTHING changed, here?", on a 4TB drive, can yield
an answer in a few SECONDS -- if the first file you go looking for is
found to be absent (have a changed name) or of a different size.

OTOH, "is *everything* UNCHANGED, here" takes a lot longer to answer
in the affirmative; you have to perform all of the tests on all of the
objects!

> If that is the case, then you will be disappointed - no such algorithm
> could exist.

See above.

Think about the ways that "file collections" can be altered.  And, think
of the relative likelihoods of those changes!

By far, I suspect accident user actions are the biggest issue.  Someone
deletes something that they shouldn't have.  Or, moves something and
the operation is aborted without their noticing it (media full, etc.).

Similarly, if users are responsible for maintaining the file sets,
then its possible a user updates X and forgets to update Y.  Or,
X *instead* of Y.

Or, a buggy protocol implementation (an aborted automated transfer that
leaves a partial result)

After that, perhaps a total drive failure (e.g., not spinning up).

Somewhere down the list are local, uncorrectable errors that affect
some portion of the drive surface.  These would tend to increase
in frequency, be more widely distributed through the dataset and
PROBABLY be noticeable if someone is aggressively watching for them.

For example, I have noticed that cheap consumer drives (e.g., 4TB)
tend to have dramatically different performance/throughput as they
"get full".  Presumably, fewer opportunities for "reliable"
sectors to be encountered vs. faulty sectors accumulating at the
unused portion of the medium.

> So now we have to consider what would give you a convenient, economical
> and reliable way to store your files safely.
>
> First, I assume that your old existing systems are irrelevant except as
> a source to copy from.  Spending time and effort on maintaining outdated
> hardware is a waste - I expect you to set up a new system.

I've already pulled the files off "old media".  There are some
obsolescent media that I have no choice but to support:  e.g.
my DSO and EPROM programmer both rely on 3" floppy media.  To
work around that, I'd have to buy new equipment -- a silly
"optimization" (easier to keep a floppy drive operational AND
capture the images off the "factory media" before it degrades!

What is considered "outdated" is subject to personal opinion.
I see no reason to replace a machine that still runs "modern"
OS's -- even if faster machines are available.  With that
criteria, you'd have to update hardware yearly -- and update
all licensed software at the same rate!

> Second, I expect you to have off-site backups.  This could be done by
> duplicating the main system, or by simply having a number of off-site
> copies on DVD, archived hard disks, cloud storage, etc.

Nope.  I've satisfied all of my contractual/legal obligations so
I'm only "on the hook" for things that have value to me, personally.
There is some value to having "past projects" accessible as
references.  But, if the house burned to the ground, the LAST
thing I would worry about would be my possessions -- both
tangible and intellectual.

[As an aside, a routine exercise I perform is querying myself
re: how I'd handle such an event:  "If the house was on fire
and you could retrieve ONE item, what would that item be?"
My goal, for as long as I've been asking myself that question,
has been to be COMFORTABLE answering "nothing".  I have been
at that point for quite some time!  :>  Of course, that doesn't
mean I'd be eager to lose *anything*!!]

> Third, I expect you to leave behind your prejudices and knowledge of
> limitations of old systems, and to look forward.

I've only mentioned old systems to indicate how I got here.
How my archive maintenance philosophy has evolved to its
current set of practices WITHOUT foreknowledge of future
developments.

E.g., dealing with RAID5 and $1K drives and why RAID1 was
"an improvement" (for me) in that timeframe.  Then, why
"manual mirroring" was an improvement beyond that (no
special hardware OR software required).

And, using SCSI as a basis for removable disks before USB
existed to provide that capability (so, I can keep a stack
of SCSI disks "cold" on a shelf and swap them into a
system as needed instead of having ALL of them spinning
ALL THE TIME).

Then, the downside (discipline required) for these approaches.

Finally, how to avoid that aspect (by automating the process
taking advantage of more modern hardware and less expensive
components).

> Files change, get lost, or corrupted due to four main causes:
>
> 1. A user changes them in some way (intentionally or unintentionally).

Exactly.

> 2. Malware or hackers change them.

Not a problem, here.  My systems are offline.  In the odd chance that
something was "infected" and spread to items CURRENTLY on-line,
my solution would detect those changes by noting the differences
in file hashes during periodic "audits".

And, as most of my archive is offline at any given time, I'd probably
notice any widespread "corruption" BEFORE mounting yet another volume.

> 3. Bit rot or sector wear causes some data to be lost on a disk.

Yes.  Nothing I can do about that short of buying better quality
components.  In practice, I've not seen this to be a real problem
(noting the consumer drive issue mentioned above).  Other folks
have reported poor durability of optical media -- I've not found
that despite using "commodity media".  I've heard of horror
stories with disk failures but have been spared that problem
(touch wood!).

I suspect part of this is due to the fact that I distribute my work
among many different machines.  So, total power on hours tends to
stay low.  For machines and their peripherals.  I suspect I've only
a hundred hours on my "most used" laptop -- cuz I don't use laptops
unless traveling or away from home (in the field, customer site, etc.)

> 4. Serious hardware failure, such as a disk head crash losing the whole
> disk.

Along the same lines, I HAVE encountered a catastrophic failure due to
a *driver* bug!  I watched a disk get "scrambled" and assumed the
problem was a disk failure (cuz everyone claims these are so common!).
Then, mounted the bytewise backup of the same drive -- and saw *it*
get scrambled just as quickly!

[No, I don't buy the idea of two independant drives dying at the same
time -- esp when they've both been sitting on a shelf for most of
their lifetimes!  What's changed?  Ah, I updated the OS yesterday...
let's roll things back to the previous release and try that, again.
Dig out the MO disks to recreate the corrupted drives (painfully
slow as the media is inherently slower than winchesters).]

> I expect you to handle 1 and 2 mainly through good security and regular
> backups.  I strongly recommend using a filesystem or setup with cheap
> snapshots - then you can take them regularly.

The second isn't a problem.  The first always is, potentially.
I only access the archive AS the superuser -- to ensure "noone"
else can screw up (i.e., me in my non-privileged guise).  So,
everything is exposed to my potential folly.

For example, I recently pruned all the "foreign language" updates from
my MS update archive.  ARA, JAP, POR, etc.  Just save ENU!  Click.
Click.  Delete.  Done.

Ooops!  "GLB" isn;t an abbreviation for a localization but,
rather "common to all"!  (sigh)  Better go back and redownload that
stuff, again!

Even automating this leaves opportunity for screwups -- if I specify
the wrong criteria/options for a particular object (and the system
dutifully follows my ill-conceived directives!)

> There is a myth that hard disk sectors can get corrupted and return
> incorrect data without an error indication - the so-called "undetected
> read error".  In theory, this /can/ occur - but unless you are using the
> data quantities of CERN or Google, you will never see it in your
> lifetime (that's why I call it a myth).  You /might/ come across
> firmware errors in disks that give such undetected read errors, though
> it is highly unlikely.  "Unrecoverable read errors", on the other hand,
> are not as uncommon as one would like - sometimes a disk is simply
> unable to return the data you ask for, but it can at least inform you of
> the failure.

I'm not worried about disks misreporting data/errors.
Likewise, I'm not worried about hash collisions -- there's
no active adversary involved.  Sure, some random set of
events could THEORETICALLY result in all those magnetic
domains flipping to some OTHER combination of states
AND still yield the same hash.

But, the Sun will burn cold long before that happens!
I'm willing to "risk" accessing that file at some
later date and wondering why its completely munged -- yet
still hashing to the correct value!

> The general method to protect against failures is redundancy combined
> with systems to spot errors.  For storage, that is almost invariably
> done using RAID.

RAID is predicated on you wanting "online" verification of EVERY
data access.  And, semi-seamless correction of any failures that
turn up.  All happening at "regular disk speeds" on volumes that
are PROBABLY seeing some significant amount of traffic.

It would be foolish to keep such an array spinning 24/7/365 for
the *rare* access that might occur.  When was the last time you
needed to access the install media for application XYZ?  Do
you REALLY need it to be available in a fraction of a second?
Are you THAT impatient?

The last time I accessed my 1976 tax forms was... in 1977!  Do
I *need* to keep them?  No.  Should I discard them?  Why?  Are
those few magnetic domains THAT valuable to me that I need
to salvage them??

You can use the redundancy aspects of RAID *without* using a
RAID appliance.  You just trade-off performance for "accessibility"
and "performance".

> So your basic setup here is to have two drives with a RAID-1 mirror.
> Anything that is written to the mirror gets written to both drives.  If
> a sector read fails (despite the drive's extensive error checking and
> correcting codes), the RAID controller (hardware or software) will read
> the second copy from the other disk.  It will then re-write this correct
> data to the first disk, which the drive will map to a spare sector.  If
> a drive fails entirely, the controller will read from the other drive.

Yes, but you can adopt a similar strategy without having both drives
on-line at the same instant.  This exposes a window in which things
can go south.  But, they are "unlikely" to do so.

E.g., when I moved from RAID1 to JBOD, I simply copied every file to two
different places -- under two parallel mount points.  The two drives
are not sector-wise images of each other.  But, at the filesystem
level, they contain the same files in the same logical places in their
hierarchies.  I could close my eyes while you rearranged the drives
and I'd not be able to tell if you'd swapped them -- or not.

If you have a means of improving your confidence in a particular
instance of a file (i.e., if you record name, size, hash in a
reliable medium and then verify this when you later access
that instance), then you can use that instance in lieu of the
original instance in creating the "mirror" copy of the file.

E.g.,

# cp file /Archive/1
# md5 file > hash
# rm file
# md5 /Archive/1/file | cmp hash -
# cp /Archive/1/file /Archive/2/file

is effectively (to "many 9's") the same as:

# cp file /Archive/1
# cp file /Archive/2
# rm file

*if* the operations are reasonably close together in time.  A RAID
array cuts the time to microseconds.  But, realistically, would it
be much different if the RAID controller ran a delay loop for
some number of seconds between each write?  (obviously, throughput
would suffer!)

My current approach exploits this.  So, a file can be copied to a volume
and its "signature" reliably recorded.  Later, that instance can be
used (after verifying it against that stored signature) to create
another instance of the file on another medium.

And, both instances can be "verified" at any time (assuming they
are available, online, at the time) -- by double-checking the
"signature", size, name, etc.

> "Bit rot", or slow bit corruption on disk sectors, is prevented by
> scrubbing.  You regularly read through the entire array, for both disks.

Yes.  But, just reading doesn't PREVENT the problem.  A disk can
degrade to a point where scrubbing just keeps revealing errors.
There;s a point when a drive's reliability is not recoverable.
The more the utilization, the less ability to adapt to grown defects.

>   If a sector has had corruption on some bits, the drives ECC coding will
> usually let it correct the error automatically, and it will re-write the
> sector (with re-mapping if necessary) to refresh the data.  If the
> corruption is too great, the RAID controller will re-write the sector
> based on the second copy.

And, the same sort of algorithm can be applied to multiple copies of a file
accessed at different points in time!  RAID keeps wanting to do everything
"now".  It doesn't have a deep memory for errors -- fix it and forget it.

And, if *the* copy of *the* file becomes irrecoverably corrupted, its
not smart  enough to know that another copy might exist -- on this same
medium OR a different medium... possibly at the other end of a network
connection!

> Thus RAID keeps your data reliable, at least until a catastrophic
> failure (such as the whole server dying) - that's why you have backups.
>
> For more features, I would recommend using BTRFS or ZFS to handle your
> RAID mirror.  These filesystems give you additional checksumming (for
> protection against drive firmware errors, or extremely rare undetected
> read errors), as well as very cheap snapshots and other goodies.
>
> On the other hand, if you want the simplest "will work on anything, even
> old Linux systems or Windows systems" solution, then use Linux md RAID
> with metadata format 1.0.  This puts the RAID information at the end of
> the disk - all the rest of the disk appears /exactly/ as a normal
> non-raid disk, and can be read from anything else.

You can't then accommodate R/O media.  E.g., I want one instance of
a particular set of files to be the "Dell Windows 7 Reinstallation DVD"
that I have in my desk drawer.  I want the *other*, probably more
vulnerable copy of that file to be "Dell_Windows_7_Reinstallation_DVD.iso"
So, I can call up the ISO file relatively easily (power up a particular
disk drive and CD to the correct directory) and access its contents.

But, being on writeable media, its conceivable that this .iso file
can get corrupted, overwritten, replaced, etc.  So, I want to be able to
verify that it is intact.  And/or "recover" it -- from the original
medium!

Or, if the original medium gets munged (scratched), to be able to recreate
a NEW physical DVD from the ISO file -- which can later be used to verify
the ISO's integrity...

>>> Disclaimer: I am not an expert on the data storage, but here are my
>>> thoughts:
>>>
>>> - Use a RAID system which has enough redundancy to detect and correct
>>> defects in the media. Depending of the RAID level used, the capability to
>>> data recovery varies.
>>
>> I started out using RAID5 "hardware" arrays (and other, more sophisticated
>> configurations).  This requires you to have multiple spindles running
>> concurrently (else the array can't *detect* discrepancies).  Compact
>> arrays also tend to suffer from cooling problems.
>
> You will need two spindles (or SSDs) concurrently - that's the only way
> to get two copies of your data on two disks.

Yes, though you can cache those files that you know will need to
be coped to a yet-to-be-mounted volume.  You know what their signatures,
sizes, names are and SHOULD BE.

So, when the desired target volume is made available to you (perhaps
because you requested the user mount it on your behalf!), you can
"resume" the DEFERRED copy to that second spindle.  I.e., where you
cache the file need not be one of the spindles IN the archive.

> Don't bother with RAID-5.  Either use RAID-1 if your data fits
> comfortably on a single disk, or use RAID-6.  Modern disks have less
> cooling requirements than old ones, if you go for RAID-6.

I made the cooling point to indicate why LARGE RAID arrays (12 and
48 drives) were problematic when I was using them.  They wanted to
be compact -- but 20W drives packed side-by-side in a small cabinet
is a lot of BTUs to disperse.

[nowadays, you can buy disks that are 10 - 1000 times larger
so you;ve already cut the equivalent power-per-gigabyte-capacity
significantly]

>> It also ties you to an often proprietary (or, at least, poorly documented)
>> implementation ("volume format").  So, a failure of the hardware can leave
>> you with possibly intact volume(s) and no straightforward means of
>> accessing their contents.
>
> Use software RAID - either Linux, or BSD if you prefer.  Avoid
> proprietary hardware solutions.
>
>> Moving volumes between arrays is seldom supported (unless you have
>> identical
>> arrays -- hardware/software -- AND a well considered means of doing this.
>> This effectively precludes having volumes off-line.  Introducing a
>> "foreign"
>> volume to an array will often result in the array prompting you to
>> "initialize" it (wipe it clean!).
>>
>>> - Duplicate or mirror the data across different physical media.
>>
>> This is RAID1.  Many of the above issues still apply.  In any case,
>> the mirrors must be spinning concurrently for the array to function.
>> (It has no way of knowing what the "other" copy of the "file"
>> looks like unless it can *see* it, now, in its entirety).
>
> If you are really desperate to avoid having two disks spinning at the
> same time, it is possible with Linux md raid to designate one drive as
> "write mostly".  It will then only be used when writing (obviously),
> when scrubbing (obviously), or when reading if the first drive fails to
> return the correct data.  I cannot think of any real benefit of this,
> but it is possible.

I'm not averse to running multiple spindles.  I often have 2 or 4
external drives spinning so I can easily move a single set of
files onto "a volume and its 'mirror'".

But, I shouldn't NEED to have two or more spindles up at the same time!

There's already a DBMS running to do all the bookkeeping (similar to
what the RAID has to do).  If you overprovision this machine  -- by
even a small amount -- then you can set aside a staging area to cache
those files that you are adding or altering in the "system" -- even
if the actual medium supporting it isn't available, presently.

Because you have the DBMS accessible, you can make a notation for the
file that you will *ultimately* be updating:  "When I see this
volume hosting this file in the future, I need to replace it with
the file currently hiding in the 'pending file cache'".

At the same time, you can choose to notify the user that you;d
eventually like to have that volume mounted -- and why its important
to you!

And, if you ever have to purge the file from that cache, you still have
a record of the fact that an update is pending for the/those other
copies of it!  So, you can remind the user that the array is
still in an inconsistent (degraded) state.

I'm doing the same thing that a RAID1 array would do -- but allowing
the timescale to be more elastic.  Letting the user decide how
long he is willing to operate in those "degraded" states.

E.g., perhaps the file needs to be updated in *3* places!  You;ve
successfully updated the first, "original" (insomuch as any copy can
be considered the "original").  You cache a copy in the holding area.
And, tickle the user to mount the one or two volumes that host
these two other, degraded copies.

Eventually, you encounter one of those columes and surreptitiously
update the copy while the user is fetching some OTHER files off the
medium.  You update the database to record the fact that this
copy is now in agreement with the "original" and no longer in
need of an update.

And, you continue to cache the file as you wait for the OTHER
medium to come along for that third copy!

You continue to pester the user -- but, can now remind him that
he no loner is operating in *AS* DEGRADED a state as previously;
he now has TWO copies of the file available!  But, you'd
really like to get that third one updated, as well!

Regardless of how much time passes, you will continue to remember the
"state" of this file and its counterparts -- because the DBMS is persistent!

Some day, the file may disappear from the cache -- pushed out by some newer
"change" that represents a more serious degradation problem:  only ONE
instance of that new file exists in the array at this time!  (the previously
cached file has TWO copies available, despite needing a third update)

Some day, you may encounter one of these first two copies (of the first
file).  You can surreptitiously reacquire a copy to have a fresh cached
copy.  And, verify that it is intact -- in the hope that you might
encounter that third volume, RSN.

[See?]

>> This begs two crucial questions:
>
> No, it /raises/ two questions.
>
>> - are BOTH copies of the file examined when you try to access it?
>
> No.  That would be highly inefficient, and does not contribute to
> improved reliability.
>
>>    Or, is a "primary" copy accessed and the "secondary" copy only accessed
>>    if the primary throws a read error?
>
> Yes.

These were rhetorical questions intended to highlight the fact that
having two copies doesn't really mean what you think!  *If* you get
a requested file from the array, there is no guarantee that both
copies are intact.  No guarantee that the copy you got HAS a
"backup" in case it is lost.

And it highlights the fact that you can only get an ambiguous
proclamation of a file's integrity with two instances:
if they agree, then, chances are, they are intact.  If they
DISAGREE, you have no way of knowing which -- if either -- is
correct.

Realistically, the failures are one (or both) copies being inaccessible.
Underlying ECC makes the chances of both drives CLAIMING to have
reliable data and that data being DIFFERENT is virtually impossible.

For similar reasons, if one of my volumes returns "intact" data,
the hash is only there to verify the file's contents haven't been
altered "while the system wasn't paying attention"  (e.g., take
the external drive and mount it DIRECTLY on a machine for ease
of access.  Alter its contents willingly.  Then, reintroduce it
to the archive system -- which will have no knowledge of your
"offline" (as far as IT is concerned) modifications.  Until,
of course, the file is scrubbed and verified.

Because of this -- the fact that a drive can be accessed outside
of the control of the archive system -- I can have two files
that claim to be correct (no ECC errors, no RECORDED differences
in their hashes) yet are very different!

>> - If a discrepancy is found between the TWO instances, how is the
>>    "winner" selected?
>
> Discrepancies don't occur with non-negligible probability unless you
> have had a massive hardware failure, or power-failure during writing.
> See <http://neil.brown.name/blog/20100211050355> for a discussion on the
> topic.
>
>> Plus, the more subtle question seldom considered:
>> - are files that are not "manually" accessed EVER verified?  Or, do you
>>    have to wait until you need/want a file to discover if it is intact??
>
> That's what "scrubbing" is for.  Hardware raid controllers should do
> this automatically, with software raid it is usually done with a cronjob.
>
>>    (Do *you* periodically check the contents of the batteries in your
>>    flashlight to verify they are functioning BEFORE you need them?
>>    Does the *flashlight* take on the responsibility of performing this
>>    test on your behalf??)
>
> A torch is not a raid device.
>
>> Consider the above when you have many terabytes of data at stake.
>
> RAID like this is /exactly/ what people use when they have TB of data at
> stake.

As volume sizes increase, the problem of a second crash during a
rebuild becomes tangible.  The fact that rebuild times increase
as the sizes of the volumes increases leads to longer "windows of
vulnerability".

> When they have more than that, they use redundant servers, or split the
> two halves of their software RAID-1 pairs across different boxes.

which is what I am doing.  But, not REQUIRING the two halves to
be online concurrently.  The DBMS "vouches" for the missing party
(i.e., instead of comparing A to B, you compare A to a function of B
(which also happens to be the same function of A).  Then, later,
when A is gone, do the same with B.

>> Do you want ALL of it "spinning" whenever you need to access *any*
>> of it?
>
> Why is that a problem?  Most hard disks have no problem spinning all the
> time.

Cheap consumer disks tend not to be well cooled and effectively
disposable.  And, there's no reason to have them spinning when
you don't need to access them.

Do you leave your computers on all the time -- in case you might want to
walk up and do something "in an instant" without waiting for it to
boot?  Do you let your disk spin down after an idle period?  Do you
let drives idle at ~1W in case you might want them to spin up, now?
(so, if you have 50 such disks, you've got all that power being
dissipated keeping wall warts warm AND electronics connected to a
potential path for lightning strikes and line disturbances)

>> How long will it take for a proactive array to examine/verify
>> ALL of the data just to reassure you?  (will you have the ENTIRE
>> array "up" for that length of time?  Or, will you power it up just
>> long enough to access the file(s) of interest, then power it back down,
>> again?  Or, will you power up ONLY the portion of the array known to
>> contain the file(s) you seek?)
>
> Again, what is the problem?
>
> If this is an array you rarely access, you can spin down all the disks
> when it is not in use.

I power down (external) drives when not in use -- unplug them from the
AC mains.  If I'm going to use a higher performance drive that's part
of an array, I will often "pop" the unused drives out of that array
to reduce the cooling load and drive noise -- as well as wear and
tear on drives that aren't DOING anything besides "keeping warm".

>> As I said, I've been down this route.  Ever hear what twenty to
>> fifty 10-15K drives sound like spinning at the same time?
>> <http://techreport.com/blog/13849/behold-thumper-sun-sunfire-x4500-storage-server>.
>>   Even a 12-14 drive "shelf" is loud:
>> <http://www.amtecgroup.co.uk/products/external-disk-storag-1071/sun-storedge-a1000-12-bay-hdd-rack-array-raid-ctrl-2x-psu-1502.aspx>.
>>   Remember,
>> they're all throwing off heat in a relatively dense package so you
>> have lots of (redundant!) fans in place to keep them from melting...
>
> Why are you considering a decade old high-end datacenter storage system?
>   These are designed for installations where noise doesn't matter, and
> you have massive cooling systems.

Reread the explanation.  Its a box that I used more than 20 years ago.
Before there were small alternatives.  And, big disks.

> Just get a simple little server box with space for a few disks.  I
> really like the HP Microservers - Gen 8 devices are dirt cheap,
> reliable, and have four neat 3.5" drive bays behind a door in the front.
>   They are perfect for file servers unless you need really high-end
> performance.

Ive adopted the approach of using external drives -- so I can move
them, readily, between hosts, power them down individually, hotplug,
etc.  The cheap consumer drives are ~$25/TB.  I can afford to buy
two -- and replace one of EACH PAIR every year and still end up ahead!

I can put 4 drives on a single box and selectively power up
any combination of them.  No fans, no heat, etc.  I can easily get
20MB/s off the drives and over the wire.  So, I may have to wait
a fraction of a minute to get a CD image.  <shrug>  If I'm just
looking for individual files, I can't click fast enough to get ahead
of them!

>> The "problem" with these solutions is they are designed for high
>> performance.
>> Serving files continuously, typically, to an "organization".  And, with a
>> high degree of reliability/availability.
>
> So don't get one.

I haven't!

>> For an *archive*, you tend to need lots of capacity but comparably little
>> bandwidth!  (how *often* will you access its contents?  how quickly will
>> you need the file "delivered" to you?)  And, you can trade reliability and
>> availability for bandwidth and cost.  Do you keep your 2013 tax returns
>> in your top left desk drawer (where they are easily accessible)?  Or, have
>> you moth-balled them to a more capacious site that's, perhaps, a bit more
>> difficult to access??
>
> What capacity are you looking for here?

I can easily account for ~30TB of "consumer grade" external drives without
considering optical media "originals", "burned" and the slew of 500G and 1T
drives around here.  Or, the higher performance drives in the Live repository,
VM server, individual workstations, etc.

As I said, at $25/TB, its not worth the time to THINK about trimming
your archive; just shell out another $200 and buy another REDUNDANT 4TB
of capacity (i.e., 8TB).

[I have a LOT of stuff in my archive as it is SO much easier to access,
there, than on DVD spindles or in specific workstations/servers]

On 10/31/2016 6:20 AM, David Brown wrote:
> On 31/10/16 12:43, Don Y wrote:
>>
>> Wanna know the IT professionals approach to all this?  Install
>> a giant filestore that is up 24/7.  Use a redundant technology
>> (RAID, ZFS, etc.)  Copy EVERY medium onto it.
>
> OK so far - anything else is madness.

No.  It's not acceptable, to me, to leave everything online and accessible
given that it is rarely accessed.  It invites failure -- even if that
failure is just human error (cuz its so easy and "natural" to
access EVERYTHING)

>> Put everything
>> under a VCS.
>
> Everything that can sensible go under a version control system (all
> development files, in particular), should be in a VCS system.  That is
> not for redundancy, availability, etc. - it is so that you have control
> of your development process, you can see who did what, when, and why,
> and you can roll back changes as needed.

That doesn't apply to things that don't change.  What's the revision
history of this MP3 file?  Or, this technical journal?  Should I
try to mimic the history of third party applications -- ensure I
keep copies of every release so I can move back and forth among them?

Instead, I only track *my* development efforts and their history.

Products that I inherit or "embrace"/adopt I leave in the VCS they were
developed under -- assuming that information is part of the file dump
made available by those projects.  (its too painful to try to convert
from one to another)

> You may also want more than one VCS - different systems can have
> different strengths and weaknesses, and you can't assume that one size
> fits all.  Of course, you don't want more different systems than necessary.

I use one for "developed works" (documents, sources, schematics, layouts,
etc.) and another for "tools" (i.e., purchased applications).  So,
the tools VCS allows me to recreate my development environment at
any point in time while the "works" VCS lets me recreate my
development EFFORT at a point in time.

Both are necessary unless one doesn't change.

>>   Add scripts to automate scrubbing and the notifications
>> outlined above.
>
> Such scripts will be part of a normal RAID installation.

So, your RAID array lets you tell it to scrub .JPG's at a
different rate than .BMP's?  Or, different portions of
the file hierarchy at different rates?

>> Install VCS clients on all hosts.
>
> For anything using the VCS, yes.

And for those things that don't?

>> Ensure all
>> users are disciplined to check things in and out of the repository
>> as required and WHEN required.
>
> For anything using the VCS, yes.
>
> Again, that is independent of your archiving or storage system - it is
> basic good practice for development work.

Its associated with "development work".  Do you keep your photographs in
a VCS?  Your music?  Text books?  etc.

Note this paragraph was introduced with:
     "Wanna know the IT professionals approach to all this?"
They wouldn't consider whether they were storing photos and songs
OR "the company books".  One size fits all.  What other solution do
they have to offer?  ("You're responsible for your own files"?)

The point of my discussion is to note how the solution is not
paired with the problem.  Just like using high performance
drives that spin 24.7 -- and are accessed for minutes per month
is a silly solution.

But, hey, if you've got the money and the manpower to throw at it,
more power to you!  I've seen people paid to dig holes and fill them
back in!

>>   Limit which users can access each
>> object and perform actions on the repository.  Put in place procedures
>> whereby staff can request additional capabilities for specific users.
>
> No, give everyone root access to each machine, and let them use whatever
> devices they want for whatever purpose.  Tell them it's fine to lend
> company laptops to their kids.
>
> Of /course/ you need procedures in place for giving the right people
> access to the right systems!  You are trying to run a business, not a
> stag party.

An IT department would spend more energy trying to decide who should
NOT have access to some "department data" than they would trying to
facilitate the access of those who *should*.

I've seen companies where free standing computers were expensed just to
get around the rules of their own IT departments.  SWMBO tracked
multimillion dollar construction projects in an MSAccess database
that she designed -- while the IT department and outside vendors
tried (and failed) to develop similar functionality ofer the course
of several years.  When she retired, their ability to track those
projects disappeared.

[Did I mention that SWMBO was a secretary without any formal CS training?]

And, this for a ~5000 employee business that has been in existence for
~50+ years!

Companies get caught up in "territorial disputes" and NIMBY issues
instead of focusing on delivering product.
    "Do we really need all these controls?  What are they trying to
    accomplish?  What will YOU guarantee as a result of their adoption??"

>> Appoint someone to be responsible for each portion of the repository.
>> Prohibit the use of laptops or other systems that can "move out of
>> the view of said system" for indeterminate periods of time.  Treat
>> any data outside of the system as "nonexistent"/unsupported.  etc.
>
> Checking out something from a repository does not mean removing it from
> the main system...

I didn't say something was "checked out".  I said something was NOT UNDER
THE CONTROL OF.  I.e., that's YOUR problem, not ours!  Instead of
"How can we get these same sorts of protections WITHOUT these constraints?
Because we can't operate within those constraints (laptops are intended
to be portable, not tethered devices!)"

>> Budget a healthy amount to replace drives that are kept spinning
>> despite only be accessed "rarely".  Increase budget for electricity
>> to keep them all spinning -- along with additional cooling, etc.
>
> A spinning disk takes a couple of watts.  If the electricity for that
> shows up in your budget, you have /big/ problems.

Now that would depend on the type of disk, interface and how much
time it spends "active".

My FC-AL drives draw about 19W.  The same is true for my SAS drives.
I can replace the FC drives -- if I want to discard my Sun workstations
and the tools that are only supported on them.  I can simmilarly
replace the SAS drives if I want to dump the server that hosts my VM's.

But, that will mean finding another suitable box, installing the
software and porting all the VM's.  I can buy a lot of electricity for
the price of several hours (days?) of my time!

> And the lifetime of hard disks bears little relationship to the amount
> of time they are kept spinning - each spin-up / spin-down cycle is worth
> many hours or even days of running time for MTBF, depending on the
> characteristics of the drive.

Yes, and disks spin up and down based on how often they are used.
Power up drive and it WILL spin up.  Wait a few minutes before you
access it and it could have spun down.  Access it and it spins up,
again.  Pause for a few more minutes while you decided if there
is anything else you want/need and it spins down again.  Eventually
shut off the host and the drive spins back up for a final sync(1)
before spinning down again.

    "each spin-up / spin-down cycle is worth many hours or even days
    of running time for MTBF"

My SAS/FC-AL/SCA drives don't spin up/down.  They just "stay warm"
all the time (i.e., throw off BTUs)

>> Make sure the entire system is on a UPS that is aggressively maintained.
>
> A UPS can be maintained quite peacefully.  The aggression I see here is
> in your posts.

UPS's routinely need new batteries.  When you have lots of UPSs, you need
lots of batteries.  A more prudent approach is to consider the amount
of time a particular load will be powered up and the chances/consequences
of an uncontrolled shutdown along with the probability of such an
event.  Offset this with the cost of maintaining that "protection".

I insist on keeping my DNS/etc. up and running despite outages.
*This* machine, router, printer and modem will stay up ~2hrs in
an outage.  Most of the machines in the office won't stay up
more than 15 minutes.  And, if they do crash, will probably fix
themselves on the next boot.  Seldom is more than one running
at any given time.

So, anything beyond "glitch protection" is largely a waste, here.
And, at $50-$100/UPS for a new set of batteries (at discount),
throwing that away every 3 years is silly.

>> Create multiple backups if disinclined to resort to alternative
>> backup media.  Put a copy off-site.  Ensure adequate fire protection.
>
> Seems reasonable to me.

Do *you* have a halon extinguisher in YOUR bedroom?  Or, anywhere
in your house?

>> Then, when Joe Employee calls looking for a file that he had
>> LAST WEEK, shrug their shoulders and say they can't find it!
>> [I've heard MANY folks make that same complaint re: their IT
>> departments:  "They do backups EVERY night -- scheduled maintenance.
>> We have to leave our computers on so they can do this, remotely!
>> So, why can't they ever retrieve something that I 'lost'?  All
>> they seem to ever do is REINSTALL WINDOWS!"]
>
> Then try getting competent IT folk.  Snapshot backups are for recovering
> lost files - user error is far and away the most common use for backups.

And the reason IT departments should have effective backup procedures.
Yet, I can bump into any number of folks in different businesses
and industries and hear horror stories re: their IT departments and
"simple" things like this!

>> SWMBO would come home with similar horror stories from her
>> colleagues almost weekly:  "They -- IT -- have got a frigging ARMY
>> wandering around the place.  What the hell do they do if they can't
>> even address this BASIC need?"  She quickly learned to keep her own
>> backups archived locally to safeguard against their folly!
>
> I can't quite figure out what you are running here.  Is it a home, or a
> company, or an antique computer museum and graveyard?  Is your wife an
> employee?  If not, why is she relevant to your company IT setup?

I am a sole proprietor.  I am *my* IT department.

I also act as her "IT" department cuz she doesn't have the skillset to
maintain her machines.  (I also act as the IT department for many friends,
neighbors and colleagues who are similarly challenged).  The story
recounted above refers to her experiences over 20 years in an organization
with an (apparently) highly dysfunctional IT department.

>> Yeah, I surely need a "professional IT department" to simplify
>> my life -- NOT!  :>
>
>  From how you describe it, no profession IT folk would go near your

I'm not looking to HIRE a professional IT person.  I can't afford
their overhead.  And, they wouldn't be able to maintain much of my
equipment ("Hey, Joe, the attenuators in the DSO need to be recalibrated.
Can you get around to that this week?  And, see if I need to update
the ARM compiler or if I can continue using the existing one -- so
I don't have to go through the whole recertification process again
with the client.")

I'm looking for a cheap way of getting what I need without having
to change MY development/business style to coincide with one IMPOSED
by some other "solution".

Is there some TECHNICAL reason that my approach won't work?
Or, does it just offend your sensibilities?

> setup.  Scrap the lot of it is the only professional answer.  So that
> leaves you as an amateur to play with your own systems in whatever way
> you want, and to shout and rant at anyone who is willing to try and
> offer useful suggestions.

As expected, instead of answering the original question I ASKED, this
discussion has digressed into the realm of "why do you want to do that".
And, because my criteria differ from those others can wrap their heads
around, the fault is obviously mine.

So, I should just adopt the discipline of not straying from a strict
focus on my original question(s).  When asked "why", I should NOT be
considerate and offer an explanation; instead, "just because" should
suffice.  If you can't answer with that sort of rationale, then
you won't be able to answer with a *detailed* one.

"Mary has 3 oranges.  She gives one to Johnny.  How many oranges does
Mary have?"

"Why oranges?  Why not apples?  What kind of oranges?  Are they juice
oranges or eating oranges?  Are they ripe, yet?  Why did she give one
to Johnny?  Why *just* one?  Why not two?  Or, all three?  Is she sweet
on Johnny?  What about Bobby?  ..."

All the while missing the obvious answer:  two.

(sigh)  Makes me wonder if any of you can work from a specification
or if you need your hands held all the way from inception to deliverables!

On 31/10/16 21:36, Don Y wrote:
>> No, you're supposed to move with the times, and move your data to
>> a system that's replaceable,
> And that's EXACTLY what I have done!
> What should I pick to safeguard against NEXT year's obsolescence?

For development machines which you need to access at any time
into the future, you should use virtualisation - assuming they
don't have specific hardware devices that can't be virtualised.
All of the "final build" machines for our products were VMs,
and every released product version involved multiple backups
(both on-site and off-site) of the entire build machine's image.

A 20GB build image is peanuts when you can buy triple redundant
RAID6 (device and drives) for $100/TB, and it idles under 5W/TB.
That's $2 per archived build VM.

No need to maintain the physical hardware when it's virtualised.
For hardware development, I prefer to rely on USB dongles, because
(most) virtualisation products manage those seamlessly.

> OTOH, really frustrating if you've just wiped the CAD tools in favor
> of the software development tools -- only to discover you've one more
> "little thing" to attend to with those CAD tools!  :<

Exactly the problem for which virtualisation is a perfect fix.

> So, I went to RAID1 and picked up some capacity without spending
> any extra money.  (these would be termed "hardware RAID" as it
> was implemented *in* the disk box, not in the host(s))

The disk box probably still used software. Proper RAID controllers
would read all copies and ECC it on every read, and would ensure
that writes to all devices were flushed (both from the controller
and the HDD's internal cache) before allowing the write to be
considered complete. Now, there are standard interfaces to this
kind of synchronous behaviour, so software RAID can achieve the
same reliability on commodity hardware.

> So, the "shelf" just became the equivalent of a modern USB (bus).
> My FreeBSD and NetBSD boxes were tolerant of powering down the
> shelf and replacing drives without rebooting so it was excellent!

NFS is good like that, though it has downsides. Of course, you
can also replace drives in a modern RAID box without powering it
down. I had to do this as recently as four months ago.

> And, each change/addition meant more things to keep track of.

> Do I do this for EVERY medium that I have?  How else will I be
> able to browse through them without physically loading each,
> individually?  Sure would be nice if there was some sort of INDEX
> of "what was where".

You're describing the exact problem that motivated the development
of "git annex". As I said in my first response.

> But, it's MY experience that I evaluate, not urban legends or
> folks with different budgets/priorities!

Like I said, you are a "special flower". Your problems mostly
seem to stem from your decision to be this way - though of
course you don't want to see it as a decision.

> On the advice of others, I'd take my RCS repositories and convert
> them all to CVS (with a few man-years of effort on the "fixups").
> Then SVN, then Hg, Git, etc.  Next week, I'll have to see what's
> /en vogue/.

If you don't know why Hg & Git are fundamentally different from
their precursors, you should educate yourself. I also used RCS, SVN,
CVS with various custom improvements, but they were all fundamentally
broken compared to the modern equivalents. Just as ZFS is a fundamental
improvement over all previous distributed filesystems.

When you move to a content-addressable memory, everything changes.

> I can't tell The Boss to hire someone to tackle all this busywork
> so I can be free to "design".

Instead you've made a mountain of different busywork for yourself.

>>> Perhaps Clifford has the luxury of an IT department to fall back on.
>> Never have had, probably never will. My clients have, but not me.
> So *they* are carrying the cost of maintaining your tools in the long
> haul.

Surprise surprise. Banks run IT departments. So do stock exchanges.
So do the militaries of OECD nations (my software runs three of these).
They are (mostly) IT businesses. IT is what they *do* (though not
very efficiently, despite my best efforts). However, is that an
error you think I should persuade them to "fix"? Or just a reality
in which we live?

>>> Or, maybe he's got an eidetic memory and can remember and recreate
>> Pretty much, yes. See below.
> Ah, I'm jealous!
> I have to label things to know what's contained within.

So do I, because I'm usually not available when something needs
retrieving. Or if I am, I don't want to do it myself.

> Since my first CS class, I've been of the mind of letting machines
> do what they do best:  remember stuff and repetitive calculations.

I think it was Guy Steele who said "when the machines see us doing
repetitive tasks, they gather at night to joke and laugh about it."

> I've little/no desire to clutter my meatware with trivia

That's very wise. Contrary to the historical view, we have limited
memory, and I regret bothering with some of the bulky stuff I've
learned - and I advise younger folk to rely on off-line memory as
far as possible.

>> I am my own employer, yet I still occasionally get paid to assist
>> with code I wrote fifteen years ago. I can still tell you which
>> file and function to look (in three million lines of code)
>> for a given feature - from memory.

>>> The problem I outlined, here, was coming up with quick algorithms
>>> to highlight likely discrepancies WITHOUT exhaustively examining
>>> the contents of all of the files in a particular SET of files.
>> There is no such algorithm.
> Of course there is!
> What you can't know is whether APPARENTLY good metadata indicates
> unaltered file contents.  (reread my above statement:  "highlight likely
> discrepancies", not "likely integrity")

If you have an environment where files get modified unintentionally
(as opposed to being corrupted by hardware failure) then you're
irretrievably broken, and any amount of "double-entry book-keeping"
will only alert you to your broken process, it cannot fix it.

> I have an application that INSISTS any file that I open "has been
> altered" and offers to save the altered file FOR me.  Yet, I *know*
> the file has not been altered.  If I allow it to save the file,
> I will end up with the exact same file on my disk -- but with a
> different timestamp!

So use a VCS. Git and Hg both do repository-wide de-duplication of
files by content. That is, they run a "content store" - content-
addressable memory - and an almost-free infinity of "naming trees"
for the contents. "git annex" is a variant that aims to provide a
single coherent content index across multiple online and offline
storage devices, where any individual file may exist in multiple
copies. When you want something from the index, it either provides
it, or tells you what media to mount to retrieve it.

Isn't that what you really want here?

Clifford Heath.

On 31/10/16 16:02, Don Y wrote:
> On 10/31/2016 6:20 AM, David Brown wrote:
>> On 31/10/16 12:43, Don Y wrote:
>>>
>>> Wanna know the IT professionals approach to all this?  Install
>>> a giant filestore that is up 24/7.  Use a redundant technology
>>> (RAID, ZFS, etc.)  Copy EVERY medium onto it.
>>
>> OK so far - anything else is madness.
> 
> No.  It's not acceptable, to me, to leave everything online and accessible
> given that it is rarely accessed.  It invites failure -- even if that
> failure is just human error (cuz its so easy and "natural" to
> access EVERYTHING)

Harddisks fail on the shelf too.  I haven't seen statistics, but I have
seen it in practice.  Starting up disks that have been off for a long
time involves a certain risk, and clearly when the archive is offline
there is no way to check its integrity.

Most human error is easily avoided by making things read-only normally,
and only re-mounting read-write when you actually need to.  (There are,
of course, other forms of human error - it is bounded only by your
imagination!)

> 
>>> Put everything
>>> under a VCS.
>>
>> Everything that can sensible go under a version control system (all
>> development files, in particular), should be in a VCS system.  That is
>> not for redundancy, availability, etc. - it is so that you have control
>> of your development process, you can see who did what, when, and why,
>> and you can roll back changes as needed.
> 
> That doesn't apply to things that don't change.  What's the revision
> history of this MP3 file?  Or, this technical journal?  Should I
> try to mimic the history of third party applications -- ensure I
> keep copies of every release so I can move back and forth among them?

The main thing is to keep /your/ data and files in the VCS.  Fixed files
and third-party files with no history (at least, none of relevance to
you) don't have to go in a VCS.  But even then, it is often convenient.
 We regularly have things like important datasheets or tools as part of
a repository for projects.  It makes the repositories bigger than
necessary, but it means we can take a blank computer, do a check-out of
the project, and have everything in place.

> 
> Instead, I only track *my* development efforts and their history.
> 
> Products that I inherit or "embrace"/adopt I leave in the VCS they were
> developed under -- assuming that information is part of the file dump
> made available by those projects.  (its too painful to try to convert
> from one to another)

Sometimes that is the best, if there is useful history in the old VCS.
Sometimes it is best to transfer the latest version into your main VCS,
and leave the old one only for reference.

> 
>> You may also want more than one VCS - different systems can have
>> different strengths and weaknesses, and you can't assume that one size
>> fits all.  Of course, you don't want more different systems than
>> necessary.
> 
> I use one for "developed works" (documents, sources, schematics, layouts,
> etc.) and another for "tools" (i.e., purchased applications).  So,
> the tools VCS allows me to recreate my development environment at
> any point in time while the "works" VCS lets me recreate my
> development EFFORT at a point in time.
> 
> Both are necessary unless one doesn't change.
> 
>>>   Add scripts to automate scrubbing and the notifications
>>> outlined above.
>>
>> Such scripts will be part of a normal RAID installation.
> 
> So, your RAID array lets you tell it to scrub .JPG's at a
> different rate than .BMP's?  Or, different portions of
> the file hierarchy at different rates?

Why would I want to do that?  Do you think the sector storing a jpg file
is likely to decay at a different rate from a sector storing a bmp file?

A sensible time-frame for scrubbing is anywhere between a week and 3
months.  How long the scrub will take depends on the size of the array
and the priority you give it.  But even with a very big array and very
low priority, it will usually be done fine over a weekend.

> 
>>> Install VCS clients on all hosts.
>>
>> For anything using the VCS, yes.
> 
> And for those things that don't?

Then don't.  Was that a rhetorical question?

> 
>>> Ensure all
>>> users are disciplined to check things in and out of the repository
>>> as required and WHEN required.
>>
>> For anything using the VCS, yes.
>>
>> Again, that is independent of your archiving or storage system - it is
>> basic good practice for development work.
> 
> Its associated with "development work".  Do you keep your photographs in
> a VCS?  Your music?  Text books?  etc.
> 
> Note this paragraph was introduced with:
>     "Wanna know the IT professionals approach to all this?"
> They wouldn't consider whether they were storing photos and songs
> OR "the company books".  One size fits all.  What other solution do
> they have to offer?  ("You're responsible for your own files"?)

I use a straight file system, shared by samba, and with dated snapshots
(also accessible as straight file systems) for files that are not in a VCS.

> 
> The point of my discussion is to note how the solution is not
> paired with the problem.  Just like using high performance
> drives that spin 24.7 -- and are accessed for minutes per month
> is a silly solution.

Who said "high performance" drives?  I would recommend "red" or "NAS"
harddrives with high capacity per dollar ratings.  IMHO, if someone
thinks they need fast SAS harddrives, you have misunderstood the problem
or the solution - they are better off with more cheap drives (for
redundancy), SSD's (if speed really is that relevant), or more RAM (to
/actually/ give you greater speed in most cases).

> 
> But, hey, if you've got the money and the manpower to throw at it,
> more power to you!  I've seen people paid to dig holes and fill them
> back in!

The setup I would use would be about $200 for an HP Microserver, and
another $300 for 4 large SATA disks, plus perhaps an hour's work for
putting it together and installing the system.  But then, I am not an IT
consultant, and my IT management work is on the side of my main embedded
development work.

> 
>>>   Limit which users can access each
>>> object and perform actions on the repository.  Put in place procedures
>>> whereby staff can request additional capabilities for specific users.
>>
>> No, give everyone root access to each machine, and let them use whatever
>> devices they want for whatever purpose.  Tell them it's fine to lend
>> company laptops to their kids.
>>
>> Of /course/ you need procedures in place for giving the right people
>> access to the right systems!  You are trying to run a business, not a
>> stag party.
> 
> An IT department would spend more energy trying to decide who should
> NOT have access to some "department data" than they would trying to
> facilitate the access of those who *should*.
> 

Your logic seems to be "An IT department would recommend X.  All IT
departments are incompetent bordering on evil.  Therefore X is bad."  Is
that a fair summary?

I am sure there are IT departments, and IT managers, who consider
Dilbert to be their guide.  And maybe you have been unlucky and seen the
worst cases.  But I think most do what they can to provide a useful
service and let people access the data as they need.

> I've seen companies where free standing computers were expensed just to
> get around the rules of their own IT departments.  SWMBO tracked
> multimillion dollar construction projects in an MSAccess database
> that she designed -- while the IT department and outside vendors
> tried (and failed) to develop similar functionality ofer the course
> of several years.  When she retired, their ability to track those
> projects disappeared.

Using MSAccess is like holding your car together with chewing gum and
duct tape.  It may appear to work for a while, but it is not a solution.
 (That does not mean that the "professional" solution was any better -
it could easily have been far worse in a different direction.)

> 
> [Did I mention that SWMBO was a secretary without any formal CS training?]
> 
> And, this for a ~5000 employee business that has been in existence for
> ~50+ years!
> 
> Companies get caught up in "territorial disputes" and NIMBY issues
> instead of focusing on delivering product.
>    "Do we really need all these controls?  What are they trying to
>    accomplish?  What will YOU guarantee as a result of their adoption??"
> 
>>> Appoint someone to be responsible for each portion of the repository.
>>> Prohibit the use of laptops or other systems that can "move out of
>>> the view of said system" for indeterminate periods of time.  Treat
>>> any data outside of the system as "nonexistent"/unsupported.  etc.
>>
>> Checking out something from a repository does not mean removing it from
>> the main system...
> 
> I didn't say something was "checked out".  I said something was NOT UNDER
> THE CONTROL OF.  I.e., that's YOUR problem, not ours!  Instead of
> "How can we get these same sorts of protections WITHOUT these constraints?
> Because we can't operate within those constraints (laptops are intended
> to be portable, not tethered devices!)"

I don't follow what you are trying to say here.  Never mind, let's move
on - there is enough material in this post without worrying about it all.

> 
>>> Budget a healthy amount to replace drives that are kept spinning
>>> despite only be accessed "rarely".  Increase budget for electricity
>>> to keep them all spinning -- along with additional cooling, etc.
>>
>> A spinning disk takes a couple of watts.  If the electricity for that
>> shows up in your budget, you have /big/ problems.
> 
> Now that would depend on the type of disk, interface and how much
> time it spends "active".

Then let's be generous and say 3 W, taking the SATA interface into account.

Note that I am talking about /new/ big, normal-speed SATA drives - not
ancient high-rev parallel SCSI devices.

> 
> My FC-AL drives draw about 19W.  The same is true for my SAS drives.
> I can replace the FC drives -- if I want to discard my Sun workstations
> and the tools that are only supported on them.  I can simmilarly
> replace the SAS drives if I want to dump the server that hosts my VM's.

My recommendation is to dump the old hardware, unless you really need it
for support of particular old projects.

> 
> But, that will mean finding another suitable box, installing the
> software and porting all the VM's.  I can buy a lot of electricity for
> the price of several hours (days?) of my time!

You don't need VMs for a file server or VCS server, nor do you need any
software beyond the installation CD/USB for Debian, Red Hat, Ubuntu
Server, FreeBSD, or whatever suits your preferences.  And if you /do/
want VMs, Linux containers (or FreeBSD jails) is often plenty - and
available out of the box, for a cost of almost nothing in disk space and
ram.


> 
>> And the lifetime of hard disks bears little relationship to the amount
>> of time they are kept spinning - each spin-up / spin-down cycle is worth
>> many hours or even days of running time for MTBF, depending on the
>> characteristics of the drive.
> 
> Yes, and disks spin up and down based on how often they are used.
> Power up drive and it WILL spin up.  Wait a few minutes before you
> access it and it could have spun down.  Access it and it spins up,
> again.  Pause for a few more minutes while you decided if there
> is anything else you want/need and it spins down again.  Eventually
> shut off the host and the drive spins back up for a final sync(1)
> before spinning down again.
> 
>    "each spin-up / spin-down cycle is worth many hours or even days
>    of running time for MTBF"
> 
> My SAS/FC-AL/SCA drives don't spin up/down.  They just "stay warm"
> all the time (i.e., throw off BTUs)
> 
>>> Make sure the entire system is on a UPS that is aggressively maintained.
>>
>> A UPS can be maintained quite peacefully.  The aggression I see here is
>> in your posts.
> 
> UPS's routinely need new batteries.  When you have lots of UPSs, you need
> lots of batteries.  A more prudent approach is to consider the amount
> of time a particular load will be powered up and the chances/consequences
> of an uncontrolled shutdown along with the probability of such an
> event.  Offset this with the cost of maintaining that "protection".
> 
> I insist on keeping my DNS/etc. up and running despite outages.
> *This* machine, router, printer and modem will stay up ~2hrs in
> an outage.  Most of the machines in the office won't stay up
> more than 15 minutes.  And, if they do crash, will probably fix
> themselves on the next boot.  Seldom is more than one running
> at any given time.
> 
> So, anything beyond "glitch protection" is largely a waste, here.
> And, at $50-$100/UPS for a new set of batteries (at discount),
> throwing that away every 3 years is silly.
> 
>>> Create multiple backups if disinclined to resort to alternative
>>> backup media.  Put a copy off-site.  Ensure adequate fire protection.
>>
>> Seems reasonable to me.
> 
> Do *you* have a halon extinguisher in YOUR bedroom?  Or, anywhere
> in your house?

I have off-site backup copies of the data.  That protects the /data/
from fire, which is the whole point.

And I don't keep my server in my bedroom - it lives in the cellar.

> 
>>> Then, when Joe Employee calls looking for a file that he had
>>> LAST WEEK, shrug their shoulders and say they can't find it!
>>> [I've heard MANY folks make that same complaint re: their IT
>>> departments:  "They do backups EVERY night -- scheduled maintenance.
>>> We have to leave our computers on so they can do this, remotely!
>>> So, why can't they ever retrieve something that I 'lost'?  All
>>> they seem to ever do is REINSTALL WINDOWS!"]
>>
>> Then try getting competent IT folk.  Snapshot backups are for recovering
>> lost files - user error is far and away the most common use for backups.
> 
> And the reason IT departments should have effective backup procedures.
> Yet, I can bump into any number of folks in different businesses
> and industries and hear horror stories re: their IT departments and
> "simple" things like this!

You don't need to tell me about what other people get wrong - you just
need to get it right yourself.

> 
>>> SWMBO would come home with similar horror stories from her
>>> colleagues almost weekly:  "They -- IT -- have got a frigging ARMY
>>> wandering around the place.  What the hell do they do if they can't
>>> even address this BASIC need?"  She quickly learned to keep her own
>>> backups archived locally to safeguard against their folly!
>>
>> I can't quite figure out what you are running here.  Is it a home, or a
>> company, or an antique computer museum and graveyard?  Is your wife an
>> employee?  If not, why is she relevant to your company IT setup?
> 
> I am a sole proprietor.  I am *my* IT department.
> 

I thought IT departments were evil and incompetent?

I still don't know if you have a company or a computer museum at home.

> I also act as her "IT" department cuz she doesn't have the skillset to
> maintain her machines.  (I also act as the IT department for many friends,
> neighbors and colleagues who are similarly challenged).  The story
> recounted above refers to her experiences over 20 years in an organization
> with an (apparently) highly dysfunctional IT department.
> 
>>> Yeah, I surely need a "professional IT department" to simplify
>>> my life -- NOT!  :>
>>
>>  From how you describe it, no profession IT folk would go near your
> 
> I'm not looking to HIRE a professional IT person.  I can't afford
> their overhead.  And, they wouldn't be able to maintain much of my
> equipment ("Hey, Joe, the attenuators in the DSO need to be recalibrated.
> Can you get around to that this week?  And, see if I need to update
> the ARM compiler or if I can continue using the existing one -- so
> I don't have to go through the whole recertification process again
> with the client.")
> 
> I'm looking for a cheap way of getting what I need without having
> to change MY development/business style to coincide with one IMPOSED
> by some other "solution".
> 
> Is there some TECHNICAL reason that my approach won't work?
> Or, does it just offend your sensibilities?

I didn't realise you /had/ an approach to the problem.  I thought you
were just asking here for the impossible, then complaining about what
"IT professionals" would do instead, and finding arguments against any
useful suggestions you got.

But if you have figured out what you need to do for your system, that's
great.

> 
>> setup.  Scrap the lot of it is the only professional answer.  So that
>> leaves you as an amateur to play with your own systems in whatever way
>> you want, and to shout and rant at anyone who is willing to try and
>> offer useful suggestions.
> 
> As expected, instead of answering the original question I ASKED, this
> discussion has digressed into the realm of "why do you want to do that".

Well, yes - that is how people usually help others in a newsgroup.  We
give suggestions about alternative approaches.  This is particularly the
case when the OP appears to be asking for the impossible ("How do I
check the integrity of an entire file without accessing the entire
file?").  Whenever someone has specific ideas that seem hard, the best
approach is to take a step back and try to look at the whole picture and
see things from a different angle.

This is an open newsgroup.  I am not a professional consultant that you
have hired to consider your setup in detail and attempt to some up with
a new archiving method to suit your needs.  I am someone who knows more
than most in this group about storage systems, and does more IT
administration than most in this group, and I am sharing some ideas from
that.  Maybe they help you, maybe not - and maybe the discussions that
people have here are of interest to others and not you.

> And, because my criteria differ from those others can wrap their heads
> around, the fault is obviously mine.

There are a number of issues here.  Your descriptions of your criteria
have changed under way, giving more information that was relevant from
the start.  Your writing about it is very verbose and unstructured - it
is extremely difficult to get a good understanding of what you have and
what you need.  You have a very strong idea that you are different from
everyone else - that your needs are different, and solutions are
different - and you prefer to stick to that uniqueness concept instead
of listening to how other people handle the storage and considering how
you could use known working solutions.

(I realise I am not being entirely complementary to you here.  I have
great respect for your knowledge and experience - but I think it is best
to be honest about how you appear in this thread.)

> 
> So, I should just adopt the discipline of not straying from a strict
> focus on my original question(s).  When asked "why", I should NOT be
> considerate and offer an explanation; instead, "just because" should
> suffice.  If you can't answer with that sort of rationale, then
> you won't be able to answer with a *detailed* one.
> 
> "Mary has 3 oranges.  She gives one to Johnny.  How many oranges does
> Mary have?"
> 
> "Why oranges?  Why not apples?  What kind of oranges?  Are they juice
> oranges or eating oranges?  Are they ripe, yet?  Why did she give one
> to Johnny?  Why *just* one?  Why not two?  Or, all three?  Is she sweet
> on Johnny?  What about Bobby?  ..."
> 
> All the while missing the obvious answer:  two.

If there was an obvious "answer" here, you would have found it yourself
and not started the thread in the first place.

> 
> (sigh)  Makes me wonder if any of you can work from a specification
> or if you need your hands held all the way from inception to deliverables!

I obviously can't answer for others, but /I/ can work from a
specification.  But this is not development from specification - this is
a brainstorming meeting from before the start of the project.

On 10/31/2016 3:20 PM, Clifford Heath wrote:
> On 31/10/16 21:36, Don Y wrote:
>>> No, you're supposed to move with the times, and move your data to
>>> a system that's replaceable,
>> And that's EXACTLY what I have done!
>> What should I pick to safeguard against NEXT year's obsolescence?
>
> For development machines which you need to access at any time
> into the future, you should use virtualisation - assuming they
> don't have specific hardware devices that can't be virtualised.
> All of the "final build" machines for our products were VMs,
> and every released product version involved multiple backups
> (both on-site and off-site) of the entire build machine's image.
>
> A 20GB build image is peanuts when you can buy triple redundant
> RAID6 (device and drives) for $100/TB, and it idles under 5W/TB.
> That's $2 per archived build VM.

Did you read my list of hosts?
"- VM server (dual wide SCSI support, HVD SCSI support,
               4T+system disk spinning)"
And, that it needs more performant drives than the "transfer a hundred
megabytes per month" portions of the archive?

[Or, that those other portions of the archive need LESS performance?]

> No need to maintain the physical hardware when it's virtualised.
> For hardware development, I prefer to rely on USB dongles, because
> (most) virtualisation products manage those seamlessly.



>> And, each change/addition meant more things to keep track of.
>
>> Do I do this for EVERY medium that I have?  How else will I be
>> able to browse through them without physically loading each,
>> individually?  Sure would be nice if there was some sort of INDEX
>> of "what was where".
>
> You're describing the exact problem that motivated the development
> of "git annex". As I said in my first response.

git annex doesn't address ALL of my requirements, as I said in a
previous response.  Paraphrasing your prior dismissal of my comments
re: git annex:
    I assume that you thought it was quicker to respond like this
    than it was to, for example, go and actually read about "my
    requirements"?

>> But, it's MY experience that I evaluate, not urban legends or
>> folks with different budgets/priorities!
>
> Like I said, you are a "special flower". Your problems mostly
> seem to stem from your decision to be this way - though of
> course you don't want to see it as a decision.

Everyone is bound by their decisions.  I could have opted NOT to
pursue hardware design activities, "photo-ready" document
preparation, specification writing, prototype fabrication, etc.
and JUST invested in this year's state-of-the-art PC and a
bunch of FREE software.

I have friends and colleagues who ONLY design analog circuitry
(and don't concern themselves with board layout beyond an advisory
role).  Or, purely digital designs.  Or, just power supplies.
Or, just web sites.  Or just desktop apps.

The folks who design power supplies don't have any compilers or VCS
to maintain; just create a folder for Rev 1 and another for Rev 2, etc.

The folks who write desktop apps don't have to worry about maintaining
test equipment or the tools/firmware to maintain them.

The folks who design web pages don't need tools to debug the virtual memory
management routines they've (not!) written.

Folks who produce electronic documents don't care if the RED they've chosen
for a particular highlight is Pantone 185C or 199C.  And, when they view
their "product" tomorrow, won't even be able to tell if the color they
are seeing is the same as the color they saw yesterday (even though
the document hasn't changed).  And, when someone sends the document
to a service bureau for printing, they'll wonder why the colors aren't
quite what they expected.

Similarly, the folks who designs embedded motor controllers has no interest
in humidity control devices, algorithms, data, etc.

But, my interests span all of these endeavors.  So, I have a much wider
variety of kit and "tools"/references that I have accumulated and rely upon.

So, it's "my fault" that my interests lie in these many areas and
require so much more than a typical "9-to-5'er" to support.

>> On the advice of others, I'd take my RCS repositories and convert
>> them all to CVS (with a few man-years of effort on the "fixups").
>> Then SVN, then Hg, Git, etc.  Next week, I'll have to see what's
>> /en vogue/.
>
> If you don't know why Hg & Git are fundamentally different from
> their precursors, you should educate yourself.

Where did I say that?  What I said was the folly of *converting*
an existing repository.  How much do I "gain" vs. the investment
it will take?  Sure seems simpler to just maintain project X
under the same VCS that was used to CREATE it!
    "Hi, client.  I'll need a few manweeks to convert the code
    repository that *I* created for you to a new VCS.  You
    should pay me for this effort as it is to your advantage!"

    "WTF?  *You* chose that scheme 10 years ago.  Why should *I* pay
    you for a bad decision on YOUR part?"

> I also used RCS, SVN,
> CVS with various custom improvements, but they were all fundamentally
> broken compared to the modern equivalents. Just as ZFS is a fundamental
> improvement over all previous distributed filesystems.
>
> When you move to a content-addressable memory, everything changes.
>
>> I can't tell The Boss to hire someone to tackle all this busywork
>> so I can be free to "design".
>
> Instead you've made a mountain of different busywork for yourself.

My past experience suggests I am way ahead of the game maintaining
this much data for this long without loss.  I was discussing a problem
with a colleague yesterday and recollected some code I had written
many years ago.

To continue that conversation, I went looking for the sources.
And, had it -- despite it being authored 35 years ago.

At the same time, found some code I'd written in 1970.  And,
some of the dataset that the 1970 code was intended to
process.

No, I wasn't looking for the 1970 code/dataset.  But, *was* looking
for the other item; the 1970 stuff was just an amusing bonus
discovery.

>>>> Perhaps Clifford has the luxury of an IT department to fall back on.
>>> Never have had, probably never will. My clients have, but not me.
>> So *they* are carrying the cost of maintaining your tools in the long
>> haul.
>
> Surprise surprise. Banks run IT departments. So do stock exchanges.
> So do the militaries of OECD nations (my software runs three of these).
> They are (mostly) IT businesses. IT is what they *do* (though not
> very efficiently, despite my best efforts). However, is that an
> error you think I should persuade them to "fix"? Or just a reality
> in which we live?

Yet, you seem to think *I* should assume similar costs, here?
Hmmm... I guess I can put the Halon fire suppression system in
the office and make sure I keep the rest of the house closed
off to contain its effects if activated...

>>>> The problem I outlined, here, was coming up with quick algorithms
>>>> to highlight likely discrepancies WITHOUT exhaustively examining
>>>> the contents of all of the files in a particular SET of files.
>>> There is no such algorithm.
>> Of course there is!
>> What you can't know is whether APPARENTLY good metadata indicates
>> unaltered file contents.  (reread my above statement:  "highlight likely
>> discrepancies", not "likely integrity")
>
> If you have an environment where files get modified unintentionally
> (as opposed to being corrupted by hardware failure) then you're
> irretrievably broken, and any amount of "double-entry book-keeping"
> will only alert you to your broken process, it cannot fix it.

You've NEVER deleted a copy of a file that you shouldn't have?
[And, of course, your RAID array promptly deleted it's "redundant
copy" as you obviously intended.]

When I make such a mistake, I curse and immediately retrieve a spare
copy to restore the deleted copy -- regardless of the machine or appliance
that I used in the crime.

And, as I've previously commented, I've had driver failures (FOSS)
that cost me entire volumes.  But, was able to recover because
my "manual procedures" protect me from these sorts of software
faults.  (Had both volumes been the ONLY volumes holding the
data in a RAID appliance, it could have been irretrievably lost.)

>> I have an application that INSISTS any file that I open "has been
>> altered" and offers to save the altered file FOR me.  Yet, I *know*
>> the file has not been altered.  If I allow it to save the file,
>> I will end up with the exact same file on my disk -- but with a
>> different timestamp!
>
> So use a VCS. Git and Hg both do repository-wide de-duplication of
> files by content. That is, they run a "content store" - content-
> addressable memory - and an almost-free infinity of "naming trees"
> for the contents. "git annex" is a variant that aims to provide a
> single coherent content index across multiple online and offline
> storage devices, where any individual file may exist in multiple
> copies. When you want something from the index, it either provides
> it, or tells you what media to mount to retrieve it.

The "Data Management Professional" approach:  treat all problems
with the same remedy.

> Isn't that what you really want here?

No.  I want something that watches the integrity of my data
and lets *me* decide on policy on an object-at-a-time basis.

I have hundreds of photographs.  If I lost most of them, I'd
just shrug.  I typically just saved them as it was easier to
save them than audit which I *should* save.

OTOH, I have other photos that are parts of publication sources.
I'd surely not like to lose them as I may not be able to
recreate such a photo -- the thing I photographed may no longer
be available to me.

If I am likely to lose the photograph by a careless act of
my own, then keeping a second copy on the same volume -- but in a
different part of the filesystem hierarchy -- is sufficient to
protect against *that* possibility.

OTOH, if I fear a volume failing (recall: just 2 in 40 years),
then I'd want another copy on a different medium -- likely
one that is NOT online (or needn't be).

I have gobs of data in my development database that I snapshot
daily, alternating between two different volumes.  It's
important enough that I'd like to know if it has been
corrupted or silently manipulated in any way; but, I can't
afford to be buying 1TB per month (RAID) just to have a deep
history.  If I want to restore a snapshot, all I need to
do is verify the hashes of the files to be reasonably
confident that everything is as it was.  If not (e.g., disk
failure), I have to move a day forward or backward to the
alternate volume and use that instance, instead.

*I* decide the value of the data that I am preserving.
And, I decide the value of data that I may be called upon
to recover; if I wipe a disk that had something I would have
liked to keep, I can go looking for another copy IF I had
the foresight to think it that important.

E.g., only one instance of those 1970 datasets exists.
They were worth keeping -- vs. discarding outright -- but
not worth so much as to double the amount of storage it
would take to preserve them "back then" (scans of hundreds
of handwritten log sheets).  And, I'm unlikely to revisit
that decision today (even though it would just be a
"cp 1970_dataset 1970_dataset_backup") despite the fact
that media costs 40 years later are several orders of
magnitude cheaper!

OTOH, I have probably half a dozen copies of my business
financial records on a variety of media!  Not to guard
against audits but, rather, as a convenient place to
"remember" certain facts (when did I buy this? who was
my contact at XYZco? etc).

---

As this conversation has spiraled far from the original
question of file metadata and shortcuts to obtain information
about file integrity, I'm out...