On 10/29/2016 6:00 PM, Clifford Heath wrote:> On 30/10/16 10:19, Don Y wrote: >> On 10/29/2016 3:47 PM, Clifford Heath wrote: >>> On 30/10/16 07:15, Don Y wrote: >>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote: >>>>> Disclaimer: I am not an expert on the data storage, but here are my >>>>> thoughts: >>>>> - Use a RAID system which has enough redundancy to detect and correct >>>> I started out using RAID5 "hardware" arrays (and other, more >>>> sophisticated >>>> configurations). This requires you to have multiple spindles running >>>> concurrently (else the array can't *detect* discrepancies). Compact >>>> arrays also tend to suffer from cooling problems. >>>> >>>> It also ties you to an often proprietary (or, at least, poorly >>>> documented) >>>> implementation ("volume format"). So, a failure of the hardware can >>>> leave >>>> you with possibly intact volume(s) and no straightforward means of >>>> accessing their contents. >>> >>> No. Almost all commercial RAID systems (Synology, Qnap, etc) >>> use Linux RAID support. If the hardware fails, you can mount >>> the drives on any Linux system that has enough SATA ports. >> >> Really? So my Sun RAID systems run Linux?? > > Ahh, my apologies. I didn't realize you were running a computer antiques > museum. I thought you were trying to ensure the availability of some data.Was I supposed to see into the future and deliberately NOT store my data on SUN and DEC hard disks until Linus got around to creating a viable solution? And, then, for someone to commercialize that? "Don, don't buy those $1K 4G drives. Just wait another decade or two and you'll be able to buy 4T drives for 10% of the price! Don't use Sun's equipment cuz they will eventually go out of business. Instead, wait around and MAYBE some kid will write some FREE software that you might EVENTUALLY be able to use in someone ELSE's box (and we KNOW *they* won't go out of business, either!)" [Hmmm... I'm getting a vision... of quantum computers and holgraphic storage matrices! Get rid of all this sill electronics and just twiddle your thumbs until the vision becomes a reality!!] I had commercial products in the market for more than a decade -- *before* Torvalds or Joy (or Stallman) decided what they wanted to do with their lives. Should those products have been delayed until they had their "solutions" available? Should I have discarded all of that history because YOUR solution wasn't available to me? I should be prescient and KNOW that someone will make a RAID server that *now* allows volume portability (seems like that sort of prescience would be more valuable to apply to picking REAL investments, not computer software and hardware that depreciate over time!) If you CAREFULLY read my posts, you will note that I no longer run RAID arrays. So, I have no need to plug a SUN (or DEC or Adaptec or Dell) RAID volume into a Linux-based appliance. Or, a GuardianOS based box. Or, ... Even the hardware RAID controllers in my machines are configured as JBOD's. None of *them* run Linux in their firmware, either! Crash a 4T drive in one of your RAID arrays. Let me know when (if!) it ever finishes rebuilding the array -- before one of the companion 4T drives dies a similar death! (Hey, you have FAITH in your RAID array, right? So, you've got NOTHING to lose by pulling the drive, dd(1)-ing /dev/zero to it, and then reinstalling it in the array! You should be able to do that TONIGHT!) Linux-weenies are just as narrow minded as Windows-weenies. They think the world is -- and always HAS BEEN -- centered solely around Linux! With a bit more of a historical perspective, you will begin to see the ever-repeating patterns that have persisted in the hardware and software industries. And, how the same "problems" keep reappearing each time someone THINKS they've invented a new solution -- that's really a rehash of something old: "Gee, why did we STOP doing it THIS WAY?? Oh. Yeah. Nevermind."
Checking large numbers of files
Started by ●October 27, 2016
Reply by ●October 30, 20162016-10-30
Reply by ●October 30, 20162016-10-30
On 10/29/2016 7:05 PM, Dimiter_Popoff wrote:> On 30.10.2016 г. 03:00, Clifford Heath wrote: >> On 30/10/16 10:19, Don Y wrote: >>> On 10/29/2016 3:47 PM, Clifford Heath wrote: >>>> On 30/10/16 07:15, Don Y wrote: >>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote: >>>>>> Disclaimer: I am not an expert on the data storage, but here are my >>>>>> thoughts: >>>>>> - Use a RAID system which has enough redundancy to detect and correct >>>>> I started out using RAID5 "hardware" arrays (and other, more >>>>> sophisticated >>>>> configurations). This requires you to have multiple spindles running >>>>> concurrently (else the array can't *detect* discrepancies). Compact >>>>> arrays also tend to suffer from cooling problems. >>>>> >>>>> It also ties you to an often proprietary (or, at least, poorly >>>>> documented) >>>>> implementation ("volume format"). So, a failure of the hardware can >>>>> leave >>>>> you with possibly intact volume(s) and no straightforward means of >>>>> accessing their contents. >>>> >>>> No. Almost all commercial RAID systems (Synology, Qnap, etc) >>>> use Linux RAID support. If the hardware fails, you can mount >>>> the drives on any Linux system that has enough SATA ports. >>> >>> Really? So my Sun RAID systems run Linux?? >> >> Ahh, my apologies. I didn't realize you were running a computer antiques >> museum. I thought you were trying to ensure the availability of some data. > > So what is he supposed to do with data which are useless outside of > a particular environment. Apparently he has plenty of past projects > spanning a few decades back to maintain - how do you suggest he will > separate movable from non-movable data. Then the sheer amount of data > he is talking is just huge.No, its much simpler than that: I am supposed to foresee the future. I am supposed to NOT purchase a CP/M system -- because, someday, there will be an IBM PC! I am not supposed to purchase DOS -- because, someday, there will be a Windows. I am not supposed to purchase a Windows -- because, someday, there will be a 386BSD. I am not supposed to purchase a 386BSD -- because, someday, there will be an OpenBSD/NetBSD/freeBSD/Linux. And, definitely don't buy expensive CAD/EDA applications -- because, someday, they MIGHT come up with something that's almost good enough to do what you need! [Hey, maybe I should not adopt any of those -- because, someday, there will be SOMETHING ELSE!] Perhaps Clifford has the luxury of an IT department to fall back on. Or, clients/employers with really deep pockets. Or, clients/employers who create very "ephemeral" products (no prolonged support nor liability issues). Or, maybe he's just designing web pages with the language-du-jour. Or, maybe he's got an eidetic memory and can remember and recreate how the schematics, artwork, specifications, contracts, software, test suites and other materials that he used to create Project X for Client Y in Year Z. Or, maybe he just chuckles and walks away from them! :> I, OTOH, take a keen interest in every project I undertake. And, a high degree of personal pride in delivering a quality product; not just cashing a paycheck and walking away. Or, dismissing the effort on the assumption that "someone is going to come along and CHANGE IT afterwards, regardless. So, in 1970, I had to decide how I was going to preserve the software I had written (though not commercially). And, the software for the first commercial product I designed (while an employee). And the patent proofs for each client after I went into business for myself. And the test suites that I used to defend the quality of my work, etc. Meanwhile, poor Linus was still thinking about what he was going to do with his life. And Stallman was still moping around LCS. And, the folks at Synology & Qnap hadn't even considered building RAID devices -- cuz they didn't have any software they could LEVERAGE for that job. [I was running SysV "at home" 5 years before Linus started *working* on Linux! I had my own microvax, pen plotter, ICE's, logic analyzers, 'scopes, etc. -- cuz I didn't WAIT until someone came up with "other alternatives"... "its only money!"]> Linux/foss may be a huge part of life but it is restricted to the needs > of the general public which feeds it directly. There are other aspects > of tech life which are simply beyond the scope of foss/linux, beyond > its capabilities as they are.The Linux camp tends to be full of zealots. Whatever the "problem", Linux (or some other FOSS product) is ALWAYS the answer! The manpower wasted on those duplications of effort is sad...> I strongly suspect Don is not an office secretary running an office archive and > I also strongly suspect he knows what he is doing.I tend to take my obligations a bit more seriously than most folks. I don't just shrug my shoulders when an old client asks a question. Or, immediately offer to bill him for my time in answering. I may not be willing to undertake NEW work (busy, not interested, etc.). But, that doesn't mean I should leave him on a limb -- esp if his "problem" may be something of my causing ("lifetime FREE bug fixes"). To do that -- to be ABLE to do that -- takes a significant investment. Call Apple/MS/Clifford's_Employer and ask about a product they released 5 years ago (10 years? 20 years??) and see how well they answer. See how EAGER they are to answer. contrast that with how eager they are to SELL YOU SOMETHING NEW! ("Gee, you're not supporting this OLD thing I purchased yet you want me to buy yet ANOTHER thing??") [If one of YOUR customers calls with a question on a NetMCA that is now no longer under warranty, how would YOU respond? Blow them off? Try to sell them an upgrade? *Or*, try to understand their problem and see how much of it is actually YOUR problem??]> Well I also suspect he knows he will find not what he is looking > for - an easier way to navigate his huge archives - but there is nothing > wrong wanting to talk or just moan about that, this is what groups are > for really.Navigation is easy. I can create an arbitrary SQL query that filters out the items that are of interest to me (e.g., all files with an 'L' as the third letter of the filename!), sort them based on whatever criteria is appropriate, and then pipe the resulting data into whatever post-processor I consider appropriate (e.g., "draw a pie chart based on number of bytes in files based on file extension"). The problem I outlined, here, was coming up with quick algorithms to highlight likely discrepancies WITHOUT exhaustively examining the contents of all of the files in a particular SET of files. "Have I made any changes to any of these files? Can I summarily discard them ALL, knowing that the originals from which they were derived are still intact?" "Does the *contents* of this ISO image agree with the *contents* of the ISO image in the archive? (the actual images can differ; the contents are what's of interest!)" etc.
Reply by ●October 30, 20162016-10-30
On Sat, 29 Oct 2016 13:15:24 -0700, Don Y <blockedofcourse@foo.invalid> wrote:> >This is RAID1. Many of the above issues still apply. In any case, >the mirrors must be spinning concurrently for the array to function. >(It has no way of knowing what the "other" copy of the "file" >looks like unless it can *see* it, now, in its entirety). > >This begs two crucial questions: >- are BOTH copies of the file examined when you try to access it?Almost always no. Although there are arrays that have that as an option, but that adds a substantial performance hit.> Or, is a "primary" copy accessed and the "secondary" copy only accessed > if the primary throws a read error?A (non-failing) read will only hit one drive (assuming it's not so large as to make it worth reading part from each), depending on which has the shorter work queue, seek, etc.>- If a discrepancy is found between the TWO instances, how is the > "winner" selected?Very implementation dependent, but usually there's a limited amount of cross-checking (except, in some case, as part of scrubbing). Most accesses only look at the ECCs, and those are very strong on modern drives (hundreds of bits per sector), so actual undetected errors are quite rare.>Plus, the more subtle question seldom considered: >- are files that are not "manually" accessed EVER verified? Or, do you > have to wait until you need/want a file to discover if it is intact?? > (Do *you* periodically check the contents of the batteries in your > flashlight to verify they are functioning BEFORE you need them? > Does the *flashlight* take on the responsibility of performing this > test on your behalf??)All good RAID implementations do scrubbing, where the contents of all drives are repeatedly read at some rate. Some arrays actually do compare data across the entire stripe (and that applies to both RAID-1 and RAID-5), but many do not. But unreadable sectors will be detected reasonably quickly. Exactly what would happen if an array detects an uncorrectable discrepancy is implementation dependent, especially if it happens to a sector not being accessed.
Reply by ●October 30, 20162016-10-30
On 10/29/2016 8:06 PM, Clifford Heath wrote:> On 30/10/16 13:05, Dimiter_Popoff wrote: >> On 30.10.2016 г. 03:00, Clifford Heath wrote: >>> On 30/10/16 10:19, Don Y wrote: >>>> On 10/29/2016 3:47 PM, Clifford Heath wrote: >>>>> On 30/10/16 07:15, Don Y wrote: >>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote: >>>>>>> Disclaimer: I am not an expert on the data storage, but here are my >>>>>>> thoughts: >>>>>>> - Use a RAID system which has enough redundancy to detect and correct >>>>>> I started out using RAID5 "hardware" arrays (and other, more >>>>>> sophisticated >>>>>> configurations). This requires you to have multiple spindles running >>>>>> concurrently (else the array can't *detect* discrepancies). Compact >>>>>> arrays also tend to suffer from cooling problems. >>>>>> >>>>>> It also ties you to an often proprietary (or, at least, poorly >>>>>> documented) >>>>>> implementation ("volume format"). So, a failure of the hardware can >>>>>> leave >>>>>> you with possibly intact volume(s) and no straightforward means of >>>>>> accessing their contents. >>>>> >>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc) >>>>> use Linux RAID support. If the hardware fails, you can mount >>>>> the drives on any Linux system that has enough SATA ports. >>>> >>>> Really? So my Sun RAID systems run Linux?? >>> >>> Ahh, my apologies. I didn't realize you were running a computer antiques >>> museum. I thought you were trying to ensure the availability of some >>> data. >> >> So what is he supposed to do with data which are useless outside of >> a particular environment. Apparently he has plenty of past projects >> spanning a few decades back to maintain - how do you suggest he will >> separate movable from non-movable data. Then the sheer amount of data >> he is talking is just huge. > > No, his amount of data is trivial compared to the petabytes > being accumulated daily and managed by internet-scale companies, > who almost all use *commodity* hardware running *open source* > software.And said companies have SCORES of staff to perform JUST those chores! As well as very deep pockets for their salaries and the equipment they want/need. How much time and money do YOU budget to maintain YOUR products? What LEGAL/contractual obligations do you have towards them? Can someone knock on your door and serve you with papers cuz someone DIED "allegedly" because of a product with which you were involved? Can a previous employer/client "slap" you with a lawsuit (to intimidate you into performing some additional work -- even if "for pay"?) Or, do you just turn off the light to your office at 5:00PM and walk home blissfully unconcerned with all of that?>> I strongly suspect Don is not an office secretary running an office >> archive and I also strongly suspect he knows what he is doing. > > Don is a "special flower". He faces problems that no-one else > has ever faced, and he analyses them with an insight that no-one > else has ever had, requiring solutions that no-one else has ever > found.No. I face problems and desire solutions that are appropriate for MY needs and my work style. Can you recreate the environment that you used to develop a product 30 years ago? Do you have hardware that will run the OS you used? Do you have the applications that ran under that OS? Do you have the documents governing that work (datasheets, specifications, contracts, records of billed time, schematics, PCB artwork, source code, test suites)? How about 20 years ago? 10? 5? Last month?? Does the ICE that you used (to expedite your development process) still work? Do you still have the manuals for it? Or, do you remember HOW to use it? Or, will you just brute force your way through the sources and HOPE you can spot the problems? Or, do you just tell your boss to buy you something that will solve that problem (even if the processor you used is no longer commercially available -- but the $1M device that it controls still has another 30-50 years of useful life!)? Or, do you just tell him "it can't be done"?> None of these descriptions are true, of course. > > For what it's worth, data management is a global profession with > a hundred major specialties and a thousand vendors, who all have > experts with more experience than Don has in his toenails. I've > just returned from a paid global speaking trip where I addressed > some of them. "Availability" isn't my specialty, I just rely on > commodity equipment to manage my measly few dozen terabytes. But > what do I know, I don't work with special problems like Don's.Great! And how many do you have on YOUR payroll? What portion of YOUR PAY do you set aside for them? How many of them can design a circuit? (I guess that will be a chunk of YOUR PAYCHECK that you'll set aside for that). How many can lay out the corresponding circuit board (oops! there goes another chunk of your pay!)? And, someone to write the code (well, maybe *you* can do that!). And, test it/certify it to the customer. And, of course, we need a bookkeeper to keep track of all that. And a lawyer to fend off folks who aren't happy with any-of-the-above. And a personnel director to administer the benefits program. And... Maybe those "data management professionals" have lots of EXTRA TOENAILS! More likely, very NARROW ranges of expertise -- "dancing bears"! Essentially USELESS without lots of OTHER support staff to lean on Or, maybe you just like giving up all those slices of YOUR paycheck so you can work with an army of (ahem) "professionals" on something that is probably as BORING as most things that involve armies! I, OTOH, have taken great pleasure in being able to work on truly novel problems throughout my career. Usually, "wild ideas" from folks WITHOUT deep pockets. Folks who can't afford a "small army" to test the viability of an idea (and need flexible/responsible folks to get them to a point where they can approach VC's with something tangible... not just an idea on a slip of paper!) But, there are scads of folks who are far more comfortable working in a bull-pen environment. The "security" of conformity and interchangeability...
Reply by ●October 30, 20162016-10-30
On 10/29/2016 8:48 PM, Dimiter_Popoff wrote:> On 30.10.2016 г. 05:06, Clifford Heath wrote: >> On 30/10/16 13:05, Dimiter_Popoff wrote: >>> On 30.10.2016 г. 03:00, Clifford Heath wrote: >>>> On 30/10/16 10:19, Don Y wrote: >>>>> On 10/29/2016 3:47 PM, Clifford Heath wrote: >>>>>> On 30/10/16 07:15, Don Y wrote: >>>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote: >>>>>>>> Disclaimer: I am not an expert on the data storage, but here are my >>>>>>>> thoughts: >>>>>>>> - Use a RAID system which has enough redundancy to detect and >>>>>>>> correct >>>>>>> I started out using RAID5 "hardware" arrays (and other, more >>>>>>> sophisticated >>>>>>> configurations). This requires you to have multiple spindles running >>>>>>> concurrently (else the array can't *detect* discrepancies). Compact >>>>>>> arrays also tend to suffer from cooling problems. >>>>>>> >>>>>>> It also ties you to an often proprietary (or, at least, poorly >>>>>>> documented) >>>>>>> implementation ("volume format"). So, a failure of the hardware can >>>>>>> leave >>>>>>> you with possibly intact volume(s) and no straightforward means of >>>>>>> accessing their contents. >>>>>> >>>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc) >>>>>> use Linux RAID support. If the hardware fails, you can mount >>>>>> the drives on any Linux system that has enough SATA ports. >>>>> >>>>> Really? So my Sun RAID systems run Linux?? >>>> >>>> Ahh, my apologies. I didn't realize you were running a computer antiques >>>> museum. I thought you were trying to ensure the availability of some >>>> data. >>> >>> So what is he supposed to do with data which are useless outside of >>> a particular environment. Apparently he has plenty of past projects >>> spanning a few decades back to maintain - how do you suggest he will >>> separate movable from non-movable data. Then the sheer amount of data >>> he is talking is just huge. >> >> No, his amount of data is trivial compared to the petabytes >> being accumulated daily and managed by internet-scale companies, >> who almost all use *commodity* hardware running *open source* >> software. >> >>> I strongly suspect Don is not an office secretary running an office >>> archive and I also strongly suspect he knows what he is doing. >> >> Don is a "special flower". He faces problems that no-one else >> has ever faced, and he analyses them with an insight that no-one >> else has ever had, requiring solutions that no-one else has ever >> found. >> >> None of these descriptions are true, of course. >> >> For what it's worth, data management is a global profession with >> a hundred major specialties and a thousand vendors, who all have >> experts with more experience than Don has in his toenails. I've >> just returned from a paid global speaking trip where I addressed >> some of them. "Availability" isn't my specialty, I just rely on >> commodity equipment to manage my measly few dozen terabytes. But >> what do I know, I don't work with special problems like Don's. > > Well, > > >> Linux/foss may be a huge part of life but it is restricted to the > >> needs of the general public which feeds it directly. There are > >> other aspects of tech life which are simply beyond the scope of > >> foss/linux, beyond its capabilities as they are. > > But I know many of the foss/linux crowd think otherwise - which is > OK, everyone sees the walls of their own room. No need to be grumpy > about other people wanting to talk about other things though.Clifford wants his answer to be "right". And, at the same time, doesn't want an explanation as to why it may be wrong in any particular case. He also probably has a much narrower focus and (almost assuredly) a much shallower historical perspective. He's probably never *held* an 8" floppy. Probably didn't know there were hard- and soft-sectored variants. Never had to live *in* a few KB. Or write his own OS, floating point package, debugger/monitor, etc. He doesn't wonder how many bits of exponent he set aside in that *24* bit float implementation for that old project. Or, why he made that choice (as well as why a 32b implementation wouldn't work). If he *thinks* he needs floating point operations in a project, he just factors in the cost of that EXISTING implementation instead of questioning whether or not it is truly necessary ("Do we REALLY need floats?") As a result, he's probably had to perform fewer trade-off analysis. Or, had to wonder how a product would perform "in overload" (just over-provision the hardware so overload isn't possible). Or, negotiate with some other aspect of a project ("If I serialize the data, can you omit this piece of hardware and, in return, give me this result in a form that's easier for me to use?" vs. "Your black box will be...")? OTOH, when you're building a "thing" that someone else is going to make in some quantity, reliability, etc., then you have to be able to justify WHY you made each decision. Why they are NECESSARY, and not just DESIRABLE! I'm sure you'd love to have twice as much memory and a processor that was twice as fast in YOUR products. But *you* have to consider what that does to your packaging, power requirments, development schedule, pricing, reliability/availability, position in the market, product perception, etc. *You* can't just call the guys in Marketing (or Manufacturing) and get a pat answer from them. *You* are The Expert for your product. In much the same way that I have been when an "idea man with a bit of spare cash" comes along wanting me to bring his idea a bit closer to a reality. I can carelessly just burn through his cash and leave him holding an empty bag. Or, I can tell him that I can't do what he wants (I don't have the skill/interest/time). Or, I can try to give him good value for money. *Not* by hiring janitorial staff, an IT guy, a bookkeeper, a PCB layout guy, a project administrator, etc. (He can go to one of the large "body shops" if he wants to throw lots of money around for the same results!) Different ballgame when you're an employee vs. proprietor.
Reply by ●October 30, 20162016-10-30
On 30/10/16 16:33, Don Y wrote:> On 10/29/2016 8:48 PM, Dimiter_Popoff wrote: >> On 30.10.2016 г. 05:06, Clifford Heath wrote: >>> On 30/10/16 13:05, Dimiter_Popoff wrote: >>>> On 30.10.2016 г. 03:00, Clifford Heath wrote: >>>>> On 30/10/16 10:19, Don Y wrote: >>>>>> On 10/29/2016 3:47 PM, Clifford Heath wrote: >>>>>>> On 30/10/16 07:15, Don Y wrote: >>>>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote: >>>>>>>>> Disclaimer: I am not an expert on the data storage, but here >>>>>>>>> are my >>>>>>>>> thoughts: >>>>>>>>> - Use a RAID system which has enough redundancy to detect and >>>>>>>>> correct >>>>>>>> I started out using RAID5 "hardware" arrays (and other, more >>>>>>>> sophisticated >>>>>>>> configurations). This requires you to have multiple spindles >>>>>>>> running >>>>>>>> concurrently (else the array can't *detect* discrepancies). >>>>>>>> Compact >>>>>>>> arrays also tend to suffer from cooling problems. >>>>>>>> >>>>>>>> It also ties you to an often proprietary (or, at least, poorly >>>>>>>> documented) >>>>>>>> implementation ("volume format"). So, a failure of the hardware >>>>>>>> can >>>>>>>> leave >>>>>>>> you with possibly intact volume(s) and no straightforward means of >>>>>>>> accessing their contents. >>>>>>> >>>>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc) >>>>>>> use Linux RAID support. If the hardware fails, you can mount >>>>>>> the drives on any Linux system that has enough SATA ports. >>>>>> >>>>>> Really? So my Sun RAID systems run Linux?? >>>>> >>>>> Ahh, my apologies. I didn't realize you were running a computer >>>>> antiques >>>>> museum. I thought you were trying to ensure the availability of some >>>>> data. >>>> >>>> So what is he supposed to do with data which are useless outside of >>>> a particular environment. Apparently he has plenty of past projects >>>> spanning a few decades back to maintain - how do you suggest he will >>>> separate movable from non-movable data. Then the sheer amount of data >>>> he is talking is just huge. >>> >>> No, his amount of data is trivial compared to the petabytes >>> being accumulated daily and managed by internet-scale companies, >>> who almost all use *commodity* hardware running *open source* >>> software. >>> >>>> I strongly suspect Don is not an office secretary running an office >>>> archive and I also strongly suspect he knows what he is doing. >>> >>> Don is a "special flower". He faces problems that no-one else >>> has ever faced, and he analyses them with an insight that no-one >>> else has ever had, requiring solutions that no-one else has ever >>> found. >>> >>> None of these descriptions are true, of course. >>> >>> For what it's worth, data management is a global profession with >>> a hundred major specialties and a thousand vendors, who all have >>> experts with more experience than Don has in his toenails. I've >>> just returned from a paid global speaking trip where I addressed >>> some of them. "Availability" isn't my specialty, I just rely on >>> commodity equipment to manage my measly few dozen terabytes. But >>> what do I know, I don't work with special problems like Don's. >> >> Well, >> >> >> Linux/foss may be a huge part of life but it is restricted to the >> >> needs of the general public which feeds it directly. There are >> >> other aspects of tech life which are simply beyond the scope of >> >> foss/linux, beyond its capabilities as they are. >> >> But I know many of the foss/linux crowd think otherwise - which is >> OK, everyone sees the walls of their own room. No need to be grumpy >> about other people wanting to talk about other things though. > > Clifford wants his answer to be "right".No. I want you to consider that your problem might not be unique, and that you should *first* look closely at existing solutions.> He's probably never *held* an 8" floppy.I still have the 8" floppy that contains the source code of the C compiler I was extending in 1979. Though my first product was released and built on a system that had 5.25" floppies - and though I don't still have that same hardware, I don't *need* it - the code still compiles today on modern hardware and C compilers.> Never had to live *in* a few KB. Or write his own OS,Built a debugger (with monitor in 256bytes) that was used to develop an RTOS for a rocket that went to the edge of space. Next question?> Different ballgame when you're an employee vs. proprietor.I've been both. Currently working in a 2 person company, ten years after I sold out of my 120 person company. See how far you traveled up the blind creek of your own imagination? That's my problem with you - so full of your own ideas, you've lost touch with reality and the rest of the world. You find it easier to write than to read, to speak than to listen.
Reply by ●October 30, 20162016-10-30
On 10/29/2016 10:14 PM, Robert Wessel wrote:> On Sat, 29 Oct 2016 13:15:24 -0700, Don Y > <blockedofcourse@foo.invalid> wrote: > >> This is RAID1. Many of the above issues still apply. In any case, >> the mirrors must be spinning concurrently for the array to function. >> (It has no way of knowing what the "other" copy of the "file" >> looks like unless it can *see* it, now, in its entirety). >> >> This begs two crucial questions: >> - are BOTH copies of the file examined when you try to access it? > > Almost always no. Although there are arrays that have that as an > option, but that adds a substantial performance hit.Yes, it was a rhetorical question. Folks *think* they have data integrity BECAUSE they are using a redundant technology. But, its like having a backup strategy and VERIFYING that backup strategy! Vastly different in practice! [I am always amused at how TERRIFIED folks are to TEST their backup/redundancy strategies. "Hey, it it doesn't work NOW, it sure as hell isn't going to work WHEN YOU HOPE IT WILL!"]>> Or, is a "primary" copy accessed and the "secondary" copy only accessed >> if the primary throws a read error? > > A (non-failing) read will only hit one drive (assuming it's not so > large as to make it worth reading part from each), depending on which > has the shorter work queue, seek, etc.Exactly. So, the other drive could have already failed -- and the failure is masked by the other (not yet failed) drive.>> - If a discrepancy is found between the TWO instances, how is the >> "winner" selected? > > Very implementation dependent, but usually there's a limited amount of > cross-checking (except, in some case, as part of scrubbing). Most > accesses only look at the ECCs, and those are very strong on modern > drives (hundreds of bits per sector), so actual undetected errors are > quite rare.And, there is no way of knowing the quality of a particular implementation! E.g., vulnerable to "write hole" failures? You *do* have a battery-backed cache on that RAID controller, right? And, the UPS that powers it doesn't EVER glitch, has orderly shutdown mechanisms in place (to ensure the RAID is idle with caches flushed when power is eventually removed). And, the battery estimate is "spot on" and the load unchanging (to potentially alter that estimate) "Trust us" (and surely DON'T verify -- cuz you may lose something! :>)>> Plus, the more subtle question seldom considered: >> - are files that are not "manually" accessed EVER verified? Or, do you >> have to wait until you need/want a file to discover if it is intact?? >> (Do *you* periodically check the contents of the batteries in your >> flashlight to verify they are functioning BEFORE you need them? >> Does the *flashlight* take on the responsibility of performing this >> test on your behalf??) > > All good RAID implementations do scrubbing, where the contents of all > drives are repeatedly read at some rate. Some arrays actually do > compare data across the entire stripe (and that applies to both RAID-1 > and RAID-5), but many do not. But unreadable sectors will be detected > reasonably quickly.This works if the array is up for reasonably long periods of time. E.g., in a business setting maintaining "live" data (with a suitable budget and care). If, OTOH, it is only powered up when needed -- e.g., to drag out a copy of last year's tax return -- and then powered down, again, the array can't *do* much scrubbing. And, a sloppy implementation won't keep track of where it left off -- because it ASSUMED it would be "up" for long periods of time. In my case,. there is no single "appliance". Instead, there is a database that is accessible to all (even people! :> ). Whenever a NAS appliance comes on-line (i.e., to serve up its files via SMB/NFS/FTP/etc.), it queries the DBMS to sort out which parts of its contents should be "verified" (not just scrubbed) AND how far it had progressed in that process before it was last powered off. Because the file services are integrated with the archive maintenance tasks, this takes into account recent file accesses. In essence, watching the "last accessed" attribute. So, if you've just recently accessed some portion of the filestore, AND, the scheduler claims that section is due for "verification" (scrubbing), the verification is skipped; it already happened when you accessed those contents! When the user unceremoniously shuts down the "appliance", the DBMS has already been update to reflect the scrubbing/verification progress made during this "session". So, if the physical volume is removed from this appliance and later appears on another (or, mounted "local" to some computer in the network), the process can resume from where it left off. As there is no "archive appliance" and no "front panel display", any communication with the user is done by email (I have a local server) and direct access to the DBMS's various schema.> Exactly what would happen if an array detects an uncorrectable > discrepancy is implementation dependent, especially if it happens to a > sector not being accessed.I.e., the only way to find out how *it* (and, thus, YOU!) will handle that possibility is to simulate such a failure. Or, grep(1) the sources for some indication of how that would ALLEGEDLY be handled (you *do* have the sources for your RAID appliances, right? :> ) IME, there are just too many "garage shop" implementations for appliances. Folks thinking they can just slap together a stock distribution and a bit of script, plus a web interface. And, it *looks* like it works. They don't worry about the inner details: "Other folks are using it for <whatever> so it MUST work!" Just like all those routers and networked security cameras, etc. The "end user" is left clueless wondering how particular failure modes will be addressed by the appliance. And, in many cases, is clueless as to the TYPES of failures that can occur or are likely to occur! Image your disks. Then, have fun with a disk editor (corrupt parts of the parity disk, one of the data disks, etc.). Or, unplug one of the drives while its spinning. Make a guesstimate as to what you THINK will happen. Then scratch your head when something unexpected happens instead (or, in addition). [I was chagrined the first time I "faulted" a drive in a RAID5 array: "Why the hell is it taking so long to recover? Crap, did I BREAK the disk??" I've had other NAS's that would "randomly" do extensive file system checks on power up -- delaying their coming on-line for many minutes! Despite the fact that they had always been systematically powered down correctly. "Why are these tests necessary today when they weren't necessary yesterday?"] [[Of course, I can freely spend MY TIME examining their GPL'd sources but without any assistance from them in doing so -- and, almost a disdain for my interest! And, work around the obviously contrived contortions they employed to put the image on the media (The BANAS440 is a perfect example of this!). Cuz they want to ENCOURAGE me to understand how their device actually works?? Not!]]
Reply by ●October 30, 20162016-10-30
On 30/10/16 15:50, Don Y wrote:> On 10/29/2016 7:05 PM, Dimiter_Popoff wrote: >> On 30.10.2016 г. 03:00, Clifford Heath wrote: >>> On 30/10/16 10:19, Don Y wrote: >>>> On 10/29/2016 3:47 PM, Clifford Heath wrote: >>>>> On 30/10/16 07:15, Don Y wrote: >>>>>> On 10/29/2016 5:17 AM, kalvin.news@gmail.com wrote: >>>>>>> Disclaimer: I am not an expert on the data storage, but here are my >>>>>>> thoughts: >>>>>>> - Use a RAID system which has enough redundancy to detect and >>>>>>> correct >>>>>> I started out using RAID5 "hardware" arrays (and other, more >>>>>> sophisticated >>>>>> configurations). This requires you to have multiple spindles running >>>>>> concurrently (else the array can't *detect* discrepancies). Compact >>>>>> arrays also tend to suffer from cooling problems. >>>>>> >>>>>> It also ties you to an often proprietary (or, at least, poorly >>>>>> documented) >>>>>> implementation ("volume format"). So, a failure of the hardware can >>>>>> leave >>>>>> you with possibly intact volume(s) and no straightforward means of >>>>>> accessing their contents. >>>>> >>>>> No. Almost all commercial RAID systems (Synology, Qnap, etc) >>>>> use Linux RAID support. If the hardware fails, you can mount >>>>> the drives on any Linux system that has enough SATA ports. >>>> >>>> Really? So my Sun RAID systems run Linux?? >>> >>> Ahh, my apologies. I didn't realize you were running a computer antiques >>> museum. I thought you were trying to ensure the availability of some >>> data. >> >> So what is he supposed to do with data which are useless outside of >> a particular environment. Apparently he has plenty of past projects >> spanning a few decades back to maintain - how do you suggest he will >> separate movable from non-movable data. Then the sheer amount of data >> he is talking is just huge. > > No, its much simpler than that: I am supposed to foresee the future.No, you're supposed to move with the times, and move your data to a system that's replaceable, not one from a company that died almost a decade ago. Especially if you are actually bothered with maintaining the integrity and availability of said data... as you claim to be.> Perhaps Clifford has the luxury of an IT department to fall back on.Never have had, probably never will. My clients have, but not me.> Or, maybe he's got an eidetic memory and can remember and recreate how > the schematics, artwork, specifications, contracts, software, test suites > and other materials that he used to create Project X for Client Y > in Year Z.Pretty much, yes. See below.> I, OTOH, take a keen interest in every project I undertake.Good for you. I respect that.> The Linux camp tends to be full of zealots. Whatever the "problem", > Linux (or some other FOSS product) is ALWAYS the answer!I agree, that's a problem. The FOSS world is almost full of software that isn't worth the price of the media that stores it. But when something exists that solves your problem... why re-invent the wheel?> To do that -- to be ABLE to do that -- takes a significant investment. > Call Apple/MS/Clifford's_Employer and ask about a product they > released 5 years ago...I am my own employer, yet I still occasionally get paid to assist with code I wrote fifteen years ago. I can still tell you which file and function to look (in three million lines of code) for a given feature - from memory.> The problem I outlined, here, was coming up with quick algorithms > to highlight likely discrepancies WITHOUT exhaustively examining > the contents of all of the files in a particular SET of files.There is no such algorithm. All the file metadata in the world will not tell you that the file has a few flipped bits not detected by error-correcting codes - you need to actually read the file. And even after you've done that, those bits could flip five minutes later.
Reply by ●October 30, 20162016-10-30
On 30/10/16 15:20, Don Y wrote:> On 10/29/2016 6:00 PM, Clifford Heath wrote: >> On 30/10/16 10:19, Don Y wrote: > Crash a 4T drive in one > of your RAID arrays. Let me know when (if!) it ever finishes rebuilding > the array -- before one of the companion 4T drives dies a similar death!I've done that (3TB drive actually). The rebuild took ten hours, but the system was in continuous use throughout the period before and during the rebuild. SATA hot-swapping rocks - I didn't even need to power it off.> (Hey, you have FAITH in your RAID array, right? So, you've got NOTHING > to lose by pulling the drive, dd(1)-ing /dev/zero to it, and then > reinstalling it in the array! You should be able to do that TONIGHT!)I don't like running degraded, but if I had chosen RAID6 instead of RAID5, no problem at all. I can pull one drive and still have single-failure redundancy, and not sweat over your challenge at all.> Linux-weeniesI would not have built my own Linux box to do this. As you say, it's better with the right HDD controllers, the right SATA interfaces, the right power supplies, the right Ethernet hardware, etc, etc... all the things that the good folk at Synology have been doing for nearly two decades. They have a lot of streamlined behaviour based on experience with millions of units... Much more than SUN ever had. Yet the fact is that because they use Linux, I can pull all drives from a volume, stick them in individual USB containers, and boot *any* PC from a USB drive to rebuild the volume or access the contents. If I chose to, instead of just buying a new NAS.
Reply by ●October 30, 20162016-10-30
On Thursday, October 27, 2016 at 11:23:58 PM UTC+1, Don Y wrote:> I currently "verify" my data archive with an algorithm similar to: > > ONCE: con 0; > Success: con 0; > Removed, Inaccessible, NoRecord, BadSize, BadHash: con Success+1+iota; > > verify(files: list of string ): list of (string, reason) > { > failures := list of (string, reason); > > while (files != nil) { > theFile := hd files; > (size, signature, result) := examine_file(theFile); > (Size, Signature, Result) := query_DBMS(theFile); > do { > if (NotFound == result) > {Result = Removed; break;} > if (BadRead == result) > {Result = Inaccessible; break;} > > // theFile exists! > if (NotFound == Result) > {Result = NoRecord; break;} > if (Size != size) > {Result = BadSize; break;} > if (Signature != signature) > {Result = BadHash; break;} > } while (ONCE); > > if (Success != Result) { > failures = (theFile, Result) :: failures; > } > > files = tl files; > } > return( failures ); > } > > [apologies for any typos; too early in the day to be working!] > > The problem this has is examine-file() is expensive -- for large > numbers of large files! > > I'd like to short-circuit this to provide an early indication of > possible problems. E.g., stat(2) each "theFile" to identify any > that are "Removed" without incurring the cost of examining the > file in its entirety. > > [Of course, all files MUST be eventually examined as you can't > verify accessibility (BadRead), actual size (BadSize) or correct > signature (BadHash) without looking at every byte! But, you can > get the simple checks out of the way expeditiously to alert you > to "obvious" problems!] > > I can get an approximate indication of "actual size" just by > stat(2)-ing the file, as well -- relying on examine_file() to > later confirm that figure. > > The hash, of course, can't be guesstimated without trodding through > the file in its entirety. > > The problem with *this* approach is I'd have to lock the entire > object to ensure the integrity of the "early results" agrees with > that of the "later results" (the original approach only requires > taking a lock on the file being examined *while* it is being > examined). > > Are there any "tricks" I can use, here? E.g., the equivalent of > CoW would ensure the integrity of my data (for the *original* > instance) in the presence of such a global lock. But, I think only > the Bullet Server would give me any behavior approaching this...? > > Also, how wary would one be relying on the files' timestamps as a > (crude) indication of "alteration"? (It seems far too easy to > alter a timestamp without altering content/size so I've been leary > of relying on that as having any "worth").I would suggest to compartmentalize this function. There are unnecessary conditional operators stacked to achieve a ‘something’ common. How about breaking apart file handler and DB results? That should provide two clear paths to work with. //late to party!
