Disk imaging strategy

In comp.arch.embedded Robert Wessel <robertwessel2@yahoo.com> wrote:

(snip)

> I've seen that before, unfortunately he's off on some of the details.
> First it's a 3390 module (an "HDA", which was actually two drives or
> "actuators"), so it's from no earlier than 1989 (not late 70s/early
> 80s), and it's not 10MB, it's about 1.89GB (assuming it's a model-1),
> or 3.78GB (if it's a double density model-2), and there were
> additional models of higher capacity later.  It's also not worth $250K
> - you could buy an entire -B28 for $275K at the time, and that
> contained six double density modules (HDAs) of the type he
> disassembled.  You'd usually buy a "string" of three units (one -Axx
> and two -Bxx units), for a total of 16 HDAs, which would set you back
> about $750K.  So the value is more along the lines of $50K (assuming
> the enclosure is free).

I think that there was also a 3390 model that ran at 1200PRM, instead
of the usual 3600RPM, and stores three logical tracks on one
physical track. Access time is longer, so ti doesn't always help.

-- glen

On 11/3/2014 10:11 AM, George Neuner wrote:
> On Mon, 03 Nov 2014 10:09:48 GMT, Jan Panteltje
> <pNaonStpealmtje@yahoo.com> wrote:
>
>
>> I think I have not 'defragmented' anything ever in my life in Linux,
>> there is no need.
>
> That isn't entirely true - at least not with inode filesystems.  The
> n-way tree structuring and inode caching reduce the need to
> defragment, but where sequential read performance is important, it
> still pays to defragment.

IIRC, the Bullet Server could (did?) create contiguously stored
files.  But, that was largely possible because of its "write once"
semantics (size declared a priori).

On 03.11.2014 г. 23:22, Don Y wrote:
> Hi Dimiter,
>
> On 11/2/2014 10:49 PM, Dimiter_Popoff wrote:
>> On 02.11.2014 г. 17:25, Don Y wrote:
>>> I'm writing a bit of code to image disk contents REGARDLESS OF THE
>>> FILESYSTEM(s) contained thereon.
>
>> since obviously there is no common solution to all filesystems
>> (unless you want to copy the entire medium which is impractical),
>> your best bet is to go minimialistic about it. Recognize which
>> file system this is, then find your way to allocated space and
>> store it in some indexed format - such that you can subsequently
>> recover it.
>> On some filesystems it will be easier than on others - e.g. on DPS
>> you will need to locate a file in the root directory, unitcat.syst,
>> which is a bitmap of the allocated clusters; and you have to read
>> logic block 0 to see how large the "device" (i.e. partition) is,
>> what block size does it assume and how many blocks are there per
>> cluster. On FAT it will be easier I think (no need to do root
>> or any directory).
>> But you can't get around this minimum I suppose. Then there are not
>> that many filesystems in mass use anyway (I think George Neuner already
>> said that), so the effort will not be that huge.
>
> I'm going to GUESS that your FS is "proprietary" (not mainstream).

Yes, the initial version (with 8.4 names) being introduced in 1994
(no way was this 20 years ago....). Now the long name directories
(within the same filesystem) are much newer, added these last few years.

>
> So, a potential test for my envisioned approach!  :>

We could do that, sure. Your compress idea is really good, I can't
think of a reason why it should not work. Then one can have
files as mounted disk images to access which should make the exchange
easy enough.

>
> First, can a user create files having arbitrary names and contents
> under your FS?

Yes of course. Pretty much everything you would expect from a
filesystem.

>
> Can he copy & rename files?

Yes, what filesystem would it be without that :D .

>
> E.g., could he <somehow> introduce a file having some particular
> contents (like "DELETEDDELETEDDELETEDDELETED...") to your FS?

I'd be tempted to go to all 0 files for the "highly compressed"
pattern - for no good reason really, except perhaps because disks
come as all 0 from the factory. But you will want to fill them
up anyway so this is not a consideration.

> Then, could he replicate it many times?  (copy to a different
> filename)
>
> Having done that until the copy failed ("No space left on device"),
> presumably, he could delete each of them?  (perhaps made simpler
> by creating them all in a single subdirectory/folder and then
> just deleting the folder AND its contents)

Yes, all of the above. Making multiple copies of a single file will
take 2-3 lines of script, to increment the name somehow. Deleting
all in a directory goes the usual del * way, if you want recursion
there is a script doing it (rm path/ -R ). I have deliberately
kept recursive disk operations in scripts, makes new bugs show up,
costs no overhead to speak of, can be retried/resumed, prompts me
to write necessary extensions when there is some new need etc.

>
> Could I then examine your disk AT THE SECTOR LEVEL and expect to find
> lots of "DELETEDDELETEDDELETED..." in sectors?

Well you can have the disk image as a file and do with it whatever
you please under any OS. Or under DPS, but you don't have a dps machine.

> In doing so, effectively know which sectors are currently "unused"?
> (or, at the very least, safe to restore with "DELETEDDELETEDDELETED..."
> as their contents WITHOUT actually having to store an instance of
> "DELETEDDELETEDDELETED..." for EACH such sector?

I am not sure I get this, we may be thinking somewhat differently on
how you will implement that. My understanding of your idea is that
you will take the disk image - say a 20G one - and copy it elsewhere
in 512 byte (say) pieces by skipping those 512 byte pieces
which are all 0 (or all deleteddeleted whatever you opt for). You just
write (say in another file)where these blocks were, position:length.

Oops, I think I get it now. You can know which sectors have been
full of "deleteddeleted" but how will you know you don't have to
restore them? What if there has been an allocated sector full of
exactly this pattern? May be it is still practical to come up
with some 512 byte pattern which will never occur on normal
disks but this will be a vulnerability, perhaps an acceptable
one. Clearly the "all 0" I thought earlier of is not applicable then.
But I'd go for restoring all data (at least have the option to
do if the shortcut does not get me anywhere). At say 20M/S a 20G
disk would get restored in what, 1000 seconds. Not that long
for such a massive intervention I suppose.

>
> [BTW, we are now below 25C.  Almost feels comfortable!!  :> ]

Hey, if 25C is "almost" comfortable then I don't know what is
"really comfortable" in your book :D . We already hover between
0 and 5C, sometimes 10C, winter is coming... (we do hate it).

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/sets/72157600228621276/

On 04/11/14 03:10, DecadentLinuxUserNumeroUno wrote:
> On Tue, 4 Nov 2014 01:31:50 +0000 (UTC), Andrew Smallshaw
> <andrews@sdf.lonestar.org> Gave us:
> 
>> No, on _any_ general purpose file system.  It is inevitable.  You
>> seem to be of the impression that Linux is somehow the ultimate
>> system,
> 
>  I was "under the impression" that the ext series of file systems, as
> well as MS' NTFS fought such occurrences with a different operation and
> management paradigm, than the old, FAT type method.
> 

There are three major things that affect the fragmentation rate on a
filesystem, and a number of things that influence the effect of the
fragmentation.

One is the type of filesystem.  Some, such as FAT, have a very simple
structure that is prone to fragmentation.  Others, such as ext4 or xfs,
have features such as "extents" and "allocation groups" that greatly
reduce the fragmentation.

Number two is the OS and the filesystem implementation.  DOS and early
Windows versions had very poor caching, and very short write cache times
(since it was expected that the system could crash any second), leading
to massive fragmentation.  Linux, and newer Windows versions, take a bit
more time and allocate more sensibly, thus leading to far less
fragmentation.  Linux still does a better job than Windows, but since
Win7 the difference is hardly noticeable.

Number three is the pattern of filesystem usage.  Fairly obviously, if
you write many large files at once, or do lots of modification of files,
the chances of fragmentation increase.


As to the effect of the fragmentation, a key point is the effectiveness
of the OS's caching and read-ahead.  If these work well (as they do in
Linux, and to a fair extent in modern Windows, but not in older
Windows), fragmentation does not matter even when reading longer files.


Fragmentation has always been here (well, not always - there were older
filesystems that did not allow file fragmentation at all, but they
suffered from other problems), and fragmentation will always be with us.
 But apart from a few extreme cases, it has never been a problem in
Linux, and it is not a problem in modern Windows.  The /problems/ of
fragmentation are a thing of the past.

On 04.11.2014 г. 09:55, David Brown wrote:
> On 04/11/14 03:10, DecadentLinuxUserNumeroUno wrote:
>> On Tue, 4 Nov 2014 01:31:50 +0000 (UTC), Andrew Smallshaw
>> <andrews@sdf.lonestar.org> Gave us:
>>
>>> No, on _any_ general purpose file system.  It is inevitable.  You
>>> seem to be of the impression that Linux is somehow the ultimate
>>> system,
>>
>>   I was "under the impression" that the ext series of file systems, as
>> well as MS' NTFS fought such occurrences with a different operation and
>> management paradigm, than the old, FAT type method.
>>
>
> There are three major things that affect the fragmentation rate on a
> filesystem, and a number of things that influence the effect of the
> fragmentation.
>
> One is the type of filesystem.  Some, such as FAT, have a very simple
> structure that is prone to fragmentation.

This is wrong. FAT has nothing to do with fragmentation. It is the
allocation strategy the OS employs which is responsible. Nothing is
stopping the OS to do worst fit on a FAT disk image (I have done it
for example, when emulating FAT disks under DPS - the latter does
enhanced worst fit so it was transferred automatically to FAT images
as well).

> Number two is the OS and the filesystem implementation.  DOS and early
> Windows versions had very poor caching, and very short write cache times
> (since it was expected that the system could crash any second), leading
> to massive fragmentation.  Linux, and newer Windows versions, take a bit
> more time and allocate more sensibly, thus leading to far less
> fragmentation.  Linux still does a better job than Windows, but since
> Win7 the difference is hardly noticeable.

Cache times have nothing to do with fragmentation either. Again, it is
just the space allocation strategy which matters. I know little of
the inner works of windows and unix (of which apparently you know even
less) but cached disk data would normally be cached on a sector
basis.
If you think the applications nowadays write in larger pieces at a time
than they did in the past this is probably correct but has nothing to
do with the type of filesystem regarding fragmentation.

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/sets/72157600228621276/

On 03/11/14 16:28, Don Y wrote:
> Hi David,
> 
> On 11/3/2014 3:36 AM, David Brown wrote:
> 
>>
>> I am thoroughly confused as to what you are trying to do here.
> 
> Because I'm trying NOT to let the thread drift!  :>

When you post to comp.arch.embedded, thread drift is a risk - when you
cross-post to sci.electronics.design, thread drift is guaranteed or your
money back :-)

> 
>> On the one hand, you want a filesystem aware process so that you can
>> image only real files, not empty space (or leftovers in deleted space).
> 
> That conclusion doesn't necessarily follow.  What I want to do is
> not waste "image space" on "dead data" (deleted files) -- WITHOUT
> EXPLICITLY KNOWING WHAT IS DEAD (because I have no metadata from
> a "file system" to tell me what is live/dead)

I think there are four ways to identify the data that you need to store:

1. Metadata to tell you what is live.
2. Metadata to tell you what is dead.
3. Identify the live data from its contents.
4. Identify the dead data from its contents.

You have ruled out 1 and 2 by not allowing the imaging system to know
anything about the filesystem.  I doubt that 3 is practical - certainly
not for a general purpose filesystem.  That leaves number 4 - identify
the dead data from its contents (alternatively, fill it with highly
compressible zeros).

It's easy enough to zero out the whole disk before using it, which gives
you a start.  Writing out a large file full of zeros, as you suggested
before, is the only general-purpose way to put zeros into the empty
space.  I gave some disadvantages of that earlier, but it may be the
only general solution given the restraints you have given yourself.

On Linux, an alternative technique would be to look at the "fstrim"
command - intercept the generated SATA TRIM commands and replace them
with commands to write zero blocks.  I believe that should be safe, and
it should let you zero out all the dead data.  You could even just store
the fstrim output and use that as a list of dead blocks, if you have
some way of getting the information to the imaging software - but be
sure nothing more is written to the filesystem after the fstrim.


> 
>> On the other hand, you want something completely independent of the
>> filesystem.
> 
> Correct.
> 
>> On the one hand, you have only a small amount of real data in use, and
>> on the other hand you might have large amounts of data copies that you
>> also want to image.
> 
> I want to image the entire disk -- because I can't know what's live/dead.
> But, I don't want to waste space/time on "dead content".  So, want a scheme
> (which may include "procedures" and not just "code") that will effectively
> give me that information without EXPLICITLY seeking it.
> 
> E.g., as I proposed, if I create files filled with some highly compressible
> data ("Dear Compressor, when you encounter me in your input stream, please
> represent me in your compressed output stream by the special token
> 'BIG_CANNED_STRING'.  In doing so, you will know exactly how to reconstruct
> me without wasting much space on my actual content.  Chances are, you will
> encounter me many, many, many times as you scan through the blocks of
> this drive..."), that data gets moved onto real platters (when disk cache
> is flushed, etc.).
> 
> Once I "run out of room" in the filesystem (more or less), I will have
> consumed the previous "dead space" (free space) with files of this type.
> I can then unlink all of these files thereby recreating the "dead space".
> But, while the previous content of this dead space was unrecognizable
> (without knowledge of the filesystem), it can now be recognized as such
> a filesystem agnostic piece of code (later).
> 
>> So lets get back to basics, and try to understand your setup.
>>
>> First, how many machines are we talking about?
> 
> Personally, about 30 or 40 drives (e.g., some "machines" have multiple
> drives).  Note that a "machine" does not have to be a PC.  Nor a
> SPARCstation.
> etc.
> 
> For my pro bono work, probably 200 - 400 yearly (but, that will hopefully
> only be 20-40 different "model numbers", 10 or more instances of each)
> 
>> What sort of different systems varieties do you have in the OS's and
>> filesystems?
> 
> Personally, three different flavors of Solaris, three Windows, three
> NetBSD,
> a couple of oddball "OTS" systems (Jaluna, Inferno, etc.) and probably a
> dozen
> different "appliance"/proprietary implementations (effectively black
> boxes).
> 
> Pro bono is much easier.  They'll either be PC's or Macs.  But, their
> OS's will largely be defined by whatever happens to *run* on that
> particular
> hardware (donations may be of various "ages").  I'm guessing three
> different Windows (though within that, there can be minor variations
> like Home, Pro, Business, etc. editions -- possibly even on the same
> make/model hardware).  Probably two different OSX versions (??).
> 

This suggests two different setups to me.  One should be a clonezilla
server system that will work for the windows systems (and also, I think
Macs and "simple" Linux or BSD setups).  This will cover the bulk of
your systems, especially all of your pro bono stuff.  The Windows
systems will all be identified automatically, and since the filesystem
is known, there is no problem imaging just the live data.

Then you have another system for your "weird" systems.  You may find
that these ones are small enough that you can just image the entire disk
with "dd".  Yes, it is a little inefficient - but it will be safe,
reliable, and easy to understand and implement.

And if you walk under a bus, or decide to retire, there is a chance that
someone else will understand how it all works.


>> What about
>> the different developers and users - are they at similar levels and are
>> they cooperative/competent, or are you going to have to do these backups
>> and images because they are not good at following version control
>> routines?
> 
> They aren't "backups" (see my thread to George).  They are "restore
> images".
> They aren't regularly performed (like backups would be).  Rather, a machine
> is imaged (typically *once*) and the image saved in order to recreate
> the machine's state at a future time (if it gets munged).
> 
> So, I expect this to be far more involved than the routine "backups"
> I do for my working files/configuration.  But, I expect the "restore"
> to be far simpler (UX) -- "push this button and wait".

For unusual systems, images will be taken much more often than they will
be restored - the image is for emergencies.  For duplicating (nearly)
identical machines, images will be "restored" much more often than they
are made.  These are different purposes, and I believe you should be
looking at different setups for these purposes.

> 
> For example, when I build a new system, here, I image the disk at
> various stages in that process.  This lets me quickly return to one of
> those points in time (if, for example, I make some annoying mistake
> in a subsequent stage and want to "undo" it).  Prior to putting the system
> into daily use, I have a final snapshot image.  I.e., I can reproduce the
> software installation and configuration process very quickly if I have
> to at a later date (because a disk died, because some app scribbled
> somewhere that it shouldn't, etc.).  Instead of DAYS to rebuild the
> system (individually installing and configuring each application, etc.),
> I can do it all in a matter of minutes.
> 
> And not worry about the things that an incremental approach might
> fail to address (have I removed ALL the cruft?  have I added all
> the changes back in?  etc.)
> 
> For the students, they tend to be careless users.  And, there's always
> a certain amount of "I didn't pay for it so I take it for granted"
> attitude involved.  ("If it breaks, I'll just ask for a new one.
> THAT won't cost (me) anything, either!")
> 
> Originally, I was pursuing a "build a set of CD's/DVD's for each machine"
> that would allow them to restore their machines (without my involvement).
> But, these would get lost, misplaced, etc.  No real incentive for the
> student to keep track of them (most are also homeless so that would be
> one more thing for them to keep track of, "just in case").  You'd be
> naive to imagine that this WOULDN'T turn into "I lost my restore DVD.
> Can you just make me another one (DVD)?"  "Well, if you can't do that,
> can I just trade this machine in and get ANOTHER machine?"
> 
> If, instead, the restore mechanism is on the disk (just like a
> factory restore partition -- but, with the *final* disk image
> instead of the *initial*/factory image), the student has no excuse
> to claim he's lost the DVD or "doesn't know how to repair/restore
> the machine".
> 
> Additionally, if the student feels his machine may have been
> "compromised" (perhaps an AV update points to the presence of
> a virus on his machine), he can "clean it" himself.  (Hysterically,
> this has resulted in machines being returned and the system
> being rebuilt from scratch.  *I* have no desire to be in that
> business -- ESP in an unpaid capacity!  :> )
> 
>> The way I would organise this all is that the server (or servers) are
>> masters.  You have full control of these - you use raid to protect
>> against hardware failures, and regular snapshot-style backups (such as
>> with rsync or btrfs snapshots).  /All/ your data is there.  Where
>> practical, it is in the form of version control repositories.  Other
>> data, especially more static data, may be just an area of shared files,
>> which is backed up with snapshots.
> 
> That;s what I do for my personal machines.  But, that doesn't mean
> it is easily usable in that form!
> 
> E.g., I have ISOs of every CD/DVD I've purchased. But, if I have
> to go through the trouble of *installing* it to be able to *use*
> it, then having the original is just a small part of the solution.
> 
> Typically, I build specific machines for specific roles/purposes.
> Once built, I image their drives and preserve the images on a
> bunch of (removable) SATA/PATA/SCA/FW drives (depending on how I
> will ultimately need to restore that image) -- along with the
> installation log that documents every step in that particular
> build process.
> 
> This just saves me from having to repeat all that labor (install
> & configure) in a hardware failure, screwup on my part or if I
> just want to upgrade the local drives to larger ones, etc.
> 
>> On local machines, you have only temporary copies of any data while
> 
> The "data" is the key, here.  I can drag the documentation, schematics,
> PCB artwork, sources, etc. for a project out of the repository, work
> on it and then discard or recommit any changes -- PAINLESSLY.  Because
> it's just "data" and not "workstation executables".  There is no
> installation or configuration involved.  I can erase it and know that
> there are no vestiges hiding somewhere unseen.
> 
>      cd /Playpen
>      rm -r *
> 
>> working.  Losing that data is an inconvenience, but should not be a
>> disaster.  Users check out from the repositories, do their work, and
>> check in changes.  If you have need of data that is not part of the
>> repositories, but should be kept safe, it is either accessed directly as
>> part of the servers shared files, or you use rsync or similar backup
>> strategies to copy from the local machine to a safe area on the server.
>>
>> Imaging of the local machines is just a convenience to get the system
>> running again faster if there is a hardware failure.  It is usually done
>> after setting up the basic system and installing key programs, and
> 
> Exactly.
> 
>> perhaps on occasions afterwards after major upgrades or installations.
> 
> I image each machine exactly once.  I rarely update applications,
> especially if the reason for the update is solely security related.
> When the machine is upgraded, I move on to the newer applications
> which get folded into the image for that newer machine.
> 
>> It is not about data backup, but merely saving time.  Usually something
>> like clonezilla or Norton Ghost to an external disk will be fine - if
>> something goes wrong with the main disk, you can simply put in the
>> imaged copy.  Imaging can also be useful if you have multiple systems
>> with the same setup - you might want images stored on a central server
>> in this case.  But you don't image the data - you only image the OS,
>> programs, and setup.
> 
> Exactly.  That is the case with the pro bono effort:  archive images for
> each type (make/model) of machine encountered.  Install the COMPLETE image
> (including the "recovery partition") from the server.  If I encounter
> another "identical" (make/model) machine in the future (often!), then
> I don't have to bother recreating a suitable image for that machine.
> 
> And, thereafter, the user (student) can restore the "system partition"
> (but none of their data -- because the recovery partition has no knowledge
> of their data!) at will instead of relying on me to perform that task for
> them.
> 
> I.e., they no longer have an "excuse" to ask (expect) someone to
> solve THEIR problem (because, chances are, the reason their computer
> is "all gunked up" is because of poor practices on their part!)
> 
> Those that try to "beat the system" by actually BREAKING their computer
> are "rewarded" by going to the end of the line:  "Gee, it's too bad
> you dropped your laptop off the bus!  We'll see if we can salvage any
> PARTS off of it.  And, put you in line to get a replacement.  But,
> there are 187 people ahead of you so you probably won't get one
> before sometime in the NEXT school year!"
> 
> [This is not an exaggeration.  :< ]
> 
>> Is there something special about your needs that makes such a system
>> impractical?
> 

OK, I've got a lot better picture of your aims now.  And as I said
above, I really think you should separate this into two systems - I
believe each will be much easier than trying to make a general system
that covers both.

On a sunny day (Mon, 03 Nov 2014 13:51:59 -0700) it happened Don Y
<this@is.not.me.com> wrote in <m38pti$rvh$1@speranza.aioe.org>:

>On 11/3/2014 11:39 AM, Jan Panteltje wrote:
>> On a sunny day (Mon, 03 Nov 2014 10:17:18 -0700) it happened Don Y
>> <this@is.not.me.com> wrote in <m38db2$nmp$1@speranza.aioe.org>:
>>
>>>>> But, this comes at the cost of not knowing which parts (sectors) of
>>>>> the medium actually have "content" that must be preserved!  So, you
>>>>> have to include every sector in your image!
>>>>
>>>>
>>>> Not if you tar a filesystem (on that partition) see the script I
>>>> posted. So mount partition, say: mount /dev/sdd1 /mnt/sdd1
>>>
>>> mount(8) brings filesystem specific code into the environment. Tell me how
>>> you are going to do this WITHOUT invoking the mount command!
>>
>> ??? Did you read I typed 'mount'???
>
>Yes!  And you should read that I said "REGARDLESS OF THE FILESYSTEM(s)
>contained thereon".  You've missed the very ESSENCE of my question!

Well by now I think I figured the essence, that what you are doing is impossible.


>By typing "mount", you are relying on mount's UNDERSTANDING of the
>filesystem.  What do you do when you type "mount" and you get
>a reply:
>     Operation not supported by device
>(or, whatever the equivalent "kernel lacks support for the filesystem
>indicated by the device's contents)?
>
>I.e., add a line after your "mount" in your script that begins
>with:
>       if [ "$?" -ne "0" ]; then
>           echo "Gee, I can't mount that sort of filesystem!  I'll handle  \
>              this in some other way..."
>
>and contains whatever commands you deem necessary to create that image!
>
>> Maybe you should read my other replies where I stated you need s similar
>> file system for 'regeneration' for exampel file name length.
>>
>>>> tar that filesystem: tar -zcvf my_sdd1_backup.tgz /mnt/sdd1/*
>>>>
>>>> If the partition has no files the tgz will be very very small.
>>>
>>> Try gzip'ing /dev/sdd1 and look at the size of the resulting file! (i.e.,
>>> /dev/sdd1 being the raw/block device without ANY knowledge of the
>>> filesystem it is currently supporting!)
>>
>> When you use mount it will TELL you the filesystem.
>
>No, it won't.  it will only tell you which filesystems it *recognizes*
>(typically by examining magic numbers).  And, will only actually *do*
>the mount if the specific mount_<fstype> is executable in your system.
>
>I have a laptop in my hands.  (actually, I have six of them)
>They were purchased in their current state.  The partition
>table contains partitions having magic numbers of 0xde, 0x07,
>0x0f and 0xdb.


Well I told you all that, in my first reply (with script).
use fdisk, remove all partitions, make new ones,
put one or more decent filesystems on it,
and untar the data (you DID have some files at some point or else this discussion is pointless or file-less)
and your done.
If not install Linux on it and start fresh all over.


<snipped rant>

Look, you want to compress a partition image without knowing what filesystem is on it,
that leaves gzip (or maybe zip).

Get a life.

On 04/11/14 09:28, Dimiter_Popoff wrote:
> On 04.11.2014 г. 09:55, David Brown wrote:
>> On 04/11/14 03:10, DecadentLinuxUserNumeroUno wrote:
>>> On Tue, 4 Nov 2014 01:31:50 +0000 (UTC), Andrew Smallshaw
>>> <andrews@sdf.lonestar.org> Gave us:
>>>
>>>> No, on _any_ general purpose file system.  It is inevitable.  You
>>>> seem to be of the impression that Linux is somehow the ultimate
>>>> system,
>>>
>>>   I was "under the impression" that the ext series of file systems, as
>>> well as MS' NTFS fought such occurrences with a different operation and
>>> management paradigm, than the old, FAT type method.
>>>
>>
>> There are three major things that affect the fragmentation rate on a
>> filesystem, and a number of things that influence the effect of the
>> fragmentation.
>>
>> One is the type of filesystem.  Some, such as FAT, have a very simple
>> structure that is prone to fragmentation.
> 
> This is wrong. FAT has nothing to do with fragmentation. It is the
> allocation strategy the OS employs which is responsible. Nothing is
> stopping the OS to do worst fit on a FAT disk image (I have done it
> for example, when emulating FAT disks under DPS - the latter does
> enhanced worst fit so it was transferred automatically to FAT images
> as well).

It is certainly possible to use FAT without fragmenting, as you have
done.  As I said, filesystem type is only one aspect - filesystem
implementation and usage patterns are also important (I haven't said
anything about which factors are /most/ important, since the answer
varies too much).

> 
>> Number two is the OS and the filesystem implementation.  DOS and early
>> Windows versions had very poor caching, and very short write cache times
>> (since it was expected that the system could crash any second), leading
>> to massive fragmentation.  Linux, and newer Windows versions, take a bit
>> more time and allocate more sensibly, thus leading to far less
>> fragmentation.  Linux still does a better job than Windows, but since
>> Win7 the difference is hardly noticeable.
> 
> Cache times have nothing to do with fragmentation either. Again, it is
> just the space allocation strategy which matters. I know little of
> the inner works of windows and unix (of which apparently you know even
> less) but cached disk data would normally be cached on a sector
> basis.

Please don't start telling me what I know or don't know - I would rather
not get into a fight again because you needlessly (and perhaps
unknowingly) write in provocative terms.

When an operating system has longer write cache periods, it is able to
wait longer to allow applications to finish writing a file.  This allows
it to allocate space more efficiently, since it has a better idea of the
final file size (and perhaps the file is now closed, and the file size
is known completely).  In particular, if there are several files open at
the same time, the OS can keep them separate and reduce fragmentation.
Thus better write caching, combined with features such as delayed
allocation or allocation groups, reduces fragmentation when there are
many things being written to the disk at the same time.

> If you think the applications nowadays write in larger pieces at a time
> than they did in the past this is probably correct but has nothing to
> do with the type of filesystem regarding fragmentation.
> 

The benefits of longer write caches apply regardless of the filesystem
type.  However, some filesystem features, such as logs and better
metadata, make advanced filesystems more suitable and safer for longer
write caches than when using simpler filesystems like FAT.

On 04.11.2014 г. 12:08, David Brown wrote:
> ......
>>> One is the type of filesystem.  Some, such as FAT, have a very simple
>>> structure that is prone to fragmentation.
>>
>> This is wrong. FAT has nothing to do with fragmentation. It is the
>> allocation strategy the OS employs which is responsible. Nothing is
>> stopping the OS to do worst fit on a FAT disk image (I have done it
>> for example, when emulating FAT disks under DPS - the latter does
>> enhanced worst fit so it was transferred automatically to FAT images
>> as well).
>
> It is certainly possible to use FAT without fragmenting, as you have
> done.  As I said, filesystem type is only one aspect - filesystem
> implementation and usage patterns are also important (I haven't said
> anything about which factors are /most/ important, since the answer
> varies too much).

So explain how is FAT "prone to fragmentation" in a way different to any
other system.


> Please don't start telling me what I know or don't know - I would rather
> not get into a fight again because you needlessly (and perhaps
> unknowingly) write in provocative terms.

So stick in your posts to things you do understand.
I will go no further, let us first see to what length you will go
to defend your "FAT being prone to fragmentation" because of its
simplicity claim.

Dimiter

On 04/11/14 12:43, Dimiter_Popoff wrote:
> On 04.11.2014 г. 12:08, David Brown wrote:
>> ......
>>>> One is the type of filesystem.  Some, such as FAT, have a very simple
>>>> structure that is prone to fragmentation.
>>>
>>> This is wrong. FAT has nothing to do with fragmentation. It is the
>>> allocation strategy the OS employs which is responsible. Nothing is
>>> stopping the OS to do worst fit on a FAT disk image (I have done it
>>> for example, when emulating FAT disks under DPS - the latter does
>>> enhanced worst fit so it was transferred automatically to FAT images
>>> as well).
>>
>> It is certainly possible to use FAT without fragmenting, as you have
>> done.  As I said, filesystem type is only one aspect - filesystem
>> implementation and usage patterns are also important (I haven't said
>> anything about which factors are /most/ important, since the answer
>> varies too much).
> 
> So explain how is FAT "prone to fragmentation" in a way different to any
> other system.
> 
> 
>> Please don't start telling me what I know or don't know - I would rather
>> not get into a fight again because you needlessly (and perhaps
>> unknowingly) write in provocative terms.
> 
> So stick in your posts to things you do understand.
> I will go no further, let us first see to what length you will go
> to defend your "FAT being prone to fragmentation" because of its
> simplicity claim.
> 

Since you failed to read what I wrote in either post, but leapt at it as
a chance to attack me again, then I see little point in posting anything
more in reply to you.