Disk imaging strategy

Hi,

I'm writing a bit of code to image disk contents REGARDLESS OF THE
FILESYSTEM(s) contained thereon.

This doesn't have to be "ideal" (defined as "effortless", "minimal
image size", etc.) but should be pretty close.

It is not intended to be performed often -- "write once, read multiple"
(i.e., RESTORE *far* more often than IMAGE).

The challenge comes in the filesystem(s) neutral aspect.  E.g., I
should be able to image a disk containing FAT32, NTFS, FFSv1/2, QFS,
individual RAID* volumes, little/BIG endian, etc. -- with the same
executable!

A naive approach to this would be to plumb dd to a compressor -- running
both OUTSIDE the native OS.  But, for large/dirty volumes, this gives you
an unacceptably large resulting image -- because you end up having to store
"discarded data" which could potentially be HUGE (consider a large volume
that has seen lots of write/delete cycles) esp in comparison with the
actual precious data!

[I'd like to be able to store the image on a (set of) optical media and/or
an unused "partition" somewhere]

I.e., without knowledge of the specific filesystem(s) involved, you don't
know how to recognize live data from deleted data.

The *hack* that I am currently evaluating is to invoke a trivial executable
UNDER THE NATIVE OS that simply creates large "blank" (i.e., highly
compressible) files until the volume is "full", then unlinks them all.
Doing this while the system is reasonably quiescent isn't guaranteed to
"vacuum" all available space but would make a big dent in it (if the
system is brought down shortly thereafter).

[Yes, I understand privilege constraints in various OS's, quotas, etc.
Those are all easy to work around...]

Then, dd | compress (on bare iron).

Again, not ideal but probably the best bang for the least buck?

On a sunny day (Sun, 02 Nov 2014 08:25:31 -0700) it happened Don Y
<this@is.not.me.com> wrote in <m35ida$150$1@speranza.aioe.org>:

>Hi,
>
>I'm writing a bit of code to image disk contents REGARDLESS OF THE
>FILESYSTEM(s) contained thereon.
>
>This doesn't have to be "ideal" (defined as "effortless", "minimal
>image size", etc.) but should be pretty close.
>
>It is not intended to be performed often -- "write once, read multiple"
>(i.e., RESTORE *far* more often than IMAGE).
>
>The challenge comes in the filesystem(s) neutral aspect.  E.g., I
>should be able to image a disk containing FAT32, NTFS, FFSv1/2, QFS,
>individual RAID* volumes, little/BIG endian, etc. -- with the same
>executable!
>
>A naive approach to this would be to plumb dd to a compressor -- running
>both OUTSIDE the native OS.  But, for large/dirty volumes, this gives you
>an unacceptably large resulting image -- because you end up having to store
>"discarded data" which could potentially be HUGE (consider a large volume
>that has seen lots of write/delete cycles) esp in comparison with the
>actual precious data!
>
>[I'd like to be able to store the image on a (set of) optical media and/or
>an unused "partition" somewhere]
>
>I.e., without knowledge of the specific filesystem(s) involved, you don't
>know how to recognize live data from deleted data.
>
>The *hack* that I am currently evaluating is to invoke a trivial executable
>UNDER THE NATIVE OS that simply creates large "blank" (i.e., highly
>compressible) files until the volume is "full", then unlinks them all.
>Doing this while the system is reasonably quiescent isn't guaranteed to
>"vacuum" all available space but would make a big dent in it (if the
>system is brought down shortly thereafter).
>
>[Yes, I understand privilege constraints in various OS's, quotas, etc.
>Those are all easy to work around...]
>
>Then, dd | compress (on bare iron).
>
>Again, not ideal but probably the best bang for the least buck?

Not quite sure whatyouwant,
but I have done this a lot:
start some recue disk, plug in some USB disk.
mount the partition you want, then:
tar -zcvf partition_sda1_image.tgz /dev/sda1

I've done this with my Raspberries SDcards by just putting the card in the PC and mounting it.


script:
/usr/local/sbin/backup-raspi-sd-card

echo "Unmount and remove any USB memory sticks" 
echo "Insert and power seagate USB harddisk, type ENTER when ready"
read user_reply

sleep 4
mount /dev/sdc8 /mnt/sdc8

cd /mnt/sdc8/backups/raspberry_pi/raspi_backups/

ls -rtl
df

echo "check for enough diskspace, <= raspi SDcard size"

echo "raspi SDcard number?"
read sd_card_number

echo "$sd_card_number"

echo "Insert raspberry SDcard, type ENTER when ready"
read user_reply

mount /dev/sdb1 /mnt/sdb1

date=`/bin/date -u +%d_%m_%Y`

command="tar -zcvf raspberry_pi_sd_card_$sd_card_number-sdb1_$date.tgz /mnt/sdb1/*"

echo "about to execute: $command",

echo "control C aborts, type ENTER to proceed"
read user_reply

echo "please wait"

$command

umount /dev/sdb1

ls -rtl

###exit
echo

mount /dev/sdb2 /mnt/sdb2

command="tar -zcvf raspberry_pi_sd_card_$sd_card_number-sdb2_$date.tgz /mnt/sdb2/*"
echo "about to execute: $command",

echo "control C aborts, type ENTER to proceed"
read user_reply

echo "please wait"
$command

umount /dev/sdb2

ls -rtl
cd
df
umount /dev/sdc8

echo

echo "unplug and power down seagate USB harddisk"

echo "Ready"

------------------------------
The above script copies and compresses data in partitions from the Raspberry Pi sdcard to my 1TB Seagate USb hardisk sdc8.

Once a card is f*cked up, you just make a new partition with fdisk,
make a new filesystem on it with mkfs,
mount it, go to it, and tar -zxvf ...tgz
and everything runs.
The wole system backed up,
this is safer than dd if=/dev/sdc of=myimage , also a lot smaller, AND allows you to restore on a bigger filesystem / card.
no idea of any of this helps.
If you want to do forensics say look for 0sama or 0bama you can dd via grep I guess if the filesystem is not encrypted,.
dd if=/dev/sdc | grep 0bama
youneverknow
then find sector and arrest the guy


 


 

In comp.arch.embedded Don Y <this@is.not.me.com> wrote:
 
> I'm writing a bit of code to image disk contents REGARDLESS OF THE
> FILESYSTEM(s) contained thereon.
 
> This doesn't have to be "ideal" (defined as "effortless", "minimal
> image size", etc.) but should be pretty close.
 
> It is not intended to be performed often -- "write once, read multiple"
> (i.e., RESTORE *far* more often than IMAGE).
 
> The challenge comes in the filesystem(s) neutral aspect.  E.g., I
> should be able to image a disk containing FAT32, NTFS, FFSv1/2, QFS,
> individual RAID* volumes, little/BIG endian, etc. -- with the same
> executable!

It doesn't get interesting until you get to IBM CKD disks...
 
> A naive approach to this would be to plumb dd to a compressor -- running
> both OUTSIDE the native OS.  But, for large/dirty volumes, this gives you
> an unacceptably large resulting image -- because you end up having to store
> "discarded data" which could potentially be HUGE (consider a large volume
> that has seen lots of write/delete cycles) esp in comparison with the
> actual precious data!

For older disks, that are usually relatively small, that is probably
the best choice.
 
> [I'd like to be able to store the image on a (set of) optical media and/or
> an unused "partition" somewhere]
 
> I.e., without knowledge of the specific filesystem(s) involved, you don't
> know how to recognize live data from deleted data.
 
> The *hack* that I am currently evaluating is to invoke a trivial executable
> UNDER THE NATIVE OS that simply creates large "blank" (i.e., highly
> compressible) files until the volume is "full", then unlinks them all.
> Doing this while the system is reasonably quiescent isn't guaranteed to
> "vacuum" all available space but would make a big dent in it (if the
> system is brought down shortly thereafter).

Be sure to fsck or chkdsk first. 
 
> [Yes, I understand privilege constraints in various OS's, quotas, etc.
> Those are all easy to work around...]
 
> Then, dd | compress (on bare iron).

I usually use an ATA or SATA to USB adaptor, and then copy from
the USB side. I presume you are copying partitions, not the whole
disk including the partition table.
 
> Again, not ideal but probably the best bang for the least buck?

-- glen

On 11/2/2014 10:09 AM, glen herrmannsfeldt wrote:

>> A naive approach to this would be to plumb dd to a compressor -- running
>> both OUTSIDE the native OS.  But, for large/dirty volumes, this gives you
>> an unacceptably large resulting image -- because you end up having to store
>> "discarded data" which could potentially be HUGE (consider a large volume
>> that has seen lots of write/delete cycles) esp in comparison with the
>> actual precious data!
>
> For older disks, that are usually relatively small, that is probably
> the best choice.

The problem comes with newer disks.  E.g., I keep ~1T on each workstation
and *only* drag "current projects" onto them counting on the file servers
to maintain most of my stuff "semi-offline".

So, you can easily have just 5 or 10% of the disk "in use" but 90% of it
"dirty".  Dirt doesn't compress nearly as well as "virgin media"  :>

>> [I'd like to be able to store the image on a (set of) optical media and/or
>> an unused "partition" somewhere]
>
>> I.e., without knowledge of the specific filesystem(s) involved, you don't
>> know how to recognize live data from deleted data.
>
>> The *hack* that I am currently evaluating is to invoke a trivial executable
>> UNDER THE NATIVE OS that simply creates large "blank" (i.e., highly
>> compressible) files until the volume is "full", then unlinks them all.
>> Doing this while the system is reasonably quiescent isn't guaranteed to
>> "vacuum" all available space but would make a big dent in it (if the
>> system is brought down shortly thereafter).
>
> Be sure to fsck or chkdsk first.

Yes, of course.  The point is that I am willing to expend a fair bit of
effort -- including "unscripted" actions -- to get the initial "master"
disk image "Correct".  But, want most of that effort to be in the native OS
instead of having to implement hooks for every conceivable file system type.

>> [Yes, I understand privilege constraints in various OS's, quotas, etc.
>> Those are all easy to work around...]
>
>> Then, dd | compress (on bare iron).
>
> I usually use an ATA or SATA to USB adaptor, and then copy from
> the USB side. I presume you are copying partitions, not the whole
> disk including the partition table.

I am actually doing both.

I originally crafted a quick and dirty solution that allowed me to build
"rescue partitions" on laptops that were being donated to disadvantaged
children.  The point was to remove the "maintenance" issue from that
(pro bono) activity:  "Your computer is 'broke'?  Well, do this when
the machine boots up and your disk will be restored to the EXACT condition
it existed at the time you received it!"  (which is typically much further
developed than any "factory restore" would have created).

In those cases, *everything* but my partition was restored (some laptops
have one or two additional "partitions" besides the "restore" partition).
This allows me to really distance myself from whatever bad practices in
which they may have engaged.

The fact that *they* have to assume responsibility for the "repair" ALONG
WITH RECOVERING ANY PRECIOUS FILES BEFORE DOING SO tends to discourage
them from putting much "at risk".

But, I also needed to be able to spool the image off onto offline media
for my own archives (so, if I come across another "Model 123" laptop,
I can just load the disk from my offline copy of the "restore partition").

As this seems to have worked pretty well, I want to extend it to address
a wider variety of machines -- esp those that I run, here!  As I keep most
precious files on servers, it is relatively easy for me to adjust to
bare metal restores (I already do this on this "Internet Machine" as a
simple way of ensuring it's integrity).

But, more machines means more file system types, etc.

>> Again, not ideal but probably the best bang for the least buck?

On Sun, 02 Nov 2014 08:25:31 -0700, Don Y <this@is.not.me.com> wrote:

>Hi,
>
>I'm writing a bit of code to image disk contents REGARDLESS OF THE
>FILESYSTEM(s) contained thereon.
> :
>The challenge comes in the filesystem(s) neutral aspect.  E.g., I
>should be able to image a disk containing FAT32, NTFS, FFSv1/2, QFS,
>individual RAID* volumes, little/BIG endian, etc. -- with the same
>executable!

And regardless of interface?  ATA? SCSI? USB? IEEE-1394? SAN?  ???
Good luck with that.

Raw block access is OS agnostic, but it has a lot of drawbacks: it
happily replicates damaged filesystems and a naive image can't be
restored to a smaller volume even if the volume could hold the live
data.

It's better to have a program that understands the volume's file table
format and can copy only live files [as files].  Most systems use some
variant of either inode or FAT, so there isn't too much to figure out.
The only oddballs I'm aware of are record and log-structured systems
which you aren't likely to encounter.

I'm not a fan of imaging backup ... too many ways to get burned.


>A naive approach to this would be to plumb dd to a compressor -- running
>both OUTSIDE the native OS.  But, for large/dirty volumes, this gives you
>an unacceptably large resulting image -- because you end up having to store
>"discarded data" which could potentially be HUGE (consider a large volume
>that has seen lots of write/delete cycles) esp in comparison with the
>actual precious data!
>
>[I'd like to be able to store the image on a (set of) optical media and/or
>an unused "partition" somewhere]
>
>I.e., without knowledge of the specific filesystem(s) involved, you don't
>know how to recognize live data from deleted data.
>
>The *hack* that I am currently evaluating is to invoke a trivial executable
>UNDER THE NATIVE OS that simply creates large "blank" (i.e., highly
>compressible) files until the volume is "full", then unlinks them all.
>Doing this while the system is reasonably quiescent isn't guaranteed to
>"vacuum" all available space but would make a big dent in it (if the
>system is brought down shortly thereafter).

The big file approach can take a lot of time and can cause failures if
something suddenly needs file space when the disk is full.

Also, quite a few filesystems shortcut writes of all zeros and just
mark the affected blocks empty in their headers.  So to be safe you
have to fill write some other value.


>Then, dd | compress (on bare iron).
>
>Again, not ideal but probably the best bang for the least buck?

Yeah, until it blows up in your face.

George

Hi George,

On 11/2/2014 11:42 AM, George Neuner wrote:
> On Sun, 02 Nov 2014 08:25:31 -0700, Don Y <this@is.not.me.com> wrote:
>
>> I'm writing a bit of code to image disk contents REGARDLESS OF THE
>> FILESYSTEM(s) contained thereon.
>> :
>> The challenge comes in the filesystem(s) neutral aspect.  E.g., I
>> should be able to image a disk containing FAT32, NTFS, FFSv1/2, QFS,
>> individual RAID* volumes, little/BIG endian, etc. -- with the same
>> executable!
>
> And regardless of interface?  ATA? SCSI? USB? IEEE-1394? SAN?  ???
> Good luck with that.

If the machine can access the medium, then what do I care about the
hardware interface?

For every (?) machine, there are certain assumptions that can be made
at IPL:  one of which is that it can retrieve some number of bytes
from the boot volume.  I.e., it need not be aware of the file system's
structure or even how to *write* to that medium.  But, it must have the
ability to fetch the initial loader.

What you load is largely a matter of your discretion.  If you don't
need to understand what you are restoring (or imaging), then all you
need to do is ensure you can extend that initial capability to one
that allows you to read (write) arbitrary portions of the medium.

If you want to image/restore a medium that is not natively supported,
tough!

> Raw block access is OS agnostic, but it has a lot of drawbacks: it
> happily replicates damaged filesystems and a naive image can't be
> restored to a smaller volume even if the volume could hold the live
> data.

Point isn't to restore to arbitrary media but, rather, to the medium
from which it came.

As to "damaged filesystems", deal with that BEFORE you make the image!

> It's better to have a program that understands the volume's file table
> format and can copy only live files [as files].  Most systems use some
> variant of either inode or FAT, so there isn't too much to figure out.
> The only oddballs I'm aware of are record and log-structured systems
> which you aren't likely to encounter.
>
> I'm not a fan of imaging backup ... too many ways to get burned.

I've not encountered a problem, yet -- and have been going the Clonezilla
route for my home machines for quite some time (storing the images on
removable SATA drives in a large shoebox).

Note, also, that it isn't a "backup" mechanism but, rather, a "restore"
mechanism.  I.e., once the image is created, it's only ever *restored*
(if you want to update the image, it's an expensive operation)

>> A naive approach to this would be to plumb dd to a compressor -- running
>> both OUTSIDE the native OS.  But, for large/dirty volumes, this gives you
>> an unacceptably large resulting image -- because you end up having to store
>> "discarded data" which could potentially be HUGE (consider a large volume
>> that has seen lots of write/delete cycles) esp in comparison with the
>> actual precious data!
>>
>> [I'd like to be able to store the image on a (set of) optical media and/or
>> an unused "partition" somewhere]
>>
>> I.e., without knowledge of the specific filesystem(s) involved, you don't
>> know how to recognize live data from deleted data.
>>
>> The *hack* that I am currently evaluating is to invoke a trivial executable
>> UNDER THE NATIVE OS that simply creates large "blank" (i.e., highly
>> compressible) files until the volume is "full", then unlinks them all.
>> Doing this while the system is reasonably quiescent isn't guaranteed to
>> "vacuum" all available space but would make a big dent in it (if the
>> system is brought down shortly thereafter).
>
> The big file approach can take a lot of time and can cause failures if
> something suddenly needs file space when the disk is full.

Sure!  You wouldn't, for example, use it on a "live server".  But, that's
the whole point -- you would image the media BEFORE the server ever went
live!

> Also, quite a few filesystems shortcut writes of all zeros and just
> mark the affected blocks empty in their headers.  So to be safe you
> have to fill write some other value.

All you need to do is ensure that whatever you write is very compressible.
Much more so than "unconstrained DEADBEEF".  E.g., you could tailor your
compressor to recognize the 512 byte sequence:
    "123234u349tuepdfjg;skjdgpa9sufwrtd....sdklfsopriujh"
and replace it with a one byte "sector is empty" code (where "empty"
really means "contains the aforementioned 512 byte sequence")

>> Then, dd | compress (on bare iron).
>>
>> Again, not ideal but probably the best bang for the least buck?
>
> Yeah, until it blows up in your face.

On 11/2/2014 8:44 AM, Jan Panteltje wrote:
> On a sunny day (Sun, 02 Nov 2014 08:25:31 -0700) it happened Don Y
> <this@is.not.me.com> wrote in <m35ida$150$1@speranza.aioe.org>:
>
>> I'm writing a bit of code to image disk contents REGARDLESS OF THE
>> FILESYSTEM(s) contained thereon.
>>
>> This doesn't have to be "ideal" (defined as "effortless", "minimal
>> image size", etc.) but should be pretty close.
>>
>> It is not intended to be performed often -- "write once, read multiple"
>> (i.e., RESTORE *far* more often than IMAGE).
>>
>> The challenge comes in the filesystem(s) neutral aspect.  E.g., I
>> should be able to image a disk containing FAT32, NTFS, FFSv1/2, QFS,
>> individual RAID* volumes, little/BIG endian, etc. -- with the same
>> executable!
>>
>> A naive approach to this would be to plumb dd to a compressor -- running
>> both OUTSIDE the native OS.  But, for large/dirty volumes, this gives you
>> an unacceptably large resulting image -- because you end up having to store
>> "discarded data" which could potentially be HUGE (consider a large volume
>> that has seen lots of write/delete cycles) esp in comparison with the
>> actual precious data!
>>
>> [I'd like to be able to store the image on a (set of) optical media and/or
>> an unused "partition" somewhere]
>>
>> I.e., without knowledge of the specific filesystem(s) involved, you don't
>> know how to recognize live data from deleted data.
>>
>> The *hack* that I am currently evaluating is to invoke a trivial executable
>> UNDER THE NATIVE OS that simply creates large "blank" (i.e., highly
>> compressible) files until the volume is "full", then unlinks them all.
>> Doing this while the system is reasonably quiescent isn't guaranteed to
>> "vacuum" all available space but would make a big dent in it (if the
>> system is brought down shortly thereafter).
>>
>> [Yes, I understand privilege constraints in various OS's, quotas, etc.
>> Those are all easy to work around...]
>>
>> Then, dd | compress (on bare iron).
>>
>> Again, not ideal but probably the best bang for the least buck?
>
> Not quite sure whatyouwant, but I have done this a lot:
> start some recue disk, plug in some USB disk.
> mount the partition you want, then:
> tar -zcvf partition_sda1_image.tgz /dev/sda1

The problem is creating "partition_sda1_image.tgz" *without* being concerned
with the underlying filesystem.  So, you have no knowledge (from the filesystem
layer) of the "valid" contents of the volume (vs. blank/deleted content).

Then, restore it *without* the rescue disk.

In sci.electronics.design Don Y <this@is.not.me.com> wrote:
> I.e., without knowledge of the specific filesystem(s) involved, you
> don't know how to recognize live data from deleted data.

You could use a program that already has this knowledge, like Partimage
http://www.partimage.org/ or (I'm pretty sure) Acronis True Image.  I
know that Partimage knows tricks like splitting the image into files of
a given size (so it fits on removable media), embedding comments into
the image files, keeping checksums on the image files, etc, and I would
assume that Acronis knows many of those same tricks.

OK, neither of these will help you back up that Amiga FFS partition
while the disk is plugged into your System/360, but for 95% of the
situations that people actually have in 2014, they work pretty well.  :)

> The *hack* that I am currently evaluating is to invoke a trivial
> executable UNDER THE NATIVE OS that simply creates large "blank"
> (i.e., highly compressible) files until the volume is "full", then
> unlinks them all.

Sometimes a fast way to do this is to create (ahead of time) a file of a
given size full of nulls, then compress it with gzip.  On your target,
unzip that file multiple times, renaming the resulting file, until the
disk is as full as you want it.  gzip is available for lots of
platforms, and its file-writing code seems to be fairly well optimized.

In other words, you do something like this ahead of time

dd if=/dev/zero of=1g.bin bs=1048576 count=1024
gzip -9 1g.bin

and bring a copy of gzip and the 1g.bin.gz file with you to the target
system.  Then, on the target system, you do something like

gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.01
gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.02
gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.03

until you fill up the disk.  (You could say something like
"zcat 1g.bin.gz > 1g.bin.01" instead, but then you're using the shell's
file handling, not gzip's.)

To get the target disk all the way full, you could make a selection of
compressed files of various sizes, and keep expanding the biggest one
that will fit.  You could have just one compressed file, and "manually"
fill up the remaining part of the disk with dd or equal.  Or you could
have just one compressed file, and decide that filling up the last
fraction of 1 GB or 10 GB or whatever isn't that important.

Some other thoughts...

I have found that sometimes NTFS doesn't really like to be full, or
close to full (like, say, over 90 or 95%).  It doesn't corrupt itself,
but it gets realllly slooooow when doing anything.  I have seen this
on NT4, 2000, and XP; I haven't tried it on anything later.

If the disk you are trying to image is an SSD, you may wish to avoid
writing more files to it than necessary.  OK, one set of extra writes
(for the null files) probably won't kill it, and it will probably be a
couple of more years before SSDs start showing up in donated computers,
but it might be worth thinking about.

Matt Roberds

On a sunny day (Sun, 02 Nov 2014 12:06:10 -0700) it happened Don Y
<this@is.not.me.com> wrote in <m35vb1$3qp$1@speranza.aioe.org>:

>>
>> Not quite sure whatyouwant, but I have done this a lot:
>> start some recue disk, plug in some USB disk.
>> mount the partition you want, then:
>> tar -zcvf partition_sda1_image.tgz /dev/sda1
>
>The problem is creating "partition_sda1_image.tgz" *without* being concerned
>with the underlying filesystem.  So, you have no knowledge (from the filesystem
>layer) of the "valid" contents of the volume (vs. blank/deleted content).

Sure you can dd that partition, but now you really are in trouble.
Its safer to tar a filesystem (that should NOT be currently running, else you are in trouble too),
you can always untar it into an other filesystem (ext2, ext4, reiserfs, etc) that is compatible.
The other thing I have noticed is that when copying partitions of disk images to bluray with my LG writer,
some bytes at the start seem to get changed, could be an error,
but this does not seem to happen when copying to a filesystem.
The other thing is in case restore of an image to a similar device,
I found that for example 8 GB card 1 (as source) has a different _real_ size than 8 GB card 2
(same make, same type, same specified size, bought at the same time)
this could be related to bad sector managing of FLASH.
If the new card (or disk) that you copy to, is smaller, then you are in trouble too.
I _always_ run a full byte by byte compare after burning a bluray...
wrote a program for that dvdimagecmp.
I use it to check FLASH images too.
All things of importance these days go to bluray kept in a dark metal case.
It has close to a thousand disks in it.

>Then, restore it *without* the rescue disk.

Na, never work on a running filesystem, UNLESS its just a file or you know _exactly_ what is going on.

Hi Matt,

On 11/2/2014 12:08 PM, mroberds@att.net wrote:
> In sci.electronics.design Don Y <this@is.not.me.com> wrote:
>> I.e., without knowledge of the specific filesystem(s) involved, you
>> don't know how to recognize live data from deleted data.
>
> You could use a program that already has this knowledge, like Partimage
> http://www.partimage.org/ or (I'm pretty sure) Acronis True Image.  I

Yes, my Clonezilla approach to date builds on partimage/partclone.
But, you shouldn't *need* to know anything about the filesystem
as that just effects "efficiency" (compressibility?)

> know that Partimage knows tricks like splitting the image into files of
> a given size (so it fits on removable media), embedding comments into
> the image files, keeping checksums on the image files, etc, and I would
> assume that Acronis knows many of those same tricks.
>
> OK, neither of these will help you back up that Amiga FFS partition
> while the disk is plugged into your System/360, but for 95% of the
> situations that people actually have in 2014, they work pretty well.  :)
>
>> The *hack* that I am currently evaluating is to invoke a trivial
>> executable UNDER THE NATIVE OS that simply creates large "blank"
>> (i.e., highly compressible) files until the volume is "full", then
>> unlinks them all.
>
> Sometimes a fast way to do this is to create (ahead of time) a file of a
> given size full of nulls, then compress it with gzip.  On your target,
> unzip that file multiple times, renaming the resulting file, until the
> disk is as full as you want it.  gzip is available for lots of
> platforms, and its file-writing code seems to be fairly well optimized.
>
> In other words, you do something like this ahead of time
>
> dd if=/dev/zero of=1g.bin bs=1048576 count=1024
> gzip -9 1g.bin
>
> and bring a copy of gzip and the 1g.bin.gz file with you to the target
> system.  Then, on the target system, you do something like
>
> gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.01
> gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.02
> gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.03
>
> until you fill up the disk.  (You could say something like
> "zcat 1g.bin.gz > 1g.bin.01" instead, but then you're using the shell's
> file handling, not gzip's.)

I figured a little piece of code that open|creat a file, pushes a particular
byte stream to that file until an error is signaled, then closes it and
tries for *another* file (in case the fwrite error was caused by max file
size) until it dies.  Then, unlinks all.  Report the total number of bytes
written (so the "user" can get a feel for whether or not the total "seems
about right").

Don't rely on much of anything from the "OS"!

> To get the target disk all the way full, you could make a selection of
> compressed files of various sizes, and keep expanding the biggest one
> that will fit.  You could have just one compressed file, and "manually"
> fill up the remaining part of the disk with dd or equal.  Or you could
> have just one compressed file, and decide that filling up the last
> fraction of 1 GB or 10 GB or whatever isn't that important.

Goal isn't to ensure every unused sector is "cleared" but, rather, that
*most* are!  E.g., you can't be sure the OS won't be dicking with the
medium while you're trying to "fill" it.  So, you could get unlucky and
find that the OS unlinks a HUGE file (or even a subdirectory full of big
files!) just after your code decided the disk was "full".  Now, all
that unlinked space has escaped your "filling".  :-/

> Some other thoughts...
>
> I have found that sometimes NTFS doesn't really like to be full, or
> close to full (like, say, over 90 or 95%).  It doesn't corrupt itself,
> but it gets realllly slooooow when doing anything.  I have seen this
> on NT4, 2000, and XP; I haven't tried it on anything later.

I would only have to do this once on each machine.  If I am going
to install that image on many machines, then the pain is spread over
those multiple instances.  E.g., if I spend a day "building" and
imaging *a* machine but can replicate that to 10 or 20 others in
*another* day, then it's a huge win!

> If the disk you are trying to image is an SSD, you may wish to avoid
> writing more files to it than necessary.  OK, one set of extra writes
> (for the null files) probably won't kill it, and it will probably be a
> couple of more years before SSDs start showing up in donated computers,
> but it might be worth thinking about.

I have two goals.  The first is to be able to create machines that users
can "maintain" (amusing choice of words given that the only hook I am
giving them is to "start over"!  :> ) without requiring assistance from
others (e.g., me, in particular).  These will tend to be PC's almost
exclusively (laptop/desktop).  But, no idea what MS will throw at them
in the future (hopefully, dealing with bare iron keeps MS's efforts out
of my hair -- even if I have to resort to pulling drives!)

[I will probably PXE boot each machine and let them pull down the
disk image from which to initialize their disk media.  For machines
that can't PXE boot, make a generic CD that does the same thing.
This should allow me to process multiple machines concurrently
(goal is to be able to do a dozen machines a week without increasing
the pro-bono time that I'm spending... ~10hrs/wk)]

The other goal is to address the machines that I build here for my own
use.  Hysterically, that has been done by installing OS, imaging disk
to secondary storage as "OS_Installed".  Installing /opt and /usr/local
stuff and then imaging as "Localization".  Then, installing whatever
apps and imaging as "Applications".  etc.

But, this is tedious to do and tedious to restore:  have to boot a
separate utility, attach secondary store with the desired image(s),
etc.

I got spoiled by watching how convenient SteadyState was!  Then, my
subsequent hack to allow *explicit* COMPLETE restores from a hacked
"factory restore" partition.  More recently, my latest "home-grown"
restore mechanism (which favors PC's).

It's just *really* cool to be able to get VM style "reinitialization"
of a system in a matter of minutes -- without the hassles of having
a VM!