Common name for a "Task Loop"

On 26.7.2016 г. 15:59, upsidedown@downunder.com wrote:
> On Tue, 26 Jul 2016 06:27:42 -0400, George Neuner
> <gneuner2@comcast.net> wrote:
>
>> On Mon, 25 Jul 2016 16:56:42 -0700, Don Y
>> <blockedofcourse@foo.invalid> wrote:
>>
>>> On 7/25/2016 2:32 PM, George Neuner wrote:
>>>
>>>>> [Note [process migration] is more involved than just packing up
>>>>> registers plus address space!]
>>>>
>>>> It isn't THAT hard: clustered mainframes in the 1960's had the ability
>>>> to migrate processes ... swap out here, swap in there.  All it really
>>>> requires is virtual addressing capability and a way to transport the
>>>> code and runtime data.
>>>
>>> Yes.  But they already had the "extra bits" (of state) that were
>>> resident IN the OS's data structures.  They either packed those
>>> up with the "(formal) process state" as it was swapped out
>>> *or* kept it in the kernel associated with the swapped out
>>> process.
>>
>> Yes.
>>
>>> For example, any network traffic that was active at the time the
>>> swap occurred still ended up with its endpoint on the current
>>> node.  You didn't have to buffer any incoming messages intended
>>> for that "to be swapped" process and later forward them to the
>>> new destination when the process is "restored".
>
> Is this any different from the situation in the old days, when you had
> to swap out complete programs from core to make room for other
> programs ? If the task had some active I/O going on, it had some
> I/O-buffers (DMA-buffers) locked in memory. The situation was quite
> nasty, especially with slow I/O such as mag tape (possibly involving a
> tape rewind).
>
> There are several alternatives:
>
> * lock the whole program in memory until I/O is complete (nasty)
>
> * just lock the I/O buffers (possibly part of a small I/O program) and
> swap that out too, when I/O is completed
>
> * abort the I/O and retry again when the program is swapped back into
> memory. Possible for read operations from mass storage
>
> The last alternative is useful also with network traffic, provided
> that the sender buffers the transmitted data until it is acknowledged
> by the receiver.
>
> With modern multicore/processors with virtual memory this should be
> trivial as long as the processors share the same physical memory bus.

Yes indeed. Then you grab your phone and discover things can be really
messed up in this context even with *gigabytes* of RAM....

Sorry for the rant, I know this is obvious to all of us here.

Dimiter

Wrong person.  DonY wrote the piece you responded to.


On Tue, 26 Jul 2016 15:59:32 +0300, upsidedown@downunder.com wrote:

>On Tue, 26 Jul 2016 06:27:42 -0400, George Neuner
><gneuner2@comcast.net> wrote:
>
>>On Mon, 25 Jul 2016 16:56:42 -0700, Don Y
>><blockedofcourse@foo.invalid> wrote:
>>
>>> :
>>>For example, any network traffic that was active at the time the
>>>swap occurred still ended up with its endpoint on the current
>>>node.  You didn't have to buffer any incoming messages intended
>>>for that "to be swapped" process and later forward them to the
>>>new destination when the process is "restored".
>
>Is this any different from the situation in the old days, when you had
>to swap out complete programs from core to make room for other
>programs ? If the task had some active I/O going on, it had some
>I/O-buffers (DMA-buffers) locked in memory. The situation was quite
>nasty, especially with slow I/O such as mag tape (possibly involving a
>tape rewind). 

If you go back to the very early days of unblocked streaming tapes,
swapping processes then was all but impossible.  But in the 60's
mainframe era, most tapes used hard block formatting, and better
drives could quickly pause and reposition the tape.

Most machines had dedicated I/O processors separate from the main
CPU(s).   The OS tracked the activity of these processors and did not
swap processes while an I/O transfer was in progress.

But since most I/O was block oriented, a large logical transfer could
be done as a series of smaller physical ones - enabling the logical
transfer to be halted and resumed.

In actual fact, early tape based mainframes quickly developed block
based tape "filesystems" that allowed data "files" on the tape to be
discontiguous.  To make it faster, multiple tapes were used as
RAID-like stripe sets - reading/writing one while seeking on another,
etc.  A file only had to be contigous on tape when it entered or left
the system.

When disks were introduced, the "filesystem" concept followed.


>There are several alternatives:
>
>* lock the whole program in memory until I/O is complete (nasty)

Which is what was done but distinguishing physical I/O from logical.

>* just lock the I/O buffers (possibly part of a small I/O program) and
>swap that out too, when I/O is completed

Too complicated.  The code/data/rss/stack segmentation of Unix
programs mirrors how early mainframes (and minis) actually worked:
pure segmentation with base:offset addressing  [the segment bases
being known only to the operating system].

I/O processors ran a single program - effectively the device driver -
which was never swapped.   Any local buffering the program may have
needed likewise would be in the processor's partitioned space.

But many I/O processors simply accessed the program's data space
buffer directly.  The buffer could not be swapped out *while* the I/O
processor was accessing it.  But again, the physical I/O could be done
incrementally with opportunity to swap between transfers.

>* abort the I/O and retry again when the program is swapped back into
>memory. Possible for read operations from mass storage

Sometimes had to be done with unblocked streaming tapes.  Once block
devices became the norm, aborting I/O for swapping never had to be
done.

>The last alternative is useful also with network traffic, provided
>that the sender buffers the transmitted data until it is acknowledged
>by the receiver.
>
>With modern multicore/processors with virtual memory this should be
>trivial as long as the processors share the same physical memory bus. 
>
>In a system with physically separate platforms, some network helper
>programs are needed to transfer data between two buffers in different
>platforms with available transfer systems, such as Ethernet.
 

YMMV,
George

On 7/26/2016 9:50 AM, George Neuner wrote:
>
> Wrong person.  DonY wrote the piece you responded to.
>
>
> On Tue, 26 Jul 2016 15:59:32 +0300, upsidedown@downunder.com wrote:
>
>> On Tue, 26 Jul 2016 06:27:42 -0400, George Neuner
>> <gneuner2@comcast.net> wrote:
>>
>>> On Mon, 25 Jul 2016 16:56:42 -0700, Don Y
>>> <blockedofcourse@foo.invalid> wrote:
>>>
>>>> :
>>>> For example, any network traffic that was active at the time the
>>>> swap occurred still ended up with its endpoint on the current
>>>> node.  You didn't have to buffer any incoming messages intended
>>>> for that "to be swapped" process and later forward them to the
>>>> new destination when the process is "restored".
>>
>> Is this any different from the situation in the old days, when you had
>> to swap out complete programs from core to make room for other
>> programs ? If the task had some active I/O going on, it had some
>> I/O-buffers (DMA-buffers) locked in memory. The situation was quite
>> nasty, especially with slow I/O such as mag tape (possibly involving a
>> tape rewind).
>
> If you go back to the very early days of unblocked streaming tapes,
> swapping processes then was all but impossible.  But in the 60's
> mainframe era, most tapes used hard block formatting, and better
> drives could quickly pause and reposition the tape.
>
> Most machines had dedicated I/O processors separate from the main
> CPU(s).   The OS tracked the activity of these processors and did not
> swap processes while an I/O transfer was in progress.
>
> But since most I/O was block oriented, a large logical transfer could
> be done as a series of smaller physical ones - enabling the logical
> transfer to be halted and resumed.
>
> In actual fact, early tape based mainframes quickly developed block
> based tape "filesystems" that allowed data "files" on the tape to be
> discontiguous.  To make it faster, multiple tapes were used as
> RAID-like stripe sets - reading/writing one while seeking on another,
> etc.  A file only had to be contigous on tape when it entered or left
> the system.
>
> When disks were introduced, the "filesystem" concept followed.

When I first adopted NetBSD (early 90's), they had no support for
9T tape (which is what I was using for "portable" tape exchange
as well as offline storage, at the time).  Originally, an F880
(800/1600bpi, 100ips)

As few folks *had* 9T transports, it fell to me to write a driver to
support my controller.  Adding the "block" device as well as the more
traditional "character" device.  So, I could build "tape filesystems"
"just as easily" as disk-based!

Conceptually, a piece of cake!  The "strategy" routine took a bit
of work to add optimizations like "read reverse"  but it was kinda cool!

However, the first time I tried to "read" a file system mounted on
the device, the "violence" of the mechanism's motions was scary!
Made me wonder if I was tearing things up with all the short
back-and-forth motions (seeks) and aggressive acceleration and
braking in the transport!  Sort of like watching a pen-plotter running
balls-out!

And, that was with files laid down in contiguous tape blocks;
the idea of the transport being commanded to seek blocks
scattered around the medium (no buffering in the transport,
formatter or controller!) led me to believe I'd quickly
wear SOMETHING out in the transport -- even if only the head
or tension arm roller!

When I got the M990 (much higher density, much deeper pockets!),
I was even *less* inclined to 'experiment'.  :<

I learned to settle for simple streaming...

On Tue, 26 Jul 2016 22:07:13 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:

>On 7/26/2016 9:50 AM, George Neuner wrote:
>>
>> In actual fact, early tape based mainframes quickly developed block
>> based tape "filesystems" that allowed data "files" on the tape to be
>> discontiguous.  To make it faster, multiple tapes were used as
>> RAID-like stripe sets - reading/writing one while seeking on another,
>> etc.  A file only had to be contigous on tape when it entered or left
>> the system.
>>
>> When disks were introduced, the "filesystem" concept followed.
>
>When I first adopted NetBSD (early 90's), they had no support for
>9T tape (which is what I was using for "portable" tape exchange
>as well as offline storage, at the time).  Originally, an F880
>(800/1600bpi, 100ips)
>
>As few folks *had* 9T transports, it fell to me to write a driver to
>support my controller.  Adding the "block" device as well as the more
>traditional "character" device.  So, I could build "tape filesystems"
>"just as easily" as disk-based!
>
>Conceptually, a piece of cake!  The "strategy" routine took a bit
>of work to add optimizations like "read reverse"  but it was kinda cool!
>
>However, the first time I tried to "read" a file system mounted on
>the device, the "violence" of the mechanism's motions was scary!
>Made me wonder if I was tearing things up with all the short
>back-and-forth motions (seeks) and aggressive acceleration and
>braking in the transport!  Sort of like watching a pen-plotter running
>balls-out!
>
>And, that was with files laid down in contiguous tape blocks;
>the idea of the transport being commanded to seek blocks
>scattered around the medium (no buffering in the transport,
>formatter or controller!) led me to believe I'd quickly
>wear SOMETHING out in the transport -- even if only the head
>or tension arm roller!

I don't doubt it.  But I think you might have hit upon a worst case
scenario for a single tape.

Mainframe tape drivers - really just an I/O program - performed
request reordering and elevator seeking to minimize thrashing of the
tapes.  Any particular tape in a big system was likely to have many
outstanding requests at any given time.

Discontiguous files were only ever used on scratch tapes that were
temporary storage.  Code images on "program" tapes always were
contiguous, as were any files on external "transport" tapes.  Input
entering the system or output leaving it always was contiguous (for
portability).

1960 era mainframes had (by today's standards) severe memory
limitations: a machine with 256K words[*] of RAM was a *BIG* system
that would be juggling dozens of jobs.   The average program used less
than 12K of memory and scripted execution of a "pipeline" of small
programs was very common.

[*] https://en.wikipedia.org/wiki/Word_(computer_architecture)


Quite a lot of routine data manipulation was done by writing/reading
temporary files, as was all passing of data between programs.  It
wasn't feasible to dedicate individual scratch tapes to every running
job, and often jobs needed multiple files simultaneously which was
hideously slow and inefficient unless the files all were on separate
tapes.

And once multitasking systems became the norm, it was no longer
politically expedient for the mainframe operators to delay jobs of
"important" users until scratch tapes were available.

Thus the discontiguous tape filesystem was born.


On my shelf, I have a textbook of "external" -  i.e. file based - data
processing algorithms.  It was written in 1988, an indication that
file based techniques were still being taught in some CS/CIS programs
then.

Modern "big data" processing has shown that file based techniques are
*still* relevant and that it was a mistake to stop teaching them:
programmers now often have no clue how to proceed when, for whatever
reason, their data can't be fit into memory. 
[Or worse: their "in-memory" processing is much slower than they
expect because because their data is swapped out due to multitasking.]

Not that still teaching file techniques today would have any great
impact on the world because the majority of programmers now have no
formal CS/CIS education.


>When I got the M990 (much higher density, much deeper pockets!),
>I was even *less* inclined to 'experiment'.  :<
>
>I learned to settle for simple streaming...

As always, YMMV.
George

Hi George,

Rain, rain, rain...  We tend to forget what 90% RH feels like when its
typically 9%!  :-/   (OTOH, never say 'no' to rain!)

On 7/27/2016 11:22 AM, George Neuner wrote:
> On Tue, 26 Jul 2016 22:07:13 -0700, Don Y
> <blockedofcourse@foo.invalid> wrote:
>
>> On 7/26/2016 9:50 AM, George Neuner wrote:
>>>
>>> In actual fact, early tape based mainframes quickly developed block
>>> based tape "filesystems" that allowed data "files" on the tape to be
>>> discontiguous.  To make it faster, multiple tapes were used as
>>> RAID-like stripe sets - reading/writing one while seeking on another,
>>> etc.  A file only had to be contigous on tape when it entered or left
>>> the system.
>>>
>>> When disks were introduced, the "filesystem" concept followed.
>>
>> When I first adopted NetBSD (early 90's), they had no support for
>> 9T tape (which is what I was using for "portable" tape exchange
>> as well as offline storage, at the time).  Originally, an F880
>> (800/1600bpi, 100ips)
>>
>> As few folks *had* 9T transports, it fell to me to write a driver to
>> support my controller.  Adding the "block" device as well as the more
>> traditional "character" device.  So, I could build "tape filesystems"
>> "just as easily" as disk-based!
>>
>> Conceptually, a piece of cake!  The "strategy" routine took a bit
>> of work to add optimizations like "read reverse"  but it was kinda cool!
>>
>> However, the first time I tried to "read" a file system mounted on
>> the device, the "violence" of the mechanism's motions was scary!
>> Made me wonder if I was tearing things up with all the short
>> back-and-forth motions (seeks) and aggressive acceleration and
>> braking in the transport!  Sort of like watching a pen-plotter running
>> balls-out!
>>
>> And, that was with files laid down in contiguous tape blocks;
>> the idea of the transport being commanded to seek blocks
>> scattered around the medium (no buffering in the transport,
>> formatter or controller!) led me to believe I'd quickly
>> wear SOMETHING out in the transport -- even if only the head
>> or tension arm roller!
>
> I don't doubt it.  But I think you might have hit upon a worst case
> scenario for a single tape.

I suspect a good part of it was due to the fact that neither the
transport, formatter nor controller did any sort of buffering.
I.e., bytes came right off the head into a latch on the ISA bus;
get it NOW or its gone!

Coupled with relatively large IRG's and low density (800/1600PE)
meant the transport had to make "noticeable" movements to do
*anything*!  (BSF/BSR, etc.)

By contrast, more modern "drives" have huge buffers, VToC's,
really high densities and much less "mass" involved so they don't
have to "work" as hard to get at data.

I think about ~50MB on a 10" BlackWatch... compared to a sizeable
fraction of a TB on a DLT-S4  (or many TB on newer LTO's <shudder>)

OTOH, modern drives have more brains in their onboard firmware
than was available in the HOSTS of most 9T's!

> Mainframe tape drivers - really just an I/O program - performed
> request reordering and elevator seeking to minimize thrashing of the
> tapes.  Any particular tape in a big system was likely to have many
> outstanding requests at any given time.
>
> Discontiguous files were only ever used on scratch tapes that were
> temporary storage.  Code images on "program" tapes always were
> contiguous, as were any files on external "transport" tapes.  Input
> entering the system or output leaving it always was contiguous (for
> portability).
>
> 1960 era mainframes had (by today's standards) severe memory
> limitations: a machine with 256K words[*] of RAM was a *BIG* system
> that would be juggling dozens of jobs.   The average program used less
> than 12K of memory and scripted execution of a "pipeline" of small
> programs was very common.
>
> [*] https://en.wikipedia.org/wiki/Word_(computer_architecture)

Yeah, my first disk (RS08) had 256K words (12b?).  Drew something like 500W
(for the drive+controller) and could source data at a whopping ~50KW/s.
If you were lucky, you could fit *four* of them in a full size rack
(for a 1 megaword store!  <drool>)

But, it was cool in that it was WORD ADDRESSABLE! (!)
[And, had a boatload of blnkenlites!]

> Quite a lot of routine data manipulation was done by writing/reading
> temporary files, as was all passing of data between programs.  It
> wasn't feasible to dedicate individual scratch tapes to every running
> job, and often jobs needed multiple files simultaneously which was
> hideously slow and inefficient unless the files all were on separate
> tapes.
>
> And once multitasking systems became the norm, it was no longer
> politically expedient for the mainframe operators to delay jobs of
> "important" users until scratch tapes were available.
>
> Thus the discontiguous tape filesystem was born.
>
> On my shelf, I have a textbook of "external" -  i.e. file based - data
> processing algorithms.  It was written in 1988, an indication that
> file based techniques were still being taught in some CS/CIS programs
> then.
>
> Modern "big data" processing has shown that file based techniques are
> *still* relevant and that it was a mistake to stop teaching them:
> programmers now often have no clue how to proceed when, for whatever
> reason, their data can't be fit into memory.
> [Or worse: their "in-memory" processing is much slower than they
> expect because because their data is swapped out due to multitasking.]

Nothing *ever* "goes obsolete" -- much like fashion; you just have to wait
for folks to rediscover *old* techniques as *new* problems manifest
and prove to no longer fit the assumptions they've artificially imposed on
their solutions!

Had to do that just last week to (brute force) process some huge "lists"
to document how I'd built the diskless workstations for my upcoming "class".
Annoying as the "files" weren't even local (NFS)!

After finishing (quick and dirty), I realized a simpler way of getting the
same results... :<

> Not that still teaching file techniques today would have any great
> impact on the world because the majority of programmers now have no
> formal CS/CIS education.

We've had this discussion before:  "teaching what industry wants"
instead of "teaching what industry WILL want".  It's the "Water Cooler
Institute of Technology" -- devoid of detail and all the caveats that
come with a more "formal" education.

I had an early employer who made no bones about this in his hiring
decisions:  "I hire from University X when I am looking for an
employee that I need TODAY; I'll hire from University Y when I'm
looking for someone that might NOT be productive, today, but will
be a better LONG TERM investment (the University *X* employee requiring
renewed education to remain viable -- almost as soon as hired!)"

It was an interesting insight to see how (some?) employers approach
their human resource needs...

>> When I got the M990 (much higher density, much deeper pockets!),
>> I was even *less* inclined to 'experiment'.  :<
>>
>> I learned to settle for simple streaming...
>
> As always, YMMV.
> George
>