Locking semantics

Hi,

[This one is probably too subtle to sort out ahead of time.
But, maybe some insights can be gleaned?]

I'm trying to sort out the best semantics for (file) locking
operations.

I don't have a traditional filesystem.  Or, in fact, *any* sort
of "filesystem".  Rather, I have "objects" (bad choice of words)
and they are referenced via "namespaces".  A namespace can
*resemble* what most folks would consider as a filesystem
hierarchy (e.g., /devices/UART0; /log/startup; /net/connection/27;
etc.).  So, for example, the "UART0" object may reside in the
"devices" container, etc.

Each "process" has its own namespace.

Unlike traditional filesystems, my namespaces are not portions
of -- not windows into -- a single, unified filesystem (as is
typically the case in most conventional systems).

With these things in mind, a single "physical" object (for example,
that UART alluded to, above) may be known as:
   /devices/UART0                   to process1
   /my/favorite/bitbanger           to process2
   /output                          to process3
   /log *and* /connections/upstream to process4
etc.  Like having multiple hard/softlinks to the same "file",
each in a separate *jail*!

Note that /output - and it's binding to that "object" -- only
makes sense in the context of (e.g.) process3.  Process2 may
not have a "/output" in its namespace.  Or, may have a /output
that is bound to a display, etc.

However, a process can *share* a namespace (or portion thereof)
with another process.  In which case, "/output" *is* the same
"/output" in that "other" process (call it process99).  Said
another way, if /output was renamed by process3, that other
process (process99) would *see* this new name -- /output would
disappear (or, be available for reuse by either or both of those
processes -- keeping in mind that process2 could still be using
it for something entirely different).

With that as background...

Imagine process3 wants to take an exclusive lock on /output
(or, any other object that is accessible via its namespace).

I think we would agree that this should prevent process1
from accessing that "physical" object -- the UART, in this
example -- for as long as the lock is held.  Whether
process1's operations block waiting on the resource, return
an immediate error or register a callback is immaterial.
The point is, the lock applies to the *object*, not the
*name(s)*!

But, what (should) happen when process99 tries to rename
the object (in the namespace that it shares with process3)?
The *name* binding isn't locked, just the object itself!

[Think about this... if you locked the name bindings, then,
conceivably, process1 shouldn't be allowed to rename
/devices/UART0 as /devices/MyUART while the lock is held!]

However, process3 may be in the process of building a namespace
for a new process that it is about to spawn -- hence the reason
he took the lock on the object... to prevent the object itself from
changing from it's current state due to actions by "others".
I.e., process3 wants to build a new namespace and reference
that UART ("/output") *in* that namespace -- perhaps by the
exact same *name*.

How does process3 "lock" the name binding to that object -- at
least for the duration of this atomic operation it is attempting?

The workaround is to create another *private* name for the object
(using the "handle" that it currently holds) and pass *that*
binding on to the new namespace.

The other, heavy-handed approach is to take a lock on the
"container" (in this case, "/") so that no changes to the container
can be made for this duration.

Namespace operations aren't particularly expensive.  OTOH, nor
are they *cheap*!  I'm mainly concerned with reducing the
possibility of latent bugs where subtle races could potentially
exist.

Imagine a *traditional* filesystem (single, unified namespace).
You have a process that has been accessing a particular "file".
It wants to spawn another process to do some further work on that
file.  So, it wants to pass "filename" to that new process.

But, at the same time, some *other* process (unbeknownst to you)
wants to rename "filename" to "myfile" -- because it is ignorant
of *your* reference to that "name".  Depending on the order in which
these competing actions interact, the new process that you spawn
may find "filename" is "not found".

As I said, I think this is probably too subtle for a simple resolution.
I imagine I'll have to implement it one way or the other... and see how
clumsy the approach ends up -- then rework it.  :<

Op Tue, 19 May 2015 09:10:38 +0200 schreef Don Y <this@is.not.me.com>:
> [This one is probably too subtle to sort out ahead of time.
> But, maybe some insights can be gleaned?]
>
> I'm trying to sort out the best semantics for (file) locking
> operations.
>
> I don't have a traditional filesystem.  Or, in fact, *any* sort
> of "filesystem".  Rather, I have "objects" (bad choice of words)
> and they are referenced via "namespaces".  A namespace can
> *resemble* what most folks would consider as a filesystem
> hierarchy (e.g., /devices/UART0; /log/startup; /net/connection/27;
> etc.).  So, for example, the "UART0" object may reside in the
> "devices" container, etc.
>
> Each "process" has its own

...potentially partially shared...

> namespace.
>
> Unlike traditional filesystems, my namespaces are not portions
> of -- not windows into -- a single, unified filesystem (as is
> typically the case in most conventional systems).
>
> With these things in mind, a single "physical" object (for example,
> that UART alluded to, above) may be known as:
>    /devices/UART0                   to process1
>    /my/favorite/bitbanger           to process2
>    /output                          to process3
>    /log *and* /connections/upstream to process4
> etc.  Like having multiple hard/softlinks to the same "file",
> each in a separate *jail*!

A jail with holes in it?

> Note that /output - and it's binding to that "object" -- only
> makes sense in the context of (e.g.) process3.  Process2 may
> not have a "/output" in its namespace.  Or, may have a /output
> that is bound to a display, etc.
>
> However, a process can *share* a namespace (or portion thereof)
> with another process.  In which case, "/output" *is* the same
> "/output" in that "other" process (call it process99).  Said
> another way, if /output was renamed by process3, that other
> process (process99) would *see* this new name -- /output would
> disappear (or, be available for reuse by either or both of those
> processes -- keeping in mind that process2 could still be using
> it for something entirely different).

That seems unnecessarily complicated and prone to security problems.

> With that as background...
>
> Imagine process3 wants to take an exclusive lock on /output
> (or, any other object that is accessible via its namespace).

Exclusive locking is selfish.  Why not send a message to the object's  
owner?

> I think we would agree that this should prevent process1
> from accessing that "physical" object -- the UART, in this
> example -- for as long as the lock is held.  Whether
> process1's operations block waiting on the resource, return
> an immediate error or register a callback is immaterial.
> The point is, the lock applies to the *object*, not the
> *name(s)*!
>
> But, what (should) happen when process99 tries to rename
> the object (in the namespace that it shares with process3)?
> The *name* binding isn't locked, just the object itself!

It would be polite of either process99 or the namespace manager to send a  
name change notification.

> [Think about this... if you locked the name bindings, then,
> conceivably, process1 shouldn't be allowed to rename
> /devices/UART0 as /devices/MyUART while the lock is held!]
>
> However, process3 may be in the process of building a namespace
> for a new process that it is about to spawn -- hence the reason
> he took the lock on the object... to prevent the object itself from
> changing from it's current state due to actions by "others".
> I.e., process3 wants to build a new namespace and reference
> that UART ("/output") *in* that namespace -- perhaps by the
> exact same *name*.
>
> How does process3 "lock" the name binding to that object -- at
> least for the duration of this atomic operation it is attempting?

Why is that needed?  Once you have a pointer to the object or a "file  
descriptor", its name shouldn't matter.

> The workaround is to create another *private* name for the object
> (using the "handle" that it currently holds) and pass *that*
> binding on to the new namespace.
>
> The other, heavy-handed approach is to take a lock on the
> "container" (in this case, "/") so that no changes to the container
> can be made for this duration.
>
> Namespace operations aren't particularly expensive.  OTOH, nor
> are they *cheap*!  I'm mainly concerned with reducing the
> possibility of latent bugs where subtle races could potentially
> exist.
>
> Imagine a *traditional* filesystem (single, unified namespace).
> You have a process that has been accessing a particular "file".
> It wants to spawn another process to do some further work on that
> file.  So, it wants to pass "filename" to that new process.

No.  Normally you fork(), causing open file descriptors to be copied.

> But, at the same time, some *other* process (unbeknownst to you)
> wants to rename "filename" to "myfile" -- because it is ignorant
> of *your* reference to that "name".  Depending on the order in which
> these competing actions interact, the new process that you spawn
> may find "filename" is "not found".

Well that is what happens when you allow renaming in a shared namespace.

> As I said, I think this is probably too subtle for a simple resolution.
> I imagine I'll have to implement it one way or the other... and see how
> clumsy the approach ends up -- then rework it.  :<

That's a common adage.


-- 
(Remove the obvious prefix to reply privately.)
Gemaakt met Opera's e-mailprogramma: http://www.opera.com/mail/

On Tue, 19 May 2015 00:10:38 -0700, Don Y wrote:

> Hi,
> 
> [This one is probably too subtle to sort out ahead of time.
> But, maybe some insights can be gleaned?]
> 
> I'm trying to sort out the best semantics for (file) locking operations.
> 
> I don't have a traditional filesystem.  Or, in fact, *any* sort of
> "filesystem".  Rather, I have "objects" (bad choice of words)
> and they are referenced via "namespaces".  A namespace can *resemble*
> what most folks would consider as a filesystem hierarchy (e.g.,
> /devices/UART0; /log/startup; /net/connection/27; etc.).  So, for
> example, the "UART0" object may reside in the "devices" container, etc.
> 
> Each "process" has its own namespace.
> 
> [a lot of stuff snipped]

Have you had a look at Plan 9 from Bell Labs? Their filesystem-based 
everything seems to be *exactly* what you have here. In fact, I'm not 
even sure you're not using Plan 9. :)

It might be a good idea to just shamelessly rip off Plan 9's file locking.

This might be a good place to start:
http://doc.cat-v.org/inferno/4th_edition/styx

Hi Don,

On 19.5.2015 г. 10:10, Don Y wrote:
> Hi,
>
> [This one is probably too subtle to sort out ahead of time.
> But, maybe some insights can be gleaned?]
>
> I'm trying to sort out the best semantics for (file) locking
> operations.

I suppose I have been through all of this last 20+ years writing
and "living" inside DPS and I think things are pretty straight
forward.

If it is a file, applications open it in a "registered" manner,
i.e. each open file has only one IOCB (input/output control block).
Tasks can "check in" to access it (there is a bitmap attached
to the IOCB for that purpose).
The IOCB can be "locked" by a task in which case the rest of the
tasks just cannot access the file. Renaming a file will take
locking the IOCB - having write access to this file - and doing
it. Lock for renaming only is nothing I have ever considered,
it would take another bit in the IOCB. But since the IOCB is
a single one system-wide there could be no serious issues with
renaming it, every registered task would have access to the
latest name.

> How does process3 "lock" the name binding to that object -- at
> least for the duration of this atomic operation it is attempting?

The more generic naming - or whatever parameter of whatever
object - I have been through (and still am when programming)
is with the DPS runtime objects. You could do all you want to
do with one of them, say "lock" for modification - save former
lock state (in case your task is locking an object it has
previously locked already) - modify - restore lock state.
The number of "lock" bits you can implement is obviously
object specific, you can grow them with new types as needed
etc.

I suppose what you are describing is still a bit too general.
While it is crucial to get things right at the foundations
I suppose your questions will get their answers automatically
as you go :-). So I posted some of my experience, some phrase
might click and be useful when you are doing this, which
I suppose is what you hope for initiation the discussion.

Dimiter


------------------------------------------------------
Dimiter Popoff               Transgalactic Instruments

http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/

On Tue, 19 May 2015 09:59:06 +0200, Boudewijn Dijkstra wrote:

> Op Tue, 19 May 2015 09:10:38 +0200 schreef Don Y <this@is.not.me.com>:

>> How does process3 "lock" the name binding to that object -- at least
>> for the duration of this atomic operation it is attempting?
> 
> Why is that needed?  Once you have a pointer to the object or a "file
> descriptor", its name shouldn't matter.

What I'm thinking as well.

You obtain a lock on the underlying object. Presumably, user processes 
talking to the kernel/namespace server can reference an object by its 
True Name, regardless of the filesystem location of the object.

But if you don't obtain locks on the object but on the filesystem/
namespace locations of objects, then obviously every filesystem/namespace 
that exposes the object needs to be made aware of the lock. Otherwise, 
how will they enforce the lock?

Or are you trying to make the third, ultra complicated option, where 
every namespace exposes the same object, and locking one exposure of an 
object by design locks only some of the other exposures while leaving the 
rest unlocked?

Hi Boudewijn,

On 5/19/2015 12:59 AM, Boudewijn Dijkstra wrote:
> Op Tue, 19 May 2015 09:10:38 +0200 schreef Don Y <this@is.not.me.com>:

>> With these things in mind, a single "physical" object (for example,
>> that UART alluded to, above) may be known as:
>>    /devices/UART0                   to process1
>>    /my/favorite/bitbanger           to process2
>>    /output                          to process3
>>    /log *and* /connections/upstream to process4
>> etc.  Like having multiple hard/softlinks to the same "file",
>> each in a separate *jail*!
>
> A jail with holes in it?

How so, "holes"?

>> Note that /output - and it's binding to that "object" -- only
>> makes sense in the context of (e.g.) process3.  Process2 may
>> not have a "/output" in its namespace.  Or, may have a /output
>> that is bound to a display, etc.
>>
>> However, a process can *share* a namespace (or portion thereof)
>> with another process.  In which case, "/output" *is* the same
>> "/output" in that "other" process (call it process99).  Said
>> another way, if /output was renamed by process3, that other
>> process (process99) would *see* this new name -- /output would
>> disappear (or, be available for reuse by either or both of those
>> processes -- keeping in mind that process2 could still be using
>> it for something entirely different).
>
> That seems unnecessarily complicated and prone to security problems.

It actually makes security *easier*!

A "process" can only access objects from which it can resolve a "name".
E.g., process77 can't reference "/output" and expect it to yield
access to the same object that process3 calls "/output".  Process77's
"/output" may have been bound to some other object (by whatever
created process77's namespace -- or, whatever *other* object process77
had a handle to (including objects that process77 may have created
or that may have been passed *to* process77 during the course of its
execution).

So, if process66 (the creator of process77) builds a namespace for
process77 (which can only reference objects that process66 can *name*),
then it is known that process77 can only reference that subset of system
objects.  E.g., if none of them are bound to the physical "UART"
device mentioned above, then there is no fear that process77 will EVER
be able to access that UART (unless someone passes a reference to the UART
*into* process77 at runtime -- and, that can only be from someone who
*has* such a reference AND has a comm link to process77.)

There are still cases where two or more processes may wish to share
a single copy of (a portion of) their individual namespaces.

>> With that as background...
>>
>> Imagine process3 wants to take an exclusive lock on /output
>> (or, any other object that is accessible via its namespace).
>
> Exclusive locking is selfish.  Why not send a message to the object's owner?

This is, in effect, what is done.  But, "owner" is the server
that is responsible for that sort of object, not some process
that may be *using* it.  I.e., all operations on the object are
actually messages to the object's server ("implementer").  So,
a lock is simply a message from someone holding a live reference
to a particular instance of the type of object that is served
stating "give me exclusive access to this object, regardless
of other 'clients' that may currently have live references to
it".   As each object is "served" from one particular server
(of thqat type), this ensures that the lock is applied (on what
I called the "physical object") universally and consistently.

>> I think we would agree that this should prevent process1
>> from accessing that "physical" object -- the UART, in this
>> example -- for as long as the lock is held.  Whether
>> process1's operations block waiting on the resource, return
>> an immediate error or register a callback is immaterial.
>> The point is, the lock applies to the *object*, not the
>> *name(s)*!
>>
>> But, what (should) happen when process99 tries to rename
>> the object (in the namespace that it shares with process3)?
>> The *name* binding isn't locked, just the object itself!
>
> It would be polite of either process99 or the namespace manager to send a name
> change notification.

Being "polite" isn't the same as *requiring* that.  Or, "enforcing"
it.

I.e., the "best practices" approach would be to create another
private portion of the namespace and bind a name to the object,
there.  So, only *you* can twiddle with that instance of the
binding.  Then, pass *this* on to the new process that you are
creating (see immediately following).

>> [Think about this... if you locked the name bindings, then,
>> conceivably, process1 shouldn't be allowed to rename
>> /devices/UART0 as /devices/MyUART while the lock is held!]
>>
>> However, process3 may be in the process of building a namespace
>> for a new process that it is about to spawn -- hence the reason
>> he took the lock on the object... to prevent the object itself from
>> changing from it's current state due to actions by "others".
>> I.e., process3 wants to build a new namespace and reference
>> that UART ("/output") *in* that namespace -- perhaps by the
>> exact same *name*.
>>
>> How does process3 "lock" the name binding to that object -- at
>> least for the duration of this atomic operation it is attempting?
>
> Why is that needed?  Once you have a pointer to the object or a "file
> descriptor", its name shouldn't matter.

You're thinking too "monolithic kernel".  What do you do when the "new
process" is on some other/remote node?  Or, when you *want* to pass
a "symbolic reference" in the form of an object name?

>> The workaround is to create another *private* name for the object
>> (using the "handle" that it currently holds) and pass *that*
>> binding on to the new namespace.
>>
>> The other, heavy-handed approach is to take a lock on the
>> "container" (in this case, "/") so that no changes to the container
>> can be made for this duration.
>>
>> Namespace operations aren't particularly expensive.  OTOH, nor
>> are they *cheap*!  I'm mainly concerned with reducing the
>> possibility of latent bugs where subtle races could potentially
>> exist.
>>
>> Imagine a *traditional* filesystem (single, unified namespace).
>> You have a process that has been accessing a particular "file".
>> It wants to spawn another process to do some further work on that
>> file.  So, it wants to pass "filename" to that new process.
>
> No.  Normally you fork(), causing open file descriptors to be copied.

I can spawn another "process" on any physical node in the system
(for which I happen to have "rights").  Likewise, a process that I
spawn can be instantiated wherever the system decides is appropriate
(based on current and future workloads).

Passing an open "file descriptor" (remember, these aren't really
"files") is the same as passing an anonymous object to another
process.  If I truly want to *share* that file descriptor (and
not just the current *copy* of it), then I have to arrange for
any changes made to it by ProcessA to be seemlessly reflected back
to ProcessB (this is expensive as A and B need not be collocated)

>> But, at the same time, some *other* process (unbeknownst to you)
>> wants to rename "filename" to "myfile" -- because it is ignorant
>> of *your* reference to that "name".  Depending on the order in which
>> these competing actions interact, the new process that you spawn
>> may find "filename" is "not found".
>
> Well that is what happens when you allow renaming in a shared namespace.

Yes.  In a traditional file hierarchy, it is a potential problem
any time *any* process changes an existing name.  So, you adopt
conventions to minimize the chances of this happening -- and/or
use ACL's to protect it from happening (sure, B can access it
but can't access the binding of its name in the directory, etc.)

When *everything* is accessed via (your) namespace, this gets
potentially more difficult.  More opportunities for conflict
(because the writer of X hadn't imagined someone writing Y
and failing to address this competition.)

>> As I said, I think this is probably too subtle for a simple resolution.
>> I imagine I'll have to implement it one way or the other... and see how
>> clumsy the approach ends up -- then rework it.  :<
>
> That's a common adage.

<frown>  But an expensive one!

Time for my nap...

On 5/19/2015 4:25 AM, Aleksandar Kuktin wrote:
> On Tue, 19 May 2015 09:59:06 +0200, Boudewijn Dijkstra wrote:
>
>> Op Tue, 19 May 2015 09:10:38 +0200 schreef Don Y <this@is.not.me.com>:
>
>>> How does process3 "lock" the name binding to that object -- at least
>>> for the duration of this atomic operation it is attempting?
>>
>> Why is that needed?  Once you have a pointer to the object or a "file
>> descriptor", its name shouldn't matter.
>
> What I'm thinking as well.
>
> You obtain a lock on the underlying object. Presumably, user processes
> talking to the kernel/namespace server can reference an object by its
> True Name, regardless of the filesystem location of the object.

What if the name is changed?

> But if you don't obtain locks on the object but on the filesystem/
> namespace locations of objects, then obviously every filesystem/namespace
> that exposes the object needs to be made aware of the lock. Otherwise,
> how will they enforce the lock?

See below.

> Or are you trying to make the third, ultra complicated option, where
> every namespace exposes the same object, and locking one exposure of an
> object by design locks only some of the other exposures while leaving the
> rest unlocked?

The issue is "how do you obtain a lock on the *name* (binding)".

A namespace is, itself, an object.  When you want to "lookup" a name
in a namespace, add a name to a namespace, remove, rename, etc.
you pass a "handle" to that namespace to the namespace server (which
is typically replicated on each node as most processes don't deal
with more than their *own* namespace -- yes, you can manipulate
some *other* process's namespace if you have the right to do so!)
along with the operation you want performed, any required credentials
and parameters, etc.

The namespace server then performs the operation for you.  (i.e., *you*
don't keep track of the contents of your namespace -- as it is possible
that they can be changed while you are accessing it... just like another
process could add a file to the global namespace that is typically called
a filesystem)

So, I can just as easily place locks on *names* as I can on the objects
that they *reference* -- using the exact same mechanisms, just imposed
in different servers (i.e., a "file/object lock" is implemented in the
server that manages that particular object.  Anyone holding a handle for
that "physical" (bad choice of word) object has to (similarly) pass
that handle ("along with the operation you want performed, any required
credentials and parameters, etc.") to the server for that object.  *It*
knows that the object is locked, who currently holds the lock, what
criteria apply to that lock, etc.  REGARDLESS OF WHERE IN THE PHYSICAL,
DISTRIBUTED SYSTEM those clients and objects reside.  The server decides
whether and how it will respond to those "other" operation requests
on the referenced object.  (Likewise, it can know to *grant* those
operations to the lock holder -- because it knows which requests
are coming *from* the lock holder... including the eventual request
to release the lock!)

Repeat this description but replace "namespace" for "object".  I.e.,
a lock on a name in a namespace can be similarly encoded in *one*
place (regardless of how many processes have access to that namespace)
and enforced as such.

Doing this in a monolithic filesystem (single namespace) is difficult,
at best.  Sure, you can lock a file (contents).  But, how do you
prevent its name from being changed?  You need very fine-grained
ACLs for the file and/or "container" (directory object) so you could
prevent changes to individual directory entries by *some* processes
while not interfering with directory operations for *other* entries.

Said another way:  write a piece of code that only allows processX
to modify the name of a particular file in a *large* directory
while not preventing processY from adding, renaming, unlinking,
etc. OTHER names in that same directory.  And, once processX is done
with its actions, allow processZ to impose the same sorts of restrictions
(while it "does its thing")

On 5/19/2015 4:13 AM, Aleksandar Kuktin wrote:
> On Tue, 19 May 2015 00:10:38 -0700, Don Y wrote:

>> I'm trying to sort out the best semantics for (file) locking operations.
>>
>> I don't have a traditional filesystem.  Or, in fact, *any* sort of
>> "filesystem".  Rather, I have "objects" (bad choice of words)
>> and they are referenced via "namespaces".  A namespace can *resemble*
>> what most folks would consider as a filesystem hierarchy (e.g.,
>> /devices/UART0; /log/startup; /net/connection/27; etc.).  So, for
>> example, the "UART0" object may reside in the "devices" container, etc.
>>
>> Each "process" has its own namespace.
>>
>> [a lot of stuff snipped]
>
> Have you had a look at Plan 9 from Bell Labs? Their filesystem-based
> everything seems to be *exactly* what you have here. In fact, I'm not
> even sure you're not using Plan 9. :)
>
> It might be a good idea to just shamelessly rip off Plan 9's file locking.
>
> This might be a good place to start:
> http://doc.cat-v.org/inferno/4th_edition/styx

I don't think Plan 9 supports file locking as an OS primitive
(at least, Inferno doesn't and much of inferno is derived from
Plan 9's concepts/implementation.

[I'm not keen on the "in channel text" by which "commands" are
invoked on "objects" -- too much parsing required:  "Did I just
receive a message to SET the baudrate?  Or, GET the baudrate?"
Likewise, too much error checking on syntax that must be done in
the object implementation:  "SET BAUD <rate>" vs. "GET BAUD".
In my approach, everything is just an RPC:  set_baudrate(device, rate)
or get_baudrate(device) -- so, the compiler can do a lot of the
checking, type enforcement, etc. for you]

My namespaces come from Inferno but the mechanisms for distributing and
manipulating them (along with other objects) is more closely attuned with
Mach.  In particular, my introduction of "credentials" and fine-grained
access control relies heavily on Mach's abstractions.

For (silly) example, I can let task1 set the baudrate on a device -- but
not alter the "word format" (number of stop bits, etc.).  And, at the same
time, prohibit it (task1) from sending or receiving characters through
that device -- while task2 can freely push/pull data yet is prevented
from altering the baudrate.

(things that aren't possible in a more conventional namespace/ACL).

E.g., a "safety" task can be given control over the brakes on a
mechanism yet prevented from otherwise altering the *operation* of that
mechanism.  There's no way to provide that level of access control using
traditional semantics.

Hi Dimiter,

On 5/19/2015 4:24 AM, Dimiter_Popoff wrote:
> On 19.5.2015 г. 10:10, Don Y wrote:

>> I'm trying to sort out the best semantics for (file) locking
>> operations.
>
> I suppose I have been through all of this last 20+ years writing
> and "living" inside DPS and I think things are pretty straight
> forward.
>
> If it is a file, applications open it in a "registered" manner,
> i.e. each open file has only one IOCB (input/output control block).
> Tasks can "check in" to access it (there is a bitmap attached
> to the IOCB for that purpose).
> The IOCB can be "locked" by a task in which case the rest of the
> tasks just cannot access the file. Renaming a file will take
> locking the IOCB - having write access to this file - and doing

So, how do you support two (or more) names pointing to the same file?

E.g., I might have, on one of my UN*X boxes, 20 or 30 file *names*
(entries in a directory) that all reference the same "set of bytes"
(i.e., "program") on the disk.  This is a technique used in creating
a "crunch'ed" file -- a single executable that actually represents
many different programs mashed together.

The first thing this sort of "program" does is figure out how it was
invoked -- i.e., by which *name* it was invoked.  Then, based on that
name, one of N "main()'s" contained in the file are given control.
The overall effect is that you can link the traditional library routines
("overhead") that most programs require into this crunched "program"
exactly once and have them accessible from any of the N "main()'s"
that are embodied in it.

int main(int argc, char **argv) {
    ...
    if (strcmp(argv[0], "copy") {
        return mainCOPY(argc, argv);
    } else if (strcmp(argv[0], "delete") {
        return mainDELETE(argc, argv);
    } else if (strcmp(argv[0], "fly") {
        return mainFLY(argc, argv);
    } ...

    } else {
        fprintf(stderr, "Not found\n");
        return -1;
    }
}

int mainCOPY(int argc, char **argv) {
    ...
}

int mainDELETE(int argc, char **argv) {
    ...
}

In this case, there would be directory entries for "copy", "delete",
"fly", etc. -- all pointing to the *one* instance of "program" on
the medium.

> it. Lock for renaming only is nothing I have ever considered,
> it would take another bit in the IOCB. But since the IOCB is
> a single one system-wide there could be no serious issues with
> renaming it, every registered task would have access to the
> latest name.
>
>> How does process3 "lock" the name binding to that object -- at
>> least for the duration of this atomic operation it is attempting?
>
> The more generic naming - or whatever parameter of whatever
> object - I have been through (and still am when programming)
> is with the DPS runtime objects. You could do all you want to
> do with one of them, say "lock" for modification - save former
> lock state (in case your task is locking an object it has
> previously locked already) - modify - restore lock state.
> The number of "lock" bits you can implement is obviously
> object specific, you can grow them with new types as needed
> etc.
>
> I suppose what you are describing is still a bit too general.
> While it is crucial to get things right at the foundations
> I suppose your questions will get their answers automatically
> as you go :-). So I posted some of my experience, some phrase
> might click and be useful when you are doing this, which
> I suppose is what you hope for initiation the discussion.

What I am concerned about is developerA deciding on a use
for some set of objects/"names" at some point in time.  He
implicitly expects his use of them (the names *and* the objects)
to be "exclusive" ("Why would anyone ever want to use this
mumbledypeg object, 'foo'?  It shouldn't be of value to anyone
OTHER THAN me!").

If the mechanism for locking a *name* is tedious/expensive/error prone,
then he will PROBABLY err on the side of laziness (after all, who could
possibly want to use that name/object?!).

Sometime later, developerB sees that developerA has done a fair bit
of his work for him in creating and manipulating "foo".  So, being
lazy, developerB tries to leverage that existing effort by referencing
"foo" -- potentially ALONGSIDE developerA's reference!

Everything *seems* to work fine... until, one day, it doesn't!
A (or B) loses a race and the corresponding code fails.  "Can't
happen" actually *does* happen!

And, identifying the actual cause of the problem may be deeply
involved/convoluted depending on how those resources are used.

Even if you can point to developerA as the initial cause of the problem,
he/she may no longer be available; their sources may be gone (or "closed");
the toolchain to support the binary may no longer be conveniently
available, etc.

So, it would be nicer/safer/more robust to have mechanisms that a
developer might need *available* instead of forcing the developer
to create an ad hoc approach -- possibly faulty -- to get the
results his code *requires*.

On 20.5.2015 г. 07:49, Don Y wrote:
> Hi Dimiter,
>
> On 5/19/2015 4:24 AM, Dimiter_Popoff wrote:
>> On 19.5.2015 г. 10:10, Don Y wrote:
>
>>> I'm trying to sort out the best semantics for (file) locking
>>> operations.
>>
>> I suppose I have been through all of this last 20+ years writing
>> and "living" inside DPS and I think things are pretty straight
>> forward.
>>
>> If it is a file, applications open it in a "registered" manner,
>> i.e. each open file has only one IOCB (input/output control block).
>> Tasks can "check in" to access it (there is a bitmap attached
>> to the IOCB for that purpose).
>> The IOCB can be "locked" by a task in which case the rest of the
>> tasks just cannot access the file. Renaming a file will take
>> locking the IOCB - having write access to this file - and doing
>
> So, how do you support two (or more) names pointing to the same file?

I could easily do but I do not on purpose. Nothing is stopping me
from having multiple directory entries pointing to the same piece
of data or to some other directory entry which points to the data
etc.
However, this has implications - new directory entry type (so the
data gets deallocated only for one of the directory entries) or just
making a "link" type directory entry which may easily turn out
to point to nothing (user deletes the file it points to). Solvable
issues but adding complexity and no serious benefit.

> E.g., I might have, on one of my UN*X boxes, 20 or 30 file *names*
> (entries in a directory) that all reference the same "set of bytes"
> (i.e., "program") on the disk.  This is a technique used in creating
> a "crunch'ed" file -- a single executable that actually represents
> many different programs mashed together.

Under DPS this is solved in a much more general - and clean - way.
Each piece of executable code is a "program module" which is accounted
for by the OS (keeping track on where it came from into RAM) and,
if reentrant/position independent (all I do are that) it is used by
any task which wants to use it.
Then each module can be of a "program library module" type containing
multiple, 64 bit identifiable, entry points. DPS keeps track by
whom a module is used so it can be discarded/deallocated when no
longer needed etc. This mechanism is widely used by various layers
(e.g. some "programs" call such module functions directly, whereas
a DPS object can have some of its actions located in a .plm file
(.plm being obviously "program libray module" :D ).

>>> How does process3 "lock" the name binding to that object -- at
>>> least for the duration of this atomic operation it is attempting?
>>
>> The more generic naming - or whatever parameter of whatever
>> object - I have been through (and still am when programming)
>> is with the DPS runtime objects. You could do all you want to
>> do with one of them, say "lock" for modification - save former
>> lock state (in case your task is locking an object it has
>> previously locked already) - modify - restore lock state.
>> The number of "lock" bits you can implement is obviously
>> object specific, you can grow them with new types as needed
>> etc.
>>
>> I suppose what you are describing is still a bit too general.
>> While it is crucial to get things right at the foundations
>> I suppose your questions will get their answers automatically
>> as you go :-). So I posted some of my experience, some phrase
>> might click and be useful when you are doing this, which
>> I suppose is what you hope for initiation the discussion.
>
> What I am concerned about is developerA deciding on a use
> for some set of objects/"names" at some point in time.  He
> implicitly expects his use of them (the names *and* the objects)
> to be "exclusive" ("Why would anyone ever want to use this
> mumbledypeg object, 'foo'?  It shouldn't be of value to anyone
> OTHER THAN me!").

Ah I see now. I suppose in my vocabulary this would be "name
reservation". Can you not simply prevent the developer from
using names which have already been used? Like in any language,
"duplicate symbol definition" etc. sort of thing error if he
tries? May be I am still not getting what you are after.

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/