RTOS popularity

On Fri, 8 Jan 2016 08:52:02 +0000, Tom Gardner
<spamjunk@blueyonder.co.uk> wrote:

>On 08/01/16 00:17, Dimiter_Popoff wrote:
>> I think you'd want to rephrase that. If system behaviour is unaltered by
>> task priorities what are they for? I believe I can guess what you mean
>> but I am not sure it is obvious enough to everyone.
>
>I've seen people notice a rare timing failure, decide that it
>was because task priorities are to blame, and that fiddling
>with the priorities will cure the problem.
>
>Occasionally such fiddling does work, but more often it either
>makes the problem more difficult to detect and cure, or creates
>a different problem.
>
>But I'm sure you know that.

In the middle of the 1980's I was running a small department with
about half a dozen programmers writing programs for the PDP-11/RSX-11
platform.

When getting a call for tenders, I first divided the problem so that
each program would fit into the 64 KiB PDP-11 address space (possibly
designing the overlay loading). After that, I split the functions into
individual tasks and assigned priorities into each task and did some
general view about messages between tasks.

After that, I made some assumption about how long it would take to
program each task. and wrote the offer. 
For some reason, the working hours in my offer was quite close to the
actual hours after the project was shipped :-).

The only times that I have had to alter process priorities was when I
had to split functionality into two or more processes.

On 08.1.2016 г. 20:57, Les Cargill wrote:
> Dimiter_Popoff wrote:
> .....
>>>
>>> I suspect if you think about it a wee bit, you'll see that priority
>>> juggling is a sign of bad system design.
>>
>> It certainly can be that - and it probably is that in the vast majority
>> of cases.
>> But the generalization is just wrong.
>>
>
> The thing that generalizes is that "if system A depends on a
> larger/more complex suite of task priority than system B, it is
> by-design inferior to system B, which has a dependency on a smaller st
> of priorities." This assuming both A & B more-or-less work.

Hmmm, I think this is a bit oversimplified. Again, I understand you
completely and probably share the same feelings about it. But what if
system A manages to do its job at say half the silicon system B
takes.... :).
If we add "all other things equal" to your rule I accept it
unreservedly of course.

> I have the conceit that we are clockmakers. When an interrupt
> fires or an event is gleaned from other input, this is like the
> escapement mechanism of a pendulum clock, that allows the system to
> deterministically move forward one notch.

Well yes but things get messier when two or more interrupts come
at the same time... IRQ latency is a key parameter in any design
really, I have gone to great length to ensure it does not vary
no matter what the OS does (this includes page fixing, in DPS this
is done with IRQ unmasked, basically DPS is a large OS with IRQ
latencies close if not equal to bare metal).

But "deterministic" means a fixed delay - i.e. no latency really;
this would be overkill to a point of being outright impractical
in the vast majority of real life situations.

> *In this case*, ...

If I ever knew what "this case" was I have long forgotten.
I entered the discussion when it became generic enough to be
interesting... :-)

>> I personally do have use for varying priorities under DPS - for
>> example in the netMCA there are various tasks some of which use
>> a lot of the CPU resource - e.g. the filtering task can take up
>> well above 50% of it depending on the incoming signal.
>
> Filters should be completely deterministic for any given bank of filters
> for a stream of fixed bandwidth.

Filters are; overall conversion calculations are not as they depend
on the incoming events (to be converted each) rate.
So this can vary a lot, from say 5-6% (no events, just filtering
for event detection) to well above 50% (maximum rate).

>
>> Then there are the VNC server tasks which check the entire
>> "display" framebuffer memory for changes, compress and send when
>> detected;
>
> If we mean the same VNC, then it's not very good software; it's just
> available.

The VNC server (like any other bit of software running on my machines)
is written by me so I'd say it is pretty good :). And It certainly was
not available before I wrote it.

On the other side there are various VNC viewers and most of them fit
your description; the best is RealVNC for windows, it works well for me.
I mean really well, no complaints whatsoever.

> But video is like that'; the formats are insane and on
> the other side, the video cards and drivers are even crazier.

RFB is a pretty sane format I'd say, as long as one picks the
right compression which the viewer can do. In my case it is
pretty much RLE, works fine, compression does not cost a lot
(up to 15% system time per VNC instance @ 400 MHz power, over
a 100 Mbps cable one just does not notice this is over VNC).

>
>> then there are the ethernet/ip inbound tasks which
>> have to cope with all the traffic...
>
> indeed; it's networking that drives much of what I am saying. You have
> to be able to drop things in a smooth and linear fashion.

Oh and it gets only "better" digging down... :). The Ethernet controller
on this SOC expects a smart DMA to do the buffering - which I had to
implement as well, multiple circular receive areas etc. Then this SDMA
does plenty of other things ("tasks"...), like disk (ATA) <-> RAM,
RAM<->RAM (off-screen buffers to "display" framebuffer), incoming 16
bit samples at several MSPS in a circular buffer, probably more.
These "tasks" have fixed (8 bit IIRC) priorities; setting these
right was crucial. Pretty advanced thing this SDMA, it is a shame
Freescale stopped making it (programming it turned out too challenging
for the typical customer today  I suppose, no other sane reason to
do that).

>
>> Not many things if any are better than simplification but it is
>> simply bound by limits like anything else. Try to simplify the netMCA
>> I talked above to equal priorities and it will become a lot more
>> complex overall.
>
> Right. I know video is always challenging.

Actually the video was the least challenging IIRC (but then I am pretty
good at it, have long since lost the count of display controllers
I have designed).

My general point is to agree with you that simplification is always
good as long as it does not turn into oversimplification.

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/

On Fri, 8 Jan 2016 12:57:35 -0600, Les Cargill
<lcargill99@comcast.com> wrote:

>Dimiter_Popoff wrote:
>> On 08.1.2016 ?. 04:44, Les Cargill wrote:
>>> Dimiter_Popoff wrote:
>>>> On 08.1.2016 ?. 02:08, Les Cargill wrote:
>>>>> upsidedown@downunder.com wrote: ......
>>>>>> The first thing in RTOS system design is to divide the
>>>>>> application to manageable tasks, estimate how long each needs
>>>>>> to run and assign priorities to the tasks.
>>>>>>
>>>>>> If there are performance problems with some tasks, never try
>>>>>> to increase the priority of the starving task, instead look for
>>>>>> other tasks, which priority can be _lowered_ without suffering
>>>>>> too much, if interrupted by a short run time high priority
>>>>>> tasks.
>>>>>>
>>>>>> Use a server task for each resource such as serial port or
>>>>>> shared data structure, so not much need for object locking
>>>>>> (with priority inversion etc.  issues).
>>>>>>
>>>>>
>>>>> I have really limited patience with designs where task priorities
>>>>> affect system behavior.
>>>>>
>>>>
>>>> I think you'd want to rephrase that. If system behaviour is
>>>> unaltered by task priorities what are they for? I believe I can
>>>> guess what you mean but I am not sure it is obvious enough to
>>>> everyone.
>>>>
>>>> Say you lower the priority of the task tracking the mouse pointer;
>>>> below some point this will inevitably become noticeable. OTOH, as
>>>> upsideddownunder suggests, the right thing is to set its priority
>>>> very high knowing it takes a minute amount of system time so it
>>>> always does get it when it needs it at an acceptable latency.
>>>>
>>>> Dimiter
>>>>
>>>
>>>
>>> I suspect if you think about it a wee bit, you'll see that priority
>>> juggling is a sign of bad system design.
>>
>> It certainly can be that - and it probably is that in the vast majority
>> of cases.
>> But the generalization is just wrong.
>>
>
>The thing that generalizes is that "if system A depends on a
>larger/more complex suite of task priority than system B, it is
>by-design inferior to system B, which has a dependency on a smaller st
>of priorities." This assuming both A & B more-or-less work.
>
>It is a heuristic which defines an ordinal relationship that's quite 
>useful in evaluating systems.

You should set the task priorities during _design_phase_ not during
testing phase !

There might be situations in which it would look like tasks X and Y
would require higher priority than the companion and vice versa. In
this case, you redesign tasks X and Y so that the priority order
becomes clear. Sometimes introducing a new task Z with high priority
and very short execution time will help, then the priority of X and Y
doesn't matter or they have a natural priority order.

Of course, the priority system should have sufficient number of
available levels, things get hairy with only two levels
(foregroun/background) or only a few (7 in early WinNT real time
priority class), but with 256 or potentially infinite number of
priorities I never have had problems assigning priorities.

BTW, the simplest way of having unlimited number of priority levels is
to create a task list in the same order as tasks are declared at
startup. At any system significant event, such as after an interrupt
or system request, the task list is scanned and the first task in
runnable state is selected for the next task to be executed. 

Of course, this becomes costly with a few dozen or few hundred tasks
to be scanned after each interrupt, but using bit masks, you can scan
for runnability for 8/16/32/64 tasks at a time, depending on the
processor instruction set.
 

Dimiter_Popoff wrote:
> On 08.1.2016 г. 20:57, Les Cargill wrote:
>> Dimiter_Popoff wrote:
>> .....
>>>>
>>>> I suspect if you think about it a wee bit, you'll see that priority
>>>> juggling is a sign of bad system design.
>>>
>>> It certainly can be that - and it probably is that in the vast majority
>>> of cases.
>>> But the generalization is just wrong.
>>>
>>
>> The thing that generalizes is that "if system A depends on a
>> larger/more complex suite of task priority than system B, it is
>> by-design inferior to system B, which has a dependency on a smaller st
>> of priorities." This assuming both A & B more-or-less work.
>
> Hmmm, I think this is a bit oversimplified. Again, I understand you
> completely and probably share the same feelings about it. But what if
> system A manages to do its job at say half the silicon system B
> takes.... :).
> If we add "all other things equal" to your rule I accept it
> unreservedly of course.
>

Yep - I never get everything typed right in all the time :)

>> I have the conceit that we are clockmakers. When an interrupt
>> fires or an event is gleaned from other input, this is like the
>> escapement mechanism of a pendulum clock, that allows the system to
>> deterministically move forward one notch.
>
> Well yes but things get messier when two or more interrupts come
> at the same time... IRQ latency is a key parameter in any design
> really,

I don't sympathize with that point of view much. I think
things are inherently event-driven and that everything is,
ultimately, a state machine.

Hardware guys agree with me and their stuff works.

For two interrupts, I need a timeline with events drawn on it to 
understand the worst case and best case.

> I have gone to great length to ensure it does not vary
> no matter what the OS does (this includes page fixing, in DPS this
> is done with IRQ unmasked, basically DPS is a large OS with IRQ
> latencies close if not equal to bare metal).
>

It's unusual for low latency to be of vital importance. If it is
of vital importance, then it's worth making sure the hardware is up
to it.

> But "deterministic" means a fixed delay - i.e. no latency really;
> this would be overkill to a point of being outright impractical
> in the vast majority of real life situations.
>

Ummm.... no, I think deterministic means "reflecting the delay
( and delay variation ) of the underlying physical processes of the
hardware." It means additional software delay/delay
variation is of low order of complexity - constant or linear.

>> *In this case*, ...
>
> If I ever knew what "this case" was I have long forgotten.
> I entered the discussion when it became generic enough to be
> interesting... :-)
>

:)

>>> I personally do have use for varying priorities under DPS - for
>>> example in the netMCA there are various tasks some of which use
>>> a lot of the CPU resource - e.g. the filtering task can take up
>>> well above 50% of it depending on the incoming signal.
>>
>> Filters should be completely deterministic for any given bank of filters
>> for a stream of fixed bandwidth.
>
> Filters are; overall conversion calculations are not as they depend
> on the incoming events (to be converted each) rate.
> So this can vary a lot, from say 5-6% (no events, just filtering
> for event detection) to well above 50% (maximum rate).
>

Ah; so you don't mean say, Butterworth filters then. Sure; you have to 
be able to calculate and measure worst case for event filters.

>>
>>> Then there are the VNC server tasks which check the entire
>>> "display" framebuffer memory for changes, compress and send when
>>> detected;
>>
>> If we mean the same VNC, then it's not very good software; it's just
>> available.
>
> The VNC server (like any other bit of software running on my machines)
> is written by me so I'd say it is pretty good :). And It certainly was
> not available before I wrote it.
>

I seem to have latched onto the VNC video *player* there; apologies.

> On the other side there are various VNC viewers and most of them fit
> your description; the best is RealVNC for windows, it works well for me.
> I mean really well, no complaints whatsoever.
>

Ah; good. That's gotta be a tough nut to crack.

>> But video is like that'; the formats are insane and on
>> the other side, the video cards and drivers are even crazier.
>
> RFB is a pretty sane format I'd say, as long as one picks the
> right compression which the viewer can do. In my case it is
> pretty much RLE, works fine, compression does not cost a lot
> (up to 15% system time per VNC instance @ 400 MHz power, over
> a 100 Mbps cable one just does not notice this is over VNC).
>

That sounds good, but I don't know enough to truly understand it :)

>>
>>> then there are the ethernet/ip inbound tasks which
>>> have to cope with all the traffic...
>>
>> indeed; it's networking that drives much of what I am saying. You have
>> to be able to drop things in a smooth and linear fashion.
>
> Oh and it gets only "better" digging down... :). The Ethernet controller
> on this SOC expects a smart DMA to do the buffering - which I had to
> implement as well, multiple circular receive areas etc. Then this SDMA
> does plenty of other things ("tasks"...), like disk (ATA) <-> RAM,
> RAM<->RAM (off-screen buffers to "display" framebuffer), incoming 16
> bit samples at several MSPS in a circular buffer, probably more.
> These "tasks" have fixed (8 bit IIRC) priorities; setting these
> right was crucial.

Huh. Yeah, he's a busy little guy then.

> Pretty advanced thing this SDMA, it is a shame
> Freescale stopped making it (programming it turned out too challenging
> for the typical customer today  I suppose, no other sane reason to
> do that).
>

Yeah, this happens a lot - you get a half-finished hardware feature.

>>
>>> Not many things if any are better than simplification but it is
>>> simply bound by limits like anything else. Try to simplify the netMCA
>>> I talked above to equal priorities and it will become a lot more
>>> complex overall.
>>
>> Right. I know video is always challenging.
>
> Actually the video was the least challenging IIRC (but then I am pretty
> good at it, have long since lost the count of display controllers
> I have designed).
>
> My general point is to agree with you that simplification is always
> good as long as it does not turn into oversimplification.
>

Always.

> Dimiter
>
> ------------------------------------------------------
> Dimiter Popoff, TGI             http://www.tgi-sci.com
> ------------------------------------------------------
> http://www.flickr.com/photos/didi_tgi/
>
>

-- 
Les Cargill

On January 5th, 2016, Grant Edwards sent:
"On 2016-01-05, Nicholas Collin Paul de Gloucester <wirklich@nicht.at> wrote:
> On January 5th, 2016, Grant Edwards sent:
> "On 2016-01-05, Nicholas Collin Paul de Gloucester <wirklich@nicht.at> wrote:[. . .]
>>
>> [. . .]
>>
>> STM threatened to sue you for reporting an STM documentation error
>> to STM?"
>
> Precisely.

Except in a different sub-thread, you state that the phone call was
about your possession of a confidential document that STM didn't think"


Indeed, STMicroelectonics did not think.


"you should have.  That's a bit different than threatening to sue you
for reporting a documentation error."


I reported an error which inspired STMicroelectronics to falsely claim
that it supposedly could sue us, therefore STMicroelectronics wanted
to sue us for reporting an error.


"I'm not saying that the phone call wasn't in error, but is it possible
they were calling because they thought"


STMicroelectronics did not think.


"you didn't have the proper NDA
in place to posess the document"


STMicroelectonics falsely claimed this.


"rather than because you reported a mistake?"


This intellectual property was disclosed to us therefore we noticed a
mistake which it contained therefore we reported it therefore
STMicroelectronics made an unfounded claim against us over this. Ta
da.

Regards,
Colin Paul de Gloucester

On January 5th, 2016, Rickman sent:
|-----------------------------------------------------------------------------|
|"[. . .] So one guy jumped a gun and accused you of violating an             |
|agreement and you think that means the entire STM company should be          |
|shunned?"                                                                    |
|-----------------------------------------------------------------------------|

Paul Carpenter and Rickman made unwarranted conclusions. There would
be a big difference between shunning and
"I advise against STMicroelectronics"
and
"If you
would have a product from STMicroelectronics which would work and
which you would be happy with, then be happy, but I would not blindly
trust a datasheet."
I do advise against STMicroelectronics - this NDA ex-event was not an
isolated example of lack of quality control at STMicroelectronics
which I found and it is not one of the examples which I cited in a
paper.

|----------------------------------------------------------------------------|
|"You still haven't provided all the relevant information and what you say is|
|not clear.  If the guy was not a lawyer what was his position?"             |
|----------------------------------------------------------------------------|

Would it be worth my while to answer this? Perhaps it would be cheaper
for thee to ask STMicroelectronics.

|--------------------------------------------------------|
|"Surely he                                              |
|identified himself before launching into an accusation?"|
|--------------------------------------------------------|

He was identified.

|---------------------------------------------------------------------------|
|"Did he explain what                                                       |
|he meant by you violated an NDA by "having" the document?  If your "having"|
|the document was a problem then it was someone else who violated the NDA by|
|giving it to you.                                                          |
|                                                                           |
|--                                                                         |
|                                                                           |
|Rick"                                                                      |
|---------------------------------------------------------------------------|

There was not an NDA violation. E.g. search court records - we were
not taken to court over this, they could not make coherent sense to
explain to a jurist what they were on about - they did not have a
case.

Regards,
Paul Colin Gloster

On 1/11/2016 12:56 PM, Nicholas Collin Paul de Gloucester wrote:
> On January 5th, 2016, Rickman sent:
> |-----------------------------------------------------------------------------|
>
> |"[. . .] So one guy jumped a gun and accused you of violating
> an             |
> |agreement and you think that means the entire STM company should
> be          |
> |shunned?"
> |
> |-----------------------------------------------------------------------------|
>
>
> Paul Carpenter and Rickman made unwarranted conclusions. There would
> be a big difference between shunning and
> "I advise against STMicroelectronics"
> and
> "If you
> would have a product from STMicroelectronics which would work and
> which you would be happy with, then be happy, but I would not blindly
> trust a datasheet."
> I do advise against STMicroelectronics - this NDA ex-event was not an
> isolated example of lack of quality control at STMicroelectronics
> which I found and it is not one of the examples which I cited in a
> paper.
>
> |----------------------------------------------------------------------------|
>
> |"You still haven't provided all the relevant information and what you
> say is|
> |not clear.  If the guy was not a lawyer what was his
> position?"             |
> |----------------------------------------------------------------------------|
>
>
> Would it be worth my while to answer this? Perhaps it would be cheaper
> for thee to ask STMicroelectronics.
>
> |--------------------------------------------------------|
> |"Surely he                                              |
> |identified himself before launching into an accusation?"|
> |--------------------------------------------------------|
>
> He was identified.
>
> |---------------------------------------------------------------------------|
>
> |"Did he explain
> what                                                       |
> |he meant by you violated an NDA by "having" the document?  If your
> "having"|
> |the document was a problem then it was someone else who violated the
> NDA by|
> |giving it to
> you.                                                          |
> |
> |
> |--
> |
> |
> |
> |Rick"
> |
> |---------------------------------------------------------------------------|
>
>
> There was not an NDA violation. E.g. search court records - we were
> not taken to court over this, they could not make coherent sense to
> explain to a jurist what they were on about - they did not have a
> case.

At this point you seem obsessed with debating every nuance of the issue 
without actually explaining anything that would make us understand the 
issue.  You had a problem with STM that was quickly resolved it would 
seem.  So what is your point?

-- 

Rick

On 1/11/2016 12:25 PM, Nicholas Collin Paul de Gloucester wrote:
> On January 5th, 2016, Grant Edwards sent:
> "On 2016-01-05, Nicholas Collin Paul de Gloucester <wirklich@nicht.at>
> wrote:
>> On January 5th, 2016, Grant Edwards sent:
>> "On 2016-01-05, Nicholas Collin Paul de Gloucester <wirklich@nicht.at>
>> wrote:[. . .]
>>>
>>> [. . .]
>>>
>>> STM threatened to sue you for reporting an STM documentation error
>>> to STM?"
>>
>> Precisely.
>
> Except in a different sub-thread, you state that the phone call was
> about your possession of a confidential document that STM didn't think"
>
>
> Indeed, STMicroelectonics did not think.
>
>
> "you should have.  That's a bit different than threatening to sue you
> for reporting a documentation error."
>
>
> I reported an error which inspired STMicroelectronics to falsely claim
> that it supposedly could sue us, therefore STMicroelectronics wanted
> to sue us for reporting an error.
>
>
> "I'm not saying that the phone call wasn't in error, but is it possible
> they were calling because they thought"
>
>
> STMicroelectronics did not think.
>
>
> "you didn't have the proper NDA
> in place to posess the document"
>
>
> STMicroelectonics falsely claimed this.
>
>
> "rather than because you reported a mistake?"
>
>
> This intellectual property was disclosed to us therefore we noticed a
> mistake which it contained therefore we reported it therefore
> STMicroelectronics made an unfounded claim against us over this. Ta
> da.

Glad we got to the bottom of this.  Case closed, we now return you to 
your regularly scheduled discussions.  :)

-- 

Rick

Les Cargill <lcargill99@comcast.com> writes:
> [...]
> Preemptive opens up a lot of disparate and ugly cans of worms. With
> "run to completion". you can get a lot closer to proving the system
> correct to within some epsilon ( sometimes a rather large epsilon ).

That sounds bogus. Whether 100 lines of "preemptive" or 100 lines
of "run to completion," you can have a LOT of bugs.

I would much, much, MUCH (did I say much?) rather use a preemptive OS.
You can always make it a "run to completion" by placing all tasks at the
same priority and ensuring nothing blocks in the body of each task
(except at the end, e.g., a yield()). So a preemptive OS just gives you
more flexibility. Of course you can always shoot yourself in the foot
with more flexibility. So be it - I'd rather have the choice.

I agree that this is not really related to managing multiple
programmers, however.
-- 
Randy Yates, DSP/Embedded Firmware Developer
Digital Signal Labs
http://www.digitalsignallabs.com

Randy Yates wrote:
> Les Cargill <lcargill99@comcast.com> writes:
>> [...]
>> Preemptive opens up a lot of disparate and ugly cans of worms. With
>> "run to completion". you can get a lot closer to proving the system
>> correct to within some epsilon ( sometimes a rather large epsilon ).
>
> That sounds bogus. Whether 100 lines of "preemptive" or 100 lines
> of "run to completion," you can have a LOT of bugs.
>

Of course you can.  But with preemption, you get Heisenbugs of a
certain sort, thread-safe issues, reentrancy and race conditions
of another certain sort.

Of course, you gain habits in how to avoid these, but still...

I just really prefer deterministic operation of
systems. And, frankly, I don't understand other people not
preferring that. I fully realize you can't always do that, but
I bet you can get closer than you think you can.

> I would much, much, MUCH (did I say much?) rather use a preemptive OS.
> You can always make it a "run to completion" by placing all tasks at the
> same priority and ensuring nothing blocks in the body of each task
> (except at the end, e.g., a yield()).


I do, generally, or I make it message passing, or
interlocking state machines with message passing...

Here's the key - it's now possible to reason about these systems in a
proof-like manner. You can enumerate all the states cross events and
check lists of invariants at each transition.

So what if it's big? Sure, you'll make mistakes but if
you keep this as a regression check, I bet it's worth it.
You can get a long way with < 100 states and < 12 events, well,
that's 1200 things. Not impossible at all.

I find that when I do things this way, people don't have to
reset the machine quite so much. If at all.

I don't wanna put words in your mouth nor make other
presumptions, but it almost sounds like you're arguing
against rigor.

> So a preemptive OS just gives you
> more flexibility. Of course you can always shoot yourself in the foot
> with more flexibility. So be it - I'd rather have the choice.
>

Sure. I get that. But so long as you block once per iteration of a loop,
it works out the same anyway.

> I agree that this is not really related to managing multiple
> programmers, however.
>

-- 
Les Cargill