EmbeddedRelated.com
Forums
Memfault Beyond the Launch

From cooperative to preemptive scheduler: a real example

Started by pozz January 6, 2020
[I'll tackle this in multiple posts to avoid conflating the issues.
But, I *really* want to extricate myself from this discussion...]

[ 8< ]

>>>>> My impression is that a very simple code is cluttered with synchronization >>>>> things that decrease readability and maintainability and increase >>>>> complexity. Why? Just to use preemption? >>>> >>>> The "clutter" is introduced because your "problem" inherently involves >>>> conflict; you're allowing two competing uses for a single resource. >>> >>> Howevere the shared resource complexity is present only when preemption is >>> used. >> >> Because it doesn't work right in the nonpreempt case! :> > > Why do you say this? This application can work flowlessy even with cooperative > multitasking.
I think the following pieces together the various bits of your example, in its "finished" form (apologies if my cut-n-paste got sloppy): First, your cooperative implementation: --- main.c --- ... while(1) { task_display(); task_serial(); } --- end of main.c --- --- display.c --- static const char msg[32]; void display_set_message(const char *new_msg) { strncpy(msg, new_msg, sizeof(msg)); } void task_display(void) { if (refresh_is_needed()) { display_printat(0, 0, msg); } } --- end of display.c --- --- serial.c --- static unsigned char rxbuf[64]; static size_t rxlen; void task_serial(void) { unsigned char b = serial_rx(); if (b != EOF) { rxbuf[rxlen++] = b; if (frame_is_complete(rxbuf, rxlen)) { char new_msg[32]; /* decode new message from received frame from serial line */ display_set_message(new_msg); rxlen = 0; } } } --- end of serial.c --- Note that you leave a lot undefined -- including "requirements"! (so, I guess that makes it easier to meet a target! :> ) So, I'll try to walk a line between being overly pedantic and "too forgiving"... The most glaring problem is that you're using "run to completion" in each task implementation -- but it's not apparent how long each of those "efforts" will take, in actual elapsed time! E.g., how long -- worst case -- between successive invocations of unsigned char b = serial_rx(); This must be less than a single character time (at the current line rate) -- unless the UART is multiply buffered AND serial_rx() is capable of completely emptying that buffer, when invoked. And, this has to be satisfied regardless of which "execution paths" each task chooses to take (i.e., conditionals). Some (fixed?) time after the serial data has arrived at the device, it becomes "available". Some VARIABLE time after that, an invocation of serial_rx() will detect it. Worst case, ONE invocation of serial_rx() will have *just* finished checking for it and, thus, MISS seeing it become available some epsilon later. [I am assuming that EOF is returned when serial_rx() determines "no data available"] So, the trailing end of serial_rx() executes (any code after the determination of the presence of data is made -- including the "return" mechanism). Then, the test for EOF (in task_serial). And, *its* return -- to main(). The crank is turned on the while() loop and task_display() invoked. No idea how "refresh is needed" is determined. Nor how long it takes to make that determination. As it undoubtedly *is* needed, eventually, display_printat() can be invoked. Which, does "something" and takes "some amount of time". Eventually, it returns (to task_display) and task_display(), itself, returns to main. Which allows task_serial() to be reinvoked. Which, in turn, invokes serial_rx() and SOME TIME LATER, data availability is again assessed. Can another character have arrived and been MISSED in this time? <shrug> Maybe. Maybe not. You have no way of knowing -- without looking through all of this code and making an assessment as to how long each statement in the above sequence will take to execute. Note that if serial_rx() returns a real character (i.e., not EOF), then even more code is inserted into this period between successive "UART checks". And, if the worst-case time is longer than tolerable (at the fastest data rate), there's no easy way for you to tweek your code to compensate. Imagine if display_printat() examines the characters in the string presented and maps each to a bitmap representation of the character -- a "font", so to speak. Then, takes the bitmap for that character and paints it into a *graphic* display, overlaying the "dots" that were already there. Then, advancing to the next character in the string and performing the same action after having advanced the "display cursor" to take into account the dots that it has painted for the previous character. This could take a fair bit of time -- especially as the display hardware might be via some I2C bus, etc. so each dot painted has a high temporal overhead! Now, imagine a Marketeer comes along and wants to display the current time alongside each message. Or, worse -- display and UPDATE that time WHILE the message remains in place! Or, have the message stay there for at most 2 seconds and then "autoblank" -- unless another message has been received in the interim. This just adds to the code (time!) that has to be executed between serial_rx() invocations. Or, the sales guy claims that he "can sell a million of them... *IF* you could just update the maximum 'baudrate' to 115200!" (Ooops! Suddenly you only have 100us to do all that work without losing data!)
And, finally, a preemptive implementation:

[Sorry, not checked for errors.  I've been more focused on consolidating
files from various disk drives and wiping the drives -- the sort of
activity that you REALLY don't want to screw up by getting sloppy!  A
total of 24TB today, alone -- and that's just the *first* SAN shelf!]

I tried to keep this structured similar to your code so you could see
the parallels...

ASSERT(MAX_CHARACTERS >= 1)
char bufferA[MAX_CHARACTERS], bufferB[MAX_CHARACTERS];

accumulator = &bufferA[0];
readout = &bufferB[0];

[N.B. I like the redundapetitive &buffer[0] notation as it
makes it clear to me that I'm referencing the start of a
buffer -- instead of the current value of a POINTER!]

void
ISR() {
     UART_character = read_UART_receive_register();
     raise_event(CHARACTER_AVAILABLE);
     // do anything else required to ack the IRQ and ensure
     // it is reactivated for the next character
}

Note that most MTOS/RTOSs place limits on the sorts of calls
you can make from within an ISR.  Event handling is often
allowed because it is so "slim".

char
get_character() {
     wait_event(CHARACTER_AVAILABLE);
     return UART_character;
}

task_t
gather_message() {
     while (FOREVER) {
         c = get_character();    // block until character available
         if (buffer >= &accumulator[MAX_CHARACTERS])
             // accumulator overrun!
             panic();             // FIXME  no one said how SHOULD be handled
                                  // so I deliberately make it an undesirable
                                  // behavior so folks THINK about what they'd
                                  // REALLY want it to do!
             return;              // NOTREACHED?

         // space exists in accumulator
         if (c != TERMINATOR) {
             // append new character to accumulator
             *buffer++ = c;
         } else {
             // message complete!
             holdme = readout;    // no one dicks with "readout" other than me!
                                  // so, safe to capture its value outside lock

             take_lock(MESSAGE_LOCK);      // block until lock available

             // swap buffers with the display task
             readout = accumulator;
             accumulator = holdme;

             release_lock(MESSAGE_LOCK);

             raise_event(NEW_MESSAGE_AVALABLE);   // signal data available

             buffer = &accumulator[0];  // reset pointer to start of accumulator
                                        // WHEREVER that happens to now reside
         }
     }
}

task_t
display() {
     char indication[4+1];            // see below

     while(FOREVER) {
         wait_event(NEW_MESSAGE_AVAILABLE);  // block until something to display

         take_lock(MESSAGE_LOCK);    // block until lock available

                                     // note: you would *THINK* that it would be
                                     // available AS SOON AS the new message was
                                     // signaled as being available.  But, if
                                     // other tasks in the system, they could
                                     // sneak in and use the processor between
                                     // wait_event() and take_lock()

                                     // However, even if DAYS pass in that
                                     // interim (which could mean hundreds of
                                     // messages have arrived in addition to the
                                     // first one), we KNOW that there *is*
                                     // work to be done (a new message to
                                     // display!) AND that we can access it
                                     // via the "readout" buffer -- regardless
                                     // of how many times that pointer may have
                                     // been changed!

                                     // Holding the lock lets us use the pointer
                                     // and "readout" buffer referenced by it;
                                     // these won't change WHILE we hold the
                                     // lock cuz we've adopted the CONVENTION
                                     // that you must hold the lock to dick with
                                     // these things

         display_printat(0, 0, readout);   // paint buffer into display

         release_lock(MESSAGE_LOCK);

         beep();                     // signal new message to be read!
         sleep(2sec);                // view message before overwriting!
     }
}

beep() {
     buzzer(ON);
     sleep(500ms);
     buzzer(OFF);
}


Note that display() holds the lock for a potentially long period of time
(defined by however long display_printat() takes to do its thing).

But, the gather_message() task will only try to acquire that lock
when it has received a COMPLETE message.  It has a place in which to
store characters received while that is happening -- the "accumulator"
buffer!

If you want to eliminate the possibility of display_printat() taking
too long (e.g., in the case of the "empty message"), then you need to
rejigger the buffering strategy.  You could use a single buffer/FIFO
and pass pointers to portions of it that are "being filled" vs. "being
displayed", etc.

If you are afraid of losing characters because display() monopolizes
the processor during display_printat(), you would set the priority of
gather_message() to be higher than display().  In this way, as soon as
the ISR raises the CHARACTER_AVAILABLE event, get_character() will
be made ready to run and preempt whatever is running currently.

[Go play!  You can't BREAK anything -- they're just *bits*!!]

And, with that, I'll consider my role this conversation "completed"...
Il 08/01/2020 23:54, Niklas Holsti ha scritto:
> On 2020-01-08 1:02, pozz wrote: >> Il 07/01/2020 08:38, Niklas Holsti ha scritto: >>> On 2020-01-07 3:08, pozz wrote: >>>> I noticed my previous post about preemptive OS involved many people >>>> and started many discussions, most of them theoric. >>>> >>>> Someone wrote the synchronization of tasks in preemptive scheduler >>>> is not so difficult, after understanding some things. >>> >>> I made some such statement. >>> >>>> Others suggested to abandon at all preemptive scheduler, considering >>>> its pitfalls. >>>> >>>> Because I know my limits, I don't think I can produce a well-written >>>> preemption system. However I'd like to understand a little more >>>> about them. Starting from an example. >>>> >>>> Suppose my system is a display where a message is written. The >>>> message can be customized by a serial line. >>> >>> So, this system consists of a display and a serial input line and has >>> requirements as follows: >>> >>> 1. The display shall at all times show a message, of at most 31 >>> characters. >>> >>> - To be defined: what the initial message should be at system reset. >>> >>> 2. The SW shall receive characters from the serial line, buffering >>> them in a "frame buffer" in memory, which can hold up to 64 characters. >>> >>> 3. After each received (and buffered) serial-line character, the SW >>> shall check if the buffered characters form a complete "frame". >>> >>> - To be defined: what to do if the frame buffer is full but does not >>> form a complete frame. (This may of course be impossible by design of >>> the "frame_is_complete" function.) >>> >>> 4. When the buffered characters form a complete frame, the SW shall >>> convert (decode) the contents of the frame into a message, of at most >>> 31 characters, display that message until another, new frame is >>> received, and erase the frame-buffer in preparation for the next frame. >>> >>> The real-time aspects are undefined, except that each message is >>> displayed until the next frame is received. >> >> The only real-time is that the new message sent through the serial >> line appears on the display in a reasonable time: 100ms? 1s? Something >> similar. >> >> The second requirement is that the display mustn't show a hybrid >> message composed by two parts of the successive messages. >> >> >>>> In cooperative approach, I would write something: > > &#4294967295;&#4294967295; [snip code] > >>>> How to convert these two tasks in a preemptive scheduler? Which >>>> priority to assign to them? >>> >>> Before that conversion one must think about the real-time >>> requirements: deadlines, response-times. This is difficult for this >>> example, because you have not stated any requirements. >>> >>> Let's assume these requirements and properties of the environment: >>> >>> A. The function "serial_rx" polls the one-character reception buffer >>> of the serial line once, and returns the received character, if any, >>> and EOF otherwise. It must be called at least as often as characters >>> arrive (that is, depending on baud rate) to avoid overrun and loss of >>> some characters. > > You asked about possible advantages of pre-emption; I made my > assumptions, above, such that the (incomplete) example you gave shows > this advantage, under these assumptions (which could be true for other, > otherwise similar example applications). > >> No, serial driver works in interrupt mode and already use a FIFO >> buffer, sufficiently big. serial_rx() pop a single element from the >> FIFO, if any. > > Ah, then your *system* is intrinsically pre-emptive (the interrupts > pre-empt the tasks), even if the *code you showed* does not show this > pre-emption.
Ah yes, interrupts are preemptive and I use them a lot, but they are confined in their works, they are very lightweight and fast. The discussion here regards medium to high complexity tasks.
> > I won't reply to your other comments on my assumptions, as they are > irrelevant to the point of where and when pre-emption can be good for you. > >> Serial driver interrupts guarantees no loss of input during >> display_printat() or other functions. > > Right, because it is pre-emptive. So there you see the advantage. > >>> For "task_display", you could replace the "refresh is needed" flag >>> with another semaphore, which is initially zero, is "given" in >>> "task_serial" when a new message is to be displayed, and is "taken" >>> by "task_display" before it displays the new message. Then >>> "task_display" consumes no processing resources until it actually has >>> to. >> >> I was thinking to a refresh made at regular intervals, such as every >> 100ms. > > In some systems that could result in annoying flickering of the display, > which could even be dangerous (seizure-inducing) to some users. >
On 2020-01-09 13:19, pozz wrote:
> Il 08/01/2020 23:54, Niklas Holsti ha scritto: >> On 2020-01-08 1:02, pozz wrote: >>> Il 07/01/2020 08:38, Niklas Holsti ha scritto:
[snip]
>>>> Let's assume these requirements and properties of the environment: >>>> >>>> A. The function "serial_rx" polls the one-character reception buffer >>>> of the serial line once, and returns the received character, if any, >>>> and EOF otherwise. It must be called at least as often as characters >>>> arrive (that is, depending on baud rate) to avoid overrun and loss >>>> of some characters. >> >> You asked about possible advantages of pre-emption; I made my >> assumptions, above, such that the (incomplete) example you gave shows >> this advantage, under these assumptions (which could be true for >> other, otherwise similar example applications). >> >>> No, serial driver works in interrupt mode and already use a FIFO >>> buffer, sufficiently big. serial_rx() pop a single element from the >>> FIFO, if any. >> >> Ah, then your *system* is intrinsically pre-emptive (the interrupts >> pre-empt the tasks), even if the *code you showed* does not show this >> pre-emption. > > Ah yes, interrupts are preemptive and I use them a lot, but they are > confined in their works, they are very lightweight and fast.
That's a design decision. Some systems do most of their work in interrupt handlers, and use "background" processing only for some non-critical housekeeping tasks.
> The discussion here regards medium to high complexity tasks.
I don't recall you saying so, and your example (perhaps naturally, for an example) did not have such tasks. Many interrupt handlers are more complex than the tasks in your example. In a pre-emptive system there is no logical difference between interrupts and tasks. The only practical difference is that interrupts are "tasks" that are scheduled by the processor HW (moderated by SW control of interrupt masking and disabling) while the tasks are scheduled by the RTOS. The advantages (and complications) of pre-emptive scheduling are mostly the same for tasks as for interrupts. -- Niklas Holsti Tidorum Ltd niklas holsti tidorum fi . @ .
Il 09/01/2020 13:03, Niklas Holsti ha scritto:
> On 2020-01-09 13:19, pozz wrote: >> Il 08/01/2020 23:54, Niklas Holsti ha scritto: >>> On 2020-01-08 1:02, pozz wrote: >>>> Il 07/01/2020 08:38, Niklas Holsti ha scritto: > > &#4294967295;&#4294967295; [snip] > >>>>> Let's assume these requirements and properties of the environment: >>>>> >>>>> A. The function "serial_rx" polls the one-character reception >>>>> buffer of the serial line once, and returns the received character, >>>>> if any, and EOF otherwise. It must be called at least as often as >>>>> characters arrive (that is, depending on baud rate) to avoid >>>>> overrun and loss of some characters. >>> >>> You asked about possible advantages of pre-emption; I made my >>> assumptions, above, such that the (incomplete) example you gave shows >>> this advantage, under these assumptions (which could be true for >>> other, otherwise similar example applications). >>> >>>> No, serial driver works in interrupt mode and already use a FIFO >>>> buffer, sufficiently big. serial_rx() pop a single element from the >>>> FIFO, if any. >>> >>> Ah, then your *system* is intrinsically pre-emptive (the interrupts >>> pre-empt the tasks), even if the *code you showed* does not show this >>> pre-emption. >> >> Ah yes, interrupts are preemptive and I use them a lot, but they are >> confined in their works, they are very lightweight and fast. > > That's a design decision. Some systems do most of their work in > interrupt handlers, and use "background" processing only for some > non-critical housekeeping tasks. >
Yes, I imagine there are a pletora of possibilites. Anyway I thought there was two typical approaches: superloop that continuously calls non-blocking functions (an example of cooperative scheduler) and a full preemptive scheduler (most of the time a full RTOS). Interrupts are always present, even in superloop.
>> The discussion here regards medium to high complexity tasks. > > I don't recall you saying so, and your example (perhaps naturally, for > an example) did not have such tasks. Many interrupt handlers are more > complex than the tasks in your example.
Of course the example was simple. Its goal was to discuss the complexities (clutters) that a preemption scheduler add to the code of tasks (not to the code of interrupts that are almost the same).
> In a pre-emptive system there is no logical difference between > interrupts and tasks. The only practical difference is that interrupts > are "tasks" that are scheduled by the processor HW (moderated by SW > control of interrupt masking and disabling) while the tasks are > scheduled by the RTOS. > > The advantages (and complications) of pre-emptive scheduling are mostly > the same for tasks as for interrupts.
Maybe I am wrong, but I implement ISRs with great care and attention, but they are very small and limited. Most of the time they are already implemented in drivers released by the silicon vendor. Anyway they are so limited that the preemption issues (synchronization) are very well defined and confined. Normally the biggest part of the firmware is related to the application/tasks, not interrupts.
Il 09/01/2020 00:39, Niklas Holsti ha scritto:
> On 2020-01-08 0:51, pozz wrote: >> Il 07/01/2020 15:51, Don Y ha scritto: >>> On 1/7/2020 2:11 AM, pozz wrote: >>>> Il 07/01/2020 03:37, Don Y ha scritto: >>>>> On 1/6/2020 6:08 PM, pozz wrote:
> What is your concern with that? You only need one semaphore to provide > mutual exclusion between two tasks, not a separate semaphore for each > shared variable. > > Are you worried about the processor time for the semaphore operations? > or the code clutter?
Code clutter. I am worried to forget to add semaphore management to a newly added function without noticing the problem (because those bugs are very rare to see).
> If you have many variables, shared in that way, you probably have some > way of identifying a particular variable by a run-time value, such as an > enumeration or a string name, and then you can write a single function > that accesses any one variable when given the identifier of that > variable as a parameter, and you can encapsulate the take/give of the > semaphore within that function.
void set_var1() semaphore_take() ... semaphore_give() void set_var2() semaphore_take() ... semaphore_give() Coding in cooperative scheduler, at first I don't like that code. But the big problem is the fear to forget to manage the semaphore when accessing some shared variable.
> In such cases, you should also consider carefully /when/ a task should > accept a change in a variable. It is often the case that failures or bad > behaviour can result if a task uses a variable, X say, in two places, > but the value of X changes unexpectedly between the first use and the > second use, because there is a "yield" or pre-emption between the uses. > Then it is better for the task to take a local copy of X, at a suitable > point in its execution, and use that local copy subsequently, until it > is time to refresh the local copy. Using the local copy of course does > not need mutex protection. > >> So the worst-case superloop duration is the sum of worst-case > >> durations of each task, plus worst-case duration of interrupts. > > If tasks are coded as non-blocking (state-machine), this worst-case > > duration could be very small and real-time requirements can be > > respected. > > You might try coding an FFT or Quicksort or other complex algorithm as a > state machine, with a variable overall length of the input and output > arrays, and then compare the "clutter" of those state machines with the > clutter of pre-emptive coding.
Those are right examples, but I think very rare in typical control systems.
>> Again, in my approach every task are *non-blocking*, > > (Just a note that this use of the term "blocking" does not conform with > its normal use in task scheduling, where a task "blocks" when it > suspends itself to wait for some event that has not yet happened, or > when it cannot execute because a higher-priority task is executing. Such > "blocked" tasks are not running and are not using processor time. A task > that just runs and computes for say 60 seconds is not "blocking" in the > normal sense of the word.) > >> so they take 100us-1ms maximum at each loop. If I have 10 tasks, >> superloop duration could be estimated in 10ms maximum. If the most >> critical real-time requirement is 100ms or more, cooperative >> multitasking is ok. > > Yes, everything depends on the execution times and the required response > times. If cooperative works, without excessively cluttered state > machines, and you are not worried about significant long-term evolution > of the SW, it may be a defensible approach. >
Il 09/01/2020 07:12, Don Y ha scritto:
 >...
> Or, the sales guy claims that he "can sell a million of them... *IF* you > could just update the maximum 'baudrate' to 115200!"&#4294967295; (Ooops!&#4294967295; Suddenly you > only have 100us to do all that work without losing data!)
I admit it wasn't clear, but I was thinking to an interrupt-based serial driver that buffers received chars as soon as they are received. serial_rx() pops chars from the buffer.
Another trouble I found with a preemptive RTOS is how to size the stack 
of tasks.

In my superloop cooperative scheduler:

   while(1) {
     task1();  // Non-blocking fast state-machined task
     task2();  // Non-blocking fast state-machined task
   }

the stack is assigned to all the unused memory available in the system 
(that is all the memory minus variables and heap size, if you use it).

If two tasks use a stack-intensive function (such as a printf-like 
function), this doesn't increase the overall stack requirement.
For example, if the stack-intensive function needs 2kB of stack, the 
global stack can be 2kB (more other stack needed by tasks for other 
operations).

With a preemptive scheduler, tasks can be interrupted in any point, even 
during printf-like function. So *each* task needs a stack of 2kB, 
reaching a global stack requirement of 4kB.

Another issue with preemptive approach is that you should be smart 
enough to size N stacks (where N is the number of tasks).
With the superloop architecture above, you should size only *one* global 
stack, that can be calculated over one task, the one that needs more stack.

Does this make sense?
Am 14.01.2020 um 13:27 schrieb pozz:
> Another trouble I found with a preemptive RTOS is how to size the stack > of tasks.
That trouble is not at all particular to preemptive scheduling.
> In my superloop cooperative scheduler:
> the stack is assigned to all the unused memory available in the system > (that is all the memory minus variables and heap size, if you use it).
And how do you know that that's sufficient?
> Another issue with preemptive approach is that you should be smart > enough to size N stacks (where N is the number of tasks). > With the superloop architecture above, you should size only *one* global > stack, that can be calculated over one task, the one that needs more stack.
If you can't calculate the stack for those small-ish tasks, then you can't do it for the functions called by your super-loop, either. If you can judge which of the super-loop's sub-tasks consumes the most stack, and how much that is, then you can do it for preemptive scheduling, too. The difficulty of (reliably) computing stack usage is the same, regardless what tasking concept is used.
Il 14/01/2020 20:10, Hans-Bernhard Br&#4294967295;ker ha scritto:
> Am 14.01.2020 um 13:27 schrieb pozz: >> Another trouble I found with a preemptive RTOS is how to size the stack >> of tasks. > > That trouble is not at all particular to preemptive scheduling. > >> In my superloop cooperative scheduler: > >> the stack is assigned to all the unused memory available in the system >> (that is all the memory minus variables and heap size, if you use it). > > And how do you know that that's sufficient?
I'm not too smart to estimate the stack usage of a function (I know the compiler can produce some useful info about this, but you should add interrupt stack usage and so on), so my approach is only tests. I fill the memory with a known value, run the application for many hours/days and eventually check the memory content to see the greatest (or lowest) address reached by the stack. This isn't a simple job, but I do it one time, because there's a single global stack in superloop architecture.
>> Another issue with preemptive approach is that you should be smart >> enough to size N stacks (where N is the number of tasks). >> With the superloop architecture above, you should size only *one* global >> stack, that can be calculated over one task, the one that needs more stack. > > If you can't calculate the stack for those small-ish tasks, then you > can't do it for the functions called by your super-loop, either. > > If you can judge which of the super-loop's sub-tasks consumes the most > stack, and how much that is, then you can do it for preemptive > scheduling, too. > > The difficulty of (reliably) computing stack usage is the same, > regardless what tasking concept is used.
I don't agree. Most of the time I can guess what is the task (or a few tasks) that consumes more stack. So the effort to estimate stack usage is limited. In preemptive scheduler, you would need to multiplicate your effort to estimate stack usage for every single task to avoid wasting precious memory. Anyway, suppose we are very smart to calculate stack usage of each task: - task1 needs 10kB - task2 needs 10kB With preemptive scheduler you need to reserve 20kB for the stack, in superloop you can reserve only 10kB.

Memfault Beyond the Launch