Delayed "printf"

Tim Wescott wrote:
> Sometimes it's necessary to float a notion just so that someone can
> affirm your suspicion that yes, you really are an idiot.
>
> On the off chance that I'm a genius, though, here goes:
>
> I don't like putting "printf" in fast tasks for obvious reasons --
> even with infinite spooling, it can take time to process a format
> string.  In my usual context, this can result in audible sounds from
> motors when I print some bit of debugging information from a
> high-priority task.  The problem can't be alleviated by spooling the
> output of the printf, because it's the execution time of the printf
> itself that's the problem.
>
> Just how dumb is it to have "printf" queue up the format string AND
> the parameters into a buffer, to be printed out in some low-priority
> task? For a system with sufficient memory, this seems like an
> astonishingly sensible thing to me.
>
> Clearly you'd have to wrap it up somehow so that if the buffer was
> full no printf at all would happen -- but I'm kinda liking the
> notion.
>
> So -- whatcha think?
>

There is nothing dumb about it at all. It can be a bit tricky, though.
But it's really just a variation on logging in general, so there
may be a logger laying around somewhere that fits your specs so you 
don't have to spend a lot of time building one.

-- 
Les Cargill

Tim Wescott wrote:
> On Mon, 24 Apr 2017 19:42:30 -0500, Tim Wescott wrote:
>
>> Sometimes it's necessary to float a notion just so that someone
>> can affirm your suspicion that yes, you really are an idiot.
>>
>> On the off chance that I'm a genius, though, here goes:
>>
>> I don't like putting "printf" in fast tasks for obvious reasons --
>> even with infinite spooling, it can take time to process a format
>> string.  In my usual context, this can result in audible sounds
>> from motors when I print some bit of debugging information from a
>> high-priority task.  The problem can't be alleviated by spooling
>> the output of the printf, because it's the execution time of the
>> printf itself that's the problem.
>>
>> Just how dumb is it to have "printf" queue up the format string AND
>> the parameters into a buffer, to be printed out in some
>> low-priority task? For a system with sufficient memory, this seems
>> like an astonishingly sensible thing to me.
>>
>> Clearly you'd have to wrap it up somehow so that if the buffer was
>> full no printf at all would happen -- but I'm kinda liking the
>> notion.
>>
>> So -- whatcha think?
>
> The responses so far are really making me think.
>
> I guess the better way to frame my desire is thus:
>
> I want some way to print free-form debug data to a serial port in
> such a way that anything slow (formatting and waiting on the UART) is
> performed in a low-priority task.
>
> Some printf-like thing may not be the best way to do it -- but I
> still want to do it!!
>

So look over a few varargs examples. If the first argument is a count of 
other arguments, and it's "format string" driven, then you should be 
able to just copy the args to .... something and pick 'em up later.

Just be careful of the lifetime of objects to be printed.

-- 
Les Cargill

On 26/04/17 03:51, Theo Markettos wrote:
> Clifford Heath <no.spam@please.net> wrote:
>> We did this with an error message system in an enterprise system
>> written mostly in C++. I had produced a type-safe ArgList class,
>> which allowed you to << any supported data type to append to the
>> list. Supporting printf-style formats with positional argument
>> numbers allowed us to fetch the format text translated into
>> foreign languages. Having an allocated copy of the typed argument
>> list made printf type-safe, but also meant that we could ship the
>> format and the arguments across a network interface, to be
>> rendered using an international format string in the language of
>> the client's locale.
>>
>> It was very nice, if rather a lot of work (mostly by me)
>
> My idea, for a system which is supposed to run code written by arbitrary
> users who expect printf, but is limited in message size that you don't want
> to send the text, is to abuse the tools for handling localisation.
>
> In other words the user writes:
> printf("Hello world foo=%d, bar=%s, baz=%f blah blah blah\n", foo, bar, baz);
>
> and the locale tools transform that into a text file containing strings like:
> STR42: "Hello world foo=%d, bar=%s, baz=%f blah blah blah\n"
>
> and code like:
> _locale_print("STR42", foo, bar, baz);

Yeah, naah, that doesn't work when a translation needs to display
the arguments in a different order from the parameter order.

For this purpose you need to be able to number the parameters in
the format string, e.g.:

message("File %2$.*1$s was not available for operation %3$s", 
max_length, filename, operation_name);

> a function which packages STR42 and binary representations of STR42 and the
> parameters into a packet and sends it on its way.
> (You might need to annotate the _locale_print with the types of the
> parameters somehow)

That's almost exactly what we did, but using C++ and my ArgList
construct meant we had guaranteed type safety at runtime, even
when using a format string fetched from a catalog. A bad catalog
cannot crash the formatting program.

I used an XML format for the message catalog source file, and
generated from that #include files which defined integer message
identifiers (as well as optionally including the default English
language format strings).

The integers were designed to mesh on Windows with HRESULT values,
so they were made up of a 12-bit "facility number" and a 16 bit
message number within that facility. Each subsystem had a manually
allocated facility number, and maintained its own messages (and
allocated its own message numbers). The message numbers were often
displayed using an sNNNmMMM format - so-called S/M numbers. This
left four bits at the top for normal Windows HRESULT code severity
markers.

Each message had an extended explanatory note which was used in
producing the "Error message manual" - which ran to almost 500 pages
in a PDF files, cataloged by SM number. The support department
loved me for it :). But when you're talking about a system that
even in 2007 was upwards of 3 million lines of code (excluding
whitespace and comments) you really need this level of discipline.

The message catalog compiler performed various gyrations, including
splitting or combining multiple XML files by language or facility,
as well as producing a standard binary indexed compiled file for
runtime. The compiler would check that each message had a translation
into each target language, and could emit a file of those messages
for which no translation was yet available, to be sent to the
translators.

The other thing we did was to encourage message authors to break each
message into three fragments: Problem, Reason and Solution. It's
alarming how much software does not tell you the things you need to
know. ENOENT -> "No such file or directory". Well, great. Was it a
file or a directory that was missing? Perhaps you could even tell
me which one?

We used to jeer at Microsoft for this kind of behaviour, with their
"Unknown error 0x80000000", but they copied "errno" from Unix, and
it has always cause the same kind of problems there too.

> At the other end, the PC uses the text file so that when it receives STR42
> in a packet it knows how to interpret the packet and prints it out (or logs
> it to a file for printing later in some GUI tool).
>
> We haven't actually implemented it yet.

Hopefully I've given you a few more things to include in your
design considerations.

Clifford Heath.

On 26.4.17 00:11, Tim Wescott wrote:
> 
> The responses so far are really making me think.
> 
> I guess the better way to frame my desire is thus:
> 
> I want some way to print free-form debug data to a serial port in such a
> way that anything slow (formatting and waiting on the UART) is performed
> in a low-priority task.
> 
> Some printf-like thing may not be the best way to do it -- but I still
> want to do it!!


Your main problem may be to know when the printf() is done, so
that the source data is free to be used again. You may need to
copy the printf argument values to a buffer, or maybe do a sprintf()
and queue the string to the output device. In any case, delaying
the output means needing to have buffer space.

The problem is similar to one in many networking hacks, where
the protocol stack forces delaying sending the data.

-- 

-TV

Clifford Heath <no.spam@please.net> wrote:
> Hopefully I've given you a few more things to include in your
> design considerations.

Thanks, that's interesting.  In this instance the use case is a bit
different - running unmodified third-party C code.  You really don't want to
go around manually restructuring every printf if you can avoid it, hence the
need for it to use source transformations.  Locales are just a pre-existing
mechanism for that; the messages never need to get translated into another
language.

Theo

> Just how dumb is it to have "printf" queue up the format string AND the 
> parameters into a buffer, to be printed out in some low-priority task?  
> For a system with sufficient memory, this seems like an astonishingly 
> sensible thing to me.

It's so dumb that I do it often. Usually as a ring buffer of {format string
ptr, arg...} with a fixed limit on the number of args. Sometime I actually
do the print offline, from coredumps or shared memory.

On 2017-04-26, mac <acolvin@efunct.com> wrote:
>> Just how dumb is it to have "printf" queue up the format string AND the 
>> parameters into a buffer, to be printed out in some low-priority task?  
>> For a system with sufficient memory, this seems like an astonishingly 
>> sensible thing to me.
>
> It's so dumb that I do it often. Usually as a ring buffer of {format string
> ptr, arg...} with a fixed limit on the number of args. Sometime I actually
> do the print offline, from coredumps or shared memory.

I do something similar.  In the latest incarnation, there's a
dedicated sourceid/instance index, two 32-bit "parameters", and the
printf "format" string is limited to somehting like 16 bytes.

There's a C macro that puts that stuff into a circular buffer in
shared memory.  Then there is a seperate utility to print the buffer
contents.

-- 
Grant Edwards               grant.b.edwards        Yow! Wow!  Look!!  A stray
                                  at               meatball!!  Let's interview
                              gmail.com            it!

On 2017-04-26, Grant Edwards <invalid@invalid.invalid> wrote:
> On 2017-04-26, mac <acolvin@efunct.com> wrote:
>>> Just how dumb is it to have "printf" queue up the format string AND the 
>>> parameters into a buffer, to be printed out in some low-priority task?  
>>> For a system with sufficient memory, this seems like an astonishingly 
>>> sensible thing to me.
>>
>> It's so dumb that I do it often. Usually as a ring buffer of {format string
>> ptr, arg...} with a fixed limit on the number of args. Sometime I actually
>> do the print offline, from coredumps or shared memory.
>
> I do something similar.  In the latest incarnation, there's a
> dedicated sourceid/instance index, two 32-bit "parameters", and the
> printf "format" string is limited to somehting like 16 bytes.
>
> There's a C macro that puts that stuff into a circular buffer in
> shared memory.  Then there is a seperate utility to print the buffer
> contents.

If you want to save even more time/space, you can require that the
format string have a static lifetime, and just copy a pointer into the
circular buffer.

-- 
Grant Edwards               grant.b.edwards        Yow! You mean you don't
                                  at               want to watch WRESTLING
                              gmail.com            from ATLANTA?