Ico wrote:
> Some more ramblings on device configuration...
>
> Don <none@given> wrote:
>> Ico wrote:
>>
>>> So, my actual question is : am I missing somehing here ? Has this
>>> problem been long solved and I just don't know about this, or am I not
>>> the only one facing this problem, time after time again. Are there
>>> existing systems that can help me handle manage the boring and hard
>>> parts of embedded systems: the configuration and user interface.
>> Ad hoc solutions usually exist because systems are rarely *designed*,
>> from scratch, with anticipation of future needs, dependencies, etc.
>> in mind.
>
> Very true, this is one of the reasons for my concerns. But this problem
> is also hard to address, since things like this just tend to grow and
> evolve - a company starts with product X version 1.0, with only one
> feature, the next version does only a little bit more, needs a tiny bit
> more configuration, et cetera. Thus another monster is born.
Shoot the Marketing Staff! :>
<on a more serious note...>
Change is inevitable. How you plan for change is what separates
the men from the boys :>
> What I am actually looking for is a general philosophy, a setup or
> framework which is flexible enough to grow with the requirements, but
> which will not grow exponentially more complex when functionality is
> added to the system.
>
> An analogy would be something like using finite state machines in
> (embedded) software: Without FSM's, it is very well possible to create
> complex systems, but each time a function is added, there is a big risk
> of creating more and more spagetti-code to solve all kinds of
> dependancies between functions. Coding an application using state
Exactly.
> machines often looks like a bit more work in the beginning, but tends to
> be much more modular, extendible and understandable, since every state
> has well defined bounds, inputs and outputs, and can be easier
Moreover, you can build small FSM's which are *entered* from
a larger FSM. Also, you can have multiple FSM's running
concurrently and let them "talk" to each other. I.e. the
"state language" becomes just another "scripting language".
> comprehended, debugged and reviewed. State machines are no magic, you
> can still mess up ofcourse, but they are a proven technique to keep
> things modular and maintainable.
>
>> [...]
>>
>> My current approach is to use a RDBMS to store all configuration
>> information. The RDBMS can then enforce "who" can set "what".
>> Server-side stored-procedures are built to handle all of the
>> dependencies/"consequences" of changes to individual "settings". This
>> helps document these dependencies in addition to ensuring that "all
>> the right things" get done each time something is changed. It also
>> makes it really easy to store/restore/update configurations as
>> everything is in one place!
>
> Yes, storing all your data in one place has advantages, but also it's
> cons. One of the problems with this is that there is knowledge about
> your configuration in more then one place: there is some kind of
> subsystem that can be configured through the RDBM, but this RDBM also
> needs to know what data this service needs, what types it's properties
> are. The frontent (cli/web/whatever) still needs to know how to present
> those properties to the user, etc.
I store data in exactly *one* place. Part of good database design is
never storing duplicate data -- as this would lead to inconsistencies
between the datasets.
Each "service" that needs configuration data formally queries the
RDBMS for that data. A service is free to *cache* that data
locally (so that it doesn't have to keep asking the same query
over and over again each time it references that configuration
parameter -- i.e. "what color should I make the error messages?").
However, when data in the database is changed, the database
issues a signal() to each process that is a *consumer* of that
data saying, in effect, "your configuration has changed; please
reread the database". This can result in a flurry of queries
at each configuration update. But, that's essentially what
happens when *any* product's configuration changes -- everything
that is configurable needs to know about those changes.
If you store the configuration in a global struct, then there is
no cost to the consumers to retrieve this data. But, then you
have to worry about which consumers may be actively examining
the data when you change it and implement mechanisms to safeguard
it's integrity during updates (e.g., with mutex's, etc.). In
my scheme, each service knows that it "must" retrieve new data
but can do so when it is convenient for that service (a more
cooperative, less authoritarian approach)
> For example, how would you solve the case ware a configuration item for
> a service needs to be selected from a list by the user (lets call it an
> 'enum'), lets say for selecting a compression protocol for some audio
> stream. Valid choices are PCM, DPCM, MP3 and uLaw. The service is
> configured through a configfile, where this choice is written to.
(trying to avoid getting too deep into database theory, here so
I'll just casually describe the mechanism used)
A table (everything is a table!) is created with, for example,
two columns ("fields"): the first may be a textual description
of the "choice" -- i.e. "PCM", "DPCM", "MP3", "uLaw" in your
example. The second (and possibly 3rd, 4th, 5th, etc.) may be a
special "code" that is what your software actually *uses*.
For example, it may be a magic constant (#define). Or, perhaps
a key parameter governing the compressor's operation. <shrug>
It is *whatever* you want/need it to be INTERNALLY. (if you have
5 "variables" that need to be set based on the choice of
protocol, you can store the five values for each of those N
different protocols... and stuff them in the corresponding
variables when that "choice" is selected)
Elsewhere, you have some configuration setting stored in the
"main" database (which is also a *table*). The tables are
*linked* in such a way that the value for that "setting"
can only be one of those listed in the first column of this
other table (just like you can say a field can only contain
text, or an integer, or a float, etc., you can also say it
is constrained to one of these N values)
The DBMS will *ensure* that this setting never gets any value
*other* than one of those N values.
Your configuration program will query this small table to get
the list of choices to display to the user. Based on what he picks,
that choice will be stored for the "setting" in question.
And, the five (?) other internal parameters will be available
for you to copy to whichever "variables" need initialization.
> Something like this is quite hard to implement properly througout the
> system. Ofcourse, the service itself knows what the valid choices are,
> since it needs to parse the configfile and act accordingly. But does
> your RDBM knows what choices are valid, and allow only these to be
Exactly! To rephrase your statement in *my* reality:
The service knows what the valid choices are -- but, relies on
the RDBMS, instead, to *remember* what those actually are! And,
what it should *do* for each of those choices (e.g., "set
magic_number to M_ULAW if uLaw is selected, etc."). Thereafter,
it *relies* on the RDBMS to remember those choices since it
will NOT!
And, the configuration program, instead of being manually
crafted to agree with the list of choices hard-coded into the
application, now ALSO relies on the RDBMS to tell it what those
choices are. I.e. adding a new choice to the RDBMS "magically"
causes that choice to be available in the configuration program.
> stored ? If so, you have knowledge about this enum in two places at
See above :>
> least. Assuming a web interface is used for configuration, the user
> should be presented a list to choose the audio format, so the web
> application will need to have knowledge about this as well, which makes
> three duplicates of the same information.
<grin> Perhaps you now see the beauty/elegance of my approach?
The web application queries the RDBMS as well and builds it's
form based on the results provided *by* the RDBMS.
With careful planning, you can even reduce configuration to
a simple "boiler-plate" that is parameterized by the name of the
setting to be updated and it fetches all of the choices for the
user, etc.
At times, this is a "forced fit" -- some things don't naturally
want to fit in this approach. But, you can often rethink the
way you specify the data in order to make the fit more intuitive.
E.g., you can design an address book in which you include a
place for each person to list their *children*. Or, you
can design that same address book with a place for each
person to list his/her *parents*! The latter case *seems*
clumsy, at first... but, if you think about the implementation,
it is much cleaner -- everyone has *exactly* two parents
(for the most part :> ) whereas a person may have 0, 1, 2...
10, 12... *children*!
> Nightmares really start when settings depend on each other - imaging
> that for some technical reason the MP3 format can only be used when the
> samplerate (another configuratble setting) is 22050 hz or higher. This
> is something the service itself knows, because it can't mix lower sample
> rates with MP3. But is this something your RDBM needs to know as well ?
> And how can I present this to the user, since I don't want the user to
> be able to select MP3 when the samplerate is 8Khz, but I also don't want
> the user to be able to set the samplerate to 8Khz when he selected MP3
> as the encoder. And the hardest part would be to inform the user *why*
> he is not allowd to set the codec: "Warning: setting A is not compatible
> with setting B, change setting A first if you want to set B to X !"
>
> Of course, it's all doable - you can just hardcode these dependancies in
> three different places, but things just grow and get buggier when more
> features are added when you have to maintain code in three different
> places to support one single function. "Don't repeat yourself" is a nice
> philosophy, but is hard to get right!
<grin> I handle this with an expert system. It embodies this
knowledge in it's ruleset. So, it "knows" what is "prohibited".
It also knows *consequences* of choices. E.g., "if the user
selects MP3, *force* the sample rate to 8KHz" (redefining your
criteria, above)
(sigh -- I had hoped not to get dragged into the bowels of this but...)
When the RDBMS is told to update a record (a line in a table),
there are "rules"/procedures that are invoked *by* the RDBMS
before and/or after (based on how they are defined) the update.
So, I can define a trigger that consults the expert system
with the *intended* data to be stored in the table. The expert
system can then tell me "go" or "no go". In the first case,
I then update the table and return "success" to the user
(the user in this case is actually the configuration program;
*it* can then figure out how to convey "success" to the HUMAN
user). In the latter case, I abort the attempt to update the
table and return "failure" to the "user".
In anticipation of other "what if" scenarios you might consider...
Each of these potential updates may be part of a larger RDBMS
"transaction". Think of them as statements in a subroutine (?).
If any of them signal failure, the transaction can automatically
be "un-done". As if it *never* occured -- including all of the
steps taken (and "successfully" completed!) up to this *failed*
step in the transaction.
The beauty is the RDBMS handles all of this for you -- ensuring
that *nothing* ever sees the intermediate results of these
changes, etc.
>> Of course, very few projects have the resources available for this
>> sort of approach :<
>
> Yes, which is a pity, and leads to a lot of bad designed devices with
> incomprehensible interfaces, showing unpredictable behaviour.
I have implemented this in a very heavyweight fashion. (I currently
have the resources available to do so) But, I think the same
approach can also be applied to a leaner environment. After all,
it is just rewrapping *how* you organize your tests and data.
It need not be stored in a "real" RDBMS... just something that
you *treat* as an RDBMS! Likewise, the "expert system" is just
a mechanism that you *treat* as an expert system...
--don