What is your strategy to save non volatile data?

On 8/11/2016 9:12 AM, bitrex wrote:
> On 08/11/2016 12:11 PM, bitrex wrote:
>> On 08/11/2016 11:27 AM, Don Y wrote:
>>
>>> OToOH, if I change the volume level of the radio -- or switch to the
>>> disk drive as "audio source" -- that fact is *also* persistent.  As is
>>> the point in the song where power was interrupted!  When I return to the
>>> vehicle, I expect it to resume playing "from disk", the particular song
>>> that was playing at the time power was lost *and* at the particular
>>> point!
>>
>> That's a cool idea, but as far as I can recall I've yet to drive any
>> vehicle which actually does this.
>>
>> My car is a 2015 Chevy, and if the vehicle is turned off with a song
>> playing from CD or Bluetooth, on restart it just returns to its initial
>> state. It doesn't even seem to remember which Bluetooth device you had
>> connected, so if you have two of them in the car and the one you were
>> using is the second on the list of paired devices, you have to manually
>> re-select it.
>
> Oh, and on hot days when you start driving after the car has been sitting in
> the sun a while, the center console "information display" will sometimes crash
> and reboot.

We walked away from the Nissans (and some other vendor) for similar
reasons:  if I can crash the "system" while it's brand new and
under the watchful care of the dealership, what would it be like
"in the wild"?

David Brown <david.brown@hesbynett.no> writes:

> On 11/08/16 14:48, pozz wrote:
>> Many times I work on embedded projects where it is required to save some
>> data in a non volatile way.  Usualy I use external (to the main
>> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
>> software layer for EEPROM emulation).
>> 
>> However my goal isn't to discuss the hw alternatives, but the data
>> format and strategy.
>> 
>> I know there are many serialization data format (XML, JSON, Protocol
>> Buffer, SQLite), but I don't think they are valid solutions for
>> medium/low end microcontrollers.  They are complex to implement and they
>> aren't really necessary.
>> Data are written and read from the same platform, so the endianess and
>> byte order are not a problem.
>> 
>> IMHO directly saving a C structure to the non-volatile memory is the
>> best method.  When you will load bytes from the non volatile memory, you
>> will have magically your C structure in RAM filled with the saved values.
>> 
>> Is it the same strategy that you use?
>
> As Don says, there is no single best method.  The answer to almost every
> question in embedded development is "it depends".
>
> But for the common case of storing a small set of parameters, I usually
> work with a struct definition.  Along with the parameters in question, I
> include a "magic number" for identification, a "parameter structure
> version number" (very useful for upgrading the parameter set if a later
> version of the firmware uses more parameters), an update serial number,
> a CRC check field, and some space reserved for future additional
> parameters.

Yay, that's *exactly* what I have ended up with.

Binary copying of structs seems to be frowned on but it saves a lot of
work (and code). AFAICS it's fine since it is defined by the ARM EABI
(in my case). And I am hardly going to be unsoldering the ARM and
replacing it with a MIPS.

> Typically I have a statically allocated struct in ram, and two copies in
> eeprom or flash.  On startup, I check both non-volatile copies to see
> which is the latest version (highest serial number) that has a valid
> CRC, and copy that into RAM.  If the parameter structure version number
> does not match the current firmware requirements, I upgrade the
> parameters.

You've copied my stuff haven't you.

> When I need to save the parameters, I use the oldest of the two NV sets.
>  First, I erase it (for eeprom, that's just clearing the magic number).
>  Then I copy in most of the data and CRC, leaving the magic number for
>  last.

> I have had occasion to use a log structured parameter "file system" in
> flash, but usually such complications are unnecessary.

It may be needed if you want to store a counter rather than some
occasionally altered settings. That's when I did it and it was a PITA.


>And certainly
> SQLite, or JSON, etc., are normally overkill.

-- 

John Devereux

On 11/08/16 16:53, Don Y wrote:
> On 8/11/2016 7:12 AM, David Brown wrote:
>
>> But for the common case of storing a small set of parameters, I usually
>> work with a struct definition.  Along with the parameters in question, I
>> include a "magic number" for identification, a "parameter structure
>> version number" (very useful for upgrading the parameter set if a later
>> version of the firmware uses more parameters), an update serial number,
>> a CRC check field, and some space reserved for future additional
>> parameters.
>
> To be clear, you are wrapping *all* of your parameters into a single
> "parameters_to_be_saved" struct.
>
> This can work if they are "configuration parameters" -- data that are
> consulted when setting up the application, restarting it, etc.

Yes, exactly.  I find this to be quite a common situation - but we all 
agree that different systems have different requirements and therefore 
different "best" solutions.

>
> This is different than trying to "save application/process state"
> across power outages.  In that case, the "state" may be scattered
> through many variables dispersed in memory.  And, more difficult to
> ensure catching a "consistent" snapshot.
>
> I.e., if *this* variable is changed, are there any others whose state
> must be reconciled *as* it is being preserved?

Usually in such cases, it is still easy enough to have everything in one 
structure.  At its simplest, all those different state variables 
scattered around the program can be fields of the one state struct.

>
> [It's unclear what the OP is addressing]

Indeed.

>
>> Typically I have a statically allocated struct in ram, and two copies in
>> eeprom or flash.  On startup, I check both non-volatile copies to see
>> which is the latest version (highest serial number) that has a valid
>> CRC, and copy that into RAM.  If the parameter structure version number
>> does not match the current firmware requirements, I upgrade the
>> parameters.
>
> I've used a circular buffer into which "tagged" parameters are placed.
> On restart, I "play back" the buffer to restore the latest versions of
> the parameters encountered, there.  This lets me mix snapshots and
> incremental updates in the same mechanism.

I've done something like that too (that's sort of what I meant be a log 
structure - though I didn't describe it in detail).  It worked well for 
a flash based system - once one sector is getting a bit full, I ran a 
garbage collection pass by running through the buffer copying the latest 
parameters into a new sector, and then erased the old buffer with the 
outdated values.

>
> My current application is effectively a hybrid approach as I'm
> effectively restoring "application state" and not "configuration data"
> (as the system can be considered to be constantly in a state of
> reconfiguration).  So, I let the RDBMS do the "log playback" from
> its WAL.
>
> In that way, I don't have to micromanage "what's important" *in*
> the application(s) but, rather, leave that to the RDBMS.
>
> ("Is this datum something that should be preserved across reboots?")

Il 11/08/2016 18:19, bitrex ha scritto:
 > On 08/11/2016 10:12 AM, David Brown wrote:
 >> On 11/08/16 14:48, pozz wrote:
 >>> Many times I work on embedded projects where it is required to save 
some
 >>> data in a non volatile way.  Usualy I use external (to the main
 >>> microcontroller) serial/parallel EEPROM or internal Flash memory 
(with a
 >>> software layer for EEPROM emulation).
 >>>
 >>> However my goal isn't to discuss the hw alternatives, but the data
 >>> format and strategy.
 >>>
 >>> I know there are many serialization data format (XML, JSON, Protocol
 >>> Buffer, SQLite), but I don't think they are valid solutions for
 >>> medium/low end microcontrollers.
 >
 > Why not? There are implementations of the FAT32 filesystem for SD cards
 > and XML libraries for platforms like the Arduino that are really compact.

Even if you're right (I don't know how simple to integrate and compact
are those libraries for microcontrollers, such as Cortex-M0+), the price
is the first reason to avoid this.  Connector and SD card are much more
expensive against a simple serial EEPROM or the internal Flash memory
that you already have for free.

Another reason could be additional space on the board.

Anyway, could you link those compact libraries for FAT32 and XML?


 > SD cards are cheap; I think I'd consider using them as an alternative to
 > external EEPROM. There's also the advantage that you can just stick them
 > in a PC and edit the data, if you want.
 >

Il 11/08/2016 16:12, David Brown ha scritto:
 > On 11/08/16 14:48, pozz wrote:
 >> Many times I work on embedded projects where it is required to save some
 >> data in a non volatile way.  Usualy I use external (to the main
 >> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
 >> software layer for EEPROM emulation).
 >>
 >> However my goal isn't to discuss the hw alternatives, but the data
 >> format and strategy.
 >>
 >> I know there are many serialization data format (XML, JSON, Protocol
 >> Buffer, SQLite), but I don't think they are valid solutions for
 >> medium/low end microcontrollers.  They are complex to implement and they
 >> aren't really necessary.
 >> Data are written and read from the same platform, so the endianess and
 >> byte order are not a problem.
 >>
 >> IMHO directly saving a C structure to the non-volatile memory is the
 >> best method.  When you will load bytes from the non volatile memory, you
 >> will have magically your C structure in RAM filled with the saved 
values.
 >>
 >> Is it the same strategy that you use?
 >
 > As Don says, there is no single best method.  The answer to almost every
 > question in embedded development is "it depends".

Of course, yes.  However if we define and restrict the requirements, the 
number of good strategies is reduced.


 > But for the common case of storing a small set of parameters,

Indeed my intent is that situation.


 > I usually
 > work with a struct definition.  Along with the parameters in question, I
 > include a "magic number" for identification, a "parameter structure
 > version number" (very useful for upgrading the parameter set if a later
 > version of the firmware uses more parameters), an update serial number,
 > a CRC check field, and some space reserved for future additional 
parameters.
 >
 > Typically I have a statically allocated struct in ram, and two copies in
 > eeprom or flash.  On startup, I check both non-volatile copies to see
 > which is the latest version (highest serial number) that has a valid
 > CRC, and copy that into RAM.  If the parameter structure version number
 > does not match the current firmware requirements, I upgrade the 
parameters.
 >
 > When I need to save the parameters, I use the oldest of the two NV sets.
 >  First, I erase it (for eeprom, that's just clearing the magic number).
 >  Then I copy in most of the data and CRC, leaving the magic number 
for last.

Interesting approach.

I couldn't understand what is "magic number" for identification purpose. 
  CRC can be used to check if the saved struct is valid or not, so the 
"magic number" is useless for this goal.

Maybe you have more than one struct saved to the non volatile memory, 
and the "magic number" identifies the type of the struct?
In this case, I think the physical address where you save the struct 
automatically identifies the struct itsel.
Do you have a different "magic number", "serial number", CRC for each 
saved struct?

Or do you have a dynamic list of structs that could be saved at 
different places of non volatile memory (non fixed address)?  In this 
case, you have a file list somewhere, similar to a FAT table.  Maybe you 
have implemented a minimal filesystem where a file is a struct.


 > I have had occasion to use a log structured parameter "file system" in
 > flash, but usually such complications are unnecessary.  And certainly
 > SQLite, or JSON, etc., are normally overkill.
 >

Il 11/08/2016 23:07, David Brown ha scritto:
 > On 11/08/16 16:53, Don Y wrote:
 >> On 8/11/2016 7:12 AM, David Brown wrote:
 >>
 >>> But for the common case of storing a small set of parameters, I usually
 >>> work with a struct definition.  Along with the parameters in 
question, I
 >>> include a "magic number" for identification, a "parameter structure
 >>> version number" (very useful for upgrading the parameter set if a later
 >>> version of the firmware uses more parameters), an update serial number,
 >>> a CRC check field, and some space reserved for future additional
 >>> parameters.
 >>
 >> To be clear, you are wrapping *all* of your parameters into a single
 >> "parameters_to_be_saved" struct.
 >>
 >> This can work if they are "configuration parameters" -- data that are
 >> consulted when setting up the application, restarting it, etc.
 >
 > Yes, exactly.  I find this to be quite a common situation - but we all
 > agree that different systems have different requirements and therefore
 > different "best" solutions.
 >
 >>
 >> This is different than trying to "save application/process state"
 >> across power outages.  In that case, the "state" may be scattered
 >> through many variables dispersed in memory.  And, more difficult to
 >> ensure catching a "consistent" snapshot.
 >>
 >> I.e., if *this* variable is changed, are there any others whose state
 >> must be reconciled *as* it is being preserved?
 >
 > Usually in such cases, it is still easy enough to have everything in one
 > structure.  At its simplest, all those different state variables
 > scattered around the program can be fields of the one state struct.

"Everything in one structure" means you have a global (as the scope in 
C) struct definition that is accessed by every piece of code in the 
project, even if they are very different section of the project.

In the past I used this approach with a global struct container:
   /* In one .h file */
   struct settings_s {
     uint8_t backlight_level;
     enum DisplayTheme theme;
     uint16_t psu_output_voltage_calibration;
     char password[7];
     ...
   };
   extern struct settings_s settings;

   /* In one .c file */
   struct settings_s settings;

   /* In display module */
   backligh_set(settings.backlight_level);

Every module that manages one or more persistent parameters should know 
the presence of a global settings variable with an exposed type 
definition (struct settings_s).
I think this is a bad approach that increase the coupling between 
modules instead of decreasing it as it should be.


 >> [It's unclear what the OP is addressing]
 >
 > Indeed.

Mainly I'm interested in saving configuration parameters/settings (color 
theme, calibration data, backlight level, and so on).
They are parameters that are changed by the user not so frequent.


 >>> Typically I have a statically allocated struct in ram, and two 
copies in
 >>> eeprom or flash.  On startup, I check both non-volatile copies to see
 >>> which is the latest version (highest serial number) that has a valid
 >>> CRC, and copy that into RAM.  If the parameter structure version number
 >>> does not match the current firmware requirements, I upgrade the
 >>> parameters.
 >>
 >> I've used a circular buffer into which "tagged" parameters are placed.
 >> On restart, I "play back" the buffer to restore the latest versions of
 >> the parameters encountered, there.  This lets me mix snapshots and
 >> incremental updates in the same mechanism.
 >
 > I've done something like that too (that's sort of what I meant be a log
 > structure - though I didn't describe it in detail).  It worked well for
 > a flash based system - once one sector is getting a bit full, I ran a
 > garbage collection pass by running through the buffer copying the latest
 > parameters into a new sector, and then erased the old buffer with the
 > outdated values.
 >
 >>
 >> My current application is effectively a hybrid approach as I'm
 >> effectively restoring "application state" and not "configuration data"
 >> (as the system can be considered to be constantly in a state of
 >> reconfiguration).  So, I let the RDBMS do the "log playback" from
 >> its WAL.
 >>
 >> In that way, I don't have to micromanage "what's important" *in*
 >> the application(s) but, rather, leave that to the RDBMS.
 >>
 >> ("Is this datum something that should be preserved across reboots?")
 >

Il 11/08/2016 17:27, Don Y ha scritto:
> On 8/11/2016 7:53 AM, Don Y wrote:
>> On 8/11/2016 7:12 AM, David Brown wrote:
>>
>>> But for the common case of storing a small set of parameters, I usually
>>> work with a struct definition.  Along with the parameters in question, I
>>> include a "magic number" for identification, a "parameter structure
>>> version number" (very useful for upgrading the parameter set if a later
>>> version of the firmware uses more parameters), an update serial number,
>>> a CRC check field, and some space reserved for future additional
>>> parameters.
>>
>> To be clear, you are wrapping *all* of your parameters into a single
>> "parameters_to_be_saved" struct.
>>
>> This can work if they are "configuration parameters" -- data that are
>> consulted when setting up the application, restarting it, etc.
>>
>> This is different than trying to "save application/process state"
>> across power outages.  In that case, the "state" may be scattered
>> through many variables dispersed in memory.  And, more difficult to
>> ensure catching a "consistent" snapshot.
>>
>> I.e., if *this* variable is changed, are there any others whose state
>> must be reconciled *as* it is being preserved?
>>
>> [It's unclear what the OP is addressing]
>
> A more concrete example:
>
> When I'm in the car,
 > [...]

I'm not sure, but I think many piece of code in the car doesn't save 
anything in a non volatile memory.  The electronics are always power 
supplied from the battery, so it is sufficient to have the variables in RAM.

Indeed when you unplug the battery for a while, all the parameters 
revert back to default.

On 8/11/2016 3:17 PM, pozz wrote:

[much elided]

> "Everything in one structure" means you have a global (as the scope in C)
> struct definition that is accessed by every piece of code in the project, even
> if they are very different section of the project.

No.  It only has to be *effectively* global.

E.g., you can have a global "update_parameter()" that takes a
"key" (name for a parameter -- this could be a literal string
*or* just a small integer) and a parameter value (or pointer to
a parameter value).  Update_parameter() then moves the "value"
into the referenced portion of the "parameters to be saved struct".

In this way, the definition of the struct can remain opaque.
The same sort of "accessor" can be used to retrieve those
parameters from the struct -- without exposing the struct
or the manner in which the individual parameters are "encoded" (!)
in the struct.

E.g., you can pack a bunch of (potentially unrelated) parameters
into a bitfield within that struct.  Data coming *into* it gets
massaged to manipulate the appropriate fields.  Likewise for data
coming *out*.

Finally, because this represents a monitor, of sorts, you can ensure
atomic operations on that "data to be preserved" -- up to and including
an action that automatically saves the struct to persistent store
when power is failing (the rest of the time, "updates" happen to
the "in-RAM" copy of the struct so you don't incur the costs of
updating FLASH more often than necessary)

> In the past I used this approach with a global struct container:
>    /* In one .h file */
>    struct settings_s {
>      uint8_t backlight_level;
>      enum DisplayTheme theme;
>      uint16_t psu_output_voltage_calibration;
>      char password[7];
>      ...
>    };
>    extern struct settings_s settings;
>
>    /* In one .c file */
>    struct settings_s settings;
>
>    /* In display module */
>    backligh_set(settings.backlight_level);
>
> Every module that manages one or more persistent parameters should know the
> presence of a global settings variable with an exposed type definition (struct
> settings_s).
> I think this is a bad approach that increase the coupling between modules
> instead of decreasing it as it should be.
>
>
>  >> [It's unclear what the OP is addressing]
>  >
>  > Indeed.
>
> Mainly I'm interested in saving configuration parameters/settings (color theme,
> calibration data, backlight level, and so on).
> They are parameters that are changed by the user not so frequent.

But, are they changed in a "configuration mode" or, rather, as part
of the normal use of the device?  E.g., if I change the time on my
clock, I don't really enter a special "configuration mode" to do so.
Nor if I change the time zone, alarm settings, etc.  The clock
continues to *be* a clock while these persistent changes are happening.

See my other example re: automobile audio system (and, think of the
similar situation for most other "digital appliances" -- your TV
remembers the channel, volume, picture settings, etc. WITHOUT you
ever having to enter "configuration mode")

On 8/11/2016 2:07 PM, David Brown wrote:
>> I've used a circular buffer into which "tagged" parameters are placed.
>> On restart, I "play back" the buffer to restore the latest versions of
>> the parameters encountered, there.  This lets me mix snapshots and
>> incremental updates in the same mechanism.
>
> I've done something like that too (that's sort of what I meant be a log
> structure - though I didn't describe it in detail).  It worked well for a flash
> based system - once one sector is getting a bit full, I ran a garbage
> collection pass by running through the buffer copying the latest parameters
> into a new sector, and then erased the old buffer with the outdated values.

It has the advantage that it also serves as a debugging tool.

I employ such "black boxes" (named for the fabled flight recorders on
aircraft) in most subsystems as cheap (fast, low resource) mechanisms
for keeping track of what is happening in a subsystem.  In a /post mortem/,
I can peruse their contents to see what was happening in each module just
prior to the "end".

For "settings" that need to be preserved as part of the normal operation
of a device (i.e., more like "state" than "configuration data"), there
can be frequent writes to these BB's -- every interaction with the user!
As such, they can reveal a lot of what else was happening in the system.

(If you just have a "configuration mode", then the amount of traffic is
probably too infrequent to be of much use in a /post mortem/ -- unless,
of course, you've a bug in your UI!)

On 8/11/2016 2:49 PM, pozz wrote:
> Il 11/08/2016 18:19, bitrex ha scritto:
>  > On 08/11/2016 10:12 AM, David Brown wrote:
>  >> On 11/08/16 14:48, pozz wrote:
>  >>> Many times I work on embedded projects where it is required to save some
>  >>> data in a non volatile way.  Usualy I use external (to the main
>  >>> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
>  >>> software layer for EEPROM emulation).
>  >>>
>  >>> However my goal isn't to discuss the hw alternatives, but the data
>  >>> format and strategy.
>  >>>
>  >>> I know there are many serialization data format (XML, JSON, Protocol
>  >>> Buffer, SQLite), but I don't think they are valid solutions for
>  >>> medium/low end microcontrollers.
>  >
>  > Why not? There are implementations of the FAT32 filesystem for SD cards
>  > and XML libraries for platforms like the Arduino that are really compact.
>
> Even if you're right (I don't know how simple to integrate and compact
> are those libraries for microcontrollers, such as Cortex-M0+), the price
> is the first reason to avoid this.  Connector and SD card are much more
> expensive against a simple serial EEPROM or the internal Flash memory
> that you already have for free.
>
> Another reason could be additional space on the board.

Another reason could be the issue of now having to deal with a
memory card that can be arbitrarily removed DURING a write!