What is your strategy to save non volatile data?

Many times I work on embedded projects where it is required to save some 
data in a non volatile way.  Usualy I use external (to the main 
microcontroller) serial/parallel EEPROM or internal Flash memory (with a 
software layer for EEPROM emulation).

However my goal isn't to discuss the hw alternatives, but the data 
format and strategy.

I know there are many serialization data format (XML, JSON, Protocol 
Buffer, SQLite), but I don't think they are valid solutions for 
medium/low end microcontrollers.  They are complex to implement and they 
aren't really necessary.
Data are written and read from the same platform, so the endianess and 
byte order are not a problem.

IMHO directly saving a C structure to the non-volatile memory is the 
best method.  When you will load bytes from the non volatile memory, you 
will have magically your C structure in RAM filled with the saved values.

Is it the same strategy that you use?

On 8/11/2016 5:48 AM, pozz wrote:
> Many times I work on embedded projects where it is required to save some data
> in a non volatile way.  Usualy I use external (to the main microcontroller)
> serial/parallel EEPROM or internal Flash memory (with a software layer for
> EEPROM emulation).
>
> However my goal isn't to discuss the hw alternatives, but the data format and
> strategy.
>
> I know there are many serialization data format (XML, JSON, Protocol Buffer,
> SQLite), but I don't think they are valid solutions for medium/low end
> microcontrollers.  They are complex to implement and they aren't really necessary.
> Data are written and read from the same platform, so the endianess and byte
> order are not a problem.
>
> IMHO directly saving a C structure to the non-volatile memory is the best
> method.  When you will load bytes from the non volatile memory, you will have
> magically your C structure in RAM filled with the saved values.
>
> Is it the same strategy that you use?

There's no universally "best" approach -- much depends on the nature of the
data, it's criticality, the frequency of update, quantity, etc.  *And*, the
hardware facilities available (i.e., if you don't have a "power fail" signal
available, saving megabytes of data *at* loss-of-power can be impractical).

[There is also a dependency on the software structure -- if you can't
get your hands on an "intact" dataset at any given instant, then that
limits what you can *do* with that data when you "need" to preserve it.]

Just "storing struct's" (assuming each has a way of identifying itself,
later -- even if that is implicitly indicated by WHERE it is stored)
means you have to ensure they are stored when they are in a "self-consistent"
state (i.e., when all members "make sense" in the context of their peers).
You also don't have any explicit way of later verifying (at "restore" time)
that the data is intact (checksum/hash/hamming code, etc.).

[likewise, are these data consistent with those *other* data that you're
also saving AT THIS POINT IN TIME?]

It also assumes all the data in a struct *needs* to be saved.  E.g., if I have
a struct that is used to define a text window/pad, it's *dimensions* may
be worth preserving -- but, probably not its *content*, cursor, etc.

Or, that data have the same "value" (i.e., the window's size/shape/position
vs. contents).

Remember, you invariably have to use the saved data to bring the NON-saved
data to some consistent state on "restore".  So, *you* decide what you need
to be able to make that "reinitialization" proper.

The underlying hardware technology also has a significant impact on how and
when you store.  Updating FLASH (limited write cycles, *long* erase cycles)
is much different than BBSRAM; updating a "local" store is much different
than updating a *remote* store (esp via a service!).

I've seen naive approaches that (literally) stored three copies of data
in an attempt to ensure its integrity (despite the inefficiency and lack
of robustness that implies).  At the other extreme, I've seen "raw" data
stored with *no* attempt made to vouch for its integrity (i.e., a three-valued
byte-wide variable could conceivably be recovered with 253 bogus states;
yet the software ASSUMED it would be in ONLY one of the three legitimate
states).

I've used approaches where I constantly maintained a live set of diff(1)'s
against some known system state -- so a "restore" was "restore to known
state followed by this set of (TAGGED!) diff's".

Currently, my only persistent store is a RDBMS.  So, I rely on *it*
to ensure data integrity (I just have to make sure i get the data *to*
the RDBMS before losing power).  I deal with the other issues (frequency
of update, underlying hardware technology, etc.) by partitioning data into
different "tablespaces" (i.e., *where*, physically, each datum gets saved
"in memory") -- so, I can decide how important various data are /ex post
factum/ (cuz I have a limited amount of each type of "memory" and don't want
to have to rewrite my code when I have to move dataX into "precious"
memory and kick dataY *out* to make room for it!).

On 11/08/16 14:48, pozz wrote:
> Many times I work on embedded projects where it is required to save some
> data in a non volatile way.  Usualy I use external (to the main
> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
> software layer for EEPROM emulation).
> 
> However my goal isn't to discuss the hw alternatives, but the data
> format and strategy.
> 
> I know there are many serialization data format (XML, JSON, Protocol
> Buffer, SQLite), but I don't think they are valid solutions for
> medium/low end microcontrollers.  They are complex to implement and they
> aren't really necessary.
> Data are written and read from the same platform, so the endianess and
> byte order are not a problem.
> 
> IMHO directly saving a C structure to the non-volatile memory is the
> best method.  When you will load bytes from the non volatile memory, you
> will have magically your C structure in RAM filled with the saved values.
> 
> Is it the same strategy that you use?

As Don says, there is no single best method.  The answer to almost every
question in embedded development is "it depends".

But for the common case of storing a small set of parameters, I usually
work with a struct definition.  Along with the parameters in question, I
include a "magic number" for identification, a "parameter structure
version number" (very useful for upgrading the parameter set if a later
version of the firmware uses more parameters), an update serial number,
a CRC check field, and some space reserved for future additional parameters.

Typically I have a statically allocated struct in ram, and two copies in
eeprom or flash.  On startup, I check both non-volatile copies to see
which is the latest version (highest serial number) that has a valid
CRC, and copy that into RAM.  If the parameter structure version number
does not match the current firmware requirements, I upgrade the parameters.

When I need to save the parameters, I use the oldest of the two NV sets.
 First, I erase it (for eeprom, that's just clearing the magic number).
 Then I copy in most of the data and CRC, leaving the magic number for last.


I have had occasion to use a log structured parameter "file system" in
flash, but usually such complications are unnecessary.  And certainly
SQLite, or JSON, etc., are normally overkill.

On 8/11/2016 7:12 AM, David Brown wrote:

> But for the common case of storing a small set of parameters, I usually
> work with a struct definition.  Along with the parameters in question, I
> include a "magic number" for identification, a "parameter structure
> version number" (very useful for upgrading the parameter set if a later
> version of the firmware uses more parameters), an update serial number,
> a CRC check field, and some space reserved for future additional parameters.

To be clear, you are wrapping *all* of your parameters into a single
"parameters_to_be_saved" struct.

This can work if they are "configuration parameters" -- data that are
consulted when setting up the application, restarting it, etc.

This is different than trying to "save application/process state"
across power outages.  In that case, the "state" may be scattered
through many variables dispersed in memory.  And, more difficult to
ensure catching a "consistent" snapshot.

I.e., if *this* variable is changed, are there any others whose state
must be reconciled *as* it is being preserved?

[It's unclear what the OP is addressing]

> Typically I have a statically allocated struct in ram, and two copies in
> eeprom or flash.  On startup, I check both non-volatile copies to see
> which is the latest version (highest serial number) that has a valid
> CRC, and copy that into RAM.  If the parameter structure version number
> does not match the current firmware requirements, I upgrade the parameters.

I've used a circular buffer into which "tagged" parameters are placed.
On restart, I "play back" the buffer to restore the latest versions of
the parameters encountered, there.  This lets me mix snapshots and
incremental updates in the same mechanism.

My current application is effectively a hybrid approach as I'm
effectively restoring "application state" and not "configuration data"
(as the system can be considered to be constantly in a state of
reconfiguration).  So, I let the RDBMS do the "log playback" from
its WAL.

In that way, I don't have to micromanage "what's important" *in*
the application(s) but, rather, leave that to the RDBMS.

("Is this datum something that should be preserved across reboots?")

On 8/11/2016 7:53 AM, Don Y wrote:
> On 8/11/2016 7:12 AM, David Brown wrote:
>
>> But for the common case of storing a small set of parameters, I usually
>> work with a struct definition.  Along with the parameters in question, I
>> include a "magic number" for identification, a "parameter structure
>> version number" (very useful for upgrading the parameter set if a later
>> version of the firmware uses more parameters), an update serial number,
>> a CRC check field, and some space reserved for future additional parameters.
>
> To be clear, you are wrapping *all* of your parameters into a single
> "parameters_to_be_saved" struct.
>
> This can work if they are "configuration parameters" -- data that are
> consulted when setting up the application, restarting it, etc.
>
> This is different than trying to "save application/process state"
> across power outages.  In that case, the "state" may be scattered
> through many variables dispersed in memory.  And, more difficult to
> ensure catching a "consistent" snapshot.
>
> I.e., if *this* variable is changed, are there any others whose state
> must be reconciled *as* it is being preserved?
>
> [It's unclear what the OP is addressing]

A more concrete example:

When I'm in the car, I can access any of the "settings" menus to configure
a variety of options pertaining to how I like the car "configured" when
I'm the driver.  I can manipulate each setting which, eventually, will be
"saved" when(?) I exit the "settings menu".  I.e., if I change 10 parameters,
only one "save" needs to be performed (hand-waving, here).

OTOH, if I'm driving and I *tune* the radio to a station, then tell it
to store that frequency as "preset #4", I've effectively created another
*single* persistent parameter.  Without entering a "setup"/settings mode.

OToOH, if I change the volume level of the radio -- or switch to the
disk drive as "audio source" -- that fact is *also* persistent.  As is
the point in the song where power was interrupted!  When I return to the
vehicle, I expect it to resume playing "from disk", the particular song
that was playing at the time power was lost *and* at the particular point!

Do you put "audio source", "title", "playback point", "volume", etc. in
*THE* "parameters_to_be_saved" struct?  And, flush that entire struct to
persistent store each time *any* of the parameters changes?  Or, do you
break up the "state" into smaller chunks that can be identified and
updated in pieces?

On 08/11/2016 11:27 AM, Don Y wrote:

> OToOH, if I change the volume level of the radio -- or switch to the
> disk drive as "audio source" -- that fact is *also* persistent.  As is
> the point in the song where power was interrupted!  When I return to the
> vehicle, I expect it to resume playing "from disk", the particular song
> that was playing at the time power was lost *and* at the particular point!

That's a cool idea, but as far as I can recall I've yet to drive any 
vehicle which actually does this.

My car is a 2015 Chevy, and if the vehicle is turned off with a song 
playing from CD or Bluetooth, on restart it just returns to its initial 
state. It doesn't even seem to remember which Bluetooth device you had 
connected, so if you have two of them in the car and the one you were 
using is the second on the list of paired devices, you have to manually 
re-select it.

On 08/11/2016 12:11 PM, bitrex wrote:
> On 08/11/2016 11:27 AM, Don Y wrote:
>
>> OToOH, if I change the volume level of the radio -- or switch to the
>> disk drive as "audio source" -- that fact is *also* persistent.  As is
>> the point in the song where power was interrupted!  When I return to the
>> vehicle, I expect it to resume playing "from disk", the particular song
>> that was playing at the time power was lost *and* at the particular
>> point!
>
> That's a cool idea, but as far as I can recall I've yet to drive any
> vehicle which actually does this.
>
> My car is a 2015 Chevy, and if the vehicle is turned off with a song
> playing from CD or Bluetooth, on restart it just returns to its initial
> state. It doesn't even seem to remember which Bluetooth device you had
> connected, so if you have two of them in the car and the one you were
> using is the second on the list of paired devices, you have to manually
> re-select it.

Oh, and on hot days when you start driving after the car has been 
sitting in the sun a while, the center console "information display" 
will sometimes crash and reboot.

On 08/11/2016 10:12 AM, David Brown wrote:
> On 11/08/16 14:48, pozz wrote:
>> Many times I work on embedded projects where it is required to save some
>> data in a non volatile way.  Usualy I use external (to the main
>> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
>> software layer for EEPROM emulation).
>>
>> However my goal isn't to discuss the hw alternatives, but the data
>> format and strategy.
>>
>> I know there are many serialization data format (XML, JSON, Protocol
>> Buffer, SQLite), but I don't think they are valid solutions for
>> medium/low end microcontrollers.

Why not? There are implementations of the FAT32 filesystem for SD cards 
and XML libraries for platforms like the Arduino that are really compact.

SD cards are cheap; I think I'd consider using them as an alternative to 
external EEPROM. There's also the advantage that you can just stick them 
in a PC and edit the data, if you want.

pozz <pozzugno@gmail.com> writes:
> IMHO directly saving a C structure to the non-volatile memory is the
> best method.  When you will load bytes from the non volatile memory,
> you will have magically your C structure

Two problems come to mind:

1) if the C structure has pointers in it, then on reload everything must
end up at the same address as before.  That constrains you a lot and can
be a source of bugs and can get in your way if you want to change the
software then reload the parameters.  So you are probably better off
serializing the data somehow--a simple binary format is fine, json etc.
are overkill.

2) As David Brown indicated, you can get in trouble if the program
crashes (loses power, etc) during the save operation.  You need some way
to deal with that, such as by writing out two copies as he describes.
Having a version number in the data as he mentions is also often
worthwhile.

On 8/11/2016 9:11 AM, bitrex wrote:
> On 08/11/2016 11:27 AM, Don Y wrote:
>
>> OToOH, if I change the volume level of the radio -- or switch to the
>> disk drive as "audio source" -- that fact is *also* persistent.  As is
>> the point in the song where power was interrupted!  When I return to the
>> vehicle, I expect it to resume playing "from disk", the particular song
>> that was playing at the time power was lost *and* at the particular point!
>
> That's a cool idea, but as far as I can recall I've yet to drive any vehicle
> which actually does this.

I was describing SWMBO's current vehicle.

If *I* drive it, it remembers which song on the thumb drive it was
playing when I last "accessed" it -- as well as where in that song
it was at the time.

The problem it has is that it restores the "audio state" to whatever
it was when the vehicle was last driven -- regardless of who the current
DRIVER happens to be!  So, if SWMBO drives the car after I have, *she*
hears the song off the thumb drive resuming -- instead of the music
from the *disk* drive (which is where HER music is stored)

> My car is a 2015 Chevy, and if the vehicle is turned off with a song playing
> from CD or Bluetooth, on restart it just returns to its initial state. It
> doesn't even seem to remember which Bluetooth device you had connected, so if
> you have two of them in the car and the one you were using is the second on the
> list of paired devices, you have to manually re-select it.

I was terribly disappointed with the quality of the electronics
(infotainment/navigation) in EVERY vehicle that we considered.  Its
as if the folks designing this stuff are clueless as to how cars are *used*
(why do I have to wait for the backup camera to "boot" before I
BACK OUT of my parking space?