Has anyone references to the compression of data in Flash memory? I
am storing temperature data and wish to use a smaller memory or even
use EEPROM.
TIA
Peter
Data compression
Started by ●April 19, 2009
Reply by ●April 19, 20092009-04-19
On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>Has anyone references to the compression of data in Flash memory? I
>am storing temperature data and wish to use a smaller memory or even
>use EEPROM.
An appropriate reference would be hard to determine without first
knowing the data rates and how much data can you hold in ram without
posting it to flash first and just how much compression you are
seeking. (It's probably reasonable to assume you want lossless
compression, without saying.) There are lots of very simple
techniques that may be perfect for you, or more complex ones. For
example, you might consider simply storing a base temperature
periodically and storing only the deltas from that base afterwards,
until you post another base again. Or deltas from the prior delta if
the temperature is a continuous record not likely to make large jumps.
Can you provide more details?
Jon
>Has anyone references to the compression of data in Flash memory? I
>am storing temperature data and wish to use a smaller memory or even
>use EEPROM.
An appropriate reference would be hard to determine without first
knowing the data rates and how much data can you hold in ram without
posting it to flash first and just how much compression you are
seeking. (It's probably reasonable to assume you want lossless
compression, without saying.) There are lots of very simple
techniques that may be perfect for you, or more complex ones. For
example, you might consider simply storing a base temperature
periodically and storing only the deltas from that base afterwards,
until you post another base again. Or deltas from the prior delta if
the temperature is a continuous record not likely to make large jumps.
Can you provide more details?
Jon
Reply by ●April 19, 20092009-04-19
At 02:47 PM 19/04/2009, you wrote:
I am storing data at 1/2 seconds maximum. I am using a AT45DB45
series with a minimum of 528 bytes of RAM in each buffer. I also will
use a MSP430F2132 with 512bytes of RAM. I could reserve 256
bytes for manipulating any compression algorithms. I would require
the data to be lossless. These loggers will go on seabirds which will
fly, dive into water up to 10 metres, sit on the water and move into
nesting burrows on land. (By the way I will also measure pressure).
The data is 16 bits wide. I have considered using Delta values but I
expect that there will be quite a variation as a bird is sitting in
the sun and then plunges into 10 metres of water. The same can be
said about pressure.
Are there any references in using delta techniques? I use assembler
in my programming.
Thanks
Peter
>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>
> >Has anyone references to the compression of data in Flash memory? I
> >am storing temperature data and wish to use a smaller memory or even
> >use EEPROM.
>
>An appropriate reference would be hard to determine without first
>knowing the data rates and how much data can you hold in ram without
>posting it to flash first and just how much compression you are
>seeking. (It's probably reasonable to assume you want lossless
>compression, without saying.) There are lots of very simple
>techniques that may be perfect for you, or more complex ones. For
>example, you might consider simply storing a base temperature
>periodically and storing only the deltas from that base afterwards,
>until you post another base again. Or deltas from the prior delta if
>the temperature is a continuous record not likely to make large jumps.
>Can you provide more details?
>
>Jon
>
I am storing data at 1/2 seconds maximum. I am using a AT45DB45
series with a minimum of 528 bytes of RAM in each buffer. I also will
use a MSP430F2132 with 512bytes of RAM. I could reserve 256
bytes for manipulating any compression algorithms. I would require
the data to be lossless. These loggers will go on seabirds which will
fly, dive into water up to 10 metres, sit on the water and move into
nesting burrows on land. (By the way I will also measure pressure).
The data is 16 bits wide. I have considered using Delta values but I
expect that there will be quite a variation as a bird is sitting in
the sun and then plunges into 10 metres of water. The same can be
said about pressure.
Are there any references in using delta techniques? I use assembler
in my programming.
Thanks
Peter
>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>
> >Has anyone references to the compression of data in Flash memory? I
> >am storing temperature data and wish to use a smaller memory or even
> >use EEPROM.
>
>An appropriate reference would be hard to determine without first
>knowing the data rates and how much data can you hold in ram without
>posting it to flash first and just how much compression you are
>seeking. (It's probably reasonable to assume you want lossless
>compression, without saying.) There are lots of very simple
>techniques that may be perfect for you, or more complex ones. For
>example, you might consider simply storing a base temperature
>periodically and storing only the deltas from that base afterwards,
>until you post another base again. Or deltas from the prior delta if
>the temperature is a continuous record not likely to make large jumps.
>Can you provide more details?
>
>Jon
>
Reply by ●April 19, 20092009-04-19
On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>At 02:47 PM 19/04/2009, you wrote:
>I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>series with a minimum of 528 bytes of RAM in each buffer. I also will
>use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>bytes for manipulating any compression algorithms. I would require
>the data to be lossless. These loggers will go on seabirds which will
>fly, dive into water up to 10 metres, sit on the water and move into
>nesting burrows on land. (By the way I will also measure pressure).
>The data is 16 bits wide. I have considered using Delta values but I
>expect that there will be quite a variation as a bird is sitting in
>the sun and then plunges into 10 metres of water. The same can be
>said about pressure.
>Are there any references in using delta techniques? I use assembler
>in my programming.
Numerical Recipes contains some information on compression that may
help a lot. Plenty of theory and practice is presented to get where
you need to go, I suspect. I haven't googled for it, but I'm sure
that if you did you would have a lot to weed through.
If the data, in net effect, is hardly discernable from "random" then
you probably cannot compress it much or at all. In fact, one of the
useful definitions some people use to imagine with, about randomness,
is the idea that truly random data doesn't compress. You need to
exploit some manner of repeatability in your data for decent
compression. Often, that is just the high order bits in data that
doesn't change rapidly over it's magnitude range, as they are usually
the same or nearly the same for nearby data values. In your case, you
may find good compression at times and bad at others.
Your comments tend to argue for something a little more carefully
crafted than just storing deltas. However, if you permit yourself a
special marker to allow you to restate the base value when you need to
(and are willing to pay that premium at times) then you may get good
compression when the data isn't changing that much and worse when it
changes a lot, but all that may be just fine.
One thing that would really help you evaluate choices would be a
sample data set. Do you have that?
Jon
>>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>>
>> >Has anyone references to the compression of data in Flash memory? I
>> >am storing temperature data and wish to use a smaller memory or even
>> >use EEPROM.
>>
>>An appropriate reference would be hard to determine without first
>>knowing the data rates and how much data can you hold in ram without
>>posting it to flash first and just how much compression you are
>>seeking. (It's probably reasonable to assume you want lossless
>>compression, without saying.) There are lots of very simple
>>techniques that may be perfect for you, or more complex ones. For
>>example, you might consider simply storing a base temperature
>>periodically and storing only the deltas from that base afterwards,
>>until you post another base again. Or deltas from the prior delta if
>>the temperature is a continuous record not likely to make large jumps.
>>Can you provide more details?
>>
>>Jon
>>
>At 02:47 PM 19/04/2009, you wrote:
>I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>series with a minimum of 528 bytes of RAM in each buffer. I also will
>use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>bytes for manipulating any compression algorithms. I would require
>the data to be lossless. These loggers will go on seabirds which will
>fly, dive into water up to 10 metres, sit on the water and move into
>nesting burrows on land. (By the way I will also measure pressure).
>The data is 16 bits wide. I have considered using Delta values but I
>expect that there will be quite a variation as a bird is sitting in
>the sun and then plunges into 10 metres of water. The same can be
>said about pressure.
>Are there any references in using delta techniques? I use assembler
>in my programming.
Numerical Recipes contains some information on compression that may
help a lot. Plenty of theory and practice is presented to get where
you need to go, I suspect. I haven't googled for it, but I'm sure
that if you did you would have a lot to weed through.
If the data, in net effect, is hardly discernable from "random" then
you probably cannot compress it much or at all. In fact, one of the
useful definitions some people use to imagine with, about randomness,
is the idea that truly random data doesn't compress. You need to
exploit some manner of repeatability in your data for decent
compression. Often, that is just the high order bits in data that
doesn't change rapidly over it's magnitude range, as they are usually
the same or nearly the same for nearby data values. In your case, you
may find good compression at times and bad at others.
Your comments tend to argue for something a little more carefully
crafted than just storing deltas. However, if you permit yourself a
special marker to allow you to restate the base value when you need to
(and are willing to pay that premium at times) then you may get good
compression when the data isn't changing that much and worse when it
changes a lot, but all that may be just fine.
One thing that would really help you evaluate choices would be a
sample data set. Do you have that?
Jon
>>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>>
>> >Has anyone references to the compression of data in Flash memory? I
>> >am storing temperature data and wish to use a smaller memory or even
>> >use EEPROM.
>>
>>An appropriate reference would be hard to determine without first
>>knowing the data rates and how much data can you hold in ram without
>>posting it to flash first and just how much compression you are
>>seeking. (It's probably reasonable to assume you want lossless
>>compression, without saying.) There are lots of very simple
>>techniques that may be perfect for you, or more complex ones. For
>>example, you might consider simply storing a base temperature
>>periodically and storing only the deltas from that base afterwards,
>>until you post another base again. Or deltas from the prior delta if
>>the temperature is a continuous record not likely to make large jumps.
>>Can you provide more details?
>>
>>Jon
>>
Reply by ●April 20, 20092009-04-20
At 09:19 AM 20/04/2009, you wrote:
I took a look at Numerical Recipes. It looks a great resource, but I
take your point on my data. I do not have any data available yet. I
am just getting the design together. Having read a little about the
techniques, I realize I can restrict the range of the sensor and so
use less bits for storage.
Thanks
Peter
>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>
> >At 02:47 PM 19/04/2009, you wrote:
> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
> >series with a minimum of 528 bytes of RAM in each buffer. I also will
> >use a MSP430F2132 with 512bytes of RAM. I could reserve 256
> >bytes for manipulating any compression algorithms. I would require
> >the data to be lossless. These loggers will go on seabirds which will
> >fly, dive into water up to 10 metres, sit on the water and move into
> >nesting burrows on land. (By the way I will also measure pressure).
> >The data is 16 bits wide. I have considered using Delta values but I
> >expect that there will be quite a variation as a bird is sitting in
> >the sun and then plunges into 10 metres of water. The same can be
> >said about pressure.
> >Are there any references in using delta techniques? I use assembler
> >in my programming.
>
>Numerical Recipes contains some information on compression that may
>help a lot. Plenty of theory and practice is presented to get where
>you need to go, I suspect. I haven't googled for it, but I'm sure
>that if you did you would have a lot to weed through.
>
>If the data, in net effect, is hardly discernable from "random" then
>you probably cannot compress it much or at all. In fact, one of the
>useful definitions some people use to imagine with, about randomness,
>is the idea that truly random data doesn't compress. You need to
>exploit some manner of repeatability in your data for decent
>compression. Often, that is just the high order bits in data that
>doesn't change rapidly over it's magnitude range, as they are usually
>the same or nearly the same for nearby data values. In your case, you
>may find good compression at times and bad at others.
>
>Your comments tend to argue for something a little more carefully
>crafted than just storing deltas. However, if you permit yourself a
>special marker to allow you to restate the base value when you need to
>(and are willing to pay that premium at times) then you may get good
>compression when the data isn't changing that much and worse when it
>changes a lot, but all that may be just fine.
>
>One thing that would really help you evaluate choices would be a
>sample data set. Do you have that?
>
>Jon
>
> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
> >>
> >> >Has anyone references to the compression of data in Flash memory? I
> >> >am storing temperature data and wish to use a smaller memory or even
> >> >use EEPROM.
> >>
> >>An appropriate reference would be hard to determine without first
> >>knowing the data rates and how much data can you hold in ram without
> >>posting it to flash first and just how much compression you are
> >>seeking. (It's probably reasonable to assume you want lossless
> >>compression, without saying.) There are lots of very simple
> >>techniques that may be perfect for you, or more complex ones. For
> >>example, you might consider simply storing a base temperature
> >>periodically and storing only the deltas from that base afterwards,
> >>until you post another base again. Or deltas from the prior delta if
> >>the temperature is a continuous record not likely to make large jumps.
> >>Can you provide more details?
> >>
> >>Jon
> >>
> >>
> >>
> >>
> >>
> >>
> >>
I took a look at Numerical Recipes. It looks a great resource, but I
take your point on my data. I do not have any data available yet. I
am just getting the design together. Having read a little about the
techniques, I realize I can restrict the range of the sensor and so
use less bits for storage.
Thanks
Peter
>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>
> >At 02:47 PM 19/04/2009, you wrote:
> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
> >series with a minimum of 528 bytes of RAM in each buffer. I also will
> >use a MSP430F2132 with 512bytes of RAM. I could reserve 256
> >bytes for manipulating any compression algorithms. I would require
> >the data to be lossless. These loggers will go on seabirds which will
> >fly, dive into water up to 10 metres, sit on the water and move into
> >nesting burrows on land. (By the way I will also measure pressure).
> >The data is 16 bits wide. I have considered using Delta values but I
> >expect that there will be quite a variation as a bird is sitting in
> >the sun and then plunges into 10 metres of water. The same can be
> >said about pressure.
> >Are there any references in using delta techniques? I use assembler
> >in my programming.
>
>Numerical Recipes contains some information on compression that may
>help a lot. Plenty of theory and practice is presented to get where
>you need to go, I suspect. I haven't googled for it, but I'm sure
>that if you did you would have a lot to weed through.
>
>If the data, in net effect, is hardly discernable from "random" then
>you probably cannot compress it much or at all. In fact, one of the
>useful definitions some people use to imagine with, about randomness,
>is the idea that truly random data doesn't compress. You need to
>exploit some manner of repeatability in your data for decent
>compression. Often, that is just the high order bits in data that
>doesn't change rapidly over it's magnitude range, as they are usually
>the same or nearly the same for nearby data values. In your case, you
>may find good compression at times and bad at others.
>
>Your comments tend to argue for something a little more carefully
>crafted than just storing deltas. However, if you permit yourself a
>special marker to allow you to restate the base value when you need to
>(and are willing to pay that premium at times) then you may get good
>compression when the data isn't changing that much and worse when it
>changes a lot, but all that may be just fine.
>
>One thing that would really help you evaluate choices would be a
>sample data set. Do you have that?
>
>Jon
>
> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
> >>
> >> >Has anyone references to the compression of data in Flash memory? I
> >> >am storing temperature data and wish to use a smaller memory or even
> >> >use EEPROM.
> >>
> >>An appropriate reference would be hard to determine without first
> >>knowing the data rates and how much data can you hold in ram without
> >>posting it to flash first and just how much compression you are
> >>seeking. (It's probably reasonable to assume you want lossless
> >>compression, without saying.) There are lots of very simple
> >>techniques that may be perfect for you, or more complex ones. For
> >>example, you might consider simply storing a base temperature
> >>periodically and storing only the deltas from that base afterwards,
> >>until you post another base again. Or deltas from the prior delta if
> >>the temperature is a continuous record not likely to make large jumps.
> >>Can you provide more details?
> >>
> >>Jon
> >>
> >>
> >>
> >>
> >>
> >>
> >>
Reply by ●April 20, 20092009-04-20
On Mon, 20 Apr 2009 11:03:42 +0800, you wrote:
>At 09:19 AM 20/04/2009, you wrote:
>I took a look at Numerical Recipes. It looks a great resource, but I
>take your point on my data. I do not have any data available yet.
Can you get some? I mean, can you produce some simulation of the data
using software on a PC? I often do this when testing algorithms. For
example, I may generate the basic data based upon theoretical, ideal
models and then add noise to them and perhaps other interfering
elements of interest and then pipe that data into the algorithm to
verify that it works acceptably. It's probably a good idea to take
some steps along this line before going out and fielding devices, with
all the associated work involved, only to find out that some simple
change would have greatly improved the length of your datasets that
you later retrieve.
Sound and pressure should be able to be simulated to a reasoned degree
and noise is fairly easy -- there are some simple routines to generate
Gaussian distributions ... in fact, Numerical Recipes includes those,
too.
>I
>am just getting the design together. Having read a little about the
>techniques, I realize I can restrict the range of the sensor and so
>use less bits for storage.
Well, that always helps, too.
Jon
>
>Thanks
>
>Peter
>>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>>
>> >At 02:47 PM 19/04/2009, you wrote:
>> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>> >series with a minimum of 528 bytes of RAM in each buffer. I also will
>> >use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>> >bytes for manipulating any compression algorithms. I would require
>> >the data to be lossless. These loggers will go on seabirds which will
>> >fly, dive into water up to 10 metres, sit on the water and move into
>> >nesting burrows on land. (By the way I will also measure pressure).
>> >The data is 16 bits wide. I have considered using Delta values but I
>> >expect that there will be quite a variation as a bird is sitting in
>> >the sun and then plunges into 10 metres of water. The same can be
>> >said about pressure.
>> >Are there any references in using delta techniques? I use assembler
>> >in my programming.
>>
>>Numerical Recipes contains some information on compression that may
>>help a lot. Plenty of theory and practice is presented to get where
>>you need to go, I suspect. I haven't googled for it, but I'm sure
>>that if you did you would have a lot to weed through.
>>
>>If the data, in net effect, is hardly discernable from "random" then
>>you probably cannot compress it much or at all. In fact, one of the
>>useful definitions some people use to imagine with, about randomness,
>>is the idea that truly random data doesn't compress. You need to
>>exploit some manner of repeatability in your data for decent
>>compression. Often, that is just the high order bits in data that
>>doesn't change rapidly over it's magnitude range, as they are usually
>>the same or nearly the same for nearby data values. In your case, you
>>may find good compression at times and bad at others.
>>
>>Your comments tend to argue for something a little more carefully
>>crafted than just storing deltas. However, if you permit yourself a
>>special marker to allow you to restate the base value when you need to
>>(and are willing to pay that premium at times) then you may get good
>>compression when the data isn't changing that much and worse when it
>>changes a lot, but all that may be just fine.
>>
>>One thing that would really help you evaluate choices would be a
>>sample data set. Do you have that?
>>
>>Jon
>>
>> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>> >>
>> >> >Has anyone references to the compression of data in Flash memory? I
>> >> >am storing temperature data and wish to use a smaller memory or even
>> >> >use EEPROM.
>> >>
>> >>An appropriate reference would be hard to determine without first
>> >>knowing the data rates and how much data can you hold in ram without
>> >>posting it to flash first and just how much compression you are
>> >>seeking. (It's probably reasonable to assume you want lossless
>> >>compression, without saying.) There are lots of very simple
>> >>techniques that may be perfect for you, or more complex ones. For
>> >>example, you might consider simply storing a base temperature
>> >>periodically and storing only the deltas from that base afterwards,
>> >>until you post another base again. Or deltas from the prior delta if
>> >>the temperature is a continuous record not likely to make large jumps.
>> >>Can you provide more details?
>> >>
>> >>Jon
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>At 09:19 AM 20/04/2009, you wrote:
>I took a look at Numerical Recipes. It looks a great resource, but I
>take your point on my data. I do not have any data available yet.
Can you get some? I mean, can you produce some simulation of the data
using software on a PC? I often do this when testing algorithms. For
example, I may generate the basic data based upon theoretical, ideal
models and then add noise to them and perhaps other interfering
elements of interest and then pipe that data into the algorithm to
verify that it works acceptably. It's probably a good idea to take
some steps along this line before going out and fielding devices, with
all the associated work involved, only to find out that some simple
change would have greatly improved the length of your datasets that
you later retrieve.
Sound and pressure should be able to be simulated to a reasoned degree
and noise is fairly easy -- there are some simple routines to generate
Gaussian distributions ... in fact, Numerical Recipes includes those,
too.
>I
>am just getting the design together. Having read a little about the
>techniques, I realize I can restrict the range of the sensor and so
>use less bits for storage.
Well, that always helps, too.
Jon
>
>Thanks
>
>Peter
>>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>>
>> >At 02:47 PM 19/04/2009, you wrote:
>> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>> >series with a minimum of 528 bytes of RAM in each buffer. I also will
>> >use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>> >bytes for manipulating any compression algorithms. I would require
>> >the data to be lossless. These loggers will go on seabirds which will
>> >fly, dive into water up to 10 metres, sit on the water and move into
>> >nesting burrows on land. (By the way I will also measure pressure).
>> >The data is 16 bits wide. I have considered using Delta values but I
>> >expect that there will be quite a variation as a bird is sitting in
>> >the sun and then plunges into 10 metres of water. The same can be
>> >said about pressure.
>> >Are there any references in using delta techniques? I use assembler
>> >in my programming.
>>
>>Numerical Recipes contains some information on compression that may
>>help a lot. Plenty of theory and practice is presented to get where
>>you need to go, I suspect. I haven't googled for it, but I'm sure
>>that if you did you would have a lot to weed through.
>>
>>If the data, in net effect, is hardly discernable from "random" then
>>you probably cannot compress it much or at all. In fact, one of the
>>useful definitions some people use to imagine with, about randomness,
>>is the idea that truly random data doesn't compress. You need to
>>exploit some manner of repeatability in your data for decent
>>compression. Often, that is just the high order bits in data that
>>doesn't change rapidly over it's magnitude range, as they are usually
>>the same or nearly the same for nearby data values. In your case, you
>>may find good compression at times and bad at others.
>>
>>Your comments tend to argue for something a little more carefully
>>crafted than just storing deltas. However, if you permit yourself a
>>special marker to allow you to restate the base value when you need to
>>(and are willing to pay that premium at times) then you may get good
>>compression when the data isn't changing that much and worse when it
>>changes a lot, but all that may be just fine.
>>
>>One thing that would really help you evaluate choices would be a
>>sample data set. Do you have that?
>>
>>Jon
>>
>> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>> >>
>> >> >Has anyone references to the compression of data in Flash memory? I
>> >> >am storing temperature data and wish to use a smaller memory or even
>> >> >use EEPROM.
>> >>
>> >>An appropriate reference would be hard to determine without first
>> >>knowing the data rates and how much data can you hold in ram without
>> >>posting it to flash first and just how much compression you are
>> >>seeking. (It's probably reasonable to assume you want lossless
>> >>compression, without saying.) There are lots of very simple
>> >>techniques that may be perfect for you, or more complex ones. For
>> >>example, you might consider simply storing a base temperature
>> >>periodically and storing only the deltas from that base afterwards,
>> >>until you post another base again. Or deltas from the prior delta if
>> >>the temperature is a continuous record not likely to make large jumps.
>> >>Can you provide more details?
>> >>
>> >>Jon
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
Reply by ●April 20, 20092009-04-20
At 11:24 AM 20/04/2009, you wrote:
I can get some data in a day or 2. I will set up a sensor and get a
full days worth. It will not alter that much as the sensor will sit
on my lab bench.
I'll come back then with it.
Cheers
Peter
>On Mon, 20 Apr 2009 11:03:42 +0800, you wrote:
>
> >At 09:19 AM 20/04/2009, you wrote:
> >I took a look at Numerical Recipes. It looks a great resource, but I
> >take your point on my data. I do not have any data available yet.
>
>Can you get some? I mean, can you produce some simulation of the data
>using software on a PC? I often do this when testing algorithms. For
>example, I may generate the basic data based upon theoretical, ideal
>models and then add noise to them and perhaps other interfering
>elements of interest and then pipe that data into the algorithm to
>verify that it works acceptably. It's probably a good idea to take
>some steps along this line before going out and fielding devices, with
>all the associated work involved, only to find out that some simple
>change would have greatly improved the length of your datasets that
>you later retrieve.
>
>Sound and pressure should be able to be simulated to a reasoned degree
>and noise is fairly easy -- there are some simple routines to generate
>Gaussian distributions ... in fact, Numerical Recipes includes those,
>too.
>
> >I
> >am just getting the design together. Having read a little about the
> >techniques, I realize I can restrict the range of the sensor and so
> >use less bits for storage.
>
>Well, that always helps, too.
>
>Jon
>
> >
> >Thanks
> >
> >Peter
> >>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
> >>
> >> >At 02:47 PM 19/04/2009, you wrote:
> >> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
> >> >series with a minimum of 528 bytes of RAM in each buffer. I also will
> >> >use a MSP430F2132 with 512bytes of RAM. I could reserve 256
> >> >bytes for manipulating any compression algorithms. I would require
> >> >the data to be lossless. These loggers will go on seabirds which will
> >> >fly, dive into water up to 10 metres, sit on the water and move into
> >> >nesting burrows on land. (By the way I will also measure pressure).
> >> >The data is 16 bits wide. I have considered using Delta values but I
> >> >expect that there will be quite a variation as a bird is sitting in
> >> >the sun and then plunges into 10 metres of water. The same can be
> >> >said about pressure.
> >> >Are there any references in using delta techniques? I use assembler
> >> >in my programming.
> >>
> >>Numerical Recipes contains some information on compression that may
> >>help a lot. Plenty of theory and practice is presented to get where
> >>you need to go, I suspect. I haven't googled for it, but I'm sure
> >>that if you did you would have a lot to weed through.
> >>
> >>If the data, in net effect, is hardly discernable from "random" then
> >>you probably cannot compress it much or at all. In fact, one of the
> >>useful definitions some people use to imagine with, about randomness,
> >>is the idea that truly random data doesn't compress. You need to
> >>exploit some manner of repeatability in your data for decent
> >>compression. Often, that is just the high order bits in data that
> >>doesn't change rapidly over it's magnitude range, as they are usually
> >>the same or nearly the same for nearby data values. In your case, you
> >>may find good compression at times and bad at others.
> >>
> >>Your comments tend to argue for something a little more carefully
> >>crafted than just storing deltas. However, if you permit yourself a
> >>special marker to allow you to restate the base value when you need to
> >>(and are willing to pay that premium at times) then you may get good
> >>compression when the data isn't changing that much and worse when it
> >>changes a lot, but all that may be just fine.
> >>
> >>One thing that would really help you evaluate choices would be a
> >>sample data set. Do you have that?
> >>
> >>Jon
> >>
> >> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
> >> >>
> >> >> >Has anyone references to the compression of data in Flash memory? I
> >> >> >am storing temperature data and wish to use a smaller memory or even
> >> >> >use EEPROM.
> >> >>
> >> >>An appropriate reference would be hard to determine without first
> >> >>knowing the data rates and how much data can you hold in ram without
> >> >>posting it to flash first and just how much compression you are
> >> >>seeking. (It's probably reasonable to assume you want lossless
> >> >>compression, without saying.) There are lots of very simple
> >> >>techniques that may be perfect for you, or more complex ones. For
> >> >>example, you might consider simply storing a base temperature
> >> >>periodically and storing only the deltas from that base afterwards,
> >> >>until you post another base again. Or deltas from the prior delta if
> >> >>the temperature is a continuous record not likely to make large jumps.
> >> >>Can you provide more details?
> >> >>
> >> >>Jon
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
I can get some data in a day or 2. I will set up a sensor and get a
full days worth. It will not alter that much as the sensor will sit
on my lab bench.
I'll come back then with it.
Cheers
Peter
>On Mon, 20 Apr 2009 11:03:42 +0800, you wrote:
>
> >At 09:19 AM 20/04/2009, you wrote:
> >I took a look at Numerical Recipes. It looks a great resource, but I
> >take your point on my data. I do not have any data available yet.
>
>Can you get some? I mean, can you produce some simulation of the data
>using software on a PC? I often do this when testing algorithms. For
>example, I may generate the basic data based upon theoretical, ideal
>models and then add noise to them and perhaps other interfering
>elements of interest and then pipe that data into the algorithm to
>verify that it works acceptably. It's probably a good idea to take
>some steps along this line before going out and fielding devices, with
>all the associated work involved, only to find out that some simple
>change would have greatly improved the length of your datasets that
>you later retrieve.
>
>Sound and pressure should be able to be simulated to a reasoned degree
>and noise is fairly easy -- there are some simple routines to generate
>Gaussian distributions ... in fact, Numerical Recipes includes those,
>too.
>
> >I
> >am just getting the design together. Having read a little about the
> >techniques, I realize I can restrict the range of the sensor and so
> >use less bits for storage.
>
>Well, that always helps, too.
>
>Jon
>
> >
> >Thanks
> >
> >Peter
> >>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
> >>
> >> >At 02:47 PM 19/04/2009, you wrote:
> >> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
> >> >series with a minimum of 528 bytes of RAM in each buffer. I also will
> >> >use a MSP430F2132 with 512bytes of RAM. I could reserve 256
> >> >bytes for manipulating any compression algorithms. I would require
> >> >the data to be lossless. These loggers will go on seabirds which will
> >> >fly, dive into water up to 10 metres, sit on the water and move into
> >> >nesting burrows on land. (By the way I will also measure pressure).
> >> >The data is 16 bits wide. I have considered using Delta values but I
> >> >expect that there will be quite a variation as a bird is sitting in
> >> >the sun and then plunges into 10 metres of water. The same can be
> >> >said about pressure.
> >> >Are there any references in using delta techniques? I use assembler
> >> >in my programming.
> >>
> >>Numerical Recipes contains some information on compression that may
> >>help a lot. Plenty of theory and practice is presented to get where
> >>you need to go, I suspect. I haven't googled for it, but I'm sure
> >>that if you did you would have a lot to weed through.
> >>
> >>If the data, in net effect, is hardly discernable from "random" then
> >>you probably cannot compress it much or at all. In fact, one of the
> >>useful definitions some people use to imagine with, about randomness,
> >>is the idea that truly random data doesn't compress. You need to
> >>exploit some manner of repeatability in your data for decent
> >>compression. Often, that is just the high order bits in data that
> >>doesn't change rapidly over it's magnitude range, as they are usually
> >>the same or nearly the same for nearby data values. In your case, you
> >>may find good compression at times and bad at others.
> >>
> >>Your comments tend to argue for something a little more carefully
> >>crafted than just storing deltas. However, if you permit yourself a
> >>special marker to allow you to restate the base value when you need to
> >>(and are willing to pay that premium at times) then you may get good
> >>compression when the data isn't changing that much and worse when it
> >>changes a lot, but all that may be just fine.
> >>
> >>One thing that would really help you evaluate choices would be a
> >>sample data set. Do you have that?
> >>
> >>Jon
> >>
> >> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
> >> >>
> >> >> >Has anyone references to the compression of data in Flash memory? I
> >> >> >am storing temperature data and wish to use a smaller memory or even
> >> >> >use EEPROM.
> >> >>
> >> >>An appropriate reference would be hard to determine without first
> >> >>knowing the data rates and how much data can you hold in ram without
> >> >>posting it to flash first and just how much compression you are
> >> >>seeking. (It's probably reasonable to assume you want lossless
> >> >>compression, without saying.) There are lots of very simple
> >> >>techniques that may be perfect for you, or more complex ones. For
> >> >>example, you might consider simply storing a base temperature
> >> >>periodically and storing only the deltas from that base afterwards,
> >> >>until you post another base again. Or deltas from the prior delta if
> >> >>the temperature is a continuous record not likely to make large jumps.
> >> >>Can you provide more details?
> >> >>
> >> >>Jon
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
Reply by ●April 20, 20092009-04-20
Peter,
well, just "play sea-bird" put your sensor in water, put in the air, in the
sun, let it stay outside night/day to get test values. How good is the
thermal coupling between the media (water, air) and your sensor? (You
should have a good coupling to get reasonable results, your sensor should
not be casted, should have a low thermal mass, direct touch to the
outside). My guess is that you have several rapid changes and a stable (low
changing) temperature over a long time. Best compression is to just store
the temperature changes (time-delta and temperature-delta) and/or use a
linear approximation for the ramps. If a block of data is full, store
start-temperature, start-time for the following deltas (you can then
restore partial data if there is any fault).
M.
Peter Grey :
> At 11:24 AM 20/04/2009, you wrote:
>
> I can get some data in a day or 2. I will set up a sensor and get a
> full days worth. It will not alter that much as the sensor will sit
> on my lab bench.
>
> I'll come back then with it.
>
> Cheers
> Peter
>>On Mon, 20 Apr 2009 11:03:42 +0800, you wrote:
>>
>> >At 09:19 AM 20/04/2009, you wrote:
>> >I took a look at Numerical Recipes. It looks a great resource, but I
>> >take your point on my data. I do not have any data available yet.
>>
>>Can you get some? I mean, can you produce some simulation of the data
>>using software on a PC? I often do this when testing algorithms. For
>>example, I may generate the basic data based upon theoretical, ideal
>>models and then add noise to them and perhaps other interfering
>>elements of interest and then pipe that data into the algorithm to
>>verify that it works acceptably. It's probably a good idea to take
>>some steps along this line before going out and fielding devices, with
>>all the associated work involved, only to find out that some simple
>>change would have greatly improved the length of your datasets that
>>you later retrieve.
>>
>>Sound and pressure should be able to be simulated to a reasoned degree
>>and noise is fairly easy -- there are some simple routines to generate
>>Gaussian distributions ... in fact, Numerical Recipes includes those,
>>too.
>>
>> >I
>> >am just getting the design together. Having read a little about the
>> >techniques, I realize I can restrict the range of the sensor and so
>> >use less bits for storage.
>>
>>Well, that always helps, too.
>>
>>Jon
>>
>> >
>> >Thanks
>> >
>> >Peter
>> >>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>> >>
>> >> >At 02:47 PM 19/04/2009, you wrote:
>> >> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>> >> >series with a minimum of 528 bytes of RAM in each buffer. I also
>> >> >will use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>> >> >bytes for manipulating any compression algorithms. I would require
>> >> >the data to be lossless. These loggers will go on seabirds which
>> >> >will fly, dive into water up to 10 metres, sit on the water and
>> >> >move into nesting burrows on land. (By the way I will also measure
>> >> >pressure). The data is 16 bits wide. I have considered using Delta
>> >> >values but I expect that there will be quite a variation as a bird
>> >> >is sitting in the sun and then plunges into 10 metres of water. The
>> >> >same can be said about pressure.
>> >> >Are there any references in using delta techniques? I use assembler
>> >> >in my programming.
>> >>
>> >>Numerical Recipes contains some information on compression that may
>> >>help a lot. Plenty of theory and practice is presented to get where
>> >>you need to go, I suspect. I haven't googled for it, but I'm sure
>> >>that if you did you would have a lot to weed through.
>> >>
>> >>If the data, in net effect, is hardly discernable from "random" then
>> >>you probably cannot compress it much or at all. In fact, one of the
>> >>useful definitions some people use to imagine with, about randomness,
>> >>is the idea that truly random data doesn't compress. You need to
>> >>exploit some manner of repeatability in your data for decent
>> >>compression. Often, that is just the high order bits in data that
>> >>doesn't change rapidly over it's magnitude range, as they are usually
>> >>the same or nearly the same for nearby data values. In your case,
>> >>you may find good compression at times and bad at others.
>> >>
>> >>Your comments tend to argue for something a little more carefully
>> >>crafted than just storing deltas. However, if you permit yourself a
>> >>special marker to allow you to restate the base value when you need
>> >>to (and are willing to pay that premium at times) then you may get
>> >>good compression when the data isn't changing that much and worse
>> >>when it changes a lot, but all that may be just fine.
>> >>
>> >>One thing that would really help you evaluate choices would be a
>> >>sample data set. Do you have that?
>> >>
>> >>Jon
>> >>
>> >> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>> >> >>
>> >> >> >Has anyone references to the compression of data in Flash
>> >> >> >memory? I am storing temperature data and wish to use a smaller
>> >> >> >memory or even use EEPROM.
>> >> >>
>> >> >>An appropriate reference would be hard to determine without first
>> >> >>knowing the data rates and how much data can you hold in ram
>> >> >>without posting it to flash first and just how much compression
>> >> >>you are seeking. (It's probably reasonable to assume you want
>> >> >>lossless compression, without saying.) There are lots of very
>> >> >>simple techniques that may be perfect for you, or more complex
>> >> >>ones. For example, you might consider simply storing a base
>> >> >>temperature periodically and storing only the deltas from that
>> >> >>base afterwards, until you post another base again. Or deltas
>> >> >>from the prior delta if the temperature is a continuous record not
>> >> >>likely to make large jumps. Can you provide more details?
>> >> >>
>> >> >>Jon
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
well, just "play sea-bird" put your sensor in water, put in the air, in the
sun, let it stay outside night/day to get test values. How good is the
thermal coupling between the media (water, air) and your sensor? (You
should have a good coupling to get reasonable results, your sensor should
not be casted, should have a low thermal mass, direct touch to the
outside). My guess is that you have several rapid changes and a stable (low
changing) temperature over a long time. Best compression is to just store
the temperature changes (time-delta and temperature-delta) and/or use a
linear approximation for the ramps. If a block of data is full, store
start-temperature, start-time for the following deltas (you can then
restore partial data if there is any fault).
M.
Peter Grey :
> At 11:24 AM 20/04/2009, you wrote:
>
> I can get some data in a day or 2. I will set up a sensor and get a
> full days worth. It will not alter that much as the sensor will sit
> on my lab bench.
>
> I'll come back then with it.
>
> Cheers
> Peter
>>On Mon, 20 Apr 2009 11:03:42 +0800, you wrote:
>>
>> >At 09:19 AM 20/04/2009, you wrote:
>> >I took a look at Numerical Recipes. It looks a great resource, but I
>> >take your point on my data. I do not have any data available yet.
>>
>>Can you get some? I mean, can you produce some simulation of the data
>>using software on a PC? I often do this when testing algorithms. For
>>example, I may generate the basic data based upon theoretical, ideal
>>models and then add noise to them and perhaps other interfering
>>elements of interest and then pipe that data into the algorithm to
>>verify that it works acceptably. It's probably a good idea to take
>>some steps along this line before going out and fielding devices, with
>>all the associated work involved, only to find out that some simple
>>change would have greatly improved the length of your datasets that
>>you later retrieve.
>>
>>Sound and pressure should be able to be simulated to a reasoned degree
>>and noise is fairly easy -- there are some simple routines to generate
>>Gaussian distributions ... in fact, Numerical Recipes includes those,
>>too.
>>
>> >I
>> >am just getting the design together. Having read a little about the
>> >techniques, I realize I can restrict the range of the sensor and so
>> >use less bits for storage.
>>
>>Well, that always helps, too.
>>
>>Jon
>>
>> >
>> >Thanks
>> >
>> >Peter
>> >>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>> >>
>> >> >At 02:47 PM 19/04/2009, you wrote:
>> >> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>> >> >series with a minimum of 528 bytes of RAM in each buffer. I also
>> >> >will use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>> >> >bytes for manipulating any compression algorithms. I would require
>> >> >the data to be lossless. These loggers will go on seabirds which
>> >> >will fly, dive into water up to 10 metres, sit on the water and
>> >> >move into nesting burrows on land. (By the way I will also measure
>> >> >pressure). The data is 16 bits wide. I have considered using Delta
>> >> >values but I expect that there will be quite a variation as a bird
>> >> >is sitting in the sun and then plunges into 10 metres of water. The
>> >> >same can be said about pressure.
>> >> >Are there any references in using delta techniques? I use assembler
>> >> >in my programming.
>> >>
>> >>Numerical Recipes contains some information on compression that may
>> >>help a lot. Plenty of theory and practice is presented to get where
>> >>you need to go, I suspect. I haven't googled for it, but I'm sure
>> >>that if you did you would have a lot to weed through.
>> >>
>> >>If the data, in net effect, is hardly discernable from "random" then
>> >>you probably cannot compress it much or at all. In fact, one of the
>> >>useful definitions some people use to imagine with, about randomness,
>> >>is the idea that truly random data doesn't compress. You need to
>> >>exploit some manner of repeatability in your data for decent
>> >>compression. Often, that is just the high order bits in data that
>> >>doesn't change rapidly over it's magnitude range, as they are usually
>> >>the same or nearly the same for nearby data values. In your case,
>> >>you may find good compression at times and bad at others.
>> >>
>> >>Your comments tend to argue for something a little more carefully
>> >>crafted than just storing deltas. However, if you permit yourself a
>> >>special marker to allow you to restate the base value when you need
>> >>to (and are willing to pay that premium at times) then you may get
>> >>good compression when the data isn't changing that much and worse
>> >>when it changes a lot, but all that may be just fine.
>> >>
>> >>One thing that would really help you evaluate choices would be a
>> >>sample data set. Do you have that?
>> >>
>> >>Jon
>> >>
>> >> >>On Sun, 19 Apr 2009 11:16:43 +0800, you wrote:
>> >> >>
>> >> >> >Has anyone references to the compression of data in Flash
>> >> >> >memory? I am storing temperature data and wish to use a smaller
>> >> >> >memory or even use EEPROM.
>> >> >>
>> >> >>An appropriate reference would be hard to determine without first
>> >> >>knowing the data rates and how much data can you hold in ram
>> >> >>without posting it to flash first and just how much compression
>> >> >>you are seeking. (It's probably reasonable to assume you want
>> >> >>lossless compression, without saying.) There are lots of very
>> >> >>simple techniques that may be perfect for you, or more complex
>> >> >>ones. For example, you might consider simply storing a base
>> >> >>temperature periodically and storing only the deltas from that
>> >> >>base afterwards, until you post another base again. Or deltas
>> >> >>from the prior delta if the temperature is a continuous record not
>> >> >>likely to make large jumps. Can you provide more details?
>> >> >>
>> >> >>Jon
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
Reply by ●April 20, 20092009-04-20
This kind of data shows long periods of stable input (mostly pressure and
even temperature during bird's sleep time). I have used a read-and-count
method where you register a measured value and then count how many samples
the value remains the same. When the data changes the counter value is
stored with the new data. If the counter has few bits (you can use the
remaining bits of this 16 bit wide data) then the data must be written also
when the counter overflows. Even a 3 bit counter would allow a data
compression in a factor of 8.
Use a non-cumulative filter (take several samples and then calculate the
mean value but do not use this calculated value in the next calculation) to
clip noise peaks that could result in low count values.
Another suggestion is to check if your FLASH memory is in fact the best
option for this application. I was used to use the AT45DB161 and found the
M25P32 better and lighter (weight). It also has a 256 byte buffer instead of
528 and this is easier to keep the low power profile (store the 256 byte
data in MSP and use the FLASH only when you have to program it, keeping the
FLASH in deep power down mostly of the time). With the AT45 internal memory
it does mean you have to waste energy storing the data in the FLASH's RAM
before writing the sector.
There are bigger serial FLASH memories with very small package you may
consider using in your application.
-Augusto
-----Mensagem original-----
De: m... [mailto:m...] Em nome de
Matthias Weingart
Enviada em: segunda-feira, 20 de abril de 2009 06:53
Para: m...
Assunto: [msp430] Re: Data compression
Peter,
well, just "play sea-bird" put your sensor in water, put in the air, in the
sun, let it stay outside night/day to get test values. How good is the
thermal coupling between the media (water, air) and your sensor? (You
should have a good coupling to get reasonable results, your sensor should
not be casted, should have a low thermal mass, direct touch to the
outside). My guess is that you have several rapid changes and a stable (low
changing) temperature over a long time. Best compression is to just store
the temperature changes (time-delta and temperature-delta) and/or use a
linear approximation for the ramps. If a block of data is full, store
start-temperature, start-time for the following deltas (you can then
restore partial data if there is any fault).
M.
Peter Grey :
> At 11:24 AM 20/04/2009, you wrote:
>
> I can get some data in a day or 2. I will set up a sensor and get a
> full days worth. It will not alter that much as the sensor will sit
> on my lab bench.
>
> I'll come back then with it.
>
> Cheers
> Peter
>>On Mon, 20 Apr 2009 11:03:42 +0800, you wrote:
>>
>> >At 09:19 AM 20/04/2009, you wrote:
>> >I took a look at Numerical Recipes. It looks a great resource, but I
>> >take your point on my data. I do not have any data available yet.
>>
>>Can you get some? I mean, can you produce some simulation of the data
>>using software on a PC? I often do this when testing algorithms. For
>>example, I may generate the basic data based upon theoretical, ideal
>>models and then add noise to them and perhaps other interfering
>>elements of interest and then pipe that data into the algorithm to
>>verify that it works acceptably. It's probably a good idea to take
>>some steps along this line before going out and fielding devices, with
>>all the associated work involved, only to find out that some simple
>>change would have greatly improved the length of your datasets that
>>you later retrieve.
>>
>>Sound and pressure should be able to be simulated to a reasoned degree
>>and noise is fairly easy -- there are some simple routines to generate
>>Gaussian distributions ... in fact, Numerical Recipes includes those,
>>too.
>>
>> >I
>> >am just getting the design together. Having read a little about the
>> >techniques, I realize I can restrict the range of the sensor and so
>> >use less bits for storage.
>>
>>Well, that always helps, too.
>>
>>Jon
>>
>> >
>> >Thanks
>> >
>> >Peter
>> >>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>> >>
>> >> >At 02:47 PM 19/04/2009, you wrote:
>> >> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>> >> >series with a minimum of 528 bytes of RAM in each buffer. I also
>> >> >will use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>> >> >bytes for manipulating any compression algorithms. I would require
>> >> >the data to be lossless. These loggers will go on seabirds which
>> >> >will fly, dive into water up to 10 metres, sit on the water and
>> >> >move into nesting burrows on land. (By the way I will also measure
>> >> >pressure). The data is 16 bits wide. I have considered using Delta
>> >> >values but I expect that there will be quite a variation as a bird
>> >> >is sitting in the sun and then plunges into 10 metres of water. The
>> >> >same can be said about pressure.
>> >> >Are there any references in using delta techniques? I use assembler
>> >> >in my programming.
>> >>
even temperature during bird's sleep time). I have used a read-and-count
method where you register a measured value and then count how many samples
the value remains the same. When the data changes the counter value is
stored with the new data. If the counter has few bits (you can use the
remaining bits of this 16 bit wide data) then the data must be written also
when the counter overflows. Even a 3 bit counter would allow a data
compression in a factor of 8.
Use a non-cumulative filter (take several samples and then calculate the
mean value but do not use this calculated value in the next calculation) to
clip noise peaks that could result in low count values.
Another suggestion is to check if your FLASH memory is in fact the best
option for this application. I was used to use the AT45DB161 and found the
M25P32 better and lighter (weight). It also has a 256 byte buffer instead of
528 and this is easier to keep the low power profile (store the 256 byte
data in MSP and use the FLASH only when you have to program it, keeping the
FLASH in deep power down mostly of the time). With the AT45 internal memory
it does mean you have to waste energy storing the data in the FLASH's RAM
before writing the sector.
There are bigger serial FLASH memories with very small package you may
consider using in your application.
-Augusto
-----Mensagem original-----
De: m... [mailto:m...] Em nome de
Matthias Weingart
Enviada em: segunda-feira, 20 de abril de 2009 06:53
Para: m...
Assunto: [msp430] Re: Data compression
Peter,
well, just "play sea-bird" put your sensor in water, put in the air, in the
sun, let it stay outside night/day to get test values. How good is the
thermal coupling between the media (water, air) and your sensor? (You
should have a good coupling to get reasonable results, your sensor should
not be casted, should have a low thermal mass, direct touch to the
outside). My guess is that you have several rapid changes and a stable (low
changing) temperature over a long time. Best compression is to just store
the temperature changes (time-delta and temperature-delta) and/or use a
linear approximation for the ramps. If a block of data is full, store
start-temperature, start-time for the following deltas (you can then
restore partial data if there is any fault).
M.
Peter Grey :
> At 11:24 AM 20/04/2009, you wrote:
>
> I can get some data in a day or 2. I will set up a sensor and get a
> full days worth. It will not alter that much as the sensor will sit
> on my lab bench.
>
> I'll come back then with it.
>
> Cheers
> Peter
>>On Mon, 20 Apr 2009 11:03:42 +0800, you wrote:
>>
>> >At 09:19 AM 20/04/2009, you wrote:
>> >I took a look at Numerical Recipes. It looks a great resource, but I
>> >take your point on my data. I do not have any data available yet.
>>
>>Can you get some? I mean, can you produce some simulation of the data
>>using software on a PC? I often do this when testing algorithms. For
>>example, I may generate the basic data based upon theoretical, ideal
>>models and then add noise to them and perhaps other interfering
>>elements of interest and then pipe that data into the algorithm to
>>verify that it works acceptably. It's probably a good idea to take
>>some steps along this line before going out and fielding devices, with
>>all the associated work involved, only to find out that some simple
>>change would have greatly improved the length of your datasets that
>>you later retrieve.
>>
>>Sound and pressure should be able to be simulated to a reasoned degree
>>and noise is fairly easy -- there are some simple routines to generate
>>Gaussian distributions ... in fact, Numerical Recipes includes those,
>>too.
>>
>> >I
>> >am just getting the design together. Having read a little about the
>> >techniques, I realize I can restrict the range of the sensor and so
>> >use less bits for storage.
>>
>>Well, that always helps, too.
>>
>>Jon
>>
>> >
>> >Thanks
>> >
>> >Peter
>> >>On Mon, 20 Apr 2009 08:53:59 +0800, you wrote:
>> >>
>> >> >At 02:47 PM 19/04/2009, you wrote:
>> >> >I am storing data at 1/2 seconds maximum. I am using a AT45DB45
>> >> >series with a minimum of 528 bytes of RAM in each buffer. I also
>> >> >will use a MSP430F2132 with 512bytes of RAM. I could reserve 256
>> >> >bytes for manipulating any compression algorithms. I would require
>> >> >the data to be lossless. These loggers will go on seabirds which
>> >> >will fly, dive into water up to 10 metres, sit on the water and
>> >> >move into nesting burrows on land. (By the way I will also measure
>> >> >pressure). The data is 16 bits wide. I have considered using Delta
>> >> >values but I expect that there will be quite a variation as a bird
>> >> >is sitting in the sun and then plunges into 10 metres of water. The
>> >> >same can be said about pressure.
>> >> >Are there any references in using delta techniques? I use assembler
>> >> >in my programming.
>> >>
Reply by ●April 20, 20092009-04-20
I have used something similar to this but may provide more compression.
If you can restrict the range from 0 to 127 then you can use the most
significant bit to indicate a counter.
For example, I have temperatures of:
70 70 70 71 72 72 72 72 73 73 74
This would take 11 bytes of storage.
Using the most significant bit as a counter flag:
83 70 71 84 72 82 73 74
8 bytes store the same result. A reduction of 28%.
Note:
this only works if the temperatures do not change to quickly.
If they do vary a lot then compression will not help.
Compression algorithms have fixed overhead associated with them call ed
a dictionary. The dictionary is stored uncompressed so that the
compressed part can be recovered.
Look here for more info:
http://en.wikipedia.org/wiki/Huffman_coding
Ray
If you can restrict the range from 0 to 127 then you can use the most
significant bit to indicate a counter.
For example, I have temperatures of:
70 70 70 71 72 72 72 72 73 73 74
This would take 11 bytes of storage.
Using the most significant bit as a counter flag:
83 70 71 84 72 82 73 74
8 bytes store the same result. A reduction of 28%.
Note:
this only works if the temperatures do not change to quickly.
If they do vary a lot then compression will not help.
Compression algorithms have fixed overhead associated with them call ed
a dictionary. The dictionary is stored uncompressed so that the
compressed part can be recovered.
Look here for more info:
http://en.wikipedia.org/wiki/Huffman_coding
Ray
