EmbeddedRelated.com
Forums

High speed data logger

Started by Chuck Bass November 20, 2005
I'm building a system using a 3300 and we are trying to log as much data as we can before we send it out for processing.   I've found the following:
 
1) Serial flash is too slow (16 msec delay for a write)
2) Using only internal ram gives too few points.
3) Using xmem is better, but I don't seem to be able to access very much (60K) before I get corruption (I haven't figured this out yet...)
4) Streaming to the ethernet is pretty good but I still end up missing data points at the data rates I'm looking for (>2kHz).
 
Is there a way with DC to get after the extra 512K of RAM?  Is serial flash really that slow?  60k/sec...  If I timesliced/ucos could I overcome the 16msec delay that I assume is spent waiting for the write to finish.
 
Any help on this would be appreciated
 
Chuck

r...@yahoogroups.com wrote:
------------------------ Yahoo! Groups Sponsor --------------------~-->
Get fast access to your favorite Yahoo! Groups. Make Yahoo! your home page
http://us.click.yahoo.com/dpRU5A/wUILAA/yQLSAA/dN_tlB/TM
--------------------------------~->

There are 2 messages in this issue.

Topics in this digest:

1. Re: Which modules are people using these days?
From: "nerdx86"
2. How to program serial communication port from/to RCM3000
From: "axn_xp112" ________________________________________________________________________
________________________________________________________________________

Message: 1
Date: Sat, 19 Nov 2005 20:30:08 -0000
From: "nerdx86"
Subject: Re: Which modules are people using these days?

I agree with Scott Henion.

My $0.02:
I think the original 3000 & 3100 are very powerfull and widely used..
Should be around for a long time, and I hope that a Rabbit 4000 is
going put into the same footprint...

The RCM3300 adds a few extras into a almost completely compatible
footprint..
________________________________________________________________________
________________________________________________________________________

Message: 2
Date: Sun, 20 Nov 2005 16:32:45 -0000
From: "axn_xp112"
Subject: How to program serial communication port from/to RCM3000

Hi,

I'm researcher from Malaysia. I'm a a first timer person in using
RCM 2000/3000. Could someone help me on programming data transmit &
receiving via serial communication from or to RCM 2000/3000?

Another thing, could someone help me on getting the source (website,
books etc.) neat/full example on programming and setting RCM 2000/3000
I/O port using the Dynamic C?

Thank you.

Addie________________________________________________________________________
________________________________________________________________________

Yahoo! Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/rabbit-semi/

<*> To unsubscribe from this group, send an email to:
r...@yahoogroups.com

<*


Yahoo! FareChase - Search multiple travel sites in one click.

Why not log to SRAM (root or xmem) then periodically, in a lower priority task, write the data to serial flash?

 

Regards,

Nathan

-----Original Message-----
From: r...@yahoogroups.com [mailto:r...@yahoogroups.com] On Behalf Of Chuck Bass
Sent:
Monday, 21 November 2005 10:24 AM
To: r...@yahoogroups.com
Subject: [rabbit-semi] High speed data logger

 

I'm building a system using a 3300 and we are trying to log as much data as we can before we send it out for processing.   I've found the following:

 

1) Serial flash is too slow (16 msec delay for a write)

2) Using only internal ram gives too few points.

3) Using xmem is better, but I don't seem to be able to access very much (60K) before I get corruption (I haven't figured this out yet...)

4) Streaming to the ethernet is pretty good but I still end up missing data points at the data rates I'm looking for (>2kHz).

 

Is there a way with DC to get after the extra 512K of RAM?  Is serial flash really that slow?  60k/sec...  If I timesliced/ucos could I overcome the 16msec delay that I assume is spent waiting for the write to finish.

 

Any help on this would be appreciated

 

Chuck

r...@yahoogroups.com wrote:

------------------------ Yahoo! Groups Sponsor --------------------~-->
Get fast access to your favorite Yahoo! Groups. Make Yahoo! your home page
http://us.click.yahoo.com/dpRU5A/wUILAA/yQLSAA/dN_tlB/TM
--------------------------------~->

There are 2 messages in this issue.

Topics in this digest:

1. Re: Which modules are people using these days?
From: "nerdx86"
2. How to program serial communication port from/to RCM3000
From: "axn_xp112"


________________________________________________________________________
________________________________________________________________________

Message: 1
Date: Sat, 19 Nov 2005 20:30:08 -0000
From: "nerdx86"
Subject: Re: Which modules are people using these days?

I agree with Scott Henion.

My $0.02:
I think the original 3000 & 3100 are very powerfull and widely used..
Should be around for a long time, and I hope that a Rabbit 4000 is
going put into the same footprint...

The RCM3300 adds a few extras into a almost completely compatible
footprint..
________________________________________________________________________
________________________________________________________________________

Message: 2
Date: Sun, 20 Nov 2005 16:32:45 -0000
From: "axn_xp112"
Subject: How to program serial communication port from/to RCM3000

Hi,

I'm researcher from Malaysia. I'm a a first timer person in using
RCM 2000/3000. Could someone help me on programming data transmit &
receiving via serial communication from or to RCM 2000/3000?

Another thing, could someone help me on getting the source (website,
books etc.) neat/full example on programming and setting RCM 2000/3000
I/O port using the Dynamic C?

Thank you.

Addie ________________________________________________________________________
________________________________________________________________________

Yahoo! Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/rabbit-semi/

<*> To unsubscribe from this group, send an email to:
r...@yahoogroups.com

<*


 

Yahoo! FareChase - Search multiple travel sites in one click.



I have some interest in this post, I am going to be reading in data
from the serial port and want to store this to memory (root to xmem).
But how do you take advantage of the full amount of memory? and also,
is there a way to read back a chunk of memory? lets say web based you
want to view some memory that has been stored, is this possible? I
havent found anything to help me with this.

--- In rabbit-semi@rabb..., "Nathan Johnston"
<nathan.johnston@o...> wrote:
>
> Why not log to SRAM (root or xmem) then periodically, in a lower
> priority task, write the data to serial flash?
>
> Regards,
> Nathan
> -----Original Message-----
> From: rabbit-semi@rabb... [mailto:rabbit-semi@rabb...]
> On Behalf Of Chuck Bass
> Sent: Monday, 21 November 2005 10:24 AM
> To: rabbit-semi@rabb...
> Subject: [rabbit-semi] High speed data logger
>
> I'm building a system using a 3300 and we are trying to log as much data
> as we can before we send it out for processing. I've found the
> following:
>
> 1) Serial flash is too slow (16 msec delay for a write)
> 2) Using only internal ram gives too few points.
> 3) Using xmem is better, but I don't seem to be able to access very much
> (60K) before I get corruption (I haven't figured this out yet...)
> 4) Streaming to the ethernet is pretty good but I still end up missing
> data points at the data rates I'm looking for (>2kHz).
>
> Is there a way with DC to get after the extra 512K of RAM? Is serial
> flash really that slow? 60k/sec... If I timesliced/ucos could I
> overcome the 16msec delay that I assume is spent waiting for the write
> to finish.
>
> Any help on this would be appreciated
>
> Chuck
>
> rabbit-semi@rabb... wrote:
> ------------------------ Yahoo! Groups Sponsor
>
> There are 2 messages in this issue.
>
> Topics in this digest:
>
> 1. Re: Which modules are people using these days?
> From: "nerdx86"
> 2. How to program serial communication port from/to RCM3000
> From: "axn_xp112" >
> ________________________________________________________________________
>
> ________________________________________________________________________
>
> Message: 1
> Date: Sat, 19 Nov 2005 20:30:08 -0000
> From: "nerdx86"
> Subject: Re: Which modules are people using these days?
>
> I agree with Scott Henion.
>
> My $0.02:
> I think the original 3000 & 3100 are very powerfull and widely
> used..
> Should be around for a long time, and I hope that a Rabbit 4000
> is
> going put into the same footprint...
>
> The RCM3300 adds a few extras into a almost completely
> compatible
> footprint.. >
>
> ________________________________________________________________________
>
> ________________________________________________________________________
>
> Message: 2
> Date: Sun, 20 Nov 2005 16:32:45 -0000
> From: "axn_xp112"
> Subject: How to program serial communication port from/to
> RCM3000
>
> Hi,
>
> I'm researcher from Malaysia. I'm a a first timer person in
> using
> RCM 2000/3000. Could someone help me on programming data
> transmit &
> receiving via serial communication from or to RCM 2000/3000?
>
> Another thing, could someone help me on getting the source
> (website,
> books etc.) neat/full example on programming and setting RCM
> 2000/3000
> I/O port using the Dynamic C?
>
> Thank you.
>
> Addie >
> ________________________________________________________________________
>
> ________________________________________________________________________ > > .
>
> ________________________________
>




You need to use xmem to take use all of the memory. Refer to root2xmem
and xmem2root functions.

Regards,
Nathan

-----Original Message-----
From: rabbit-semi@rabb... [mailto:rabbit-semi@rabb...]
On Behalf Of james_r_hebert
Sent: Monday, 21 November 2005 11:07 AM
To: rabbit-semi@rabb...
Subject: [rabbit-semi] Re: High speed data logger

I have some interest in this post, I am going to be reading in data
from the serial port and want to store this to memory (root to xmem).
But how do you take advantage of the full amount of memory? and also,
is there a way to read back a chunk of memory? lets say web based you
want to view some memory that has been stored, is this possible? I
havent found anything to help me with this.

--- In rabbit-semi@rabb..., "Nathan Johnston"
<nathan.johnston@o...> wrote:
>
> Why not log to SRAM (root or xmem) then periodically, in a lower
> priority task, write the data to serial flash?
>
> Regards,
> Nathan
> -----Original Message-----
> From: rabbit-semi@rabb... [mailto:rabbit-semi@rabb...]
> On Behalf Of Chuck Bass
> Sent: Monday, 21 November 2005 10:24 AM
> To: rabbit-semi@rabb...
> Subject: [rabbit-semi] High speed data logger
>
> I'm building a system using a 3300 and we are trying to log as much
data
> as we can before we send it out for processing. I've found the
> following:
>
> 1) Serial flash is too slow (16 msec delay for a write)
> 2) Using only internal ram gives too few points.
> 3) Using xmem is better, but I don't seem to be able to access very
much
> (60K) before I get corruption (I haven't figured this out yet...)
> 4) Streaming to the ethernet is pretty good but I still end up missing
> data points at the data rates I'm looking for (>2kHz).
>
> Is there a way with DC to get after the extra 512K of RAM? Is serial
> flash really that slow? 60k/sec... If I timesliced/ucos could I
> overcome the 16msec delay that I assume is spent waiting for the write
> to finish.
>
> Any help on this would be appreciated
>
> Chuck
>
> rabbit-semi@rabb... wrote:
> ------------------------ Yahoo! Groups Sponsor
>
> There are 2 messages in this issue.
>
> Topics in this digest:
>
> 1. Re: Which modules are people using these days?
> From: "nerdx86"
> 2. How to program serial communication port from/to RCM3000
> From: "axn_xp112" >
>
________________________________________________________________________
>
>
________________________________________________________________________
>
> Message: 1
> Date: Sat, 19 Nov 2005 20:30:08 -0000
> From: "nerdx86"
> Subject: Re: Which modules are people using these days?
>
> I agree with Scott Henion.
>
> My $0.02:
> I think the original 3000 & 3100 are very powerfull and widely
> used..
> Should be around for a long time, and I hope that a Rabbit 4000
> is
> going put into the same footprint...
>
> The RCM3300 adds a few extras into a almost completely
> compatible
> footprint..
________________________________________________________________________
>
>
________________________________________________________________________
>
> Message: 2
> Date: Sun, 20 Nov 2005 16:32:45 -0000
> From: "axn_xp112"
> Subject: How to program serial communication port from/to
> RCM3000
>
> Hi,
>
> I'm researcher from Malaysia. I'm a a first timer person in
> using
> RCM 2000/3000. Could someone help me on programming data
> transmit &
> receiving via serial communication from or to RCM 2000/3000?
>
> Another thing, could someone help me on getting the source
> (website,
> books etc.) neat/full example on programming and setting RCM
> 2000/3000
> I/O port using the Dynamic C?
>
> Thank you.
>
> Addie >
>
________________________________________________________________________
>
>
________________________________________________________________________
>
>

> > .
>
> ________________________________
>


Yahoo! Groups Links



At 06:23 PM 11/20/2005, you wrote:

>I'm building a system using a 3300 and we are trying to log as much
>data as we can before we send it out for processing. I've found
>the following:
>
>1) Serial flash is too slow (16 msec delay for a write)

Yes it can be slow. I get about 30kbytes/sec write speeds through fat.

>2) Using only internal ram gives too few points.
>3) Using xmem is better, but I don't seem to be able to access very
>much (60K) before I get corruption (I haven't figured this out yet...)

I use about 80k on an lowly old 2200 (128k RAM) and works with no
problems. May be how you are using the xmem pointer allocated by
xalloc(). I guess calling it a pointer is a bad term. It is a mem
address and can't be used as a C pointer. Just make sure you are
using longs in the address calculations. 16-bit values will wrap
negative. DC is bad about this as it will not give a warning.

>4) Streaming to the ethernet is pretty good but I still end up
>missing data points at the data rates I'm looking for (>2kHz).

You will have to have incoming data interrupt driven to a large
buffer or multiple buffers. I prefer two buffers. Have data ISR fill
one then switch to the other. A main task waits for the switch and
sends the old buffer. Can also be done with a circular buffer. The
mentioned 2200 used a circular buffer to store data when the local
server was down.

>Is there a way with DC to get after the extra 512K of RAM?

Only with root2xmem() and cmem2root() for blocks. There is also
xsetint()/xgetint() (and long versions). On the 3300, you need to
compile to flash, run in RAM. Then all mem is RAM; although DC fragments it.

xavail() will tell you how much RAM you have. On the 3300 you should
have 256k or more.

Of course Softools could address this mem directly. ;)

> Is serial flash really that slow? 60k/sec... If I
> timesliced/ucos could I overcome the 16msec delay that I assume is
> spent waiting for the write to finish.

If the input side is interrupt driven to a buffer, then the delay
should not be a problem.

<Scott ------------
| Scott G. Henion | shenion@shen... |
| Consultant | SHDesigns |
------------
Rabbit Libs: http://www.shdesigns.org/rabbit/
--



I have been looking at that example, but where i get lost is, they use

physaddr = xalloc(1024); // physical memory address (SRAM)

is this where they start writing to memory? or is this the amount of
memory they are allocating for memory? I understand how to use
xmem2root and root2xmem and have successfully write and array and read
it back from memory. just dont know how to take advantage of that.

also i would like to view what is in memory via a web browser.
so...how would i be able to take a chunk of the memory and view it?

--- In rabbit-semi@rabb..., "Nathan Johnston"
<nathan.johnston@o...> wrote:
>
> You need to use xmem to take use all of the memory. Refer to root2xmem
> and xmem2root functions.
>
> Regards,
> Nathan
>
> -----Original Message-----
> From: rabbit-semi@rabb... [mailto:rabbit-semi@rabb...]
> On Behalf Of james_r_hebert
> Sent: Monday, 21 November 2005 11:07 AM
> To: rabbit-semi@rabb...
> Subject: [rabbit-semi] Re: High speed data logger
>
> I have some interest in this post, I am going to be reading in data
> from the serial port and want to store this to memory (root to xmem).
> But how do you take advantage of the full amount of memory? and also,
> is there a way to read back a chunk of memory? lets say web based you
> want to view some memory that has been stored, is this possible? I
> havent found anything to help me with this.
>
> --- In rabbit-semi@rabb..., "Nathan Johnston"
> <nathan.johnston@o...> wrote:
> >
> > Why not log to SRAM (root or xmem) then periodically, in a lower
> > priority task, write the data to serial flash?
> >
> > Regards,
> > Nathan




At 08:13 PM 11/20/2005, you wrote:

>I have been looking at that example, but where i get lost is, they use
>
>physaddr = xalloc(1024); // physical memory address (SRAM)

The parameter to xalloc() is th number of bytes to allocate.

bytes_avail=xavail(NULL);

That will let you know how much is available. I usually allow enough
for the ethernet stack and allocate the rest.

I wrote a tech-tip on using xmem in DC in a way closer to ANSI C:
http://shdesigns.org/tips/xmem.html

<Scott ------
| Scott G. Henion| shenion@shen... |
| Consultant | Stone Mountain, GA |
| SHDesigns | PGP Key 0xE98DDC48 |
| http://www.shdesigns.org |
------
today's fortune
"Nuclear war can ruin your whole compile."
-- Karl Lehenbauer --




> physaddr = xalloc(1024); // physical memory address (SRAM)
>
> is this where they start writing to memory? or is this the amount of
> memory they are allocating for memory?

It's the amount.

> I understand how to use
> xmem2root and root2xmem and have successfully write and array and read
> it back from memory. just dont know how to take advantage of that.

The way to take advantage of that is that you make your "real" arrays
(which reside in root data) smaller and only xmem2root the section that
you need. So, rather than making an array LogEntries[1000], you could
make an array LogEntries[10] and copy chunks of 10 entries from xmem
when processing them.

To view the data from the web, you'd have to make a cgi function that
copies a chunk from xmem to the http buffer on each tick, and waits
until the buffer is flushed (cgi functions can span multiple ticks).
That is, since I'm on Dynamic C 8, I'm not sure if the http handler of
Dynamic C 9 can be told to send out data from xmem directly; if it can
it will be even easier. If not, you can still use the xmem tcp functions
to send the data directly to the http socket, but I think it's better
not to bypass the http handler's own workings too much. :o)

Maurits.

--

HiTECHnologies Industrial Automation B.V.
Industrieweg 30, 3401 MA, IJsselstein, The Netherlands
P.O. box 242, 3400 AE, IJsselstein, The Netherlands
Tel: +31 30 6875335
Fax: +31 30 6875333
E-mail: info@info...
Web: www.hitechnologies.nl

===================================================================
The information contained in this communication is confidential and may
be legally privileged. It is intended solely for the use of the
individual or entity to whom it is addressed and others authorized to
receive it. If you are not the intended recipient you are hereby
notified that any disclosure, copying, distribution or taking any action
in reliance on the contents of this information is strictly prohibited
and may be unlawful.
HiTECHnologies B.V. is neither liable for the contents, nor for the
proper, complete and timely transmission of the information contained in
this communication.
===================================================================