EmbeddedRelated.com
Forums

Downloading a large log file to a PC

Started by diosper2 November 21, 2007
Hello:

We have a logger using MSP430F1612, which is connected via an RS-232
line to the PC controlling it.

The logger periodically wakes up, makes a measurements and writes the
data to the external flash of the PC.

After a while, it is time to transfer the data accumulated in the
flash to the PC, in order to generate a local log file.

Now, the external flash is pretty large, 64MB or so.

We're concerned about the errors introduced in the transmission
process as well in its duration, and naturally in the power consumption.

In your experience:

- did you go for a single transfer of tens of MBs, say with a single
CRC; seems very risky;

- did you go for several smaller transfers (and then which size did
you consider for the pieces?), each checked individually, and then
assembled, on the PC side;

- if in pieces, did you go for a CRC for each individual piece (we
guess so);

- did you implement a resend/retry strategy, based on the failures
occuring during transmission?

- what other precautions should one introduce to be on the safe side?

- what baud rates can we reliably employ during this download? Of
course, we would like to perform the transfer ASAP, but then the power
consumption may be higher per time unit.

Any pointers would be appreciated.

Jerry.

Beginning Microcontrollers with the MSP430

Yes, XMODEM isn't all that great, with its simple, 8-bit checksum, but it is very easy to code and does give some protection and handles retries. But XMODEM-1K or XMODEM-CRC use the much better CRC (a 16-bit CRC) for error detection. It's the usual trade-off between time/code space versus better protection. Our company had used the simple XMODEM for years with few problems but we used short RS-232 connections at 9600. We were using underpowered processors that made generating a CRC too time consuming. In later years we switched to XMODEM-1K. Kermit is nice, too, with its varying sized blocks. There is lots of code available for all of those protocols.

Jim

----- Original Message ----
> From: microbit_virgin
> To: m...
> Sent: Tuesday, November 20, 2007 9:39:20 PM
> Subject: RE: [msp430] Downloading a large log file to a PC
>
> I always thought Xmodem wasn't all that reliable ?
> Zmodem is complicated to implement, but has a nice streaming
> topology
>
compared to (what I think
> is) the inferior XModem ?
>
> Best Regards,
> Kris
>
>

____________________________________________________________________________________
Get easy, one-click access to your favorites.
Make Yahoo! your homepage.
http://www.yahoo.com/r/hs
--- In m..., Kipton Moravec wrote:
> I have lost bytes dumping data to a PC at 9600 baud, and at 115.2K baud.

First, thank you all for the very many pointers.

Now, is 115.Kbaud realy achievable with MSP430 (MSP430F1612 in our
case for now)?

Do you people feel that the mentioned apps, such as xmodem-crc, offer
everything required in terms of handshaking and retries in order to
provide a safe connectivity? Or should we put something else above
them in order to improve on that?

Thanks again.

Jerry.
I use 460.8 kb/s with F135.

--- In m..., "diosper2" wrote:
>
> --- In m..., Kipton Moravec wrote:
> >
> >
> > I have lost bytes dumping data to a PC at 9600 baud, and at 115.2K
baud.
>
> First, thank you all for the very many pointers.
>
> Now, is 115.Kbaud realy achievable with MSP430 (MSP430F1612 in our
> case for now)?
>
> Do you people feel that the mentioned apps, such as xmodem-crc, offer
> everything required in terms of handshaking and retries in order to
> provide a safe connectivity? Or should we put something else above
> them in order to improve on that?
>
> Thanks again.
>
> Jerry.
>
--- In m..., "old_cow_yellow"
wrote:
>
> I use 460.8 kb/s with F135.

That's speed:-)

Is this a custom comm app (which you designed) or one of those
mentioned here, such as kermit, xmodem-crc, etc?

Thanks again.

Jerry
I use a Prolific USB to virtual COM port chip, fixed block length and
simple checksum, and ACK/NACK response. Real simple and stupid.

--- In m..., "diosper2" wrote:
>
> --- In m..., "old_cow_yellow"
> wrote:
> >
> > I use 460.8 kb/s with F135.
>
> That's speed:-)
>
> Is this a custom comm app (which you designed) or one of those
> mentioned here, such as kermit, xmodem-crc, etc?
>
> Thanks again.
>
> Jerry
>
On Wed, 2007-11-21 at 14:52 +0000, diosper2 wrote:
> --- In m..., Kipton Moravec wrote:
> >
> >
> > I have lost bytes dumping data to a PC at 9600 baud, and at 115.2K baud.
>
> First, thank you all for the very many pointers.
>
> Now, is 115.Kbaud realy achievable with MSP430 (MSP430F1612 in our
> case for now)?
>

I have not tried 115.2K baud with the MSP430 with the internal clock. I
have only done it with an external crystal. It will work with a 1.8432
MHz, 3.6864 MHz, or 7.3728 MHz crystals.

In all cases you have to worry about the OS doing something in the
background to screw you up. I have used serial to USB converters with
Windows and Linux and still lost data on occasion when I did not use
hand shaking. I was just logging data to see what was going on inside
the processor for debugging, and did not want a bunch of overhead to
slow it down. The PC seems to work better with bursty data, or with
pauses in the data.
> Do you people feel that the mentioned apps, such as xmodem-crc, offer
> everything required in terms of handshaking and retries in order to
> provide a safe connectivity? Or should we put something else above
> them in order to improve on that?
>

That is what they are made for. Back before the Internet people logged
into BBS systems with a phone modem and used these protocols to
successfully transfer data over a noisier line than your RS-232 wire.

Kip

--
Kipton Moravec KE5NGX
"Always do right; this will gratify some people and astonish the rest."
--Mark Twain