Forums

I2C hanging

Started by merapcb November 1, 2011
In my application there is an EEPROM that is used to store data. The app has been hanging sporadically (can be hours or days), but consistently. I managed to catch it while on debugger and it is hanged on the line indicated. But, I don't know *why* it is hanging. Any suggestions please on how to approach/debug? Thanks.

unsigned char I2C_ByteRead(unsigned char Addressing_Mode, unsigned int Address, unsigned char *(Data_Ptr))
{
while (I2CDCTL&I2CBUSY);

I2CWriteInit();
I2CIFG &= ~ARDYIFG;
I2CTCTL |= I2CSTT;

while ((~I2CIFG)&ARDYIFG); // Application hangs here
I2CReadInit();
I2CNDAT = 1;

I2CIFG &= ~ARDYIFG;
I2CTCTL |= I2CSTT+I2CSTP;
while ((~I2CIFG)&ARDYIFG);
*Data_Ptr = I2CBuffer[0];
return (0);
}

Beginning Microcontrollers with the MSP430

The slave device state machine can get into a hang-up state if it misses one of the clocks sent to it and so will not complete the transfer. I have had a problem when interfacing an I2C device to an AVR32 using its TWI interface hardware and the processor is reset part way through the I2C transaction. The EEPROM required the manual transmission of up 9 clock pulses on the I2C lines to restart the interface.

If this is your problem then you may be operating the I2C too fast for the slave device or may have a noise problem on the SCK line. In either case it should be possible to recover the interface by adding a timeout in the while loop and then restarting the interface by manually inserting 9 cycles on the SCK line. You will then have to restart the entire transaction.

Ian

From: m... [mailto:m...] On Behalf Of merapcb
Sent: 01 November 2011 13:17
To: m...
Subject: [msp430] I2C hanging

In my application there is an EEPROM that is used to store data. The app has been hanging sporadically (can be hours or days), but consistently. I managed to catch it while on debugger and it is hanged on the line indicated. But, I don't know *why* it is hanging. Any suggestions please on how to approach/debug? Thanks.

unsigned char I2C_ByteRead(unsigned char Addressing_Mode, unsigned int Address, unsigned char *(Data_Ptr))
{
while (I2CDCTL&I2CBUSY);

I2CWriteInit();
I2CIFG &= ~ARDYIFG;
I2CTCTL |= I2CSTT;

while ((~I2CIFG)&ARDYIFG); // Application hangs here
I2CReadInit();
I2CNDAT = 1;

I2CIFG &= ~ARDYIFG;
I2CTCTL |= I2CSTT+I2CSTP;
while ((~I2CIFG)&ARDYIFG);
*Data_Ptr = I2CBuffer[0];
return (0);
}



Thanks Ian for the pointer. I definitely can't rule out noise, but would it not stand to reason that if there was noise the problem would manifest more frequently than many hours or even days?
Regarding the 9 SCL cycles to reset, is this a standard I2C thing?

Apart from this I think the challenge would/will be (if this is the case and a reset is needed) to reset the entire read transaction and in fact the entire application logic. What I mean is, if say I am reading 12 bytes and the problem occurred on any of them, I would have received only partial data but my pointers (to records in my case) would be all messed up. Meaning if say I read only 5 bytes and then it hanged and I had to recover, then effectively I would have to "roll back" the pointers and read again.... (assuming the data was still there, which is another thing....)

--- In m..., "Okey, Ian" wrote:
>
> The slave device state machine can get into a hang-up state if it misses one of the clocks sent to it and so will not complete the transfer. I have had a problem when interfacing an I2C device to an AVR32 using its TWI interface hardware and the processor is reset part way through the I2C transaction. The EEPROM required the manual transmission of up 9 clock pulses on the I2C lines to restart the interface.
>
> If this is your problem then you may be operating the I2C too fast for the slave device or may have a noise problem on the SCK line. In either case it should be possible to recover the interface by adding a timeout in the while loop and then restarting the interface by manually inserting 9 cycles on the SCK line. You will then have to restart the entire transaction.
>
> Ian
>
> From: m... [mailto:m...] On Behalf Of merapcb
> Sent: 01 November 2011 13:17
> To: m...
> Subject: [msp430] I2C hanging
>
> In my application there is an EEPROM that is used to store data. The app has been hanging sporadically (can be hours or days), but consistently. I managed to catch it while on debugger and it is hanged on the line indicated. But, I don't know *why* it is hanging. Any suggestions please on how to approach/debug? Thanks.
>
> unsigned char I2C_ByteRead(unsigned char Addressing_Mode, unsigned int Address, unsigned char *(Data_Ptr))
> {
> while (I2CDCTL&I2CBUSY);
>
> I2CWriteInit();
> I2CIFG &= ~ARDYIFG;
> I2CTCTL |= I2CSTT;
>
> while ((~I2CIFG)&ARDYIFG); // Application hangs here
> I2CReadInit();
> I2CNDAT = 1;
>
> I2CIFG &= ~ARDYIFG;
> I2CTCTL |= I2CSTT+I2CSTP;
> while ((~I2CIFG)&ARDYIFG);
> *Data_Ptr = I2CBuffer[0];
> return (0);
> }
>
>
>

merapcb wrote:
> Thanks Ian for the pointer. I definitely can't rule out noise, but
> would it not stand to reason that if there was noise the problem
> would manifest more frequently than many hours or even days?
> Regarding the 9 SCL cycles to reset, is this a standard I2C thing?
>

No, but I2C is prone to hanging with some devices. It's like the starter
on some older Vauxhall or Opel cars where sometimes you had to whack it
with a hammer to make it turn over the engine :-)

If you have the choice, SPI is usually better.
> Apart from this I think the challenge would/will be (if this is the
> case and a reset is needed) to reset the entire read transaction and
> in fact the entire application logic. What I mean is, if say I am
> reading 12 bytes and the problem occurred on any of them, I would
> have received only partial data but my pointers (to records in my
> case) would be all messed up. Meaning if say I read only 5 bytes and
> then it hanged and I had to recover, then effectively I would have to
> "roll back" the pointers and read again.... (assuming the data was
> still there, which is another thing....)
>

It is IMHO always prudent to roll back because what has been read so far
is suspect data. At least I'd see it that way.

Provided you didn't exceed any speed limits it is not normal that the
I2C bus hangs up in the middle of a transmission. Maybe there is
excessive ringing? I'd check that at both ends.

--
Regards, Joerg

http://www.analogconsultants.com/

--- In m..., Joerg wrote:
>
> If you have the choice, SPI is usually better.

I wish I could, but not at this point.... I have an EEPROM and RTC on I2C and that's what I must use for now, until I can revise the board.
> Provided you didn't exceed any speed limits it is not normal that the
> I2C bus hangs up in the middle of a transmission. Maybe there is
> excessive ringing? I'd check that at both ends.

I can definitely put a scope on it, but the problem is that the unit fails in the field and I am not there 99% of the time. After a failure, it is power cycled and can work for days, including thousands of read/write cycles to the EEPROM, and then hangs. So what I mean is that I can put a scope to it but I can't be with it all the time, and mostly that's when the problem happens (that's Murphy for you all right!).

I've used DS1307 with an 24C64 EEPROM and sometimes the RTC hang-up due a
noise in the battery voltage. The DS1307 disable the
I2C when VCC is small then 1.25xVbat. Try putting an 100nF ceramic capacitor
in parallel to the battery, it solve my problem.

--Paulo

-----Mensagem original-----
De: m... [mailto:m...] Em nome de
merapcb
Enviada em: ter-feira, 1 de novembro de 2011 14:55
Para: m...
Assunto: [msp430] Re: I2C hanging

--- In m..., Joerg wrote:
>
> If you have the choice, SPI is usually better.

I wish I could, but not at this point.... I have an EEPROM and RTC on I2C
and that's what I must use for now, until I can revise the board.
> Provided you didn't exceed any speed limits it is not normal that the
> I2C bus hangs up in the middle of a transmission. Maybe there is
> excessive ringing? I'd check that at both ends.

I can definitely put a scope on it, but the problem is that the unit fails
in the field and I am not there 99% of the time. After a failure, it is
power cycled and can work for days, including thousands of read/write cycles
to the EEPROM, and then hangs. So what I mean is that I can put a scope to
it but I can't be with it all the time, and mostly that's when the problem
happens (that's Murphy for you all right!).

merapcb wrote:
>
> --- In m..., Joerg wrote:
>> If you have the choice, SPI is usually better.
>
> I wish I could, but not at this point.... I have an EEPROM and RTC on
> I2C and that's what I must use for now, until I can revise the board.
>

Sometimes man's got to do what man's got to do :-)

>
>> Provided you didn't exceed any speed limits it is not normal that
>> the I2C bus hangs up in the middle of a transmission. Maybe there
>> is excessive ringing? I'd check that at both ends.
>
> I can definitely put a scope on it, but the problem is that the unit
> fails in the field and I am not there 99% of the time. After a
> failure, it is power cycled and can work for days, including
> thousands of read/write cycles to the EEPROM, and then hangs. So what
> I mean is that I can put a scope to it but I can't be with it all the
> time, and mostly that's when the problem happens (that's Murphy for
> you all right!).
>

So I assume it never happens with a test unit in your own lab. Could it
be some big noise spikes that only occurs at some of the field
locations? Any correlation with nearby thunderstorms? I have seen
electronics be influenced by lightning that was many miles away.

As Paulo wrote, make sure the decoupling is ok and very close by.

--
Regards, Joerg

http://www.analogconsultants.com/

merapcb, I feel your pain!

In my experience, the upfront cost of making the robust software is worthwhile. If your hang-ups are occurring say once an hour, this is a problem. If you were to implement noise suppression, extra clocks, whatever..., you might get the hang-up once a week. That's a bigger problem! You'll have customer complaints, but can't reproduce them.

So, I think adding some software robustness to your code may be worthwhile. I suspect it's not quite as bad as it seems. Also in my experience, when adding more robust software and debugging it, I have stumbled on the root cause of the problem.

Good luck. Please report back your solution.

Stuart

--- In m..., "merapcb" wrote:
> Thanks Ian for the pointer. I definitely can't rule out noise, but would it not stand to reason that if there was noise the problem would manifest more frequently than many hours or even days?
> Regarding the 9 SCL cycles to reset, is this a standard I2C thing?
>
> Apart from this I think the challenge would/will be (if this is the case and a reset is needed) to reset the entire read transaction and in fact the entire application logic. What I mean is, if say I am reading 12 bytes and the problem occurred on any of them, I would have received only partial data but my pointers (to records in my case) would be all messed up. Meaning if say I read only 5 bytes and then it hanged and I had to recover, then effectively I would have to "roll back" the pointers and read again.... (assuming the data was still there, which is another thing....)
>
> --- In m..., "Okey, Ian" wrote:
> >
> > The slave device state machine can get into a hang-up state if it misses one of the clocks sent to it and so will not complete the transfer. I have had a problem when interfacing an I2C device to an AVR32 using its TWI interface hardware and the processor is reset part way through the I2C transaction. The EEPROM required the manual transmission of up 9 clock pulses on the I2C lines to restart the interface.
> >
> > If this is your problem then you may be operating the I2C too fast for the slave device or may have a noise problem on the SCK line. In either case it should be possible to recover the interface by adding a timeout in the while loop and then restarting the interface by manually inserting 9 cycles on the SCK line. You will then have to restart the entire transaction.
> >
> > Ian
> >
> > From: m... [mailto:m...] On Behalf Of merapcb
> > Sent: 01 November 2011 13:17
> > To: m...
> > Subject: [msp430] I2C hanging
> >
> >
> >
> > In my application there is an EEPROM that is used to store data. The app has been hanging sporadically (can be hours or days), but consistently. I managed to catch it while on debugger and it is hanged on the line indicated. But, I don't know *why* it is hanging. Any suggestions please on how to approach/debug? Thanks.
> >
> > unsigned char I2C_ByteRead(unsigned char Addressing_Mode, unsigned int Address, unsigned char *(Data_Ptr))
> > {
> > while (I2CDCTL&I2CBUSY);
> >
> > I2CWriteInit();
> > I2CIFG &= ~ARDYIFG;
> > I2CTCTL |= I2CSTT;
> >
> > while ((~I2CIFG)&ARDYIFG); // Application hangs here
> > I2CReadInit();
> > I2CNDAT = 1;
> >
> > I2CIFG &= ~ARDYIFG;
> > I2CTCTL |= I2CSTT+I2CSTP;
> > while ((~I2CIFG)&ARDYIFG);
> > *Data_Ptr = I2CBuffer[0];
> > return (0);
> > }
> >
> >
> >
> >
>

Hi Stuart and thank you. Yes, it is "painful" (actually without the quotes), especially when an advanced proto is in the field and this crops up. 99% is not 100%.

As for a solution, the first thing I will try, as a work around, is to implement a timer in the while statement for reading the byte and if the bus hangs, that will get me out of there. But then the real issue starts, which is to recover the state of the application (data pointers etc) and since this function is called almost in 100 places in the application, I suspect much pain....
--- In m..., "Stuart_Rubin" wrote:
>
> merapcb, I feel your pain!
>
> In my experience, the upfront cost of making the robust software is worthwhile. If your hang-ups are occurring say once an hour, this is a problem. If you were to implement noise suppression, extra clocks, whatever..., you might get the hang-up once a week. That's a bigger problem! You'll have customer complaints, but can't reproduce them.
>
> So, I think adding some software robustness to your code may be worthwhile. I suspect it's not quite as bad as it seems. Also in my experience, when adding more robust software and debugging it, I have stumbled on the root cause of the problem.
>
> Good luck. Please report back your solution.
>
> Stuart
>

Have you checked the errata-sheet of your controller? The I2C had many bugs -
I changed to my own bit banging routines after a similar nightmare. (btw. the
code of the software-I2C was smaller than the code using the I2C hardware
;-).

M.

"merapcb" :

>
> Hi Stuart and thank you. Yes, it is "painful" (actually without the
> quotes), especially when an advanced proto is in the field and this
> crops up. 99% is not 100%.
>
> As for a solution, the first thing I will try, as a work around, is to
> implement a timer in the while statement for reading the byte and if the
> bus hangs, that will get me out of there. But then the real issue
> starts, which is to recover the state of the application (data pointers
> etc) and since this function is called almost in 100 places in the
> application, I suspect much pain....
> --- In m..., "Stuart_Rubin" wrote:
>>
>> merapcb, I feel your pain!
>>
>> In my experience, the upfront cost of making the robust software is
>> worthwhile. If your hang-ups are occurring say once an hour, this is a
>> problem. If you were to implement noise suppression, extra clocks,
>> whatever..., you might get the hang-up once a week. That's a bigger
>> problem! You'll have customer complaints, but can't reproduce them.
>>
>> So, I think adding some software robustness to your code may be
>> worthwhile. I suspect it's not quite as bad as it seems. Also in my
>> experience, when adding more robust software and debugging it, I have
>> stumbled on the root cause of the problem.
>>
>> Good luck. Please report back your solution.
>>
>> Stuart