[TriEmbed] Update on my Stability Issue

[John Vaughters]

Chip,
Tough problem, Thanks for the feedback. I have been curious if you solved it. Interesting specifics on the boards.
So you have a watchdog workaround which is a common fix for tough issues like this. The actual lock up could still be a tough one to solve, I wish you luck on that task. Could be power and/or timing issues from wake up.
John Vaughters 

    On Tuesday, March 14, 2017 10:02 PM, Chip McClelland via TriEmbed <triembed at triembed.org> wrote:
 

 All, 
For those who missed it, I sent a note about a month ago asking for advice on improving the reliability of my sensors.  These sensors would go weeks and measure tens of thousands of cars and then lock up.  I was struggling to fix a problem I could not easily reproduce so I asked this group for ideas.   
Thank you for all the great advice it has been very helpful.  I think I have made significant progress in my testing and, this week, I have deployed the updates to the sensors in the park.  Given that many of you took the time to give me your thoughts, I wanted to share what I have learned.
 1) A few of you suggested making the problem worse to help improve the reproducibility of the issue - that was key.  I was eventually able to get the issue to surface in under ten minutes which helped me test potential fixes. 2) My problem seemed to be related to the Arduino’s ability to wake from sleep and reliably access data on the i2c bus.  I was able to find the specific function and eventually the  i2c command - t = RTC.get(); that would lock up the i2c bus and the Arduino.  The incidence of this issue was increased significantly when the Arduino was recently brought back from sleep. 3) The solution was to start a watchdog timer prior to using this command and set up the Arduino to reboot if it did not complete in a reasonable period.  Worked fine on the Uno but this locked up the Arduino Pro Mini.  It seemed that the Pro mini does not use an Optiboot boot loader like the Uno and there is a known bug that prevents the watchdog timer from functioning correctly.  I found a new boot loader - MiniCore that fixed the issue and delivered on some cool new features. 4) The problem was that I have a number of boards in production which do not have ICSP headers.  I was able to find a great adapter  that let me refresh the chips in place - highly recommended.    5) Changing the boot loader also changed my tool chain and I ended up asking the embedXcode folks for help and they delivered a new release that let me integrate the new boot loader with the board definition files.  If you are a Mac user and want a better IDE, I can highly recommend embedXcode.  With these changes and some improvements to the process of waking from sleep from Nick Gammon and the Arduino forums, I was able to put the device through significanttesting without any lock ups.  I added a “reboot” counter to EEPROM so I can see if more changes are needed in the future.  I just finished updating all the sensors in the park and will see how the real world reacts to my efforts. 
I hope there are some ideas in here that might be helpful.  Please let me know if you have more suggestions.  I will also be looking at my power and signal integrity on the boards but, for now, I think I have a solution.  If you would like to see what the code looks like, you can find it here.  Thanks again for all your suggestions.
Chip



_______________________________________________
Triangle, NC Embedded Computing mailing list
TriEmbed at triembed.org
http://mail.triembed.org/mailman/listinfo/triembed_triembed.org
TriEmbed web site: http://TriEmbed.org


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.triembed.org/pipermail/triembed_triembed.org/attachments/20170315/aa6477a6/attachment.htm>