[TriEmbed] Update on my Stability Issue

Chip McClelland chip at mcclellands.org
Tue Mar 14 21:02:37 CDT 2017


For those who missed it, I sent a note about a month ago asking for advice on improving the reliability of my sensors.  These sensors would go weeks and measure tens of thousands of cars and then lock up.  I was struggling to fix a problem I could not easily reproduce so I asked this group for ideas.   

Thank you for all the great advice it has been very helpful.  I think I have made significant progress in my testing and, this week, I have deployed the updates to the sensors in the park.  Given that many of you took the time to give me your thoughts, I wanted to share what I have learned.

	1) A few of you suggested making the problem worse to help improve the reproducibility of the issue - that was key.  I was eventually able to get the issue to surface in under ten minutes which helped me test potential fixes.
	2) My problem seemed to be related to the Arduino’s ability to wake from sleep and reliably access data on the i2c bus.  I was able to find the specific function and eventually the  i2c command - t = RTC.get(); that would lock up the i2c bus and the Arduino.  The incidence of this issue was increased significantly when the Arduino was recently brought back from sleep.
	3) The solution was to start a watchdog timer prior to using this command and set up the Arduino to reboot if it did not complete in a reasonable period.  Worked fine on the Uno but this locked up the Arduino Pro Mini.  It seemed that the Pro mini does not use an Optiboot boot loader like the Uno and there is a known bug <https://forum.arduino.cc/index.php?topic=435172.0> that prevents the watchdog timer from functioning correctly.  I found a new boot loader - MiniCore <https://github.com/MCUdude/MiniCore> that fixed the issue and delivered on some cool new features.
	4) The problem was that I have a number of boards in production which do not have ICSP headers.  I was able to find a great adapter  <https://www.amazon.com/gp/product/B00V2W467I/ref=oh_aui_detailpage_o04_s00?ie=UTF8&psc=1>great adapte <https://www.amazon.com/gp/product/B00V2W467I/ref=oh_aui_detailpage_o04_s00?ie=UTF8&psc=1>r  that let me refresh the chips in place - highly recommended.   
	5) Changing the boot loader also changed my tool chain and I ended up asking the embedXcode folks for help <https://embedxcode.freshdesk.com/support/tickets/660> and they delivered a new release that let me integrate the new boot loader with the board definition files.  If you are a Mac user and want a better IDE, I can highly recommend embedXcode <http://embedxcode.weebly.com/>.
With these changes and some improvements to the process of waking from sleep from Nick Gammon and the Arduino forums, <http://forum.arduino.cc/index.php?topic=336789.0> I was able to put the device through significant
testing without any lock ups.  I added a “reboot” counter to EEPROM so I can see if more changes are needed in the future.  I just finished updating all the sensors in the park and will see how the real world reacts to my efforts. 

I hope there are some ideas in here that might be helpful.  Please let me know if you have more suggestions.  I will also be looking at my power and signal integrity on the boards but, for now, I think I have a solution.  If you would like to see what the code looks like, you can find it here <https://github.com/chipmc/Logger-Arduino-PIR>.  Thanks again for all your suggestions.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.triembed.org/pipermail/triembed_triembed.org/attachments/20170314/d00c0d58/attachment.html>

More information about the TriEmbed mailing list