<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">All, <div class=""><br class=""></div><div class="">Thank you for the great suggestions. Based on your responses, here is what I am thinking:</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- Given the number of “events” and days over which the counter runs, the issue is likely hardware not software</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- I checked the fuses on my Arduino and I do have brownout protection turned on. I am running at 8MHz, 3.3V and brownout is set to 2.7V </div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- The ATMEGA 328p can run up to 10Mhz at 2.7V so should be good there</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- I took two hours of measurements at 1/2 second intervals while the device was under test out of the 14400 test points - average 3.282V and lowest was 3.279</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- I have taken a screen shot of the test <a href="https://dl.dropboxusercontent.com/u/21074878/Vcc%20Log%20over%20two%20hours.png" class="">here</a> I think that this looks fairly clean but please let me know if you see anything that needs more exploration here.</div><div class=""><br class=""></div><div class="">I did find more noise and voltage sag under load in my Solar power converter than I expected. I contacted TI support and they suggested I change my layout. You can see the exchange and how I have improved the layout <a href="https://e2e.ti.com/support/power_management/non-isolated_dcdc/f/196/t/569106" class="">here</a>.</div><div class=""><br class=""></div><div class="">I am now wondering if there is some dynamic between the Solar Panel board and the LDO converter on my trail counter. I plan to start testing this once I figure out how to do so. It may be there are some transients when the Solar Board changes to Solar charging, LiPo battery or Solar batter charged. </div><div class=""><br class=""></div><div class="">Again, thank you all for your suggestions, I will let you know what I figure out. I am also going to respin the boards to improve the power supply layouts.</div><div class=""><br class=""></div><div class="">Chip</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 23, 2017, at 1:00 PM, <a href="mailto:triembed-request@triembed.org" class="">triembed-request@triembed.org</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Send TriEmbed mailing list submissions to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><a href="mailto:triembed@triembed.org" class="">triembed@triembed.org</a><br class=""><br class="">To subscribe or unsubscribe via the World Wide Web, visit<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>http://mail.triembed.org/mailman/listinfo/triembed_triembed.org<br class="">or, via email, send a message with subject or body 'help' to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>triembed-request@triembed.org<br class=""><br class="">You can reach the person managing the list at<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>triembed-owner@triembed.org<br class=""><br class="">When replying, please edit your Subject line so it is more specific<br class="">than "Re: Contents of TriEmbed digest..."<br class=""><br class=""><br class="">Today's Topics:<br class=""><br class=""> 1. Re: Suggestions Needed to Improve Quality / Reliability<br class=""> (Pete Soper)<br class=""> 2. Re: Suggestions Needed to Improve Quality / Reliability (Brian)<br class=""> 3. Re: Suggestions Needed to Improve Quality / Reliability<br class=""> (Craig Cook)<br class=""> 4. Arrow offering 12% discount for one week (Pete Soper)<br class=""> 5. Re: Suggestions Needed to Improve Quality / Reliability<br class=""> (Pete Soper)<br class=""><br class=""><br class="">----------------------------------------------------------------------<br class=""><br class="">Message: 1<br class="">Date: Mon, 23 Jan 2017 10:03:05 -0500<br class="">From: Pete Soper <pete@soper.us><br class="">To: triembed@triembed.org<br class="">Subject: Re: [TriEmbed] Suggestions Needed to Improve Quality /<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Reliability<br class="">Message-ID: <9f4f1f8a-0130-38a3-e52a-dc2167afbb45@soper.us><br class="">Content-Type: text/plain; charset="windows-1252"; Format="flowed"<br class=""><br class="">On 01/22/2017 03:37 PM, Chip McClelland via TriEmbed wrote:<br class=""><br class=""><blockquote type="cite" class="">- When I do get the system to lock up, his is what I see:<br class="">- The Arduino is the one that is locked up not the Simblee<br class="">- When the Arduino locks up, the Simblee is unable to take control of <br class="">the i2c bus<br class="">- The Arduino is in the ?sleep? state and will not wake up - with the <br class="">pin change interrupt set and _even if I manually bring the interrupt <br class="">pin low_<br class="">- The Arduino will not reset _even if I manually bring the reset pin <br class="">low_ - this is odd and what I have come to call the "fugue state".<br class="">- The Simblee will be visible on Bluetooth LE but will lock up when I <br class="">attempt to connect and it tries to read the i2c bus (I do not have a <br class="">timeout on taking the bus)<br class="">- The only way to recover is to power cycle the device<br class=""><br class=""></blockquote>(Since your board is full custom and doesn't look like an Arduino, I'm <br class="">going to just refer to the IC.)<br class=""><br class="">Well, the Atmega not doing the right thing in response to the reset pin <br class="">being pulled down seems like the smoking gun. This is of course <br class="">"impossible", so the question is how this could be happening. The only <br class="">two things I can think of are a too short reset sequence and an <br class="">intermittent power supply problem. Is the Simblee holding the reset line <br class="">low for at least 2.5 microseconds? The rest of this assumes "yes".<br class=""><br class="">What's the clock speed of the Atmega? What's the Atmega's supply voltage <br class="">when it won't respond to reset? Will the Simblee run with a lower supply <br class="">voltage than the Atmega? I wonder if the fugue state is to do with the <br class="">onboard DC to DC converter vs the Atmega. If you could watch and record <br class="">the Atmega's VCC pin while trying to reset it out of the fugue state you <br class="">might see it sagging as the Atmega tries to start up: the Simblee is <br class="">able to keep running (or the sag might be causing the Simblee to stop <br class="">and restart. Can you detect that?). Is there any correlation with <br class="">temperature or battery state? Can the Atmega get it's work done at a <br class="">lower clock speed? Purely as an experiment you might lower the speed to <br class="">reduce its power requirement. If it rides through and "outlasts" all the <br class="">other units in the park, then it might suggest a power supply stability <br class="">issue. Systematically testing the power supply might be in order.<br class=""><br class="">-Pete<br class="">-------------- next part --------------<br class="">An HTML attachment was scrubbed...<br class="">URL: <http://mail.triembed.org/pipermail/triembed_triembed.org/attachments/20170123/f274ffc7/attachment-0001.html><br class=""><br class="">------------------------------<br class=""><br class="">Message: 2<br class="">Date: Mon, 23 Jan 2017 10:06:12 -0500<br class="">From: Brian <triembed@undecidedgames.net><br class="">To: triembed@triembed.org<br class="">Subject: Re: [TriEmbed] Suggestions Needed to Improve Quality /<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Reliability<br class="">Message-ID: <58861BE4.8030708@undecidedgames.net><br class="">Content-Type: text/plain; charset=windows-1252; format=flowed<br class=""><br class="">I was thinking a power issue might be the culprit as well. Is there any <br class="">difference in power supply between your test fixture and the real world? <br class=""> Presumably batteries are involved in the real installation, and <br class="">batteries are not very clean power supplies; be sure you have plenty of <br class="">filtering capacitance at the input. I'd recommend two or three decades <br class="">(10 uF, 1 uF, .1 uF e.g.) of capacitors in parallel on the supply; each <br class="">represents a low-impedance path for different bands of high-frequency <br class="">energy.<br class=""><br class="">The mention of memory management is worth investigating, though if that <br class="">were the cause I'd expect there to be a stronger correlation to number <br class="">of events. The Arduino software won't tell you if you're exceeding the <br class="">available RAM. Stack overflows can easily sneak up and bite you.<br class=""><br class="">Good luck!<br class="">-B<br class=""><br class=""><br class=""><br class="">On 01/23/2017 09:41 AM, Shane Trent via TriEmbed wrote:<br class=""><blockquote type="cite" class="">Chip,<br class=""><br class="">Forgive me but I do not recall if you have the Brown-out detector<br class="">enabled on the Arduino. It is common to disable the brown-out detector<br class="">to save power but these are the kind of symptoms the brown-out detector<br class="">is designed to prevent. Odd lockup can frequently (but not always) come<br class="">from voltage excursions out of spec. Basically the brown-out detector<br class="">resets the microcontroller if the voltage drops below the brown-out<br class="">level. Worth testing it is not currently enabled. No pun intended.<br class=""><br class="">Shane<br class=""><br class="">On Mon, Jan 23, 2017 at 9:23 AM John Vaughters via TriEmbed<br class=""><triembed@triembed.org <mailto:triembed@triembed.org>> wrote:<br class=""><br class=""> Chip,<br class=""><br class=""> If the symptom is at regular intervals, whether they are regular<br class=""> events, or regular in time, I would consider a memory leak. I do not<br class=""> no have recommendations on how to check that, but I did a quick<br class=""> search and it appears there are some suggestions. I have never<br class=""> search for a memory leak in an Arduino before, but I am just<br class=""> throwing out an idea based on what appears to be regular intervals<br class=""> and a lock up condition.<br class=""><br class=""> If you want to create a cheap reboot device, consider an ATTINY to<br class=""> wake up the device and look for a Digital IO to come on and off<br class=""> during a certain period of time, and if not cycle power. I know this<br class=""> just more power to Engineer which is a major negative for your<br class=""> product. Another option is a low cost, low power 555 timer set to<br class=""> trigger every so many hours or days, whatever you need. I am not<br class=""> sure which idea would give you the best power solution.<br class=""><br class=""> Just throwing out ideas.<br class=""><br class=""> Good Luck,<br class=""><br class=""> John Vaughters<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><br class=""> On Sunday, January 22, 2017 3:37 PM, Chip McClelland via TriEmbed<br class=""> <triembed@triembed.org <mailto:triembed@triembed.org>> wrote:<br class=""><br class=""><br class=""> All,<br class=""><br class=""> I have been installing trail counters at Umstead park for some<br class=""> time. This has been an ideal situation as I often run or ride in<br class=""> the park and it allows me to keep an eye on things. For some time<br class=""> now, I have been working to improve the reliability of my counters<br class=""> and I feel like I have hit a wall. I am writing you all not in the<br class=""> hope that you can find the source of my issue but to offer<br class=""> suggestions on approaches I could use to better isolate what the<br class=""> root cause might be.<br class=""><br class=""> Background and what I know thus far:<br class=""> - My sensor has two micro controllers (Arduino and Simblee) that<br class=""> share an i2c bus. As they are both masters, I have a scheme to<br class=""> allow one or the other to take control. I mention this because it<br class=""> may be the source of my issue. More on how I architected this here<br class=""> <https://www.hackster.io/chipmc/arduino-i2c-multi-master-approach-why-and-how-93f638?ref=user&ref_id=6903&offset=0><br class=""><br class=""> - The Arduino manages the sensor and the Simblee manages<br class=""> communications using Bluetooth LE.<br class=""> - I have tested the software extensively and have a test rig that<br class=""> can test the full functionality of the system to randomly timed<br class=""> ?events?. I can only occasionally get the error condition with this<br class=""> approach typically requiring over 100,000 ?events? before it<br class=""> happens. Because of the timing in the system, this requires testing<br class=""> the systems continuously for days on end. In the parks, the sensors<br class=""> typically run 25-30 days before they lock up during which time they<br class=""> will log around 30,000 ?events?.<br class=""> - When I do get the system to lock up, his is what I see:<br class=""> - The Arduino is the one that is locked up not the Simblee<br class=""> - When the Arduino locks up, the Simblee is unable to take control<br class=""> of the i2c bus<br class=""> - The Arduino is in the ?sleep? state and will not wake up - with<br class=""> the pin change interrupt set and _even if I manually bring the<br class=""> interrupt pin low_<br class=""> - The Arduino will not reset _even if I manually bring the reset pin<br class=""> low_ - this is odd and what I have come to call the "fugue state".<br class=""> - The Simblee will be visible on Bluetooth LE but will lock up when<br class=""> I attempt to connect and it tries to read the i2c bus (I do not have<br class=""> a timeout on taking the bus)<br class=""> - The only way to recover is to power cycle the device<br class=""><br class=""> I am not sure how to best proceed. I could spend more time / effort<br class=""> trying to figure out what the root cause is OR I could design an<br class=""> external watchdog that would simply recognize a lock up and reset<br class=""> the power. In my application, the cost of a reset every 20-30 days<br class=""> is acceptable but, part of me says that is the easy way out.<br class=""><br class=""> So, I am open to any suggestions you might have even if they are the<br class=""> ?have you checked to see if it is plugged in? variety. If you want<br class=""> to see the code, it is in two parts: Arduino<br class=""> <https://github.com/chipmc/Connected-Logger-Arduino> and Simblee<br class=""> <https://github.com/chipmc/Trail-Counter-Simblee> Any input is<br class=""> appreciated.<br class=""><br class=""> Thanks,<br class=""><br class=""> Chip<br class=""><br class=""> _______________________________________________<br class=""> Triangle, NC Embedded Computing mailing list<br class=""> TriEmbed@triembed.org <mailto:TriEmbed@triembed.org><br class=""> http://mail.triembed.org/mailman/listinfo/triembed_triembed.org<br class=""> TriEmbed web site: http://TriEmbed.org <http://triembed.org/><br class=""><br class=""><br class=""> _______________________________________________<br class=""> Triangle, NC Embedded Computing mailing list<br class=""> TriEmbed@triembed.org <mailto:TriEmbed@triembed.org><br class=""> http://mail.triembed.org/mailman/listinfo/triembed_triembed.org<br class=""> TriEmbed web site: http://TriEmbed.org<br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">Triangle, NC Embedded Computing mailing list<br class="">TriEmbed@triembed.org<br class="">http://mail.triembed.org/mailman/listinfo/triembed_triembed.org<br class="">TriEmbed web site: http://TriEmbed.org<br class=""><br class=""></blockquote><br class=""><br class=""><br class=""><br class="">------------------------------<br class=""><br class="">Message: 3<br class="">Date: Mon, 23 Jan 2017 15:09:19 +0000 (UTC)<br class="">From: Craig Cook <cncook001@yahoo.com><br class="">To: "triembed@triembed.org" <triembed@triembed.org><br class="">Subject: Re: [TriEmbed] Suggestions Needed to Improve Quality /<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Reliability<br class="">Message-ID: <263538251.1930074.1485184159176@mail.yahoo.com><br class="">Content-Type: text/plain; charset="utf-8"<br class=""><br class="">How many trail counters are in operation?<br class=""><br class=""><br class="">Do they all show the same lockup behavior?<br class="">The 30,000 number sounds suspicious.? Close to 32k.? Do you have some sort of integer or number issue going on?? i.e. you ran out of space to hold numbers?<br class="">Craig<br class=""><br class="">-------------- next part --------------<br class="">An HTML attachment was scrubbed...<br class="">URL: <http://mail.triembed.org/pipermail/triembed_triembed.org/attachments/20170123/2b5868c6/attachment-0001.html><br class=""><br class="">------------------------------<br class=""><br class="">Message: 4<br class="">Date: Mon, 23 Jan 2017 11:45:28 -0500<br class="">From: Pete Soper <pete@soper.us><br class="">To: Triangle Embedded Computing Discussion <TriEmbed@triembed.org><br class="">Subject: [TriEmbed] Arrow offering 12% discount for one week<br class="">Message-ID: <d289e963-7940-ac9d-100d-53706652fafe@soper.us><br class="">Content-Type: text/plain; charset=utf-8; format=flowed<br class=""><br class="">Arrow kicks the stuffings out of other distributors for some components, <br class="">and a 12% discount can take the sting out of shipping costs. They just <br class="">announced this discount for one week with promo code "ROOSTER17". (I <br class="">only occasionally buy from them and there's nothing in this for me.)<br class=""><br class="">-Pete<br class=""><br class=""><br class=""><br class=""><br class=""><br class="">------------------------------<br class=""><br class="">Message: 5<br class="">Date: Mon, 23 Jan 2017 12:07:56 -0500<br class="">From: Pete Soper <pete@soper.us><br class="">To: triembed@triembed.org<br class="">Subject: Re: [TriEmbed] Suggestions Needed to Improve Quality /<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Reliability<br class="">Message-ID: <0c7f5451-9bef-9fee-05dd-2756b7abb7e7@soper.us><br class="">Content-Type: text/plain; charset=windows-1252; format=flowed<br class=""><br class="">Also, is it a sure thing that the Simblee is pulling the Atmega reset <br class="">line low enough? How much current is the Simblee having to sink to do <br class="">the reset? It might require cutting/repairing a trace, but if the <br class="">current necessary to reset the Atmega could be measured it might show <br class="">this is only working part of the time by serendipity because of a <br class="">problem with the board. We need to be careful to get any regular flux <br class="">off the boards and moisture might create a situation where the Simblee <br class="">hasn't got the grunt to reset the Atmega. If they don't already have <br class="">them your boards may need unique IDs (e.g. a few bytes of the 328's <br class="">EEPROM) to allow for tracking what's going on with them.<br class=""><br class="">There seem to be one or two things that need their root causes found:<br class=""> 1) Failure to reset ("fugue state")<br class=""> 2) Atmega "lock up" if it's the cause and not the side effect of (1)<br class=""><br class="">Could the Simblee detect the I2C bus "release" by the Atmega failing and <br class="">tell you exactly when that happens? This might be correlated with <br class="">voltage measurement logs (but you're only logging battery voltage, not <br class="">the IC's VCC, right?) Tacking on a tiny daughterboard to detect and <br class="">signal something to the Simblee would would not be too terrible. In my <br class="">opinion having two processors the way you do is a terrific demonstration <br class="">of the value of postponing optimization. :-)<br class=""><br class="">-Pete<br class=""><br class=""><br class=""><br class=""><br class=""><br class="">------------------------------<br class=""><br class="">Subject: Digest Footer<br class=""><br class="">_______________________________________________<br class="">TriEmbed mailing list<br class="">TriEmbed@triembed.org<br class="">http://mail.triembed.org/mailman/listinfo/triembed_triembed.org<br class=""><br class=""><br class="">------------------------------<br class=""><br class="">End of TriEmbed Digest, Vol 44, Issue 27<br class="">****************************************<br class=""></div></div></blockquote></div><br class=""></div></body></html>