Field testing finds what the bench can’t

I pulled askew-auk off the cold frame this week and brought her inside. She had gone silent on May 10, which was strange because the last check-in showed the battery at 72%. The story I expected was a dead battery, a loose wire, maybe a corroded contact. I knew something was off before I even pulled the head apart.

Then I cracked the enclosure.

A real colony. With workers, with eggs, with the unmistakable little pile of larvae that says “we live here now.” They had set up shop inside the enclosure. The warm electronics, the protected cavity, the breathable vents I had so generously designed for them. Phoebe was no longer a sensor. Phoebe was a condo.

The bug was a bug

I have spent weeks of field testing waiting for the last component on a slow boat from China. It arrived today. The entire PCB order to JLCPCB was queued up, ready to go. I was an hour away from clicking submit on the alpha boards.

Then I opened up askew-auk and found the colony.

I have been calling these things “bugs” my entire professional life and never once meant it literally. Now I have video of the bug. The bug has legs. The bug has children.

What the data said before I knew about the ants

Before I cracked the case, I pulled the last hour of readings to see if the silence had been preceded by anything weird. It had.

The soil moisture was railed at exactly 45.30%. Same value, every tick, three decimals deep. Real soil doesn’t do that. Real soil drifts, even if only by a tenth of a percent. Real soil breathes. This was a sensor pinned by something across its contacts, not a plant.

Her air-quality readings had gone wobbly in the same window. She was already half-blind before she stopped reporting, and I had not noticed because I was waiting on parts and assuming “no news is good news.” That assumption is now retired.

The actual kill

I plugged her into USB to see what was left. She powered on but said nothing. The little chunk of memory that holds her WiFi password and her name had been wiped clean. She had forgotten who she was.

She had spent the previous five days in setup mode, waiting patiently for a phone to come pair with her and tell her again. No phone was coming. She was in a cold frame full of ants.

Best guess at the chain of events: the ants on her power rail caused a tiny power dropout, the dropout landed in the middle of her writing to memory, and the write got scrambled. From there, the rest follows.

The fix on the firmware side

The post-mortem turned up something worth fixing on the software side too. Phoebe had been rewriting her own memory way more often than she needed to. Every boot, every reconnect, every little settings change was another write. Lots of small chances for a power dropout to land in the wrong instant.

The fix is boring and good. She now writes that memory once, when she is first set up, and then leaves it alone. Fewer writes, fewer chances to get clobbered.

That is the part I can fix in software. The bigger fix is mechanical.

The enclosure has to change

The alpha enclosure design assumed open-air vents were a feature. The air-quality sensor needs to actually breathe the air. The temperature sensor wants to feel the real outside. The light sensor wants to see the sun. So I gave Phoebe big friendly openings on the body, mesh-free, and patted myself on the back for thermal management.

I had designed an ant hotel with sensors on it.

The vents have to stay, because gas exchange is the whole point. They also clearly cannot stay the way they are. I do not yet know what the right answer is. I have a pile of ideas and none of them feel obviously correct, which is usually a sign I should sit with the problem for a few days instead of committing to the first thing that sounds clever.

The PCB order is on hold while I figure out the body. Better to find this now than after a hundred enclosures have been printed.

The lesson worth pulling out

The failure here was not really electrical. It was an enclosure problem that became a power problem that became a memory problem. Hardware in the field is a stack of unrelated layers, and the bug always lives in whichever layer you were not thinking about.

I was thinking about circuit boards. The ants were thinking about real estate.

— Ben