Tolerating Electrical Interference: reconnecting USB ports

UAF’s Aurora Robotics team entry in the NASA Robotic Mining Competition is using a standard Arduino Mega as the central microcontroller, which has worked well except for one intermittent disconnection problem, which we believe is caused by electrical interference from our big motors.

When it disconnects, the Linux kernel shows a dmesg error like “ehci_hub: USB disconnect on port 3 (EMI?)”.  We had previously built a system to tolerate this, by simply waiting a few seconds and reopening the USB serial connection, but this was only a partial fix since eventually the kernel disables the port completely.  This is bad because we could only fix it with a complete restart.  During our second practice mission yesterday, the kernel disabled all USB ports just as we began the run, resulting a completely scrubbed practice mission.

John Pender found a very simple way to re-enable the USB ports by running these commands as root:

echo "0000:00:1a.0" > /sys/bus/pci/drivers/ehci_hcd/unbind
echo "0000:00:1d.0" > /sys/bus/pci/drivers/ehci_hcd/unbind
echo "0000:00:1a.0" > /sys/bus/pci/drivers/ehci_hcd/bind
echo "0000:00:1d.0" > /sys/bus/pci/drivers/ehci_hcd/bind

This basically just turns off the kernel’s EHCI USB driver, then turns it back on, allowing the USB ports to be used again. Use “lspci | grep USB” to verify those addresses are correct for your machine (they seem right for most laptops).

The long-term fix for electrical interference issues on Arduino is using opto isolators on each digital control channel connected to a noisy source, but it’s often difficult to fully separate the ground planes in real robotics designs, so this is a simple fast way to get your robot moving again!

Arduino millis() jumps by 2 every 43 milliseconds

I’ve been doing some software-modulated infrared light detection work, and for the signal to be sent correctly I need millisecond-accurate timings.  I realize professionals do this sort of thing with interrupts, but I hate debugging interrupt code, and I’m always afraid there’s some rare timing glitch that will kill the entire project at the worst possible moment.

I was using Arduino’s millis() function as my time base, and I kept getting weird screwy timing every 43 milliseconds.  I assumed it was my code, which does a bunch of other stuff like serial communication, but the glitch didn’t change regardless of what I did.  Turns out, the Arduino Uno’s oscillator runs at a power of two rate, 1.024 milliseconds per overflow, so every 1/0.024 = 41.666 milliseconds, it’s a full millisecond off.  Arduino wiring.c fixes this by keeping track of the fractional milliseconds, and adds a sort of “leap millisecond” every 43 milliseconds to keep things in sync.  This results in millis() instantly jumping up by more than one.

Here’s an example Arduino Uno sketch that demonstrates these timing jumps, and shows two fixes.

/* Demonstrate timing gaps in the millis() function */
void setup(void) {
 Serial.begin(57600); // to report the horror
}

extern "C" volatile unsigned long timer0_overflow_count; // from wiring.

unsigned long last=0; // last value of millis()
void loop(void) {
 unsigned long next;
 do { // watch the timer until it changes
   next=millis(); // jumps by 2 every 43 ms
   //Two possible fixes:
   // next=micros()/1000; // slow, but works
   // cli(); next=timer0_overflow_count; sei(); // faster, but 1024 microseconds/tick
 } while (next==last);

 int delta=next-last;
 if (delta!=1) {
   Serial.print(delta);
   Serial.print(" jump at ");
   Serial.println(next);
 }

 if (last%10000==0) {
   Serial.print("Still running at ");
   Serial.println(last);
 }
 last=next;
}

On my Arduino Uno, the millis() version of this reports regular timing glitches:

Still running at 0
2 jump at 43
2 jump at 86
2 jump at 128
2 jump at 171
2 jump at 214
2 jump at 256
2 jump at 299
2 jump at 342
...

The easy way to avoid these timing jumps is never to use millis() at all!  It works to just use micros()/1000, although looking at wiring.c’s implementation of delay(), you need to be a bit careful about when micros() overflows every hour.  You can also directly read the raw timer0_overflow_count as shown above, although the actual speed this increments depends on the Arduino clock rate, and you need to do cli/sei around the read to avoid jumps of +-256, as the interrupt function updates the two bytes of that value.  Both fixes never have leap milliseconds, and result in dependable timing.

Ubuntu USB serial circuit boards vs “modem-man”

For CYBER-Alaska we built a custom Arduino-style circuit board (“Rovoduino”) that connects to the PC via USB.  It generally worked, but there was always a weird half-minute delay when you plugged it into a Linux PC before anything would start to work.  Tonight I was testing a new sensor on an Arduino Mega, a similar board, and noticed the plug-in-to-work delay there was only a little over one *second* (due to the bootloader).

Back to my custom board, trying a “sudo lsof | grep ttyUSB” during the half-minute delay shows the unexpected process “modem-man” spending this time sending commands like “AT+++” and trying to convince my circuit board that it is, in fact, a modem.

It’s not a modem.

You can explicitly blacklist your custom USB device from modem-man on Ubuntu in /lib/udev/rules.d/77-mm-usb-device-blacklist.rules, but it’s more reliable to just remove the darn thing entirely:

sudo apt-get remove modemmanager

Because seriously, who uses a modem?

This makes plugging and working on my USB serial board a lot snappier!