Baling Wire and Spit: Reliable Remote Computer Control

A recent work project involved remote, unmanned computers set up to record weather data.  Every so often, we want to log in to the computers and download the data for processing.  The question is, if you are using off-the-shelf parts, can you make the remote computer reliably available?

Requirements

Why Download Data Manually?\\What is preventing us from just uploading the logged data to a remote server?  The data acquisition software is designed for user-interaction.  It has no batch processing or scripting features.  If I could just stop logging and send the data home, I would.  Human intervention is needed, for now.

The project has the following requirements:

  1. Alert users if computer boots.
  2. Allow users to connect to the computer.
  3. Maintain network connection.

Parts

This project uses a commercially available small-form-factor Windows 7 Embedded PC because the data acquisition hardware requires software that is only available for Windows.  The sites where we installed the computers are unmanned, with no chance of installing data lines or piggy-backing existing wired Internet connections.  We chose to use a Pantech UML295 USB Aircard on the Verizon network for our Internet access, because we knew there would be coverage in the areas we were monitoring.===== Alerting the User =====

This requirement was the simplest:

  1. Download sendEmail.  This is a Windows program that sends email via an SMTP mail server.  It works well with common hosts like Google Mail because it supports encryption.
  2. Write a batch file to send email.  The batch file tries 3 times.  Initially, it waits 10 minutes then sends, another 10 minutes then sends again, and finally it waits for an hour before sending one more "I'm rebooting" email.  This "three tries" approach was done in case the network wasn't available immediately upon boot.
  3. Create a schedule task to run the batch file at boot time.

This system does not take into account a long network outage.  I may add a check for Internet connectivity, but for now this setup is working.===== Allow Users to Connect =====

At our company, TeamViewer is the preferred software for controlling remote computers.  We installed TeamViewer on each PC, and linked the PCs into our company's account.  Whenever the PCs are online they show up in our TeamViewer computer list.  TeamViewer's performance is smooth, even over 4G.==== The Problem With TeamViewer ====

TeamViewer is great, and works 90% of the time (percentage is 100% made up).  But if it stopped working, what would we do?  If the network connection temporarily drops, sometimes the PC never comes back online.  It may even show up as "online" in TeamViewer, but connections cannot be made.  An alternative would be needed in case of TeamViewer failure.==== A Solution: VNC ====

The VNC family of programs is old and well-tested.  VNC does basically the same thing as TeamViewer - remote desktop sharing - but in a slightly more difficult way.  Whereas TeamViewer uses TeamViewer.com's servers to mediate connections between a client and a remote computer, VNC puts you in charge of connecting the client and the remote computer.This works out to our advantage because of something else, namely:==== The Problem with Verizon's 4G System ====

Verizon has a good, fast 4G network.  However, the IP addresses they give out are private, non-routable addresses.  I originally thought I could simply write a batch file that would email us a public IP to use with VNC, but that cannot work with a private IP address.  Verizon does have static IP addresses available, but I needed to finish this system right now, and didn't have the luxury of setting up static IP address for all the aircards we were using.==== TightVNC to the Rescue ====

The TightVNC program is made of a client and a server.  The server runs on a remote computer, and the client runs on your own computer.  You tell the client to connect to the server, and voila, you can see the server's desktop.  This assumes one little, minor, teensy detail: that you know the server's IP address.  But, alas, I do not, because of the Verizon network.All is not lost!  TightVNC's client has an interesting mode that makes it listen for servers.  Then you can just configure the server to "phone home" to the client, and the server's screen will, almost magically, appear on your computer.  This feature, called, smartly, listening, would be the cure for all our problems.  Except...==== What if the Remote Computer Loses Internet Access? ====

If the remote computer loses its Internet access, I'd need a way to re-establish the VNC session when access is restored.  Fortunately, the VNC server (running on the remote) can be controlled via batch file.  So, I wrote another batch file.  This file does the following:

  1. Tells the server to connect to a listening client.
  2. Waits 20 minutes.  Now is the time to get some work done.
  3. Disconnects from the listening client.
  4. Waits a minute.
  5. Goes back to Step 1.

The reason for connect/disconnect loop is that the server has no way to tell me that the client is unavailable, so I cannot automatically retry connecting.  Although VNC can be controlled via batch file commands, it only supports stopping, starting, connecting, and disconnecting.  There are no status commands, so it is difficult to figure out what's going on.  To work around this limitation, I just set up a connection, and restart it every 21 minutes.  It isn't pretty, but it works.It does have the advantage that if I don't want a client computer to be flooded with requests from all the remote computers, I can just turn off the client, and then turn it on when I need it.  The servers' phone home attempts silently fail, then they try again in 20 minutes.==== The Final Hurdle: Flaky Aircards ====

While a field crew was installing the first three monitoring systems on the opposite side of the country, I ran into a rather horrifying problem while prepping more systems in the lab.  One of the lab computers dropped its Internet connection.  There was no indication of a problem from the aircard's green=good/red=bad dummy lights, but there was also no Internet access.  This means that emails wouldn't work, TeamViewer wouldn't work, and VNC wouldn't work.  I patiently waited 20 minutes in the hopes that something would change.  Nothing changed.This was sort of a big deal, because if the three being deployed dropped their connections, there'd be no one around to fix it.  So, after a bit of searching, I came across a Microsoft tool called DevCon, or Device Control.  DevCon can remove a specific piece of hardware, and then add the hardware back again.  This is akin to using Device Manager, which is included with Windows.  DevCon's selling point was that it, like TightVNC, could be used with batch files.  I found an article from National Instruments (makers of the over-wrought LabView software system) that details how DevCon can be used to re-initalize an entire USB hub, thereby resetting all the devices attached to it. Devices like, oh, maybe aircards?So now we have a third batch file, but this is just two lines long.

  1. Tell DevCon to remove the USB hub that the aircard is connected to.
  2. Tell DevCon to rescan for hardware changes.  This resets the aircard, which forces it to start looking for Internet access.

This batch file is set to run every 6 hours.  So every 6 hours, the Internet connection is terminated, and then reestablished.  Not necessarily convenient, but better than nothing.===== Reliable, for Certain Definitions of Reliable =====

This system will be used over the coming weeks to monitor temperature data at locations all across the country.  The system was finished on a tight schedule, and although almost nothing is ideal, everything works.Probably.

This article was updated on February 5, 2014

You should also read: