The Only Complete OSM Tileserver Guide

With Google’s recent announcement to start charging for the use of Maps, we
started to worry. After running the numbers and assuming reasonable growth (the
same growth we had in 2010), http://ridewithgps.com was looking at around
$40,000 per year in usage! This is considerably higher than our budget, so I
finally took the plunge
I have just spent the last two weeks setting up a tileserver on the OSM stack.

Using the Edge 800 with turn by turn directions

I finally got around to putting up some comprehensive information about the Edge 800, specifically in regards to route navigation and Open Street Maps. Detailed information can be found here on how to use your Edge 800 for turn by turn directions – additionally, the page has information for how to download and load OSM (Open Street Maps) onto an Edge 800 for people who didn’t purchase Garmin’s City Navigator maps.

Garmin Communicator broke for IE9 users

If you have the Garmin Communicator API integrated into your site, and you have noticed IE9 no longer works with the new FIT enabled devices, it’s not your fault! While I don’t know if it’s a bug in the Communicator plugin or an intentional move by Garmin, Connect is the only site which allows IE9 users with FIT enabled devices like the Edge 500 and Edge 800 series to sync.

However, there’s a simple solution. You just have to jump into the Communicator javascript API and change a few things. The problem is that the actual browser plugin says that the device doesn’t support the FIT file format. The solution is “faking” the results for users with the newer devices.

To fake out the plugin, I simply use a regular expression to check that the actual name of the GPS device matches a FIT enabled device. In this solution I only have a regex that detects Edge 500 and Edge 800 devices, and you may want to expand the expression to other devices.

First, open up GarminDevicePlugin.js and head to the initialize function, which is around line 51.

  initialize: function(pluginElement) {        
      this.plugin = pluginElement;
      this.unlocked = false;
      this.deviceName = "";
      //console.debug("DevicePlugin constructor supportsFitnessWrite="+this.supportsFitnessWrite)
  },

Notice how I added the field deviceName to the class object. We will use this to assign the first detected device’s name to the plugin, so that it can use a regex later. In order to assign this value when the device itself is detected, we have to modify GarminDeviceControl.js at line 240:

   _finishFindDevices: function() {
        if(this.garminPlugin.finishFindDevices()) {
            this.devices = Garmin.PluginUtils.parseDeviceXml(this.garminPlugin, this.getDetailedDeviceData);
            this.numDevices = this.devices.length;
            this.deviceNumber = 0;
            if(this.devices.length > 0) this.garminPlugin.deviceName = this.devices[0].getDisplayName();
            this._broadcaster.dispatch("onFinishFindDevices", {controller: this});
       } else {
          setTimeout(function() { this._finishFindDevices() }.bind(this), 500);
       }
   },

We only assign the device name if there actually is a device detected. Also it is important that we do this before the onFinishFindDevices event is dispatched.

Final piece of the puzzle is actually using this information. To do this, we head back to GarminDevicePlugin.js and head to line 161 and the function “getSupportsFitnessDirectoryRead”. There are a few functions we want to add this check to, including “getSupportsFitDirectoryRead”. I simply added another function to do the actual checking which returns true or false, then call that function wherever I need to.


  isFitEnabledEdge: function() {
      if(this.deviceName.match(/edge (5|8)00/i)) return true;
      return false;
  },
    
    /** Lazy-logic accessor to fitness write support var.
     * This is used to detect whether the user's installed plugin supports fitness writing.
     * Fitness writing capability has a minimum requirement of plugin version 2.2.0.1.
     * This should NOT be called until the plug-in has been unlocked.
     */
    getSupportsFitnessWrite: function() {
        if(this.isFitEnabledEdge()) return true;
        return this._getPluginFunctionExists("StartWriteFitnessData"); 
    },
    
    /** Lazy-logic accessor to fitness write support var.
     * This is used to detect whether the user's installed plugin supports fitness directory reading,
     * which has a minimum requirement of plugin version 2.2.0.2.
     * This should NOT be called until the plug-in has been unlocked.
     */
    getSupportsFitnessDirectoryRead: function() {	
        if(this.isFitEnabledEdge()) return true;
        return this._getPluginFunctionExists("StartReadFitnessDirectory");
    },

    /** Lazy-logic accessor to FIT read support var.
     * This is used to detect whether the user's installed plugin supports FIT directory reading,
     * which has a minimum requirement of plugin version 2.8.1.0.
     * This should NOT be called until the plug-in has been unlocked.
     */
    getSupportsFitDirectoryRead: function() {		
        if(this.isFitEnabledEdge()) return true;
        return this._getPluginFunctionExists("StartReadFITDirectory");
    },
    
    /** Lazy-logic accessor to fitness read compressed support var.
     * This is used to detect whether the user's installed plugin supports fitness reading in compressed format,
     * which has a minimum requirement of plugin version 2.2.0.2.
     * This should NOT be called until the plug-in has been unlocked.
     */
    getSupportsFitnessReadCompressed: function() {
        if(this.isFitEnabledEdge()) return true;
        return this._getPluginFieldExists(this.plugin.TcdXmlz);
    },

There may be more places where this is needed, but this works for my simple use cases, merely reading files from GPS devices. Hope this helps!

Did Mongoid Break Your Migrations?

When using Mongoid and ActiveRecord side by side, a default install will break your ability to run rails migrations and generators. This is because Mongoid pushes aside ActiveRecord and assumes the tasks of migrations, which if you know anything about MongoDB or other document stores, is not needed. The solution is simple – tell ActiveRecord that you want to use it for generators with the following snippet added to your application.rb file:

    config.generators do |g| 
      g.orm :active_record 
    end

How to catch a (rodent, not informant) rat

I like to consider this blog first and foremost about startups and programming. However, dealing with disposing of a rat living in my house proved to be just as difficult as setting up a recurring billing system! So, this article is about hacking a rat, instead of programming computers. If you don’t want to read the whole post, here is the gist: to get rid of a rat who will touch no trap, simply tie bacon onto a spool of fishing line, and when the rat takes the bacon to his nest you can then follow the line to find the nest and finish him off.

Now, some people who get rats are lucky: they have dumb rodents that will fall for snap traps smothered in peanut butter. Boy, do I wish the rat that chose my freaking oven as a home was dumb. Nope, instead I get the rat that will not fall for any type of trap. I tried humane rodent traps, a custom trap made with a bucket and a spinning bottle smothered in peanut butter (cool design, rodent runs up a ramp to the bucket rim, tried to walk on the soda bottle suspended with wire across the bucket opening, which then spins on them dropping them into water below). I then resorted to snap traps with a variety of different baits, including peanut butter, cheese, dog food, bacon and tuna. Nothing. The damn thing was stock piling food underneath the kitchen stove with impunity!

I took apart the stove and disturbed his nest, which happened to be in the insulation above the oven and below the stove top. He had pissed and shat in the insulation, which cost me $100 to replace. In any event, he was scared out of his oven-nest and proceeded to skitter across the kitchen floor, and down to the basement. Dumb me forgot to close the basement door. Here is a pic of him playing dead when I opened the oven:

So, getting desperate, I bought a pellet gun and set out to shoot the damn thing that had been keeping me up at night for weeks now. I spent several hours each night perched on the steps in my basement, pellet gun loaded in my lap, scope zeroed in and focused at my 10 yard range to the pile of bacon I kept replenishing on a ledge across the basement. Still, the damn thing wouldn’t eat the bacon until it had been out for at least 24 hours, and wouldn’t eat it at any consistent time.

So now I am really desperate and my girlfriend is starting to question my manhood. Come to think of it, even I was questioning my manhood so I decide to campout in the basement until I killed the damn thing. I made a hammock snipers nest, hung from the joists supporting the floor of the house above me. I put up a red 25w light bulb illuminating my target area, tied a piece of fishing line to the two pieces of bacon in the event I dozed off, then proceeded to read/doze in my snipers den until 4am. The other end of the fishing line was looped around my wrist, which would hopefully tug me awake if my target went for the bait. By the time 4am comes, I give up reading by almost no light, and decide to go to bed. Next morning the damn bacon is gone, but I have a length of fishing line I can follow to find his new home!

From here, the most manic 4 hours of my life occurred, which resulted in my digging up 20 square feet of dirt in the crawlspace of the house, unearthing rat tunnels, flooding them, ripping all the vapor barrier plastic out, chasing a rat with a shovel, screaming at the sky, then finding the bastard under the steps in front of the house. I got him with a pellet then did a victory dance with my rifle in the front yard, scaring the shit out of all my neighbors.

Sorry PETA, and sorry rat. But, it was you or my girlfriend plus my sanity. Easy choice.

nginx error log filling up with rewrite rules

I will be the first to admit that I am not a stellar sysadmin, however I met a new low when I realized my nginx error log for my bike route mapping site was filled with 5 gigs of rewrite notifications. After thinking the solution would be easy, simply setting rewrite_log to off, I found out that didn’t actually do anything.

So, if you are trying to turn off rewrite rule logging in your nginx error log, you must do two things:

First, add

  *rewrite_log off;*

to your main configuration. Then, make sure each error_log declaration has a level other than critical. I picked ‘warn’, and we’ll see if this fills up quick. My guess is for heavy hit sites, ‘error’ is the correct value. Your error log line should look like:

  *error_log /var/log/nginx.ridewithgps.com.error.log warn;*

With those changes you should have a much smaller error log footprint, and grepping through your logs won’t grind on your disks too much.

Converting DEM from GeoTIFF to GridFloat with C Source

I just recently acquired a patched and improved SRTM elevation dataset for the entire world. This will allow our route planner to use only our own elevation service rather than relying on the USGS and geonames for elevations outside the United States. However, the format choices for the DEM was either GeoTIFF or an ASCII grid. Since the ASCII grid of elevations just means the elevations are literally space delimited in a plain ASCII file, I decided that was a huge waste of space.

However, after spending a bit of time learning about GDAL then writing a program in C (made a ruby module), I found out that accessing individual pixels in a GeoTIFF using GDAL is slowwww. So slow, that it was unusable for our application. After poking around and thinking, I decided to drop the GeoTIFF format and to convert the files to a good old GridFloat. The GridFloat format is very simple: it’s a binary file where each 32 bits is an elevation. You can easily calculate where in the binary file to access your elevation by knowing the latitude and longitude of the corner, and that the binary file is a flattened 2D array of elevation data in row-major order. Meaning, the first X bits of the file are row1 of the grid, followed by row2, etc etc.

My first conversion program was naive, and it read the TIFF pixel by pixel, writing each pixel to the output file individually. As expected, this was incredibly slow considering each TIFF was a 6001×6001 grid of pixels. Converting a single 69mb TIFF took 3-5 minutes.

My second iteration of the program read row by row, writing each row to the output file, and was considerably faster, clocking in at only 3 seconds per file. However, I realized that each read then write was sending the disk head all over the platter, and was definitely inefficient. So, I sacrificed some RAM and just read the entire TIFF into an array, then wrote that array to disk using a single fwrite() call. This ended up being incredibly fast, taking only 1 second to read a 69 meg TIFF and writeout a 138 meg GridFloat file.

Here is the final conversion code:

#include <stdlib.h>
#include <math.h>
#include "gdal.h"

int main(int argc, char *argv[]) {
        char *filename = argv[1];
        char *outfilename = argv[2];
        char *headerfilename = argv[3];
        char str[128];
        GDALDatasetH dataset;
        GDALRasterBandH *band;
        int xSize;
        int ySize;
        int i, j;
        double geoTransform[9];
        int sizee = sizeof(float);
        int count = 6001*6001;
        float *outt = malloc(sizeof(float)*count);
        float *window;
        FILE *outfp;

        GDALAllRegister(); //initialize GDAL

        dataset = GDALOpen(filename, GA_ReadOnly);
        if(dataset == NULL) {
                fprintf(stderr, "Couldn't open geotiff file %s\n", filename);
                return 0;
        }   

        band = GDALGetRasterBand(dataset, 1); 

        xSize = GDALGetRasterXSize(band);
        ySize = GDALGetRasterYSize(band);
        window = (float *) CPLMalloc(sizeof(float)*xSize);

        GDALGetGeoTransform(dataset, geoTransform);
        if((outfp = fopen(outfilename, "w")) == NULL) {
                printf("shit, something went wrong opening file");
                fflush(stdout);
                exit(1);
        }   

        /*  
         * If I naively just loop through each pixel and write that pixel, it takes so long to actually
         * perform the conversion.  So we loop through row then order them in-memory
         * in an array, then write that array in a single fwrite call.  This is orders of
         * magnitude faster!  1 second vs 3-5 minutes per 69 meg in -> 138 meg out file.
         */
        for(i = 0; i < xSize; i++) {
                GDALRasterIO(band, GF_Read, 0, i, xSize, 1, window, ySize, 1, GDT_Float32, 0, 0); 
                for(j = 0; j < ySize; j++) {
                        outt[j + i*xSize] = window[j];
                }
        }   
        fwrite(outt, sizee, count, outfp);
        fclose(outfp);

        outfp = fopen(headerfilename, "w");
        sprintf(str, "ncols         10812\nnrows         10812\n");
        fputs(str, outfp);
        sprintf(str, "xllcorner     %f\nyllcorner     %f\n", geoTransform[0], geoTransform[3] - 5.0008333333333);
        fputs(str, outfp);
        sprintf(str, "cellsize      0.0008333333333\nNODATA_value  -32768\nbyteorder     LSBFIRST\n");
        fputs(str, outfp);
        fclose(outfp);

        exit(0);
}

Error

If you see this error popup while working with ActionScript 3, it’s most likely because you are trying to access a property of a null object. It happens most often to me when looping through an array and having an element be null unexpectedly. It’s a pretty undescriptive message, but easily fixed.

Installing ggplot2 with R on Ubuntu

Had an issue installing the ggplot2 library for making a few graphs using R. I ran into a snag when I opened up the R console and tried to install the library, getting an error:

  > install.packages("ggplot2")
  install.packages("ggplot2") : package ‘plyr’ is not available

Turns out, Ubuntu has an older version of R than the latest ggplot requires, so, I had to pull down and install R from source. Not too hard, you can get the source at The Comprehensive R Archive Network

Installing from source is as simple as the standard:

  ./configure
  make -j8
  sudo make install

where -j8 is taking advantage of the quad-core processor in my machine.
Enjoy!

Garmin Communicator getting FIT support

After the Garmin Edge 500 came out with its new fileformat, FIT, I tried to get support for the device on the ridewithgps Garmin Sync page. However, using the exact same method Garmin Connect uses in the API, I was unable to get anything to work. Turns out, they blocked third party websites from using that portion of the API. Best guess says it is for bug testing purposes…It’s easier to debug something in a controlled environment you control, rather than hearing grief from all the developers using the API. In any event, they announced a beta developers plugin with support for FIT devices.

Announced in their forums here:

https://forums.garmin.com

ridewithgps might be one of the first sites to support direct sync with FIT devices!