Accuracy of Garmin Forerunner 10 vs RunKeeper vs MapMyRun

Even though I’m the engineer in the family, my wife is more of a gadget freak than me. This year, she beceame interested in tracking her workouts.  She started out with a FitBit, and found that to be rather useless, so we returned that. Lately, she’s been into training for a half marathon, so we started using GPS workout tracking apps on an iPhone 5 and iPhone 4S.

We played with several different apps:  Kaiser EB Walk, Nike+, MapMyRun, NeoTreks, and Runkeeper. It’s interesting how you can run several apps simultaneously on an iPhone, and get wildly varying metrics from them.  The distance and pace vary quite a bit between apps, and I think the main reason is that the realtime geodata coming out of the iPhone is crap. Therefore, the app developers implement different filtering strategies on the data, leading to the different calculated metrics.

I find it amazing that the iPhone has an onboard GPS, accelerometer, and compass, and still manages to output such bad location data. Granted, the GPS antenna is quite small, but GPS watches are pretty small these days, and work much better. My guess is that Apple needs to do some work in iOS’ sensor data processing code. If only they would give us access to raw GPS data, rather than forcing us to use their garbage location API’s…

Following the recommendations of friends, my wife recently bought a Garmin Forerunner 10 (FR10) GPS watch.  I was skeptical that it was worth the $130, price tag, and was very interested in finding out how much better it would perform than the iPhone.  After using it for only a few days, I must say that I am very impressed.  Unlike my 10 year old Garmin eTrex GPS, the FR10 gets a fix very quickly, and manages to keep the fix even under moderate tree cover and in canyons.

Below, I will compare the simultaneously recorded outputs of RunKeeper, MapMyRun, and the FR10.  I chose RunKeeper and MapMyRun as the iPhone apps, because they offer easy access to the logged track data in the form of GPX files, and both websites allow you to upload/edit track data.  Note that the MapMyRun and RunKeeper data were recorded simultaneously on my iPhone, so they share the same sensor data.

Route plotted on Google Maps:

google

Calculated distance: 1.3mi

RunKeeper

rk

Despite the fact that the route is largely in a flat area, with very little tree cover, notice how the track wiggles and meanders.

MapMyRun

mmr

The MapMyRun track, as expected, also wiggles quite a bit.

MapMyRun Track on RunKeeper.com

mmrrk

The above plot is taken from a GPX file exported from MapMyRun.com of the workout, as recorded by MapMyRun. Notice how RunKeeper.com, using the same track data, calculates the distance/pace/calories differently. I have found that MapMyRun consistently calculates the longest distances, and highest calorie count of the bunch.

Garmin FR10 on Garmin Connect

gc

 

Note how the Garmin track is very clean and smooth, and tracks the roads reasonably closely.

Garmin FR10 Track on RunKeeper.com

fr10rk

The above plot is of GPX data exported from Garmin Connect, and then uploaded to RunKeeper.com. The FR10 stores the track data in a proprietary FIT format, but luckily, their website is able to output GPX. Again, notice that RunKeeper.com used the same data, and yet, calculated different results.

Comparison

Distance (mi) Calories
RunKeeper 1.40 153
MapMyRun 1.51 178
FR10 (Garmin Connect) 1.28 81
MapMyRun Track on RunKeeper.com 1.44 147
Garmin FR10 Track on RunKeeper.com 1.29 121
Google Maps 1.3 N/A

Below are some data from a couple of other routes that I tested:

Euston Loop

Distance (mi) Calories
RunKeeper 2.53 279
MapMyRun 2.71 306
FR10 (Garmin Connect) 2.44 156
MapMyRun Track on RunKeeper.com 1.44 147
Google Maps 1.3 N/A
Manual Plot on RunKeeper.com 2.55 N/A

Winston Loop

Distance (mi) Calories
RunKeeper 1.79 186
MapMyRun 1.83 205
FR10 (Garmin Connect) 1.66 103
Manual Plot on RunKeeper.com 1.73 N/A

 

Discussion

While my sample size is small, some trends emerge.  The Garmin FR10 consistently calculates the shortest distances and calories.  MapMyRun consistently calculates the longest distances and highest calorie counts.  There is a difference between the calculation methodology between RunKeeper.com and MapMyRun.com, because given the same GPS track, MapMyRun calculates a longer distance (1.51 vs 1.44mi) and a higher calorie burn (147 vs 178cal) than RunKeeper.com.  On the other hand, Garmin Connect calculates shorter distances and calorie burns than RunKeeper.com, given the same track.

I have tested several other times, and the above trends hold.  Furthermore, the Garmin FR10 gives the smoothest, most accurate GPS track, especially when in a canyon, or under heavy tree cover. Below is a canyon track, Garmin FR10 in red, RunKeeper in blue:

ecrkfr

Notice the vastly improved accuracy of the Garmin FR10 track vs RunKeeper running on the iPhone.

Finally, a canyon track which was recorded on my iPhone 4S, running RunKeeper and MapMyRun simultaneously. RunKeeper is in red, MapMyRun in blue:

ecrkmmr
The image above shows the vast differences which sometimes occur between the way that RunKeeper and MapMyRun interpret the same location data stream.  Neither track closely follows the actual path.  It’s pretty obvious that the Garmin FR10 is a worthwhile upgrade from using the iPhone as a sensor.

Next Related Post: Garmin Forerunner 10 vs Pyle PSWGP405BK

7 thoughts on “Accuracy of Garmin Forerunner 10 vs RunKeeper vs MapMyRun”

  1. I’ve been using RunKeeper for a couple of years, and have had only sporadic issues related to location tracking (maybe because I live in NYC and the GPS tracking is, I would imagine, coming from quite a robust sample), however I’ve been thinking about switching to a Garmin device for one simple reason–to save $$ on data usage every month. I believe at least 50% of my data is used maintaining an uplink to RunKeeper.com while I’m out on the road. So, I’ve been looking for a single comprehensive article to point me in the right direction, and congrats–this one is it. Thanks for the clarity, and for taking the time to spell all this out.

    1. John W:

      If you’re on an iOS 7 device, go into your Settings->Cellular and scroll down. There will be an entry for Runkeeper which will tell you exactly how much cell data the app has used since the last manual data reset. There is also a slider next to the app to turn off its ability to use cell data (so it’ll just use WiFi as available). Scroll down further for the button to perform a manual reset of the stats. I’ve been using Runkeeper regularly for over a year and it has used less than 30MB of data in all that time. It does not seem to use much data (if any) during a run.

      OB Article: I too am puzzled how variable my run tracks are. I suspect it’s partly due to the compromises that must be made when managing the power used by all the competing apps on such a general purpose device as a phone. I wonder how much of the positioning during a run is established from WiFi triangulation and how much from GPS/accelerometer measurements. A fair bit of the inaccuracy I am seeing comes from nonexistent altitude changes (which will only tend to lengthen the track, never shorten it, and might explain why the Garmin gives consistently shorter tracks). AFAIK altitude is only established from a 4 or more satellite GPS lock which implies that the phones are just not that good at maintaining real GPS satellite data streams. I suspect the power-use tradeoff is a large factor in the choice within phones to use more WiFi data than constant and consistent GPS.

      This may explain why phone tracks are so poor in canyon runs where WiFi triangulation is likely non-existent (i.e. pure phone GPS is used).

      Thanks for the research and well written article. I hope Apple sees it and takes note.

      1. Thanks for your feedback. It doesn’t matter how many competing apps are using the location data … the way that Apple does it is that they have a location engine in IOS, which processes the data, and then makes it available to all of the apps simultaneously.

        Yes, you’ll get slightly better accuracy in areas with WiFi, but I don’t think it will help significantly in the accuracy. WiFi fix will only get you into the general area. You’re right that you need at least 4 satellites for a 3D fix (altitude data), but two things I would like to point out: 1) GPS altitude fixes are inherently bad quality, no matter how many satellites you use for your fix. AFAIK, Runkeeper and MapMyRun use their own terrain data to determine altitude, rather than your GPS data. 2) if you have only 3 satellites fix, your location fix will be horrible accuracy, anyway.

        The reason that the tracks are so poor in canyons is not due to lack of WiFi signal.. GPS signals are hard to receive when inside canyons. I suspect that the Garmin GPS watch has a much bigger antenna, and it also benefits from not having the extra interference of the phone’s radio. Even so, as a software developer who implemented one of the first portable GPS navigators, I wish Apple would give access to the raw GPS data stream. I am sure that I could get better accuracy out of it than the location API’s are sending.

  2. Hi, have you done any further comparing of Garmin Forerunner 10 vs. MapMyRun on the iPhone? I fear I’ve noticed the same discrepancy, with MMR overestimating the distance covered and pace (I say ‘fear’, because I used it for all my runs before recently getting a Garmin). Today they marked the first mile of the run exactly the same, but by the end of a ~6.5 mile run, MMR said I’d covered 6.75 miles while Garmin said I’d covered 6.2 miles (at a commensurately slower pace, of course….) Thanks for your analysis above.

    1. No, I haven’t done any more controlled tests. But in general, the smartphone apps tend to overestimate the distance. This may be partly due to the wobbly GPS track, which increases the distance.

  3. I have been using mapmyrunner on my smartphone until I bought a new Garmin FR 15. I compared the two today and Garmin FR 15 had terrible wiggles compare to mapmyrun. I don’t know what’s going on. I expected smoother routes like you have shown with FR 10. I would like to share the screen shots of the two but I don’t know how to post pictures here. But wow! I am very disappointed with FR15…

Leave a Reply to Alen S.Cancel reply