Follow along as I create a map of where I’ve been in 2013

(Follow along in my process of creating a simple heat map – from initial idea, to brainstorming, trial and error, coding, designing, and more trial and error in between)

Before I start:
goal: create a heat map that displays all of my traces of where I was in 2013.

what’s the map’s purpose: – create something purdy and find out where I have been the most in 2013.

Context: In 2013, I had taken 140 gps traces* using osmtracker, my favorite gps logging software, on the android. Nearly all traceswere taken while I was driving or riding my bicyle.
(This does not include traces that I had taken in Haiti in May-June while with the Humanitarian OpenStreetMap Team.

Here’s some brainstorming:
– what tool to use? Most of my projects and cartographic exploration have been with Tilemill.
In this case, using tilemill isn’t a smart move here, because I had hundreds of layers that i’d have to manually add.

I thought of manually combining all of the traces together – which I did using this simple bash script and a
gpx file manipulator called gpsbabel:

gpsbabel -i gpx $(echo $* | for GPX; do echo " -f $GPX "; done) \
-o gpx -F appended.gpx

I added the merged gpx file as a new layer to a tilemill project and received an error:
“This datasource has multiple layers:
(pass layer= to the Advanced input to pick one)”

Instead of looking for a workaround for the error, I thought about, hey, let’s give cartodb a shot!
I have been looking for an excuse to use cartoDB for ages. I had played around with it minimally over the past year and a half, but my comfortability with Tilemill and the fact that I simply had not had a project where I thought it would be a
really appropriate tool for the job, hadn’t come up.

Let’s give cartodb a shot now…
uh-oh :( )
appended.gpx is too large (it’s 5.6mb). You can import files up to 4.53 MB.

Perhaps I should simplify the GPX traces I thought? I also thought, let’s just convert it to geoJSON and see if geoJSON would resize it’s size… so using ogr, I thought to do:

ogr2ogr -f "GeoJSON" traces.json appended.gpx
ERROR 6: "GeoJSON driver doesn't support creating more than one layer"
ERROR 1: Terminating translation prematurely after failed
translation of layer routes (use -skipfailures to skip errors)

What?! I thought “what do you mean by multiple layers?”
– Does ‘layers’ in the error message mean geometry types (points, linestrings) ? or multiple geometry collections? Some extensive google searching just led to the gdal source code which didn’t give me any more clues..

I walked away for an hour or 2, then thought something might have been wrong in my gpx file.
I opened the gpx file in qgis, and saw, hey it asked me which layer to open of the file and it referred to:


The layers created from my GPX file consisted of 5 layers, including points where I made voice recordings, text notes, or had taken a picture. The layer that I want – the linestrings that I recorded, ended up being the Tracks layer. (The other layers: routes was blank; track_points were the points that make up my linestring; waypoints were points where I had taken a voice recording, text notes, or a picture)

My traces and points in qgis

So, I thought I could just specify the layer within my geojson with the following
ogr2ogr -clipsrclayer tracks -f “GeoJSON” tracksonly.json appended.gpx

I thought this would select the layer that I’m exporting to, but it does not.

Alas, that didn’t work, I received an error as mentioned earlier in the post.

So, fellow readers:

Any suggestions for tools to use or workflow for this?

(Updates to this post will be made as I progress along)

What I plan to do :

– Some more google and gis.stackexchange searching for some inspiration.
– save the gpx file (selecting only the tracks layer) as JSON in qgis and open in
cartodb and/or Tilemill.

I also wonder:
Simply convert each of them individually to geojson files, and then add all of them as a layer in a leaflet instance?

Update 1, 2014/01/02, 7pm:
– I converted the tracks layer in my gpx file to JSON (only 1.3mb!) and created a cartodb project for it . Success!
– I’ve also created a tilemill project and loaded my JSON in there. (I’ll post links soon)

I thought of a new idea: increasing or changing a section of the line based on its proximity to another line…
For example, if there’s another line within 60 meters of it at a specific point, increase the intensity of color of the line for a few meters…

This cannot be done out of the box in tilemill or cartodb… but I have a hunch that this sort of calculation could be done in postgis… I would then reimport this data into cartodb and style there.

HMM, this also looks interesting –

Post a comment or leave a trackback: Trackback URL.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s