Category Archives: openstreetmap

Mapping Cleveland’s Proposed Ward Boundaries of 2014

Monday March 25, 2013

Cleveland City Council President Martin Sweeney released the proposed ward boundaries for 2014. This is just one day before he presents them to be voted on in City Council.

City of cleveland issues this map. A JPEG. Not even georeferenced.
– it has no street names, all features (including rivers and railroads) are all styled the same. Nothing more.

This has sadly been characteristic of the City of Cleveland’s approach to open data, particularly spatial data…

Cleveland’s approach to open data, particularly in this instance, isn’t acceptable. Nor does it help foster a culture where civic hacking flourishes.

Great maps and other visualizations including this great slippy map of new Districts of NYC by WNYC (led by jkeefe) that inspired me to do this, shouldn’t be exclusive to the tech cultures that we usually hear (NYC, SF, CHI, Austin, SEA, PDX, on and on) or wherever Code for America stops in for the year.

Later that afternoon, Current Ward 14 Councilman Brian Cummins, had received PDF maps of most of the proposed ward and posted them on his blog.
A step above from what I had before. A Shapefile would be too much to ask.

(At least Kudos to him and my current councilman, Joe cimperman, for having twitter accounts and responding to their constituents on there. )

So I began by opening up a blank layer in JOSM, loaded in the Cleveland boundary from OpenStreetMap (less things to draw that way) and began simply tracing out the boundaries over Openstreetmap tiles. I was switching windows every couple minutes, looking at the JPEG boundary, then drawing the same lines in JOSM, repeat.

Had to be a better way, was going to take a couple hours (and it did).

(Side Question: What You use to draw geometries that you’ll later process in your maps/visualizations/analysis ? )

Behold: The georeferencing tool in qgis, which would let you load an image as a layer. This, I thought would be a shortcut. I could create the polygons of the wards by tracing right over the boundaries in the image, without hauving to switch windows.

The biggest problem was that I didn’t know the projection of the JPEG.
Unfortunately, these tutorials assume that your image is georeferenced.

To georeference in qgis, you should know what the projection of your original image is in before you start. If you don’t know, you’ll have to do some guessing and trial and error. I made a few guesses of the most popular projections (4326, 3857) and then tried several ohio ones. An hour or 2 later, none of the projections worked out.

So, I scrapped that idea, I began to draw the ways again in josm.
From there, I did my usual workflow which with I’m most comfortable into tilemill;

use osm2pgsql to convert my .osm file of boundaries (which were in the form of relations, specifically multipolygons) to load into a postgis enabled database.

(I’ve been meaning to become more comfortable with geoJSON and I would have tried to save my file in josm as json but I read there’s a bug in the json export of josm that doesn’t export relations correcty, I didn’t bother to verify this yet)

Next, I used Mapbox’s (which consists of data from OpenStreetmap) technique to create a custom map that I can use as my reference base layer.

Style my layer of proposed ward boundaries in Tilemill….

Then in mapbox.js, I simply put the two layers together. voila, as shown in my map at:

http://maps.jhfeichtnerfund.com/wards/index.html

Still more to do with this:

– finish up the documentation

– Tweak the colors
– Add in the census tracts so when a user hovers over an area, they can see the population of a particular place.
– add the existing boundaries too would be nice (as a separate layer for the online map)
– convert the 2014 ward boundaries (currently as a .osm) to a shapefile so others can use it.
Right now, it is available as an .osm in my github repo…

Follow this along in its github repo. https://github.com/skorasaurus/cleboundaries/

Advertisements

Cleaning up .GPX files for OpenStreetMap and Visualizations.

While going through the rest of my GPS traces from the HOT exploratory mission in Senegal [some write-ups on the HOT blog], I had found one GPS trace that has 2 problems that I’ve occassionally experienced while collecting gps traces for OpenStreetMap.

– The GPS signal is temporarily lost and the distance between 2 points of your trace is abnormally lengthy,
and cuts across roads and features.

An example: Did I time-travel across that area ? Nope. Did I time-travel across that area ? Nope.

– And what I call the ‘scribble effect’ (is there another name for it?), where I accidentally forget to turn off the gps after I arrive at my destination and end up with multiple points near each other, looking like a child scribbled in a coloring book.
An example:

If you’re planning to make any visualizations with your traces, these two symptoms allow viewers to misinterpret where you’ve been and make your visualizations inaccurate.

The scribble effect also negatively affects OpenStreetMap because GPS traces that are uploaded in the future in the same area as your scribbled GPS trace, other users will be able to not be discern the newer gps trace from your scribbled gps trace. There’s also still a few places on earth (including some regions of Senegal) that don’t have any satellite imagery to (legally) use for drawing ways for OSM.

So, after you complete a trace, how do you eliminate these symptoms described above?

I’ve tried a combination of different filters with gpsbabel – (explained more here)

but have yet to find a single command that would solve both issues.

gpsbabel -t -i gpx -f gpx_louga_03062012-withmay.gpx -x track,pack,sdistance=0.6k -o gpx -F test7.gpx
– this had fixed the temporary signal loss … but it didn’t fix the scribble effect.

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x discard,fixnone,fixunknown -o gpx -F test.gpx
(resulted in a blank GPX File.

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x discard,fixnone -o gpx -F test.gpx
(no effect)

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x position,distance=3m -o gpx -F test3.gpx
(this removed the long points between the 2 segments, but didn’t solve the scribble effect)

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x discard,hdop=5,hdop -o gpx -F test6.gpx
(no effect)

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x position,distance=10m -o gpx -F test8.gpx
– removed 1100 of the trace’s 2400 points and the , but still left me with the scribble effect.

I’ll continue to explore cleaning up the GPS traces with GPSbabel filters and hope to find a way but if you know of a specific filter in gpsbabel or any other way to do this, besides manually deleting the offending points in gpsprune, I’d appreciate it.

Update: Dec. 1, 2012. I also shared this in an OpenStreetMap Diary Entry in Sept. Other OSM users have also experienced this problem and hadn’t found a solution besides what I’ve described above. So, I’ve found that the test3 example above and manually removing the points in gpsprune (from the scribble effect) is the easiest way to clean up your GPS points.

Downtown Cleveland Parking Map

Upon hearing about the (ill-fated) proposed parking garage for the May Company building a couple weeks ago, I wanted to highlight the existing amount of parking that already exists downtown.

So, here’s a map of Downtown Cleveland’s parking,
here’s a map of Downtown Cleveland parking broken down by garage, underground, and surface, hosted by mapbox. The map design is based on osm-bright and the source is available on my github page.

(Update: There are still a handful of lots that are exclusively for some downtown apartment complexes – particularly on West 9th, that are unmapped at the moment).

Copying an existing postgis db to use in Tilemill

Furthering my exploration in cartography, I’ve been experimenting with Tilemill lately, particularly using data from openstreetmap [that’s in a postgis db].

[The rest of this entry assumes you’re at least familiar with Tilemill and its terminology].

I started learning to make my own Tilemill stylesheets [written in carto]. As of now, there’s only 3-4 other stylesheets for maps on a street level. (we’re talking like 0.25mile=72 pixels, roughly; enough to see several city blocks on your computer monitor]. (I also plan on comparing these in a future post in the coming days).

One of the most prominent ones is open-streets, a stylesheet made by developmentseed/Mapbox, the dev’s of Tilemill.

Although open-streets has been invaluable for me learning tilemill, I was questioning how they structured their stylesheet and wondered if there were any other approaches.

I then found high-roads by Michal Migurski, whose work and mapping I’m beginning to admire.

So, for the past couple days, on and off, I was figuring how to try out high-roads without making a whole new postgis database from scratch (preparing OSM data, etc, etc) and instead copying an existing one and using that copied db to experiment with using high-roads.

Caveat: You could use the same database in another Tilemill project, but I wasn’t sure if using views.sql from high-road would alter the database; and wanted to be safe than sorry.

So, after some trial and error, this what I did to do that:

Found this helpful page by Ramesh Natarajan concisely explaining how to export a single mysql db:
did the following to back up the postgis database whose copy I would use for high-roads:
(obviously, user-name is the one that is the owner of the postgis db):

pg_dump -U {user-name} {source_db} -f {dumpfilename.sql}

pg_dump -U myusername thecleve -v -f theclevebackup.sql

Log into your postgres account:
sudo -u postgres -i

created the new database that you’ll be using for your new Tilemill project:
createdb thenameofyournewdb (in the following commands, it's named thecleve-hr)

Then run the couple commands to prepare your mysql for postgis:
createlang plpgsql thecleve-hr
psql -d thecleve-hr -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
psql -d thecleve-hr -f /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
echo "ALTER TABLE geometry_columns OWNER TO skorasaurus; ALTER TABLE spatial_ref_sys OWNER TO myusername;" | psql -d thecleve-hr

(I ran this last ‘echo..’ command to move the ownership back to my postgis user name.. I know that I didn’t create my postgis user name correctly, so I have add myself as the
owner again…)

Then, I was going to restore the database…
$ psql -U myusername -d thecleve-hr -f theclevebackup.sql
psql: FATAL: Ident authentication failed for user "myusername"
nope, that didn't work.

Next idea, log out of postgres, and do it as my normal user name…

So I did…
$ psql -U myusername -d thecleve-hr -f theclevebackup.sql
[received tons of errors after this].

So I figured, let’s see if the new db works anyways.
psql “dbname=thecleve-hr host=localhost user=myusername password=mypassword”
I can connect to the database by….
but when issuing \d:
no relations are made... that’s a bad sign !

but on a whim, I decided to remove my username and try:
$ psql -U -d thecleve-hr -f theclevebackup.sql

and now the data is there after issuing \d !

Then, I just went ahead and created my project in Tilemill =)

Creating an extract of a city in Openstreetmap

I have been rendering maps of my local city [Cleveland, Ohio] recently. Because of Cleveland’s administrative borders, I had to extract the city of Cleveland using a large bounding box that included a lot of data outside of the city of Cleveland. As a result, I had to manually remove many ways and nodes (via josm) outside the city’s borders before I later used this data within the city’s boundaries for rendering in osmarender, and as a postgis db in tilemill and mapnik.

To automate this, I had been looking to create an extract of my city (and hopefully to be able to replicate this with other cities). This extract would only contain data within the city’s administrative boundaries.

I found a similar project to mine, The Extractotron – which is a great project, but its extracts are for cities including areas outside a city’s administrative borders.

UPDATE: March 2013; I finally figured out how to do this:

(this tutorial expects that you know OSM pretty well, how to use osmosis, and that you’re running OS X or linux).

1. Obtain the id for the relation of the boundary. (You can do this in josm by clicking on the boundary and then press cntrl + shift + i).

the way that is displayed in the webpage IS NOT the entire boundary for the city, it’s only a part of it. Instead, go down to the bottom of the new webpage and find “Part of: Relation: Name (THISID####)”

2. Obtain the .poly for this file. There’s multiple ways to do this, I find overpass, although powerful, the server can be overloaded, so I use this tool…

2A] Open this great tool by jocelyn. http://osm102.openstreetmap.fr/~jocelyn/polygons/index.py

Enter in the relation ID that you just found. Click, boom. Click Poly.

new window will appear, copy its contents into a new file, name it whatever you’d like with .poly at the end. for example, summit.poly

Use this polygon for the –bounding-polygon file=nameofyourpolygon.poly parameter in osmosis.

For example, if you just want the administrative boundaries within that area, you could you do:

osmosis --read-pbf file=ohio.osm.pbf --bounding-polygon file=summitcounty.poly --tf accept-ways boundary=administrative --used-node --write-xml summit.osm

or generally, if you want everything within it, do: osmosis --read-xml file="planet-latest.osm" --bounding-polygon file="county.poly" --write-xml file="australia.osm"

Voila, you’re done and have your OSM file.
========================================================================================================
(This is my old, original post and I left it up for posterity).

I have unsuccessfully tried the following:

1] Created an extract that contains the administrative borders of my city in its entirety from a larger extract (in my case, my state) from geofabrik

bzcat ohio.osm.bz2 | osmosis\

–read-xml enableDateParsing=no file=-\

–bounding-box top=41.606 left=-81.88 bottom=41.3903 right=-81.5323 clipIncompleteEntities=true –write-xml file=-\

| bzip2 > extracted.osm.bz2

2.] Filtered the extract again in osmosis to only contain the administrative boundaries [and highway motorway link as a reference for me]

Specifically I did:

osmosis –read-xml extracted.osm –way-key-value keyValueList=”highway.motorway_link,boundary.administrative” –used-node –write-xml entireclevelandboundingbox.osm

4. Opened the file in josm, selected the multipolygon that consisted of the city’s administrative boundary, and then right-clicked on it, Selected Members, and copied it to a new layer

5. In the new layer, Removed the tags from the city’s boundaries and then combined all of them into one single way [in one direction] and saved it as a new osm file.

http://dl.dropbox.com/u/17003223/clevelandboundaryonly-dell.osm

6. Used osm2poly to create a polygon:

perl ./osm2poly.pl clevelandboundaryonly-dell.osm > clevelandboundaryonly-dell.poly

polygon can be found:

http://dl.dropbox.com/u/17003223/clevelandboundaryonly-dell.poly

7. Use the polygon as the boundary from the first large extract [from the state of ohio].

bzcat ohio.osm.bz2 | osmosis\

–read-xml enableDateParsing=no file=-\

–bounding-polygon file=clevelandboundaryonly-dell.poly clipIncompleteEntities=true –write-xml file=-\

| bzip2 > extracted.osm.bz2

An example of these broken boundaries can be can see in a screenshot of josm at http://i.imgur.com/Tw8Ba.png

In this screenshot, the underlying gray line is an inactive layer in josm and the same OSM file that I used to create the polygon in Step 5. The

administrative boundary, shown as orange dashed line in the screenshot, is from the extract in step 7 and should be over the gray line.

Although we were unsuccessful, thank you to stereo in #osm for troubleshooting this with me yesterday.

Cleveland Neighborhood Map

The Cleveland Neighborhood Map was updated last week, check it out. I added Brooklyn Centre and Lake Erie is now included on the map.

Yes, Osmarender can render coastlines [sort of]!

There’s been a nagging error in the Cleveland Neighborhood Map: No Lake.

Lake Erie will be in the upcoming CNM update as I figured out how to configure my rendering software, or/p, (a perl implementation of osmarender) to include the coastline.

Here is how I did it:

The problem:

When downloading data from OSM, the geographic area is (not always, but for the sake of this post, most of time) shaped in a rectangle defined by the bounding a north, east, south, and west coordinates (known as a bounding box).

When it comes to coastlines in OSM, the entire area is not downloaded, just the data within the bounding box. This can be a problem if your data partially contains a coastline and the data does not specify which side is land and which is water !

Thus, here’s what you should do if you have a large coastline to render in osmarender.
(thanks to bobkare in the OSM IRC channel for the help).

  1. download the tiles@home client.
  2. Extract the t@h client, copy both ./close-areas.pl and the folder png2tileinfo to the folder that contains the .OSM file you wish to render.
  3. Run ./close-areas.pl on the .OSM file
    As follows: ./close-areas.pl minlat minlon maxlat maxlon < theosmfile.osm > output.osm

close-areas.pl will download the additional coastline that was not in the bounding box and specify the area(s) near the coastline which contain water.

After running close-areas.pl, my output .osm file, once rendered, still did not have water in it.
Why ?!
Through a couple trial and errors of running close-areas.pl, I also learned that the the bounding box order IS NOT the same as your minlat minlon maxlat maxlon . The bounding box order (used in the OSM API, [v.06]) is left,bottom,right,top sides of your bounding box;

but the parameters of the
minlat is the bottom (of your bounding box) (southern portion),
maxlat is the top/north;
minlon is the left/west;
maxlon is the right/east.

I had mistakenly entered the bounding box coordinates in the same order of the API, not the specific order used in close-areas.pl

The correct close-areas.pl was:./close-areas.pl 41.49503 -81.7291 41.51245 -81.69391 < input.osm &rt; output.osm

EDIT: I haven’t fixed it yet I was able to render the coastline using an area of bounding box that consisted of the mall portion of downtown Cleveland (the above coordinates), for the sake of simplicity during this debugging process. However, when I attempt to run the above using the bounding box of the entire city of Cleveland (the same used for the Cleveland Neighborhood Map), the water is displayed everywhere on the rendered image, instead of just Lake Erie.

Progress Indicator for the Cleveland Neighborhood Map

First off, a group Naplab, in Indy have made a great neighborhood map. Aaron Renn of the Urbanophile has an interview with the creators here. I hope to talk to them soon to kick around a few ideas with the Cleveland Neighborhood map and discuss our projects.

The Cleveland Neighborhood map has a background layer of roads, rivers, streams, and railways from Openstreetmap (OSM). However, this background layer of data from OSM was mostly based from the 2007 US Census Data. As I worked on the Cleveland neighborhood map, I noticed quite a few instances where the data (the roads no longer existed, were incorrectly named, or new roads have since been created, and didn’t mark any streets as one ways) was wrong.
Through the past few months, I have traveled around the city to verify the data in Openstreetmap and made many corrections. (This is also quite addictive too!)

July 18, 2011 edit: see the code map for more information

The green roads have been verified or corrected (as of June 16th)
red have not been verified. Also thanks to someonecallAlejandro (CWRU campus), badandy23 and anyone else in the OSM community who had contributed.

I plan on rolling out an updated version of the Cleveland Neighborhood Map in the next week or so, hopefully adding Glenville and Shaker Square.

(For me, back to rebuilding my sister’s garage!)