User-editable Road Maps Downloadable as Shapefiles

[Scott Lincoln]

I posted this thread on the GR owners forum and thought it might be of interest here. The shapefiles mentioned can be downloaded and added to GR products.

OpenStreetMap Project
I wanted to bring everyone's attention to something new for road shapefiles - the OpenStreetMap project. The OpenStreetMap (OSM) project is a type of Volunteered Geographic Information (VGI). It is database of global road, trail, water, locations, and boundary features freely editable by registered users, just like Wikipedia. For areas in the United States, the OSM project uses the TIGER Census shapefiles as a starting point, but this data can be corrected or have new roads added much more frequently when open to the public. To view the interactive maps rendered weekly from this database in a manner similar to Google Maps, visit the OSM website: http://www.openstreetmap.org

You will notice right away some of the obvious inconsistencies in road types that come from using the TIGER Census shapefile data. I have already been working to correct some of these issues for the central Iowa region, especially around Ames. To make changes to the database, you must first register an account so they can track who makes what changes. After that, you simply have to go to the "Edit" tab and a flash editor pops up. Once you make a change to a line or point, it is saved automatically. Before making any changes, I suggest you use the sandbox editor because changes made are put into the database right after you make them.

CloudMade Downloads
A website called CloudMade uses data from the OpenStreetMap project for mapping projects and even includes a simple (with a some bugs) trip router. The most fascinating and useful product they produce, however, is a set of downloads rendered from the OSM database. They have downloads for handheld Garmin devices and for other mapping programs, as well as the ability to download a statewide shapefile of the roads. Here is the download page for the Iowa data:
http://downloads.cloudmade.com/north_america/united_states/iowa

In theory, map enthusiasts could keep the map database updated for everyone, and have the new results rendered each month.

The Shapefiles
Here within the next few weeks, I will be using ArcGIS to clip these state-wide road shapefiles (updated April 2009) to some of the local WFO boundaries or to the WSR-88d radar ranges. I have yet to decide which would be more useful. I also will be developing a style file that can display shapefiles from CloudMade's OpenStreetMap project renderings. When I get this done I will share it with people if there appears to be interest in this project.

For People Editing Maps
Since this project was started in Europe, they have slightly different ways of describing road types than we do here. These are the road types I have been using. We should try to be as consistent as possible.

Motorway - Interstate highways or highways that are four-laned with controlled access more than 1 mile at a time
Motorway link - Interchange ramps for a motorway
Trunk - Four lane highways that do not have controlled access
Trunk link - Interchange ramps for a trunk
Primary - Federal highways, 2-lane
Primary link - Interchange ramps for a primary, such as a super2
Secondary - State highways, 2-lane
Tertiary - County or other local, numbered highway. 2-lane
Residential - City street, county road, gravel road.
Service - Road that is not supported by government maintenance. A minimum maintenance county road. (I rarely use this)

[Scott Lincoln]

DMX roads shapefile, 150mi range
Data from OpenStreetMap project, updated April 15th, 2009. Style file included.
http://www.meteor.iastate.edu/~slincoln/GRS/shapefiles/DMX_OpenStreetMap_Roads.zip

Style file for OpenStreetMap project shapefiles
http://www.meteor.iastate.edu/~slincoln/GRS/shapefiles/OpenStreetMap_Roads.sty

[CZandbergen]

Scott,

This is just what I have been looking for. Do you have links to any states other than Iowa?

[Scott Lincoln]

I just got done writing an ArcGIS model that I'm going to batch-produce several more shapefiles. Right now I only have radar sites surrounding Iowa in the queue. After I get those done I'll work my way down Nebraska, Kansas, and Oklahoma if people find them useful.

[CZandbergen]

I just got done writing an ArcGIS model that I'm going to batch-produce several more shapefiles. Right now I only have radar sites surrounding Iowa in the queue. After I get those done I'll work my way down Nebraska, Kansas, and Oklahoma if people find them useful.

Good deal. I look forward to seeing what you come up with. Keep us in the loop!

[Scott Lincoln]

I still havent figured out the best way to keep these updated, but for now I have some sites surrounding Iowa completed:
http://www.meteor.iastate.edu/~slincoln/GRS/shapefiles/ (http://www.meteor.iastate.edu/%7Eslincoln/GRS/shapefiles/)

Since the OpenStreetMap database can be updated on a continuous basis, and because CloudMade re-compiles the state shapefiles every month or so, I need to create an automated way to do this in ArcMap. That way when a new update comes out, I can download the files, and over a weekend just start the script and have it create the shapefiles we want.

[Scott Lincoln]

I finished building and running the ArcMap script/model this afternoon and evening. Right now it looks like it is going to be a two-step, 1.5 hour process to update the shapefiles. I now have road shapefiles for 25 NEXRAD sites in the Upper Midwest and Plains.

Depending on my free time, I may be able to take and process other requests and add them to the script/model. CloudMade re-renders the shapefiles from the OpenStreetMap database roughly every month. If there appears to be major updates to roads (especially if someone lets me know that they have made major changes to roads via the OpenStreetMap project), then I will update the site specific shapefiles at that time.

[CZandbergen]

Thanks Scott. This is great. For those that haven't tried this yet you will love it. Same roads as Delorme or Google Maps, but in GR3. Really great stuff.

[Scott Lincoln]

I will probably add a KLSX shapefile to the mix later today, based on the chance for significant severe weather tomorrow.

Update: St Louis NEXRAD (LSX) now added to the site-specific shapefiles for GR.
http://www.meteor.iastate.edu/~slincoln/GRS/shapefiles/ (http://www.meteor.iastate.edu/%7Eslincoln/GRS/shapefiles/)

[Jordan Bell]

Hey Scott,
How do I get these exactly to work in GR3. I can get them unzipped and am confused from there.

Thanks

[Jeff Duda]

Jordan,
Just move the files into your GRLevel3 folder and then load the shapefile and stylefiles in GR3 itself.

[cdcollura]

Good day all,

Jordan,
Just move the files into your GRLevel3 folder and then load the shapefile and stylefiles in GR3 itself.

I tried this myself and here is the problem...

1). I tried the TLX files and put them in the Grlevel3 directory.

2). In Grlevel3, chose load shapefile.

3). I loaded the TLX (*.shp file) and chose the "tigerroads.sty" file and clicked "import".

4). Go to the TLX radar site.

5). Nothing happens ;-(

Note: There were no *.sty files other than "tigerroads.sty".

Something I am missing?

[Jeff Duda]

You should see a file called "openstreetmap_roads.sty" stylefile that you can open in conjunction with the given road shapefile. Use that as your stylefile instead of tigerroads.sty. If you don't see that file available, then redownload the package and make sure to copy ALL files in the downloaded folder into your GRlevel3 folder.

[Jordan Bell]

cdcollura,
put this link in the stylefile :
http://www.meteor.iastate.edu/%7Eslincoln/GRS/shapefiles/OpenStreetMap_Roads.sty

are the roads not labeled?
Great product

[Jeff Duda]

I've noticed it causes my versions of GR2 and GR3 to crash occasionally while they're loaded. I wonder if that has to do with the shapefiles themselves or if some random conflict is occurring with my software and other components of my system.

[cdcollura]

Good day all,

cdcollura,
put this link in the stylefile :
http://www.meteor.iastate.edu/%7Eslincoln/GRS/shapefiles/OpenStreetMap_Roads.sty


Thanks, that worked!

[Scott Lincoln]

I've noticed it causes my versions of GR2 and GR3 to crash occasionally while they're loaded. I wonder if that has to do with the shapefiles themselves or if some random conflict is occurring with my software and other components of my system.

Someone else mentioned that to me as well. Best guess is that the size of the shapefile is causing issues. It is definitely a large file. Mike Gibson has optimized the loading of GIS shapefiles so that the program only loads the files that are visible in the main window, and only the road features in the file as specified by the zoom levels of the style file.

By merging all the features from individual shapefiles into a massive shapefile like this, a larger block of memory will be required at any given time. I have noticed, however, that the combined size of all of the smaller shapefiles is much larger than the size of these individual files.

I have yet to have any crashing issues on any of the machines I have loaded these on, but if many people start reporting issues, the files will lose their usefulness. Perhaps I could reduce the range from the radar used to clip the roads? Reducing the range from 150miles to 140miles, for example, will lower the area included in each file by ~9100 sq miles, or 13%.

[Scott Lincoln]

are the roads not labeled?
Great product

Thanks.

As far as roads not being labeled.... Some may not be. The style file I wrote specifies which field in the shapefile to use for the status bar label, and as far as I know, I set that to be correct. Theoretically, the names for the roads should be the same as with the TIGER Census shapefiles, since the OpenStreetMap Project used those files as the starting point for mapping the United States. Although there are many more possible uses available when using the OSM data, the accuracy will likely suffer at first until people step up and edit the roads. Perhaps people who notice errors could create accounts on OpenStreetMap.org and make the necessary corrections so that we may all benefit?

[Scott Lincoln]

Has anyone done any major edits to the roads database via OpenStreetMap? I just noticed that CloudMade did another render/export of the database into shapefile data again recently. If any major changes have been made, I could recreate the site-specific files for the affected areas.

[cdcollura]

Good day,

I will probably add a KLSX shapefile to the mix later today, based on the chance for significant severe weather tomorrow.

Update: St Louis NEXRAD (LSX) now added to the site-specific shapefiles for GR.
http://www.meteor.iastate.edu/~slincoln/GRS/shapefiles/

The link to LSX does not work...

Thanks for putting all this out there for us!

[Mike Hardiman]

An interesting project for sure.

One complaint...Several Texas Farm/Ranch-to-Market roads are labeled as "residential" when they're really more like 2-lane highways. Since the maps are based on Tiger 2008 shapefiles, the "Name" fields are strife with naming inconsistencies. For example... one road may be labeled "Farm-to-Market Road 1111" in one place, then "FM 1111" farther down the line, then "CR 1111" ... etc.

Also, the state highways tend to "disappear" in towns as they become "residential" roads.

[Scott Lincoln]

The link to LSX does not work...


It's fixed now. Mis-typed one character in the ZIP file name.

[Scott Lincoln]

One complaint...Several Texas Farm/Ranch-to-Market roads are labeled as "residential" when they're really more like 2-lane highways.

My suggestion would be to go in to the Open Street Map project, register a username, zoom in to areas of interest, and change the road type from "residential" to "tertiary." Tertiary seems to be the category used for county/local highways. Residential, however, is being used as a rural road as well and shouldn't just imply city streets.

Since the maps are based on Tiger 2008 shapefiles, the "Name" fields are strife with naming inconsistencies. For example... one road may be labeled "Farm-to-Market Road 1111" in one place, then "FM 1111" farther down the line, then "CR 1111" ... etc.


Unfortunately, there is not much I can do about that. The only thing to do is to go and edit those road names to something consistent as you find the errors. That is what I have been doing. Since alot of the road information came from the TIGER data, this should be the same regardless of your shapefile source. At least with the OpenStreeMap project data you have the ability to fix it. The other problem is that alot of this data was digitized locally by different counties and states, and they each have different ways of doing it, and to different degrees of accuracy.


Also, the state highways tend to "disappear" in towns as they become "residential" roads.

Do you know of a place in particular where this happens? This doesn't make alot of sense to me, since the roads should be the same regardless of the zoom. If you can give me more info on a specific example, I can try and look into it.

[cdcollura]

Good day,

It's fixed now. Mis-typed one character in the ZIP file name.

Thanks again...

[Darren Addy]

Best guess is that the size of the shapefile is causing issues. It is definitely a large file.

Would it be more accurate to say that the problem is the amount of RAM needed to run with your shapefiles?

Might be interesting to know the system configs (OS, RAM, etc) of those having problems (and how many programs they are running concurrently - including spyware that they may not know they are running, etc.). Then one could come up with some minimum system requirements, in order to make use of the shapefiles.

[Scott Lincoln]

Would it be more accurate to say that the problem is the amount of RAM needed to run with your shapefiles?


My best guess is that it is either a RAM issue or an issue with not having the most up-to-date GR version. I'm not sure what types of things will cause GR to crash, but I suppose we could make a best guess if it seems that the RAM capacity of some users with crashes are lower than that of those that do not experience crashes.

[Jeff Duda]

I've actually been having a similar problem, although it only seems to occur when I'm looking through archived data. Some of the images will become blank white discs of about 130 nmi around the site, then after about 60 seconds, GR2 will crash. It only happens when I have the shapefiles loaded. My computer has 4 GB of RAM and a 512 MB graphics card, so I don't see how I could be lacking in memory.

[Scott Lincoln]

I've actually been having a similar problem, although it only seems to occur when I'm looking through archived data. Some of the images will become blank white discs of about 130 nmi around the site, then after about 60 seconds, GR2 will crash. It only happens when I have the shapefiles loaded. My computer has 4 GB of RAM and a 512 MB graphics card, so I don't see how I could be lacking in memory.

Get me all the files needed to replicate the situation and I will try and see what it takes to make the problem happen on another machine.

[Scott Lincoln]

All available sites now updated to May 20th, 2009, roads update.

[J Kinkaid]

All available sites now updated to May 20th, 2009, roads update.


Thanks for the update and keep up the good work:D

[Blake Michaleski]

Excellent work!

With today being the first day of hurricane season, are there any plans to set this up for the gulf coast and east coast sites? Obviously not as useful in hurricane chasing as in plains chasing but would love to finally integrate my navigation and radar into one program, obviously the GR products being the base program.

[Jim Hunt]

I would love to see KIND and KIWX done too.

[Scott Lincoln]

With today being the first day of hurricane season, are there any plans to set this up for the gulf coast and east coast sites?

At this time I have just planned to do sites in tornado alley. Although most of the tedious, annoying labor is done by ArcMap automatically over the course of like 4 hours, it still takes some time and labor on my part to download each of the zipped-up shapefiles and extract the files I want. I might try and automate this in the future when I can figure out how.

Because of that, I probably wont add any hurricane sites to my automatic queue at this time. Perhaps if I have some free time some day I might do some shapefiles for the Gulf Coast broken up into larger pieces than the radar-specific ones - but on a non-updated basis. If you want files for that area now (and have a computer that can handle big shapefiles), you can download state-wide shapefiles from the CloudMade site. They will work with the style file I created.

http://downloads.cloudmade.com/north_america/united_states/

[Scott Lincoln]

I would love to see KIND and KIWX done too.

I should be able to get you KIND for sure here pretty soon as I already download the road data for Indiana so that I can cover the Illinois sites. KIWX, I'm not sure about yet.

[Scott Lincoln]

Updated to add KIND, as per request.

http://www.meteor.iastate.edu/~slincoln/GRS/shapefiles/ (http://www.meteor.iastate.edu/%7Eslincoln/)

[Scott Lincoln]

After a few more requests off of the GR forums, I decided it would be best if I tried to semi-automate the whole process of creating the shapefiles. This way I could keep them updated more easily with less work from me, and I could also add more sites. With that in mind, I decided to go all out, and doubled the number of available sites/shapefiles.

MAF, SJT, GRK, DYX, FWS, SHV, LZK, NQA, PAH, VWX, IWX, ILN, GRR, DTX, CYS, PUX, FDX, GGW, GRB, MKX, LOT were added.

I also added available shapefiles from Canada to the sites with ranges that extend beyond our northern boundary.

The coverage area is now basically the entire plains and midwest. The entire update process now takes about 6 hours, and is over 3GB in size.

http://www.meteor.iastate.edu/~slincoln/GRS/shapefiles/ (http://www.meteor.iastate.edu/%7Eslincoln/GRS/shapefiles/)

Please let me know if there are any problems.

[Jim Hunt]

Thanks Scott for the work and adding Indiana.

So I am not seeing any road names. Is there a setting I have wrong?

[Scott Lincoln]

People have mentioned to me before about the road names. As far as I know there isn't much that can be done. I've set the style file to place the "name" field of the shapefiles in the status bar of GR. I have noticed that some roads seem to have this blank. I can double-check to make sure that none of the operations that I do in ArcGIS take away the road names, but I am pretty sure that it comes that way from the database.

Since it is a community supported and updated mapping database, the best I can say is that if you see important roads that are unlabeled, go in an edit them on openstreetmap.org using the "edit" tab. Make a new field down at the bottom, and type the name in. It should save automatically and after a few weeks when I do a new update, it will be available to everyone.

[Jim Hunt]

People have mentioned to me before about the road names. As far as I know there isn't much that can be done. I've set the style file to place the "name" field of the shapefiles in the status bar of GR. I have noticed that some roads seem to have this blank. I can double-check to make sure that none of the operations that I do in ArcGIS take away the road names, but I am pretty sure that it comes that way from the database.

Since it is a community supported and updated mapping database, the best I can say is that if you see important roads that are unlabeled, go in an edit them on openstreetmap.org using the "edit" tab. Make a new field down at the bottom, and type the name in. It should save automatically and after a few weeks when I do a new update, it will be available to everyone.

Doh! I was not looking there. Sorry.

[Scott Lincoln]

Just did a new update that fixes several little bugs reported here and on the GR product forums.

-Shapefiles from Lousiania will now show up on Shreveport NEXRAD
-Shapefiles from Ohio will now show up on Michigan sites
-Many files sizes have been reduced as I realized that my batch script is putting shapefiles into the zip files twice in some instances
-Shapefiles now on par with the June 3rd, 2009 CloudMade update

[Danny Neal]

The requested URL /~slincoln/GRS/shapefiles/LOT_OpenStreetMap_Roads.zip was not found on this server.

I tried LOT and got this message just an FYI..... Appreciate all your hard work Scott.

[Scott Lincoln]

The requested URL /~slincoln/GRS/shapefiles/LOT_OpenStreetMap_Roads.zip was not found on this server.

I tried LOT and got this message just an FYI..... Appreciate all your hard work Scott.

Should be fixed now.