CivicMaps Tile Engine v0.5

Status | Overview | Installation | Technical | Resources | Contributers | Download License |
a pretty compase rose


Status

Overview


This is a web mapping engine that you can put in your site to present maps with clickable thumbtacks. It has these features:

Installation

  1. Download
    . Get the latest version at tile.js. Put this on your website.
  2. Copy HTML
    . Look at the source code for this page to see how to insert it in your own page - or read the comments below.

If I was sticking this in my own site I would include tile.js like so: 

		HTML
			HEAD
				SCRIPT type="text/javascript" src="tile.js" SCRIPT

You will also want to type this code into your html page on your site (optionally also providing your own real feed url instead of my example): 

		DIV id="tiles1" style="border: blue 2px dashed; WIDTH: 512px; HEIGHT: 256px;" DIV
		SCRIPT
			new tile_engine_new(
				'tiles1',
				'FIT',
				'local_url_to_your_feed.xml',
				'http://civicmaps.org/cgi-bin/mapserv?map=/www/sites/civicmaps.org/maps/world.map&service=WMS&WMTVER=1.0.0&REQUEST=map&SRS=EPSG:4326&LAYERS=bluemarble,landsat7,lakes,rivers,cities,majorroads,minorroads,tiger_polygon,tiger_landmarks,tiger_lakes,tiger_local_roads,tiger_major_roads,lowboundaries,boundaries,coastlines&FORMAT=image/jpeg&STYLES=&TRANSPARENT=TRUE',
				0,0,0,0,0
				);
		SCRIPT

Technical

API

Application Stack

This mapping application can be thought of as a vertically integrated 'stack' of components. Each of the pieces is built upon services provided by pieces beneath it. You are welcome to just use the javascript core portion of the stack. This is very easy to do and is described in the "Install" section above. However some users may also want to do their own CMS integration or even have their own copy of the map data running on their own hardware and may also want their own tile caching mechanism.

Integration into Content Management Systems

This phase is the most interesting but is also not critical if you just want to do simple website integration. Please skip this part of the stack if you're not doing CMS related work.

This engine may be run within any website. It may also be integrated into a CMS. Since the map data is provided off-site there is no need to copy map data to your own server.

To plot thumbtacks your website or content management system will need to produce an RSS/XML or RDF/XML feed with at least geo:long and geo:lat tags as shown here:

    Locative Packet

Our research here was focused on ultimately doing a CMS integration with Drupal and Civicspace which is demonstrated here:

    CivicMaps

Our Drupal / Civicspace integration leveraged Ankur's Location Module. This provides a way to decorate any blog entry or user with location information. It uses a zipcode based geocoder. There were a few small gotcha's in this module that were corrected and are included below. These issues may trip you up if you are trying to integrate this mapping solution into CivicSpace - briefly they are as follows:

Building your own CMS integration on top of this foundation should take you less than a week at worst case.

Here is our time including the learning curve - hopefully we save you this time:

Javascript Tile Engine

The core of the application is a lightweight javascript application that runs in both Internet Explorer and Firefox. The approach is similar to the one taken at SVG Tile Engine which I wrote last summer. The difference is that this one talks to conventional WMS compliant mapping sources rather than a pre-tiled blue marble database and relies only on Javascript - not on SVG. This javascript engine is actually just a straight port of a java client based tile mapping engine which is visible at Java Tile Engine . The problem with the java applet approach however is that it cannot do cross domain image loading due to flaws in the security policy of java.

Javascript is not supported across all browsers but it is becoming increasingly present, stable and compliant. A 'virtuous circle' effect is taking place where as more people rely on Javascript, the javascript support improves. Recently the term used to talk about javascript is 'ajax'.

Mapserver

Mapserver is a small C based cgi mapping program that can publish jpeg based maps based on supplied ESRI shapefiles. It has a number of key features that make what we do possible including support for WMS queries, spatial indexing of shapefiles, stylistic rendering of vector data and support for proxy access to other third party WMS servers (such as the demis blue marble server).

Here is an example of a typical WMS query that the Javascript application will issue (I simply right clicked on a tile in the tile mapping engine and did a 'view properties' to see the URL of that tile):

   An example Tile Query

The configuration file that I use to configure mapserver is available here:

   world.map

Data

Organizing the data for Mapserver is probably the most tedious part of providing a mapping solution.

All of the prepared map data can be downloaded directly here:

   CivicMaps Map Data Folder

To prepare the VMAP0 data ( Vector data of the world coastlines, rivers and major highways among other things) I did something like this as a shell script on my windows box under cygwin: 

		# ogrinfo -summary -ro gltp:/vrf/z:/p2p/geodata/vmap/v0noa/vmaplv0/noamer
		for file in $( cat summary.info )
		do
		  emit=${file%@*}
		  echo "generating layer $file"
		  rm one.shp two.shp tre.shp for.shp
		  ogr2ogr one.shp gltp:/vrf/z:/p2p/geodata/vmap/v0noa/vmaplv0/noamer "$file" -nln "$emit" "$file"
		  ogr2ogr two.shp gltp:/vrf/z:/p2p/geodata/vmap/v0soa/vmaplv0/soamafr "$file" -nln "$emit" "$file"
		  ogr2ogr tre.shp gltp:/vrf/z:/p2p/geodata/vmap/v0eur/vmaplv0/eurnasia "$file" -nln "$emit" "$file"
		  ogr2ogr for.shp gltp:/vrf/z:/p2p/geodata/vmap/v0sas/vmaplv0/sasaus "$file" -nln "$emit" "$file"
		  echo "merging $emit"
		  ogr2ogr "$emit.shp" one.shp -nln "$emit" one
		  ogr2ogr -update -append "$emit.shp" two.shp -nln "$emit" two -nln "$emit" "$file"
		  ogr2ogr -update -append "$emit.shp" tre.shp -nln "$emit" tre -nln "$emit" "$file"
		  ogr2ogr -update -append "$emit.shp" for.shp -nln "$emit" for -nln "$emit" "$file"
		  echo "done $file"
		  # ogr2ogr -f PostgreSQL PG:dbname=vmap0 "$emit.shp" -lco OVERWRITE=yes
		done

To prepare the tiger data I first fetched it all from the US Census as zipfiles and then did something like this: 

			mkdir tiger2004fe_shpsmall
			for i in $( ls tiger2004fe_unzipped ); do
				for( j in $( ls tiger2004fe_unzipped/$i ); do
					echo visiting tiger/$i/$j
					mkdir tiger2004fe_unzip/$i/$j
					/opt/gdal/1.2.5/bin/ogr2ogr -f "ESRI Shapefile" -t_srs EPSG:4326
						tiger2004fe_shpsmall/$i/$j tiger2004fe_unzip/$i/$j
				done
			done

Another thing that can be done above is to run a shapetree over each of the shapefiles. This incantation is something like "shptree filename.shp" and helps mapserver to deal with large shapefiles. I did not do this but may do so later.

Schulyer manufactured polygons over tiger using a python based tool. I don't document this process here yet but may do so perhaps later. You can google for it. Polygons are not needed but are nice to have.

Then I ran a tile index generator over the whole thing so that mapserver could be efficient: 

			for i in $( ls tiger2004fe_shpsmall ); do
				for( j in $( ls tiger2004fe_shpsmall/$i ); do
					/opt/gdal/1.2.5/bin/ogrtindex landmarks_index.shp tiger2004fe_shpsmall/$i/$j/Landmarks.shp
					/opt/gdal/1.2.5/bin/ogrtindex chain_index.shp tiger2004fe_shpsmall/$i/$j/CompleteChain.shp
					/opt/gdal/1.2.5/bin/ogrtindex names_index.shp tiger2004fe_shpsmall/$i/$j/EntityNames.shp
					/opt/gdal/1.2.5/bin/ogrtindex polygon_index.shp tiger2004fe_polygons/$i/$j/Polygon.shp
				end
			end

Squid

Squid is a server side caching utility. For general information about this application see:

   http://meta.wikimedia.org/wiki/Squid_caching.

Squid can optionally be used to cache tiles that we generate in mapserver. Mapserver can take several seconds on a slow machine to generate a requested tile from the raw vector data. By using a tile-engine we can take advantage of any queries that were ever made before. If we were to allow exact rectangle queries for a segment of geography then it is likely that most user queries would be unique - and mapserver would be required to spend time to manufacture the requested image for each query. Using a cache probably gives us 1000x performance improvement.

Squid also opens the possibility of inter-server tile caching. Squid servers can talk to each other transparently. Squid is in fact used by a number of ISP's to help reduce bandwidth costs so even if there is no conscious ISP policy to cache tiles from our tile server - tile caching may still take place.

Squid can be a bear to setup. I tried both 3.0 and 2.5 and decided 2.5 was better documented. My key settings (after disabling almost everything in squid) are:

If you are using squid you will also want SquidPurge.

Apache

One key trick in using squid is that you must teach apache to set its 'ExpiresActive on' on. I do this with these two lines on the very last line of the apache 'httpd.conf':

Resources

Contributers and Helpers

This work was sponsored and funded by Civicactions LLC who also donated the hardware and bandwidth.

Download

If you just want to embed this engine in your site then you just want this (see install section above for how to use it):

http://civicmaps.org/tile.js

Here are tarballs of our modifications to the location module and our copy of the civicmap module. These will likely become obsolete and replaced by versions on the drupal site however I will try to keep this documentation current. You want only one of these two files not both. If your location module is working then just take the second file (it is much smaller to download as well):

License

This work is public domain.