Thinking in GIS

a blog about GIS from a urban geogeek working for humanitarian emergencies

Feed, Categories, Archives


A day with TileCache: generating KML Super-Overlays

Posted: August 06, 2008
Categories: TMS, GIS, SharpMap, WMS, Python, MetaCarta, kml, FeatureServer, devs, Apache, MapServer, TileCache, OpenLayers, GeoServer, Tutorials, GDAL, Uncategorized
Feedback: View Comments

My friend Diego Guidi is the smartest GIS/.NET developer I personally know here in Italy. He is the developer of NetTopologySuite, the port in the .NET world of the popular Java's JTS Topology Suite from VIVID Solutions. I wanted, sooner or later, write some stuff here about WMS and TMS, and now I am very happy that Diego asked me to publish this brilliant article about this topic.

First of all, let me thanks Paolo for hosting this post! I hope that this article can be interesting and useful like other stuff that you can find here...

Introduction

There are a lot of discussions out there about how to define Google Earth, Google Maps, and related apps... are they GIS? Viewers? Video games? Even a neologism was created: Neogeography. I think that all the folks out there have the same idea in mind: maybe Google don't make the same business as Esri, but Google Earth is cool, and it's funny to play with it!

For a simple user, the verb "play" could be meaning: "wow! I fly over my house!"; for me, as a GIS developer, the verb "play" means something slightly different: "How can I show my GIS data, like shapefiles, geodatabases and/or raster images, in Google Earth?". The simpler answer to this question is to publish your data as Web Map Service (WMS) and then consume it from any compatible clients, like OpenLayers or Google Earth; this solutions works, and works well, if you don't matter performances... let's give a look to a simple real-world example. Your customer calls at the phone and say something like: "I have a bunch of data, some shapefiles and some GeoTIFF images: can you please help me to show these data in Google Earth?". You may say: "Absolutely! Google Earth is a WMS client, so I help you building a WMS service". The problem is that the "bunch of data" is 10+ gigabytes of raster images, and the customer, as usual, love to see immediately his data on screen (and not to wait seconds - or minutes - before WMS sends the response for each request), just because "if google can do it, why you can't do the same?"... The way to achieve this goal is not to buy an extremely powerful hardware for your service, but simply to convert the data, published by the service itself, in a way that is faster to manage... like a ton of little images (a.k.a. tiles) stored to disk.

Enter KML Super-Overlays

Super-Overlays are a way, offered by the KML standard , to manage extremely large datasets; from the KML tutorial:

Q: How can I share a 47MB image with the world? A: One piece at a time.

So, basically create a Super-Overlay involves a three-step procedure:

  • 1. Prepare your data.
  • 2. Generate the tiles.
  • 3. Prepare a valid KML document to publish this tiles.

Although the first step is relatively easy to do with well-known products, both open-source (MapServer, GeoServer or SharpMap) or commercial, the next steps are something new and unknown, at least for me.

TileCache to the rescue

TileCache is an open-source project made by the smart guys at Metacarta Labs - check also their other amazing apps, like OpenLayers and FeatureServer - and it's essentially a proxy between generic GIS clients and a Web Map Service: TileCache's purpose, as the name suggests, is to manage a request from a client, ask a Web Map Service for the response and cache it, so for next requests for the same data the proxy itself could use his cached copy and avoid requests to the Web Map Service.

TileCache Sequence Diagram

I've talk about generic "GIS services" as clients, because TileCache proxy could act not only as Web Map Service (requests like this one) but also as a Tile Map Service (requests like this one): in particular, TMS handling capabilities are used in KML Super-Overlays, so later we talk a little bit more about this service. One thing that must be clear: cached tiles are generated by a Web Map Service; so, the first step of our procedure ("Prepare your data", do you remember?) means that we need to build and configure a WMS that is accessible from TileCache (and TileCache only: you could hide this WMS behind a proxy because the only service that is needed to be exposed over the Internet is only TileCache).

TileCache is not the only one...

Although TileCache, at least for my experience, works really well, other software is available to make the same purpose; one of the most known is GDAL2tiles, that is part of the GDAL project. GDAL2Tiles it's beyond the scope of this article, but I want to specify some bigger differences between both projects, so maybe you could understand if GDAL2Tiles is better for you:

  • While TileCache needs a configured WMS service as data source, GDAL2Tiles works with any of the formats supported by GDAL. For my experience, It's easier to prepare a WMS, test how it works (and how our data is rendered) using a WMS Client and then generate the tiles using TileCache.
  • While GDAL2Tiles needs to pregenerate all the tiles, TileCache could generate a tile on the fly when it's asked for the first time (of course, for the next requests, the cached tile is used): this saves a lot of time because often the pregeneration process is extremely slow.

Finally @work

Prerequisites

Remember: TileCache does nothing all alone, so first of all we need a Web Map Service; we want to try TileCache immediately, then we use a WMS exposed via Internet, NASA's OnEarth (at this url), that publish a "current global view of the earth", as described in capabilities document, called daily_planet. Building a complete WMS server is an interesting topic itself, and maybe in a next article we will write about that; for now, if you're interested to use your own data, check this tutorial about how to configure and use MapServer as Web Map Service. The only mandatory requisite is that the WMS must support EPSG:4326 -WGS 84 LatLong as Spatial Reference System for published data, because Google Earth could manage only this kind of SRS. A suggested requisite is that the service could publish data using at least one image format that supports transparency - like png - because often Google Earth's users love to show your data upon other stuff, without hiding it.

Apache HTTP Setup

Before installing TileCache, we need Apache HTTP Server up and running: I've downloaded and installed 2.2.9 version from the download section, that creates a ready-to-use web server that runs at the address localhost:8080; I've also downloaded and installed 2.5.2 version of Python interpreter and 3.3.1 version of Apache's Mod Python, an Apache module that embeds the Python interpreter within the server. Once that software is installed, configuring Mod_Python is pretty straightforward (but If you encounter some problems, TileCache's mailing list is a good resource for technical solutions): open httpd.conf file under the Apache conf folder and add:

LoadModule python_module modules/mod_python.so

at the end of LoadModule declarations. To check if Python interpreter is well configured, create a folder called test under Apache htdocs folder, and add a directory section like this:


    AddHandler mod_python .py
    PythonHandler mptest
    PythonDebug On

where apacherootfolder is the path of your Apache installation folder ("C:ProgrammiApache Software FoundationApache2.2" in my Apache installation).

Now create a file called mptest.py under the already created test folder, with this content:

from mod_python import apache
def handler(req):
    req.content_type = 'text/plain'
    req.write("Hello, World!")
    return apache.OK

Now start Apache HTTP Server and go to the url:

http://localhost:8080/test/mptest.py 

and see what's happen... if you read the familiar "Hello, World!", you're the man!

TileCache time

Now download version 2.0.4 of TileCache, packed as zip file that you could extract under Apache "htdocs" folder; I've also renamed the root folder from "TileCache-2.0.4" to simply "tilecache", so I have a hierarchy like this: "htdocs/tilecache/" that contains all the tilecache files and executables. Looking at tilecache files, you see "tilecache.cgi"... this means that it's not mandatory to use Apache HTTP Server and Mod_Python: installation readme contains all information you need to use TileCache under cgi, fastcgi, and so on... and if you like to use IIS, Vish Uma has made a great tutorial that you must read.

Re-open http.d conf file and add a Directory section like this:


    AddHandler python-program .py
    PythonHandler TileCache.Service
    PythonPath "['apacherootfolder/htdocs/tilecache'] + sys.path"
    PythonOption TileCacheConfig "apacherootfolder/htdocs/tilecache/tilecache.cfg"
    PythonDebug Off

In the http.d file we have specified as TileCacheConfig parameter a reference to a file that contains all configuration options: tilecache.cfg is a text file (What The Hell? Why a simple and perfectly readable TXT and not an unusable XML file? Are you crazy at MetaCarta?) that contains some global options (like cache strategy) and a list of published layers, each with his specifications (like WMS source, SRS, tile image format, and so on). The configuration file itself contains documentations for all options supported, but for our purposes we need a really simple configuration file, like this:

[cache]
type=Disk
base=apacherootfolder/htdocs/cache
[NASA_JPL_WMS]
type=WMS
url=http://onearth.jpl.nasa.gov/wms.cgi
layers=daily_planet
extension=png
size=256,256
bbox=-180.0,-90.0,180.0,90.0
srs=EPSG:4326
levels=20

So we use a cache-to-disk strategy (we decide, but it's not a mandatory, to save our tiles under Apache "htdocs" folder because we may want to access directly from internet), and a single TileCache layer called NASA_JPL_WMS, published with SRS EPSG:4326: this layer uses specified WMS URL to ask for 256x256 tile images from daily_planet layer, requested as image/png mime type and stored as PNG files; we need images for entire world extension, so a valid bounding box in WGS 84 LatLong coordinates is specified, and we decide to use twenty levels of detail for our tiles. These are all information TileCache needs to build a WMS request to our service, get a tile to store as cache, and then send it to clients requesting for our data.

The "web 2.0" way to make tests

It's time to show our tiles, but before using Google Earth we start with a simple html page, that could be helpful to verify if we made some mistakes, just because with Firefox and Firebug we can easily see all the requests to our services and see what happens. We use OpenLayers library, a javascript framework for building WebGIS applications: this library is really, really, really an interesting piece of software because helps to build mapping applications easily and with nice features: take a look at some examples and explore the applications gallery! I hope to write a blog post soon about OpenLayers... but after my vacation indeed, sorry :( So, we change the content of the script tag of default index.html page (that is part of the TileCache distribution) with this javascript code:


In this page we have build several layers, asking for our tiles in different ways:

  • 1. The old and glorious Web Map Service.
  • 2. The new upcoming star Tile Map Service.
  • 3. The outsider, and free runner, TileCache "service".

Looking at the urls we may recognize a familiar WMS request:

http://localhost:8080/tilecache/tilecache.py?LAYERS=NASA_JPL_WMS&FORMAT;=image%2Fpng&SERVICE;=WMS&VERSION;=1.1.1&REQUEST;=GetMap&STYLES;=&EXCEPTIONS;=application%2Fvnd.ogc.se_inimage&SRS;=EPSG%3A4326&BBOX;=90,-90,180,0&WIDTH;=256&HEIGHT;=256

a not-so-familiar but quite-simple TMS request:

http://localhost:8080/tilecache/tilecache.py/1.0.0/NASA_JPL_WMS/1/3/1.png

and an even more simple TileCache request:

http://localhost:8080/cache/NASA_JPL_WMS/02/000/000/007/000/000/003.png

Look deeper at the request urls, we see an interesting thing: the first two requests uses TileCache service, that take care of requests and manage in some ways a response, but third request is simply an url to an image! This is why we have cached our tiles under "htdocs" folder... isn't cool? Unfortunately, for our purposes we need to generate KML documents on-the-fly: actually, Google Earth requests are TMS requests, but with "type: kml": try to write this url directly in your browser and see what happens...

http://localhost:8080/tilecache/tilecache.py/1.0.0/NASA_JPL_WMS/1/3/1.kml

Greetings from the Earth

Preparing our KML

We're finally at the final step: create a KML document and see our data on Google Earth! This job is quite easy, just because TileCache distribution contains a kml template called overlay.kml (under the "htdocs/tilecache/docs/examples" folder), that we could modify to redirect calls to our ready-to-use TileCache service:



    
  
    
      http://localhost:8080/tilecache/tilecache.py/1.0.0/NASA_JPL_WMS/0/0/0.kml
    
  
  
    
      http://localhost:8080/tilecache/tilecache.py/1.0.0/NASA_JPL_WMS/0/1/0.kml
    
  
  

Simply, this document instructs Google Earth about how to find external resources (other KML documents) via Network Links, that may basically be anything the KML files can support, included kml's and/or images. TileCache uses a 'whole world' extent split into two tiles that are western and eastern hemisphere, so we're actually asking data for the entire world; if I open this document in Google Earth, here's what I see:

Google Earth

i.e. our raster dataset overlays default content :)

A deeper look

We're curious, so let's take a look at one of the generated KML's. Type an address like this in your browser:

http://localhost:8080/tilecache/tilecache.py/1.0.0/NASA_JPL_WMS/0/1/0.kml

and open generated response with a text editor, looking for interesting tags like:

  • 1. GroundOverlay: as kml reference suggests, draws an image overlay draped onto the terrain.
  • 2. Icon: the image to draw as overlay, is a link to our TileCache Service using TMS specifications.
  • 3. NetworkLink: four references to other kml documents that defines next levels of detail for the shown image.

kml tiles

So, this means that each document defines how to retrieve subsequents documents that provides higher levels of detail: simple, easy and perfectly working.

Speeding your responses

Playing with the test page you may see a little but obviously problem: when TileCache needs to generate a tile from scratch, you need to wait! Perhaps wait-time is not significant at lower visualization scales, but at larger scales WMS service (that is the "real" tile generator) often needs to manage and process a very-large dataset, and a response delay of thirty seconds or more is not accettable. Fortunately, we could resolve this issue using tilecache_seed.py script (shipped with TileCache distribution) that simply generates "fake" requests to our service, forcing TileCache to generate tiles. Script usage is straightforward; simply open a console prompt, go to Python 2.5 installation folder (C:Python25 in my PC) and type

python "apacheroot/htdocs/tilecache/tilecache_seed.py" "http://localhost:8080/tilecache/tilecache.py?" NASA_JPL_WMS 0 10

where the parameters are:

  • 1. Full absolute path of "tilecache_seed.py".
  • 2. Url of our TileCache service.
  • 3. TileCache layer to use in tiles generation
  • 4. Minimum zoom level, the start-point in tiles generation process.
  • 5. Maximum zoom level, the end-point in tiles generation process.

Note that tiles generation process is extremely slow: for my experience, often is useful to pregenerate only first levels of detail (like in the example above) and let TileCache generate on-the-fly remaining levels. An acceptable compromise is to start using your service when tilecache_seed is still working: this degrade performances only a little bit if dataset's management is not too expensive for your WMS service.

Time to say goodbye

TileCache is cool, no doubt about this, but, as many other open-source projects, documentation is not so cool like the code itself; in addition, Super-Overlays isn't clear (at least for me), so I hope that this blog post could be helpful for anyone interested. This is my little contribution to TileCache documentation: any suggest or criticism is well-accepted :) Diego Guidi

P.S: in this journey we've encountered some interesting "extension points" for this article... installation issues in a non-windows environment, how to build a WebGIS application with OpenLayers, how to create a Web Map Service... actually, my thought was to stay focused on TileCache only: I hope this goal was achieved, and I promise to contribute with more articles soon.

blog comments powered by Disqus