Newbie Notes for GIS Web Development

I have spent some time over the past couple of weeks getting to know the Open Source GIS arena, from spatial databases (PostGIS) to server software (GeoServer) to web display techs (OpenLayers). When I started, I decided to map my trail as I explored this space since I will have co-workers following along behind me as new projects in our group ramp up. Hopefully, these breadcrumbs will help other newbies in this area as well.

begin tangent
Have you ever felt that sometimes the worst people to write documentation for something are actually those that know the most about it? Once you are an expert in something, it's really hard to consciously remember all the questions that came up and problems you had to solve and research you had to do to reach nirvana. That's why with this project I tried to jot down questions that I had as I went, and kept the question list even after I figured out the answers. It helped me not forget what I didn't know. I've used this technique before, and strongly encourage new hires who come onto my projects to do the same so we can fill in the holes in our developer documentation. Hmmm... sounds like another blog entry in there...
end tangent


This first entry is about the "what's what" GIS and "who's who" in the Open Source area. It is not intended to provide all the information you need to work on a GIS application. It is intended to tell you where to get that information. It also explains a few questions and misconceptions I had as I started down this path.

What is GIS?

First, to get your head around the GIS terms and concepts, read this short overview on GIS concepts from developerworks. One thing the article does is define what a layer is and relate that to the term feature. What it leaves out is the term FeatureType. A FeatureType defines a type of data, listing the attributes that go along with it, such as name, shape, and other meta-data (e.g. population for cities, road type for a set roads (highway, secondary, dirt, etc)). The layer concept is a way of visualizing a bunch of features of the same FeatureType. For example, in GoogleEarth, you don't think about turning on and off the road FeatureType or the city FeatureType. You think about showing/hiding those layers. Some software (like GeoServer) uses the term FeatureType in places where we more naturally think in layers. Just get used to moving back and forth between the two terms.

What's a Map?

Something that is missing from all the literature is a strict definition of what is a map. Tutorials describe layers, FeatureTypes,and features. Standards define ways of storing geographic information, retrieving that information, applying styles to it, and rendering it. But I couldn't find anything that strictly defines what a map is. What this means is that each application or library has it's own concept for what comprises a map, or it may not have a unified concept of a map at all. OpenLayers (the most popular way of rendering maps in web pages) is based around the concept of a map. It's map has layers of data that can be shown/hidden independently. The map also has tools for zooming, paging, measuring, highlighting, and possibly even editing the data. GeoServer (an Open Source map server), on the other hand, is more focused on individual data sets (i.e. independent layers or FeatureTypes). How those sets of data are combined into a single, visual display that we would call a map is up to the consumer of the data (such as a web app using OpenLayers or a desktop app like uDig).

If you are looking for the least common denominator for the concept of a map, think of a display of layered geospatial information with one or more "base layers" comprised of static (or nearly so) data (e.g. geographic features, political boundaries, rivers, roads, cities, etc) and zero or more "live layers" comprised of data that can change with relatively high frequency (e.g. weather images, traffic patterns, earthquake epicenters, recent Elvis sightings, etc).

Who's Who

To learn about the Open Source standards and tools in the GIS space, flip through Scott Davis' GIS for Web Developers presentation while you listen to his GIS podcast. He has a lot of other great content on his mapmap site. If you like his presentation style, pick up his GIS for web developers book. It comes in a PDF format for instant gratification.

Now that you've been exposed to some of the concepts and heard mention of the major players and most popular apps, you can read what Wikipedia has to say about them. As usual, the Wikipedia pages have links to the organizations' sites as well as important related concepts.

• Open Geospatial Consortium (OGC) - the first thing to know about the OGC is that they publish the WMS and WFS standards. They have many other standards as well, but those two are the primary protocols by which you will get data from a map server (like GeoServer) to a UI (like OpenLayers).


• ESRI ArcGIS - ESRI is the 800 lbs. gorilla in the GIS space. It is sort of like the Oracle of GIS. It has it's own commercial, proprietary software suite called ArcGIS. If you have heard of Shape files, this is the company that invented that format. It's not open source, but it's good to know who they are. In my situation, I have existing systems that feed into ArcGIS layers, so I have to work with it as well as with other data via the OGC standards.


• GeoServer - Highly extensible, open source WMS/WFS server. A good application to keep in mind if you want to run an application that is a single source for both "base layer" data and your app-specific data. Something that was critical for my project is that it can pull data from an ESRI ArcGIS server as well as other sources like PostGIS or raw images. It's online user manual contains some good sections on basic concepts for serving and formating geo data over HTTP, including Styled Layer Descriptor (SLD), WMS, and WFS.


• PostGIS - Geo-spatial extensions to PostgreSQL. This is the most popular (and powerful) OpenSource geo-enabled DB. MySql also has geo-extensions, as does MS SqlServer and Oracle (called Oracle Spatial). BostonGis has a great tutorial on installing PostGIS and the basics for using it.


• OpenLayers - JavaScript library for displaying maps and mapping tools. If you see a map on a web site and it isn't an embedded GoogleMap or MapQuest map, it's probably being rendered by OpenLayers.


• Open Source Geospatial Foundation (OSGeo) - not to be confused with the OGC, above. The OGC is a standards organization. OSGeo is a non-profit that supports open-source geo software projects and related initiatives. They support web mapping, desktop apps, geospatial libraries, and other types of projects, including GeoTools and OpenLayers.

If you thought that keeping OGC and OSGeo straight was confusing, just wait! There's one more. OpenGeo is a company (sorry, a "social enterprise") that integrates the most popular Open Source GIS technologies (like those listed above) into a single, supported stack or application framework. I have no experience with them, but if you need to get a GIS app up and running quickly, they sound like good people to call. I am sure there are other such organizations out there that can help write your software or train your dev team. I just now found that Scott Davis' ThirstyHead company is offering 3-day GIS training course.

GIS Blogs

There are probably a 100 GIS-oriented blogs. Start with planetgs. That is an aggregator for many others. If you find that articles coming from a particular source are good, you can follow it directly. I happen to like Fuzzy Tolerance for it's good content on GIS and Open Source web development in general and concise monthly roundups.

Where to Get Data

If you want to get "base layer" data (geographic, political, structural, etc) to display underneath your app-specific data, browse through these sites:

• http://www.mapcruzin.com/download_mapcruz.htm
• http://www.census.gov/geo/www/gis_gateway.html
• http://exploreourpla.net/
• http://www.bostongis.com/
• ftp://data.massgis.state.ma.us/pub/shape/state/towns.exe
• http://onearth.jpl.nasa.gov

Geocoding

If you have address data or other geographic text and want to find out how to plot it on a map, there are a few free geocoding services. geocoder.us is a good starting place for testing your app if you just have data in the US. There are sister services for other countries. Google also has a geocoding service, but the license requirement says you have to use the data to display on a google map. (At least, it does in one place. In another place, it just says display on a map, without specifically stating google map.)

Getting Data Via WMS versus WFS

Before going into more detail about some of the above apps and libraries, I want to clear up something that confused me at first. How data is stored (vectors or rasters) is a separate concept from how it is distributed and displayed. When you request data from a WMS service, you will get back an image. It doesn't matter if the data is stored as a jpeg or tiff or as a Shape file or a set of XML files or it is in a database table. Whatever the source, the map server converts that data into an image using some standard styling rules and sends that image over the wire. On the other hand, if you request the same data via WFS, you will get some form of data list, usually in an XML format known as GML. How that data is then transformed into some visual display is up to the client.

The difference between WMS and WFS has an impact on how you can combine data from different sources in the same web-based map when using OpenLayers. WFS layers can be subject to the cross domain scripting limitation. But that is the basis for another post.

Reference Apps

Finally, here are some apps that let you drool over the possibilities of what GIS tools can do for you.

• MarineMap
• PostGIS case studies

Have your own cool web-based GIS app? Post a comment and I'll add it in.

Next up... A few notes on setting up datasources in GeoServer.

Bus Carpet (c) 2009

We have several new projects spinning up in my group, as well as several new releases or add-ons for existing projects. It's good when your products are popular and are needed by many different contracts/customers. It's bad when you are one of the very few people who has worked on (or led, in my case) nearly all of these products and so people keep asking for "just a little bit of time" to review a design or answer a question from the field or interview some new folks. I haven't been able to touch my "real" assignment (designing a replacement for our geospatial + full text alerting app) in a couple of weeks.

When another developer here in my group suggested me along with another overloaded teammate as folks who may be able to do some testing of a new REST api on one of the products, he was kind enough to phrase the request as "tossing some names under the bus," indicating that he knew the people attached to the names were overbooked. My response? "There have to be more bus carpet names than Jeff and I for everything."

While not a full Googlenope as defined by Gene Weingarten, I think it is original in its usage. So I hereby submit my addition to the blogsphere lexicon:

Bus Carpet - Someone who is tossed under the bus with great frequency.

Eclipse, Maven, and Groovy, oh my!

The groovy support in Eclipse leaves a bit to be desired. "Tell me something new" you say. How about - The groovy support in maven projects within eclipse leaves a lot to be desired, but it does at least exist, so long as you are willing to work at it a bit.

Rather than posting the details off in the wilderness of my own blog, I responded to a post on the m2eclipse (the maven-eclipse plugin) mail list from last Fall where someone was asking how to add groovy support to maven projects. The author of the m2eclipse plugin stated that the groovy plugin needed to support some hooks into m2eclipse. While that may be true to get fully automated setup and synchronization, you can get the two plugins working together. See this thread for the info:
http://www.nabble.com/Using-M2Eclipse-together-with-Groovy-td19261256.html

Subversion (subclipse) corruption in Eclipse

I use subclipse to manage my subversion interactions in eclipse. I also use TortoiseSVN for pulling clean source code trees that are separate from my development workspace to produce releases. Normally, "never the two shall meet" is my policy. Eclipse/subclipse manage projects in my workspace directory and Tortoise handles everything else. I just found out what happens if you let them cross paths.

I had a project in eclipse that seemed to have corrupted it's svn meta-data. The Synchronize view insisted that I had added some files when I hadn't. After restarting eclipse, the Synchronize view insisted that the whole source tree had changed. Trying to narrow the problem, I quit eclipse, opened Windows Explorer, and told Tortoise to check for updates and local mods in the project. Tortoise turned up a big nothing; it didn't think there were any changes (which was correct). So I went back into Eclipse to see if I could find a pattern to it's madness. (I know, I know - "This is eclipse, " you say, "to which madness are you referring?"). Trying to synchronize the project again resulted in this helpful messages in the SVN console:

update C:/Dev/workspace/myproject -r HEAD --force svn:
'C:\Dev\workspace\myproject' is not a working copy
svn: Cannot read from 'C:\Dev\workspace\myproject\.svn\format': C:\Dev\workspace\myproject\.svn\format (The system cannot find the file specified)

For any who don't know the inner workings of subversion (like me), under subversion 1.4 and below, the format file contains the version of the meta-data. 9 for version 1.4, 8 for 1.2 (I think), etc. With subversion 1.6, that file is removed and the meta-data version is written in the top line of the entries file.

Lesson #1: The subversion version used by eclipse may be different from the version used by your standalone subversion client.

I use Tortoise 1.6.2. This uses subversion client 1.6 libraries. My subclipse plugin is from the 1.4.x branch. As the Tortoise FAQ will tell you, subversion versions 1.4 and above will automatically upgrade the working copy format of your meta-data. So the message was correct - the format file wasn't there. I just didn't yet understand what had happened to cause that. I thought I was still dealing with the same unknown error that I started with when, in fact, I had managed to create a whole other problem to deal with.

Somehow (and the steps get blurry here) I got Eclipse to give me a better error:

Problems reported while synchronizing SVNStatusSubscriber. 0 of 1 resources were synchronized. An error occurred synchronizing /myproject: Error getting status for resource P/myproject org.tigris.subversion.javahl.ClientException: Unsupported working copy format

svn: This client is too old to work with working copy 'C:\Dev\workspace\myproject'. You need to get a newer Subversion client, or to downgrade this working copy.
See http://subversion.tigris.org/faq.html#working-copy-format-change
for details.

Ah-hah! So I at least knew what had caused the latest behavior. Upgrading my subclipse plugin to the 1.6.x branch got me back to my original problem. Nice to be back in known territory.

I knew that the subversion meta-data that I could see (stuff in the .svn directories) was okay since Tortoise functioned. So eclipse must keep some other meta-data about the synchronization state of a project. Time to delve into the dark world of the workspace .metadata directory. I searched through that directory for anything with the name of my project. After eliminating a few hundred hits in .markers and .history files and whatnot, I found a likely sounding culprit - .syncinfo. Sure enough, armed with that file name, I searched the web for references to my problem and turned up an archive of the subclipse-users mail list with exactly what I needed.

Lesson #2: If the subversion integration in Eclipse ever goes haywire for a project, delete this file:
[workspace dir]\.metadata\.plugins\org.eclipse.core.resources\.projects\[project name]\.syncinfo

Problem solved. Now I just have to get maven, eclipse, and groovy playing nicely with each other...

Offline Browsing in Firefox with Scrapbook

Got a long plane flight or car trip? Have several articles you've been wanting to read, or maybe need some online documentation that's a subset of a big site? The Scrapbook plugin for Firefox lets you access the latter while on the former. It is nearly perfect for assembling an offline reading list or a library of things you want to keep indefinitely. It has many options for capturing various parts of a page (selection, frame, etc) which is handy if you just want to save the main content and not things like ads or navigation menus. I also like the markup features available from a toolbar that appears at the bottom of the window when viewing captured pages.

The online docs cover the full feature set, so I won't repeat those. I do have a few tips for specific use cases that I came across though:

1. I wanted to capture an article that spanned multiple pages. In theory, ScrapBook has a filter capability that should be available in the In-depth Capture section of the Capture Detail dialog. I couldn't access that capability, so instead I would highlight the page links for the article (you know... the "1 2 3 Next" links that are usually at the top/bottom of a page) and then use the URL Detector in the Capture Multiple URLs option to capture all the links in the selection. I would then save those links to a new folder with the name of the article. It's not perfect since the pages are independent of each other (you can't navigate between them), but it's better than having to manually walk through all the pages and capture them.
   
2. While reading captured pages when I didn't have network access, I would often come across links that I hadn't captured, but wanted to mark to grab later when I was back on the net. You can grab the link do this with the Bookmark with Scrapbook option and then Capture Again to pull the content when you are backonline. (Oops - correction the Bookmark with Scrapbook option is not available from the context menu when you right click on a link. I'll have to add that to the list of features suggestions I posted for the author.)
   

On a related note - while looking for something that provided the capabilities of Scrapbook, I found some other interesting plugins to view the page cache (CacheViewer) and index pages you have visited so you can easily find where you read a certain factoid that you stumbled across, but can't remember where (Breadcrumbs). I'll see how much I end up using those before reviewing them.

Useful Links for Java Memory Issues

Every once in a while I have some problem with memory usage in Java (usually related to either Eclipse or JBoss running out) and I have to remember various settings or tools for dealing with the problem. Here are some pages that I return to on occasion:

Heap memory and MaxPermSize interaction
JVMStat tool
Open Source Profilers
Random Info
Interesting thread I came across when researching the Groovy eclipse-plugin memory leakl

If you want a good setting for JBoss (and you are limited to 32-bit Java), try:
-Xms128m -Xmx1536m -XX:MaxPermSize=128m

Eclipe Groovy Plugin - decent for scripts, PITA for classes

I have been using the Groovy plugin for Eclipse for about 18 months now. First let me say thanks to the team who has put it together. I can't really complain about any problems since it is a volunteer effort and I haven't volunteered. And over the time I have used the plugin it has gotten some pretty nice improvements, especially for code refactoring and auto-completion. It's debugging capability has always been good (with a few quirks). The fact that I can debug scripts as well as regular classes kept me from switching to NetBeans when the major upgrade of that product came out in November without that feature (hmmm... I see there are some upgrades for NB. Yet another thing to add to my to-do list to check out). Overall I find the plugin perfectly capable when writing scripts, especially in small projects. For writing classes however, especially in a large project or when mixed with Java classes, it's a PITA.

For those that cannot or do not want to move from Eclipse to IDEA or NetBeans, here are a few tips for getting by with the plugin for now:

1) The largest issue is a memory leak. I posted to the groovy user list about it over a year ago. (My explanation of the problem in that post was slightly incorrect. The problem will occur in any project with a lot of classes on the class path, not just one that includes another project.) I think the issue has gotten better in some circumstances, but it still makes it impossible for me to easily integrate groovy classes into my major projects.
Work Around: I have a small "GroovySandbox" project that has the class directory and key required libraries from my main project on its classpath. (Do not include the main project as a dependent project because you will have the exact same memory problem.) I write new scripts and classes in this project and then when they are ready, I copy them into the main project. The memory issue still has an affect, but I can work a full day on groovy files without having to restart eclipse rather than having to restart every 20-30 minutes.

2) Whenever I modify the class path of a project that includes groovy classes (thus triggering a clean build by Eclipse), the groovy class files are wiped with everything else and are not regenerated. I have to go in and touch each groovy class to get it to compile.
Work Around: I now have enough groovy classes that this isn't feasible, so I created an ant target that compiles only the groovy classes and I trigger it from within Eclipse when necessary. Eclipse detects the newly generated class files and the remaining java classes I have that are dependent on the groovy classes finish compiling. (Why do I have to mix groovy and java so much? See #1, above and #3, below. Unless I want to take advantage of groovy syntax features a lot, I stick with Java.)

3. When I open a groovy class in the editor, it often doesn't populate the Outline view. Makes it hard to navigate the file.
Work Around: Touch the file, or add-and-delete a space. This was a pain when using CVS for our SCM since the files were tagged as modified, but since switching to Subversion there hasn't been a problem. I guess Subversion uses something beyond file dates to track mods.

4. When debugging, if I try to step into a class (java or groovy) that is included from another project (which I do frequently since nearly all classes are in the main project due to #1, above), I get an error window saying that the class is part of the Groovy Libraries plugin path and that the path is not modifiable. The screen doesn't give you a button to modify the source lookup path for the directory/jar. (This one is relatively new. It started happening in some update around the beginning of the year.)
Work Around: Go into the project properties->Java Build Path->Libraries and manually set the "Source Attachment" property for the offending directory/jar. The next time you start the debugger, it will let you step in without a problem.