I’m taking part in the NYC BigApps competition this winter. The competition’s aim is to reward the “most useful, inventive, appealing, effective, and commercially viable applications for delivering information from the City of New York’s NYC.gov Data Mine to interested users”. Well, interested users, my partner Frank and I would like your feedback on how we’re doing.

Today we’re launching PushpinWeb, a website and API dedicated to organizing your government’s information. We like to imagine ourselves a Google for open data, a Wolfram Alpha for government. Your feedback and insights are an important part of us translating that vision to reality.

Hello Bureaucratic World!

Programmers are generally smart folks. Most of their time is spent bridging the gap between possibility and execution. They solve problems. They identify the failure point in a system and work around it.

Government creates a brick wall for that innovative spirit. By some level of necessity, our city, state, and federal organizations are massive bureaucracies. They move slowly, and you cannot work around them. When Frank and I began the BigApps competition, the DataMine was our first hurdle.

The New York City DataMine is a difficult to navigate collection of zip files, each containing CSVs, TSVs, Plaintext, GDB, or SHP files (that’s only what we’ve found so far!). Some files are fixed width character fields, some are point data, some are shape data, and some formats are actually proprietary. Add to that the companion metadata, and the challenge to create something “useful, inventive, appealing, effective, and commercially viable” quickly becomes a challenge to write an effective parser.

Some developers have been very vocal about pushing the city to present data in a more accessible format. We support that push, but we also want to create apps around government data today, not after the dinosaur wakes up.

PushpinWeb is Open Data. Today.

Frank and I have begun parsing datasets from the DataMine onto superfast servers and publishing it via a RESTful JSON API. Query with a location and get back data on any dataset we parse. Build a client from scratch with the API specification, or take any of the examples we’ve written in Ruby, Python and Perl as a starting point.

Right now, we want your feedback as you start using the API and website. We have an exciting direction to take PushpinWeb as more datasets come online, but the goal here is to build something that caters to it’s users. To take government data, and convert it to something that caters to it’s citizens.

Stay informed of important updates by signing up for PushpinWeb, following PushpinWeb on Twitter, or emailing Frank and I at pushpinweb@gmail.com. We’re looking forward to your feedback and ideas!