I’ve been working on a major upgrade of the search technology used behind Zenbu. Lucene is a giant in the open-source search software community and we’ve been using a Ruby port of Lucene called Ferret since inception. Ferret is blazing fast but hasn’t been maintained or gained some of the additional features that it’s Java cousins have. We have now made a complete transition to Solr - a popular, blazing fast open source enterprise search platform.
On the surface you won’t notice any difference to existing functionality; we’ve spent a lot of time making sure it ‘just works’. It does give us an exciting platform on which to add new features, two of which I’ll discuss today.
Faceting allows you to constrain a search by a particular attribute of the results. Let’s give a simple example.
A search for ‘mexico bar’ on Zenbu NZ has 3 results. You probably wouldn’t need to refine such a small set of results further but it serves as a good example of the idea.
On the right hand side of the search page you can see the Category Facets.
Refine by Top Categories
Clicking on any of these would show you new results with only entries from that category. You might note the facet numbers add up to 4; this is because one of the entries is in two categories.
Maybe you really meant “lunch bar” rather than a drinking “bar” so clicking the Bakery category would likely help you find what you were after.
This is cool stuff and even cooler when you can combine it with a Location facet as well; something I hope to have in the near future.
This ones really for the data nerds like me; it gives an interesting insight into the history of Zenbu data updates and a way to see ‘old’ data.
It’s a table of data (that will populate a histogram eventually) showing the number of entries last updated split by month. This doesn’t really show you how much activity happened in a month as an entry only shows up once in the most recent month it was updated.
What it does show you and allow you to easily find is entries that haven’t been updated for a long time. This isn’t necessarily a bad thing, some entries just haven’t changed in 5 years, but it’s certainly a clue for chasing down potentially outdated information. This is something we’re going to be focusing on a lot more in future; maintaining the accuracy of existing entries is just as important as extending the coverage.
One dedicated Zenbu user noticed that there were some entries that did have addresses, but didn’t have a map and wanted to be able to find these to possibly put them on the map. It’s undocumented as yet but you can do this now with the following search. http://www.zenbu.co.nz/search?q=address%3A%5B*+TO+*%5D+location_ll_0_coordinate%3A0 nifty.
Bit rot: noun. The gradual decay of a software program over time.
One of the wonderful things about the internet today is the availability of APIs and 3rd party services that you can bring together to create a unique service — like Zenbu. One of the problems of this system is the rapid pace of development and change which means that APIs and software are updated and/or deprecated on a regular basis. If you don’t keep up, your site will fall apart. You could think of it like the maintenance required on a house or car, except the time frames are far, far shorter.
We’ve been using the Google Maps service since day 1 at Zenbu. Google has a massive team working on this product and it is continually developing. The ‘version 2′ service that we have always used has been officially deprecated for 18 months and will be turned off in May 2013. We looked at migrating early on but it wasn’t as complete as the old version initially and then it just sat in the To Do list, until now.
We’ve finally upgraded to Google Maps v3 API which will bring some nice enhancements to the site.
The performance of v3 makes the maps load much snappier.
The Street View integrations is much tighter. “Peg Man” is shown on the map and you can drag him onto the map to activate Street View at your desired location. The entry location markers actually show up right inside Street View. Nifty!
It should be an intuitive upgrade, the user interface is almost identical. I hope you all enjoy using it as much as I do.
One of the down sides to the Creative Commons Attribution license is that there is no requirement to tell us what you do with all the freely available Zenbu data. But we love hearing about it, so please do let us know! Today we heard from Dr. Stacy Rendall about how he has been using Zenbu data. In his own words…
We use Zenbu for work focused upon understanding energy issues relating to transport within urban forms. I recently completed my PhD which developed a GIS based tool to estimate the ability of people to adapt to energy supply issues, which is a function of transport infrastructure and destination availability. For that project we used Zenbu to show us destination locations.
Our research has developed a methodology for estimating the ability of households to adapt to future transport energy supply constraints. This characterisation requires a significant amount of data, including transport networks, household locations, and employment and activity facilities. The activity facilities data is most difficult to come by, as no New Zealand national body reliably collects this information in a GIS-ready format.
We found out about Zenbu through the Koordinates website. The Everything from Zenbu dataset is the best data source available for our requirements, as it is nation-wide and has a large number of contributors, including, for example, the Ministry of Education for school destinations.
Another project we have just started is looking at is some redevelopment within Christchurch to house people displaced from the earthquakes, and we will use Zenbu data to get an idea of current activity destinations (Employment, Education, Shopping, Medical etc) as context for the project.
Dr. Stacy Rendall
Post Doctoral Research Fellow
Department of Mechanical Engineering
University of Canterbury
OpenStreetMap (OSM) is the torch bearer for Volunteered Geographic Information (VGI) projects on the web. It has over 500,000 users and very usable maps in most parts of the world. Like any collaborative project the data is strong, or even amazing, in places where users have dedicated time but patchy in many other parts. Zenbu is no different in this regard.
What is the difference between Zenbu and OSM?
- OSM is primarily about the map, with POIs (Points of Information) as a helpful component of the map.
- Zenbu is primarily about the POI, with the map as a helpful visualisation tool.
This might seem like a minor difference but it is at the core of each product and the direction it takes.
- OSM has a ODBL license
- Zenbu has a Creative Commons Attribution license
Both licenses allow access to the data for free. The Zenbu CC by A license is significantly less restrictive in what you can do with it as there is no ‘share-alike’ clause. One outcome of this is that Zenbu data can be pushed into OSM but OSM data can not be pulled into Zenbu.
OSM has a significantly larger user base and coverage internationally. In New Zealand Zenbu has a far more authoritative set of POI information.
I’d like to see the Open Street Map of New Zealand be as good as it can. We’re working to enable systems for people to easily re-purpose Zenbu content onto OSM. Stage 1 of this is complete now with a mapping of the Zenbu Categories to OSM categories. There are a few more steps to go to make this more accessible, http://wiki.openstreetmap.org/wiki/Zenbu.
There is a lot of work going on at the moment to build tools to import NZ government spatial data, from Land Information NZ, into OSM. https://groups.google.com/forum/?fromgroups#!forum/nzopengis. We’ve been in touch with the team at Koordinates who are working on this and hope to have more to report soon.
At 1.35pm on Fri Sep 7, we suffered a major server failure at our primary data centre. The secondary server picked up the load and there was little down time but the database backup had not happened for 24 hours.
At the moment we are missing all entries and users made or updated from approximately 4.30am on Sep 6 to 1.30pm on Sep 7.
We sincerely apologise if the changes you made were affected. We hope to be able to restore them when the affected machine is investigated.
UPDATE: All data was recovered: http://www.zenbu.co.nz/entries/updatesummarybyday/2012/09/06
There is now (finally!) a free native app for Zenbu for those of you with Android smart phones.
BizHunt uses the data from Zenbu to power their search database. It’s the data you know and love in a handy wee Android package. One of the early (5 star) reviews said:
Just like findnz for Android but better
Go check it out.
It is worth noting that the ratings and advertising opportunities on BizHunt have no connection with Zenbu, are unique to the BizHunt application and will not reflect on the Zenbu website.
This is one for the power users, the data crunchers, the hardcore Zenbu users who are committed to making sure the information on the site is as accurate as possible.
We’ve always had a complete version history on Zenbu so you could see the entry as it existed through time. Now we’ve added a side-by-side difference engine so that you can compare two edits and easily identify what changes were made. Changes use red and green highlighting to show what was deleted and added. Depending on the field, it could be by character or word; on the map we’ll display both locations.
You can see this screen on any entry with a history by clicking on the “Back in time” link on the entry listing.
I trust this will help give better visibility for changes. Wise men have said “With enough eyes, all bugs are shallow“; on Zenbu I think that means that with enough people looking at and using the data we can make the data as current and accurate as possible.