Marenkay.com

During my adventures with the phpArmory class I have taken a look at WoW databases out there. I browsed through loads of sites to see if any of them would be Open Source or at least under a free license.

The database contestants

Ever since World of Warcraft became public there have been websites out there, which gathered game data by using a Lua addon and some sort of data uploader. In the beginning there was one database site and nowadays there are many.

• Thottbot: Probably the father of all WoW database websites. Thottbot.com started somewhere during the World of Warcraft Closed Beta and has been a primary source of information for many players. Quests, Items, Maps, NPCs, Spells, and more await the curious player, and closely coupled with Thottbot comes Cosmos, a User Interface modifcation pack, to which players were drawn in the early days of WoW aka. the days when World of Warcraft was not available in any localization.
• Allakhazam: Allakhazam entered the market of WoW databases a bit later. They already had experience with this thing, as they started with an Everquest database. Their WoW site has gone through various iterations, and AFAIK was the first database site to offer support for the various localizations in which WoW was made available. Another feature spearheaded by them was the monitoring of Auction House prices for items. Yay for that!
• Wowguru: Another side project of an MMO database site, and one that initially had some success. WoWguru.com entered the scene in Summer 2005, and later spearheaded with features like honor tracking, and most prominently with monitoring the individual realm status during the Ahn’Qiraj War Effort.
• Wowhead: Here comes an impressive contestant. Ever since their launch Wowhead impressed with an information-centred design that helped users in finding the information needed very quick. Wowhead comes with a well made Lua addon, and a sleek uploader. Once of their primal bonusses was that they had been very active with making their Lua addon work for the Public Test Realms, and soon this resulted in Wowhead leading the WoW databases. Features like a well-designed commenting system, their item and character preview, and others still remain on top of the list.
• WoWDB: The latest entry in the list of contestants. Made by the MMO portal site Curse (which alos provides hosting for most WoW Lua addons), WoWDB entered the scene with some features that Wowhead could not offer: item comparison, wishlists, and a look more oriented on the WoW look. AFAIK their character preview was first, and Wowhead added that a bit after them.

What do all these databases share?

As it seems, the market for World of Warcraft databases is a hot one. Even four years after WoW made its’ initial release. And as we can see by the release of new and different databases, none of the contestants listed above yet has managed to reach a state where WoW players are satisfied.

If we compare the features of all available WoW databases we can see that it’s not all about features, but about design and usability, too. Some features did not really make it into the market, as players did not seem to accept them.

• Auction House monitoring: Yes, it is there, and some people might use it, but apparently this is not interesting or good enough to be used anywhere else than on Allakhazam. Apparently using spreadsheets still is superior. Also, there is Auctioneer which works in-game and automates a lot of the process.
• Crafting requirements: Wowguru introduced this quite some time ago, and I thought it was a great feature. When displaying a crafted item, you could see all the items required to create it, summing up the total vendor price for all needed materials. Nice, but oviously not needed. Gold seems to be an irrelevant factor, and thus only material lists survived.

This leaves us with a list of features that seem mandatory for a mature World of Warcraft content database.

• Searching: Of course, searching is mandatory. Every site has it, and the only difference to be found is in the way search results are returned. Older sites just take an argument to look for and return an result page with any matching object, Wowhead and WoWDB sport a bit more elegance and make use of AJAX to return drop down lists with matching data instead of requiring you to wait for a result page.
• Filtering: Even though the official forums are full of people demanding more content because the existing content would not be sufficient, a look at a database of your choice proves the opposite. There is lots of game content, and there is much to be seen. Even after four years, I am sure you will find content that you have not seen yet, even if you started your character in February 2005. Thus most database allow you to filter for various parameters, and most important: you can apply multiple filters to any kind of data. Sleak and useful.

And why would you want that Open Source’d?

It’s simple. Many people contribute data to these databases, and only with your or my contributions of data gathered while playing the game, those databases exist. It’s a community effort, just like the game itself is meant to be some sort of social community. But why are all of these databases closed, or commercial? Some even want you to buy premium subscriptions to access guides or more data.

Now what if there was an Open Source database for World of Warcraft? Where you could not only contribute data you gathered while playing, but also access that data or even download it to your computer and do something with it? Or where you could see what you have gathered while playing?

So I have decided to give myself a new pet project to work on, and it will become Open Source very soon, once I have ironed out the basic details. Work already begun, and a very basic implementation is there. The tools I have used are:

• Visual C# 2008 Express Edition to build a .NET 2.0 client application which handles uploads,
• Lua to write a WoW addon which gathers data while you play,
• Zend Framework and Doctrine ORM to write the server-side backend which accepts uploads.

The process for now is very simple:

• the .NET client will parse the WoW data cache (which you will find in your WoW installation directory under WDB/your-locale/*.wdb),
• it can show you the contents of your WDB files, e.g. you can see which NPCs you have seen, or what books you have read,
• it will parse the saved Lua data generated by the addon, and can show what data besides your WDB cache is there,
• and it can convert both WDB and Lua data to XML,
• and finally it can upload the data to my server (GZipped of course, to reduce bandwidth usage).

Parsing the data is something that will be done next. For now my server-side application only handles storing the uploaded data, and manages updates of the client. Not very much, but at least my data is stored. Parsing will be a bit more work since I have to make a modular parser that can deal e.g. with different data formats, as the data formats may and surely will change with every WoW client update.

What do you think? Would you use an Open Source World of Warcraft database, where you have full access to the data you contributed? Would you like to download contributed data in a parsed format, perhaps as XML files to reuse them on your guild website or your blog?

More on the project soon, after the finishing touches have been put on the data client. Screenshots coming soon.