The database client

While I am currently using .NET 2.0 / C# to prototype a client which takes care for downloading and updating the WoW addon for data gathering, I am of course aware that .NET 2.0 is not available on every platform where you can play World of Warcraft. There is Windows in various incarnations, and there is Mac OS X for us to play the game on, and some people even use Linux to play. I probably have not mentioned this, but I am using all of these platforms to play the game, and thus I am very interested in making the database client work on all of these platforms.

My current plan is to finish my .NET prototype, as I need some quick test tool, to define and build a proper web interface for managing the WoW addon, and to test the client updater. Once this is done, and the web interface works, I plan to turn the whole client functionality into a plain C++ library, which can be build and used on Windows, Mac OS X and Linux.

For now the library is supposed to contain the following features:

• Parsing the WDB cache: as many players know, the game client uses a simple data cache for books you have read, creatures you saw, game objects seen and used, items seen, and quests done. This data will be required for several reasons. E.g. if we want to display rendered views, we need the data from the WDB cache files.
• Detecting WoW: of course, the library needs to find WoW on your system, and there are various different methods to do so. WoW is e.g. installed in different ways for 32-bit and 64-bit operating systems.
• Parsing the addon Lua data: while this is not necessary it would be beneficial to embed Lua into the library. This would allow us to validate the saved Lua data, and it might provide options for blocking invalid data, too. Why should we upload broken data?
• Authentication: I would love to have the database site requiring a login. Why? Because it would be great to give credits where they are due. If someone contributes to the database, it should be recorded and visible in some place.
• Update management: we need to keep up-to-date, and this includes both the database client, and our WoW addon. The game client changes with every release, and bugs are everywhere. There is no development method that results in zero bugs.

On top of the library custom GUI clients can be used for each platform. On Windows, we could use plain C++ with MFC or a .NET GUI, on Mac OS X we can use Carbon API to integrate into the system.

Of course, suggestions are welcome.

The database server

Once of the things I have considered over the last few days was to turn the server into something that could run in a distributed fashion, much like the version control system Git which I use for my own projects at home and at work. Concerns have been raised that the current databases are commercial because you will have large hosting bills.

But what if the database server would have a central server somewhere, and all available installations would connect to this server for data sharing? Thus every installation (e.g. on guild sites) could contribute their data even without knowing the main server, and every installation would benefit from uploads.

Anyway, the only thing I have decided on yet is that the database server will be developed using a model-view-controller pattern. This may be done with ASP.NET, on PHP using the Zend Framework, or on Python using Django, or on Ruby using the famous Ruby on Rails.

Personally I would prefer a Python or Ruby based solution. But still, for my prototyping I will stick to using PHP and the Zend Framework for one simple reason: this solution works on most hosted environments. You will not depend on owning a root server (which not everyone can afford).

The code

There already is a bit of code, and I am working hard to get this in a readable state and documented. It will appear in my GitHub account in a few days.

Contributions are welcome, in the beginning I would be very interested to hear what addon developers need. E.g. what would systems like QuestHelper, TourGuide, or LightHeaded need? After all, quest guides are the simplest form of giving data back to the community, thus you would want to have it done right from the beginning.

For now, I am considering XML output in a format similar to the armory format, but would that work for addon developers?

The database contestants

Ever since World of Warcraft became public there have been websites out there, which gathered game data by using a Lua addon and some sort of data uploader. In the beginning there was one database site and nowadays there are many.

• Thottbot: Probably the father of all WoW database websites. Thottbot.com started somewhere during the World of Warcraft Closed Beta and has been a primary source of information for many players. Quests, Items, Maps, NPCs, Spells, and more await the curious player, and closely coupled with Thottbot comes Cosmos, a User Interface modifcation pack, to which players were drawn in the early days of WoW aka. the days when World of Warcraft was not available in any localization.
• Allakhazam: Allakhazam entered the market of WoW databases a bit later. They already had experience with this thing, as they started with an Everquest database. Their WoW site has gone through various iterations, and AFAIK was the first database site to offer support for the various localizations in which WoW was made available. Another feature spearheaded by them was the monitoring of Auction House prices for items. Yay for that!
• Wowguru: Another side project of an MMO database site, and one that initially had some success. WoWguru.com entered the scene in Summer 2005, and later spearheaded with features like honor tracking, and most prominently with monitoring the individual realm status during the Ahn’Qiraj War Effort.
• Wowhead: Here comes an impressive contestant. Ever since their launch Wowhead impressed with an information-centred design that helped users in finding the information needed very quick. Wowhead comes with a well made Lua addon, and a sleek uploader. Once of their primal bonusses was that they had been very active with making their Lua addon work for the Public Test Realms, and soon this resulted in Wowhead leading the WoW databases. Features like a well-designed commenting system, their item and character preview, and others still remain on top of the list.
• WoWDB: The latest entry in the list of contestants. Made by the MMO portal site Curse (which alos provides hosting for most WoW Lua addons), WoWDB entered the scene with some features that Wowhead could not offer: item comparison, wishlists, and a look more oriented on the WoW look. AFAIK their character preview was first, and Wowhead added that a bit after them.

What do all these databases share?

As it seems, the market for World of Warcraft databases is a hot one. Even four years after WoW made its’ initial release. And as we can see by the release of new and different databases, none of the contestants listed above yet has managed to reach a state where WoW players are satisfied.

If we compare the features of all available WoW databases we can see that it’s not all about features, but about design and usability, too. Some features did not really make it into the market, as players did not seem to accept them.

• Auction House monitoring: Yes, it is there, and some people might use it, but apparently this is not interesting or good enough to be used anywhere else than on Allakhazam. Apparently using spreadsheets still is superior. Also, there is Auctioneer which works in-game and automates a lot of the process.
• Crafting requirements: Wowguru introduced this quite some time ago, and I thought it was a great feature. When displaying a crafted item, you could see all the items required to create it, summing up the total vendor price for all needed materials. Nice, but oviously not needed. Gold seems to be an irrelevant factor, and thus only material lists survived.

This leaves us with a list of features that seem mandatory for a mature World of Warcraft content database.

• Searching: Of course, searching is mandatory. Every site has it, and the only difference to be found is in the way search results are returned. Older sites just take an argument to look for and return an result page with any matching object, Wowhead and WoWDB sport a bit more elegance and make use of AJAX to return drop down lists with matching data instead of requiring you to wait for a result page.
• Filtering: Even though the official forums are full of people demanding more content because the existing content would not be sufficient, a look at a database of your choice proves the opposite. There is lots of game content, and there is much to be seen. Even after four years, I am sure you will find content that you have not seen yet, even if you started your character in February 2005. Thus most database allow you to filter for various parameters, and most important: you can apply multiple filters to any kind of data. Sleak and useful.

And why would you want that Open Source’d?

It’s simple. Many people contribute data to these databases, and only with your or my contributions of data gathered while playing the game, those databases exist. It’s a community effort, just like the game itself is meant to be some sort of social community. But why are all of these databases closed, or commercial? Some even want you to buy premium subscriptions to access guides or more data.

Now what if there was an Open Source database for World of Warcraft? Where you could not only contribute data you gathered while playing, but also access that data or even download it to your computer and do something with it? Or where you could see what you have gathered while playing?

So I have decided to give myself a new pet project to work on, and it will become Open Source very soon, once I have ironed out the basic details. Work already begun, and a very basic implementation is there. The tools I have used are:

• Visual C# 2008 Express Edition to build a .NET 2.0 client application which handles uploads,
• Lua to write a WoW addon which gathers data while you play,
• Zend Framework and Doctrine ORM to write the server-side backend which accepts uploads.

The process for now is very simple:

• the .NET client will parse the WoW data cache (which you will find in your WoW installation directory under WDB/your-locale/*.wdb),
• it can show you the contents of your WDB files, e.g. you can see which NPCs you have seen, or what books you have read,
• it will parse the saved Lua data generated by the addon, and can show what data besides your WDB cache is there,
• and it can convert both WDB and Lua data to XML,
• and finally it can upload the data to my server (GZipped of course, to reduce bandwidth usage).

Parsing the data is something that will be done next. For now my server-side application only handles storing the uploaded data, and manages updates of the client. Not very much, but at least my data is stored. Parsing will be a bit more work since I have to make a modular parser that can deal e.g. with different data formats, as the data formats may and surely will change with every WoW client update.

What do you think? Would you use an Open Source World of Warcraft database, where you have full access to the data you contributed? Would you like to download contributed data in a parsed format, perhaps as XML files to reuse them on your guild website or your blog?

More on the project soon, after the finishing touches have been put on the data client. Screenshots coming soon.