Design Documents – Yes, you need them!

Hallsofvallhalla started a conversation on Friday questioning the value of design documents. I think he is mostly wrong and partially right. I’ll start with why I agree because that is easiest. Most design documents that I have seen are not terribly valuable because they focus solely on the color of the bike shed. They discuss things like level layout, class abilities, races, etc. and ignore the deep decisions about the physics of the world; how it actually works. The design doc should be useful to all team members; not just artists and designers. Now the other side of the argument:

Architecture Matters

Persistent worlds are certainly complex enough to require serious planning. Part of preproduction – and an important part of the design document – is figuring out how to do the hard problems. How will inventory work? How can it be implemented? Etc. Etc. Unless you want to constantly be throwing your work and starting over again, these things need to be worked out before you write a single line of code. All too often, I see freshly started projects publishing screenshots, which hints that they have skipped preproduction entirely.

While writing the Angela interpreter engine for Memotica, I’m reaching the point where I’ve got the core of the interpreter to the point where I’m writing the brokering mechanisms for actions and stimuli. I designed the overall, high level architecture of the engine over a year ago and looking back on it now allows me to continue without flailing about and redesigning the whole choreography. In short, the time spent then is saving me considerable time now.

The ship needs a rudder

Where is the project going? Is it still going in the same direction as six months ago? Or is it adrift and rudderless, buffeted by whatever is being talked about on the project’s forums?

A dirty secret about community/indy teams

One of halls’ argumenents against design documents is that it acts as a straightjacket that restricts creativity and makes the project less interesting to portential team members. One of the dirty secrets is that half of the people who show an interest in helping out a project are useless. They have neither the work ethic, nor the patience and tenacity to stick with the project and be productive. They will happily shoot the breeze on your forums and pontificate endlessly. Do you really want these people setting – and changing – the direction of your project? 90% of the other half are only valuable if they were recruited at the right time and there is consensus on vision; two things that are only possible after the valley of tears phase.

The valley of tears

Every PW project goes through the valley of tears. This is the phase in the project when the interesting theoretical discussions have been had and it is clear that it won’t take weeks to realize the world, but months and years. It will seem that “nothing is happening and the project is going nowhere” for long stretches as the foundations are laid. This is a time when no flashy screenshots are created, but instead the plumbing is laid in. In the worst case scenario, you will be the only person sticking with the project during this time. Projects whose leads allowed them to turn into designed by committee, vampire/puppy/rainbow monstrosities tend to die tragic and lonely deaths in the valley of tears; identified only by the lonely husks of once active forums that will one day disappear into the great 404. You didn’t let it turn into something that you are not entirely happy with during the pontification phase did you? Because it is your passion and drive that will see the project through to the promised land on the other side.

Please tell me that you are recruiting your team at the end of prepreproduction!

You have a prototype. It may be butt ugly. It may not be very performant and will need the attention of better programmers than yourself. The important thing is that the project is past the valley of tears phase, is picking up speed and has direction. The people you recruit at this phase will be attracted to:

• The fact that it has direction. Hey, we have all been on teams with no rudder or team stuck in perpetual pontification mode. Being on a team that is on the move has its own attraction and for years has been one that Luca Pancallo (Talad) of Planeshift has used in dev team recruiting drives.
• The vision. In the pontification (a.k.a. early design) phase, you will inevitably find that others want your goth-vampire world to be about puppies. Perhaps those puppies could have a vampire option as a compromise, but rainbows are much more important. The kinds of people who join because they read your design doc are true believers.

The live team that runs the PW won’t be the same as the one that initially works on it

I don’t say “built” or “delivered” as community worlds are never finished and are always works in progress. People come and go on the project and over time the composition of the dev crew will change. The people coming in will need to sign onto that vision. When a player (always a self identified “ideas” person) starts pontificating about how you need to change your world, you can point him to the vision. Not having a clearly defined vision contributed to Etilica’s death spiral. From the perspective of the leads, I was probably a rogue developer. If there was a design doc to refer back to, we probably would not have been in the habit of surprising each other with changes.

A final caveat. The document need not be set in stone

A design document needs to be a living document. You’ll want to prevent the pontificators from rewriting your design document, but you’ll also want the committed team members to effect change as long as they earn the right. As these committed team members get their ideas accepted, the design doc should be updated to reflect the new status quo.

Postmortem: The Etilica Soulflash – Part II

Continued from part I – (I realized that I’d never finished the Etilica Soulflash story)

When the soulflash went live on Etilica and the lore changes went into the documentation, along with a note on the forums detailing how death worked. There was uproar!

Generally, Etilica’s community fell into three camps:

1 – Those who did not think too much about the ramifications of everyone respawning – or were okay with it – and just found it kind of cool to find dead bodies from player character now and then. If you were on the server during off hours when it was very quiet and happened to be in a far flung zone, coming across a shell from a respawn hours ago was a concrete sign that you were not alone. Also, PCs could get a small amount of experience from cremating a shell and it served as a roleplay device

2 – Those who did not think too much about the ramifications of everyone respawning – or were okay with it – and were simply happy to rezz shells and do bizarre things with them. It quickly became de rigueur for evil characters to rezz shells of fallen enemies to mock them. The whole “turn your worst enemy’s cast off corpse into your mindless, living zombie butler” was part of the intent of the system and I heard that my own character’s shell was used for precisely this kind of roleplay. I was also given vague hearsay warnings about inappropriate usage, but was never told the details.

3- The last group was the ones who were not okay with the everyone respawn lore. They were extremely vocal about it. They were completely fine with the fact that their character was more or less immortal. What bothered them was the fact that slaying the orc king would now seem utterly pointless because he returned to life. They would be trapped in a groundhog day scenario forever.

They had something of a point. The problem was this was in fact already the case. There were a number of repeatable quests where the PCs had to kill named villains and bring back their heads/rings/etc. or rescue the children that they had kidnapped. Everyone conveniently overlooked the fact that these named villains – whose lairs we would probably now call raid instances – would probably be reset and ready to fight again even before their ring was turned into the quest giver. Where is the closure in rescuing the children when you go back out and there they are again? This is a problem that every PW, from the small text MUDs to WoW experiences. Players consume content. In the absence of compelling GM activity and in the absence of player driven roleplay plots, players will grind and run quests during quiet times. But this is another problem for another time.

These resistant players were correct about the groundhog day effect. The problem was that they were not really thinking through the effect of having a sub-population that was effectively immune to death, while the rest were. And for the record, if a god restores you to life every time you die, you are pretty fucking immune to death. They despaired at the pointlessness of it all. I despaired at the thought that the logical way to play the alternative out would be to effectively turn most NPCs into second class citizens at best and slaves at worst with the respawners as a divinely chosen aristocracy. Eventually I implemented a compromise. Everyone – PC and NPC alike – still soulflashed. All PCs and 10% of NPCs (all named bosses and 10% of spawns) also took their gear to the fuge plane. There people were destined to be respawned and were collectively the immortals. With such people, the shell left behind was naked. The rest of the NPCs left their gear behind when they soulflashed. They were truly dead. It was a sloppy compromise and I was never really happy with it; not least because I still had to paper over the cracks in the lore and pretend that human nature was not at play here and that the world would not develop a caste system. Nevertheless, the changes did not go over well and some people left. Ironically, one of the most vocal of the resistant players was a member of the dev team. He had ignored the posts on the builder forums leading up to the insertion of the new death system/lore into the world and only spoke up afterwards.

I took two lessons away from this:

1- “Everyone Respawns” is not an acceptable alternative to “Only Players Respawn”. Some players will happily ignore the knock on effects of their own serial resurrection, while getting hot and bothered about NPC resurrection. This reality and consideration of possible alternatives eventually led to HSM.

2 – In addition to being competent team players, devs actually have to participate in the developer discussions. This is kind of ironic in a way as – at least at the level of technical skill required to work on NWN1 – all development PW discussions are essentially bike shed discussions. Users don’t pontificate about ERP software, but they’ll do so endlessly about PWs. On the forums, they’ll give unsolicited advice on changes to the world that are either technically impossible to implement or miss the point of the world’s vision. If they are not posting on the forums, they are blogging. I know this phenomenon quite well. Over on Project Angela, the brainstorming discussions of what was to eventually become the Memotica design language were full of debate. I had trouble keeping up sometimes. When the discussions reached the advanced stage where deep thought was required in solving tricky design problems, I found myself being the only meaningful participant as often as not. When it came to actually doing the grunt work in the formal language design and writing the interpreter, I was completely alone.

In the case of that guy on Etilica, I still don’t know why he did not know what was going on.

Portmortem: Etilica – Management Lessons

I took away some management lessons from Etilica.

A world needs a “vision” that everyone on the team signs onto - That vision needs to include technical standards and accepted approaches to problems. In short, there needed to be a design document. Amateur hobbyists don’t usually write design documents, but among hobbyist teams, it is very common for the initial builders to pass the torch, even when the admins remain the same. ETI was built by multiple generations. The people working on the server when it left beta were not those who did the initial work and they were also not the ones who took over as the live team. Every builder that ever worked on the server had his own way of doing things. The one before me who called himself a “lazy scripter” in his comments made me want to poke my eyes out with a fork. Sometimes, those ways did not mesh and sometimes they actively undermine the vision.

Selling builders on the vision is a marketing task.

Know when to ignore the forums - Also in the face of player whining. If you want your world to be about cute fluffy bunnies that also happen to be demon worshipping undead, then stick to that vision. Somewhere out there, there are players looking for exactly your vision. If you are constantly modifying it to the whims ofthose for whom your vision does not coincide with their idea of perfection, then you’ll turn your world into something that you no longer love; and those players will still leave. Unless it fits their idea of perfection, or they have a social anchor, they will be fickle and leave sometime.

This requires discipline. The temptation to change the server to fit the players that it has is strong; especially in a genre or platform that is on the wane and difficult to find players for. I was severely guilty of this. On more than one occasion, I put the server into a gyration because of what the players were saying on the forums. This lack of an anchor inadvertently helped its downfall I think.

Changing the nature of the server overnight – The admins and builders who brought the server through beta started a new server plot shortly after the end of beta. A far away undead nation ruled by a lich emperor took over the main city and started a war. Overnight, the server turned very dark. It was a good server plot and gave the players a “reason” for their characters actions. Unfortunately, it was too dark for many players. There was loud protest and the playerbase rapidly dwindled. The leads lent on hiatus shortly afterwards, probably due to a feeling of discouragement, and there was a complete turnover of the live team. Generic D&D fantasy and goth undead lovers are two distinct crowds. You can’t gear populate the server with one and then gear it towards the other. This means that if you do have large, overreaching, server plots, they have to be consistent with the types of players that you are attracting to the server in the first place.

Wait! Is this not contradictory to the “know when to ignore players” rule? Not really. Just as selling your vision to the team is a marketing task, so is selling it to the players. BUT… you can’t bait and switch.

No PW builder/dev should ever say “I can’t script/program”. They should say “I’m going to learn how to script/program” – Too many people who tended to say the former worked on Etilica. NWN has a nice add on tool created by Lilac Soul. It is a script generator. It greatly helps builders who lack the skills and are not willing to learn them to build modules quickly. The problem is that persistent worlds are complex enough that if you have to use a script generator, then you are going to end up with a buggy, Frankenstein monster.

Thrym is an excellent example of how to do it right. He knew nothing about NWN’s scripting language (or modding that engine in general) when he started building Markshire. He had a vision and the drive to learn the necessary skills and became very competent.

The Managers need to be technically competent enough to have a clue – What I said above for builders hold especially true for the management. You need to be prepared to wear all hats. Yes, this might mean learning to code or animate. Everyone has a dream that they would like to build. The difference between a lead and a builder is that the latter has decided that his vision is too big to build himself and that he’d rather work on something else that sees the light of day rather than his own project that might never do this. A manager who is not willing to learn the requisite skills is entirely dependent on others; finding those builders who don’t want to waste time on their own visions. In doing so, he is making his world less attractive. Seriously, who wants to do all the gruntwork for someone else’s vision if they are not willing to do it themselves? It also makes the world vulnerable to being altered by a builder who does not share that vision.

Communication is like oxygen to the live team– It should be so obvious that it does not need to be said, but there needs to be a central communications hub. The builders, including and especially myself, were very active on Etilica’s forums; but used IM very little. The head admin was rarely on the forums, but held court on IM. Some of the DMs used IM to varying degrees. This was a recipe for non-communication among the live team. It actually surprised me on more than one occasion when someone who was not active on the forums or in development was made acting lead.

I’m strongly partial to forums for their asynchronous nature. This keeps the whole team in the loop regardless of time zone or the hours they keep. Even when IRC and IM chat logs are archived for those not present, the signal to noise ratio usually makes them so painful to read that nothing is gained.

Postmortem: Etilica’s Lag – Part II

Since this is post two of the lag theme, I’ll catalog the things that I would have done differently as a live team dev. Later in the week I can delve into design and management lessons. I’ll try to keep the lessons generic and skip the platform specific ones. What I’d do differently taking over running the live team of a world now:

Perform an exhaustive survey of the world – Let’s not trust the documentation; even if it exists. Even among professional teams, the design docs are likely to be outdated. Let’s assume that we don’t have a large block of time for a handover. Up to date design docs and handover meetings are luxuries that professionals have. The alternative that amateurs have is to survey the server. Don’t change anything until all of the resources are understood.

Run scheduled stress tests – Do you have a new AI package? Have you completely revamped your honey pot zones? Do you have a fancy schmancy new dynamic content generator or crafting system? Run a scheduled stress test on a dev server before making those changes live. Advertise it to your playerbase like you would an event. Beg, borrow and steal players from other places just to be there for the stress test.

Make sure that there is only ONE definitive version of the module – From Thrym’s observations of the Etilica in the comments on part one, I suspect that he is looking at a different version than I worked on. I know that there were multiple, unsynchronized versions of the module lying about. Sometimes it was a mystery as to who had a version of the module with which changes. I made the biggest changes and made the most cleanup, but other versions sometimes slipped into the live position or were passed to builders. Etilica should have been committed to a version control system such as Perforce or SVN, so that changes could be tracked, rolled back and there would be only one head revision.

What I would not do is start overhauling the module at the kind of rate that I did last time - There were certain things that needed to be done to whip Etilica into shape that were not really practical, or were just so painful to do that nobody wanted to do them. First and foremost, there was the issue of the half-implemented and orphaned script packages and other resources. A huge portion of the module’s resources were deprecated, but could not safely be removed without an exhaustive and time consuming survey of the mod. There are no tools for that engine that allow you to search for deprecated and unused resources. Moreover, many obscure resources WERE used somewhere. You would be surprised how many different AI scripts were in use and how many item and creature templates popped up somewhere; but making changes too rapidly in a cruft encrusted environment is asking for trouble.

Postmortem: Etilica’s Lag – Part I

Several former Etilica players have commented about the lag on that world. That is my fault and I learned a great deal from it. Allow me to explain. In part II, I’ll list what I would have done differently if I were to work on that world again.

The buzzword on ETI is cruft. ETI was built by three generations of builders who overlapped very little or not at all. It had about 250 areas (small zones) and an immense number of placed NPCs when I started overhauling it in earnest; even in places where players rarely went. I think I counted 400 placed NPCs when I started cataloging them. ETI’s NPCs also had a heavy heartbeat script (which fires every six seconds on that platform) and heavy combat scripts; courtesy of me. I think in many respects, it catalogs a list of mistakes that can be made when building and running a server. There are many things that can be learned from examining the life and death of ETI, but here I will concentrate on my lag. When I took over as lead builder, I made two server killing mistakes:

I did not completely familiarize myself with the mod as much as I should have – This was ultimately fatal. There were big parts of the module – especially in the desert (about 2/3 of the world was in the desert) – that were so hard to get to that I only ever saw them in the toolset or via DM client teleport when testing. The very first thing that I should have done was to catalog all of the areas and familiarize myself with them; even the rarely visited ones. I quickly grew intimate with the high traffic portions of the mod, but not the low traffic areas. Had I done this, I would have been aware of the placed NPC count.

I did not stress test the server before making changes live - I brought in a couple of AI systems that I had worked on for my NWN version of Epoch. They included custome a waypoint walker, a new heartbeat routine and a heavily modified version of Jasperre’s combat AI package to spice up combat.

Altogether, they were heavy in terms of CPU. They worked fine on Epoch, which was small and heavily dynamic. Most NPCs were only around when there was actually a PC in the area. On ETI, this caused an enormous amount of overhead on the server. It only took one player getting into a heavy fight somewhere on the server to cause serious lag. Also, there was a rebuild of the main high traffic areas. They were arguably nicer, prettier and more immersive. They were also prone to lag afterwards as well. Sometimes a plain soundstage is the best. A proper scheduled stress test would have revealed this.

Postmortem: The Etilica Soulflash – Part I

Since subject of death systems came up in the last post, I thought I’d elaborate on the death system used on the long defunct NWN persistent world Etilica. Etilica’s final death system came about in an accidental way. It originally had a fugue plane death system; where the player is teleported to the afterlife on death, leaving a corpse behind. It had stopped working when an admin who could not script installed a script package that overwrote one of the server events that the old death system used and I had more or less a free hand in fixing it. So I got to working on the appropriate NWscript fu.

My goals were simple. I was not looking to overhaul both the mechanics and lore of the death system, but fix what was there and possibly make it nicer. I always disliked the jarring instant teleport that is typical of fugue plane systems. One moment you are in combat and then suddenly you are looking at a load screen. I set it up so that the player got a popup on death asking them if they wanted to wait for a rez or respawn with a penalty. If they respawned, then they went to the fuge plane from whence they could exit and go to the bindstone that they had last selected. I also disliked the generic corpse object that was left behind. Since that version of NWN did not have the SetName() script command, players could not tell who’s corpse it was. Why not make a copy of the deceased PC, kill it and leave it behind so that everyone knows who’s corpse it was? It can be removed when the player leaves the fugue plane. Great! I fixed the problem and added a nice tweak!

Except that I did not properly test it before putting it into the live server. The problem was that the clone was rezzable. Sooner or later, a well meaning character was going to pull out a raise dead scroll after a player had hit the respawn button, but before the clone was cleaned up. When this happened, they effectively had a zombie on their hands. It did not even have a heartbeat script, so it was more of a standing vegatable. This being a roleplay server, the players ran with it and it was the source of some fantastic RP. I could have just fixed the clone so that it was not rezzable anymore, but after seeing the RP, I had an idea; and thus was born “the shell”.

I saw a way to logically resolve the whole PCs respawn and are not lootable, but NPCs perma-die and are lootable problem. After a few seconds (or when respawn was clicked in the case of PCs), everyone respawned and everyone got a new body. An empty shell was left behind. The difference was that the PC’s shells took their gear with them. To give some sort of plausible cover, 10% of all NPCs (randomly determined) also took their gear as well. The cloning, killing and stripping process caused the corpse to jerk about a bit, so I added a flashy effect to cover it. This was the soulflash; when the soul left the body (and sometimes took possessions as well).

I came up with some lore that shells could be rezzed (and made the appropriate technical moves to make them viable mobs) to be what would effectively be a zombie; a golem if you will. As this golem was a mindless similarcum of a living person and was in fact their old body, the idea of a shell would be horrifying to the populace. People burned shells quickly to prevent either necromancers from turning them into undead, or other malcontents from raising them from the dead and using them as living zombies. I also reckoned that since in some cases the soulflash also takes the possessions, that there would be a prohibition against public nudity; lest that person be mistaken for a shell.

To be honest, I never really delved too deep into the ramifications of everyone respawning. The background that I came up with was that everyone respawned, but I was never really happy with this. I realized what a profound effect this would have on life, but I was not prepared to venture into the sort of society that would create. Nor was I about to start those changes on a live server with an active playerbase.

When I tried to discuss it among the builders and GMs, there was a distinct lack of interest in the subject; either positive or negative. As the server owner was too busy playing WoW at the time to even check in on the world that was ostensibly his and I was the member of the builder/GM team with the highest degree of involvement, I was the de-facto lead. Finally I posted on the developer boards that I planned to put the lore and technical changes in if nobody objected. There were no objections and after a week I made the changes live.