On Architecture
Chapter 1, "Introduction to Virtual Worlds," described the way that virtual world servers are constructed in terms of four different layers of functionality (driver, mudlib, world model, incarnation). This is the breakdown of most interest to designers. However, an understanding of how the rest of the system hangs together is required, at least at an abstract level. That's what's discussed next.
Overall Architecture
Actual architectures differ from developer to developer, but they can almost all be regarded as variations of a single generic approach that emphasizes reliability, scalability, and maintainability. Figure 2.1 illustrates this overall architecture.
)
Figure 2.1 Overall architecture.
Here's how it works.
Central to the system is the user database. This is a powerful piece of software running on its own, fast machine with lots of storage. It contains records for all the players registered for the virtual world. Before anyone can access content, they must first log in; this means checking with the user database. Other parts of the complete system (not shown in Figure 2.1) for billing, customer service management, and patching also have access to the user database. It's a very important, industrial-strength system, and is therefore usually bought in rather than programmed in-house.
Players have a number of options for connecting to the virtual world. The main one is to use a client from a PC (or console or Macintosh or Linux box—the clients will all present the same interface, so the virtual world neither knows nor cares what operating system they're running under). Players may also connect to the host by using a browser, but not to the same extent as with a bespoke client; although there are text-based virtual worlds that do have good browser-based (Java) clients, no-one has yet been reckless enough to try the same kind of thing for a full-blown 3D graphics-based one. Similarly, mobile phones fall way short of being usable for actual play. However, both browsers and phones can be used to obtain information from a virtual world (for example, news, the virtual weather) or to make changes to standing orders (for example, training regimes, the prices of goods offered for sale). Phones and email can be used to inform players of unfolding events, but they don't usually offer the chance to participate in them.
When an incoming connection is established, it is handled by a front end. Front ends can communicate with both their target platform (client, browser/email, mobile phone) and the user database. Figure 2.1 shows the various front ends as separate entities because they're separate processes; however, in practice they may be consolidated onto a single machine or be split across several (for example, by real-world geography).
Individual incarnations of virtual worlds run on server clusters (also known as shards17). The architecture of shards is described in detail shortly, but for the moment a shard can be regarded simply as a unitary server entity. Having approved a player for access to content, the front end will either query the server to which the content relates (for browser or phone connections), or pass control to the server itself (for client connections). The server does not need to access the user database in order to support the virtual world, because servers deal with characters rather than players. Customer support staff, however, deal with players rather than characters, so they do need to be able to check the user database from within the virtual world. This is sometimes done using a separate database tool that they can invoke while simultaneously using the regular client, but not always. In particular, if the client owns18 the player's PC, a special admin client is needed with the database tools built in.
If this is so, then either the client has to be able to maintain two connections (one to the server, one to the user database) or—more easily—the server itself can issue database requests when required, passing the results back to the client.
Server Architecture
Server clusters implement instances of the virtual world. There may be programmed differences between them (attacking other players might be allowed on one but not on another, for example), but these are generally minimal in form if not effect. The number of servers present in a cluster varies from implementation to implementation, but it's usually around half a dozen or so. The number of clusters also varies, with more clusters being added as a product's popularity increases (EverQuest hit 40 server clusters in 2000, averaging around 1,500 players simultaneously on each one at peak time).
Figure 2.2 shows how a server cluster is typically configured.
)
Figure 2.2 Server cluster architecture.
Individual clients are connected to individual sub-servers. Ideally, each sub-server does the same amount of work (which in practice means it handles the same number of players) as every other sub-server. Sub-servers have access to a shared character database19 that stores the persistent data relating to players' characters on this server20. This may or may not be part of a systemwide accounts database for managing player access. For maintainability, a large-scale world generally keeps a separate database for environment data, which may be partitioned into a template database, a scripting language database, and an instantiation database (as described in Chapter 1).
Furthermore, whether these world/environment databases are shared (as shown here) or local to each sub-server depends on how load-balancing works. EverQuest's zoning system, for example, can get away with having smaller environment databases that are controlled by individual sub-servers; only the character database needs to be shared with the other sub-servers, for when characters move between zones. This is paid for in other ways, of course, which we will discuss shortly.
The hardware implementation of a server cluster is as a bunch of PCs or beefier hardware (typically running some flavor of Unix) connected over a LAN. Figure 2.2 shows the usual pathways between elements of the server, but in fact any sub-server can talk to any other sub-server should the need21 arise. Thus, passing players between sub-servers can be achieved either formally (through the databases) or informally (by direct negotiation between machines). Individual sub-servers may also be in contact with the user database, as shown in Fig. 2.1 (but not in Fig. 2.2, so as to prevent its suffering from dashed line death).
I should point out that academics are experimenting with other architectures, particularly the distributed kind much loved by Science Fiction (Idoru and Otherland both use it, for example). These, and those involving multicast, are not, however, likely to be used beyond academia due to the security and liability issues they raise.
Load Balancing
Fortunately, the arcane subject of load balancing is not something with which designers need concern themselves directly. However, they do need to be aware of the consequences of whatever solution is adopted by the technical experts, especially because they are likely to be consulted on the matter.
Ideally, a virtual world would run on a single, very powerful computer. For textual worlds, this is already the case22. Graphical worlds may go the same way23, but for the moment there are still plenty of things on which newly available computational resources could be spent at the server side:
Increasing the number of players present in each incarnation (100,000 players on 50 servers is one thing, but 100,000 players on 1 server is something else).
Increasing the level of detail at which the virtual world functions (for example, leaving footprints in the snow that fade as it melts or as new snow falls).
Improving the AI of the virtual world's denizens (both in quality and quantity).
It is therefore likely that virtual worlds will soak up whatever additional computing power is thrown their way for quite some time. Inexpensive machines will still be clustered, because it will always cost less to use eight computers of power X than to use one computer of power 8*X (although management overheads degrade overall efficiency as new machines are added, which is why no one would use 64 computers of power X/8)24.
So, given that distributed servers are here to stay, how does this affect the designer?
Well, the virtual world is too big to fit on one computer; therefore it must be partitioned over several computers.
Sort of....
If the virtual world were inert, that is, nothing ever happened in it, it wouldn't need any computers at all, it would just sit on a disk pack. It only needs computers when things happen in it. The issue is therefore one of ensuring that activity is spread across sub-servers such that they can all cope with the work they have; in other words, the computational load must be shared in a balanced way (hence load balancing).
The greatest source of activity in a virtual world is the player community. Every moment that they are in the virtual world, players are interacting with it. Merely moving from one location to another entails informing every other character that can (in the virtual world) see you do so. Yes, some activities are more of a drain on resources than others, but they tend not to be specific to particular playing styles. The issue is the sheer quantity of actions being performed, not the efficiency of individual actions. Load balancing in virtual worlds therefore generally means ensuring that roughly the same number of players is connected to each sub-server in the cluster.
The obvious way to do this would be to assign each incoming player to whichever sub-server has the fewest players. It turns out that it's quite difficult to do this without introducing big overheads, though. To update the instantiation database to reflect an action, records need to be locked to prevent other sub-servers from also changing them at the same time (for example, if two characters attempted to pick up the same object simultaneously). The sub-server needs to lock all records that it could need during an action, perform the tests to ensure that the action is possible, make any necessary update requests, and unlock the records. This is a lot of locking/unlocking. It would be really handy if there were some way to block-book records in the instantiation database for long periods without relinquishing control of them. Are there any types of records for which this could easily be done?
Statistically, most actions performed by players involve movement25.
The player wants to move their character from A to B, so the sub-server has to lock location B, check if it's empty, if so then move the player into location B, then unlock it. Location A must also be locked, so anyone wanting to do anything to the character that assumes it is in location A (for example, teleport it to location C) will not inadvertently screw up things. Many other common commands (particularly get/drop and those to do with communication) are also location-based.
For these reasons, servers typically partition responsibility for the virtual world along (virtual) geographic lines. A sub-server can lock location records in the instantiation database for extended periods; indeed, if the system is programmed correctly, it doesn't have to lock them at all—it has implicit control by mere virtue of the fact that none of the other sub-servers do.
To summarize the argument so far: We want to spread the players fairly evenly across sub-servers, but the obvious way would introduce too many overheads on database access; a far more efficient way to do it is to partition by geographical location. The question is: Would using this partitioning model give us load balancing?
The answer is that yes, it would. How, exactly, depends on the virtual world.
There are essentially two approaches: fixed load balancing and dynamic load balancing. The former, exemplified by EverQuest, assigns a predefined geographical location (a zone) to one sub-server26; the latter, exemplified by Asheron's Call, moves responsibility for geographical locations between sub-servers.
Fixed load balancing:
Is easier to implement.
Can partition the instantiation database and keep it local to the sub-server, for greater efficiency.
Allows the client to figure out in advance what texture maps will be needed and preload them into graphics card memory.
Dynamic load balancing:
-
Has seamless terrain (you can see to the horizon27).
Has boundaries that are not physical (monsters chasing you don't get stuck at zone edges).
Balances the load better.
What does all this mean for design?
It's an example of where technology imposes constraints. For fixed load balancing, zones can be created with greater individuality: The 'physical' barriers between them allow for radical change. Players can cross from one to another and expect to see something different on the other side. For a seamless system, sudden changes have to make more sense or they'll seem out of place.
The fact that content such as monsters can't cross zone boundaries means that players will use different tactics in such a world than they would in one where there was always the possibility that a creature they had royally annoyed could pursue them relentlessly. Victorian London's police forces couldn't (legally) cross precinct boundaries: Jack the Ripper would commit a murder in one police district and then run into another where the police were not allowed to follow; if they could have followed, he would have had to rethink his getaway strategy.
Zoned worlds have something of a problem with flash crowds (people appearing instantly in the same vicinity in response to interesting news; the term comes from a 1973 Larry Niven short story28). Most of the time, each sub-server will be handling similar numbers of players. However, sometimes something happens that causes everyone to want to be in the same general locale29. Maybe it's a rare spawning of an impressive dragon, or a social event such as a wedding or guild rally. Whatever, all of a sudden more people want access to a server than it can handle. The designer has to decide what to do when this happens. Do they simply show a "zone full" message if people try to enter it? Do they let them in and leave Customer Service to handle the resulting complaints about lag? Do they organize the virtual world such that it would be counter to its fiction for everyone to want to be in the same zone?
Dark Age of Camelot partitions its player base into three "realms." Members of one realm can't enter territory belonging to members of another realm. They can enter a no-man's land between two realms, however, which is where realm versus realm combat takes place. The upshot of this is that unless there is a serious skewing of the DAoC player base, only a third of the players online will usually be present in any one realm. That's excellent for load balancing. What's more, realms are aggregations of zones, they're not zones in themselves. Each realm is made up of 13 outdoor zones (64K by 64K squares) plus five dungeons plus one city. There's no reason why a sub-server has to handle zones from only one realm; if (for some reason) 75% of the players are all in one realm, the sub-servers handling that realm will automatically have less load from the other (sparser) realms they're controlling.
Players of EQ notice how the sub-servers take responsibility for zones, but players of DAoC don't. Why not? Because of the latter game's world design.
Other Things Happen
Although players are the main source of the load on virtual worlds' supporting hardware, they are not the only one. Things can happen whether or not players are present. Some of it is mechanistic: The virtual world's sun rises and sets, its weather comes and goes, all irrespective of whether there are players around. For a highly detailed world, this could amount to considerable work (a breeze rustles individual leaves on a tree, one of which falls off to land in a stream that carries it lazily to a river and thence to the sea). Virtual worlds of this complexity are some way off at the moment, though.
What's more of an issue is the presence in the virtual world of virtual creatures. These are commonly known as mobiles30 (mobs for short), and they represent the monsters and non-player characters that inhabit the virtual world. They are discussed in more detail in Chapter 4, "World Design," but what concerns us about them right now is that they need to behave in a believable manner. This requires artificial intelligence techniques, which gobble up computational resources like nothing else. Even simple path-finding is an insatiable consumer of CPU cycles. It would be great to have a virtual city with 100,000 virtual inhabitants, each making real-time decisions as to how to spend their virtual lives. We may have to wait some time before we get this, though.
If a designer wants more mobiles in the virtual world than the servers can handle, they have to offer solutions for managing these mobiles. The classic answer is to suspend processing of those mobiles whose actions would not be witnessed directly by players. What causes AI load is not the number of mobiles on a sub-server, but the number that are active at any one time.
Consider a group of goblins in a village. With no players in the vicinity, there's no point in having them do anything. Sure, a sub-server can move them around a bit when it's not unduly loaded, but players get priority. Only when a group of adventurers shows up is it time to activate the goblins so they can behave intelligently and give the players a run for their money. When the players leave, the dead goblins can respawn and wait for the next batch of adventurers.
Designers who want more mobiles than the programmers tell them they can have might be tempted to use this proximity activation approach. They have to realize, though, that every decision they make has consequences.
In this particular case, the consequences are on causality. If a tree falls in a desert, does it make a noise? Using proximity activation, it would never fall in the first place.
Consider a second goblin raiding party. It emerges from its camp, kills some villagers' sheep, and then returns home with the spoils. The villagers get angry and offer to pay players to kill the goblins.
It's an evening's quest.
It's an evening's quest that would never happen if the goblins stayed in their camp until a player happened upon them.
Yes, of course, plenty of ways around this instantly present themselves, but that's not the point: What's important is that the designer has to recognize that there may be a problem in the first place. Would your proposed solutions to the problem come with problems of their own? Would you have thought about that if I hadn't asked?
Virtual world design is about consequences.
The Client/Server Model
The server embodies the virtual world; the client translates it into a form the player can comprehend. Because the client is the player's window on the virtual world, designers have a lot to say about its look and feel. Much of this is a matter of taste and convention, though, and will therefore not be discussed here. Clearly, it does make a difference if your proposed spell-casting system is too complicated to be implemented for mouse and keyboard31, but if this kind of thing isn't obvious to you, then you've no business being a designer anyway.
There are, however, engineering considerations specific to the implementation of virtual worlds that impact directly on the virtual world itself. These are unavoidable, and designers must be aware of them.
To illustrate: In theory, the same virtual world could be presented using different skins, so whereas one player might see a defense robot discharging an energy weapon, another might see a wizard wielding a wand. This is something that players would find novel, but it presents a major challenge to designers. Few genres map onto each other to this extent, so compromises would have to be made.
As it happens, this particular example is a red herring because the amount of artwork necessary to support just one genre, let alone two, is considerable. The benefits aren't worth the cost. Textual worlds are easier to reskin than graphical ones in this respect, of course, but have less reason to want to do it32. In practice, using skins for graphical clients means altering the look and layout of the client's interface, not the look of what it displays of the virtual world. Hanging different curtains doesn't change the view through the window.
It turns out that there are only two issues relating to the client/server relationship that have concrete effects on the design of virtual worlds, but both of them are very important: synchronization and security.
Synchronization
It takes time for information to travel between the client and the server. It takes time for the server to execute commands. During this time, the client has to maintain its display of the virtual world. What it shows may not, therefore, be a true representation of what the server defines to be the current world state: The two aren't synchronized. In the real world, the sun might have spontaneously exploded 19 seconds ago, but you're not going to find out for another eight minutes. In a virtual world, someone may have quit four seconds before you loosed an arrow at him.
Lag due to server load can be addressed by buying in faster hardware and by optimizing code. There's little that can be done about lag due to communications, though. Even with a perfect connection, the speed of light through glass33 is such that someone in Sydney playing in a virtual world with servers in San Diego34 would experience a delay of over 0.06 seconds in each direction with a direct cable connection. The fact that their communication has to go through routers and isn't in a straight line brings it up to more like a third of a second. Throw in an analogue modem and you can add another third of a second. Lag happens.
Designers have to account for this by making nothing too time-sensitive. In a regular computer game, players can be expected to make timed runs through windmill sails or giant steam pistons or pendulum scythes, but in a virtual world there's no guarantee that when the player sees a gap on their client the server is actually implementing one. For this reason, players can never be called on to make reflex actions (although their characters can be), which means virtual worlds have little or no twitch.
Timed actions are possible, but the window for success needs to be at least four seconds in duration or some players are going to miss it while believing they hit it. Nevertheless, improvements in Internet reliability have led to attempts to bring the kind of response times common in first-person shooters to virtual worlds: Planetside35 is the best-known pioneer.
Precise timing issues aside, virtual worlds are fairly robust in the face of lag. The exact moment that a player initiates an action is rarely important; when players agree to do something "at the same time," they know it's a fuzzy concept. As long as designers avoid doing anything deliberately that requires speed, there's rarely a problem.
Although most of the lag that players endure is fairly constant across a connection, not all of it is. Sometimes, lag can be intermittent: Things can work fine then suddenly halt for no apparent reason. This kind of lag is actually due to bandwidth issues that cause a service provider to invoke some kind of resource allocation scheme. It doesn't matter how good your broadband connection is, if your ISP isn't sending you the packets, then you're not going to see them. This kind of lag can last several seconds, whereupon all the packets that are buffered up are dispatched at once and service resumes as normal.
Clients expect regular packets of data from the server to update their local state. Most of the time, these arrive in a timely manner. However, it only takes someone using the same router as you to start downloading copious quantities of pornography and packets will inevitably be delayed. What does the client do in such circumstances?
The easiest solution is to do nothing. Just sit and wait until packets arrive, then update them in sequence. The problem with this is that the virtual world effectively freezes for the player until the updates arrive. The player can try to do things, but gets no response until the packets start flowing again. This isn't always a problem in textual worlds, but it looks very disturbing in graphical ones.
Most graphical clients therefore use a predictive model, whereby they continue moving objects along whatever course they were taking the last time information was available. If a character is running east, the chances are that when an update packet finally arrives it will still have the client running east, so the screen will be right without ever having frozen. Predictive models work well when their predictions are correct, which (fortunately) is most of the time. However, there are problems when they're wrong. If the character who was running east had stopped and turned north, there could be a serious discrepancy between their actual position and the position the delayed client is displaying them at.
To correct failed predictions, there are two approaches. The traditional way is to use the new information and just forget about the predictions. This results in an effect called warping, which originated in Air Warrior. Players would be on the tail of an enemy plane when suddenly it would disappear from their sights to rematerialize instantly a short distance away like it had made a hyperspace jump. Players even found ways to induce it, so they could plan the reappearance to gain a tactically superior position.
More modern clients apply gradual translations to the displayed position of an object so that it moves to its correct position smoothly (if a little more slowly and still looking highly suspicious).
For designers, this means that not only can't they use relative time (in three seconds) but they can't use relative space (dead ahead) either. The client might think a particular coordinate is slightly to the left of the player whereas the server knows it's slightly to the right. Commands that need coordinates therefore have to use absolute ones rather than relative ones. Again, when designers are aware of the problem and don't call for players to follow complex instructions in mazes or anything, virtual worlds are usually sturdy enough to cope; a little error in absolute positioning is fine. They do have to be aware first, though.
There is a special case, however, in that sometimes players want to do things to other players at a distance, for example, shoot an arrow at them. In this situation, not only might the archer's position be at odds with the server's definitive version, but so might the target's. The client knows that the player wants to shoot an arrow at a character, but doesn't know for sure that character's coordinates. The designer must decide whether the command is "shoot at a target" or "shoot at a location" (which hopefully contains the target). If the arrow has a timed flight (and if it's modeling a real arrow it certainly ought to), the potential for error increases even more.
Bleah!
Graphical virtual worlds are presented as being continuous. Although characters might actually occupy integral coordinates, they are animated such that they move smoothly between them, thereby giving the impression that they at times occupy the spaces "between" the coordinates.
Textual worlds can be continuous too, but they are usually contiguous. Locations can represent an area rather than a point, and several characters can occupy such locations36 without causing an anomaly. Restricting horizontal movement to eight compass points means that using relative direction in such virtual worlds is a definite possibility. True to form, there are textual worlds that allow players to use both absolute (north, northeast, east, and so on) and relative (forward, ahead right, right, and so on) directions for movement, with the room description format (absolute or relative) also under player control. Graphical games can take relative coordinates from a client, but they have to transmit them as absolute ones.
A common mistake among inexperienced client authors is to take all this with a pinch of salt. So what if the client and server have slightly different ideas as to what is where pointing in whatever direction? There isn't going to be that much divergence. Okay, so maybe occasionally you see someone run through a solid windmill sail because there is a gap on their client, but that's hardly a show-stopper. Let the client decide if an arrow hits, rather than putting the burden on the server. There might be a few hits that should be misses and misses that should be hits, but it'll all even out over time.
This brings us neatly to the issue of security.
Security
Your client software will be hacked. For some virtual worlds (such as those with no game aspect to them), this won't necessarily matter. For the rest, it matters a great deal.
At the very least, it means that all packets sent from the client to the server have to be checked to see if they make sense. Error-correction at the hardware and transport layers should ensure that what arrives at the server is what was sent by the client, so why waste time checking for nonsense that isn't going to come? Well, the fact is that the server may not be talking to a bona fide client. It could be talking to a piece of code a player has written to masquerade as a client or to insert data into the client's packet stream. If the server receives nonsense but has no way of handling it, what happens then? Has someone acquired an ability to crash the server?
That's if it's even your server. Sometimes, groups of players will write their own server and persuade clients to speak to that instead. They can then design and play their own world in preference to yours—and all for free37!
Virtual world programmers don't have to make life easy for hackers—they can use encryption, own the screen, make very infrequent identity checks that packet-sniffing software may miss—but eventually their code will be reverse-engineered and people will figure out what's going on. Then they'll change it38.
Important: Absolutely no decisions with regard to what happens in a virtual world can be delegated to a client. No decisions. That's no decisions.
Air Warrior's first client performed the necessary calculations to determine if a shot hit or missed. It was considered unfair for players to line up their sights exactly on a target, pull the trigger, but miss because their client was showing the target to be somewhere the server didn't think it was. All very laudable, until someone wrote a hack for the client such that whenever you fired, it sent a packet claiming you'd hit whichever plane was closest to your sights—irrespective of whether it was actually in them. Kesmai had to bring out a patch to fix it.
Programmers should never put world-critical code in the client. If they do, it can mean major, big-time fraud. Designers only need worry about this if they have to come up with a strategy for repairing the damage should it go undetected for too long (for example, groups of players giving themselves money and spending it in inflation- causing amounts).
Designers do have to worry about things such as automated play. Anything that requires similar actions to be performed repeatedly is usually easy to automate—it doesn't even need a client hack in many cases, as there are off-the-shelf tools that will do it. Ultima Online's craft system was so boring that players were overjoyed when they discovered macro software intended for typists that enabled them to save their index fingers from repetitive stress injury. Computers can generally issue commands faster than players, too: Someone wrote their own client for MUD1 that stuffed commands down the line so quickly that 30 seconds after having started to play, the automated character would be standing with arms full of vicious weapons and other useful kit, while everyone else was still pretty well empty-handed.
Designers should therefore avoid calling for anything that involves doing something again and again and again with no respite (and that can include movement). If you really want an action to take a lot of time, let characters do it as a background task while their players are offline. If speed is occasionally important (for example, for dramatic reasons), insist that the server programmers institute delays between processing the commands from any one player39. If some level of searching or exploration is required (for example, for a puzzle), put in moving obstacles or traps so a program can't easily find a solution using a brute-force method.
People usually want to automate tasks that are tedious. If you design something that you think many players might like to automate, consider the possibility that it could be intrinsically uninteresting. For example, players will come to understand a virtual world far better if they make their own maps, so a designer might want to encourage them to do so. However, the actual mechanics of mapping are so mind-numbing that there are auto-mapping programs around (for textual worlds) that try every exit from every room until they have produced a complete map that can then go on a web site. If you were hoping to stimulate a climate of exploration by holding back on embedded mapping tools in the client, it didn't work.
There are other ways in which wily players can subvert a client to put themselves at an advantage. Consider the case of a character executing a 360° turn. The client must be in close, rapid contact with the server to ensure that the necessary information is at hand to display the world as the sweep progresses. If it weren't, then the player could be looking at a half-blank screen or seeing characters pop up in the middle of the view that weren't there an instant earlier. However, as was pointed out earlier, the client can't usually rely on a super-fast connection to the server.
To solve this problem, the EverQuest server was configured to send the client more information than it strictly needed; if a player wanted to turn, then the positions of nearby objects would already be known and could be displayed without lag effects. To this end, the client was told the locations of all objects and players in the vicinity—not just those that character could "see"—along with other handy tidbits, such as what they were carrying and how many hit points they had. The geography also was kept permanently available on the player's PC.
This was an efficient approach, in that anyone could rotate or otherwise change the camera angle with impunity without suffering staggered images. However, a number of players soon realized that if the information was present, it could be displayed whether or not it was in actual use. They wrote programs to give radar-like readouts of everyone and everything in the neighborhood. No longer did mobiles attack them from behind; no longer did they need to guess which were carrying loot. One such program, ShowEQ, was so useful a tool that Verant felt compelled to ban its use (leading to one of its many public relations disasters40).
It doesn't even have to be the client that's hacked. Ultima Online used the PC graphics card to control brightness, which meant that opportunistic players could override the client (by turning up the gamma correction) to get full daylight when they were supposed to be in total darkness. Great for ambushes!
If at all possible, designers should be adamant that no items of data are sent to the client that convey any information beyond what the player's character is entitled to know. Client programming being the inexact science that it is, however, chances are some additional information will have to be sent in advance, even if it's only texture maps on a CD-ROM. Designers should be appraised of this in advance, so they can adjust their designs accordingly. If you know that players are going to figure out where all nearby objects are anyway, give that information to them officially and be sure it's never of much use. Have new mobiles that teleport in from nowhere, if you want to keep them on their toes.
Okay, so, your programmers assure you that nothing is in the client that should be in the server, except for a few elements that you can design around. That's the security issue sorted, then!
Aw, you know it isn't. Your design itself could have security problems.
For example, suppose (having decided that fixed prices are a bad idea) you make your economy determine the price of goods based on local supply and demand. Suppose also that it allows people to buy and sell in bulk at the price for a single item. Normally, this wouldn't be a problem: If there are 300 swords for sale, it doesn't matter whether you buy one at 20 UOC41 or 10 at 20 UOC each. The price will rise the fewer swords are left, which is what you want—having bought 10 swords, the new price might be 22 UOC. You'd make the buy-back price be much lower than the sale price, so that anyone trying to sell back 10 swords they just bought for 200 UOC would receive maybe only 110 UOC for them instead of 220 UOC.
So far, so good.
What happens, though, for high-cost, low-production items? Maybe the local diamond mine unearths only five diamonds a week. When all five are available, the price at the diamond mart is 1,000 UOC; if only one is available, the price soars to 4,000 UOC. If none are available, the supply is exhausted. In that situation, how much could a single diamond be sold back to the diamond mart for? Even with a 50% mark-down on the purchase price, it would still be at least 2,000 UOC. So, if someone were to buy all five diamonds at once for 1,000 UOC each, then sell them back all at once for 2,000 UOC each, they'd make 5,000 UOC each transaction. Your design has given them an engine for generating however much money they like! Augh!
When something is allowed by the virtual world but the designers wished it wasn't, it's known as an exploit. Exploits—design bugs—can ruin a virtual world42 overnight.
Exploits aside, there are plenty of other ways that players can subvert a designer's well- meaning intentions. Identity theft—pretending to be someone else in real life—is fairly easy but is hardly the responsibility of the designer43. Character theft—pretending to be someone else's character in a virtual world—should not be easy, because it is the responsibility of the designer. In particular, if two players can use the same name for their characters then it's partly the designer's fault if one of them subsequently successfully pretends to be the other. This "name problem" is discussed in some depth in Chapter 3.
So, your design will have bugs. Players will find these and wring every advantage from them that they can. Even if they report them immediately, they'll feel you owe them for their honesty. So what do you do?
You can't prevent exploits, but you can take steps to minimize their number and impact. Detection and recovery are of critical importance. If possible, log everything. In MUD2, players would regularly complain that line noise44 had severed their connection to the game and led to their character's demise. Instituting a "log everything" policy (player input/output transcripts and all server decisions) solved the problem at a stroke—90% of the time, players were exposed as having been active right up until the moment the dragon incinerated them or the wolf bit off their head or whatever, and therefore their impassioned pleas for resurrection amounted to cheating.
This degree of logging is not always possible for virtual worlds that use a lot of bandwidth, for example graphical game worlds. There are two approaches to it, but neither is particularly satisfactory: store events (from which exact circumstances can be reconstituted) or store client communications (which gives the player's actual viewpoint). Both of these create huge quantities of hard-to-search data. Customer service administrators may be able to snoop45 on players as situations unfold, which can partially alleviate the problem, but if they arrive at a scene too late then the only alternative to logs is the presence of impartial witnesses (like there'll be many of those).
Detecting possible bugs and exploits is important, but it isn't itself enough. When it transpires that something really has gone unfortunately wrong, the ability to correct the consequences of it should absolutely always be available (even if it means a wholesale reinstallation of the character database from a timed back-up). An incomplete ability to discover when things are going awry is inconvenient; an incomplete ability to recover damaged data is incompetent46.
Part of the satisfaction of virtual world design lies in seeing your creation evolving, with things occurring within it that you hadn't anticipated but which make perfect sense. One of the less fortunate consequences, though, is that some of what happens you'll almost certainly wish hadn't, and will have to fix. The more contingency plans you have in place, the better, but you'll never be able to cover everything.
Still, if you want the unpredictable, you can't complain when you get it.