Words are hard. Game servers are nontrivial.

Engines of Delight is a blog about multiplayer game server architecture.

What are we talking about here?

Words are hard. Too often in software development, we use overloaded terms, and their meanings become diluted or confused. To avoid this I’ll define what I mean by multiplayer game server architecture. 

For us, a multiplayer game is any computer game where multiple players interact with the game and each other using a client connected over an IP-based network, usually the Internet. This generally includes most types of FPS, MMO, MMOFPS, MMORTS, MOBA, and the like. It excludes, however, the venerable MUD-style games, play-by-email, and peer-to-peer networked games played over LANs. I’m not making any value judgments about these other game types, just trying to limit our scope a bit.

I’ll use the term server to describe software that runs on a central computer and provides some sort of service to game clients or other servers that work together to form the complete game. Sometimes I’ll also use the term server to refer to the set of all non-client processes that make up the game from the player’s perspective. Our focus at Engines of Delight will be mostly on game servers. I’ll discuss clients when it makes sense, usually in the context of client-server interactions. This is mainly because that’s where my experience lies. Also, there’s already a lot of client-oriented information available that I could do little to improve on.

Finally, we have the term architecture, or, more precisely, software architecture. I think this is one of the more ambiguous terms in software development, as I’ve found a number of definitions for it. The one I find most useful is, “anything and everything related to the significant elements of a software system; from the structure and foundations of the code through to the successful deployment of that code into a live environment.” [1]“What is software architecture?”, Software Architecture for Developers, by Simon Brown, 2015-04-01, Chapter I, Section 3. http://leanpub.com/software-architecture-for-developers

I hope these definitions both clarify these key concepts and bound the scope of the content provided at Engines of Delight. Going forward, the handy Engines of Delight glossary will define most important or commonly used terms found on this site.

Why Software Architecture Matters

Why should we think about software architecture?

Software architecture is important because it’s the foundation on which every meaningful part of the software rests. It’s the “big picture” view of a system that defines how the pieces fit together. Some obvious elements of an architecture are the high-level design, public interfaces, programming language(s), libraries, networking protocols, persistence technologies and the like. Less obvious are the tools and processes used to develop, test, and deploy the software, and the techniques, practices and standards the team employs.

In the tired but true analogy of building a house, the foundation is the most important element of construction. The entire house’s integrity depends on it. A poorly designed or flawed foundation will create problems later in construction, and certainly in the finished house. In some cases the problems might not show up for years. Yet, the eventual outcome is often disastrous. I once knew someone whose home had this very problem, and his only option was to demolish and rebuild the house.

So it is with software. If the architecture is bad, the software is likely to be bad. Inadequate testing prevents early problem discovery and correction. A monolithic application design limits the ability to scale. Undisciplined source control practices cause integration errors. Failures at the architecture level are always costly. In fact, another way to identify an architecture element is to ask, “what happens if I get this wrong?” Anything where the answer has a high cost is part of your architecture.

All software has an architecture, whether or not its developer intended. [2] “Is software architecture important?”, Software Architecture for Developers, by Simon Brown, 2015-04-01, Chapter I, Section 5. http://leanpub.com/software-architecture-for-developers.  This is true for both small and large systems. Think of a simple script that copies files from one host to another. The script’s architecture includes, among other things:

  • The scripting language used.
  • How the script handles command-line arguments.
  • Any external tools the script invokes.
  • The way in which the script is maintained.

The decisions made for each of these points could significantly affect the use and maintenance of the script over time:

  • If the script author uses bash because it’s his favorite language, but the dev team is a Python shop, it’s a potential problem. It will either limit who can make changes to the script, or increase the cost of maintaining it.
  • If the author designed the script to accept arguments in a specific order, it could make the script hard to extend. Simply adding more arguments to the end of the list will eventually break down. Fixing that will mean implementing named arguments, a major change. This will force users of the script to adapt, and will break any other scripts that might have called this one.
  • If the script calls ftp to do the file transfer, and its author later converts it to use scp for security reasons, this will introduce other dependencies such as ssh certificates or user/password management. This could also have a cascading effect requiring more changes to the command line arguments.
  • If the author chooses not to commit the script to the team’s source control repo, a myriad of problems may arise. The author may hard code host-specific values into the script. A file system failure or user error may destroy the file. Another developer may copy and change the script for use on another host. This will quickly become unmanageable.

This is a trivial example, and it may seem contrived. After all, it’s just a script, right? Yet, I’ve seen cases where mission-critical operations jeopardized by one or more choices similar to those above. That’s not to say that every choice above is always wrong. It comes back to that question, “What happens if I get it wrong?” How you answer that question determines the choice you should make.

Software architecture matters because we must be intentional about these choices. While it’s true that software architecture “emerges” over time, the architecture you get may not be the one you want.[3]Agile Architecture Myths #3 Good Architecture Emerges”, from The Responsible Designer blog, by Rebecca … Continue reading If you intentionally choose one or more of the above courses, then so be it. I’ll assume that you have weighed the options and will accept the outcome.

We need to think about software architecture because:

  • The quality of the architecture determines the quality of the software built on it.
  • Failing to make good architectural decisions is costly.
  • We must make intentional choices to create a good architecture.

What’s Different About Game Servers?

I don’t want to overstate this, but game server development is… nontrivial. Depending on the type of game, the server’s architecture is likely to include most of the following:

  • Distributed processing / inter-server messaging.
  • Relational data persistence.
  • NoSql / document data persistence.
  • Transaction processing.
  • In-memory caching.
  • Custom object serialization.
  • Custom networking protocols.
  • Web-based or RESTful APIs.
  • Artificial Intelligence and complex state machines.
  • Pathfinding.
  • 2D or 3D movement and physics.
  • Per-client object visibility computation.
  • Highly specialized use cases for game play scenarios.
  • Lots of non-deterministic or emergent behavior.
  • Support for dozens to thousands of connected clients.
  • Perceived latency from tens to hundreds of milliseconds.

I’m sure lots of other software development is nontrivial, and I don’t want to diminish anyone else’s life’s work. But, personally, I’ve not heard of many other systems that encompass such a range of complex problem domains.

Yet over the years, I’ve seen cases where game server architecture seems to be treated as an afterthought. Yes, even on large, complex MMO projects. Symptoms of this include game designs that are green-lit without a server-side technical assessment, overoptimistic guesstimates given to execs in a hasty hallway conversation, or kick ass client-only demos that makes it look like the game is mere weeks from “first playable” when the truth is more like months.

Maybe it’s because the game, as experienced by the player — and producers, and execs — appears to happen entirely on the client. Everything that the server does is either displayed through the client, or is entirely invisible. Of course, anyone involved in creating multiplayer games should know about the server’s important role in bringing the game fantasy to life. But for some, it requires a conscious effort to break their “suspension of disbelief” and acknowledge it.

With Engines of Delight, I hope to show how good software architecture is both relevant to multiplayer game projects and essential to their success.

What’s next?

In the next few posts, I hope to cover these topics at some level:

  • A survey of common multiplayer online game types.
  • A survey of common game server architecture patterns and/or anti-patterns.
  • An attempt to match game types with recommended architecture patterns.
  • A discussion on server authority.
  • A discussion on various approaches to concurrency.
  • I’ll also experiment with writing a few postmortems on specific architectural decisions I have experience with. I hope to make these a regular feature if possible.

Thanks for taking the time to check out Engines of Delight! If you have questions, topic suggestions, or constructive feedback, please leave a comment! I’d love to hear what you think. Or, feel free to contact me privately if you prefer. .




Leave a Reply

Your email address will not be published. Required fields are marked *

WordPress Anti-Spam by WP-SpamShield

This site uses Akismet to reduce spam. Learn how your comment data is processed.