This page presents a collection of multiplayer game serverarchitecture patterns that describe solutions to common problems in multiplayer game server development. Over time, I hope to build a library of patterns based on real-world experience, research and experimentation by game server developers.
Problem How do we distribute the load of running core game play functionality across multiple processes to support thousands of concurrent players? Context We are developing the server for a massively multiplayer online game with a client-server architecture. The game design seeks to create an immersive play experience by enabling thousands of players to interact with each other in a … Continue reading Pattern: Map-Centric Game Server→
Problem How do we distribute core game play load across multiple processes in a way that supports flexible scaling and efficient allocation of computing resources? Context We are developing the server for a massively multiplayer online game with a distributed architecture. The game design seeks to create an immersive play experience by enabling thousands of players to interact … Continue reading Pattern: Responsibility-Oriented Game Server→
Problem How do we create a game server that permits unlimited freedom of movement and game play on an expansive map with no apparent boundaries? Context We are developing the server for a massively multiplayer online game with a client-server architecture. The game design seeks to create an immersive play experience by enabling thousands of players to interact with … Continue reading Pattern: Seamless World Game Server→
The feature impact assessment for the MMORPG game design requires a number of assumptions that aren't documented in the main posts or in the "reference problem statement" for the game. I've provided them here to serve as a rationale for impact rankings that I chose.
Ubiquity and importance of items will make for large numbers in inventories and various game systems. Expect high frequency of both reads and writes. Caching definitely going to be needed. Again, Redis perhaps.
Huge variety of items with modifiers/enhancements/persistent effects means key/value store over table storage. Also, BLOBs not worth the trouble; too likely for object attributes to change.
Economy, player trades, missions and other use cases where items are transferred between owners will require transaction support.
Multiple shared spaces roughly 100 sq km. Intended to support average 3000 players / map (30 / sq km). Populated areas will max 200 players / sq km in non-combat activity. Battleground and raid areas max 100 players / sq km with combat activity during events.
Using non-contiguous shared maps, with instanced spaces for certain missions and group activity. Will need lots of testing and iteration of visibility vs. performance using various terrain geometry scenarios. Also, need to do serious investigation on NPC populations vs. players. Probably need some "clever tricks" to make things feel right.
Items are second only to character, but without items you really don't have much character. Inventory will be massive; nearly all game systems will interact. Lots and lots of state updates on the server; hopefully somewhat minimizing transfers to client.
What can I say? Items touch everything: combat, crafting, economy, visuals, missions, and world interaction.
(repeated from character visuals) Need a good - no, great - visibility and distance culling system. Also may need server-side LODs (not graphical, but governing number of item slots that may be visible to others). Need to work with client devs on pre-caching of items in the client. Also need general item/object catalogue for classifying what objects can be seen at what locations and/or with which NPCs to minimize load, because we can't really do that with Player characters.
It will be a challenge to get the variety we'd like without overwhelming bandwidth with state updates.
Need a good - no, great - visibility and distance culling system. Also may need server-side LODs (not graphical, but governing number of item slots that may be visible to others). Need to work with client devs on pre-caching of items in the client. Also need general item/object catalogue for classifying what objects can be seen at what locations and/or with which NPCs to minimize load, because we can't really do that with Player characters.
We have to insist that all behavior/game play item stuff is fully decoupled from the visual stuff so we can aggressively fall back if needed.
We want thousands of concurrent users. This drives all aspects of scale in all systems. Need careful attention to data serialization and network optimization at the low level.
State updates will be constant, both between server and clients and also between server-side processes, and persistent storage.
At higher levels, we need to distribute load laterally across an undetermined number of services and hosts. This will also require the ability to scale laterally as well. No single points of failure or bottlenecks, please!
This is what puts the "M" in MMO.
We need to scale, but we can make design concessions to get it. Distribute load based on functional responsibilities, orthogonal to game play and world geometry. Consider actor model for ease of single-threaded-ish implementation.
Mission data is almost surely graph-based. Note fairly tight integration with items and other systems that use them; need to make sure we cleanly decouple mission and item state. Probably key/value store, *maybe* BLOB if absolutely necessary, but try (hard) to avoid. Versioning essential from the start; don't cobble it on later.
We can keep this fairly tightly scoped if we do it right. Most integration between characters and other game systems is above the persistence layer. Need to scale frequent writes, infrequent reads (assuming cache).
Probably some kind of key/value store for attributes. BLOBs seem attractive at first, but IMO they'll bite you in the ass later. Basic CRUD operations with a write-through cache on top - Redis, anyone?
Chat tech is generally well understood, with support from existing solutions if desired.
Integration with game features will be limited to global, regional, alliances, party, and private channels.
Just don't try to implement a custom XMPP server! Not worth it.
I'm arbitrarily deciding that alliances will be pretty tightly scoped. It will be focused almost exclusively on supporting social interaction and offline coordination of group play. Not tightly integrated with other game systems.
I have a Seiko analog watch that needs repair. When I pull out the crown to set the time, the movement stops. This is normal. But when I push the crown back in, the movement doesn’t resume right away. It sometimes takes several minutes for it to start ticking again.
I love this watch. My wife gave it to me for our anniversary years ago, and I’ve worn it daily ever since. I’m definitely going to have it repaired. Administrivia isn’t my strong suit, though, so I haven’t gotten around to it yet.
Meanwhile, I keep wearing it. I’ve learned that if I just stop messing with it, I can still rely on it to keep time. Now, I just remember the offset between the actual time and my watch’s time and do the head math. This works because the watch is consistent, and I’ve become consistent in using it this way. Continue reading Consistency FTW!→
Online game development is a school of hard knocks. Over the years I’ve made many mistakes, shared some successes, and learned a hell of a lot. I love collaborating and sharing knowledge with my teammates. Yet, the transient nature of game development often dilutes our shared wisdom when projects end and teams split up. I’ve often wished for a venue where I could share in a larger pool of wisdom from like-minded game server developers. This venue would give us with a way to not only capture our shared experience, but refine it, build upon it, and most important, make use of it in future work. I’m not sure I can describe clearly what I have in mind yet, but I hope to explore it through Engines of Delight.
My mission for Engines of Delight is to enrich the online game development community by advancing our shared understanding of software architecture as it applies to multiplayer game server development. I hope to do this through a blend of shared real-world experience, professional discussion and feedback, and curated references to external works.
This is a tall order: the term “architecture” is broad, complex, and ill-defined. It’s certainly bigger than me, and I can’t do it alone. With Engines of DelightI also hope to offer an environment where other server developers can teach as well as learn, share their failures and successes, and apply their unique insights to common problems and solutions.
Some features that I hope to explore with Engines of Delight include:
Regular blog posts on a variety of topics relevant to multiplayer game server architecture. Standard blog fare, so to speak.
A catalog of known patterns and architectural “styles” found in the industry.
A catalog of common multiplayer game types.
A cross-reference between known patterns and game types, identifying combinations that work well and those not so much.
Postmortem articles that document architectural decisions from real projects, their outcome, and lessons learned.
Curated references to recommended third-party sources, including blogs, podcasts, books, and other works.
A vehicle to encourage contributions from community members in all the above areas.