Vodafone Innovation Park Labs

Pattern: Distributed Network Connections

Problem

How do we support the greatest number of concurrent client connections to our multiplayer game server?

Context

We are developing the server for a massively multiplayer online game with a client-server architecture. The game design seeks to create an immersive play experience by enabling thousands of players to connect simultaneously to the same virtual world.

Forces

  1. The game design depends on having many players online at the same time.
  2. Client-server communication requires TCP for reliable, persistent connections.
  3. TCP connection management overhead must not impact the player’s game play experience.
  4. The game server uses a distributed deployment architecture.
  5. The project’s business plan seeks to optimize hardware and operating expenses.
  6. The project’s operating plan seeks to minimize downtime by building redundancy and failover into the architecture.

Solution

Build a type of server responsible for accepting and managing client TCP connections and multiplexing client network traffic to game servers. This type of server is commonly called a connection server, user servergateway server, front-end server or player server.

The server design should:

  1. Be dedicated to managing client connection state and delivering client network messages to the right game servers. This may include decoding compressed and/or encrypted data streams.
  2. Provide a single server connection for all network traffic from each connected client.
  3. Use some sort of stable hash of unique player metadata (e.g. user ID) to consistently route messages from a given client to the same back-end server.
  4. Avoid introducing other responsibilities such as authentication, server directory management, server-side load balancing, and the like.
  5. Limit state management to client connections, player sessions, and related metadata. Avoid interacting with any type of game state.
  6. Support multiple instances of this server type running on one or more hosts for optimal resource allocation.
  7. Maintain service uptime via automatic crash recovery and failover handling of incoming connection requests.
  8. Optionally support dynamic scaling by adding new user server instances on demand.
Distributed Network Connections Pattern
Distributed Network Connections Pattern

Resulting Context

We have an MMO game server that can handle thousands of client connections without adding network overhead to the game servers. We can increase the client connections we handle by adding new connection servers. Also, the number of connection servers is independent of the number of game servers.

Rationale

Creating TCP connections incurs both CPU and memory overhead. This overhead increases linearly with each connected client, or O(N) where N is the number of clients. Decoding compressed data streams, and especially encrypted data streams, adds more overhead. That cost increases more like O(N*M) where N is the number of clients and M is the number of packets received by each client.

Most MMO games involve walking, running, jumping, combat, and other activities that require repeated player input.  This generates a constant stream of network messages from each client, so that the M part of our O(N*M) becomes pretty significant. Trying to handle that load in the same process that handles the game logic (and physics, AI, crafting, etc.) will degrade the player’s experience. If we decouple that work from the rest of the game, however, we have both more capacity and greater flexibility for handling that load.

Related Patterns

Client Side Load Balancing

Known Uses

  • Most commercial large-scale massively multiplayer online games.

 

Pattern Template: Coplien, James. “PLoP95_telecom – Info.” PLoP95_telecom – Info. James Coplien. Web. 30 Sept. 2015. .

Leave a Reply

Your email address will not be published. Required fields are marked *

WordPress Anti-Spam by WP-SpamShield