Software architecture is about the big picture of development. It’s also about identifying and managing engineering risk. The development team is at the nexus of these two concepts. The team is your greatest asset, and the greatest contributor to the success of any software architecture. The big picture revolves around the development team and its ability to create the software for your game.
The team’s development environment is a cornerstone of its ability to deliver. A team with inadequate tools or support to do the work required is at least inefficient, if not ineffective. A team with an inferior development environment is a likely source of engineering risk.
In this post, I’ll identify some ways to create a development environment with which your game server programming team can excel. These come from personal experience doing MMO server development. I won’t claim that the list is comprehensive or right for all circumstances. I will claim that these things have worked for me.
I’ll start by saying, “be agile.” The rest of this article assumes at least a modest understanding and acceptance of agile development practices. If you’re not quite there yet, the web has plenty of great resources on agile development. Go find one and follow it, then come back here.
Here are some starting points:
- The Agile Manifesto
- Agile Software Development: A gentle introduction
- Agile Guide
- Scrum Methodology
- What Is Agile? (10 Key Principles of Agile)
- The Most Important Agile Practice Of All
At least, do these things:
- Short iterations. Two-week sprints promote accurate estimates, minimize uncertainty, and allow low-cost course corrections based on frequent feedback.
- Automated testing. Early defect detection and correction is the best kind. Tests that run every time the code changes catch errors as early as possible.
- Continuous Integration. Always commit changes to the mainline to minimize code divergence. Continuously build and test. Drop everything to fix breakages. Profit.
- Refactoring. Make frequent, small, evolutionary improvements instead of sweeping changes. Code breaks less often.
- Daily standups. Talk with your team every day. Stay on point: what you did, what you’re going to do next, and call out any blockers. Stand up to keep the meeting short.
Seriously, adopting the basic practices of agile development is probably one of the most effective ways to improve your development environment.
Have Enough Hardware
Game server development is resource intensive. Developers need to be able to edit, build, test, execute and debug the server code. They often need to build and run client code too. On top of that, they might need to run a database, infrastructure services other development tools, and possibly a virtual machine or two.
There’s no reason a game server developer should have less than 16GB of RAM a 500GB disk (preferably an SSD), and at least a 4-core CPU (preferably 8 including hyperthreads). And, don’t forget a current graphics card with enough VRAM/cores and support for the shader/rendering tech your client uses. It may not need to be absolute top-end, but it needs to run the game client at 30 FPS or better.
Also, dual monitors are a necessity these days. Most server devs need to run:
- An IDE (with dockable/floating panes); sometimes more than one instance at a tim.
- A version control system (VCS) tool (with tree view of the repo).
- Various other dev tools and supplemental editors, such as visual diff tools, DB client/editor, terminal windows, and the like.
- Custom game data editors for testing server-side game systems.
- Email, web browser, chat client(s) and the usual stuff everyone else uses.
In short, server developers need all the screen real estate they can get.
In some cases, a server developer may need two machines. This is especially true when developing systems that involve interactions between multiple players or server processes. In a distributed server architecture, these interactions often span multiple hosts.
If the developer has only one machine on which to test and debug, s/he might miss important edge cases that don’t appear until later in testing. And even then, it’s hard to reproduce and fix those cases without a second machine handy.
Systems development that may require a second machine include PvP combat, trade, grouping, chat and other social systems. They also include world travel systems like teleportation, map instance selection, and the like. In some cases server-side visibility culling, relevance, or client view filtering might also require more than one machine.
Don’t be stingy. Every dollar spent up front on good development hardware returns many more in developer productivity.
Get your Head in the Cloud
Consider allocating small Amazon, Microsoft Azure, or other cloud VM instances to server developers for their individual use.
The AWS T2 “Burstable Performance Instances” or the Azure “General Purpose Compute: Basic Tier” provide low-cost access to VMs. Having 1, 2, 4 or 8 cores, with appropriate memory and disk, these have enough power to handle local builds or run a small server cluster.
With a little training on best practices, and perhaps some automation, server devs have a great resource on demand that can be easily shutdown when not needed. These might easily satisfy the need filled by the second developer machine I mentioned above, at a lower cost.
Also use the cloud to host shared development clusters. These should be small-scale versions of your production game server cluster. The QA team should use these for daily acceptance and regression testing. They’re also perfect for regular team play sessions.
Standardize Your ToolsI’ve worked on teams where developers could choose their OS, IDE, and other tools based on personal preference alone. In some cases, they even chose third-party libraries or frameworks such as, or even languages based on what they liked to work with, without getting consensus from teammates.
This approach may seem egalitarian, but it levies a heavy penalty on the team. This penalty usually looks something like this:
If the team doesn’t choose a standard, anything goes, so every option must be supported. No real owner exists, so the burden appears to be shared, but is really only diffused through the team. When compatibility problems arise, it’s usually the poor fool blocked by it who has to fix it.
If the team designates an “official” tool of a given type but allows developers to use their preference as an “unsupported” option, then only the user of the unsupported tool is affected. Except, that’s not really true. Even though s/he may have to unblock her/himself when problems occur, others on the team are probably indirectly blocked, waiting for a check-in, a build breakage, or what have you.
In either case, onboarding new developers becomes an unwanted adventure. First, the newbie is confused about what tool s/he should be using. Next, when s/he has questions about using the tool, s/he must find a teammate who uses the same tool. It’s an inefficient, demoralizing mess for a newbie to have to deal with.
It may seem obvious, but I’ll say it anyway: don’t do this, please!
Version All the Things
Being agile, your team’s source code is already managed in a version control repository, right? That’s great, but it’s not enough. For the development team to excel, you should try to put (nearly) everything in version control.
As a rule of thumb, you should version everything needed to edit, build, test, debug, deploy, and run your entire game project without further manual intervention. 1)“What to Put under Version Control?” – Stack Overflow. 10 Dec. 2009. Web. 10 Nov. 2015. <http://stackoverflow.com/questions/1880817/what-to-put-under-version-control>.
Here’s a list of things that you should keep in your version control system. 2)Binstock, Andrew. “Putting Absolutely Everything in Version Control.” Dr. Dobb’s Journal. UBM Tech, 3 Sept. 2013. Web. 10 Nov. 2015. <http://www.drdobbs.com/tools/putting-absolutely-everything-in-version/240160762>. Some are obvious, others not so much.
- Source code.
- All game content files, whether used by the client or server. This includes:
- Graphical and audio source files (i.e. the files content creators edit in their tools).
- Game rules, object specifications and properties and other data that defines the play experience.
- Exported game-usable versions of all content.
- Tests, test scripts and test data.
- Database scripts, including DDL, migration scripts, or schema diffs.
- Generated code with external dependencies that can’t be pulled into the project, such as API clients and message structures for remote REST APIs.
- Third party libraries and other external dependencies of your project. Store these in the form in which your project uses them; i.e. not archived or requiring manual installation. These include:
- Statically or dynamically linked native libraries in source or binary form, as referenced by the build project.
- Package manager (e.g. NuGet, npm, etc.) packages and configuration data.
- Compiled Jar files, .Net DLLs, and the like, from remote repositories.
- Project documentation that isn’t generated from code, such as requirements, specifications, and technical designs.
- Build scripts for all build targets and runtime configurations.
- Deployment scripts for all environments.
- Configuration files for all environments.
- Compilers, interpreters, and runtime engines (e.g. JVM), along with any configuration/environment settings.
- Current versions of the team’s officially supported editors, IDEs and other development tools.
- Official build artifacts, approved for deployment to a production environment or equivalent (e.g. public test, internal test, beta, etc.).
Here are some things not to include:
- Files that contain absolute paths or hard-coded host-specific data.
- Output files from local builds (i.e. those not meant to be published).
- Individual developer preferences and settings. However, creating a separate user area in your repository where developers can keep these things is great.
By putting everything in version control your team reaps these benefits:
- You can recreate the entire project from a single source, without needing downloads, installs, or configuration. Sometimes catastrophic errors make it into production, in spite of our best efforts. When that happens, you need a way to roll back to a known good state quickly and with minimal risk of introducing more problems.
- Explicit and implicit dependencies between source code, data, and tools are always in sync at any given revision. This can be critical to solving hard-to-find version-specific bugs.
- It ensures that everyone who modifies, builds, and runs the code has the same environment as everyone else. This significantly reduces your chances of encountering the “works on my machine” syndrome.
- New team members can hit the ground running by starting with the current working versions of everything s/he needs to be productive. Onboarding server programmers on large game projects can take hours and sometimes even days. Bootstrapping the process in this way can eliminate uncertainty and reduce that time tremendously.
- It positions the project to adopt continuous delivery, an agile practice that promotes frequent small releases to respond quickly to changes and continuously improve your software. 3)Binstock, Andrew. “Putting Absolutely Everything in Version Control.” Dr. Dobb’s Journal. UBM Tech, 3 Sept. 2013. Web. 10 Nov. 2015. <http://www.drdobbs.com/tools/putting-absolutely-everything-in-version/240160762>.
It’s ideal to store all versioned assets in a single repository. That way you can easily identify the state of the entire project as of any given time. Yet, in practice this is hard, if not impossible.
Some version control systems, such as Git, don’t handle large files or repositories well, although various workaround do exist.4)Daityari, Shaumi. “Managing Huge Repositories with Git.” SitePoint. SitePoint Pty. Ltd., 1 Sept. 2015. Web. 14 Nov. 2015. <http://www.sitepoint.com/managing-huge-repositories-with-git/>. Some build tools such as Maven use separate repositories for managing build artifacts. These provide important benefits, but may not integrate well with traditional version control systems. Finally, some build management systems such as Jenkins or Bamboo have integrated systems or plugins for versioning their scripts and the build artifacts they produce.
When faced with these issues, your team will probably have to compromise on some hybrid solution. This may involve storing some versioned files in more than one repository. Or, it may require building custom tools and processes to link file versions across two or more repository types to define a set of revisions for the entire project.
Always be Running
Exercise your game server code continuously, every day. The more often it runs, the sooner you can expose and fix problems. Keeping the code in a runnable state makes frequent team play sessions possible. It also makes it easy to demo code for execs and other stakeholders on short notice.
Make sure game servers can be easily run on developer machines. This means running a self-contained local cluster with a subset of real game data that handles a couple of game clients. This may require a special runtime configuration to run in a low-resource mode. Avoid using build-time options or special development-only server code to do this unless you have no other choice.
Make it easy to start and stop the local cluster using a custom development cluster management tool. Make it a GUI or web client, as user-friendly as possible. Put any special case code for development use cases into this tool, and not into the server code. Build into the tool support for running at least two of each type of server process. Also include support for controlling a single process as well as the entire cluster.
Such a tool may seem like a luxury for developers, but it’s not. It will pay off easily by reducing iteration and testing time for server code. It will also make running a game server cluster accessible to non-technical team members, including designers and artists. This greatly reduces iteration time on content development, which increases the entire team’s velocity.
Run all automated tests as part of every continuous integration (CI) build. If you have long-running integration tests as well as unit tests, consider having two or more CI builds. One build job can run at every check-in and exercise only fast-running unit tests. Another can run on a slower cycle and exercise all tests. Pay attention to code coverage metrics and strive to meet at least 80-90%.
Create a build system that allows any dev team member to start a build and deploy a server cluster to a shared development environment at any time. Modern build management systems such as Jenkins or Bamboo, together with a little careful planning, make this possible and safe to do. Self-service builds will free your engineers to focus on building functionality and improving tools instead of supporting routine operations.
One Build to Rule them All
Use the same set of build project and configuration files for all builds. This includes builds that run on developer machines, CI builds, and those that generate packages for production.Push all build variables into the projects themselves so that they can be set by any script or IDE that invokes the build. Ensure that any build scripts that wrap the projects don’t add new rules that affect the final build result.
This approach ensures that local developer builds are similar to CI, QA and production builds. This reduces the chance that a build error will fail to appear on a developer’s machine only to appear in the CI build or QA build.
Don’t Share Databases
It’s sometimes tempting to create a centralized game database shared by multiple development servers. Several forces may drive you towards this:
- The high cost of licensing expensive commercial databases such as SQL Server or Oracle.
- A wish to spare team members from dealing with complex database changes.
- A perceived benefit of always having game data available for ad-hoc testing.
Resist this temptation. It creates an artificial external dependency for development game servers. This creates of risk of problems that occur only in development, and is sure to slow your team down in the long run. Examples include:
- Any outage or maintenance activity on the shared database will create downtime for developers.
- Schema changes to the shared DB could break code, blocking some or all developers.
- Subtle and hard to debug errors may occur when someone changes or deletes shared data unexpectedly.
- Data collisions may occur with keys generated by game code, such as object IDs based on specific value ranges.
Several alternatives to a shared development DB exist:
- Use database migrations for all schema changes. Two free DB migration tools are Flyway (Java) and Fluent Migrator (.Net). Commercial solutions also exist. Automate the migration process using scripts to minimize the burden of making DB changes.
- Use free open source SQL database products such as MySQL, PostgreSQL, and their peers to minimize licensing costs. Often these products are as performant and reliable as their commercial counterparts.
- Use a free DB product for development databases, running commercial products for QA and production environments. This will probably require maintaining separate schemas, though, so choose this option only if your team can handle that burden. If you do go this way, choose a lightweight option such as SQLite for the dev database to keep things very simple.
- Use a NoSQL database instead of a relational one. This eliminates the issue of managing schema changes entirely. It also eliminates the licensing cost, as many excellent free NoSQL databases exist. Some examples are Cassandra, RavenDB, and MongoDB, but there are many more to choose from. Finally, the tree-like structure of most MMO game data is often well suited to non-relational data stores anyway.
Strive to make your development servers a faithful model of your production servers. The closer your development environment is to production, the easier it is to simulate real-world conditions during development. Of course, you can’t run development servers at the scale expected in production. Still, there are things you can do to reproduce many aspects of your production environment. These include:
- Avoid special-case development logic in your server code when possible.
- When you must implement special case development logic, enable it via runtime options instead of compile-time directives. This ensures that the binaries running locally are the same as those that would run in production.
- Configure your development cluster to run at least two processes for each server type that supports it. This minimizes the risk of accidentally relying on a specific code path.
- Avoid relying on artificial test values (e.g. 1, 42, 9999) that tend to always exercise the same code path. When possible, use random values, or randomly select from a set of pre-created values.
- Create a simulated production host for individual developer use. This should be a virtual machine that runs either on a developer’s local machine or in the cloud. Provision the VM identically to production hosts if possible, or get as close as you can. Run local builds of the development cluster on this VM when possible. Consider using Vagrant to manage the VM, allowing for quick and easy iteration on a fresh host when needed.
- Consider using Docker to build deployable containers. A Docker container created by a developer can run in any environment, including production. With the right tool and process support, using containers can significantly reduce overall time to build, test, and release to production.
Embrace DevOpsYou might think DevOps has nothing to do with your development environment, but I think it does. With DevOps, your dev team partners closely with Ops and becomes a co-owner of the deployment process. You can leverage this to bring DevOps deployment automation practices into your server development environment. This will enhance your team’s ability to model production, deploy more often, and improve release quality.
You can embrace DevOps in your development environment by:
- Keep deployment scripts and configurations in version control. They are critical to your project’s success and should be treated as peers to other project assets.
- Synchronize deployment changes with game changes. Always be able to identify the exact revision used to deploy a specific revision of the game. The easiest way to do this is by keeping both deployment and game assets in the same repository. If you can’t do this, then build tools and processes to automatically link them. See Version all the Things, above.
- Allow server developers and Ops team members to commit changes to deployment and game assets as needed. Changes on one side will sometimes mean changing the other. Reinforce collaboration by removing the silos. Trust your teams.
- Integrate deployment into your development process. “Dog food” your deployment process by using production deployment tools in your development environment. Support deployments to local development VMs to allow development testing and iteration. Add deployment to your CI builds, using production tools to push the latest code to a shared development cluster. Do the same for your QA environments to ensure acceptance testing criteria match production requirements.
Your developers want to be productive. They need to be. Give them the tools and support to do this, then get out of their way. Encourage them to develop a core value of continuous improvement. Empower them to make changes to the way they work as well as the product they’re working on. Make improvement activities first-class tasks in your backlog to give them the appropriate priority and resources.
There are no guarantees in game development. It’s hard work, expensive, and full of risks. Investing in your development environment will generate returns directly to your game. It won’t guarantee success, but it will make your programmers more productive. They’ll iterate more quickly on new development and bug fixes. This will improve your game’s stability and quality. It will also reduce your team’s time to release changes to production.
In the end, this means a better chance of hitting your target ship date and delighting your players.
I’ve covered a lot of points in this post, but I know there are many more. How has your development environment contributed to your game’s success, or impeded it? I’d love to hear your stories. Please share your thoughts on this, or leave feedback about this post by submitting a comment.
Thanks for playing!