One of our best friends, in game development, is the build server. The build server makes clean builds when we forget. It keeps our artists supplied with up-to-date binaries. The simple things we just assume will always work, it will test without complaining. I thought I’d take some time and share some of the lessons we have learned in the past, and how we build our projects.
Why Use a Build Server
Why use a build server? You’ve got version control, why not just check in binary files? Most version control software is based around text files, and does not handle binary data very well. Since version control software will try and ensure that any version of a given file can be recreated, it will store a large amount of differences in binary files. This will quickly eat-up space in your version control. If you use a distributed version control solution, this is especially bad.
Our projects include a BAT file that allows non-coders to pull down the latest binaries from the build server, and unpackages them into the recipients working copy; .gitignore contains entries for these files. This prevents repository bloat, and ensures that an up-to-date binary is always available. It provides a consistent platform on which to build; using the same SDK versions, compilers, and dynamic C libraries each time.
A previous game project,
imported from SVN, created a 4GB Git repository, for what was a ~300MB game directory. This previous project checked in ZIP files of archived assets, and distributed builds to artists by checking builds in to the SVN repository. This was close to a worst-cased scenario, but if you don’t believe the bloat, just think about how many differences a 100MB ZIP file of a deploy-able package would incur each update. Using a distributed version control solution means that each developer would be downloading, and storing a 4GB package.
You should never check your game builds into your version control repository, and you should prefer using text-based file formats for assets whenever possible. Instead, with proper use of version control, and a build server, you can generate binaries and packages using any current, or past version of your project.
A Little Help From Our Friends
We are game developers, not service hosts. The best service is the kind someone else can host; companies that do hosting as their business. We chose to manage our own build hardware, but use hosting providers for all our other needs. There are some wonderful projects out there, that are critical to our work-flow.
Our precious source code is hosted by GitHub (we will talk about Git, and our work-flow, in future development blogs). GameClay has a paid account that is used only to create, and manage repositories; only the build servers have an RSA keys linked to this account. I manage the GameClay GitHub account in my “off browser”; either Safari or IE since I am a Chrome user and this helps to avoid any badness with cookies and sessions.
The Build Server
Our server is a 16-core Nehalem Xserve with 12GB of RAM running off an 128 SSD, using a DroboPro for a Time Machine target. Our office does not have a dedicated server room, instead we use a Kell Systems PSE12. It is well soundproofed, thermal controlled, and it also makes a great printer table! We use Parallels Server to host virtual build machines for Xbox360, Windows, Mac and Linux builds. CI Joe gives us an easy-to-use, easy-to-extend, continuous integration server, runnable on all platforms.
This machine is almost entirely firewalled to the outside world, and only provides shell access through RSA-key authenticated SSH. The limited, externally accessible web services are located on virtual machines, and are backed up by Time Machine. Mac OS X Server is very easy to maintain, and because it needs so little external visibility, I burn almost no time administrating our limited, internal services.
The GameClay fork of CI Joe adds the ability to send push notifications to an iPhone using Prowl, and some other features. CI Joe listens for GitHub to send information from a Post-Receive Hook, and will automatically perform a build every time there is a push to the repository. When a build is successful, it packages the required binary files, and makes them available for our non-coders.
By Our Powers Combined…
So what do these nodes, wired together with a little Ruby contribute to our development process? Well first of all: it works, I set all up in less than a man-week (accumulated over time), and most of the elements are hosted by people with better bandwidth, redundant everything, and great security.
A consistent build platform allows us to dodge issues with D3DX DLL’s and Visual Studio run-time libraries that plague teams of coders who all work in their own, personal development environments.
During development of a previous PC title, we delivered incremental updates via a patcher solution. This worked fine until one release magically broke for around 20% of our user-base. The culprit? A different developer had done the build, and was linking with a different version of DirectX, requiring a different D3DX DLL. Our solution at the time? Make the same developer do the builds. This was inconvenient many times, like if the testers needed a build, and he was not at his computer, or if he was on vacation and we needed to do an emergency fix.
With a build server, the official binary comes from one place, one place only, and it is always available. It allows developers to do what developers do: try out new tools, and new versions without compromising the ability to deliver an up-to-date build at any time. By using virtual machines, we have given ourselves the ability to create several versions of environments and make the most of the hardware we do host in-house.
Knowing is Half the Battle
I chose to use services and projects which are specialized in purpose, like CI Joe, GitHub, and Lighthouse (more on our bug tracking in a future post). In my experience, you always want to tweek something, or add one tiny feature to your build system. All of these projects use Ruby for their functionality, and this is intentional.
A common interface between all the required pieces of your build system is very advantageous. Elements of the system can easily be modified and integrated with each-other and Ruby is host to a powerful set of tools to build new elements. Among them:
Sinatra provides an amazing platform to write custom web services, and
grit brings the battle-tested Git interface that powers GitHub.
The project isn’t complete yet, but we do have a build system that meets our needs with a minimum investment of set-up time, and maintenance. It’s ability to stand up to a full game development cycle is yet to be proven. We refined our process many times during our time at GarageGames, and each iteration improved our workflow. This has been the easiest, and most flexible build system set-up by far.
We will be making more blog posts about our development process during this still-unannounced game project. We hope this will help small game teams gain the advantage of some of the great, open (and closed) source friendly, services and tools that make our lives easier.
Awesome blog Pat. I love build servers and fresh binaries since we know how well artists trying to compile code goes.
One thing that has worked out pretty well for the last few projects on the art side strangely enough is Dropbox. It keeps a 30 day history of files and allows us to share and keep art source files in one spot without bloating SVN for everyone else. Love me some Dropbox.
Awesome post indeed!
Im curious about what each build process involves, specially under windows, a bat file? How deep into this processes you use ruby? Would be intersting to hear some experiences on the low level, as how you made the handled the process of building, packaging and uploading to a repo the latest build, in the different plataforms.
For the wip art assets, we just use a different folder in the svn, so the standard update doesnt get cloged with them. Why zip’em into big files?
The CI Joe builder command is a ruby script which runs the generateProjects.bat and (slightly modified) compile.bat files from Torque3D. If the build worked, CI Joe executes the .git/build-worked hook, and that hook is coded to add the .EXE and .DLL into a ZIP file and make it available via a Sinatra route.
There is a bat file in the project that looks like this:
util\curl\curl.exe https://server:port/route > latest.ziputil\7za.exe e -ogame -y latest.zip
del latest.zip
This uses a CURL command (which can be customized to include authorization etc) to download the zip file which was built by the CI Joe server, and extract the needed EXE/DLL files to the non-coder’s machine. I should probably do a blog post on this part of the process as well.
Do NOT zip assets into big files. That was part of the “bad” example hehe. It does terrible things to the size of the repository, and is really not a correct use of version control.
Excellent, much appreciated!
Indeed, there you have material for another post
Another fine post indeed, Pat.