How Riot Games Used Open Source to Rework Its Software Infrastructure
Life at a gaming company isn’t always fun-and-games. It’s also a demanding IT environment with a huge amount of data to manage. Using various Hadoop open source tools, the gaming company behind League of Legends supported hypergrowth and delivered more timely analytics.
Building a software infrastructure that can support hypergrowth isn't easy under the best of circumstances. But add to this the business need to move from a traditional SQL database to Hadoop, and to develop better real time monitoring tools; These are some of the challenges faced by Riot Games last year.
The company, which has offices in Santa Monica and St. Louis, is the maker of the very popular online game League of Legends. Launched four years ago, the game regularly has 32 million monthly active players who accumulate more than a billion hours of game play each month.
But the game's popularity was causing their software development team fits. "It was taking us too long to gain insights into our software performance, and our database servers were getting bogged down," said Barry Livingston, Director of Engineering at Riot Games. "You wouldn't think a five-year-old company would have legacy software issues, but we did because we have grown very rapidly," he explained in a presentation at the St. Louis StampedeCon conference in July.
Riot Games began its operations with a monolithic SQL platform for its data warehouse. It required a great deal of manual, custom-coded processes. Queries were written in MySQL and most of the reporting was done in Excel.
The company had several reasons to change from that approach.
Part of Riot Games' mandate is to provide the best game play possible to its end users. To that end, the company releases new gaming virtual worlds and in-game content every two weeks.
The company’s staff also wanted to be responsive to their gaming customers by quickly resolving problems or removing cheating players. The old system wasn't working; it wasn't designed to operate at the current level of players. As a result, several critical processes would bog down. For instance, the daily data extract update was approaching 24 hours to complete. Plus, debugging software errors meant digging deep into log dumps to figure out what went wrong.
Last year, Riot Games senior staff realized that they were at a breaking point and needed a new software data infrastructure. "The time to arrive at insights was taking too long, and our solution required too many dev team members to make changes to our data schemas or other updates," he said.
They also wanted to accomplish two other goals. "First, we wanted to democratize our data access, making available player statistics and game play analysis to a wider portion of our staff. We also wanted to create actionable insights into the underlying software components of the game, so our staff could more readily do upgrades and fix problems," said Livingston.
To upgrade the software infrastructure meant incorporating Hadoop along with a cloud-based data warehouse and an end-to-end automated software development pipeline. The Hadoop transformation touched on several tools and add-on programs for various purposes:
- Honu: Streaming log collection and event processing pipeline
- Platfora: BI analysis and visualization
- Oozie: Workflow job scheduler
- Hive: Data warehouse and queries
- Chef: Code deployment and configuration management
- GitHub: Versioning and tracking of programs
- Jenkins: Build system management
- Eureka: Service discovery process
As they added to their seven PB Big Data collection, the staff kept in mind how they could leverage the Hadoop APIs to tie them together and provide the automation to link the different tasks.
The new infrastructure took three months to build, with a lot of help from different open source communities as well as hiring a former Netflix developer who had worked on Honu. "We couldn't have built this from scratch without relying on the open source forums and knowledge base to put all the pieces together," said Sandeep Shrestha, a Riot Games developer who also presented at StampedeCon.
Riot Games used Honu as a key technology in its software development pipeline.
As a result, the company process to push out software updates went from taking two weeks to less than ten minutes, and removed most of the software downtime as well. Their processes are very elastic and scalable to ten times their current game play traffic, and, Livingston said, they are looking to do further automation and extend their platform further in the future.
Riot Games transition to Hadoop wasn't easy, but the gaming firm found some serious lessons after they have implemented their various Big Data technologies.
1. Be vigilant about upgrading your Hadoop stack to the most recent version.
Given that things change quickly and new versions are released continuously, it is important to stay on top of the changes. "The community is constantly making updates and you don't want to be the one asking about a bug in the forums that has already been fixed in a later update," said Matt Goeke, one of Riot Games' developers, who also presented at StampedeCon.
2. Make your software pipeline as automated and easy as possible.
Complexity creates lag and confusion and introduces error. They now process more than seven billion pipeline events daily, something that would be impossible without this automation.
3. Extend when you can.
Do this even if the existing APIs do not support what you need. "Don't be afraid to roll up your sleeves and write some code to fill in where there are no APIs," said Goeke. "That is what open source is all about." Riot Games has contributed more than 70 programs to GitHub on various Hadoop add-ons. They want to share more of their innovations in the future, too.
4. Don't be afraid to ask for help.
One of the advantages of using the various Hadoop tools is that there are extensive and well-developed communities that you can turn to when you are stuck. "You should look at the docs in the wikis and archived listserv responses. Chances are, someone else has already asked and answered your question," said Goeke.
Your comments are like a life form from which we feed and survive. Let us know what you think.
About the author:
David Strom has written thousands of articles on Internet and communications for a variety of IT and business publications and is the author of two books on computer networking. You should follow him on Twitter @dstrom.