Architecture and Infrastructure Aspects of Mobile Game Testing

One important thing is to deep dive into a few important topics in mobile game testing and how to adopt this with test automation for your development process. So, let’s start with the infrastructural and architectural aspects of mobile game testing…

One of the most common problems in mobile game testing – similar to app testing – is how to access all possible devices that your game users are using. As said, despite thousands of unique models already out there for Android (and with all possible iOS variants with different OS versions) only a few hundred of them are really relevant in the big picture. And naturally, this keeps changing all the time; new devices are introduced by tens of major OEMs every quarter and we’re naturally determined to keep our device farm as up-to-snuff as possible.

Architecture and Infrastructure for Testing Mobile Games

Let’s start with software architecture. Many of today’s mobile games are based on open source or commercial game engines, such as Unreal, Unity3D, CryEngine, Construct, PlayCanvas,  Cocos2D, and many others! (DISCLAIMER: I’m not taking sides of which game engine works the best or delivers the greatest features, just listing a few popular ones for reference.) Furthermore, these game engines provide great tools and frameworks to get your game quickly built, with all those nice graphical features.

Looking this from the traditional software point of view, this is like a “tool – app – middleware” offering that these products will provide you, as well as compiling your game to specific platform (many of these products are cross-platform and enable you instantly to compile the game for Android, iOS and other platforms as well). For the platform part, let’s use Android as an example: an “open” operating system with a stack of software components which can be roughly divided into four layers: Application, Application framework (manager layer for content, resource, package etc.), Libraries (e.g. OpenGL ES software, Fonts, WebKit, SGL etc.) and the Linux kernel (drivers for graphics, audio and keypad, power management etc.). In addition, the platform here includes the actual hardware running whatever chipset on it, with GPU and displays of different resolutions.

After that, your mobile game needs to communicate with back-end servers using your own or Google/Apple/other services via WiFi, radio or some other type of communication channel. The need for testing the connectivity between your mobile game and back-end services is nowadays very important. For example, ads are shown via connectivity and without those running properly on your device, you are taking an extra risk in front of your core audience – mobile gamers. Sure, they would probably like it better without ads, but from the monetization point of view, you surely want to test this thoroughly before submitting your app to app stores – and make sure you are not losing revenue because of faulty implementation.

As said, testing connectivity should be one of the key things to do when building mobile games. Test automation can bring you a lot of benefits for automating the connectivity testing, and scaling the infrastructure from one user to hundreds of thousands of users. There a good check-list of connectivity testing aspects in our game blog series.

Hermetic Environment and Testing

Today, nearly all mobile games have a server-client interaction, some requiring a login, uploading of data (e.g. game results/progress) and downloading of data (e.g. ads). When adopting these kinds of services to be part of your games, developers need to ensure that all their changes done in code, to either server-side or client-side do not break the functionality of the service. Furthermore, this type of environment can be also set up on a local machine/server where critical connectivity and interaction between user and server can be tested. In fact, testing for some of these types of use cases could be done efficiently with a hermetic environment – however, not jeopardizing the integrity of your game.

Hermetic testing can be seen and done as easily as switching your device to flight mode. Nothing gets sent out and nothing gets in.

Compatibility and Porting – with Real Devices

When your mobile game is getting developed, you should always test on the real platform that the end-users are using as well. This means that you must test your game on real devices. Emulators won’t simply provide you as an authentic environment as it would be required to get a full understanding of real behavior in hands of real users. What makes a difference between using an emulator or real hardware? Typically, the lack of real hardware is slowing down and forces developers to use emulators, but this can be shrunk into 4 categories why games should be only tested on real devices:

1) SOFTWARE. It’s just not only about vanilla platform software that makes the difference, but nowadays OEMs customize their software a lot: UI layers, customized legacy software, only certain widely used apps are pre-integrated with the device (e.g. Twitter, Facebook).

2) HARDWARE. There are LOTS of differences in chipsets, GPUs, even connectivity hardware, displays, sensors, and memory. This is very hard to emulate appropriately – and even it would get as close to the real-world scenario – all of that is executed on PC x86 hardware, and not ARM-based hardware.

3) USER EXPERIENCE. This is heavily related to performance. The graphics performance of the system makes the user experience, either good or bad. And there is an absolute must to get mobile games tested on real hardware.

4) NETWORK. The system can experience network issues and the slowness of the network is not possible to be tested on emulation. In terms of network configuration, emulators run on the PC, connect to the LAN and access the internet via your corporate firewall. Using real handsets, the network is connected to the radio interface and from there to the internet.

Whatever platform (e.g. native mobile platforms, web, etc.) you are building your mobile game on, remember that only real devices give you real results. Furthermore, the depth of testing (e.g. how many devices are used, how many different combinations those cover) is important as you don’t want to minimize your target audience who have the potential to play your game – make you money!

Smoke and Reference Testing

When the mobile game is in the state that is testable in the context of 1) game itself, 2) platform-compiled and 3) connect with back-end services (or just platform APIs) the full-blown reference testing can give great results and understanding of how well a game does. But even much earlier than that, it is possible to start doing smoke testing and using even hundreds of real Android/iOS devices for that. When bugs are revealed and fixed earlier, it will save you a lot of time (and money) and gets you earlier to publish your game for gamers. In addition to regular smoke and reference testing, it’s a good idea to include real performance and battery drain tests as part of your testing pattern.

The reference testing gives a great understanding of how your game does, but it doesn’t cover everything. For example, a huge number of users can quickly take down the back-end, slow down the game-play and even worse, force users to stop playing the game. More about this in the next chapter…

Performance Testing Against Back End Services

Good performance is very closely related to good user experience. Users want to see constant progress with the game, do the smooth game-playing, graphics performance needs to be up to snuff and work across all different mobile devices (phones, tablets, etc.) the very same. In addition, it is important to understand how well back-end(s) can handle a massive number of users and a significant load.

There are lots of different ways to build back-end testing and scale up the performance so that simulates the huge load for your service. One easy and very straightforward way to do this is to use a device farm on the cloud to get all of those devices to do the same thing for your service. It literally takes minutes to set it up and get results back instantly.

Conclusion – Why You Must Test Game On Real Devices

It’s important to highlight that with mobile games, the graphics performance capability is the most important thing in order to form either good or bad user experience of that game. Naturally, not all devices are capable to run smoothly that high-end game, but even with smaller changes and optimization you can
really make a difference and get your game running properly on a wide range of different hardware. The more devices you get supported, the better chance you have getting your game played by hundreds of millions of gamers.

Building a successful mobile game starts with understanding how intuitive and great game-play can be enabled for gamers. Making it sure that users understand that they are fully capable to use all great features (and graphical stuff) you’ve built in your mobile game, makes a difference. The usability, user interfaces, graphics assets, OpenGL ES content and implementation are actually those that form the playing experience. And eventually, when game-play is great and gamers stick with it, you’ll see that as an increased potential to monetize that enthusiasm.