Mobile Game Testing – Part #1: The Importance and Difference from App Testing

Mobile game testing differs from the regular mobile app testing. Effective mobile game testing derives from a well-structured and systematic approach, use of test automation framework and seamless integration with your agile process. Naturally, the simplistic view of testing is to identify bugs and help developers to remove them. However, to aim for efficient, result-driven testing it is mandatory to have test automation as a part of the overall development process.

This is the first actual blog post on the series of the Best Practices in Mobile Game Testing. Each Wednesday for the next 5 weeks we will deep-dive into a new topic, focused on something important for mobile game developers.

Introduction

Testing in general, but applying also to game testing, there are two different forms of testing that can be categorized as a Black-Box and White-Box testing. These definitions are well-known in software testing world but process-wise aren’t different with game testing. Only their goals are slightly different in-game testing:

Black-box testing – focuses on the functional and overall playability aspects of the game. In this type of testing, for example, testing the graphical user interface, user experience or any visual appearance is in the key role. Menus, graphical elements, special effects, animations, and the actual gameplay are those under test with Black-box approach.

White-box testing – focuses on the architectural, integration and systemic aspects of the mobile game: how third-party components, databases, social media/external entities, as well as graphics/game engines, audio player and so on are integrated into your game.

In the test automation context, black-box testing focuses on controls, capabilities, features, gameplay, and automates the flow of testing. For this, it is typically recommended to build test scripts that know how the game is progressing, what kinds of faulty states could there be, and also focus on image recognition. As many graphics cannot be recognized as a control mechanism – buttons, menus, sliders etc. – image recognition will help to identify those that are meant for controlling (and not just as a graphical element).

Methods Used in Mobile Game Testing

The testing methodology and used forms of testing aren’t too different from app testing. The most important ones when it comes mobile game testing are as follows:

Functional testing is probably the most common method in mobile game testing. Typically functional testing is associated with manual testing and playing ‘game through’. However, to be agile, all functional testing should be automated. Functional testing – with help of test automation frameworks- requires some basic understanding of programming and setting up an environment for testing. Automated functional testing can reveal issues related to user interface (and graphics), stability, game flow/mechanism, and integration of graphics assets.

Compatibility testing is all about making that game compatible across different devices. The typical misconception with compatibility testing is that it will be only done with the final version of the game. If compatibility testing is done during all development phases – known also as smoke testing – it can help to reveal incompatibility issues with any parts of the game, its third-party components or integrations with those actual devices that end-users use. This is very important to get the game working robust and well across all possible device variants – hardware and software.

Performance testing is important for game testing. Sluggish and badly performing games can ensure that the game won’t be successful, gets bad ratings and won’t make its creator name. Related to performance testing it is important to understand how used device ecosystem varies and what are actual requirements of the game for its users. Many games are by default developed (and tested) on high-end devices, without any chance to get decent performance level on low-end devices.

Localization testing becomes highly important when your game is targeted for the global markets. The word ‘global’ means your game needs to be ‘local’ for everyone. When your game titles, texts, and content needs to be translated and tested with devices in multiple languages, these types of tests can be easily performed automatically (with help of cloud-based device access and test automation) to change the language of devices and even do it again with the same test runs. The problems layouts can be easily picked from screenshots.

Regression testing needs to happen when anything changes in your software. Most of the mobile games today have a server-client interaction, requiring a login, uploading of data (e.g. results) and downloading of data (e.g. data, images). If you’re developing these kinds of services around your game, users need to ensure all their changes are done in code, to either server side or client side do not break the functionality of the service. Automated testing is really the only option, but manual testing doesn’t cut it.

Load testing tests the limits of a system, such as the number of players on a server, the graphic content on the screen (e.g. frames per second, FPS), or memory consumption (allocation and deallocation of it). Load testing can be built seamlessly in test automation context and heavy loads of data, usage or interactions can be simulated with test scripts and real content form backend servers. In this category, you can count in multi-player testing as it can involve several different scenarios to be tested.

6 Key Areas in Mobile Game Testing

#1: User Interface and Functionality

The User Interface and overall functionality of it has a direct impact on how successful your game will be. These two things, including visual appeal and the gameplay, are the most important thing to get right – and ensure device fragmentation doesn’t break any of these. In the UI various things need to be tested:

– UI layouts and elements: All games are typically targeted for a maximal set of different screen resolutions and different types of screens. Regression testing should be done each and every time when UI layout changes to ensure that game works.
– Menu structure and functions: Testing menu structured, functionality and correct behavior can be automated with instrumentation and help of different test automation frameworks.
– Screen orientation: Surprisingly this gets so wrong with many games out there. For example, if screen orientation changes during the game what happens, and what actually supposed to happen? Does the game work well in both landscape and portrait modes?
– Screen resolution: There are lots of screen resolutions especially on Android and typically autoscale can help game developers. However, it is highly recommended to test your game across these different resolutions to ensure that the graphics do not get stretched.

#2: Graphics Performance

Performance needs to be consistent across all different device variants that your game’s users have. Because of this, it is highly recommended that you use as many real devices for testing as possible. In order to determine how well your game responds to various levels of usage, you should consider creating tests which last for hours to observe how well the game as well as how much of battery is used during long usage. To determine if your game will be able to run successfully for a sustained duration under heavy load, Load/Stress tests can be used. The performance tests will measure how responsive your game can be on a real device.

#3: Usability and User Experience (=Good Entertainment)

Testing usability, navigation flow, and eventually what user experience your game delivers to gamers simply cannot be done on a desktop with a mouse and keyboard. So forget emulators and use only real devices for testing. And to get the best possible understanding of how great usability and entertainment your game is providing, here are two important things you should consider when testing ‘user experience’:

User Interactions and Excellent Responsiveness – It is important to test the game for its performance as this will make either the positive or negative user experience. For example, performance lags are easy to expose using actual devices.

Real occurring events on background – Interrupts, battery consumption, how charger effects to overall performance and usage. These all have a significant impact on user experience – and good entertainment level.

#4: Multi-player/User Features

Nowadays, the multi-player/user feature is very common in mobile games. A need to test multi-player abilities is there and is naturally more challenging and requires the other end to perform as a real counterpart. For example, a typical case is that the game communicates with the backend server. In these cases, connectivity is essential and the synchronization between backend and devices that need to get information about the gameplay. There are tons of different scenarios you should be testing and many of those will severely impact the game’s experience, resulting in negative feedback and gets uninstalled by users.

#5: Social Integrations

Social network integration is also a very important part of your game. In many games, it is essential to be able to share results across your ecosystem, friends or just for yourself. This should be thoroughly tested in the context of real Android and iOS devices, with different OS versions and different device configurations, to understand the functionality and easiness-of-use.

#6: Security and Liabilities

Nearly all game developers use some open source components as part of their game. This is highly recommended and well-accepted development practice as it offloads the task of developing code for non-core functions in your mobile game. However, identifying third-party code, its vulnerabilities, and license restrictions are very often neglected by game devs.