The fact is, there is no major field of software engineering more rife with potential hazards than that of mobile development, and we’ve all experienced for ourselves the sudden surge of frustration that results in a deleted application. In the rapidly growing mobile market—unlike the more familiar realm of desktop- and web-based software—your customer is often on the move, and the proverbial moment of truth is just a narrow slice of time, so you really don’t get a second chance to make a first impression. An ugly GUI, a confusing UX, or a series of interminably slow API calls can lead to your app’s tragic demise before it ever has a chance to grow and thrive.
This is why mastering the art of mobile testing is increasingly important for any company that wants to remain competitive. Far too many developers, it seems, are content to just throw their latest rushed apps at the App Store wall and see what sticks. The approach is certainly understandable. Software testing can be hard work. To remain competitive in a fast-paced and crowded market, sometimes cutting corners makes sense. But choosing speed and quantity over care and quality has never been a good long-term business strategy, and the app-store halls of Apple, Google, Amazon, Windows, BlackBerry, et al., are littered with thousands of one-star reviews that tell a tragic tale.
It doesn't have to be this way. The vast majority of software firms already recognize the value of investing in the development of mobile apps, so it only makes sense to extend that investment into testing mobile apps as well, despite all of the added cost and complexity that programming and testing for a vast array of mobile platforms entails. In the long run, the extra work involved in quality assurance usually pays off. And in a world where Gartner estimates that $26 billion dollars in revenue was generated from 102 billion mobile app downloads in 2013, and we may be seeing over 268 billion downloads per year by 2017 (or around $68 billion dollars), that potential payoff for automating android testing may be worth taking seriously.
In this article, we’ll take a broad look at the state of mobile testing and development today, exploring its tremendous potential, its unique perils, and the exciting promise of what lies ahead in this continuously evolving field.
Rise of the Planet of the Apps
Ever since January 9, 2007, when Steve Jobs held aloft his magic iPhone and brought the world’s early adopters to their knees, we’ve been living in a different world. It’s a world where, according to the International Telecommunication Union’s 2014 report, the total number of cellular subscriptions is fast approaching seven billion—one mobile phone for nearly every person on the planet—and 22% of those subscriptions are for smartphones. Research by Business Insider indicates that prior to the iPhone, only 2% of the global population had smartphones—and the old BlackBerry and Treo, as far as app development went, weren’t particularly exciting. But now, with one out of every five persons on the earth carrying an advanced computing platform in his or her pocket and more acquiring them every day, the creative potential and financial incentive for developers interested in mobile applications has never been greater.
In a “State of Mobile” study of 1,040 software developers, testers, and consumers conducted by SmartBear and published in early 2014, we discovered, among other things, that:
- Nearly 30% of those building any kind of apps were building mobile apps
- 54% of respondents who were building mobile apps had entered the space within the past two years.
- 84% of those who were not currently building mobile apps planned to enter the space in the near future.
- 30% of companies were planning to develop 5-20+ new apps in 2014.
- 40% of consumers download 5-20+ apps in a single month.
Stats like these drive home the growing significance of the mobile market—especially the last two figures, which suggest that the monthly consumer demand is currently exceeding the entire annual supply of apps produced. You don’t need to be an economist to see the enormous market potential here. And if you or your organization are among those arriving fashionably late to the mobile party, perhaps due to the common impression that most smartphone, tablet, or phablet apps are for social networking and gaming, be advised: a 55% majority of mobile app developers in our survey were working on applications for Business Productivity, Utility, and Finance. The rest of them were attending to the other usual suspect categories—Social Networking, Games, Lifestyle, Navigation & Travel, and Entertainment—and yet not one of these genres claimed, in itself, more than 6% of the developers polled. So it clearly isn’t all just fun and games, but no matter what your personal or corporate focus may be, there is probably sufficient demand out there to justify creating an app for that.
Still, with so much competition out there, which is only increasing with each passing day, what’s going to ensure that your app succeeds while others fail? Once again, apart from originality there’s only one market factor in software development that you can reliably control, and that’s quality. According to the consumer reports collected by our survey, nearly 50% of users will delete a mobile app if they encounter a single bug. And as the following chart shows, they care about quality—particularly functionality and speed—more than developers and testers currently do:
(A slide from SmartBear’s State of Mobile 2014 webinar and eBook, available here.)
It’s therefore safe to say that when it comes to the success of mobile apps, nothing is more important to focus on than quality . . . or more challenging to attain.
Understanding the Challenge of Mobile Testing
Naturally, if quality assurance for mobile apps was easy, more people would be doing it (or at least doing it better). But as anyone who’s ever taken a single step into the mobile minefield knows, things get rather tricky, fast. Developers and testers are immediately confronted by multiple operating systems (and different versions of each OS, especially with Android), multiple devices (different makes and models of phones, tablets, phablets), multiple carriers (including international ones), multiple speeds of data transference (3G, LTE, Wi-Fi), multiple screen sizes (and resolutions and aspect ratios), multiple input controls (including BlackBerry’s eternal physical keypads), and multiple technologies—GPS, accelerometers—that web and desktop apps almost never use. There are countless testing tools to select from, and even then a home built piece of software may be an option.
Yet even before we take all of those shifting factors into account, we have to pin down what kind of app we want to develop or test in the first place. There are native apps, web apps, and hybrid apps to choose between, and they’re each built differently.
Native apps, the most common form, are applications built to run on a specific mobile device’s hardware, sometimes with and sometimes without a data connection—like the Kindle app, for example, or Angry Birds. Web apps, however, almost always require a data connection, are usually designed to run across all platforms using the same basic code, and are really not “apps” in the app-store sense but are simply, as the name suggests, web-based applications. The difference between an ordinary website and a web app in this context, though, is that these apps range from mobile-friendly versions of websites to websites that are custom-built (usually with HTML5, JS, CSS), from the ground up, specifically for display and use on mobile devices. Finally, with regard to hybrid apps, these are essentially web apps packaged in a native-app wrapper—available in app stores and designed for use only on mobile devices, but also relying heavily on communication with a website or web-hosted database, such as via APIs, in order to function at all. (One common example is the Wikipedia app, which requires an active connection to Wikipedia.org’s online database to deliver its Apache Lucene search results.)
In other words, opening the Pandora’s Box of mobile testing and development is not for the faint of heart. But for those courageous souls who have dedicated their lives (or at least careers) to mastering the art and science of software quality assurance, let’s continue with a closer look at five of the biggest challenges facing QA testers of, say, native apps. (While we’re focusing on testers in this article, it’s good to bear in mind that the majority of developers, especially on agile production teams, also perform testing throughout the mobile-app production cycle—around 72% of them, according to our survey.)
(1) Device Fragmentation: These days, it isn’t simply a matter of testing an iOS app or an Android app; you have to test each app’s functionality and appearance across a huge variety of physical devices, which, as a result of possessing distinct hardware characteristics, utilize even the same versions of installed software differently. If you’ve ever seen the selection in an AT&T store, browsed the collection at Verizon, or played with the phones in an O2 shop, you know that the actual variety of mobile devices on the market at any given time is itself enough to boggle the mortal mind. You can, of course, choose to test your software on only the most popular models, but new devices are constantly being released, not making that task any easier. Just think of the perennial Mac vs PC difficulties, especially before Macs adopted Intel chipsets. Then multiply that by Samsung, LG, HTC, Sony, Motorola, ZTE, TCL, Huawei, Nokia, Amazon, Apple, and others—plus all of the ROM, display, and OS variances within each of those manufacturers’ product lines. Yikes.
(2) Data Consumption: It’s no secret that mobile apps can wreak havoc on your phone bill. For those not lucky enough to have unlimited data plans or a perpetual Wi-Fi connection, remaining mindful of one’s app usage is a very real concern. Google Maps, Waze, YouTube—all data-hungry applications can suck your allotted data-plan dry, and too many exorbitant payments to Verizon can make one rethink his or her smartphone habits. The simplest solution is often to simply delete the most nonessential but data-hogging apps. Don't let yours be one of them; test to see how much data your applications really use. And while you’re at it, check for data-speed consistency too (i.e., does your app run as well on 3G or LTE as on Wi-Fi?).
(3) Processing Power and Battery Life: Gamer, much? Just like with laptop PCs, nothing taxes the processing power, memory, and battery life of a mobile device more reliably than video games. Rendering high-res framerates smoothly always takes its toll, but games aren’t the only apps worth worrying about. Mobile web browsers excel at draining memory resources, and certain streaming-music apps don’t fare so great, either. But these are just the obvious culprits. Any kind of app could be a resource hog. If you don’t run performance tests to vet your application properly, you could be the next app creator responsible for draining an iPad battery before the morning subway commute is done.
(4) GPS, Biometric Scanners, Gyroscopes, Accelerometers: Some of the things that most desktop and web application developers rarely, if ever, need to take into account are the very things that make mobile apps so incredibly useful. But programming apps to use a phone’s GPS, gyroscope, fingerprint sensor, or accelerometer capabilities can be terribly problematic. Does your geolocation-enabled dining app cease to function if it loses a GPS connection . . . completely? Partially? Can your e-reader app use the accelerometer to switch between horizontal and vertical screen orientations? Can you program your Flappy Bird clone to use the iPhone’s gyroscope and make the game even harder?
(5) Touchscreen UX and GUI: The thing that changed everything, spawning the entire mobile-app world as we know it, was the original iPhone’s jaw-dropping touchscreen. The fluidity of it, the natural, effortless user experience, simply dragging one’s finger across a rounded machine of smooth metal and glass resting gently in the palm of one’s hand—nothing like it had ever been seen before. And nearly every smartphone and tablet computer today is modeled after its design. But it wasn’t the hardware alone that made it work; it was the seamlessly integrated GUI in the form of iOS and its touch-sensitive apps. The bar was set high from the start, and any app developers today have to clear it. So when testing your app, ask yourself: Is this app intuitive? Is it easy to figure out and use? Does it respond to my every gesture and whim as it should? Most of all, is it pretty?
So, as you can see, although there are clearly some big challenges facing testers of mobile apps, fortunately the same familiar tools and techniques that are used for desktop and web applications can also be used to test mobile ones—along with a few additional tools, like emulators, to make testing a vast array of devices just a little bit easier.
Choosing a Mix of Mobile Testing Solutions
Because 50% of mobile users will delete an app if they find just one bug, putting in the time and effort to make sure that your applications are error-free is critical. But eradicating programming glitches is only part of the story. An equally important task for testers is to help ensure the overall quality and value of the product itself, striving to look at an app more objectively than its developers typically can. Does it perform as expected? Is it intuitive? Is it well-designed and easy to use? And most importantly: Will your end-users love it and want to share it with their colleagues and friends?
Those kinds of questions can usually be answered by the most common quality-assurance practice for mobile apps: manual testing. Using the actual smartphones or mobile-OS tablets at their disposal, testers can perform a variety of usability and functional tests on apps under development, checking features such as sign-up and login fields, menus and navigation, touch gestures and scrolling, input key functionality, screen size and GUI appearance, connection speed and data handling, interruptions (such as how your app responds to an incoming phone call or text), operating system versions, crashes and error messages (does your app display an actionable message or simply stop working?), accelerometer response, etc. Seeing how an app functions in real time on a real device is, of course, an irreplaceable testing method, and testers should try this on as many of the most popular devices that they can get their hands on (or enlist beta testers to help).
But with so much device variance in the marketplace, it’s impossible to physically test them all—and yet it’s getting close to being possible, virtually. To take manual testing all the way, emulators and simulators are an app tester’s best friend, and software tools like Genymotion—along with common SDKs (software development kits) for Android and iOS, including the iOS Simulator—allow testers to check their apps’ functionality and behavior across a wide variety of virtual devices. These digital versions of smartphones and tablets replicate not only different OS versions, but also much of the hardware performance of the most popular mobile devices on the market today. Granted, emulators won’t necessarily reveal things like apps’ excessive battery use, struggles with bad Wi-Fi connections, or the consequences of a suddenly lost GPS signal, but they do allow a much greater number of devices to be tested, at least generally, than could ever be achieved in a reasonable timeframe through physical testing alone.
In our survey, as seen in the chart below, we found that about 28% of QA professionals relied on manual testing as a component of their overall quality strategy. Still, it can be impractical, to say the least, to run manual tests after every code modification, especially in a mobile DevTestOps environment committed to a rigorous continuous-deployment schedule. That’s why the vast majority of testers (94%) are also using at least two other traditional quality processes, such as automated testing, API testing, and load testing, in order to get the job done (and 33% use four to six processes).
(A slide from SmartBear’s State of Mobile 2014 webinar and eBook, available here.)
Different apps may benefit from these approaches in different ways, such as employing load testing and performance monitoring for likely resource hogs. But the use of automated regression testing is a pretty safe bet across the board, always ready to ensure that your latest round of updates doesn’t break your app (often testing in comprehensive ways that manual methods simply can’t). Tools like SmartBear’s TestComplete deliver a powerful fusion of both robust regression scripts and functional use tests, enabling a full range of automated mobile tests to be easily written and deployed in batches on regularly programmed schedules. As a crucial complement to hands-on mobile testing, automated mobile testing can save time and money by significantly expanding test coverage and catching more bugs—faster and more thoroughly—than manual testing alone.
The Future of Mobile Testing
Given the complexities involved with the mobile market, it’s unlikely that we’ll ever find a one-size-fits-all testing solution. There’s a reason why one-third of testers use a combination of four to six distinct methods. Every app is be different, with its own unique purpose, context, and technical requirements. This means that certain testing approaches will be more appropriate—and therefore far better investments of time and money—for some dev projects over others. But again, despite the complications, proper mobile testing should never be simply bypassed or ignored. The mobile market is only on the move, and the apps that stand out will be the ones to ensure that their creators survive while their competitors fall by the wayside.
As uTest writes in their eBook on mobile testing, “Companies that recognize this trend for what it is—a technological revolution—and take preemptive action to make the quality, security and usability of their mobile applications top priority, will find themselves with a tremendous advantage.”
This technological revolution is all the more apparent when we consider that the term “mobile app,” as traditionally defined, is on the verge of exploding in all directions over the coming months and years. The sheer demand for apps is speeding up daily, and as applications are becoming more powerful and more complex, the hardware they run on is shrinking. Beyond the old smartphones, tablets, phablets, and iPod Touches, an onslaught of wearables—smartwatches, fitness bands, and even rings—are already overtaking us. When you add to this picture Google Glass (and forthcoming augmented-reality contact lenses), Oculus Rift, Google Cars, Nest, and every other soon-to-be-omnipresent manifestation of the Internet of Things . . . well, let’s just say that the market for desktop and website applications has met its match, at last. We are living in the age of the mobile app, and it’s only just begun.