Testing desktop apps is comparatively straightforward, but there can be dozens of machines for a given mobile OS, each one slightly different in display, controls and usability. Fortunately, there are resources that let you test more easily and affordably.
When its QA team began testing the company’s mobile expense tracking application, "We started out buying one of everything," says ProOnGo CEO Phillip Leslie.
At the time, ProOnGo was targeting Windows Mobile and BlackBerry smartphones for its ProOnGo Expense software. But, Leslie acknowledges, "This approach to testing on devices doesn't scale." As the company expanded its target smartphone environments — ProOnGo now also has versions for Android, iPhone, and iPad — ProOnGo needed better, more cost-effective ways to do testing and QA.
And ProOnGo is hardly the only mobile app developer with this problem.
Unlike desktop computers, the physical user interfaces of mobile devices vary tremendously, points out Joe Conway, software engineer at Big Nerd Ranch, which trains mobile developers. There's different screen resolutions, screen sizes, and button placements. Not to mention differences in features like GPS, front/back cameras, video, and accelerometers (or gyroscopes) for motion and orientation-sensing.
Combine this with two other dimensions — operating systems/versions, and telecom carriers — and the total number of unique device combinations is substantial. And you thought it was tough to test against desktop and notebook environments? Ha!
Research in Motion (RIM), for example, currently lists seven families of BlackBerry products, with each having from one to four models. The Wikipedia list of current Android devices has over two dozen smartphone vendors including Acer, Dell, Garmin, Lenovo, Motorola, Samsung, and Sony Ericsson… and that's before you get to tablets, e-readers, or other devices. And a pie chart released by TweetDeck in October 2010 of the phones that downloaded their beta Android Twitter client showed around 350 different Android-based devices. In terms of OS versions, over half these Android tweeters were using version 2.2 of the OS, and nearly a third using 2.1.
The potential total, for a developer who wants to ensure software quality across any mobile client application, is staggering. DeviceAnywhere, which offers remote testing access to mobile devices, has over 2,000 unique models, according to Leila Modarres, the company’s VP of marketing. Few CFOs will consent to provision a software testing group with every one of them.
Fortunately, there are alternatives. Here's a look at strategies and resources to let you test mobile applications on target devices more easily and affordably.
Choose Your Target Platforms First
One way to minimize device testing is to limit the environments and devices you support.
This isn't an option for all developers, but testers in enterprise environments have it a little easier, when IT limits which mobile devices are provisioned and supported. Developers writing their software for small businesses and consumers don't have quite this degree of control; the best they can do is select which OSs and versions their applications will run on.
Make sure you identify which models and versions will support your app, cautions Chris De Herrera, a smartphone and mobile device guru, and creator of the TalkSites mobile computing sites. "Some apps may require certain features, which constrains OS or hardware," he says. "The OS version is much more sensitive than model in terms of features."
Use Simulators and Emulators
Chris De Herrera suggests, "Start by testing on the emulators and simulators, since every vendor offers them."
The Android and BlackBerry emulators are target-specific, De Herrera points out. "You can choose screen sizes, and with BlackBerrys, you can specify a model name or number. Normally you would focus on the most popular devices."
Browser extensions can help, too. "To test or at least simulate a mobile device from my desktop, one thing I frequently use is the FireFox plugin, User Agent Switcher," says Dan Bittner, senior mobile engineer at R2Integrated, a digital marketing firm offering creative and technology development, including websites, portals, and mobile apps. The web service can provide a BlackBerry or iPhone look-and-feel, suitable for early testing phases at least.
Bittner also uses Adobe Device Central, part of Creative Suite 5, which has simulators, including skins for the look and feel of the device, and similar interactions for some hardware features. "It doesn't reproduce the rendering issues or nuances in specific mobile devices or mobile device browsers,” he says. “ But it does give you the correct dimensioning -- pixel height and width.” The Adobe tool can give developers and testers a feel for how the mobile application works, as well as the user interaction method (thumbpad, scroll wheel, touchscreen).
But, Bittner acknowledges, "It's not 100%." For that, you need to get your hands on the mobile device.
Buy a Representative Sample of Devices
"The problem with any web-based or simulated environment is that there are some tests that are difficult to impossible to do," says Uriah McKinney, QA manager at developer Übermind. "For example: trying to stress test with multitasking; what happens if you tap two buttons; can you crash the device," says McKinney. "That's hard to test in a simulated environment."
Also, states Chuck Hriczko, application developer at Accella, which develops applications for various mobile environments, "You have to test on multiple real-world devices, since they have differences. For example, Samsung's new Nexus S comes with Android 2.3, which no other phone has, and it has hardware that no other phone has, like a gyroscope instead of an accelerometer."
"Using the computer as an input device is so different from using a touchscreen," says Big Nerd Ranch's Conway. "It's not about how it looks; it's about how it feels."
But while individual devices can be inexpensive, acquiring many devices quickly stops being cheap. "We can't test on every device," says Lance Parker, president of iTag.com. iTag’s mobile security app (for locating lost phones and other security activities) is currently available for Android and BlackBerry, with iPhone and Nokia versions in the works. "We have to move pretty quickly and nimbly. It's almost impossible to test on every platform or device, and put out a release." For their Android version, says Parker, "We have a handful of Android devices, about 8 to 10, and we may buy another few Android phones next month."
"The general strategy I use is bounds testing," says Übermind's McKinney. "Know what the bottom and upper limit are, and test those. If there are specific devices or carriers you want to target, get them."
Equally, focus on the most popular devices. For example, says Big Nerd Ranch's Conway, in terms of Android devices, "Based on TweetDeck's tracking of device and OS, the majority of people were sticking with 1.6 or 2.2. So if you get the more common devices, chances are it will run pretty well on those other devices. And you have to take a leap of faith, that based on the performance and how it looks on this device, that it will look and run OK on these other devices."
Check for Availability of Affordable Testing Units
One important resource to consider is vendor loaner or rental programs for development partners. Some provide short-term access for testing purposes affordably, perhaps even free.
"Most of the hardware vendors have developer programs, which offer hardware for reduced costs or even free," says Chris De Herrera. "This includes Samsung and Motorola — and even Apple has one."
For one client’s application, R2Integrated determined the company would need to test nine different devices. "Because of our relationship with RIM and the carriers, we got about half those on loan, and bought the rest," said Chris Chodnicki, CTO, R2Integrated.
Developer program membership can help you stay on top of your game. "Last fall, we heard that the BlackBerry Torch was going to hit the market, and it was very important for us to support it well," says ProOnGo's Leslie. "I had every expectation I'd be going to AT&T to pick one up — but that day, three [units] arrived from BlackBerry.”
According to Research In Motion, through the Smartphone Loaner Program, "BlackBerry smartphones are available to BlackBerry Alliance Members for testing and development, customer trials, and demonstration purposes." For example, in North America, members can order up to ten smartphones at any one time for a 60-day trial."
RIM isn’t the only one. For example, Nokia has an online service free to Forum Nokia members, promising, "The remote device access services enable access to a live device over the Internet. This offers a possibility for the developers to reduce the effort on testing, demoing and content and application development."
So if you aren't already in developer partner programs, it's worth exploring.
Design for Expandability
You might not like the idea of limiting your support to a small set of devices upon which you can reasonably test your software. Good thing, too, because "You do have to support everybody that wants your app," says ProOnGo's Leslie. What’s the solution?
"We realized that the way to support larger numbers of phones (like, several hundred) was to build incredibly detailed diagnostics into the app itself," says Leslie. As a result, when a user tries the software on a phone ProOnGo hasn’t tested, the user can press a button “that sends us every imaginable detail about what went right and wrong," Leslie explains.
These built-in, easy-to-invoke diagnostics, reports Leslie, "let us test our ProOnGo software on 240 mobile operators worldwide, and on hundreds of devices. The diagnostics help tell whether it works right, and if not, what went wrong so we can fix it."
Mobile app developers traditionally have not been as pro-active in gathering these diagnostic dumps, according to Leslie. "But they're starting to realize the necessity. It's the only viable way, when there's hundreds of phones in hundreds of countries."
Use Online Mobile Device Petting Zoos
If you don’t want to test against many mobile devices yourself, There are plenty of third-party remote access test labs. Hire a company banks and racks of smartphones and other mobile devices that can be accessed and used via the web. In some cases, these companies allow developers to come in and use devices on-site.
Using a service like CrossBrowserTesting can save you thousands of dollars a year, and offers you a scalable solution to a problem that isn't going away.
"Remote access" comes in two flavors. In one, the hardware is wired in, accessed, and controlled from the tester's computer. With remote video, the device is monitored by a camera.
With directly-connected devices, "We connect all the inputs and outputs hardwired to a physical server, so you get access to all the features, and can interact with it," says DeviceAnywhere's Modarres. "You can turn it on and off, power cycle it, interact with camera, accelerometer, etc."
The camera monitor approach, while requiring an on-site tester as well, has some advantages over direct-connect. For example, it's often quicker to provision a new test device, and the tester sees the actual device in action.
Some of these services simply offer remote access; others, like DeviceAnywhere, also offer automated scripted testing.
Not all apps, or all app features, may be a match for remote testing. For example, iTag looked at web-based testing services, but, says iTag's Parker, "To truly test an app with hardware like ours -- it's difficult to do that over the web, versus having a bank of devices at hand."
Pricing for remote access testing varies, and can be affected by the number of devices you want to test, for how long, and project complexity (whether you're doing something simple like remote testing or sophisticated like automated scripting). For example, DeviceAnywhere's basic Test Center SaaS service has a $200 subscription fee and then around $12-15 per hour per session to test up to six devices at a time; customers often spend a few hundred dollars per month for a few months, while enterprise-class tests that support scripting and more run into five and six-figures per month.
Use Crowdsourcing for Beta Testing
Still another approach is to "crowdsource" some betatesters -- and here, too, there are Internet-based offerings, such as Mob4Hire and uTest, Inc.
By crowdsourcing your QA process, you get access to a broader range of devices. You may also get regional input, to learn how your application works in multiple languages (which may also mean feedback that's harder to understand).
That’s a lot of resources. Between simulators, devices on hand, developer loaners, remote access, and crowdsourcing, you should be able to expand the number and quality of mobile application tests you perform.
See also:
