Managing Responsive Design Risk With Selenium
The first iPhone was released by Apple in April of 2007, with the iPad following in 2010. Before these two devices, yes, we had cell phones and BlackBerrys -- but these two changed the experience of browsing the web on a portable device, moving it from novelty to mass consumption.
Add Android and Windows Phone and today the market is full of mobile devices with hundreds of options for screen size. That thriving market forces us to either spend less time per device, spend more time and money testing (good luck with that), or innovate how we do our mobile testing.
Enter Selenium and Responsive Design
The classic solution to the "mobile" problem was a "mobile" website at around 300 pixels of width - but that doesn't work for Tablets, and certainly not for mini-tablets(phablets). Responsive design embeds tools in the website to calculate the screen resolution and resize the side to meet that resolution. In theory, the same content will look and work well on a laptop or desktop computer, tablets, and different sized mobile phones. When shifting from a larger to a smaller display, menus become more compact, and text and controls move, images may shrink. There are also other changes focused on usability, for example data pickers and drop lists will often replace text fields.
This solves the programming problem, but it does not solve what we call the combinatorial problem. There are many tests to run, multiplied by the number of important platforms and display sizes. This equals more testing that you have time for in your project schedule.
Making Selenium Work For You
Using Selenium, you might be able to get by with creating two scripted checks for each test idea -- one for mobile, and one for non-mobile. That works for the 300-pixel screens. If your development problem is simpler, maybe you only need one set of tests that can be run on a browser set to a few different sizes. There is a trade-off involved in this decision, you have to figure out if creating and maintaining user interface checks in selenium is better than the time burden of a person doing the work.
If you decide to use Selenium, you can start by recording scenarios using the Firefox plugin and running those across all of the form factors that are important to you.
Even with this scenario, you have a set of checks that need to be run one at a time taking up computer resources and time. You might be able to run this set over night and check the results in the morning. That means, at best, you are discovering important information the day after a change has been made.
A virtualization service can tighten up your feedback loop quite a bit. Using virtualization, you can schedule your tests to be run on remote servers that emulate all of your important environments as soon as a code change is in the build. Making the tests run in parallel, all at the same time, will make things even faster.
Here are some things I focus on when designing checks to find risk around responsive design.
One of the simpler things you can check using Selenium, is whether or not all of the fields you think should be on the page, are really there. There are a few different ways you can do this. You can count the number of elements of a certain type, for example all buttons on a page, to see that this number is consistent across platforms and builds. You can look for specific attributes of the controls on a page, like labels, IDs, as well as DOM attributes like visibility and if the field is editable or read only.
Form submission is a short hand way of talking about whether people can complete important tasks on your software. Being able to submit a form means that you can not only find the important fields on a page, but more importantly you can find all of the required fields, enter data in them, and click the button that sends data to your server. Without this, most products wouldn't be much good to anyone.
Navigation gets tricky with responsive web pages. As the available real estate shrinks, most navigation links get centralized all into one place. Often that is an expandable menu that becomes available if you click a button or hover over a special place on the page. Being able to access navigation tools and move around in your software is important and something that is usually pretty simple to do in a scripted check.
The Next Level
One of the downsides of using automation to check responsive design is that the scripts will only notice the parts of the web page that you chose to make assertions on. That is one of the reasons why you see me call this a check, rather than a test- you can read a bit more about checks in testing here. Scripts are good for things that can be answered with a 'yes' or 'no'.
I like to reduce this risk by selecting the most important platforms, maybe the top five, and watching the checks run. While the script checks that buttons are clicked, forms are submitted, and fields are read-only, the skilled observer will notice far more. There are a few important classes of bugs that might go unnoticed if we never take the time to directly observe the software. This will a run that is faster than a human can run, yet involves the human ability to spot problems.
While watching the script run, you can notice that text is overlapping a button, a text field is hanging off the edge of the viewable area, and other controls shifting in unexpected ways. Image scaling falls into this category, too. With automation, we can check that the image is present in the DOM, but we can't know that it looks decent on the page. That requires a person making judgements based on their experience and values.
Responsive design is about shrinking or expanding, and rearranging web pages automatically so that content will display regardless of the available space. Selenium scripts cannot tell how well a person can use this rearranged web page.
So far, I have introduced a few typical problems we see with responsive design projects and talked about how Selenium can help you to discover information faster. The ideas are pretty simple, but when used together as part of your testing strategy, can be a powerful source of information.
Try it out and let me know what you think.