A Google search for the phrase "UI Testing Framework" yields 1.7 million results. Your manager wants a proof of concept in a few of these to see how they will or will not help test your product. Which do you choose? And why?

News flash: Choosing test frameworks is confusing.

There are lots of questions -- do we choose a free and open source tool, do we go with an enterprise automated test framework? How do I know this will work with my technology stack? The answers aren't always easy to find. This should make your choice a little easier.

Drive A Browser

This set of tools is designed to control the web browser as it were being controlled by an invisible hand.  Some offer support across nearly every important kind of web browser, some create an easy way for non-technical folks to get started quickly, and others put power in the hands of the programmer to test just the way they want. All of the options you are about to read about are free and open source.

Selenium:

Selenium is an incredibly popular open-source tool. All the popularity means more than just a fan-club; it means a large potential support network. There are two main parts to Selenium. WebDriver is the base framework that helps you to do things like click buttons, set text in fields, and check values on the screen. The other part is called Selenium IDE, a plug-in for FireFox that you can use to record the actions you take and then export then to the language of your choice to run later.

Once you get a hang of the WebDriver framework, things will feel pretty intuitive. To get help get you started you can read our Selenium Guide. Here is a quick example in Ruby:

driver = Selenium::WebDriver.for :chrome
driver.navigate.to "http://www.kurtdanziger.com/"
String pageTitle = driver.getTitle()
assertEquals(pageTitle, "Kurt Danziger")

This code snippet creates a new instance of WebDriver and opens the Chrome browser to www.kurtdanziger.com. Once the browser is open, we check to see that the page title is equal to the string "Kurt Danziger".

A few browsers have implemented versions of WebDriver specifically designed for how that browser works, so you can be comforted that the latest version of Internet Explorer or FireFox will probably have corresponding WedDriver updates on the way.

Selenium has a very large support group, partly from the community of developers and consumers of the framework, and partly from vendors building tools to help support testing with Selenium. Although there is a large support group for Selenium, there are still limitations.  TestLeft is an automated UI test framework for creating and running functional tests for Windows desktop and Web applications. With TestLeft, you can generate application models for webpage or desktop applications in just two clicks - saving you hundreds of hours of test creation. The industry leading UI Spy helps identify objects in seconds and your tests can be done all in your IDE.

WATIR:

WATIR, Web Application Testing In Ruby, is one of the oldest frameworks designed to help people automate testing a web browser. Much like Selenium, WATIR is a group of tools. Each library in the WATIR suite provides special functionality. While WATIR will only support Internet Explorer running on the Windows Operating System, you can access most others using a special implementation of WebDriver called Watir-WebDriver.

If you don't want to roll your own scripts to organize and run your scripts, WATIR is compatible with other frameworks like RSPEC and Cucumber.

Because WATIR is built on Ruby, you will need to manage your Ruby version and WATIR gems carefully to make sure everything is compatible - this can add an extra layer of effort and debugging.

TestStack.White:

Despite what it looks like, the entire world isn't developing software for the web. There are still lots of companies developing win32 applications that get installed on a client computer. TestStack.White is developed to satisfy this exact need.

Aside from the difference that you will be working in an installed product instead of accessing your software through a browser, White functions much like the other frameworks I already mentioned. You can take control of a window, look for text, click buttons, and make check box selections all by using the framework.

...Or Go Headless

Even when checks in the browser are automated, things go slowly. Checks can only complete as fast as the browser will load and display content, and a few more minutes are added for every check you add to the suite. Mixing up automated UI checks with Continuous Integration systems gets tricky because of timing. Your test suite may not finish before a new line of code gets checked in and triggers the next test set to run.

A headless browser like PhantomJS has most of the guts of a web browser, but none of the UI. Phantomjs is built on webkit, the underlying technology behind Chrome and Safari. TrifelJS might be what you are looking for if you want to focus on Internet Explorer.

Phantomjs and Trifeljs aren't the test frameworks, they are the browsers. To get headless tests running, you'll want to write tests like you normally would (in WebDriver, for example), but instead of running them through your browser you'll use a test runner that is compatible with the framework you are using.

There are some trade-offs to consider when running headless. You get much faster test runs since there is no user interface to load and wait on at each step in the test. But, you also get an even less realistic and more narrow check. Rather than firing an event that clicks a button that submits an order form, all of this happens on a DOM (document object model) that you never see. Any issues that are specific to how your code works in a particular browser will never be seen.

Choose speed or choose thoroughness. You can't have both.

To Code, Or Not To Code

Lots of tools boast the selling point of not needing any programming skills to use squeeze every bit of value from that product. In my experience, the amount of coding you will need to do to get a working UI test is on a spectrum. You can code a little, or you can code a lot. But at some point you, or someone on your team, will need to write a few lines of code to create the check you want.

At one end of the spectrum are tools like WATIR. This tool is fantastic for programmers that want the power to create the exact test they want. The people that use this will probably be programmers that have moved toward the testing and tool-smithing side of the business. To make this tool useful, you'll need to have working knowledge of your programming language of choice as well as the testing framework.

Tools like Selenium IDE, and SmartBear’s TestComplete can move you toward the "code a little" side of the equation. These two tools in particular provide a test recorder.

To create a test, instead of opening my IDE and typing the line driver = Selenium::WebDriver.for :chrome, I hit the record button, open up a browser and navigate to my test environment, and actually perform the actions that will become a scripted check. When I'm done recording, those actions I performed are converted to code.

This is where the code part comes in to play. What you get from this isn't a powerful test. I almost always revisit that test to add in assertions for things like the value on a text field, or the state of a radio button, and also make a few tweaks in case there are steps that don't need to be there. My colleague, Matt Heusser, suggests that you start with the test with hard data. For example: Entering in "happy birthday!" in a text field, saving it, then expecting to see "happy birthday!" on the screen. Once the test is done, pull the "happy birthday!" out and replace it with a variable in two places - now you can call the test with parameters. One common place to start this is login, with parameters for username, password, and text to expect on the next screen.

Bottom line: Don't neglect the skill level and ambitions of the people that will be using a tool when you select it.

Final Thought: Operating Environment

The first UI I tried to create a test suite for was built on Adobe Flash Flex. This was a display technology based on the ActionScript language. At the time it was the hottest thing around, a brand new tool. Everyone wanted to get experience with Flex. Since the technology was so new, there were no out-of-the-box tools available that would get us driving a browser. We couldn't just grab the latest fork of WATIR and start being productive.

Today the hottest UI technologies are JavaScript libraries that either change version or completely go out of style once a month, proprietary technologies like Flash and Silverlight (which thankfully aren't as popular as they once were), and newer and more comprehensive versions of CSS and HTML. Figuring out how well your automation framework handles your technology stack might save a lot of customizations later.

In addition to the important choice of what framework, most of the time you will have to select a language as well. My instinct is usually to select the language that most people in the development group are familiar with, even if that is not my specialty. Choosing that way improves the odds of turning the production programmers into a support when the testers when you run into stumbling blocks on the language. It can also help when the tool is struggling to interface with the Software Under Test (SUT), for example how to know when a page is completely loaded.

You can use many different tools to test your UI, and there is no "best" option. Thinking ahead of time about the testing problems you want to solve and the people you have can help make the effort a success.