Agile development and testing is growing in popularity and smart QA/testing teams keep pace with current development trends. Agile testing tools vary from project management tools to automated testing tools. We recently spoke with JoEllen West, Agile tester from VersionOne, to discuss automated testing and its’ role in Agile environments. VersionOne is recognized by agile practitioners as a leader in Agile project management tools and JoEllen is the lead tester on their agile development team.
Automated Testing of Agile Projects vs. Waterfall Projects
JoEllen had this to say about the difference: "Automation is a critical component to maintaining agility, and is a priority for the entire team through established practices/disciplines and a focus on continuous improvement. Continuous integration/builds, unit, functional & integration test execution and continuous/automated deployment are common examples of applying automation beyond the scope of traditional automated tests."
"In contrast, waterfall projects will tend not to invest in automation as a priority, primarily due to the phased investment and handoff mindset that accompanies that approach. At best, builds may leverage some automation, and a phase of a waterfall project may be dedicated to functional test automation for long-term support/future release needs, but it’s rare to find much more in a back-end-loaded waterfall project."
Agile Tester Skills
Regarding agile tester skills, JoEllen said that an Agile tester will have a more team oriented approach and use a wide variety of tools. "In addition to a quality/test-oriented outlook, an agile tester will incorporate a wide array of testing tools and technologies into their testing efforts. They are an integral member of the development team, participating fully in the development process, from design through delivery & deployment. A deep understanding of the domain and features under development is essential to developing tests that provide meaningful feedback. Testers may also work on several stories or features at one time, so tools to organize and juggle multiple testing activities are critical. They will have adapted to a more collaborative, team-based approach to the iterative/continuous delivery of high-quality, working software as a measure of progress."
Automation Responsibilities on an Agile Team
"The team! Literally anyone who does anything repetitively more than 2 or 3 times in a short period of time should and will automate it, preferably with a well known tool such as TestComplete, open source options or homegrown scripts/utilities. Testers generally shoulder the development of functional test automation, since that activity can parallel the actual feature development."
Deciding what Tests get Automated in Agile Environments
"The individuals doing detailed planning for a given user story (i.e. product owner(s), developers, testers) may make some of those decisions at that time. Developers and testers working on the story make detailed decisions about where automation can be used to trigger acceptance/signal completion of the story. During testing, testers may see opportunities for adhoc, on the fly automation to increase test coverage."
Start Automating Tests with a TestComplete Free Trial
Maintaining and Reviewing Automated Tests in Agile Environments
"Since tests are often created by tester-developer and/or tester-product owner pairs, there’s a built-in review process of the test scope and validation level as part of story acceptance. Otherwise, the collaborative planning to define the tests, then the collaborative acceptance with the product owner provides ample shared knowledge and review. The end result is robust, resilient automated tests which provide the backbone of regression testing, and any test failures from regression testing may also trigger a test review."
"Ideally, the entire team maintains all tests, whether unit, functional, or any other. However, testers may use specialized tools for GUI-level tests that require their involvement to maintain tests over time. To reduce the possible silo-ing of this information, the tools should be carefully chosen to be accessible to all team members, information should be readily available, and the test inventory should be continuously culled to prevent test bloat. Beware of the tester building a mammoth suite of tests in a specialized environment as a job security strategy.”
Agile Teams Automate Black Box and White Box Tests
"For functional/acceptance tests, it’s most common for there to be a combination of white-box (under-the-covers) and UI automation (black-box) tests. The white-box tests tend to be less fragile, so are less costly to maintain, whereas the black-box UI-driven tests need to be leveraged judiciously due to the cost to develop and maintain the tests over time. Ideally, testers are involved in both, so that the black-box tests can supplement instead of duplicating white-box tests."
Implementing Test Driven Development
"By definition, TDD is a developer coding practice (via a unit test framework), not an agile tester practice. Simply, a developer will write a failing unit test, add the associated code to satisfy the test, then re-run those tests to ensure success. Subsequent refactoring/redesign of the resulting code is done whilst keeping the unit tests running ‘Green’. The standard TDD mantra is Red-Green-Refactor."
"However, the same concept (i.e. test-first) is definitely applied to higher-level functional and acceptance testing, whether automated or executed manually. Essentially, detailed requirements for a user story are expressed in the form of acceptance tests, which can then be captured in advance in a test automation environment such that they will eventually run ‘green’ once the functionality is in place, signaling completion (they may run ‘yellow’ on incomplete stories)."
When Not to Use Automation in an Agile Environment
It's important to know when and when not to automate tests, this holds true no matter what methodology is used. JoEllen said that if a test is only performed once then it shouldn't be automated. "One-time tasks and exploratory testing/edge cases shouldn't be automated. Edge cases are, by definition, usually one-off test cases, and the effort to automate edge cases generally does not pay off. Exploratory tests are best used to gain knowledge of a new feature and then tweak or revise tests based on the new knowledge."
Switching from Waterfall to Agile
When asked if she had any advice to give testers who are switching from a Waterfall project to an Agile project, JoEllen had this to say: "Be aware of the change in mindset/culture. Experiment with modern tools. Keep the big picture of the application in mind and consider individual user stories within the bigger context. Think holistically, and encourage the rest of the team to do so as well."
Functional, GUI, Regression and Performance Testing in an Agile Environment
JoEllen explained that all testing types have a place in an Agile shop, it is just a matter of how they are used. "Unit, functional, acceptance, GUI (some of which become regression), performance and load testing are all used. We try to select the most appropriate tool for the team and the type of test we’re performing, so we have several different tools in our arsenal."
"All of these testing types have a place in an agile environment. However, it’s important to incorporate them in an Agile way, rather than a prescriptive way. Balance the quality goals of the team with the requirements and risk of individual user stories."
Automated Testing Starts on Day One
The automated testing process begins the same time as development in an Agile shop. JoEllen said, "The testing process begins at the same time as development - Day One."
Automated Testing Tool Features - What is More Important to Agile Testers?
"Most agile teams, regardless of any specific methodology, are looking for fast, reliable feedback from their automated tests. Fast, reliable feedback means timely results with minimal false failures, ideally incorporated as part of the continuous integration (build) pipeline."
Automated Testing ROI in Agile Environments
"It’s really a prerequisite to becoming a truly agile team. Automated tests can provide faster feedback than a manual test, reducing rework and long feedback cycles. Thus, if ROI is producing high-quality software iteratively, then yes, you see a nice return on your automation."
Automated Testing and Agile Methodologies
JoEllen explained automated testing's role in different Agile methodologies: "In XP, automation is explicitly called out in a variety of disciplines (TDD, continuous integration, automated acceptance testing). Scrum teams may have incorporated those practices on their path to agility, but automation is not prescribed by the base methodology. Lean product development assumes automation in the pursuit of waste elimination."
Implementing Automated Testing in Agile Environments - Challenges
JoEllen explained the challenges frequently seen in Agile shops: "Availability of people & skills, a holistic team-oriented approach, striking the right balance between automation and excess test inventory which can result in drag. Some teams struggle with when to automate functional tests – whether to delay acceptance of a user story until the automated tests are complete, or to have automated test development as a separate activity. We’ve found that with the right tools, development of GUI-level tests can easily take place as standard work on a user story. However, automating functional tests on a story-by-story basis may lead to test bloat and may not provide the higher-level coverage that’s desired from these tests. Teams may choose to write user stories to prioritize and schedule test development at that level."
Automated Testing Benefits in an Agile Environment
JoEllen said that the same benefits seen in other areas of testing are also seen in Agile shops. "Speed, quality, malleable code base (i.e. adaptable/open to change), feedback, efficiency are all benefits that we experience using automation for our tests."
TestComplete in Agile Environments
JoEllen had nice things to say about TestComplete for Agile testing: "A co-worker’s prior company chose it over the high-end commercial offerings and really digs it for automating acceptance testing with both client-server and browser-based applications. It’s both developer and tester-friendly, so it reduces the risk/tendency of specialization/single-role maintainability of the test suite."
About VersionOne
Since 2002, companies such as Adobe, BBC, Siemens, Disney, Dow Chemical, IBM, Lockheed Martin, Sony, 3M and Business Objects have turned to VersionOne to help provide greater value to their customers. Today more than 10,000 teams and 70,000 users in 50 countries use VersionOne’s Agile project management tools to streamline and standardize their Agile development efforts.
About TestComplete
TestComplete is a comprehensive automated testing tool with integrated support for a wide range of both internal and UI based testing. Designed for use by developers and testers alike, TestComplete offers both visual and scripted test generation and robust test management facilities. TestComplete supports Microsoft Windows, .NET, Delphi, Java, Web and many other applications.
Learn more about TestComplete and download a free trial.