What is Agile Software Development?

An agile methodology is an approach to software development where requirements and solutions evolve through the collaborative efforts of software teams and their customers or end users. If your software team is doing agile testing and development, then it's important to remember that testing isn't a role on agile teams, it's an activity that involves the all members of a cross-functional agile team. On agile teams, everyone is equally responsible for the quality of the product or the success of the project. This means agile testing is done by the whole team, not just designated testers or quality assurance professionals, including team members whose primary expertise may be in programming, business analysis, database or system administration. Team members whose expertise is in software testing or using a particular testing tool aren't limited to only doing that activity--they can also collaborate with customers or business owners on product requirements and work with other members of the team to develop high-quality code that fulfills those requirements. It’s this focus on the people doing the work and how they work together that separates Agile from other approaches to software development like Waterfall. On agile projects, solutions evolve through collaboration between self-organizing, cross-functional teams who’ve discovered—often through trial and error—the best processes, practices, and tools to use in different contexts.

What Is Agile Testing?

Agile development takes a test-first approach, rather than the test-at-the-end approach of traditional development. Agile testing and coding are done incrementally and interactively, building up each feature until it provides enough value to release to production. The main reasons to do agile testing are to save money and time. Because agile testing relies on regular feedback from the end user, it also addresses a common problem many software teams have, which is building the wrong solution because the team misinterprets a feature and they align what they're seeing with their development expertise, rather than what the requirement says or what the end user wants.

The Agile Testing Lifecycle

Unlike the Waterfall methodology, Agile Testing is not sequential--or done after a coding phase--but continuous. Continuous Testing is one of several continuous activities that take place simultaneously on most agile projects, including

• Continuous Build;
• Continuous Integration (CI);
• Continuous Delivery (CD); and
• Continuous Deployment.

Continuous Build or build automation is the first stage in implementing an agile software delivery pipeline. If your developers are practicing test-driven development (TDD), they'll write unit tests for each piece of code they write, even before the code itself is written. An important part of the agile methodology in testing, TDD helps developers think through the desired behavior of each unit of software they're building, including inputs, outputs, and error conditions. New features implemented by developers are then checked into a central code base prior to the software build, which compiles the source code into binary code.

Continuous Integration is a practice where members of a software development team use a version control system and integrate their work frequently to the same location, such as a master branch. Each change is built and verified by means of tests and other verifications in order to detect any integration errors as quickly as possible. With build automation, the software build happens automatically, using tools such as Makefiles or Ant, rather than when a developer manually invokes the compiler.

Agile Testing Lifecycle

In the last stage of a CI/CD pipeline, once an application passes all the required tests, it's then released into production. For all intents and purposes, this means releasing every good build to users.

Agile Testing Quadrants

Because Agile is an iterative development methodology, testing and coding are done incrementally and interactively, where features can evolve in response to changing customer requirements. Agile testing covers all types of testing, including unit, functional, load and performance tests. The following Agile Testing Quadrants diagram is a useful model for cross-functional agile development teams to use to plan and execute testing activities.

Image Source: SearchSoftwareQuality

Agile expert Lisa Crispin developed these four Agile testing quadrants as a guide for managers and development teams to use to create test strategies. It's important to realize that the Agile Testing Quadrants diagram is simply a model or taxonomy to help teams plan their testing and that there are no hard and fast rules about which tests belong in which quadrant and in which order the different tests need to be done. (For example, it's not necessary to work through the quadrants from Q1 to Q4 in a Waterfall style.)

The four quadrants are described in more detail below:

Quadrant Q1: These are technology-facing tests that guide development, such as Unit tests, API tests, Web Services testing, and Component Tests that improve product design. Tests in Q1 are often associated with automated testing and continuous integration.

Quadrant Q2: These are business-facing tests that guide development, such as those used for Functional Testing, Story Tests, Prototypes, and Simulations that make sure your software products are properly aligned with the business. Tests in Q2 are often associated with both automated and manual testing.

Quadrant Q3: These are business-facing tests used to evaluate or critique the product. Q3 covers tests such as exploratory testing, scenario-based testing, usability testing, user acceptance testing, and alpha/beta testing and can involve product demos designed to get feedback from actual users. Tests in Q3 are often associated with manual testing.

Quadrant Q4: These are technology-facing tests used to evaluate or critique the product. Q4 covers test such as performance, load, stress, and scalability tests, security tests, maintainability, memory management, compatibility and interoperability, data migration, infrastructure, and recovery testing. These tests are often automated.

The clouds at the quadrant corners signify whether tests in that quadrant generally require automation, manual testing or specialized tools. The division of tests into quadrants allows teams to strategize whether they have the right skills to accomplish each of the different types of testing, or if they have the necessary hardware, software, data and test environments. It also makes it easier to customize your agile testing process on a project-by-project or skill-by-skill basis. So, for example, if you don't have a tester on your QA team with appropriate load or performance testing skills, it helps you to see the need to bring in a contractor or outsource that particular test. A testing strategy based on the Agile Testing Quadrants requires effective workgroup communication, which is made easier by a test management solution that allows the team to work collaboratively in real-time.

Agile Methodology in Testing

In traditional waterfall testing and development, software developers typically concern themselves with three types of requirements, often addressed at different stages in a software project: Business requirements that describe why the product is being built and identify the benefits both customers and the business will reap; User requirements, which describe what tasks or business processes a user will be able to perform with the product; and Functional requirements that describe the specific system behaviors that must be implemented. In traditional waterfall development, functional requirements often reside in a software requirements specification (SRS) document, which is used by analysts to communicate detailed requirements information to developers, testers, and other project stakeholders.

User stories, a less formal approach, are used in agile development to help shift the focus on software projects from writing about software requirements to talking about them. User stories are short, simple descriptions of a feature told from the perspective of the person who wants the new capability, usually a user or customer of the system. On agile projects, user stories are the smallest units of work done by a development team and usually follow this standard user-story template:

As a {type of user}, I want {goal} so that I {receive benefit}.

Here's a simple example from a banking website application that illustrates a user story:

As a bank customer, I want to be able to check my bank account balance in real-time so that I can see if any purchases I'm about to make will result in overdraft charges.

Agile is all about short, flexible development cycles that respond quickly to customer demand. Since you are, in effect, building a continuous, two-way DevOps software pipeline between you and your customers, you should have the idea of Continuous Delivery (CD) in mind from the start as the destination for your digital transformation journey. DevOps continuous delivery is a software development strategy that optimizes your delivery process to get high-quality software into the hands of your customers as quickly as possible. Scrum and Kanban are currently the two main types of agile process frameworks for doing that. For organizations just getting their feet wet with the agile methodology in testing, here are a few differences between the two frameworks.

Scrum takes a time-boxed, incremental approach to software development and project management by advocating frequent interaction with the business during what are known as Sprints (which are called iterations in other agile frameworks). The simplest Scrum project team (as shown in the figure below) is made up of a customer/ business unit stakeholder (known as a Product Owner), the team facilitator (called a ScrumMaster) and the rest of the agile development team. Team members interact frequently with business users, write software based on requirements that they pull from a product backlog (a prioritized list of user stories maintained by the Product Owner) that they then integrate frequently with software written by other team members.

Overview of the Scrum Agile Process Framework

Image Reference: wikimedia

Scrum projects employ fixed-length sprints, each of which usually lasts from one to four weeks, after which potentially shippable code should be ready to be demonstrated. The notion of releasing a prototype, or minimum viable product (MVP), is also an important best practice in Scrum for getting early feedback from your customers. Once the MVP is released, you're then able to get feedback by tracking usage patterns, which is a way to test a product hypothesis with minimal resources right away. Every release going forward can then be measured for how well it converts into the user behaviors you want the release to achieve. The concept of a baseline MVP product that contains just enough features to solve a specific business problem also reduces wasted engineering hours and a tendency for feature creep or 'gold plating' on agile software teams.

Another Scrum best practice is for the ScrumMaster is to organize a workshop with the customer/end user and the Scrum team so they can create the Product Backlog together. This exercise will allow the Development Team to estimate the implementation effort and the customer to determine the business value (possibly by assigning business value ‘points’), which helps prioritize the Product Backlog without delegating the responsibility entirely to the Product Owner.

Traditional software teams estimate their work effort in a time format such as days, weeks or months. Many agile teams, however, have transitioned to story points, which are numbers used to measure the relative complexity of a story in terms of risk or effort involved. The Scrum best practice described in the paragraph above can be further refined by an exercise called “planning poker” in which the Development Team will take an item from the backlog, discuss it briefly, and then everyone holds up a card with the number that reflects their estimate. The goal is to help team members reach agreement on the relative difficulty of different user stories on the Product Backlog.

The Kanban agile process management framework is designed to aid decision-making about what software to produce, when to produce it, and how much to produce. Unlike the time-boxed approach that Scrum takes, Kanban is designed around a continuous queue of work, which goes through a number of stages of development until it's done. Kanban teams usually write their user stories on index cards or sticky notes that are arranged on walls, such as the Kanban Board shown below, to visualize workflow in a left-to-right manner. When work is completed in a stage, it moves into the next-stage column to its right. When someone needs new work to do, they pull it from a left-hand column.

Kanban Board

While Scrum relies on fixed time-boxes for estimating and planning, Kanban puts more emphasis on the concept of cadence, or continuous flow, that an agile team establishes by working together and reliably delivering software at a set pace. Kanban emphasize two main best practices. One involves visualizing the flow of work, which requires you to map your team's workflow stages and configuring the lanes on your Kanban board to match. Visualizing workflow enables teams to analyze the amount of work on their Kanban board, better understand how they process it, and then ultimately optimize their process by introducing incremental improvements. The second best practice is to constrain the amount of work in progress, which requires you to set work-in-progress (WIP) limits, such as only allowing five items in the In Progress lane on the Kanban board above.

Scrum versus Kanban

Every organization is unique and you should choose an agile methodology in testing that works best within your culture and the skill-sets of your development and testing teams, which can be a mix of the best features of both Scrum and Kanban. Introducing Scrum is quite a change for teams not used to agile software development: they have to start working in iterations, build cross-functional teams, appoint a product owner and a Scrum master, as well as introduce regular meetings for iteration planning, daily status updates and sprint reviews. On the other hand, Kanban is much less structured and has a looser, more free-flowing style. Kanban may be easier for your organization to adopt since it encourages incremental improvements to your existing software delivery process. You can apply Kanban principles to any process you already have in place, even Scrum.

Atlassian offers the following resources that Zephyr Squad tool users can use to better understand the two frameworks:

• “Learn Scrum with Jira Software Tutorial”: https://Jira.atlassian.com/agile/tutorials/how-to-do-scrum-with-Jira-software (Step-by-step instructions on how to drive a Scrum project)
• “Learn Kanban with Jira Software Tutorial”: https://Jira.atlassian.com/agile/tutorials/how-to-do-kanban-with-Jira-software (Step-by-step instructions on how to drive a Kanban project with Jira Software)

Agile Testing Methods

As mentioned earlier, test driven development (TDD) is an agile programming technique that requires developers, before they write any unit of code, to write an automated test for that code. Writing the automated tests is important because it forces the developer to take into account all possible inputs, errors and outputs. TDD allows an agile team to make changes to a project codebase and then quickly and efficiently test the new changes by running the automated tests. The result of using TDD is that agile teams will accumulate a comprehensive suite of unit tests that can be run at any time to provide feedback that their software is still working. If the new code breaks something and causes a test to fail, TDD also makes it easier to pinpoint the problem and fix the bug.

Behavior-Driven Development (BDD)

Behavior-driven development (BDD) is an extension of test-driven development (TDD) that encourages collaboration between developers, QA and non-technical or business participants on a software project. It extends TDD by writing test cases in a natural language called Gherkin that non-programmers and domain experts can read. BDD features are usually defined in a GIVEN WHEN and THEN (GWT) format, which is a semi-structured way of writing down test cases. A BDD feature or user story needs to follow the following structure:

• Describe who is the primary stakeholder of the feature
• What effect the stakeholder wants the feature to have
• What business value the stakeholder will derive from this effect
• Acceptance criteria or scenarios

A brief example of a BDD feature in this format looks like this:

In TDD, the developers write the tests while in BDD the automated specifications are created by users or testers (with developers writing the underlying code that implements the test.) A key BDD best practice is to use Specification by Example, a collaborative approach to define the requirements and business-oriented functional tests for software products based on capturing and illustrating requirements using realistic examples instead of abstract statements.

User stories, which we described earlier, are used in agile development to help shift the focus on software projects from writing about software requirements to talking about them. User Stories aren't formal documents in the way traditional requirements are. They’re intended to be placeholders for conversations among the stakeholders on a project in order to get agreement on acceptance criteria for a particular piece of functionality. Another best practice to encourage this kind of conversation about functionality is to use "Three Amigos" meetings that involve a product owner (or a business analyst), a developer and a QA tester, who get together (either face-to-face or on-line) to review the requirements, tests and dependencies of a feature request on the backlog.

Use Three Amigos meetings to define and refine BDD User Stories

Image Source: BDD in Action

Fuel your software testing program using BDD

Download Behavior Driven Development Whitepaper

Download

Acceptance Test Driven Development

Acceptance test driven development (ATDD) is another enhancement of test-driven development that promotes collaboration between business users, testers, and developers to define automated acceptance criteria before coding has begun. ATDD and TDD are complementary techniques: ATDD helps describe the high-level business objectives, while TDD helps developers implement them as requirements. ATDD helps ensure that all project members understand what is being implemented since failing ATDD tests provide quick feedback that requirements are not being met.

Although they don’t have to be written in the Gherkin language used on BDD projects, User Stories on ATDD projects should be well-defined. A best practice for doing this in a "Three Amigos" meeting is to follow the "3 C's" formula, devised by Ron Jeffries, that captures the components of a User Story:

1. Card – stories are traditionally written on notecards, and these cards can be annotated with extra details
2. Conversation – details behind the story come out through conversations with the Product Owner
3. Confirmation – acceptance tests confirm the story is finished and working as intended

A key part of ATDD tests is that they are run automatically whenever a change is made to the source code. In addition to testing the application, automated acceptance tests are useful in measuring the progress your project team is making since, on an agile project, working software is considered to be the only objective measure of progress.

Exploratory Testing

Exploratory tests are tests where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests. Exploratory testing is done in a more freestyle fashion than scripted testing, where test cases are designed in advance. Exploratory testing is important on Agile projects since developers and other team members often get used to following a definite process (such as the those dictated by TDD, BDD or ATDD testing approaches) and can stop thinking outside the box. Because of the desire for fast consensus among self-organizing Agile teams (even globally distributed ones), collaboration can devolve into groupthink. Exploratory testing combats this tendency by allowing a team member to play the "devil's advocate" role and ask tough, ‘what if’-type testing questions.

An example of exploratory testing on an on-line shopping website would test the following functionality from the perspective of an end-user (in any order the tester chooses): Login, List of Products, Loading and Unloading of a Shopping Cart, Payment Processing, Order History and Product Shipment.

Exploratory testing is especially useful in complex testing situations, when little is known about the product, or as part of preparing a set of scripted tests. The complexity of modern applications makes it difficult for prescribed manual or automated tests to find all of the edge cases needed to fully capture your end user’s experience. Exploratory testing excels in allowing non-testers to go through an application and provide clear, captured, annotated data for your team to replicate any issues that are found.

Because exploratory testing is a time-consuming process, a best practice requires the Project Owner to budget enough time so that the tester can understand the business requirements and the purpose of the app being tested. Testers should also use testing tools and other resources that can help document any bug found in a clear and concise manner so that developers can recreate and fix the bug. Exploratory testing is best done in a collaborative environment where testers work closely with developers.

Session-Based Testing

Session-based testing is a type of structured exploratory testing that requires testers to identify test objectives and focus their testing efforts on fulfilling them. Test sessions differ from test cases in two ways: First, more than one test can be carried out in a single session. Second, like test cases, test sessions can tell you who tested what, but they can also tell you how, when and why the tests were done.

Image Source

Pre-written test scripts can be run during a session, but because test sessions emphasize test objectives over specific test cases, testers are encouraged to create and execute more tests based on what they've discovered and learned. This type of exploratory testing is an extremely powerful way of optimizing test coverage without incurring the costs associated with writing and maintaining test cases.

Formal Session-Based Testing uses a charter, which a goal or agenda for a test session that is created by the test team prior to the start of testing from a specification, test plan, or by looking at the results from previous sessions. An example of a charter for session-based testing of the shopping cart application (the example used in the Exploratory Testing section above) would be: “Test adding items to the shopping cart.” Since a session is an uninterrupted period of time spent testing, typically lasting one to two hours. session-based testing is an ideal way to test the performance and functional of your apps, i.e. how long it takes to add or remove items from the shopping cart or the speed and accuracy of calculations performed.

Best practices for Session-Based Testing include setting aside uninterrupted time for a session and limiting the session to just one charter (charters should be prioritized and can be broken down into smaller logical units that can be tested in separate sessions). It’s also important to do a session debriefing, which is a meeting between test leads and the testers who completed the session. Debriefing topics should include what happened during the session, what was achieved, challenges faced, pending issues and feedback from the tester to improve future sessions.

Agile Testing Tools

Test Automation Pyramid

Outside-in versus Inside-out Testing

Agile teams generally follow one of two approaches when it comes to testing their applications, either outside-In or inside-out. In the outside-in approach, teams start by focusing on the end user's perspective and attempt to describe high-level desired functionality and goals for the software under test in the form of user stories. In every iteration or Sprint, user stories are refined until the agile team and the Product Owner/Customer Representative can agree on the acceptance criteria, which determine that a User Story works as planned. Testing then goes 'inward' and code is written to test smaller and small components until you reach the unit-test level.

In the inside-out or bottom-up approach, agile teams start with unit tests using JUnit or other unit testing frameworks (collectively known as xUnit) at the lowest level of the Test Automation Pyramid (see figure above). As the code evolves due to refactoring, testing efforts evolve as well as the team moves upward to acceptance level testing, which tests business logic at the API or service level using tools like SoapUI, Fitnesse or Cucumber [Cucumber runs automated acceptance tests written in the behavior-driven development (BDD) style described above.] The top of the pyramid and the last thing tested is the user interface (UI), which can be tested using tools like TestComplete or Selenium.

Inside-out and outside-in are different but complementary approaches to testing. Software quality control relies on the related notions of verification and validation (V&V) that check to see that a software system meets specifications and that it fulfills its intended purpose. The terms verification and validation are often used interchangeably but have different meanings:

• Verification: Checks the software with respect to specifications ("Is our team building the code right?")
• Validation: Checks the software with respect to customer's expectations ("Are we building the right code?")

On modern agile testing projects, validation and verification steps overlap and take place continuously since agile team members must engage with customers and other stakeholders throughout the project-- to do things like prioritizing bug fixes and enhancements on the team's project backlog --and not just after a separate test phase at the end of the project. This requires effective communication at all levels of the business since team members need to understand what features need to be built and who needs each feature.

Limitations of Spreadsheets

In addition to specialized tools for agile testing, agile teams can benefit from test management or test case management tool that can help manage their testing efforts, record test results and generate reports that provide useful feedback to testers, developers and project leads. Many agile teams often rely on a spreadsheet application like Microsoft Excel as a test case management, documentation and reporting tool. There are significant risks to using spreadsheets to store and process test cases, however, especially on multi-team projects where individual teams often adapt spreadsheets to their specific needs, which can cause problems when it comes to getting uniform reports. If two or more people are working at the same time on a spreadsheet file, there's also the danger of corrupting the file or creating other security risks. This is one of the many reasons that, as agile teams and projects get larger, they’re likely to switch to a test management tool that provides a central repository for all their testing efforts.

Test Management Tools

There are a variety of test management tools available for different needs, with both cloud-based or on-premise installation options. Some tools like Zephyr (Standalone Edition) and HipTest come with integrated issue and requirements tracking, while other test case management tools are integrated with popular issue tracking tools such as Atlassian’s Jira or Bugzilla (both Zephyr and HipTest provide Jira plug-ins).

Agile Test Automation

Automation testing works on agile projects by running a large number of tests repeatedly to make sure an application doesn’t break whenever new changes are introduced—at the Unit-. API- and GUI-level. For many agile development teams, these automated tests are executed as part of a Continuous Integration (CI) build process, where developers check code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect errors and conflicts as soon as possible. CI tools such as Jenkins, Bamboo, and Selenium are also used to build, test and deploy applications automatically when requirements change in order to speed up the release process.

Test automation allows agile teams to execute more tests in less time, increasing coverage and freeing human testers to do more high-level, exploratory testing. Since automation test scripts are reusable, they can be used to do more comprehensive testing by testing repetitive steps with different data sets, such as those for cross-browser or cross-device compatibility.

Among the risks of automation are those related to the need for version control and maintainability of test scripts and test results. Choosing the right automation testing tool is critically important since you want to avoid ones that are incompatible with other software testing tools in your test environment. Once you’ve got a test automation tool that works well with your other testing tools, automated tests should also be run on a regular basis to provide continuous feedback about the health of your entire system, preferably through a continuous integration approach described above, as opposed to manually.

Guide Your Agile Testing with Zephyr

As noted earlier, the testing pyramid is a popular strategy guide that agile teams often use in planning their test automation strategy. The base or largest section of the pyramid is made up of Unit Tests--which will be the case if developers in your organization are integrating code into a shared repository several times a day. This involves running unit tests, component tests (unit tests that touch the filesystem or database), and a variety of acceptance and integration tests on every check-in.

At the top or eye of the pyramid are the last tests that should be considered for automation, the manual exploratory tests, which are tests where the tester actively controls the design of the tests as those tests are performed and use information gained while testing to design new and better tests. Exploratory testing is done in a more freestyle fashion than scripted automated testing, where test cases are designed in advance. With modern test management software, however, it's possible to semi-automate these kinds of tests, which entails recording and playing back the test path taken by an exploratory tester during a testing session. This helps other agile team members recreate the defect and fix the bug.

Capture for Jira helps software testers on agile projects create and record exploratory and collaborative testing sessions, which are useful for planning, executing and tracking manual or exploratory testing. As described earlier, session-based test management is a type of structured exploratory testing that requires testers to identify test objectives and focus their testing efforts on fulfilling them. Because session-based agile testing gives testers the flexibility to respond to changes and re-plan their testing quickly, it’s a powerful way to optimize agile testing coverage without incurring the costs associated with writing and maintaining test cases. Like Zephyr Squad, Capture for Jira has a deep integration with the Jira platform, allowing users to capture screenshots within browsers, create annotations, and validate application functionality within Jira.

The Best Agile Test Management Tool

The best agile test management tool is one that enables agile teams to work collaboratively in the different areas described above--Continuous Build, Continuous Integration, Continuous Delivery, and Continuous Deployment-- in order to speed up the release of high-quality software. Doing that effectively means it should be able to integrate with other project management, issue tracking and automation tools in your agile development and testing toolchain. It also should have live reporting features since you need to maintain real-time visibility into the products in your software delivery pipeline. This is important so that information about bugs, inefficiencies or other issues can be shared and acted on in real-time. If you're a small or medium-sized business, it also helps to choose a tool that can scale to fit the needs of an enterprise or very large enterprise team.

About SmartBear

SmartBear is behind the software that empowers developers, testers, and operations engineers at over 20,000 of the world’s most innovative organizations including Adobe, JetBlue, MasterCard, and Microsoft. More than 6 million people use our tools to build, test, and monitor great software, faster. Our high-impact tools are easy to try, easy to buy, and easy to use. These tools are backed by a team of people passionate about helping you create software that transforms our world. Those tools are SmartBear tools. That team is SmartBear. For more information, visit: http://smartbear.com, or follow us on LinkedIn, Twitter, or Facebook.