Developing a DevOps Testing Strategy: Benefits, Best Practices & Tools

 

DevOps is all about streamlining and automating your entire software delivery lifecycle. For most organizations, a successful DevOps testing strategy starts with adopting the agile best practice of Continuous Integration (CI), where developers check code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect errors and conflicts as soon as possible. Automation frameworks and CI tools such as Jenkins and Bamboo are also used to build, test and deploy applications automatically when requirements change in order to speed up the release process.

What Is DevOps?

The goal of DevOps is to build better, faster and more responsive software by bringing Development and Operations teams together. DevOps is not a methodology or a suite of tools but a cultural shift to remove the barriers between Dev and Ops in order to meet the need for shorter and more frequent software deliveries. This will allow your organization to respond in a more agile manner to changing business requirements. The DevOps cultural shift depends on continuously optimizing workflow, architecture, and infrastructure in order to deliver high-quality applications.

{^widget|(buttontext)Download|(headline)Stay+ahead+of+the+latest+testing+trends|(buttonurl)https%3a%2f%2fpost.getzephyr.com%2febook-how-the-world-tests-2018%3fcta%3ddevops-core-content|(name)InlineBlogAd|(backgroundcolor)White|(content)Download+Our+Annual+Testing+Report|(richtextcheckbox)False|(widget_displayname)Inline+Blog+Ad|(width)|(height)^}

From the Agile Model to the DevOps Model

Software developed with the Agile Model adheres to the following four basic priorities stated in The Agile Manifesto:

1. Individuals and interactions over processes and tools;
2. Working software over comprehensive documentation;
3. Customer collaboration over contract negotiation; and
4. Responding to change over following a plan.

Agile development takes a test-first approach, rather than the test-at-the-end approach of traditional development. In agile testing, code is developed and tested in small increments of functionality. Agile project management promotes frequent interaction between an IT department and business users and tries to regularly build and deliver software that meets business users’ changing requirements. The flexible approach of the Agile Model helps to ensure a streamlined software development process that responds quickly to customer demand in a wide variety of use cases.

Agile software development can be implemented with a number of methodologies, including Scrum, Kanban, Scrumban (a mix of Scrum and Kanban), extreme programming, and many more. In the Scrum methodology, the development effort is broken down into a series of short ‘sprints,’ or iterations, typically lasting 2 to 4 weeks. The goal of each sprint is to create potentially shippable software by adding new or enhancing existing application features.

The DevOps Model is the ‘Agile Model on steroids,’ which extends the cross-functional Agile team made up of software designers, testers and developers to include support people from Operations. The focus of this expanded agile team is the overall service or software your company delivers to its customers, instead of just “working software” (the 2nd of the 4 Agile principles listed above).

Building a successful continuous, two-way DevOps software pipeline between your company and your customers means creating a DevOps culture of collaboration among the various teams involved in software delivery (developers, operations, quality assurance, business analysts, management, and so forth), as well as reducing the cost, time, and risk of delivering software changes by allowing for more incremental updates to applications in production. In practice, this means teams produce software in short cycles, ensuring that the software can be reliably released at any time. Where regular agile teams strive to deliver shippable software in 2 to 4 weeks, the ideal DevOps goal is to deliver code to production daily or every few hours.

{^widget|(buttontext)Download|(headline)Why+is+shifting+left+is+critical+to+agile+DevOps%3f|(buttonurl)https%3a%2f%2fpost.getzephyr.com%2finfographic-shift-left-devops%3fcta%3ddevops-core-content|(name)InlineBlogAd|(backgroundcolor)White|(content)Download+DevOps+Infographic|(richtextcheckbox)False|(widget_displayname)Inline+Blog+Ad|(width)|(height)^}

Finding the Right DevOps Testing Strategy

The aim of most DevOps projects is to automate as many manual processes in their software delivery process as possible. Some roadblocks in a DevOps pipeline that lead to slow deployment include error-prone manual processes such as handoffs from a development group to a QA group, including ones that require signatures or bureaucratic approval. These kinds of handoffs mean there is a lack of shared ownership of the end product, which is contrary to the basic agile testing and development methodology that says all members of a cross-functional agile team are equally responsible for the quality of the product or the success of the project. Because of this, testing on an agile DevOps project is done by the whole team, including developers, testers and operations people. In the DevOps model, testing is done continuously—early, often and even after applications are in production.

Continuous Delivery vs. Continuous Deployment vs. Continuous Integration

Pros and Cons of DevOps Continuous Testing:

Continuous Testing needs to be a key element your DevOps testing strategy if you want to successfully implement a DevOps pipeline. Continuous Testing, which is often called shift-left testing, is an approach to software and system testing in which testing is performed earlier in the software lifecycle, with the goal of increasing quality, shortening long test cycles and reducing the possibility of software defects making their way into production code.

Continuous Testing is one of several continuous activities that should take place simultaneously in a DevOps pipeline, including

• Continuous Development;
• Continuous Integration (CI);
• Continuous Delivery (CD); and
• Continuous Monitoring

Continuous Development or build automation is the first stage in implementing a Continuous Delivery DevOps pipeline. If your developers are practicing test-driven development (TDD), they’ll write unit tests for each piece of code they write, even before the code itself is written. An important part of the agile methodology, TDD helps developers think through the desired behavior of each unit of software they’re building, including inputs, outputs, and error conditions. New features implemented by developers are then checked into a central code base prior to the software build, which compiles the source code into binary code.

Continuous Integration is a practice where members of a software development team use a version control system and integrate their work frequently to the same location, such as a master branch. Each change is built and verified by means of tests and other verifications in order to detect any integration errors as quickly as possible. With build automation, the software build happens automatically, using tools such as Makefiles or Ant, rather than when a developer manually invokes the compiler.

In the last stage of a Continuous Delivery pipeline, once an application passes all the required tests, it’s then released into production. For all intents and purposes, this means releasing every good build to users. The upside of Continuous Delivery is that it allows delivery of new functionality to users within minutes whenever it’s needed, as well as instant feedback to the DevOps testing team that, in turn, allows them to respond rapidly to customer demand. A big CD downside is a risk of releasing buggy software into production faster, which will alienate end-users and cause your company to lose customers. To mitigate the risk involved with continuous deployment, many DevOps teams also test in production, otherwise known as shift-right testing, in order to obtain user feedback and incorporate it into successive builds. This last phase is often aided by Continuous Monitoring tools (more on this below).

Multiple teams work together as a DevOps team, so DevOps has an impact on not only QA but also developers, operations, business analysts, production support, etc. Testing is a critical element in promoting cohesiveness and collaboration between the various teams, automation of everything possible. Testing also allows for continuous improvement in the test process delivered via iterations.

Developing Your DevOps Testing Strategy

A successful DevOps testing strategy is one aimed at building, testing and releasing software faster and more frequently. If you’re lucky enough to start out in a “greenfield” organization without an established coding culture, it’s a good idea to try to create and automate your software delivery pipeline upfront. If you’re successful out-of-the-gate in creating a Continuous Delivery DevOps pipeline, your business will be much more competitive since you’ll be able to get higher-quality software into the hands of your users and customers faster than your competitors, and you’ll be able to react to business demand and change much more rapidly.

If you are adding a DevOps pipeline to an existing organization, where you start depends on your current development and testing practices and the bottlenecks in your software delivery process. These bottlenecks can include slow, error-prone manual processes as well as poor-quality, big-bang rollouts that fail in production, leading to unhappy users.

There are several ways to get a handle on the current state of your deployment processes, including using workflow visualization tools like flowcharts and business process maps to breakdown and understand your current delivery processes. One of the simplest visual process management tools you can use to help make these kinds of decisions is a Kanban board.

Kanban boards, like the one pictured below, are typically just sticky notes on a whiteboard that are used to communicate project status, progress, and other issues.

The main advantage of using Kanban in your DevOps roadmap is that it encourages your teams to focus on improving workflow. It does this by highlighting bottlenecks in the pipeline that cause work to slow work down or otherwise impact the performance indicators your entire DevOps team has mutually agreed on–such as application time-to-market, systems uptime, code quality, customer satisfaction, and so on.

Kanban also helps your Dev and Ops people to work in a collaborative manner, which they may not be used to doing. Once they start talking about how to eliminate duplicate work and how to optimize your company’s release management processes, you’ll likely see an evolution over a period of time into a single team and single workflow that includes all of Dev and Ops activities–and a transformation to a DevOps culture.

It’s also likely that developers and testers on your DevOps teams will have some familiarity with Kanban if they’ve been using Scrum or other Agile methodologies, and they’ll be able to help your Operations people get up to speed with the concept of flow, which basically is what Kanban is all about.

[Atlassian offers the following tutorial that Zephyr for Jira tool users can use to better understand Kanban: “Learn Kanban with Jira Software Tutorial”: https://www.atlassian.com/agile/tutorials/how-to-do-kanban-with-jira-software (Step-by-step instructions on how to drive a Kanban project with Jira Software)]

Best Practices for Continuous Testing

Automate as Many Tests as You Can

Continuous Testing means testing early and often. This is obviously a major challenge for your DevOps team if the applications in your pipeline are constantly being updated via Continuous Integration, where the code is integrated into a shared repository on a constant basis. A best practice is to use test automation to eliminate much of the risk that comes with CI and to get quick feedback on application quality. Pairing CI with automation enables teams to easily test every new code iteration and reduces the opportunity for team members to add errors. Use the Test Automation Pyramid below as a strategy guide to planning your DevOps testing strategy. As shown in the illustration, the base or largest section of the pyramid is made up of Unit Tests–which will be the case if developers in your organization are integrating code into a shared repository several times a day. This involves running unit tests, component tests (unit tests that touch the filesystem or database), and a variety of acceptance and integration tests on every check-in.

Use Pair Testing on Tests You Don’t Automate

Knowledge transfer via pair testing is another best practice and one of the most efficient ways to train new members and maintain DevOps pipeline velocity, assuming your team has senior testers or developers who are willing to share their knowledge and time with new team members. Pair testing is similar to pair programming, which is an agile software development technique in which two programmers work together at one workstation. In pair testing, two team members work together at one keyboard to test the software application. One does the testing and the other analyzes or reviews the test results. This can be done between a tester and a developer or a business analyst or between two testers with both participants taking turns at driving the keyboard. Pair testing is useful in breaking down communication barriers between developers, QA testers and application/platform operators and creating interdependency between people in functional silos– thus building strong workplace alliances.

{^widget|(buttontext)Watch+Webinar|(headline)Future-proof+your+DevOps+testing+strategy|(buttonurl)https%3a%2f%2fpost.getzephyr.com%2fvideo-2020-vision-for-devops-and-ai-webinar%3fcta%3ddevops-core-content|(name)InlineBlogAd|(backgroundcolor)White|(content)Watch+DevOps+and+AI+Webinar|(richtextcheckbox)False|(widget_displayname)Inline+Blog+Ad|(width)|(height)^}

Who is Involved in DevOps Testing?

Testers on DevOps teams need to leverage best practices in agile testing, continuous integration, and test-driven development to accelerate their test QA processes and reduce cycle time. A fundamental aspect of DevOps pipelines is the notion of shared responsibility for attaining the highest quality. On agile DevOps teams, everyone is equally responsible for the quality of the product or the success of the project. This means testing is done by the whole team, not just designated testers or quality assurance professionals, including team members whose primary expertise may be in programming, business analysis, database or system administration. Team members whose expertise is in DevOps testing or using a particular testing tool aren’t limited to only doing that activity–they can also collaborate with customers or business owners on product requirements and work with other members of the team to develop high-quality code that fulfills those requirements.

DevOps QA testers need to have flexible working skills that allow them to move comfortably from one area of the DevOps software pipeline to another, whether it’s development, integration, testing, releasing or deployment. Starting with an understanding of the people, the culture, and how the organization runs, it also helps if they have strong communications skills since the DevOps testing strategy they’ve built is focused on simplifying the overall development, testing and operating environment to achieve the goal of continuous delivery.

Among the technical skills DevOps QA testers need is a broad understanding of some of the following tools and technologies (see the DevOps Tool Section below for more detail):

• Source Control (using Git, Bitbucket, Svn, VSTS etc)
• Continuous Integration (using Jenkins, Bamboo, VSTS )
• Infrastructure Automation (using Puppet, Chef, Ansible)
• Deployment Automation & Orchestration (using Jenkins, VSTS, Octopus Deploy)
• Container Concepts (LXD, Docker)
• Orchestration (Kubernetes, Mesos, Swarm)
• Cloud (using AWS, Azure, GoogleCloud, Openstack)

Another useful skill is the ability to write scripting code in languages like JavaScript, Python, Perl, or Ruby in order to implement automation technologies and tools at any level, from requirements to development to testing and operations.

{^widget|(buttontext)Download|(headline)Prepare+now+for+the+future+of+testing|(buttonurl)https%3a%2f%2fpost.getzephyr.com%2febook-continuous-testing-2020%3fcta%3ddevops-core-content|(name)InlineBlogAd|(backgroundcolor)White|(content)Download+Continuous+Testing+Agility+Whitepaper|(richtextcheckbox)False|(widget_displayname)Inline+Blog+Ad|(width)|(height)^}

DevOps Tools

As mentioned earlier, software applications pass through five different stages when developed in a DevOps pipeline:

1. Continuous Development
2. Continuous Testing
3. Continuous Integration
4. Continuous Delivery
5. Continuous Monitoring

Because DevOps is a cultural shift that promotes collaboration between development, QA and operations, there is no one single product that can be considered the definitive DevOps tool. Often a collection of tools, from a variety of vendors, are used in one or more stages of the DevOps toolchain. The DevOps tools used in each stage will vary from organization to organization, based on tools already in place, the skill level of your developers and testers, and the types of applications you’re deploying.

Here is a short list of some the most common DevOps tools and an explanation of how they’re used in each stage:

Continuous Development with Jira

The de facto standard for the agile DevOps software development and testing, Jira Software has powerful collaborative functionality that visually highlights issues as they move through the project workflow, which makes the Jira platform easy to use for planning development, tracking daily work and reporting on project progress.

Continuous Testing with Zephyr for Jira

Although Jira Software is designed for issue, project and workflow tracking on IT projects, many DevOps teams are also using it for test case management so that development and testing teams can work together in one system. Test issues can be created, executed, tracked and reported on just like any other Jira issue in Zephyr for Jira, which has the same look n’ feel of Jira.

Continuous Integration with Jenkins

Jenkins is a CI/CD server that runs tests automatically every time a developer pushes new code into the source repository. Because CI detects bugs early on in development, bugs are typically smaller, less complex and easier to resolve. Originally created to be a build automation tool for Java applications, Jenkins has since evolved into a multi-faceted platform with over a thousand plug-ins for other software tools. Because of the rich ecosystem of plug-ins, Jenkins can be used to build, deploy and automate almost any software project–regardless of the computer language, database or version control system used.

Continuous Delivery/Deployment with Puppet or Chef

Automated configuration tools such as Puppet or Chef allow you to avoid the problem of “snowflake servers” in your delivery/deployment environment. “Snowflake servers” are long-running production servers that have been repeatedly reconfigured and modified over time with operating system upgrades, kernel patches, and/or system updates to fix security vulnerabilities. This leads to a server that is unique and distinct, like a snowflake, which requires manual configuration above and beyond that covered by automated deployment scripts.

Using definition files (called manifests in Puppet, or recipes in Chef) to describe the configuration of the elements of a server is a key best practice in Infrastructure as Code, which allows environments (machines, network devices, operating systems, middleware, etc.) to be configured and specified in a fully automatable format rather than by physical hardware configuration or the use of interactive configuration tools.

Continuous Monitoring with Splunk or Elk

Both Spunk and Elk are log monitoring tools that allow you to do data analysis on transactions that take place in your IT applications after they’ve been deployed to ensure uniform performance, availability, security, and user experience. Both Splunk and ELK supply a scalable way to collect and index log files and provide a search interface for users to interact with the data in order to create visualizations such as reports, dashboards or alerts.

Splunk is a paid service that bills according to the volume of your transactions. The ELK Stack is a set of three open-source products—Elasticsearch, Logstash, and Kibana—all developed and maintained by Elastic.

DevOps Testing Tools

Using some or all of the following DevOps testing tools provides numerous benefits to DevOps teams– including improving your teams’ code quality, accelerating the time-to-market of software in your DevOps pipeline and providing fast and continuous feedback that will increase collaboration among development, testing, and operations teams.

The first tests that should be done in a DevOps pipeline are unit tests that developers write as part of the test-driven development (TDD) process. If your developers are practicing TDD, they’ll write unit tests for each piece of code they write, even before the code itself is written. An important part of the agile methodology, TDD helps developers think through the desired behavior of each unit of software they’re building, including inputs, outputs and error conditions.

TDD tools include JUnit, one of a family of open source unit testing frameworks that are collectively known as xUnit. Used by Java developers to write repeatable cases that increase programming speed and improve code quality, JUnit provides features such as fixtures, test suites, JUnit classes, and test runners. TestNG is an enhanced version of JUnit that covers a wider range of test categories: unit, functional, end-to-end, integration, etc.

In addition to doing unit and component testing, many agile DevOps teams also practice test-first approaches such as acceptance test-driven development (ATDD) and behavior-driven development (BDD) for tests higher up the Test Automation Pyramid (described above). This allows testing to be repeated in increments as software components are assembled upon each other.

Cucumber is a tool based on behavior-driven development that encourages a culture of collaboration between developers, QA and non-technical or business participants on DevOps and other software projects. BDD bridges the communication gap between business and IT by writing test cases in a natural language that non-programmers and domain experts can read. BDD features are usually defined in a GIVEN WHEN and THEN (GWT) format, which is a semi-structured way of writing test cases.

DevOps Test Automation

Choosing the right DevOps test automation tool is vitally important for your organization since the right software testing tool in the right hands can foster team collaboration, drive down costs, shorten release cycles, and provide real-time visibility into the status and quality of applications in your DevOps pipeline.

Test automation works by running a large number of tests repeatedly to make sure an application doesn’t break whenever new changes are introduced—at the different Unit-, API- and GUI-levels of the Test Automation Pyramid. In addition to Continuous Integration tools such as Jenkins described above, here are some other useful tools for building, testing and deploying applications automatically when requirements change in order to speed up the release velocity of your pipeline apps.

{^widget|(buttontext)Download|(headline)Leverage+automation+to+speed+your+DevOps+delivery|(buttonurl)https%3a%2f%2fpost.getzephyr.com%2febook-test-automation-speed-devops%3fcta%3dautomation-core-content|(name)InlineBlogAd|(backgroundcolor)White|(content)Download+Test+Automation+Framework+Whitepaper|(richtextcheckbox)False|(widget_displayname)Inline+Blog+Ad|(width)|(height)^}

Bamboo

Bamboo is a CI/CD server from Atlassian. Like Jenkins and other CI/CD servers, Bamboo allows developers to automatically build, integrate, test and deploy source code. Bamboo is closely connected with other Atlassian tools such as Jira for project management and HipChat for team communication. Unlike Jenkins, which is a free and open source agile automation tool, Bamboo is commercial software that is integrated (and supported) out of the box with other Atlassian products such as Bitbucket, Jira, and Confluence.

Selenium

Selenium is a suite of different open-source software tools used for automated software testing of web applications across various browsers/platforms. Most often used to create robust, browser-based regression automation suites and tests, Selenium–like Jenkins–has a rich repository of open source tools that are useful for different kinds of automation problems. With support for programming languages like C#, Java, JavaScript, Python, Ruby, .Net, Perl, PHP, etc., Selenium can be used to write automation scripts that run against most modern web browsers.

TestComplete

TestComplete has a powerful and comprehensive set of features for web, mobile, and desktop application testing. In addition to having an easy-to-use record and playback feature, TestComplete allows testers to use JavaScript, VBScript, Python, or C++Script to write test scripts. The tool also an object recognition engine that can accurately detect dynamic user interface elements, which makes it especially useful in applications that have dynamic and frequently changing user interfaces. Since it’s a SmartBear product, TestComplete can be integrated easily with other products offered by SmartBear.

SoapUI

SoapUI is a test automation tool for functional testing, web services testing, security testing, and load testing. Specifically designed for API testing, SoapUI supports both REST and SOAP services. SoapUI provides drag and drops options for creating test suites, test steps and test requests to build complex test scenarios without having to write scripts.

{^widget|(buttontext)Download|(headline)Choose+the+right+testing+solution+the+first+time|(buttonurl)https%3a%2f%2fpost.getzephyr.com%2febook-test-management-solution-buyers-guide%3fcta%3ddevops-core-content|(name)InlineBlogAd|(backgroundcolor)White|(content)Download+Test+Management+Buyer%27s+Guide|(richtextcheckbox)False|(widget_displayname)Inline+Blog+Ad|(width)|(height)^}

Establish A DevOps Testing Strategy that Works

Since both speed and quality are at the heart of the DevOps approach, QA testing needs to be a key part of any DevOps pipeline. Zephyr fulfills this critical need by providing the right set of QA tools to perform functional, integration and performance testing, that are essential to validating DevOps quality, feature and infrastructure changes, and ensure that every code check-in is production ready.

The right test management tool for DevOps is one that enables agile teams to work collaboratively in the different areas described above–automated build, automated testing and automated provisioning of infrastructure for deployment– in order to speed up the release of high-quality software. Doing that effectively means it should be able to integrate with other project management, issue tracking, and automation tools in your DevOps toolchain.

Both Zephyr Standalone and Zephyr for Jira are advanced test management tools that offer end-to-end traceability across both manual and automated test cases, Jira and an array of CI/CD tools. This traceability will allow you to gain visibility over your entire development and QA cycle, and better manage your testers and other users in a collaborative environment where development, testing, and operations teams all work together to meet frequent iterations, releases, and delivery goals. Zephyr makes it that much easier for DevOps teams to realize the benefits of an integrated tool ecosystem since both tools are fully integrated with all of the best-of-breed agile and DevOps systems—such as Atlassian’s Jira, Confluence, Crowd and LDAP; Continuous Integration tools like Jenkins, Bamboo; and Automation tools like Selenium, QTP/UFT, and so forth.

The latest Zephyr release also makes it easier to do centralized Test Case management by allowing Projects and Releases to share and copy Test Cases from the Global Test Case Repository.

You’re able to create, edit and delete Test Cases at the Project Level as well as share and copy test cases among releases within the same project or across other projects. In addition, Zephyr gives users a release-specific Requirements View, which allows a user to manipulate requirements in one release without impacting other releases. This is especially useful if your company is using Jira to manage your agile projects and/or requirements since Zephyr’s multi-Jira capability enables organizations to configure Jira instances at both global or individual project levels, based on their requirements. Users are able to perform Jira requirements synchronization within Zephyr that is release specific. This makes it easier to do consolidated reporting on quality initiatives across multiple projects in your DevOps pipeline since you’re better able to track Test Case usage across different projects.

Last but by no means least, both tools offer live reporting features that you need to maintain real-time visibility into the products in your DevOps pipeline. This is important so that information about bugs, inefficiencies or other issues can be shared and acted on in real-time.

For a tutorial on Zephyr testing solutions, visit https://www.guru99.com/zephyr-enterprise.html

{^widget|(buttontext)Watch+Demo|(headline)See+why+18%2c000%2b+Teams+rely+on+Zephyr+for+Jira|(buttonurl)https%3a%2f%2fpost.getzephyr.com%2fvideo-instant-demo-zfj%3fcta%3ddevops-core-content|(name)InlineBlogAd|(backgroundcolor)White|(content)Watch+the+Zephyr+for+Jira+Demo|(richtextcheckbox)False|(widget_displayname)Inline+Blog+Ad|(width)|(height)^}

About SmartBear Software

SmartBear is behind the software that empowers developers, testers, and operations engineers at over 20,000 of the world’s most innovative organizations including Adobe, JetBlue, MasterCard, and Microsoft. More than 6 million people use our tools to build, test, and monitor great software, faster. Our high-impact tools are easy to try, easy to buy, and easy to use. These tools are backed by a team of people passionate about helping you create software that transforms our world. Those tools are SmartBear tools. That team is SmartBear. For more information, visit: https://smartbear.com/, or follow us on LinkedIn, Twitter or Facebook.