The Role of System Testing in Automated Infrastructure
Test and Monitor | Posted August 19, 2016

In today's world of delivering software, the automation of infrastructure has become a key component in providing consistency and reliability. The ease by which we can "spin-up" tens to thousands of virtual machines hides the fact that there is significant complexity in infrastructure that needs to be considered.

Complexity in virtualization, networking, security, storage and other technology require appropriate level of expertise in proper configuration. Below is a small sample of the various configurations a simple Spring-Boot service may employ for infrastructure using Docker for AWS:

  • IAM policies
  • VPC(s)
  • Elastic Load Balancer(s)
  • Subnets
  • Security Groups
  • Route 53 Hosted Zones
  • EC2 AMI(s)
  • Network Interfaces
  • Autoscale Groups
  • Docker images / DockerHub
  • Volumes
  • RDS Instance(s)

How do we know if we have the configuration right?

Consider system testing. System testing provides an effective means to verify that the software and its supporting environment behaves as expected to your functional and non-functional requirements.

What does system testing look like in action?

One example we performed for a client is simulating the effect of cluster members stopping during production load. Using a fully integrated staging environment we measured the impact on client latency, the count of HTTP errors the client received, and server performance metrics.

These were collected while the cluster was loaded and members were removed and then reintroduced. Simulation of the load was done through LoadUI and client metrics were collected using SoapUI Assertions. The result of this test helped us understand the volume the cluster could support as we reduced available members. It also confirmed that the addition of new members occurred smoothly with no impact to the client or other cluster members.

System Testing of Automated Infrastructure

Organizations that have leveraged automated infrastructure in cloud providers such as Amazon Web Services, Microsoft Azure or Google Compute test to make sure their infrastructure configuration supports resiliency. Netflix has led the effort in developing open source tooling to kill / disable infrastructure (e.g. Chaos Monkey). These outages will often test features such as:

  • Container self-healing
  • Auto-scaling groups
  • DNS Failover
  • Elastic Load Balancing
  • Multi-AZ failover
  • Multi-Region failover

Combined with the power of an open source API testing tool like SoapUI, or an API testing framework like Ready! API, you can simulate system level outages and develop a deeper understanding of how infrastructure behaves.

The Beginner’s Guide to Agile API Development

I recently had the opportunity to partner with SmartBear Software on a new educational resource: The Beginner’s Guide to Agile API Development: A Look at DevOps, CI, and CD in API DeliveryIn addition to sharing best practices for system testing, I also look at how selecting and implementing the right tools can transform your API delivery strategy.

We look at:

  • Shifts in the API development landscape, and the challenges and opportunities they present to organizations seeking to evolve
  • What is DevOps? How are API teams structured in DevOps organizations?
  • What are Continuous Integration/Continuous Delivery in the context of DevOps? How can you determine which methodology is most suitable for your API team?
  • Best practices for modern software delivery, and a thorough explanation of the tooling that goes along with it

Interested in learning more? You can download the resource today for free.