Back to Software Testing Practices

Case Study: Blue Ridge Networks

AppGuard by Blue Ridge Leverages SmartBear Expertise to Validate Scalability to Support Millions of Concurrent Users

About Blue Ridge Networks

Headquartered in Virginia, Blue Ridge Networks has offered field-proven, reliable, scalable, certified, easy-to-use, and affordable cybersecurity products and services to government, enterprises and consumers for more than 15 years. Blue Ridge’s comprehensive solution set includes AppGuard®, a patented next-generation antivirus protection that stops zero-day malware. AppGuard uses a unique isolation and containment method to protect against persistent threats and targeted attacks without whitelisting or scanning.

Business Need

Blue Ridge Networks needed to test whether AppGuard could handle millions of concurrent users for one of its leading customers, AOL, which was about to commence the distribution of AOL Tech Fortress powered by AppGuard to AOL members. Previous testing revealed AppGuard could handle hundreds of thousands of concurrent users, but the AOL application presented a much bigger challenge—scaling for millions of users.

AOL Tech Fortress, powered by AppGuard, plays a critical role for AOL customers by protecting Windows PCs, laptops and tablets from zero-day malware attacks. “We developed a back-end system that we maintain 24x7 and which interfaces to the AOL back-end for license control and ongoing product service, such as software and configuration updates,” explains Fatih Comlekoglu, a co-founder and the Chief Software Architect for Blue Ridge Networks. “We wanted to test AppGuard to ensure our infrastructure could withstand millions of transactions per day executed by the large volume of AOL customers for software downloads, license activation, and software updates.”

Blue Ridge Networks also wanted to determine whether it would need to scale up its existing server resources or scale out by adding additional servers. In addition, the company needed to validate how dedicated HTTP SSL offload and load distribution worked under various system loads.

“In the past, we relied on our own homegrown scenarios to test each component,” says Comlekoglu. “We already knew our individual components could hold their own, but we were not sure if that would be the case when accessed by millions of website visitors, a scenario for which we could not create realistic simulations.”

“We experienced a first-class customer engagement with SmartBear. From our account rep to the project manager and the top-notch engineering team, we always received prompt and professional service. With the help of SmartBear, our collaboration with AOL turned into a successful product launch experience.”
Fatih Comlekoglu, Co-Founder and Chief Software Architect Blue Ridge Networks

Targeted Solution

Blue Ridge Networks hoped to find a reputable and controllable load-testing solution that did not require extensive script development for simulation. While conducting initial research, Comlekoglu first discovered two primary options. “We found many cheap “blast” type load testers as well as other solutions with lengthy script development that required us to customize the load testing to our needs,” he explains. “Neither was suitable. The former lacked control and reality while the latter required unnecessary software development.”

After considering testing tools that require significant time to learn how to use the tool as well as solution providers that require extensive interaction with internal client engineers, Comlekoglu was not sure if he could identify a partner with the necessary skills. A chance meeting with representatives from SmartBear at an industry conference provided Comlekoglu with the solution he needed.

“SmartBear can run tests using their own professional services team and without asking us to invest a lot of our time,” Comlekoglu says. “They rely on their own advanced tools that go well beyond a simple botnet, and they offer experts who know how to conduct application load tests as well as interpret the results.”

The SmartBear professional service team addressed the needs of Blue Ridge Networks by leveraging the SmartBear SoapUI Pro and LoadUI Pro solutions. “SmartBear’s expertise and product set provided the perfect balance,” Comlekoglu says. “They engineered customized scripts and ran the load tests for us within one week—without requiring extensive script development from us.”

The SmartBear team first used SoapUI Pro to build tests for confirming that the AppGuard interface worked properly for one user as far as activating licenses and confirming license validity as well as the ability to turn off licenses. SmartBear then uploaded the SoapUI Pro functional tests into LoadUI Pro and distributed the tests to 10 virtual machines running in the cloud. The servers tested how well AppGuard holds up while executing one thousand transactions per second, which simulates approximately one million simultaneous users checking their licenses within a single second.

Benefits and Results

With SmartBear’s help, Blue Ridge Networks verified that its current system could handle 25 million new users per hour as well as 80 million user and configuration updates per hour. “We also learned that we can easily scale to 100 million new users per hour and 400 million users and configuration updates per hour,” adds Comlekoglu. “Without SmartBear, we could not have determined the boundaries of our infrastructure developed for AOL Tech Fortress powered by AppGuard.”

In addition to validating the Blue Ridge Networks infrastructure, engaging with SmartBear also reduced the risk of unknowns when going live and helped Blue Ridge Networks predict with confidence exactly how to scale as more AppGuard customers come online. “The cost-effective, adaptable load-testing solution offered by SmartBear makes it possible to create a simulated load environment within a matter of a few hours,” Comlekoglu adds. “The other solutions we evaluated required extra development time and resources that would result in delays making such alternative solutions impractical for our go-to-market timeline.”

The SmartBear reports also allowed Blue Ridge Networks to compare test results to raw server data and identify any changes that the server infrastructure required to support the AOL Tech Fortress for AppGuard solution. Comlekoglu and his team found the graphical reports generated by LoadUI Pro were very easy to interpret.

The test results helped Blue Ridge Networks determine that virtual CPUs added on the fly would eliminate any performance issues. The AppGuard support team then created a procedure to add virtual CPUs in case the need arises and prepared additional warm standby servers to bring online when specified load thresholds are reached.

“We experienced a first-class customer engagement with SmartBear,” Comlekoglu says. “From our account rep to the project manager and the top-notch engineering team, we always received prompt and professional service. With the help of SmartBear, our collaboration with AOL turned into a successful product-launch experience.”

Business Challenges

  • Test whether a major client’s application can scale to support millions of users.
  • Complete testing within a short time frame to support the go-to-market timeline.
  • Partner with a testing provider that requires minimal input from internal resources.
  • Identify a solution that generates easy-to-interpret reports for identifying necessary changes for supporting additional users.

SmartBear Professional Services

  • Executes and completes application load testing within short time frames.
  • Simulates application usage by millions of concurrent users.
  • Generates reports that correlate testing results to raw server activity.
  • Removes testing burden so testing teams can focus on application development.

Measurable Results

  • Reduces application performance risks before going to market.
  • Verifies number of new users and configuration changes per hour that applications can handle.
  • Projects number of new users and user/configuration updates that applications can scale to.
  • Determines whether virtual CPUs added on the fly will eliminate performance issues.
  • Identifies threshold points for when to add warm stand-by servers for additional users.

All company names and/or products referenced herein are either registered trademarks or trademarks of their respective trademark holders.

Related Webinars and Ebooks

API Documentation with the OpenAPI Specification & Swagger Tools
Writing Better BDD Scenarios
REST Testing Made Easy with SoapUI Pro