• Products
    Test Automation
    BitBar Cloud-Based Web and Mobile App Testing VisualTest Automated Visual Testing TestComplete Automated UI Functional Testing
    App Production Visibility
    BugSnag Error and Real User Monitoring
    Performance Testing
    LoadNinja Automated UI Performance Testing
    API Lifecycle
    AlertSite Global, Synthetic API Monitoring ReadyAPI Automated API Testing Platform SwaggerHub Design, Model, & Share API Definitions SwaggerHub Explore Evaluate the functionality of any API PactFlow Complete Contract Testing
    Test Management + BDD
      Zephyr Standalone & Jira Test Management
    Collaboration
    Collaborator Code, Document, & Artifact Review
    Open Source
    Cucumber Open Validate Specs Against Your Code SoapUI Create & Execute API Test Automation Swagger Interact With API Resources Pact Consumer-driven Contract Testing VIEW ALL PRODUCTS
  • Solutions
    Solutions
    API Lifecycle Observability Performance Testing
    Test Management UI Testing
    Integrations
    Source Code Management Collaboration
  • Resources
    Support
    AlertSite AQTime Pro BitBar Collaborator CucumberStudio LoadComplete LoadNinja PactFlow QAComplete ReadyAPI
    SwaggerHub SwaggerHub Explore TestComplete TestEngine TestLeft VisualTest Zephyr Enterprise Zephyr Squad Zephyr Scale
    New Support Case
    Manage My Cases
    Resources
    Case Studies Webinars eBooks White Papers
    Academy Community Docs
    Blog
    Recent Posts
    What is Visual Testing?
    How to Use ChatGPT in Your Automated Tests
    About Us
    News
    Partners
  • Careers
  • Cart
  • EN DE
  • Login
  • Try Any Tool for Free
    TestComplete ReadyAPI BugSnag BitBar SwaggerHub SwaggerHub Explore PactFlow VisualTest Collaborator Zephyr Scale Zephyr Squad Zephyr Enterprise LoadNinja AlertSite

Delivering an Exceptional Firefox Experience to Android and iOS Users

Mozilla Corporation prides itself on keeping the internet open and accessible to all. With the rise of smartphone and mobile internet, it was a natural move for the Mozilla team to bring its Firefox Browser to Android and iOS users. Today, over a billion web browser users around the world enjoy the Firefox experience on desktop, mobile phone, or tablet.

Bringing the same level of Firefox Desktop experience to Firefox Mobile has always been a goal for Mozilla’s development teams. The automation team led by Joel Maher, Lead Automation & Tools at Mozilla, wanted to go through the full set of tests on a variety of physical phones with different combinations of hardware and software.

“We worked with Sakari [the CTO at BitBar], who helped us set up a Docker image container connected directly to a phone in their private cloud. And it ended up being great.”

Joel Maher, Lead Automation & Tools at Mozilla

Unable to Scale Real Device Testing with a Homegrown Device Lab

The company started the Firefox Android project in 2012, and the automation team built many automation scripts to run against their in-house devices.

“It didn’t go well,” Joel admits. “The phones were distributed at different desks in the office, or to people working remotely. Running tests became very dependent on that person being around to fix the phones in case of any issues. We started moving to emulators.”

Due to the lack of a robust real device testing solution, the team opted to test on emulators, since it was impractical to run tests on physical devices. In 2017, the year before they adopted BitBar Private Cloud, the team was hosting half of a rack of devices at Mozilla’s Mountain View office.

“This time, we set up 30 Android phones connected to a couple of servers in a closet. We were able to run the tests, but devices would go offline on any given day,” explains Joel. “We didn’t have any paid staff to look at the devices. When some phones went offline, it always required somebody hands-on to get into the closet and spend a half hour getting those phones online. It wasn’t a robust setup, and we were frustrated with that. So we started to consider other options.”

Putting BitBar to Work Out Unique Testing Requirements

Joel’s team looked at a variety of options, including Google Firebase and Amazon Device Farm. However, none of these could meet Mozilla’s unique testing needs.

“Our web browser tests aren’t built using APIs. Instead, all our commands are built into a browser, and we need to load a webpage from a server. That doesn’t work with the typical way to run APIs on a phone,” explains Joel. “What we’re testing is whether we can load this web page and if it can render properly. Because of that, everything depends upon a certain framework or a list of different automation frameworks.”

In addition to necessary support for custom frameworks, Joel’s team needed rooted phones to make tests run more smoothly. Given BitBar’s previous success for custom OS flashing (Firefox OS) on Android devices for Mozilla, Joel’s team reached out to the BitBar team.

“We didn’t have Dockerized containers or high-quality hubs connecting them. We got by with what we had, as we didn’t have time to experiment and make things better,” says Joel. “Then we worked with [then BitBar CTO] Sakari Rautiainen, who helped us set up a Docker image container connected directly to a phone in their Private Cloud. It ended up being great.”

The POC with BitBar set up Joel’s team for success in a much faster way, and turned out to be a more reliable and repeatable process.

Realizing the Full Potential of BitBar’s Technologies for CI/CD and Full Automation

Every week, Joel’s team has over a thousand builds in the integration and release branches. They also have a “try server” where developers test their stuff before it gets checked in. With around 2,500 builds in any given week, there’s a lot of testing. To improve test efficiency and support its continuous integration and release processes, Mozilla created Taskcluster, an easy-to-configure and robust scheduling software that helps to distribute test runs.

“For a full set of tests against one Android build,” Joel explains, “we would launch a hundred emulator jobs and 40 tasks on BitBar’s real devices. If we were to run all of them sequentially, that would take 75 hours to complete – 60 hours for the emulators and 15 hours for real devices – without repeating them. Mozilla created Taskcluster to distribute multiple tests across available Workers [a Worker could be an emulator or a phone at BitBar] at the same time. Now we’ve taken that 75 hours down to 5 hours to get through a test cycle.”

For real device testing, a proxy will call a BitBar API and insert the Taskcluster test jobs into the proper queue. With Taskcluster managing test orchestration, and BitBar’s support for unlimited device concurrency, Joel’s team has brought 15 hours of real device testing down to an average of 2 hours.

To help Mozilla enhance its full automation and continuous testing practice, BitBar is hosting a Private Cloud with a multitude of devices, dedicated to Mozilla, in its Santa Clara data center.

“As far as BitBar’s real device environment, we’re over 90% utilization during the peak hours,” says Joel. “During the core workday, from 6 a.m. to 8 p.m. Pacific Time, we’re guaranteed to have test jobs in the queue. We’ve always been running full automation. BitBar’s solution has enabled us to scale up our automated testing. We’re running a much higher throughput, and these tests are depended on by a much larger volume of people.”

Monitoring and Scaling to Achieve a Robust Environment

“It was hard to scale our testing efforts before switching to BitBar,” Joel adds. “We needed better organization, and we didn’t have good tools. If one of our servers needed updating, for example, all eight devices connected to it would go down. Now we run test automation on a hundred phones hosted by BitBar, and we have alerts on the phone status online. It’s a much more robust environment where it’s more self-healing, and we’ve got more eyes on it.”

By using the BitBar API, the automation team at Mozilla is able to monitor the system and infrastructure to get proper alerts on device status.

The frustration with the homegrown testing solution led the team to think about outsourcing device hosting and management while allowing them to run test automation on real devices.

“Since we didn’t have any paid staff to look at the devices, when some devices went offline, it required somebody to get into the closet and spend a half hour getting these phones online. It wasn’t a robust setup, and we started to consider other options.”

Joel Maher, Lead Automation & Tools at Mozilla
Try BitBar for Free
Delivering an Exceptional Firefox Experience to Android and iOS Users
Download the PDF
Industry
Enterprise Software

Key Products

Webinar
6 Steps to Achieving Performance Nirvana in 2016
Performance improvements are multifaceted: they can be challenging and sometimes
1 hour 2 minutes
Webinar
6 Steps to Become a .Net Performance Expert
To get you started, we are offering a tutorial-like webinar for testing performance
60 minutes
Webinar
6 Ways to Measure the ROI of Automated Testing
Join SmartBear's testing experts to discover what it means to transition from manual to
60 minutes
  • 0
  • 1
  • 2

Explore SmartBear Products

  • AlertSite
  • AQTime Pro
  • BitBar
  • BugSnag
  • Capture for Jira
  • Collaborator
  • Cucumber for Jira
  • Cucumber Open
  • LoadNinja
  • Pact
  • PactFlow
  • ReadyAPI
  • SoapUI
  • Swagger
  • SwaggerHub
  • SwaggerHub Explore
  • TestComplete
  • TestEngine
  • TestLeft
  • VisualTest
  • Zephyr
  • About Us
  • |
  • Careers
  • |
  • Solutions
  • |
  • Partners
  • |
  • Responsibility

  • Contact Us
  • |
  • +1 617-684-2600 USA
  • |
  • +353 91 398300 EUR
  • |
  • +61 391929960 AUS
© 2023 SmartBear Software. All Rights Reserved.
  • Privacy
  • |
  • Terms of Use
  • |
  • Site Map
  • |
  • Website Terms of Use
  • |
  • Security
Log In
Products
AlertSite
BitBar
CrossBrowserTesting
CucumberStudio
LoadNinja
QAComplete
SwaggerHub
VisualTest
Additional Links
SmartBear Community
Case Management
License System