An app has to do a lot of things for you to love it—and do only two things wrong to make you hate it: perform sluggishly and consume a lot of power.
Facebook operations engineer Antoine Reversat revealed what Facebook does to prevent those two things and keep users from installing its apps. He showed me the automated mobile testing lab, which has never before been disclosed to the public, that he designed and operates. The system automates testing of Facebook’s mobile apps, Messenger and Instagram for iOS and Android.
Almost 2,000 Android and iOS smartphones are housed in 60 racks at Facebook’s Prineville, Oregon, data center. The standard-sized, specially designed racks hold 32 phones each, interconnected to a server. Linux servers interconnect Android phones, and Mac Minis interconnect iPhones. The phones are remotely controlled and monitored during the testing using an automated system that, like much of what Facebook does, will be open sourced for other mobile testers to use and contribute improvements.
The system automatically installs apps for testing, monitors the apps, measures the execution of the apps and reports the test results to the Facebook developer, who can be at any of Facebook’s software development locations. The test phones are instrumented to measure the performance and power consumption of the apps. Called a regression, the test compares the app’s performance and power consumption to the baseline of the previous version. If the app runs slower or consumes more power than that baseline, the developer is asked to fix it.
The developer doesn’t take the initiative. Every twentieth iteration of the app, called a commit, is pulled from the software repository, installed on some of the phones and tested. While Facebook’s apps seem static and this scale of a test lab unnecessary, they are not. Apps are dynamic because of the mobile designers’ and developers’ fanatic zeal for testing new features, user interface subtleties and new user interactions within the apps.
Updates to the apps are frequently released to selected cadres of users for user experience testing. The users’ experience is measured with instrumentation built into the app, and the results are compared to the stable release to determine if the users like the changes and new features. If the tests prove a better user experience, the limited test version is released to Facebook’s user population. The mobile test lab ensures that the app’s performance and power consumption don’t drift in a sluggish or power-draining direction that could raise users’ ire.
Facebook's test racks
The racks that house the test phones look like any of the thousands of racks in Facebook’s 350,000-sq.-ft. lights-off data center, except they are fully enclosed. The enclosure prevents the Wi-Fi access points connecting the 32 phones in each rack from leaking and interfering with phones under test in the other racks.
Like all of the server racks, the mobile test lab racks are designed for unattended operation except for intervention by a technician when the system sends an alert of a hardware malfunction. The phones’ iOS and Android operating systems can be automatically upgraded or downgraded to match the test scenario. Similarly, the device states can be managed, configuring parameters such as Wi-Fi settings and security certificates.
After testing, the apps are removed and the system restores the phones to the pretested state. If the test instrumentation indicates a reboot is necessary to return them to the initial pretest state, the system sends a reboot commanded and confirms state-readiness for the next test.
Reversat’s demonstration brought home another point. Mobile platform upgrade cycles are much different than those for the PC platform. Intel and PC manufacturers could forecast their revenues and product introductions based on Microsoft’s operating system and application software upgrades because each upgrade required more powerful hardware.
Facebook is representative of a different trend in mobile application development to increase app features without consuming more smartphone hardware resources. Apple, Samsung and other smartphone makers have to design phones that attract buyers without counting on app developers to consume more resources. Richard Windsor of Radio Free Mobile fully explains how this dilemma affects Apple in a recent blog post.
