Skip to main content

Enterprise scale live device test automation for Android on Jenkins

"No darlin', don't make me explain it. I don't know why but I don't want to."
Update: the included test system in Gradle supports parallelized test automation on all connected devices as of version 0.3. The current version is 0.7.3. See the following link. Obviously there is a lot of bluster about Google disrespecting scalable live device test automation which I could edit out at this point but A) Gradle Android support is not yet at v1.0 and B) that would be a wee dishonest. This is what I wrote. I was wrong. Check out Gradle. It is awesome-sauce! (edited: 2014-01-16)

Android's developer documentation is excellent and voluminous. There is an excellent mix of training, API guides, and JavaDoc references. So it piques my attention forcing me to ask "why" when a large topic I'm interested in is conspicuously absent. With Xcode 5 and Mavericks Server, Apple has made a serious play to support the enterprise continuous integration practices and live-device test automation for iOS. What system is provided by Google for their Android platform supporting scalable, live-device test automation? Clearly with its Java heritage with ages of continuous integration support, extensive command-line API in ADB, and massive JUnit library, all the pieces are there. But the documentation doesn't talk about scale, Ant and now Gradle can't handle multiple devices, the ADT plugin only automatically serializes test execution on multiple devices but that doesn't apply to the command-line tools. I could go on. So what gives? Why aren't the dots finally connected? 

I just assume that Google's massive, proprietary internal testing system is partly to blame. They can't package it with Android. But since it solves all the enterprise scale issues for them, the engineers don't prioritize solving that for others, or even just explaining it. Heck, despite all the reasons to loathe the emulator, Google not only leverages their massive server resources to reduce the pain of testing on emulators, they go so far as to decry live device testing for being expensive of all things. See their presentation and slides from GTAC 2013. To Google I say, "no, live device testing is irreplaceable." It is the foundation for one of my favorite robust and growing companies.

So now that we've covered the history and context, let's get down to brass tacks. My work over the last 3 years has included building a device lab, deploying a Jenkins CI system, and evangelizing its use among the various product teams here at the studio. I wrote an article on this blog touching lightly on the components of our system last year and the time has come to really lay out the details, particularly the crucial adaptation of Jenkins' master and slave node architecture which enables scalability through parallelization. I'm not going to write the book on how to set up Android test automation on Jenkins. That's already been done by Diego and it is a great resource. What I am going to do instead is the following:
  1. Describe the node hierarchy and the method for configuring it in a device lab
  2. Provide sample template jobs for building and testing an application which you can import DIRECTLY INTO YOUR JENKINS DEPLOYMENT (all caps!!!)
  3. Provide an open-source release of the script we use to tie the whole system together
Jenkins-ci.org has good documentation on setting up distributed builds. They even go so far as to provide a step-by-step guide here. The key to understanding parallelization of Android test automation is to configure a slave node for each device you want to include in your test system. What makes this scalable is that you can host multiple slave nodes on the same computer. This means you can have a computer with an array of USB hubs and devices connected to it all talking to the same ADB service at the same time. There is an upper limit on the number of connected devices ADB can manage though I'm uncertain of what that number specifically is. I have heard it is on the order of 40 or 50. Let me illustrate what this can look like. Given 1 build server, 1 lab computer, and 3 devices connected to the lab computer via USB, we'll create 3 slave nodes.
  • Build server
    • hosts Jenkins process for master node
    • Android SDK is installed for building projects via Ant
  • Lab computer
    • configured with DNS (or at least just static DNS) - e.g. 10.100.10.100
    • supports SSH - e.g. devicetest@10.100.10.100
    • Android SDK is installed for supporting device testing and app inspection
  • Device A
    • USB debugging enabled
  • Device B
    • USB debugging enabled
  • Device C
    • USB debugging disabled. Ha ha! Just kidding, it's enabled.
Basically once you get the pattern for the first slave node, you repeat it exactly for all devices on the lab computer only changing the Name and a device serial environment variable value. Here are the important config details to consider and sample settings:
  • Name: "Device A"
    • this will be used across Jenkins for linking jobs to the slave node under the "Restrict where this job can be run" setting on your test job
  • # of executors: "1"
    • this just ensures that only one test runs per device at a time, seems obvious but don't forget it.
  • Launch method: "Launch slave agents on Unix machines via SSH"
    • Host: "10.100.10.100"
    • Credentials: "devicetest (10.100.10.100)"
    • I'm using this as an example because of the config I found convenient and in this case "devicetest" is a user on the Lab computer with SSH access and credentials stored on the master node
  • Environment variables
    • name: "DEVICE_SERIAL"
      • NOTE - in this case "DEVICE_SERIAL" is inherited by any job that is executed on that slave node's context (as opposed to the machine itself, hence running in parallel is threadsafe).
    • value: "00000000000000A"
For each successive slave node config you can just copy the config for "Device A". Just make sure to update the name to reflect each new device and update the value for the DEVICE_SERIAL environment variable. This supplies the plumbing Jenkins needs to manage distributing your test automation in parallel across your slave nodes (even though technically it will be the same host computer for each one, trust me, it works in parallel).

To take advantage of this architecture, I recommend using individual test jobs for each node and separating the test job from the build job. If your build job triggers downstream test jobs, they can each run independently, fail in any unique way, and if you're really adventurous provide direct debugging per device within the Eclipse IDE using the Mylyn connector plugin. Follow me here, that means your developers can debug test results across hundreds of devices from their IDE as though the devices were connected to their development machine, all while offloading the processing and time cost of the test passes to the CI system.


Words, words words! Give me something I can use!


Because I'm a generous test monkey, I've gone ahead and created and tested two jobs in our Jenkins instance which perform what I've just described: an upstream build job and a downstream test job. To scale this, you simply duplicate the test job and point it at a second, third, forth, etc, slave node (making sure to modify the name appropriately because Jenkins needs unique job names). These jobs have been exported as XML via the Jenkins CLI tool. Using this same tool in your environment will allow you to reverse the process and import the XML as a new job using the following syntax:
java -jar jenkins-cli.jar -s http://server create-job newmyjob < myjob.xml
...where http://server is the web address of your Jenkins instance and myjob.xml is the name of the copy of my XML. It should even automatically link the two jobs which already identify their dependency in the XML. You can find the XML for each project in the GitHub repo at the end of this article. I'm running a Master node in Mac OSX Mountain Lion and a slave node running Ubuntu Linux 12.04 LTS. The configs include shell commands tailored for those environments. Please adjust as needed for your environment.

[edit] On the excellent suggestion from Christopher Orr, I've added a Jenkins Recipe to the repo which allows you to not only import the configs more quickly and easily than using the Jenkins-CLI (just make sure you install the plugin first), it will also import the plugin dependencies. That's some serious heavy metal stuff, Kohsuke. /horns [/edit]


Speaking of scripts, there's more...


Buried in the test job config is a reference to a script called "test_runner_script.sh". This script does a lot of really convenient things:
  • Takes parameters from the test job identifying:
    • Test ID
    • Project APK
    • Test APK
    • Device ID
    • Test Suite
    • Test Runner
    • Coverage Enabled
  • It can handle screenshots from Robotium
    • specifies directory to collect them in
    • retrieves them for processing
  • It can handle the XML output from the Polidea Instrumentation Test Runner (highly recommended for reporting)
  • It can handle the Emma code coverage file output
    • that's right, who needs Ant?
    • note that the upstream build job is configured already to generate this for you
  • It can uninstall any previously installed versions of the app based on package name
  • It can accept all test suites supported by Android's InstrumentationTestRunner including annotated tests
That's a lot of extras to factor in so please take the time to CAREFULLY inspect what I'm doing in the build and test jobs I've exported. The best way to do so is after importing the XML, trust me. The XML is just fluff, the script is open source. Please feel free to adapt any of this to suit your needs and work best in your environment. 

Looking to the future, especially in the Android Test Kit, I'm hoping that an XML test runner version of the GoogleTestRunner can be developed (possibly by just importing the Polidea test runner's only class into the source for the GoogleTestRunner and extending it via the Test Runner Bridge; look at the source code for Espresso and Polidea's runners, you'll see what I mean).

Here's the repo:

Have fun! Happy testing!

Comments

  1. As per a suggestion from Christopher Orr, I've updated the Github repo with a recipe file for the Jenkins plugin. Try it out and let me know if it works for you.

    ReplyDelete
  2. Hello there!

    "The key to understanding parallelization of Android test automation is to configure a slave node for each device you want to include in your test system."

    Could you expand on this? I'm guessing that this is so Jenkins can deal with the queue of jobs running on the devices, but we are toying with the idea of building a simple library that would handle it, it shouldn't be a big deal.

    But maybe we are missing out on some other issues we could run into?

    Thanks : )

    ReplyDelete
  3. Sorry for the long delay in replying, Juan. The distributed build architecture model Jenkins uses is exactly what you leverage for queuing. You constrain the number of jobs your device nodes can process to 1. In your test jobs, you restrict where the job can be run by either using node labels (i.e. "Android_4.4") or the specific desired node name itself.

    ReplyDelete
  4. Hi, I have a question, is not related to this topic, but maybe you can help me. I am building a device lab, I connect the devices with adb over wifi, but if the device turn off, I have to configure it again. How do you handle device connections? (any device should be rooted).

    ReplyDelete
  5. Is connecting to the devices via wifi a hard requirement for your device lab? You may have a problem there since that is a two-way configuration. You might want to invest in a network of powered USB hubs instead. I haven't spent any serious time trying to get devices to only connect to ADB via wifi.

    I've always preferred physical USB connections and I tend not to use rooted devices. My preferences come down to costs though so if you have the time and money I'd go for redundancy and have a pool of rooted devices along with normal devices. It should be said though that even with physical USB connections, sometimes the device resets, sometimes the ADB daemon dies, and there are any number of other similar interruptions. It is not uncommon to have to restart ADB periodically. I've written a separate post about dealing with locked screens (from rebooted devices).

    ReplyDelete
  6. Hi Russell, I want to implement this scenario but with "Rundeck" instead of "Jenkins". I have never use Jenkins, how is the connection between Jenkins master node and the slave nodes? ssh? this connection is made everytime a test is executed in each node?

    ReplyDelete
    Replies
    1. Jenkins uses JNLP to handle communications between the master and slave nodes. When you configure the node on the master instance, you give the configuration an address and a connection protocol. I prefer SSH. The master node installs a process on the slave node that keeps in touch and provides node data back to the master.

      The key to the scenario I've described is to limit the number of processes assigned to each device. This is tricky if you're wanting to use gradle instead of adb directly since the stock connectedCheck task in gradle will automatically try to use every device. It is much better to assemble your androidTest apk directly in the upstream job and then call 'am instrument' from your slave node's device shell in the downstream job. See the following project's readme documents for more information about manually calling am instrument along with existing functional test code to check it out on: https://github.com/googlesamples/android-testing-templates/tree/master/AndroidTestingBlueprint

      Delete
  7. I really enjoyed reading your article.

    ReplyDelete
  8. Android (stylized as android) is a mobile operating system developed by Google, based on the Linux kernel and designed primarily for touchscreen mobile devices such as smartphones and tablets. Android's user interface is mainly based on direct manipulation, using touch gestures that loosely correspond to real-world actions, such as swiping, tapping and pinching, to manipulate on-screen objects, along with a virtual keyboard for text input. In addition to touchscreen devices, Google has further developed Android TV for televisions, Android Auto for cars, and Android Wear for wrist watches, each with a specialized user interface. Variants of Android are also used on notebooks, game consoles, digital cameras, and other electronics.

    ReplyDelete

Post a Comment

Popular posts from this blog

Jenkins + Devices + AndroidJUnitRunner

New Android build system and test runner, same goose chase using undocumented features and hacks

As I've posted before, I am a big fan of Jenkins. It is extremely flexible, open source, and supported by a staggering array of plugins actively developed by engineers running over 100,000 instances of the server worldwide. With it's distributed node model, you can even build your own device cloud for hosting enterprise-scale automation, economizing hardware investments by sharing resources across multiple projects as well as speeding up automation by parallelizing test runs. I had been using a Jenkins-based system in the past to support instrumentation automation with Robotium quite happily. For the last couple years however, my work hasn't required that as much and I've found myself doing a lot more manual testing and using UiAutomator which didn't require a tight integration between the product codebase and the test code. As a result, I've been slow to adopt the…

UiAutomator and Watchers: Adding Async Robustness to UI Automation

"I'm looking over your shoulder... only because I've got your back." ~ Stephen Colbert
After my recent UiAutomator review a user brought up an important question about the use of UiWatcher. The watchers serve as async guardians of the test flow, making sure the odd dialog window doesn't completely frustrate your tests. Having a tool that automatically watches your back when you're focused on the functional flow of your tests is awesome. 100% pure awesomesauce. Since the API documentation on watchers is scant and the UI Testing tutorial on the Android dev guide doesn't cover their use in depth, I figured I should add a post here that goes over a simple scenario demonstrating how to use this fundamentally important UI automation tool.

In my example code below, I'm using uiautomator to launch the API Demo app (meaning run this against an Emulator built in API level 17 - I used the Galaxy Nexus image included in the latest ADT and platform tools). The te…

Run-As Like the Wind: Getting private app data off non-rooted devices using adb run-as and a debuggable app

"You're some kind of big, fat, smart-bug aren't you?"
~Johnny Rico, Starship Troopers (1997) One of the most important things about writing bugs is making them descriptive but concise. Screenshots, video, debug logging, and hardware snapshots are all fairly standard or available to Android testers these days and can save you enormously on text when communicating what constitutes the bug. Sometimes though, the app gets into a weird state due to some transient data issue where you may not own the API or the complexity with forcing the app data into a certain state is prohibitively high. In those cases it is very handy to directly inspect the data the app has in its own directories.

Getting at this data is trivial on emulators and rooted devices but due to file system permissions, this data is otherwise completely private to the app itself. If you're like me, you tend to test using devices rather than emulators and you probably prefer not to root your devices since t…