Skip to main content

Deploying Android Automation in Parallel

So here we go, right off the deep end.

One of the things that has bugged me about Android since I first started with it 18 months ago is the half-complete features and tools supporting on-device automation at any kind of enterprise scale (as opposed to just a few developers and devices in some garage somewhere). So much of the platform just integrates so beautifully with other scalable components of a build and test system that it felt like a hack when we managed to get parallelization up and running in our device lab. Hey, at least the platform is open and flexible enough to do so in the first place, right? A startup in Portland is currently building their business model off this functionality (holla, AppThwack, go on with your bad selves!).

So it is there for the clever. I just wish it didn't take being clever to manage running and monitoring multiple, simultaneous jobs on actual, live devices. Why? Well because my cleverness is not nearly as strong as my laziness (one enables the other, I suppose). Knowing it is there isn't necessarily all that reassuring because there are plenty of snares along the road from single device testing to scalable parallelization.

The tools that we use in the Seattle Deloitte Digital studio are open source for the most part except for the single most important piece of the loop. Let's start by talking about the loop itself though. For the unfamiliar, continuous integration usually involves the following pieces:

  1. source control - in our case, Github
  2. build server - in our case, Jenkins
  3. test framework - in our case, JUnit in Android
    1. third party JAR from Polidea for XML output 
    2. third party JAR from Jayway (the ever-popular Robotium library) for easy UI automation
    3. a custom device I/O script hosted on the lab server
  4. reporting and IDE integration - in our case, Jenkins and a Jenkins Eclipse plug-in, respectively.
There are plenty of blog posts about most of those pieces except for 3.3 and that's what I'll focus on here. With that script and the use of slave node configs on Jenkins, we can parallelize test jobs, queue jobs for specific devices, and build comprehensive coverage passes with multiconfig jobs. Jenkins slave node handling is well documented, as is multiconfig job setup. So let's get to that script already, eh?

Without pasting the whole thing in here (since your mileage may vary), let me just break it down by functional step. I am positive you, dear reader, will easily think of how to put this together in a shell script and your implementation will likely be pretty snazzy by comparison. You're awesome like that and we both know it. So awesome that the following functions probably leap off the screen in perfect syntactic BASH within seconds of viewing them...

  1. Collect parameters passed by Jenkins into local variables. These are defined here and also here.
  2. Define directories on device for file output for screenshots, test results xml, and code coverage output.
  3. Clean up the device in prep for a test run
  4. Install new build and test APKs
  5. Run parameterized tests based on the variables from step 1. Typical parameters are: package name, test suite, XML and emma code coverage filenames and output directories, test runner, and  any boolean flags enabling required features.
  6. Collect test artifacts and copy them to Jenkins
Because we've configured device-oriented slave nodes on the Jenkins server managing the test jobs, we can use a common server SSH config for the machine in the lab to which all the devices are connected and then simply supply the device ID you'd get from running 'adb devices' as an environment variable for that slave node. With that in place, each test job config on the Jenkins server calls something like '/working/directory/path/and/testShellScript.sh parameter1 parameter2 ... parameterN'

Jenkins' own slave node job queuing and job reporting dashboards handle the rest.

If you really want to get tricksie, watch this blog for future posts on how we do the same things for iOS devices.

Comments

Popular posts from this blog

Jenkins + Devices + AndroidJUnitRunner

New Android build system and test runner, same goose chase using undocumented features and hacks

As I've posted before, I am a big fan of Jenkins. It is extremely flexible, open source, and supported by a staggering array of plugins actively developed by engineers running over 100,000 instances of the server worldwide. With it's distributed node model, you can even build your own device cloud for hosting enterprise-scale automation, economizing hardware investments by sharing resources across multiple projects as well as speeding up automation by parallelizing test runs. I had been using a Jenkins-based system in the past to support instrumentation automation with Robotium quite happily. For the last couple years however, my work hasn't required that as much and I've found myself doing a lot more manual testing and using UiAutomator which didn't require a tight integration between the product codebase and the test code. As a result, I've been slow to adopt the…

UiAutomator and Watchers: Adding Async Robustness to UI Automation

"I'm looking over your shoulder... only because I've got your back." ~ Stephen Colbert
After my recent UiAutomator review a user brought up an important question about the use of UiWatcher. The watchers serve as async guardians of the test flow, making sure the odd dialog window doesn't completely frustrate your tests. Having a tool that automatically watches your back when you're focused on the functional flow of your tests is awesome. 100% pure awesomesauce. Since the API documentation on watchers is scant and the UI Testing tutorial on the Android dev guide doesn't cover their use in depth, I figured I should add a post here that goes over a simple scenario demonstrating how to use this fundamentally important UI automation tool.

In my example code below, I'm using uiautomator to launch the API Demo app (meaning run this against an Emulator built in API level 17 - I used the Galaxy Nexus image included in the latest ADT and platform tools). The te…

Run-As Like the Wind: Getting private app data off non-rooted devices using adb run-as and a debuggable app

"You're some kind of big, fat, smart-bug aren't you?"
~Johnny Rico, Starship Troopers (1997) One of the most important things about writing bugs is making them descriptive but concise. Screenshots, video, debug logging, and hardware snapshots are all fairly standard or available to Android testers these days and can save you enormously on text when communicating what constitutes the bug. Sometimes though, the app gets into a weird state due to some transient data issue where you may not own the API or the complexity with forcing the app data into a certain state is prohibitively high. In those cases it is very handy to directly inspect the data the app has in its own directories.

Getting at this data is trivial on emulators and rooted devices but due to file system permissions, this data is otherwise completely private to the app itself. If you're like me, you tend to test using devices rather than emulators and you probably prefer not to root your devices since t…