Skip to main content

Lies, Damn Lies, and Android Non-Rooted Device Code Coverage

"There are three kinds of lies: lies, damn lies, and statistics."
~Mark Twain, aka Samuel Clemens
In a previous post I mentioned rather blithely that we were collecting code coverage data from live devices using Emma. If you're unfamiliar with Android's use of Emma, which comes with the Android SDK tools when you install them, the documentation won't do you much good. It isn't to say that the Android developer portals' "Testing from Other IDEs" section isn't helpful or that the online documentation for the SDK's InstrumentationTestRunner isn't either. Far from it. But if that's all you read, you may find yourself getting confused shortly after getting started.

Let's say for example, that you begin your deployment of code coverage tracking via the robust set of Ant commands included in the default build.xml. You may notice from the Testing from Other IDEs section above that there is a simple command you can run that allows you to also get code coverage data from your build and subsequent test jobs. If you attempt to run this on a device, via the "ant emma debug installt test" command the process will run smoothly until you get a strangely unforeseen message aka "the big emma lie":
WARNING: Code Coverage is currently only supported on the emulator and rooted devices.
The folks on Stack Overflow have tackled this to some success and so have the folks in the Android Developers Google Group. They're smart people and very clever but all of them seem to get hung up on that big emma lie. You may be asking, "what's the lie? It works when I use it on an emulator and doesn't when I use it on a non-rooted device. This seems consistent." Well, I'll tell you but first a little story about JUnit XML output in Android.

In Android's security paradigm, the sandboxing of an application to writing to its own directory keeps non-rooted devices under control. Files installed by your application including databases, images, etc all wind up in '/data/data/com.your.application/'. That is true of code coverage output and even XML output if you're using Polidea's version of the InstrumentationTestRunner. I mentioned our use of this third party JAR in a previous post because it is great for getting standard XML output from Android's JUnit tests.

Running continuous integration on live devices would be kind of pointless if you couldn't pull all your test results back from the device. Hence why Polidea named their project the "missing xml junit test runner". It is something folks in the Java development community outside of Android have enjoyed for years so what's the issue with Android? I don't know. All I can assume is that the development model focuses on the use of emulators and Eclipse. This is why using the command line to execute test automation is included in the Android Developer Portal's section on "other IDEs". It is non-standard. So non-standard in fact that simply including the Polidea JAR does not grant you access to your XML on devices either. It works fine with emulators but like everything else in your project, it is sandboxed. The question is how can we get out of the sandbox without rooting the device or resorting to emulators?

At first we got clever. We flexed some coding skills and adapted the open source Polidea JAR to fork the test data output stream into LogCat. We added a script to parse LogCat for specific tags and collect those parses into XML on the USB host server in our lab. We thought we were quite clever with this because it got us our nice, portable JUnit output off live devices and that was our first goal. It worked very well, despite feeling a little TOO hack-ish, for about a year. But when I started to revisit the purpose of the lab and testing on devices, I couldn't shake the feeling that we missed something fundamental about the Android sandbox. Turns out my hunch was correct and I cracked the lie of XML and code coverage from live, non-rooted devices. It was right in front of me all along.

Elsewhere in our build process, we had previously included a step on the Jenkins job to update the AndroidManifest.xml with a build name that made our test builds more human friendly. This simple method of inserting code into the build at compile time became the model for how we subverted that lie about emulators and rooted devices. Because we were doing this code insertion at build-time, nothing changed in the source code for the project and outside of that specific job. That same manifest also controls the scope of the sandbox by the 'uses-permission' flag. That flag is what I had previously experimented with in order to get screenshots from devices using Robotium (expect a post on this topic soon). And because I am lazier than clever, I didn't immediately at that minute use the same flag to push XML files and code coverage output to a shared location on the device sdcard.

It took about 2 days instead.

So there you have it really. Without making this into a step-by-step description of the whole process of getting code coverage data off live devices (I can include that in a different post if desired), the secret was using a specific Jenkins build job step to insert the following line into the application's AndroidManifest.xml
"uses-permission:android.permission.WRITE_EXTERNAL_STORAGE"
Now suddenly I could get all the data that the emulator-driven paradigm gave but instead from every Android device in our lab, simultaneously even. All I did afterwards was update the test runner script on the USB host server in our lab to point the XML output, screenshot output, and Emma code coverage output to a single directory on the device sdcard that gets erased after every run. YAY! Ponies for everyone!

A word of caution though. While you may find this unlocks a whole new world of device automation coverage (both the code and the platform diversity), you should be aware that Code Coverage data is a form of statistical data. Code coverage, like any statistic can tell a useful story but be careful to understand the meaning of the data in context so that you aren't trapped in "give me 100% coverage on all automation" Hell. The proper use of code coverage statistical data is beyond the scope of this post but keep in mind the old saying, "there are lies, damn lies, and statistics."

Comments

  1. For starting an any new project development for any company you should have to study or understand the whole big data of the company. But this is but natural not a common people can understand the big data record of the company, for making this easy for you the https://activewizards.com/ is here for you which provides you the best data scientist for summarizing your big data.

    ReplyDelete

Post a Comment

Popular posts from this blog

Jenkins + Devices + AndroidJUnitRunner

New Android build system and test runner, same goose chase using undocumented features and hacks

As I've posted before, I am a big fan of Jenkins. It is extremely flexible, open source, and supported by a staggering array of plugins actively developed by engineers running over 100,000 instances of the server worldwide. With it's distributed node model, you can even build your own device cloud for hosting enterprise-scale automation, economizing hardware investments by sharing resources across multiple projects as well as speeding up automation by parallelizing test runs. I had been using a Jenkins-based system in the past to support instrumentation automation with Robotium quite happily. For the last couple years however, my work hasn't required that as much and I've found myself doing a lot more manual testing and using UiAutomator which didn't require a tight integration between the product codebase and the test code. As a result, I've been slow to adopt the…

UiAutomator and Watchers: Adding Async Robustness to UI Automation

"I'm looking over your shoulder... only because I've got your back." ~ Stephen Colbert
After my recent UiAutomator review a user brought up an important question about the use of UiWatcher. The watchers serve as async guardians of the test flow, making sure the odd dialog window doesn't completely frustrate your tests. Having a tool that automatically watches your back when you're focused on the functional flow of your tests is awesome. 100% pure awesomesauce. Since the API documentation on watchers is scant and the UI Testing tutorial on the Android dev guide doesn't cover their use in depth, I figured I should add a post here that goes over a simple scenario demonstrating how to use this fundamentally important UI automation tool.

In my example code below, I'm using uiautomator to launch the API Demo app (meaning run this against an Emulator built in API level 17 - I used the Galaxy Nexus image included in the latest ADT and platform tools). The te…

Run-As Like the Wind: Getting private app data off non-rooted devices using adb run-as and a debuggable app

"You're some kind of big, fat, smart-bug aren't you?"
~Johnny Rico, Starship Troopers (1997) One of the most important things about writing bugs is making them descriptive but concise. Screenshots, video, debug logging, and hardware snapshots are all fairly standard or available to Android testers these days and can save you enormously on text when communicating what constitutes the bug. Sometimes though, the app gets into a weird state due to some transient data issue where you may not own the API or the complexity with forcing the app data into a certain state is prohibitively high. In those cases it is very handy to directly inspect the data the app has in its own directories.

Getting at this data is trivial on emulators and rooted devices but due to file system permissions, this data is otherwise completely private to the app itself. If you're like me, you tend to test using devices rather than emulators and you probably prefer not to root your devices since t…