Android end-to-end test automation
Recapping: In the A journey through end-to-end tests article (Please, read it first) we was introduced to the end-to-end
, user journey test
or smoke tests
terms. We also saw how to write test scenarios using Gherkin Syntax to structure our Domain Specific Language and here we are...
Looking to the following scenario, how can we automate it?Given my application is installed
And I tap on the app icon
When the app is launched
Then I see the Login screen
Rules
By observing the given test scenario It's possible to extract some rules and infer some others.
Test scenarios are composed by ordered steps
Each step should be asserted
It's possible to find the exact step that caused a failure.Each step execution result should be printed
As each step should be asserted, each step execution result should also be printed as It is described in the test scenario (e.g. And I tap on the app icon FAILED).Test scenarios should also be ordered
As we're simulating the real world, it should not be possible to open the account screen before authenticate an user (If the application requires authentication, of course)
Now that we have the test rules, let's select the tools that we're going to use to automate our tests.
Tools
On android we have two kind of instrumentation tools that helps to simulate the user interactions, Espresso
and UiAutomator
.
With espresso
we have a sort of gray test box, as it allows the developer to access and modify some application values, on the other hand the UiAutomator simulate a real black box, where we interact with the app and the system (not possible on espresso) as a real user, and that's why we'll choose the UiAutomator.
Ok, what about the Junit? Is it possible to use Junit5? May it have some new feature that can help? Yes, it does, but the Junit5 is not officially supported on android (They also don't have plans for that) and the setup is not simple for the instrumentation tests. Let's keep using the Junit4 instead of adding a bunch of unnoficial library to our project.
And what about cucumber? You can give it a try, but It's not easy to configure, especially with UiAutomator. We can have something simpler by creating a custom test runner with the Junit4, let me show you how.
Test Runner
We have three important rules set for our test steps, they should be ordered, asserted and properly logged, but Junit4 doesn't offer the @Order
and @DisplayName
annotations as Junit5, so, how we can comply with these step rules?
On Junit4 we have the possibility to extend BlockJUnit4ClassRunner
and override the getChildren
and describeChild
methods, let me explain...
The test methods in the Junit4 are filtered/found by the @Test
annotation inside the Junit Test Runner
, this process happens in the getChildren
method, and then each method found is executed by calling the runChild
(Undefined execution order), which also calls the describeChild
method to define the test name to be displayed. Now we have all the information needed to modify the Junit Runner
as we want.
So, the @Test
annotation is just an information to Junit, and the same is valid for the @Ignore
annotation, that tells Junit to just logging the given method as ignored instead of running it. With that, we can also add more information to the test method, as telling Junit the sort order
of a test for example.
So let's create an annotation called @Order
and try it.
@Retention(AnnotationRetention.RUNTIME)
@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION)
annotation class Order(@IntRange(from = 0L, to = 100) val value: Int = 0)
Now it's time to create our custom test runner extending BlockJUnit4ClassRunner
class AutomatorRunner(
private val testClass: Class<*>
) : BlockJUnit4ClassRunner(testClass)
And then we can sort the test methods by overriding the getChildren
method for that
override fun getChildren(): MutableList<FrameworkMethod> {
return sortTestClassMethods()
.map { FrameworkMethod(it) }
.toMutableList()
}
private fun sortTestClassMethods(): List<Method> {
return testClass.methods
.filter { it.getAnnotation(Test::class.java) != null }
.sortedBy { it.getAnnotation(Order::class.java)?.value ?: 0 }
}
In the code above we first filter the existent test (annotated with Test::class
) methods inside the Test Class
sorting them using the Order::class
annotation value.
Now let's create our test class using our the AutomatorRunner
along with Order
annotation.
@RunWith(AutomatorRunner::class)
class ApplicationStartScenario {
@Test
@Order(1)
fun givenMyApplicationInstalled() ...
@Test
@Order(2)
fun andITapOnTheIcon() ...
@Test
@Order(3)
fun whenTheApplicationIsOpened() ...
@Test
@Order(4)
fun thenISeeTheLoginScreen() ...
}
Running the test above will give us the following output:
givenMyApplicationInstalled SUCCESS
andITapOnTheIcon SUCCESS
whenTheApplicationIsOpened SUCCESS
thenISeeTheLoginScreen SUCCESS
Not that readable, right? Let's fix it. We need a display name and we have the option to create a DisplayName
annotation or enhance the Order
annotation, which is the selected approach here.
The @Order
annotation can be renamed to @Step
, as our test Scenarios
are composed by Steps
, then we just need to add the displayName: String
field to the Step::class
@Retention(AnnotationRetention.RUNTIME)
@Target(AnnotationTarget.FUNCTION)
annotation class Step(
val displayName: String,
@IntRange(from = 0L, to = 100) val order: Int = 0
)
To modify the test method display name we need to override the describeChild
method
override fun describeChild(frameworkMethod: FrameworkMethod): Description {
val displayName = getDisplayName(frameworkMethod.method)
return Description.createTestDescription(
testClass.name,
displayName,
frameworkMethod.method.annotations
)
}
private fun getDisplayName(method: Method): String {
val annotation = method.getAnnotation(Step::class.java)
return annotation?.displayName ?: method.name
}
That's it, if the method don't have the @Step
annotation we return the test method name, if it's annotated, then we return the displayName
value from @Step
.
Here's the final version of our Automator Class
class AutomatorRunner(private val testClass: Class<*>) : BlockJUnit4ClassRunner(testClass) {
override fun getChildren(): MutableList<FrameworkMethod> {
return sortTestClassMethods().map { FrameworkMethod(it) }.toMutableList()
}
override fun describeChild(frameworkMethod: FrameworkMethod): Description {
val displayName = getDisplayName(frameworkMethod.method)
return Description.createTestDescription(
testClass.name,
displayName,
frameworkMethod.method.annotations
)
}
private fun sortTestClassMethods(): List<Method> {
return testClass.methods
.filter { it.getAnnotation(Test::class.java) != null }
.sortedBy { it.getAnnotation(Step::class.java)?.order ?: 0 }
}
private fun getDisplayName(method: Method): String {
fun formatMethodName(name: String): String {
return name.replace("_", " ")
}
val annotation = method.getAnnotation(Step::class.java)
return annotation?.displayName ?: formatMethodName(method.name)
}
}
Time to update our ApplicationStartScenario
test class
@RunWith(AutomatorRunner::class)
class ApplicationStartScenario {
@Test
@Step("Given My Application Installed", 1)
fun givenMyApplicationInstalled() ...
@Test
@Step("And I tap on the app icon", 2)
fun andITapOnTheIcon() ...
@Test
@Step("When the app is opened", 3)
fun whenTheApplicationIsOpened() ...
@Test
@Step("Then I see the Login Screen", 4)
fun thenISeeTheLoginScreen() ...
}
The test example above would print:Given My Application Installed SUCCESS
And I tap on the app icon SUCCESS
When the app is opened SUCCESS
Then I see the Login Screen SUCCESS
Cool, but how we can have Ordered Scenarios
? Test Suite
JUnit Test Suite
Ordered scenarios means that we should call the test classes in a sort order, and Junit4 already give us this feature, the Test Suite
, here's how to use it..
@RunWith(Suite::class)
@Suite.SuiteClasses(
ApplicationStartScenario::class,
LoginScenario::class,
HomeScenario::class
)
class SmokeTestSuite
That's it, the class order that goes inside @Suite.SuiteClasses represents that execution order of the SmokeTestSuite
.