Scripted Test

Monitoring supports creating scripts, specifically Appium scripts in Java to run checks on mobile applications running on simulators and real devices.

Creating Build

Currently iOS builds need to be simulator builds. To create a simulator build the -sdk iphonesimulator9.3 parameter needs to be passed to the xcodenbuild command. Your build is available under the location: ~/Library/Developer/Xcode/DerivedData.

The .ipa can also be created using Xcode by selecting “generic ios device” as build target. The archive is available under the Product menu.

Creating a Test Zip

The tests are assumed to be in a zip-package that is unzipped and executed at check test run. Once the package is unzipped in an empty folder, that folder should contain any of the following for the test to be started successfully.

  • a shell script that has all the necessary steps to start the execution of the test steps. This file is not mandatory if the unzipped file structure has a pom.xml file.
  • In case the above script is not available, then Bitbar Monitoring tries to run the Maven pom.xml assumed to be at the root, as mvn test. This should take care of all the steps needed to run the test.

An example zip package could look like this.

$ zipinfo
Zip file size: 3504 bytes, number of entries: 11
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/test/
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/test/java/
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/test/java/com/
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/test/java/com/bitbar/
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/test/java/com/bitbar/monitoring/
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/test/java/com/bitbar/monitoring/samples/
drwxr-xr-x  3.0 unx        0 bx stor 17-Jan-16 14:45 src/test/java/com/bitbar/monitoring/samples/ios/
-rw-r--r--  3.0 unx     2187 tx defN 17-Jan-16 14:45 src/test/java/com/bitbar/monitoring/samples/ios/
-rw-r--r--  3.0 unx     1764 tx defN 17-Jan-16 14:45 pom.xml
-rw-r--r--  3.0 unx      152 tx defN 17-Jan-16 16:58

In this example the is redundant as the pom.xml file would be sufficient for the test to get started.

Script Test Results

The results of the test script should be in a standard Junit XML format for them to be properly parsed and presented in Bitbar Monitoring UI.

Appium Test

Appium supports running tests against native, hybrid and web iOS and Android mobile applications. Appium supports running web tests using Android Chrome and iOS Safari browsers.

To start an Appium test, a zip package as described above needs to be created. Appium is pre-installed on the hosts running tests and is started automatically. The or pom.xml files only take care of starting test run execution.

$ ls -lrt
total 394
drwxr-xr-x  1 userA  staff     102 Jan 16 04:45 src
-rw-r--r--  1 userA  staff    1764 Jan 16 04:45 pom.xml
-rw-r--r--  1 userA  staff     152 Jan 16 06:58
-rw-r--r--  1 userA  staff    3504 Jan 16 06:59
-rw-r--r--@ 1 userA  staff   67480 Jan 16 07:11 page_loading.png
-rw-r--r--@ 1 userA  staff  125295 Jan 16 07:12 page_loaded.png

In this example the file is very simple.

$ cat

# starts execution of the Java Appium test using Maven and ./pom.xml file.

echo ">>> Starting test from script <<<"
mvn test


To get screenshots visible in the Monitoring cloud checks view, all screenshots should be stored to a ./screenshots directory.

Eg. this is how screenshots are taken in the sample scripts:

private void takeScreenshot(String name) {
    try {
        File scrFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
        File testScreenshot = new File("screenshots", name);
        FileUtils.copyFile(scrFile, testScreenshot);

Postman/Newman Tests

Postman is an API development tool that is often used with Newman to create API tests for regression and monitoring.

Currently our script running servers do not have Postman installed by default but you can install small tools like this at the beginning of the monitoring run.

## Install newman
npm config set proxy null
npm config set https-proxy null
npm install newman
export PATH=$PATH:$(pwd)/node_modules/.bin
## clone Postman sample for testing
git clone

Now you are all set to run your Postman or Newman tests against your API endpoints.

newman run -k --reporters "cli,junit" --reporter-junit-exporter "$LOGS_DIR" newman/examples/sample-collection.json;

There is a sample shell script for running the Postman sample here.

Supported Environment

In Bitbar Monitoring service, the default Appium setup supports tests written in Java. Sample tests are available at Bitbar’s Github samples.

The used software stack available for scripted tests includes among other things these tools and versions.

  • Appium 1.6.4
  • Java 8 (1.8.0_112) and Apache Maven 3.5.0
  • Android SDK 7.0
  • Xcode 7.3.1
  • Python 2.7.10
  • Ruby 2.0.0p645
  • and most Unix tools that you would expect there to be