Custom Framework Support

It is possible to run generic testing frameworks in Bitbar testing cloud. To do so, select eg. the Android Appium server side project type and create your own test environment setup and launching of the tests. If you are testing an iOS application then select the iOS Appium server side project type.

The following example uses Android Appium server test project type and goes through the different steps, optional or required, to have your tests run using your own test environment.

Test Script

First Things First

With server side tests, your zipped and uploaded test suite and the uploaded application are always located in the working directory as /test/tests.zip and /test/application.apk (or .ipa for iOS applications). Make sure the path is reflected accordingly in your tests, if needed.

Scripting a Test Run

When running a server side test run, Bitbar cloud always seeks to execute a run-tests.sh script. This script should contain the environment setup and finally the call to start execution of your test suite. The below goes through a sample script with explanations of why and what is done.

For the impatients, a working sample is available on Bitbar Github repository.

# unzip the test package
echo "Extract test package to current directory"
unzip tests.zip

When ever you upload a zipped test suite, it becomes visible in your working directory during the test run as test.zip. Your test run working directory is /test/. In the above the zipped and uploaded test suite is unpacked to the working directory.

Similarly if you need it, the uploaded application is always present in your working directory as: /test/application.apk.

# App file is in present working directory
PWD=$(pwd)
ls -lrt ${PWD}/application.apk

At this point the application and test suite are available in your working test directory. It is now possible to install any additional libraries or frameworks to customize your test environment.

Installing Python Tools

As an example you might want to use some specific Python libraries to post process your test results so you might need Pip. The best way to install Python libraries is to have them listed in requirements.txt file that should be part of the uploaded test zip-package. The below block shows how to install additional Appium Python dependencies.

echo "Installing pip for python"
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py

echo "Installing Appium Python Client 0.24 and xmlrunner 1.7.7"
chmod 0755 requirements.txt
sudo pip install -r requirements.txt

Installing Node Tools

Maybe instead of Python or Java you are using Node-based tools for app testing? To install Node tools you need using the npm tool.

echo "Trying out node"
node -v

echo "Trying out npm"
npm -v

In October 2017 the above prints out:

Trying out node
v6.11.2
Trying out npm
3.10.10

Installing Ruby

Installing Ruby is possible inside of the Appium (Android) server side test run, using default Linux package management tools.

echo "Installing Ruby"
apt-get update
# When installing Ruby also tzdata gets installed that requires your current
# location to set date and time preferences. To avoid questions during
# installation:
export DEBIAN_FRONTEND=noninteractive
# now we can install Ruby
apt-get install -y ruby-full

Starting Installed Framework

Once you have all the required additional packages and tools installed, you are ready to start your test framework. In this case the Appium server is started.

# start the Appium server
echo "Starting Appium ..."
appium-1.6 --log-no-colors --log-timestamp

# Export some desired capabilities in case they are not set in test
# Appium server URL is same for local & Cloud executions
export APPIUM_URL="http://localhost:4723/wd/hub"
export APPIUM_DEVICE="Local Device"
export APPIUM_PLATFORM="android"
export APPIUM_APPFILE="${PWD}/application.apk"

# Get device API level with adb
APILEVEL=$(adb shell getprop ro.build.version.sdk)
APILEVEL="${APILEVEL//[$'\t\r\n']}"
echo "API level is: ${APILEVEL}"

# Older API level needs to use Selendroid for APPIUM_AUTOMATION
if [ "$APILEVEL" -gt "16" ]; then
  echo "Setting APPIUM_AUTOMATION=Appium"
  export APPIUM_AUTOMATION="Appium"
else
  echo "Setting APPIUM_AUTOMATION=selendroid"
  export APPIUM_AUTOMATION="Selendroid"
fi

Now everything is setup to launch an Appium server side test run. Similarly any other test framework run could now be started. If your test suite is in Javam you could now start testing with calling mvn test or Python eg. python MyPythonTestSuite.py.

To get results visible in Bitbar cloud UI, the test results should be output using the Junit XML format. There are multiple libraries that produce this output format in any scripting or programming languages. The location where results are generated varies between the libraries.

The xml test results need to be copied to the local directory as test-all.xml file for the test post processor to attach the results to the test run in cloud UI. For this we need to add still one additional line to our run-tests.sh script.

# Make results available to cloud UI
cp <test output dir>/*.xml TEST-all.xml

# E.g. Maven Surefire plugin generated results
#cp target/surefire-reports/TEST-*.xml TEST-all.xml

Dedicated Dockers

Bitbar cloud supports running private test frameworks or whole environments in customer defined Docker images. With this users can run tests using same test environment locally while developing and in Bitbar testing cloud.

  1. To get started create your own Docker image that you use to run your mobile tests from.
  2. Get in contact with us (feedback@bitbar.com) and be prepared to send us the dockerfile of your test environment.
  3. We’ll review and validate it before making it available as a new project type for you in Bitbar testing cloud.

This service is currently offered as an additional add-on starting from Team plans.