Custom Framework Support

When running tests on Bitbar’s servers in parallel, the user is able to configure the test environment the test is executed. On Public cloud the Android or iOS Server side Appium framework should be selected depending wether testing on Android or iOS devices.

The following examples use Android Appium server side framework and go through different steps, optional or required, to have your tests running using your own test environment.

Test Script

Test Package to Upload

With server side tests your tests need to be uploaded to cloud for execution in a zip-package. Here is an example content of such a zip package. Note the run-tests.sh script must be at the root of the unzipped package.

$ ls -lrt
-rwxr-xr-x  1 user  staff    60 Feb 28 16:31 requirements.txt
-rw-r--r--  1 user  staff  4667 Mar  9 15:38 TestdroidAppiumTest.py
-rw-r--r--  1 user  staff  2687 Mar 13 11:04 BitbarSampleAppTest.py
-rwxr-xr-x  1 user  staff  1263 Mar 14 16:10 run-tests.sh

On the cloud servers, the run-tests.sh script is called by the test execution environment. This script you can edit and make modifications to when defining your required environment.

The uploaded test package and the uploaded application are always located in the working directory as /test/tests.zip and /test/application.apk (or .ipa for iOS applications). Make sure the path is reflected accordingly in your tests, if needed.

Scripting a Test Run

When running a server side test run, Bitbar cloud always seeks to execute the run-tests.sh script. This script contains the environment setup and finally the call to start execution of your test suite. The below goes through a sample script with explanations of what is done.

For the impatiens, a working sample is available on Bitbar Github repository.

# unzip the test package
echo "Extract test package to current directory"
unzip tests.zip

Your uploaded test suite, becomes visible in the test run working directory as tests.zip. The above shell script lines unpack the test package you uploaded to the cloud, into the current working directory.

Similarly if you need it, the uploaded application is always present in your working directory as: /test/application.apk.

# App file is in present working directory
PWD=$(pwd)
ls -lrt ${PWD}/application.apk

Your application and test suite are available in your working directory. You can now install any additional libraries or frameworks to customize your test environment. If you don’t require any changes, you would instead start a test run.

Installing Python Tools

As an example you might want to use some specific Python libraries to post process your test results so you might need Pip. The best way to install Python libraries is to have them listed in requirements.txt file that should be part of the uploaded test zip-package. The below block shows how to install additional Appium Python dependencies.

echo "Installing pip for python"
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py

echo "Installing Appium Python Client 0.24 and xmlrunner 1.7.7"
chmod 0755 requirements.txt
sudo pip install -r requirements.txt

Installing Node Tools

As an example maybe instead of Python or Java you are using Node-based tools for app testing? To install Node tools you need using the npm tool.

echo "Trying out node"
node -v

echo "Trying out npm"
npm -v

In October 2017 the above prints out:

Trying out node
v6.11.2
Trying out npm
3.10.10

Installing Ruby

Installing Ruby is possible inside of the Appium (Android) server side test run, using default Linux package management tools.

echo "Installing Ruby"
apt-get update
# When installing Ruby also tzdata gets installed that requires your current
# location to set date and time preferences. To avoid questions during
# installation:
export DEBIAN_FRONTEND=noninteractive
# now we can install Ruby
apt-get install -y ruby-full

Starting Installed Framework

Once you have all the required additional packages and tools installed, you are ready to start your test framework. In this case the Appium server is started.

# start the Appium server
echo "Starting Appium ..."
appium-1.6 --log-no-colors --log-timestamp

# Export some desired capabilities in case they are not set in test
# Appium server URL is same for local & Cloud executions
export APPIUM_URL="http://localhost:4723/wd/hub"
export APPIUM_DEVICE="Local Device"
export APPIUM_PLATFORM="android"
export APPIUM_APPFILE="${PWD}/application.apk"

# Get device API level with adb
APILEVEL=$(adb shell getprop ro.build.version.sdk)
APILEVEL="${APILEVEL//[$'\t\r\n']}"
echo "API level is: ${APILEVEL}"

# Older API level needs to use Selendroid for APPIUM_AUTOMATION
if [ "$APILEVEL" -gt "16" ]; then
  echo "Setting APPIUM_AUTOMATION=Appium"
  export APPIUM_AUTOMATION="Appium"
else
  echo "Setting APPIUM_AUTOMATION=selendroid"
  export APPIUM_AUTOMATION="Selendroid"
fi

Now everything is setup to launch a server side test run. If your test suite is using Java you would start testing by calling mvn test. With Python the call would be python MyPythonTestSuite.py.

To get results visible in Bitbar cloud UI, the test results need to be output using the Junit XML format. There are multiple libraries that produce this output format in any scripting or programming languages. The location where results are generated varies between the libraries.

The xml test results need to be copied to the local directory as test-all.xml file for the test post processor to attach the results to the test run in cloud UI. For this we need to add still one additional line to our run-tests.sh script.

# Make results available to cloud UI
cp <test output dir>/*.xml TEST-all.xml

# E.g. Maven Surefire plugin generated results
#cp target/surefire-reports/TEST-*.xml TEST-all.xml

Dedicated Dockers

Bitbar cloud supports running private test frameworks or whole environments in customer defined Docker images. This allows users run tests on same test environment locally while developing and in Bitbar testing cloud. This same functionality is also supported on Private and On-premise cloud setups.

  1. To get started create your own Docker image that you use to run your mobile tests from.
  2. Get in contact with us (feedback@bitbar.com) and be prepared to send us the dockerfile of your test environment.
  3. We’ll review and validate it before making it available as a new project type for you in Bitbar testing cloud.

This service is currently offered as an additional add-on starting from Team plans.