Run-in-Cloud Using Pipeline (Jenkinsfile)

This guide contains instructions for using the Bitbar Run-in-Cloud plugin from a Pipeline script (Jenkinsfile). For instructions on how to use the UI build step in the same plugin, click here.

Credentials setup

Before you can use the Pipeline plugin, you need to save your Bitbar Cloud login details into Jenkins. To do this, from the Jenkins main page:

  1. Navigate to Credentials -> System -> Global credentials -> Add Credentials
  2. Select the “Kind” Username with password, and fill in your Bitbar Cloud login details
  3. Note that the Username should be the Bitbar Cloud email you use to login:

image0

Pipeline script - snippet generator

For those users that use befor Freestyle project there is a convenient jenkins pipeline syntax generator:

  1. Create New Item. Choose Pipeline
  2. Navigate to Configure
  3. Click Pipeline Syntax

image1 4. Choose Sample Step - runInCloud - Start a run in Bitbar Cloud

image2 5. Set up parameters like in Freestyle project

image3 6. Click Generate Pipeline Script

image4 7. Use generated snippet in Your pipeline

Pipeline script - simple use case

Read more about all possible parameters below. Note that the example is not a complete Jenkinsfile - just a snippet.

stage('Device tests') {
  steps {
    runInCloud(
      // parameters to start test with...
      cloudUrl: "https://cloud.bitbar.com",
      credentialsId: "my-bitbar-cloud-credentials-01",
      projectId: "12345",
      deviceGroupId: "110",
      appPath: "/path/to/app.apk",
      testPath: "/path/to/tests.zip",
      frameworkId: 252,

      // ... don't just start test, also wait for results!
      waitForResultsBlock:
         [
             downloadScreenshots: true,
             resultsPath: 'results',
             testRunStateCheckMethod: 'API_CALL'
         ]
    )
  }
}

Parameters

List of parameters to go inside the runInCloud() method shown above.

appPath

Type: String
Mandatory: no
Description: The absolute or workspace relative path to the app executable (.apk or .ipa) that should be uploaded and targeted in this test run.
Example: "/path/to/app.ipa"
Default: use the latest app uploaded for this project.

cloudUrl

Type: String
Mandatory: no
Description: The Bitbar Cloud URL this run should target. **Note:** also specify `credentialsId` if you specify this option!
Example: "https://cloud.bitbar.com"
Default: the cloud URL that was specified in the Jenkins /config page Bitbar Cloud section.

credentialsId

Type: String
Mandatory: no
Description: Login credentials ID for the Bitbar Cloud. The `ID` that you specified for your *Username and password* credentials (guide how to do this, above).
Example: "my~bitbar~cloud~credentials~01"
Default: use the credentials (email/password) that were specified in the Jenkins /config page Bitbar Cloud section.

dataPath

Type: String
Mandatory: no
Description: The absolute or workspace~relative path to a .zip file that your tests uses as additional data during a run.
Example: "/path/to/extra~data.zip"
Default: don't use an additional data package.

deviceGroupId

Type: String
Mandatory: yes
Description: Used to specify which device(s) the test run will target. It's a number which you can find by visiting /#testing/devices in the Bitbar Cloud, and looking to the right where it says "Device Groups". Under each "box" of devices, there is a number "ID 12345" or such. That number, 12345 in this case, is the `deviceGroupId`.
Example: `"12345"`

downloadScreenshots

Type: boolean
Mandatory: no
Description: Specifies whether or not to download the screenshots (if any) after the test run completes. **Note:** also specify `waitForResults: true` if you set this option to `true`, otherwise it won't have any effect.
Example: `true`
Default: `false`; don't download screenshots (only logs) after the test run.

failBuildIfThisStepFailed

Type: boolean
Mandatory: no
Description: Specifies whether the `runInCloud` build step should cause the whole Jenkins build to fail if there is an error. You can use this option to allow your Jenkins build to pass, even if an error occurred in the Bitbar `runInCloud` step. **Note:** this has nothing to do with whether or not the tests in the cloud pass; it's only concerned with whether the step executes successfully or encounters exceptions.
Example: `true`
Default: `false`; don't fail even if there is an error.

forceFinishAfterBreak

Type: boolean
Mandatory: no
Description: Whether or not to abort the Bitbar Cloud run after waiting for the results for a set amount of time after the run started. If set to `false`, the cloud run is allowed to finish on its own even after the Jenkins job stopped waiting for it. **Note:** if `true`, requires `waitForResults: true` and `waitForResultsTimeout` to be set.
Example: `true`
Default: `false`; don't abort the cloud run after waiting for it to finish.

hookURL

Type: String
Mandatory: no
Description: URL to which the Bitbar Cloud will send a `POST` request when the test run finishes. For obvious reasons, this address needs to be accessible by the global internet, or have the Bitbar Cloud whitelisted. This can be used by enterprise systems that need to react to completed test runs.
Example: "https://example.com/bitbar~cloud~endpoint"
Default: "JENKINS_URL/plugin/testdroid~run~in~cloud/api/json/cloud~webhook". This default value can be used if `waitForResults: true` is specified, and `testRunStateCheckMethod: "HOOK_URL"` is set. The hook URL info is then used by the Bitbar Run~in~Cloud plugin to determine that a test run finished.

The data sent in the POST request is a short UTF8 snippet, with the following format:

testRunId=12345&projectId=45678&status=SUCCESS&deviceRunId=123,456,789

keyValuePairs

Type: String
Mandatory: no
Description: Additional environment variables provided to the test run. This can be used for advanced logic in test cases, where the test might behave differently depending on some value. Key~value pairs should be semicolon~separated (`;`) and key and value should be colon~separated (`:`)
Example: "SOME_PARAM:some~value;ANOTHER_PARAM:another~value"
Default: "" (empty string)

language

Note: Android only

Type: String
Mandatory: no
Description: The locale (language) that should be set on the phone when the run starts. Has to be a locale supported by Android.
Example: "zh-TW"; Chinese, Taiwan
Default: "en-US"; English, United States

notificationEmail

Type: String
Mandatory: no
Description: Email address that'll receive notifications about the test run when it finishes. Emails are sent by the Bitbar Cloud, not by Jenkins.
Example: "firstname.lastname@example.com"
Default: the Bitbar Cloud account email that's being used for this Jenkins build

notificationEmailType

Type: String
Mandatory: no
Description: Defines what types of events should trigger an email to the email address specified by `notificationEmail`.
Default: don't send emails

Options:

  • “ALWAYS”; send an email after the test run, regardless of whether it passed or failed
  • “ON_FAILURE”; send an email when a test run fails
  • “”; (empty string) don’t send emails

projectId

Type: String
Mandatory: yes
Description: Defines which Bitbar Cloud project this test run should be included in. By extension, this also selects the target platform Android/iOS. This value is a number that you can find by going to "Projects" in the Bitbar Cloud and reading the URL when you selected a project. For instance, if the URL in the browser is "https://cloud.bitbar.com/#testing/projects/12345", then the project ID is "12345".
Example: `"12345"`

resultsPath

Type: String
Mandatory: no
Description: The absolute or workspace~relative path to a folder where results should be downloaded after the run completes. **Note:** requires `waitForResults: true` to be specified or this field won't have any effect.
Example: "./my~results"
Default: ""; Jenkins workspace

scheduler

Type: String
Mandatory: no
Description: Defines in what order tests should be run in the Bitbar Cloud, in case multiple devices are targeted.
Default: "PARALLEL"

Options:

  • “PARALLEL”; tests will execute on multiple devices in parallel.
  • “SERIAL”; tests will run sequentially on devices, one device at a time.
  • “SINGLE”; the cloud will pick one device from the group and run the tests only on that device.

screenshotsDirectory

Note: Android only

Type: String
Mandatory: no
Description: The location on~device from where test screenshots should be pulled after a test run.
Example: "/sdcard/screenshots"
Default: "/sdcard/test~screenshots"

testCasesSelect

Only applies to projects of type “Android Instrumentation” - Espresso tests

Note: requires testCasesValue to be set to something meaningful

Note: Android only

Type: String
Mandatory: no
Description: Value specifies a way (package/class) to narrow down the test cases that should be run.
Default: "PACKAGE"

Options:

  • “PACKAGE”; only execute test cases defined in a certain package.
  • “CLASS”; only execute test cases defined in a certain class.

testCasesValue

Only applies to projects of type “Android Instrumentation” - Espresso tests

Note: Android only

Type: String
Mandatory: no
Description: If `testCasesSelect` is specified, this parameter defines the package/class by which to filter test cases.
Example: "com.mycompany.espressotests.AuthenticationTest"; An example of a class that might be defined in your tests.

testPath

Type: String
Description: The absolute or workspace~relative path to a test package that should be uploaded and run in the Bitbar Cloud. Most often a .zip or .apk file. For AppCrawler projects, this parameter should be omitted.
Example: "/path/to/my/tests.zip"
Default: use the latest test package that was uploaded to this project.

testRunName

Type: String
Mandatory: no
Description: Custom name for the test run in the Bitbar Cloud.
Example: `"My test run"`
Default: by default, the cloud will assign a name like `"Test Run 45"`, depending on the number of previous runs in the cloud project.

testRunner

Only applies to projects of type “Android Instrumentation” - Espresso tests.

Note: Android only

Type: String
Mandatory: no
Description: The test runner class that will execute Espresso tests. Further reading at: https://developer.android.com/reference/android/test/InstrumentationTestRunner
Example: "android.support.test.runner.AndroidJUnitRunner"
Default: "android.test.InstrumentationTestRunner"

testRunStateCheckMethod

Type: String
Mandatory: no
Description: Specifies the method by which the plugin will check if the Bitbar Cloud run has finished.
Default: `"HOOK_URL"`

Options:

  • “API_CALL”; the plugin will continuously poll the Bitbar Cloud to see if the run has finished.
  • “HOOK_URL”; read more about this option by the hookURL parameter.

Note: requires waitForResults: true to be set.

testTimeout

Type: String
Mandatory: no
Description: Value in seconds, of how long the cloud test can run before it is forced to shut down.
Example: "3600"; one hour
Default: "600"; ten minutes

waitForResults

Type: Array
Mandatory: no
Description: Specifies whether or not the build step should wait for results from the test run, or merely launch it. If present, the jenkins build step won't finish until the Bitbar Cloud tests do.
Default: absent

waitForResultsTimeout

Type: String
Mandatory: no
Description: Value in seconds, specifies how long the build step should wait for cloud results. **Note:** requires `waitForResults: true` to be set. Also, unless `forceFinishAfterBreak: true` is set, setting `waitForResultsTimeout` won't abort the test run, it will just silently stop waiting.
Example: "3600"; wait for up to one hour after the test run was started
Default: wait indefinitely

withAnnotation

Only applies to projects of type “Android Instrumentation” - Espresso tests

Note: Android only

Type: String
Mandatory: no
Description: Run *only* test cases that have a certain Java annotation specified.
Example: "com.android.foo.MyAnnotation";
Default: ""; don't filter by annotations but instead run all test cases

withoutAnnotation

Only applies to projects of type “Android Instrumentation” - Espresso tests

Note: Android only

Type: String
Mandatory: no
Description: Do *not* run test cases that have a certain Java annotation specified.
Example: "com.android.foo.MyAnnotation";
Default: ""; don't filter by annotations but instead run all test cases