Testmate Setup and Administration

Owned by Zane England
Last updated: Jul 30, 2024 by Glen Steedman
17 min read

Contents

Testmate Setup

The setup chapter describes the required services and apps for Testmate to operate

Firewall Rules

Required Services

To setup Testmate, Log in to Yarnman, open Administration App->Services

Testmate will require three services described below

  • Testmate App

  • Interconnect Service x 2

  • Terminator Service

From Top hand corner on services screen, select Testmate App from drop down

Define Service Name, select Yarnman Node and Authentication policy from drop downs and optionally Customer to be associated → Submit

Select Services Menu → Select Proxy Service

From Proxy Configuration Page, select ‘Service Routing’ from top right corner

Select ‘Add Yarnapp' and from drop down select the Testmate App ->Submit

The Testmate App has now been added and may be selected from top Right Drop down

Testmate requires 2 x Interconnect services. By default, there will always be one interconnect service running in Yarnman. To create a second service

Go Services → ‘Add Service’, select 'Interconnect Service

Name service, select Node/Arm, then Submit

Terminator service is created from Services → ‘Add Service’

Select Node/Arm and Submit

 

JTAPI node Setup

Any test_mate node that user JTAPI requires downloading the jtapi.jar from CUCM – this is required for test_mate to be able to communicate with CUCM

  1. Browse to administration application and select the required node

  2. Download and restart jade berlin service – it will ask what CUCM IP to download from. Enter the IP and OK

This is the error seen if jade-berlin is not setup and test_mate tries to run

Error: connect ECONNREFUSED 127.0.0.1:6666 ERROR Found test-run marked for this service that is "processing", marking test-run "failed" and aborting its tests.

 

CUCM Configuration

Yarnman CUCM app user requires

  • Standard CCM End Users

  • Standard Admin Users

  • Standard CTI Enabled

  • Standard CCM Server Monitoring

  • Standard CTI Allow Control of Phones supporting Rollover Mode

  • Standard CTI Allow Control of Phones supporting Connected Xfer and conf

  • Standard AXL API Access

Standard AXL API Access Role may need to be added to a custom Access Control Group, an alternative approach is to use

Standard CCM Super Users

Interfaces

Before initiating Testplans, Interfaces to CUCM needs to be configured. This is described in Yarnman Setup Guide under interfaces chapter Yarnman Administration | Interfaces

JTAPI Remote Setup

  1. You will require a CTI route point with 2 DN's (lines) each with external number mask (e164). The CTI route point must be associated with the Testmate User

  2. Click on CUCM instances then select the CUCM interface that the Remote (CTI Route Point) is on and add it

 

Administration

Test Plans - Overview

All test plans can be viewed on overview page (click on 'Test Plans' on the left).
Overview shows a few rows (tests), but more rows can be viewed by clicking "Load More Results" or "Load All Results" (directly underneath the overview).
Test plans can be filtered by name, customer (if linked to a certain customer), type, test, last modification user and date. To do so, click on the corresponding arrow next to the field name and select choice from the drop-down bar.
In order to see the test results in excel format, click on "Download Results" underneath the displayed tests. An excel file download will start, which you can then open directly.
Clicking on the Test plans name will bring you directly to that Test plans details.
To create a new Test plan, click on "+ Create Test Plan" in top right corner.

Dialplan

The Dialplan test will perform testcalls from a selected phone, using configured CSS’s, to the PSTN to numbers with automatic answer or alternatively to another phone via back to back trunks. A sample configuration to set up a trunk to CUBE using TCL script for test may be found below in Testmate Configuration section below

Other Prerequisites for running a Dialplan test is that CSS Groups (Required CSS’s to be tested), Number Groups (Numbers to be used for the tests) and Optionally Geographic Locations have been configured. See below in Testmate Configuration section

On completion of the tests a report is displayed and may be downloaded with the results of the tests

Select "+ Create Test Plan" in top right corner. Following page opens:

  • Enter a name for the new test plan.

  • Select requested type of test plan "Dialplan" and press 'Submit' underneath the drop-down-menu.

  • (Pressing "Cancel" will bring you back to the 'Test Plans Overview'.)

  • After selecting 'Submit', the following page is displayed:


You can now specify the inputs for the test plan.
Select the following:

  • CSS Group

  • Number Group

  • Geo-Location (optional)

The Test Timeout Options settings generally does not need to be modified but may be required to be adjusted if timeout issues occur:

  • After Connect Failure

  • After Connect Success

  • JTAPI Connect Command

  • Pressing "Save" will save all changes made to the Dialplan test plan, that you have made.

  • Pressing "Refresh Tests" will update data for this very test, that might have been changed in existing CSS or Number Groups before.

  • Pressing "Start" will bring you to the following screen:

  • First, select a CUCM Instance.

  • After that, select the corresponding Device Pool.

  • Then select Device to be used for the test, Registration status of the selected device will display once selected.

  • Select the Line to be used and if to be scheduled, pick a date and time for the test (once you have clicked on the 'Schedule Test-Run' field, a calendar will open and after you picked a date, take you to pick a specific time automatically).

  • (Pressing 'Cancel' will bring you back to the previous page, where you can select Test Timeout and Test Generation Options.)

  • Pressing "Start" will take you to the following screen:




You can now find your test under "Test Runs" on the very left of the page:


To view a report of the Dialplan tests, select the Report tab. The report may be downloaded by selecting the ‘Exporting Report XLSX’ button for report in Excel format or ‘Exporting Report CSV’ button for CSV format:


Site Migration


Click on "+ Create Test Plan" in top right corner. Following page opens:

Enter a name for the new test plan.

  • Select "Site Migration" and press 'Submit' underneath the drop-down-menu.

  • (Pressing "Cancel" will bring you back to the 'Test Plans Overview'.)

  • After clicking 'Submit', the following page opens:



First you select the corresponding Test Devices, such as CUCM Instance and Included Device Pools (one or more are possible).

  • You can also enter, how many devices you want to test (percentage).

  • You can then specify (under 'Internal Call Test Cases'), what type of call / action you want to actually test.

  • You can also select a Remote Endpoint (External Call Test Cases) or Media Terminator (Test Case Mixins), if required (if you do, you can choose more options for this endpoint).

  • You also have several other options to choose from (under Test Case Mixins), if requested.

  • Pressing "Save Only" will only save the Site Migration, but not do anything else.

  • Pressing "Cancel" will take you back to the Test Plans Overview.

  • "Delete Test Plan", marked in red on the right side of the page, will delete the whole Site Migration you created.

  • Pressing "Save and Run" will bring you to a review of your Site Migration Test (before you actually run it), that looks as follows:



You can then select a date and time for the test (once you have clicked on the 'Schedule Test-Run' field, a calendar will open and after you picked a date, take you to pick a specific time automatically).
Pressing "Cancel" will bring you back to the previous page, where you can select your Site Migration Test settings.
Pressing "Start Test-Run" will bring you to the following screen:

You can now find your test under "Test Runs" on the very left of the page:




Results view:

PSTN Load Test


Click on "+ Create Test Plan" in top right corner. Following page opens:

  • Enter a name for the new test plan.

  • Select requested type of test plan and press 'Submit' underneath the drop-down-menu.

  • (Pressing "Cancel" will bring you back to the 'Test Plans Overview'.)

After clicking 'Submit', the following page opens:

  • Select the source and target endpoint

  • Number of calls

  • Codec

Note, that the endpoints (CTI route points) need to be added to Testmate in the CUCM instances configuration.
There are options for:

  • Disable DTMF identification

  • Continue calls on error

  • Timeout for call talking

  • Timeout for call identification

Post Verification Testing (PVT)


Click on "+ Create Test Plan" in top right corner. Following page opens:

  • Enter a name for the new test plan.

  • Select requested type of test plan and press 'Submit' underneath the drop-down-menu.

  • (Pressing "Cancel" will bring you back to the 'Test Plans Overview'.)

  • After clicking 'Submit', the following page opens:

 

  • You can then select your options for Internal Call Test Cases.

  • You can also select a Remote Endpoint, if required (if you do, you can choose more options for this endpoint).

  • You can also select options for Polycom VVX Phone Test Cases, Mobile Gateway Test Cases, Manual Test Cases and Test Case Mixins.

  • You can then select a date and time for the test (once you have clicked on the 'Schedule Test-Run' field, a calendar will open and after you picked a date, take you to pick a specific time automatically).

  • "Delete Test Plan", marked in red on the right side of the page, will delete the whole PTSN Load Test Plan, you created.

  • (Pressing "Cancel" will bring you back to the 'Test Plans Overview'.)

  • Pressing "Save Only" will save the test plan, but not run it. You will already receive an error message here, if you haven't selected enough devices (minimum of 2 devices).

Select Devices to use for the test

System Acceptance Testing


Click on "+ Create Test Plan" in top right corner. Following page opens:

  • Enter a name for the new test plan.

  • Select requested type of test plan and press 'Submit' underneath the drop-down-menu.

  • (Pressing "Cancel" will bring you back to the 'Test Plans Overview'.)

  • After clicking 'Submit', the following page opens:


You can now select your CUCM Instance and select Included Device Pools (this option will only be selectable, once you have entered a CUCM Instance).

  • You can then set options for 'Internal Test Call Cases', 'External Test Call Cases' and 'Test Case Mixins'.

  • You can also add a Remote Endpoint. If you do so, several more options will open, which you can then choose.

  • "Delete Test Plan", marked in red on the right side of the page, will delete the whole SAT Test Plan, you created.

  • (Pressing "Cancel" will bring you back to the 'Test Plans Overview'.)

  • Pressing "Save Only" will save the test plan, but not run it.

  • Pressing "Save & Run Immediately" will run the corresponding test right away and save the corresponding test plan. It will then take you to the corresponding Test Details.

  • Pressing "Save & Preview" in the SAT Test Plan, will show you a preview of the SAT Test Devices.

  • Pressing 'Cancel' in the preview will take you back to the SAT Test Plan.

  • Pressing 'Regenerate Devices' in the preview will select new devices from the device pool.

  • Pressing "Start Test-Run" will then take you to the corresponding Test Details.

Test Run Schedules

  • All test run schedules can be viewed on overview page (click on 'Test Run Schedules' on the left).

  • Overview now shows all the schedules created.

  • Test Run Schedules can be filtered by name, type, total runs and next execution. To do so, click on the corresponding arrow next to the field name and select choice from the drop-down bar.

  • In order to see the test run schedules in excel format, click on "Download Results" underneath the displayed schedules. An excel file download will start, which you can then open directly.

  • Clicking on the schedules name will bring you directly to that schedule's details.

  • To create a new schedule, click on "+ Add Schedule" in top right corner and select the corresponding test

  • The following page will open:

  • You can now give your schedule a name and set a start and end date and time for your schedule.

  • You can also enter, how frequently you want your tests to run.

  • Under "Configuration", you can select the corresponding test plan, that you are setting the schedule for.

  • You can then "Save Changes" in the upper right corner (once saved, a new field "Delete Schedule" will appear on the right, where you can delete the just created schedule).

  • On the top right it will now also show the next execution date, that you have just set (where it said 'Not Started / Completed' before).

  • You can now also click on "Test Runs" on the upper left, right next to "Schedule":

  • Here you can see the planned Test Runs.

  • In order to see the planned test runs in excel format, click on "Download Results" underneath the displayed test runs. An excel file download will start, which you can then open directly.

  • You can also delete the planned test runs, by clicking "Delete Schedule" in the upper right corner.

  • Clicking on the Test runs name will bring you directly to that test runs details.

Test Run Archiving

Test Runs may be archived as required. To access go to Test Runs menu item, then select ‘Bulk Change Test Runs’

Select either specific Test Runs to archive or All Test-Runs, then select date followed by ‘Archive Test Runs’ button or ‘Delete Test Runs’ to delete selected

Archived Test Runs may be displayed by selecting ‘All’

 

Testmate Configuration

Configuration – CSS Groups


To perform Dialplan tests requires configuration of CSS groups. To configure these, select "CSS Groups" on the menu on the left side of your screen (under "Configuration") and the following screen displays:


You can now see a list of existing previously created CSS Groups. The CSS groups can be added manually by selecting ‘Create CSS Group’ or imported by selcting ‘Import CSS Groups’
The existing CSS Groups may be downloaded and viewed in excel format, click on "Download Results" underneath the displayed CSS Groups. An excel file download will start, which you can then open directly.
CSS can be filtered by Group Name, Calling Search Spaces, Features and the date, when they were last modified. To do so, click on the corresponding arrow next to the field name and select choice from the drop-down bar.
Clicking on the CSS Groups name will bring you directly to that CSS groups details.

On the CSS Group Overview screen, you can a new CSS Group by clicking "+ Create CSS Group" in the upper right corner. The following screen will open:

Give your CSS Group a name, the call type boxes will be automatically populated with ‘CONN’. Change the corresponding codes for each call type as per required result for CSS in question.
When all selections completed, select ‘Submit’


Configuration – Number Groups

In addition to CSS Groups, the DIalplan test also needs Number Groups configured. This is a list, including any PSTN breakout code, of all the target numbers to be tested for each call type.
Click on "Number Groups" on the menu on the left side of your screen (under "Configuration") and the following screen opens:

You can now see a list of existing configured Number Groups.
In order to see the created Number Groups in excel format, click on "Download Results" underneath the displayed Number Groups. An excel file download will start, which you can then open directly.
CSS can be filtered by Label, Country and Numbers. To do so, click on the corresponding arrow next to the field name and select choice from the drop-down bar.
You can view any number group, by simply clicking on the corresponding Label.

To create a new Number Group, select "+ Create Number Group" in the upper right corner. The following screen will display:

Enter a name for the new number group and the corresponding country code (i.e. GER for Germany).
You can then enter the call type, the label (optional), the number and an corresponding geo location (optional). By clicking on the little arrow next to each field name, you can also select more choices for that very field (such as 'clear column', read only' etc.).
In 'Call Type' and 'Geo Location', you can select a value from a drop-down-menu. Label and number, you can enter manually (double-click in the field, to see the drop-down-menu).
When all entries complete, select 'Submit' will bring you to the following screen:


Number grops may also be imported from a CSV file
Clicking on ‘Import' in the upper right corner, will bring you to the following screen:

Select ‘Choose file' , then browse to your configured CSV file with list of numbers. As per instructions, the CSV must at a minimum include callType and number - label is optional but desirable, location is optional, but if not populated requires empty field

Existing Number groups may be exported by selecting 'Export' in the upper right corner, will display the following:

Select "Download as CSV" and then you can open the corresponding file directly.
Clicking "Back" will bring you back to the corresponding number group.

Configuration – Geographic Location


Select "Geographic Location" on the menu on the left side of your screen (under "Configuration") and the following screen displays:

You can now see a list of the created Geographic Locations.
In order to see the created Geographic Locations in excel format, click on "Download Results" underneath the displayed Geographic Locations. An excel file download will start, which you can then open directly.
Geographic Locations can be filtered by Name, Description, Country and the date and name of the last modification. To do so, click on the corresponding arrow next to the field name and select a choice from the drop-down bar.
You can view any Geographic Location, by simply clicking on the corresponding Name.
You can now create a new Geographic Location by selecting "+ Create Geographic Location" in the upper right corner. The following screen will open:

You can now give your Geographic Location a name and a short description and add a country.
Select "Submit"

Configuration – Sample Trunk for Dialplan PSTN simulation

To test Testmate Dialplan without sending call to PSTN, a trunk may be setup to a CUBE that may be configured as per below sample configuration. The CUBE will answer the outgoing call from the CUCM without actually forwarding call till PSTN.

Only CUBE configuration below, the test will also require a CUCM SIP trunk configured in CUCM with CUBE as destination

The configuration uses a Cisco Dev Support sample TCL (Tickle) script - available below followed by sample CUBE CLI config

Sample CUBE configuration

application service robodest bootflash:robodest.tcl param robodest_maxtime-ringback 10 param robodest_mintime-ringback 0 ! dial-peer voice 9004400 voipservice robodestsession protocol sipv2 incoming called-number . dtmf-relay rtp-nte codec g711ulaw ! dial-peer voice 200 voipdescription *** Calls to Primary CUCM*** destination-pattern [2-9]......... (Dial Peer if return trunk required to dest CUCM adjust dial peer as required) session protocol sipv2 session target ipv4:<ip address of target CUCM here> dtmf-relay rtp-nte codec g711ulaw

Firewall Configuration

Local Firewall Configuration 

Configure on each host as required using ufw:  

sudo ufw allow ssh 
sudo ufw allow http 
sudo ufw allow https 
sudo ufw enable

 Additional rules are required on the core node for each arm deployed 

sudo ufw allow from <ip address of arm> to any proto tcp port 5984,5986 
sudo ufw allow from <ip address of arm> to any proto tcp port 6379,6380

Default Terminator Configuration (Testmate) 

sudo ufw allow from any proto udp port 6700:6799 

Configuration – Codec Lists
Click on "Codec Lists" on the menu on the left side of your screen (under "Configuration") and the following screen opens:

You can now see a list of the created Codec Lists.
In order to see the created Codec Lists in excel format, click on "Download Results" underneath the displayed Geographic Locations. An excel file download will start, which you can then open directly.
Geographic Locations can be filtered by Name, Description, Codecs and the date and name of the last modification. To do so, click on the corresponding arrow next to the field name and select a choice from the drop-down bar.
You can view any Codec List, by simply clicking on the corresponding Name.
You can now create a new Codec List by clicking "+ Create Codec List" in the upper right corner. The following screen will open:

You can now enter a name for your new Codec List and a short description.
Clicking into the 'Codecs' field will give you a drop-down-menu of different codecs to choose from.
Pressing "Cancel" will bring you back to the Codec List Overview.
Pressing "Submit" will tell you, that your Codec List has been created successfully. It will now also give you the possibility to delete it again, if you wish. See following screen:

You can now see the created codec list in the codec list overview correspondingly, see below:

Configuration – Call Types
Click on "Call Types" on the menu on the left side of your screen (under "Configuration") and the following screen opens:


Pressing the arrows up or down, will re-order the call types and you will receive an corresponding information, as per screen below:

To create a new Call Type, please press "+ Create Call Type" in the upper right corner. The following screen comes up:

You can now enter a name and a short description for your new call type.
Pressing 'Cancel' will bring you back to the "Call Type Overview".
Pressing 'Submit' will let you know, that you have successfully created a new Call Type.
It also gives you the option to delete the newly created Call Type (and would then bring you back to the Call Type Overview). See screen below:

You can now see the newly created Call Type in the Call Type Overview.

Configuration – CUCM Instances


Click on "CUCM Instances" on the menu on the left side of your screen (under "Configuration") and the following screen opens (1. Left side / 2. Right side):

You can now see a list of the created CUCM Instances.
In order to see the created CUCM Instances in excel format, click on "Download Results" underneath the displayed CUCM Instances. An excel file download will start, which you can then open directly.
CUCM Instances can be filtered by Name, Description, Customer, IP-Address, User Name and Endpoints. To do so, click on the corresponding arrow next to the field name and select a choice from the drop-down bar.
You can view any CUCM Instance, by simply clicking on the corresponding Name. The following screen will open:


You can now see the details of the CUCM Instance.
You can select a CTI Route point and add it correspondingly by hitting the "Add CTI Route Point Remote" button.
The corresponding points you just added, will automatically be saved and now be shown in the list below, as per following screen:



Configuration – Settings


Click on "Settings" on the menu on the left side of your screen (under "Configuration") and the following screen opens:




On this page you can now set and change the time settings for Interconnect Testing, Test Runs and Testmate. Every line has a short description underneath, what it is controlling and what is the default value for this line.
Once you have set all the values you want, you can press "Save Global Settings & Restart Effective Services".
The following screen will now show you, that you have set changes and that all services will be restarted correspondingly:

yarnlab website