Open

Contents

Overview

The Testmate platform has been designed to provide tightly integrated and relevant test automation for Cisco Unified Communications. Specific test plans have been built for the following functions:

• Commissioning Testing (new customer)

• Post Migration (new sites)

• Post change (assurance BAU)

• System Accreditation (new CSR)

• Stress Testing

• IP Readiness Assesment (Prattler)

Open

Testplans

Dial Plan

Dial plan testing in HCS environments is a complex and time-consuming task, which becomes significantly more complex and time consuming, when new countries and their associated dial plans are added to the HCS solution over time.
The dial plan test plan provides automated testing, that enables a robust and consistent test regime, that ensures a quality user experience, before and after planned updates and changes in the HCS dial plan.
Use Cases

• Testing, to assist with verification of the new dial plan, used by CUCDM 10/VOSS4UC/Kurmi.

• Reduce resources required for testing for the dial plan.

• Minimize potential human errors, by using test automation.

Migration

The migration capabilities are made of the following components:

• Site Migration

Enables automated testing by deployment engineers during a migration, to automatically test, that the porting has been successful and that media services and basic calling is functioning correctly.

• PSTN Load Test

Enables stress testing with a configurable number of calls between two CTI objects, controlled by Testmate, to ensure sufficient capacity is available, which is not practical to test, using other means.

• Prattler

Providing a mechanism, to test Jabber and media services beyond the network boundary, to support assurance and deployment teams, by testing the required DNS records, certificates and media paths for Jabber.

Site Migration Test Plan

The site migration test plan has been designed for testing and verification of porting and site migration activities. During the setup and execution of this test plan, Testmate takes control of registered devices in the nominated device pool and orchestrates a variety of tests between phones and the JTAPI remote.
This test plan focuses on the test of site specific PSTN and media services and is intended for use by deployment engineers, to be able to quickly perform pre-site commissioning testing in an automated manner.
Use Cases

• Samples DID's from all ranges in device pool(s) under test and verifies, if both in- and outbound calls are correct.

• Basic media service testing (MOH, MTP, CONF, transfer)

• Extension mobility

• Media path verification (packet statistics)

• Re-run test run, following the resolution of issues

• Public work space testing

• Simulated emergency call testing

PSTN Load Test

The PSTN load test plan enables load testing and verification of a number of calls, that can be established between two CTI route points. The CTI route points can be located on either two different clusters, connected by either PSTN or private trunks, to determine the maximum number of calls supported on the trunking infrastructure – or for testing Call Admission Control limits between two different locations on the same cluster.
Use Cases

• Verify number of concurrent calls (channels) between two JTAPI remotes (CTI Route Points) – these can either be across the PSTN, internally (CAC testing/AAR) or across private trunks.

• Test SIP Connect Services

Prattler

Supporting Jabber beyond the network boundary has become a complex and time-consuming task, due to the limited control of the environment. When issues arise, the traditional methods of troubleshooting require a high level of expertise, operational resources and direct involvement by the end user, with issues normally taking several interactions to resolve. This leads to diminished customer satisfaction with the solution.
Providing effective assurance faces the same challenges beyond the network boundary, to determine, if call quality is being affected by network degradation or impairments.
Use Cases

• Reduce resources, required for troubleshooting Jabber.

• Improve customer satisfaction with soft phones.

• Minimize potential human errors, by using test automation.

• Provide a mechanism for site pre-commissioning testing, for Jabber to ensure, that the environment is configured correctly.

The solution is broken down into three separate components:

• Client-Side Testing (Prattler)

This component provides an application, that can be downloaded by end customers, to perform testing for reactive support. This does not require any infrastructure, to be deployed in the customer network.

• Ad-Hoc Media Probe (Prattler Remote)

Extends the client-side, to enable media testing end to end.

• Problem Report Analytics

Provides a mechanism, to easily provide Jabber problem reports to assurance teams and performs automated log analysis.

PVT test plan

The PVT test plan is targeted for post change verification testing, that allows for a selection of affected shared services and customer specific systems, to determine, what systems require post change verification testing. This test plan will be modular, which allows adding future extensions, to support additional test cases, as well as backup, health checks and functionality checks of impacted systems.
The first step of the PVT test plan will be, to select customer systems and nominate, what devices to use for testing, then select the test cases required, then select any manual mix-ins, then select the JTAPI remote to use, then start testing.
PVT also has the manual test case, that allows interactive testing with the Testmate users mobile phone or other external destination, to make and receive calls from, as well as verify call statuses.
Use Cases

• Post change verification for operational teams, to standardize the change management process.

• Centralize testing results and metadata.

• Standardize change management pre-change and post-change tasks and verification.

System Acceptance Testing

The System Acceptance capabilities are made of several modules, that can be added, to extend test coverage. The base and mandatory module is basic calling, that enables control and testing of supplementary services and acts as the primary test integration point for calls to and from other remotes.
Goals and Objectives

• Automate testing activities for testing and accreditation teams.

• Enable automated testing, prior to upgrades in a lab environment, if upgrades are required outside the standard upgrade lifecycle (customer specific bugs).

• Provides the basis of the Orcamate upgrade process, where environments are tested both before and post upgrade.

Basic Calling

Testing of phone features for all possible scenarios grows exponentially, as new features are added. Testmate provides a simple interface, to add mix-ins, that add relevant test cases.
These mix-ins include:

• Dialing discovery, that dynamically works out, what possible ways there are to dial between endpoints (e164 / FNN / Local / on-net / URI / etc).

• Device selection, to ensure, that at least one of each device type is tested for each test case.

• Variable verifications automatically determine the appropriate verification steps for each test case (DTMF / media path / media services / codec).

These basis functions can, when combined with targeting remotes on other different CUCM clusters, have test coverage of a large number of scenarios and configurations.

• Internal Calls

• Inter Cluster Calls

• PSTN Calls

• SIP URI Calls

The remote enables making end to end calls between two separate systems. It can either use CTI RP or dedicated test phone(s), to originate and terminate calls.
Use Cases

• End to end call path verification

• Inbound call testing

• CLIP / CLIR verification

• Call transfer testing

• Call forwarding testing (Internal and External)

• DTMF verification

• Call hold and resume testing (MOH)

• Call Park

• Call Pickup

• Verification of numbers during call states (Called, Connected, Calling and Redirecting)

• Long running calls (SIP Refresh)

Remotes

It is possible, to expand the test coverage, using the following remotes, that include other destinations:

• JTAPI Remote

The JTAPI Remote is included with Testmate and utilizes a CTI route point on CUCM as the external party for test calls. This is combined with the Yarnman Terminator service for sending and receiving RTP.

• Video Control Remote

The Video Control Remote enables testing of Cisco TC and CE endpoints for video scenarios, to ensure that video and content streaming functions correctly. This is also required for the video specific test cases.
This remote can be placed in the customers network.

• Polycom Remote

The Polycom Remote enables the incorporation of Polycom VVX phones, registered to TIPT (Broadsoft), for making and receiving calls, using the Polycom webservice API. This will also enable media statistics.
This remote can be placed in the customers network.

• Cloud PSTN Remote

The Cloud PSTN Remote will enable making and receiving calls from a cloud VOIP service (Twilio), to enable testing of various numbers on demand.
This remote would be placed in the model network.

• Mobile Gateway Remote

The Mobile Gateway Remote will enable making and receiving calls, to use the mobile network, multiple SIM cards can be added to the gateway, to enable testing of multiple carrier networks.
This remote would be placed in the model network.
The following diagram provides an overview of how Testmate PVT would be deployed with the various remotes.

JTAPI Remote

The remote enables making end to end calls between two separate systems. It can either use CTI RP or dedicated test phone(s), to originate and terminate calls
This enables:

• End to end call path verification

• Inbound call testing

• CLIP/CLIR verification

• DTMF verification

• Verification of numbers during call states (Called, Connected, Calling and Redirecting)

Video Control Remote

Video remote takes control of video endpoints, using the xCommand API via HTTPS, that allows for greater control of video endpoints, which offer functionality, not available via JTAPI.
This enables:

• Making calls and receiving video calls.

• Adjust CUCM configurations, to test different scenarios, such as BW and SIP settings.

• Checking video negotiates and 2 way media.

It will provide a configurable number of endpoints and destinations, that includes DN, FNN, e164 and URI destinations. The following table shows the various video destinations, combined with video specific test cases.

The video control remote requires direct HTTPS connectivity from the Arm to the endpoint.

Polycom VVX Remote

The Polycom VVX uses the phone webservice, to make and receive calls directly from the phone.

Cloud PSTN Remote

The cloud PSTN remote enables testing, using any numbers, offered by Twillio or Tropo. These numbers can be added as required.

Mobile Gateway Remote

The mobile gateway remote allows for end to end testing of mobile calls; multiple mobile services can be added to the gateway and used for making and receiving calls.

Virtual Endpoint

The virtual endpoint enables testing registration of SIP services, registering through the service provider NAT to CUCM. This allows for assurance teams, to test customers systems, without requiring the use of customer endpoints. The media path requires the use of an MTP on CUCM, to enable end to end RTP.