The resources for cloud computing are generally made available through applications with web interfaces that provide functionality to allocate, configure, and maintain the resources. Hence cloud computing adds some ambiguity to the systems hosting your applications causing some concerns around performance and security. And for this reason any cloud integration should have a comprehensive performance testing plan.

Firstly have an end to end understanding of the application performance; ranging from knowing the requirements to understanding the impact on the systems. And with cloud some additional unknowns that need to be considered and planned for are response times, impact on the systems and most importantly the shared resources.

Once deployed into the Cloud, the only practical testing model is the external testing model – that is driving load into the application from a secondary location over the internet. This can easily provide information about response time compared to traffic or load. There will be a few hurdles to work out with the impact on the systems, but you can get a pretty good indication if carefully planned and coordinated. To measure and understand the impact of traffic on computer systems you must understand the environment and the infrastructure, generate realistic traffic, analyze the capabilities and the limitation of the system under test, and find the remedy for performance related defects.

The cloud is a shared environment. This must be taken into consideration for performance aspects of your applications that you deploy into the cloud. Few considerations in this regard are, who you are sharing the resources with, how much of resource is being consumed, how it is impacting the performance of your application, and is usage and performance consistent.

Performance monitoring is going to be essential to assure that your application remains responsive. Constant and on-going monitoring is a good way to verify that the performance continues to remain stable. For this you need to monitor over time, watch for degradation, do changes in performance correspond to changes in your application. Unexplained changes in performance may be attributed to, shared resources and cloud provisioning tool

Several factors that can lead to inadequate performance testing are, lack of requirements, lack of understanding the fundamentals of performance testing, misinformation provided by tool and simulator vendors, limited budget or funding and enormity of the task.

The application system evolved as per the organizational and end user need. It eventually became an integral part of business, banking, trading, shopping, and even entertainment, demanding perfect performance. Testing in the cloud adds testing capabilities that were not available in the past like multiple geographic locations, lots of unique real IP addresses, to formulate traffic patterns. Most performance testing clouds are built out in controlled cloud environments. To perform a controlled cloud testing, access to the full systems, know where the load originates- IP Address, geographic locations, control the distribution of the load, use same test system/systems for retesting, perfect coordination of the user traffic, and correlate the load with end-user response times and the impact of the systems under test.

Performance Failures = Lost Revenue

The appalling news of performance failure leading to loss of revenue, emerging every now and then creates a chill in the spine. The reason, in a single word is ‘negligence’.  The reluctance to define performance requirements usually comes from: lack of understanding the performance requirements, time constraints, limited budget, non-existent or incomplete performance specifications and ownership of systems which is not understood. All that we need to do at this juncture is, curb these.

Performance Goals and their importance

The first step in this regard is to realize the Performance Goals and their importance. To achieve the goals we need to describe considerations for a systems performance, break down performance goals into tasks and then emphasize the importance of making performance goals part of the IT software requirements definition.

Planning the performance goals should be the second move. In this stratum we need to realize the performance requirements in terms of Consumer Requirements, Data, Traffic and Business, plan the Requirements Phases and Responsibilities, and assign the roles and responsibilities accordingly.

Inter team communication is the third move, wherein the objectives should be set and requirements defined,  but sometimes the business needs change. Note that the key to successful implementation of performance testing is continual communication among team members – manage change accordingly, because an effective mechanism of communication and collaboration allows the team to make good performance related decisions, whilst changes are easy to make. The methods you use to communicate plans, priorities and changes are not that important as long as you are able to adapt those changes without requiring significant re-work.

But before sorting out anything, we need to have an anatomy of the process- the performance testing process, which includes, Requirements, Discovery, Test Planning, Automation, Testing, Measurements, Analysis and Tuning and Re-Testing.

With this skeletal in the planning schedule, both the product and the process emerge as flawless, hence giving a breath of relaxation.

Key factors to be considered in managing outsourcing relationship are, business drivers, different outsourcing test scenarios, and potential expectations from the client. Lack of performance measurement framework can often lead to the below situation.

• Excessive communication
• Micro management by client
• Supplier spends too much time in reporting
• Every stakeholder feeling out of control

There is a strong need for performance measurement framework that can prevent the above potential mishaps. A Performance Measurement Framework (PMF) is an essential part of any test-outsourcing project. It defines the boundaries of the project in terms of the services that the service provider will offer to their clients, the volume of work that will be accepted and delivered, and acceptance criteria for responsiveness and the quality of deliverables. A well-defined PMF correctly sets expectations for both sides of the relationship and provides targets for accurately measuring performance against those objectives. At the heart of an effective PMF is its performance metrics. During the course of the test outsourcing engagement, these metrics will be used to measure the service provider’s performance and determine whether the service provider is meeting its commitments or not.

‘5P’ performance measurement framework is introduced to establish accountability on both the sides (Client and Vendor), jointly manage and achieve a win-win situation. The 5P’s are - product, project, process, people and price. ‘5P’ performance measurement framework is easy to apply, proven and practical in nature and was developed based on knowledge and experience. This framework provides collection of metrics to choose from multiple dimensions of the testing engagement namely project, process, product, price and people. Metrics can be provided that can cater to wide variety of testing engagements namely test automation, performance testing, certification testing, functional system testing, white box testing, security testing etc.,

Sample metrics against each category are mentioned below to give you some ideas on direction to think about.

• Project: Test effort Vs development effort, Productivity.
• Process: Cycle time improvement, defect leakage index.
• Product: Time to find a defect, test coverage.
• People: Attrition, average experience.
• Price: $ amount saved, Price variance.

Vendor and client have to understand the business drivers of the testing engagement. Identification of the key results areas have to be done based on the business drivers. Appropriate test metrics selection happens based on the nature of the project, test types, test phases etc., Metrics selection is based on the principle that every metric in isolation gives information to track business drivers. The idea of multiple measurements is to put together a pattern of information that collectively gives a complete and accurate picture of the system. Install a metric system in place that allows you to collect the needed information to measure and analyze information and steer projects into the right direction.

Benefits of the model:

Implementation of proper performance measurement framework for outsourced test activities has numerous benefits. Few of them are listed below.

• Helps companies manage their test service providers in an optimal manner for win-win relationships.
• Proper visibility on the return on investment by the outsourced service provider.
• Consideration of all the quality measures into account while analyzing the performance.
• Introduction of a standard evaluation process across the company.
• Identification of the potential risk areas that affect the productivity of the test team.
• Higher level of abstraction with carefully choosen test metrics and the presentation format enabled management to spot the critical issues quickly.
• Past history of the results from the framework can help the success probability of future projects.

As trading has become electronic, trading applications need to be reliable and effective; emphasizing more on the speed of delivery with peaks and troughs in demand. And the answer to these demands is Performance Testing, a beneficial testing strategy in terms of cost -the cost of fixing a performance defect (flaw) is many times more compared to fixing a functional defect which could just involve rewriting a few lines of logic (code).
AppLabs, with its wide spread experience in this arena, has conceptualized certain strategies to help clients in addressing all the risks related to performance testing of trading applications. This successful test strategy hinges on the following aspects:

Fit for ‘purpose’
Ability to address all the pain points

AppLabs has implemented ‘TRADES’ Heuristic Model to help strategize performance testing for trading applications. This model promotes a structured ‘thought process’ whilst taking into account critical drivers; ‘business needs’, ‘performance objectives’, ‘risk’ and ‘budgets available’, thereby helping to choose the ‘right’ test types and techniques to be employed.

A careful analysis of the technology and application stack that is within its control can help identify the performance bottlenecks and come up with recommendations for right infrastructure, configuration and tuning. Challenges in building a real time performance test environment may be overcome through emulators and simulators. Additional probes may need to be written to augment the performance metrics capture capability of Commercial Off The Shelf (COTS) performance testing tools. Only competent performance test analysts can draw inferences from the performance test results and come up with recommendations to meet the performance objectives.

AppLabs, with a decade of experience in performance testing, is a leader in this space. Our extensive expertise in market tools, simulators, emulators and custom scripting needs of complex performance testing engagements has provided business benefits to a number of banking, financial services and insurance enterprises.

The Dark Pool Trading System is mainly used by the institutional traders who trade in large volumes. The system helps institutional investors in getting more liquidity and less transaction cost. They also help fund managers’ strategies that are best implemented with the use of algorithmic trading. In spite of these benefits, few issues like inadequate price transparency, regulatory requirements and uniform information access to all kinds of investors are still debatable.

Considering the complexity of these trading systems in terms of technology, speed, functionalities and system performance, it is very important that the functional testing along with the gateways testing and performance testing need to be done.

Functional Testing:

As these applications are multi product, multi segment and multi user, an end to end functionality testing of the application is required, wherein each functionality needs to be identified, smoke and unit test cases of each needs to be developed. As these trading systems are linked and interdependent, authoring integrations test cases is needed. To ensure that the existing functionalities are not affected, regression testing on a continuous basis is needed.

Gateways (Interface) Testing:

Dark Pool Trading systems communicate with various internal and external systems and it is very important that the communication between these interfaces is very fast without any data loss. As these systems are mainly used by the institutional traders and bulk executions occur in a short span of time, interface testing plays a crucial role in validating the data flow.

Testing of the interfaces basically ensures the existence of communication between the client and server. The validation of backend messages is done by connecting to the various FIX gateways like DMA, IDCE, etc. The various message tags are also validated for the protocols like FIX (Financial Information Exchange), SIP (Securities Information Processing) and ITS (Inter market Trading System) in the respective gateways.

So all the gateway testing is to be done using any of the protocols (usually FIX is used for trading applications) to ensure smooth flow of communication between various gate ways associated with the dark pool trading systems.

Performance Testing:

The dark pool trading systems are used mainly by the large investors whose volumes are very high. They want their orders to be executed as and when the opportunity exists for their strategy implementation. As these applications are accessed simultaneously by multiple users, the load testing of the same must be done before implementing them in the production environment.