Gall’s Law: “A simple system may or may not work. A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.” - John Gall, Systemantics

This is a true principle that holds true to sports, education, family life, computer systems, and even software development and testing. If we don’t have a working simple system in any of these areas—if we haven’t mastered the basics, the core—then the difficult, challenging, and more complex challenges will be beyond our capability. We can’t go from zero to complex without paying the price beforehand to master the simple basics. It is not saying that we seek complexity. I believe that in any situation we should seek the minimum level of complexity that meets our needs.

It especially applies to software development and testing. When developing a web application, it is common practice to start with a simple working and tested core of functionality and grow in complexity from there. As we go, we test functionality, performance, and security along the way. We add a bit here and a bit there as we go, until we are satisfied with the final product. We don’t start with the complex and save our testing to the end. We release early and often, testing as we go.

The implications of this principle on software development and testing are profound. Start by creating a simple system or process that works for you and then add to it incrementally until it is just complex enough to meet all of your goals.

The intensity of global warming is aggravating with every passing day, and so is the environmental condition. Even the IT Industry is reputed to issue 2% of the world’s CO2 emissions (the same as the much maligned airline industry). Hence emerges the need to reduce energy wastage, CO2 and other greenhouse gas pollution. The IT world, aware of its significant and growing impact on carbon emissions and climate change, must take steps to reduce this trend.

1,000 PCs running 24/7 without energy saving modes activated can consume as much as £70,000 worth of electricity in a year. Each year, enterprises waste nearly $4 billion powering devices that are not in use. The network traffic and the data centre too are the contributor to energy waste. As a result, the IT world must look at ways of reducing the output of harmful emissions and heat waste.

Poorly defined roles and responsibilities – diluted lines of accountability, development lifecycle chaotic – testing chaos, defect/bug management needs structure, no supporting configuration management processes, 100+ Unix environments – under utilised, replicating overnight process utilises huge CPU resource, lead to energy wastage, delays, resource misuse and environmental impact.

A number of options are available in this field and ever evolving technical solutions are coming forward, such as multi-core processing, virtualisation, data compression solutions, etc. These solutions not only reduce the environmental impact, they also reduce the financial impact. Here are certain testing approaches to reduce the impact.

• Pertaining to the project necessities, travel only when necessary, utilize communications technology; video-/tele-conferencing, minimize paper trails, use electronic media and define risk criteria.
• When handling Test Management, commission to sustainable, reusable environments, liaise closely with development community and be focused; perform impact analysis of change and direct testing accordingly.
• When dealing with the Test Process conduct an environmental analysis and design, order work methodically by maximising efficiencies, and adhere to structured testing methodology (Static Testing, Business Process Testing, V-Model and so on).
• To ensure environmental concerns the existing testing roles should be evolved. The roles should be captured during requirements gathering phase, they should be prioritized accordingly, alongside functionality and other business requirements, suitably tested (throughout development lifecycle), outstanding concerns are properly assessed, prioritised, risk managed and scheduled for future release.

Having a more energy efficient IT infrastructure and highly focused and driven people will minimize the organisations exposure to energy price increases and subsequent limitations imposed on the business.

Many leading venture capital firms from the U.S. are willing and have begun to bet on investing in niche Indian players. Consider this, till five years ago, an investment of $7-8 million in a niche services company, operating on an outsourcing model from India, was considered big. Today, the average size of an investment a leading venture capital company makes, is anywhere between US$20-30 million.

This growing trend among VCs of investing in ‘niche players’, gives me a sense of satisfaction for what I ventured out to do six years back, which was to set up a company focused on ‘testing and quality management’.

With an increasing number of organizations automating their business processes, there has been a growing need for these companies to test the efficiency of their software and IT systems. More than often, they prefer to engage the services of an independent testing and quality management company, to get an unbiased appraisal. As an entrepreneur, I realized the potential of the testing market and set up AppLabs in 2001, to capture the exciting and tremendous business opportunity that ‘testing’ provided. And since then, AppLabs has been growing significantly.

Contrary to what comes to mind, when thinking of a ‘specialized niche company’, AppLabs hasn’t been built around a small opportunity. In fact, it is possible to build a $ 1 billion company in this niche segment of ‘testing and quality management’. Let me illustrate my point.

Once a strong value proposition has been built to focus on a particular ‘niche segment’, one needs to identify an opportunity that is scalable. Here, I would like to clarify that, while in the Indian context, scalability often refers to how quickly a company can increase its headcount, in the U.S., it is measured by the market opportunity the company has the potential to capture. Hence, it requires determining the quantum of business already being generated by the big players in the ‘niche market’, as well as implementing an effective ‘go to market strategy’ to compete in the market for the business.

Having reached a considerable size in its industry segment, it is also advisable for the niche company to diversify into peripheral or related services. For example, a player in the ‘testing’ market, could look at providing ‘security or certification services’ as well. Also, various niche players with related business interests, could work through ‘partner referral’ programs, which would enable them to offer the entire range of specialized services, required by the customer to meet his business needs. The attempt here is not to expand by becoming an ‘end-to-end player’, but to grow the business, while being focused on one’s core service offerings. Additionally, a niche player needs to invest in and build new competencies in the focused areas. The measurement of success is when your services/products provide ‘true value’ to your customer’s business, which will help to retain as well as increase your client base.

To reinforce the ‘value proposition and branding’, ‘niche players’ periodically, need to get their messages out in the market they operate in. This could also involve ‘a re-branding’ exercise, wherein the company considers changing its logo, tagline, external and internal messaging to its customers and employees respectively. This is also a positive statement in communicating its ‘key differentiators’ from its competitors and/or to maintain its ‘leadership position’ in the industry.

A niche player could also communicate its value proposition to its customers, through new and innovative pricing and packaging. For example, a testing firm could charge for its services appropriately, after analyzing the amount the customer currently spends on technical support (post product deployment). If the spend is minimal, it could result in increased returns for the testing firm. The pricing could also vary according to the number of bugs one finds in the product.

On the packaging side, one could draw up a list of ‘top 20’ functions and charge it as a package, rather than charging for all the services. For example, in the arena of security testing, this ‘top 20’ list could constitute of a set of services deployed to detect security flaws.

A niche player should clearly articulate all the business benefits that would accrue to the customer, by engaging the company’s services, including the ‘financial savings’ in the entire product lifecycle, rather than limiting it to specific services concerned. Experience has taught me, that this is one of the strongest factors that helps win and retain customers.

‘Niche players’ are extremely effective at working closely with their customers to build and maintain long term relationships by innovating and challenging the norm in the industry, thus adding value at the project, program and organizational level. The best organizations in the world want to work with the best, and if you (as a ‘niche player’) can be the best at what you do - and this can really only happen if you focus in one area – then you will have great success.

In conclusion, I would like to add that, the toughest challenge an entrepreneur faces while building a $1billion ‘niche company’, is to convince people to believe in the ‘value proposition of offering only specialized services’. While a niche firm often succeeds in attracting the best talent with the requisite specialized skills, it does encounter challenges to retain them in the face of competition from the big ‘end to end players’ in the industry. Hence, it is imperative for a niche company to reinforce to its employees, its value proposition and belief of being a ‘big player’ in the industry, as well as offer them promising career paths.

Performance Failures = Lost Revenue

The appalling news of performance failure leading to loss of revenue, emerging every now and then creates a chill in the spine. The reason, in a single word is ‘negligence’.  The reluctance to define performance requirements usually comes from: lack of understanding the performance requirements, time constraints, limited budget, non-existent or incomplete performance specifications and ownership of systems which is not understood. All that we need to do at this juncture is, curb these.

Performance Goals and their importance

The first step in this regard is to realize the Performance Goals and their importance. To achieve the goals we need to describe considerations for a systems performance, break down performance goals into tasks and then emphasize the importance of making performance goals part of the IT software requirements definition.

Planning the performance goals should be the second move. In this stratum we need to realize the performance requirements in terms of Consumer Requirements, Data, Traffic and Business, plan the Requirements Phases and Responsibilities, and assign the roles and responsibilities accordingly.

Inter team communication is the third move, wherein the objectives should be set and requirements defined,  but sometimes the business needs change. Note that the key to successful implementation of performance testing is continual communication among team members – manage change accordingly, because an effective mechanism of communication and collaboration allows the team to make good performance related decisions, whilst changes are easy to make. The methods you use to communicate plans, priorities and changes are not that important as long as you are able to adapt those changes without requiring significant re-work.

But before sorting out anything, we need to have an anatomy of the process- the performance testing process, which includes, Requirements, Discovery, Test Planning, Automation, Testing, Measurements, Analysis and Tuning and Re-Testing.

With this skeletal in the planning schedule, both the product and the process emerge as flawless, hence giving a breath of relaxation.

The client, one of the largest and most respected insurance brands in North America, wanted to implement a large transformation program in terms of development, assurance and testing, which was complicated by the number of external vendors (including two Tier One offshore companies, as well as one of the worlds largest System Integrators) and internal teams.

To meet the client requirements, AppLabs, a chosen testing service provider, recommended the introduction of its Test Compliance and Governance Framework service. AppLabs firstly undertook a formal Test Review. This was to assess the client’s existing testing and key test dependencies, identify the gaps and establish a baseline. This provided the client and AppLabs a clear view of the level of testing maturity, and more importantly, this helped in identifying the improvement areas and recommendations to address the production issues. AppLabs then conducted the Test Improvement program, which aided the implementation of the recommendations identified during Test Review by the vendors and internal groups. AppLabs provided a detailed improvement roadmap, and recommended for, implementing a process to ensure adherence to a new Release Management Process; implementing the use of code coverage tools and objective based integration testing for delivering quality software to the validation teams; and identifying appropriate tests for automation. After the Test Review and Test Improvement components were completed, AppLabs implemented the Test Compliance and Governance Framework. This involved four key elements: implementing the Governance Framework; implementing the “Audit/Compliance” processes and templates; implementing the guidelines to measure the ROI for each of the implemented recommendations; and agreeing with the “Audit/Compliance” schedule. AppLabs then undertook the first of the Test Compliance Reviews and continued to provide these across the program at regular checkpoints, resulting in the following benefits:

• 82% increase in test coverage and 91% reduction in test process defects has helped capture 22% more defects during the testing phase, with this the overall cost of the project is reduced by $5,183,640
• 85% reduction of production defects has driven up client satisfaction indices and reduced the overall cost of release delivery by $9,151,260
• 65% increase in authoring of test objectives and a 71% increase in test execution productivity has resulted in reduction of the team size, from 20 to 8, equating to a cost saving of $1,985,000 over the last 2 years

Key factors to be considered in managing outsourcing relationship are, business drivers, different outsourcing test scenarios, and potential expectations from the client. Lack of performance measurement framework can often lead to the below situation.

• Excessive communication
• Micro management by client
• Supplier spends too much time in reporting
• Every stakeholder feeling out of control

There is a strong need for performance measurement framework that can prevent the above potential mishaps. A Performance Measurement Framework (PMF) is an essential part of any test-outsourcing project. It defines the boundaries of the project in terms of the services that the service provider will offer to their clients, the volume of work that will be accepted and delivered, and acceptance criteria for responsiveness and the quality of deliverables. A well-defined PMF correctly sets expectations for both sides of the relationship and provides targets for accurately measuring performance against those objectives. At the heart of an effective PMF is its performance metrics. During the course of the test outsourcing engagement, these metrics will be used to measure the service provider’s performance and determine whether the service provider is meeting its commitments or not.

‘5P’ performance measurement framework is introduced to establish accountability on both the sides (Client and Vendor), jointly manage and achieve a win-win situation. The 5P’s are - product, project, process, people and price. ‘5P’ performance measurement framework is easy to apply, proven and practical in nature and was developed based on knowledge and experience. This framework provides collection of metrics to choose from multiple dimensions of the testing engagement namely project, process, product, price and people. Metrics can be provided that can cater to wide variety of testing engagements namely test automation, performance testing, certification testing, functional system testing, white box testing, security testing etc.,

Sample metrics against each category are mentioned below to give you some ideas on direction to think about.

• Project: Test effort Vs development effort, Productivity.
• Process: Cycle time improvement, defect leakage index.
• Product: Time to find a defect, test coverage.
• People: Attrition, average experience.
• Price: $ amount saved, Price variance.

Vendor and client have to understand the business drivers of the testing engagement. Identification of the key results areas have to be done based on the business drivers. Appropriate test metrics selection happens based on the nature of the project, test types, test phases etc., Metrics selection is based on the principle that every metric in isolation gives information to track business drivers. The idea of multiple measurements is to put together a pattern of information that collectively gives a complete and accurate picture of the system. Install a metric system in place that allows you to collect the needed information to measure and analyze information and steer projects into the right direction.

Benefits of the model:

Implementation of proper performance measurement framework for outsourced test activities has numerous benefits. Few of them are listed below.

• Helps companies manage their test service providers in an optimal manner for win-win relationships.
• Proper visibility on the return on investment by the outsourced service provider.
• Consideration of all the quality measures into account while analyzing the performance.
• Introduction of a standard evaluation process across the company.
• Identification of the potential risk areas that affect the productivity of the test team.
• Higher level of abstraction with carefully choosen test metrics and the presentation format enabled management to spot the critical issues quickly.
• Past history of the results from the framework can help the success probability of future projects.