The Test Maturity Model Integration (TMMi) is a framework that is developed to complement the existing Capability Maturity Model Integration (CMMI). It provides a structured presentation of maturity levels, allowing for standard TMMi assessments and certification. It enables a consistent deployment of the standards and collection of industry metrics. The TMMi has a rapidly growing uptake across Europe, Asia and the USA and owes its popularity to being the only independent test process measurement method.

The independent TMMi Foundation initiative has been established with the sole intent of developing the TMMi standard. The model it promotes can be used in isolation or in support of other process improvement models. It provides an independently managed data repository to support TMMi assessment method accreditation. It also provides Assessment Method Accreditation/Audit Framework for TMMi in accordance with ISO15504 and the process to certify commercial assessment methods against the standard model. Further it provides the certification and training/examination process, procedures and standards for formal, public accreditation of Assessors and Lead Assessors and the on-going management.

As with the CMMI staged representation, TMMi has a staged architecture for process improvement. It contains stages/levels from 1 to 5 through which an organization passes as its testing process evolves from one that is ad-hoc and unmanaged, to one that is managed, defined, measured, and optimized.

In September 2008 AppLabs became a supporter of the TMMi Foundation, after having ensured that the objectives of the TMMi Foundation were compatible with our views and objectives. This initiative was valuable to us as a company, to the testing industry as a whole, something which AppLabs could add a valuable contribution to and help advance. As AppLabs was already a CMMI Level 5 company, the TMMi alignment with CMMI proved beneficial.

On the whole, these industry metrics and information are sorely missing from the software quality management industry and AppLabs would like to be a key driver in rectifying this process outage. The decision was not taken lightly though, as there were and indeed still are some negatives. The TMMi Foundation is still busy defining the TMMi Levels, at the time of AppLabs decision to support the Foundation, only Level 2 was available. Level 3 has since moved on, but there is still significant work to be done on Levels 4 and 5. There is no guarantee that the ‘Industry’ will recognize this model, despite these negatives, AppLabs still felt it was a valuable investment and could reap rewards for all sponsors of the TMMi Foundation, including AppLabs, but more importantly, for the testing industry as a whole.

AppLabs is currently in the process of accrediting our TMMi model and our experienced Test Consultants with the TMMi Foundation. AppLabs will be offering TMMi reviews as part of our Consultancy Service line in the near future.

Warranty Statement:

The TMMi Foundation makes no warranties of any kind, either expressed or implied, as to any matter included, but not limited to, warranty of fitness for purpose or merchantability, exclusivity, or results obtained from use of the material. The TMMi Foundation does not make any warranty of any kind with respect to freedom from patent, trademark or copyright infringement.

The published material is produced with the permission of the TMMi Foundation. Contents should not be reproduced without the permission of AppLabs and the TMMi Foundation. TMMi® is a registered trademark of TMMi Foundation.

The resources for cloud computing are generally made available through applications with web interfaces that provide functionality to allocate, configure, and maintain the resources. Hence cloud computing adds some ambiguity to the systems hosting your applications causing some concerns around performance and security. And for this reason any cloud integration should have a comprehensive performance testing plan.

Firstly have an end to end understanding of the application performance; ranging from knowing the requirements to understanding the impact on the systems. And with cloud some additional unknowns that need to be considered and planned for are response times, impact on the systems and most importantly the shared resources.

Once deployed into the Cloud, the only practical testing model is the external testing model – that is driving load into the application from a secondary location over the internet. This can easily provide information about response time compared to traffic or load. There will be a few hurdles to work out with the impact on the systems, but you can get a pretty good indication if carefully planned and coordinated. To measure and understand the impact of traffic on computer systems you must understand the environment and the infrastructure, generate realistic traffic, analyze the capabilities and the limitation of the system under test, and find the remedy for performance related defects.

The cloud is a shared environment. This must be taken into consideration for performance aspects of your applications that you deploy into the cloud. Few considerations in this regard are, who you are sharing the resources with, how much of resource is being consumed, how it is impacting the performance of your application, and is usage and performance consistent.

Performance monitoring is going to be essential to assure that your application remains responsive. Constant and on-going monitoring is a good way to verify that the performance continues to remain stable. For this you need to monitor over time, watch for degradation, do changes in performance correspond to changes in your application. Unexplained changes in performance may be attributed to, shared resources and cloud provisioning tool

Several factors that can lead to inadequate performance testing are, lack of requirements, lack of understanding the fundamentals of performance testing, misinformation provided by tool and simulator vendors, limited budget or funding and enormity of the task.

The application system evolved as per the organizational and end user need. It eventually became an integral part of business, banking, trading, shopping, and even entertainment, demanding perfect performance. Testing in the cloud adds testing capabilities that were not available in the past like multiple geographic locations, lots of unique real IP addresses, to formulate traffic patterns. Most performance testing clouds are built out in controlled cloud environments. To perform a controlled cloud testing, access to the full systems, know where the load originates- IP Address, geographic locations, control the distribution of the load, use same test system/systems for retesting, perfect coordination of the user traffic, and correlate the load with end-user response times and the impact of the systems under test.

The environment and global warming pose a disastrous threat, hence organizations world wide focus on reducing energy wastage, CO2 and other greenhouse gas pollution. The IT world is no exception; testing and test professionals too play a pivotal role in reducing the impact of this threat.

Unfortunately due to timeline issues testing gets squeezed leading to inefficient and ineffective testing and this in return leads to resource wastage, ranging from people, money, hardware and ultimately energy. Poorly defined roles and responsibilities leading to diluted lines of accountability, the chaotic development lifecycle leading to testing chaos, defect/bug management that needs structure, no supporting configuration management processes, 100+ Unix environments that are under utilized, replicating overnight process that utilizes huge CPU resource, lead to energy wastage, delays, resource misuse and environmental impact.

Certain testing approaches to reduce the impact are:

• Pertaining to the project necessities, travel only when necessary, utilize communications technology; video-/tele-conferencing, minimize paper trails, use electronic media and define risk criteria.
• When handling Test Management, commission to sustainable, reusable environments, liaise closely with development community and be focused; perform impact analysis of change and direct testing accordingly.
• When dealing with the Test Process conduct an environmental analysis and design, order work methodically by maximizing efficiencies, and adhere to structured testing methodology (Static Testing, Business Process Testing, V-Model and so on)
• To ensure environmental concerns the existing testing roles should be evolved. The roles should be captured during requirements gathering phase, they should be prioritized accordingly, alongside functionality and other business requirements, suitably tested (throughout development lifecycle), outstanding concerns are properly assessed, prioritized, risk managed and scheduled for future release.

Having a more energy efficient IT infrastructure and highly focused and driven people will minimize the organizations exposure to energy price increases and subsequent limitations imposed on the business.

Small and medium-sized companies for fast secure and scalable IT infrastructure chose to migrate to Cloud Computing, a solution that would help organizations focus on their core business rather than worrying about the investment and maintenance of their business IT infrastructure. Though the solution offers significant benefits, it has its own challenges in terms of security, reliability, and manageability. To mitigate these risks, a rigorous testing is mandatory.

Migrating to a cloud environment requires an understanding of the new business needs and the inherited challenges associated with it. Accordingly, the scope of the software testing also needs to be widened to fully cover those business requirements and the inherited risks associated with cloud computing. In order to meet these testing requirements, organizations need to be equipped with resources adroit in different testing skills.

The strategies that testers need to follow while performing testing in a Cloud environment are:

Identify Applicable Testing Types

For this one needs to thoroughly understand cloud characteristics, the business characteristics and the risks/challenges involved.

Some of the quality risks of Cloud computing are Reliability, Flexibility, Multi-tenancy, Self Healing, Pricing

Band on SLA’s, and Location Independence; some of the inherited risks are Data Governance, Data Security, Virtualization Security, Reliability, Monitoring and Manageability. These risks with Cloud computing pose a threat to applications, hence the testing team must understand such threats and accordingly identify the additional amount of testing involved. Also, the team must understand the applicable Cloud models to be tested like Software-as-a-Service (SaaS), Platforms- a-Service (PaaS), Infrastructure-as-a-Service (IaaS) etc. The various services offered to customers via these models play an important role in deciding the applicable testing types to be used for Cloud testing.

The types of testing the team can perform while testing the Cloud are, System Integration Testing (SIT), User Acceptance Testing (UAT), Interoperability Testing, Compatibility Testing, Performance Testing, Load Testing, Stress Testing, Recovery Testing, Security Testing.

Selection of cloud test environment

The infrastructure requirements for test environment is another important consideration for Cloud testing. The two possible options for choosing the right test environment are: simulating in-house Cloud test environment, and choosing the right Cloud service provider - the selection of right infrastructure is critical to testing.

Special considerations

Certain general considerations that are common to cloud testing which the tester must focus are, supporting multiple browsers, user session management related issues, testing against security vulnerabilities, in a multi-tenant environment, restricting users to access their data only and so on.

To mitigate the risks and issues introduced by Cloud Computing, the testing teams should develop a good strategy to test their applications in a Cloud environment that will help the organizations meet their business needs. The testing strategy should discuss the scope of software testing to meet the business requirements and characteristics of Cloud computing. The scope of software testing needs to be widened, which will cover additional testing, improve infrastructure, test environment and the test engineer’s skill set.

The intensity of global warming is aggravating with every passing day, and so is the environmental condition. Even the IT Industry is reputed to issue 2% of the world’s CO2 emissions (the same as the much maligned airline industry). Hence emerges the need to reduce energy wastage, CO2 and other greenhouse gas pollution. The IT world, aware of its significant and growing impact on carbon emissions and climate change, must take steps to reduce this trend.

1,000 PCs running 24/7 without energy saving modes activated can consume as much as £70,000 worth of electricity in a year. Each year, enterprises waste nearly $4 billion powering devices that are not in use. The network traffic and the data centre too are the contributor to energy waste. As a result, the IT world must look at ways of reducing the output of harmful emissions and heat waste.

Poorly defined roles and responsibilities – diluted lines of accountability, development lifecycle chaotic – testing chaos, defect/bug management needs structure, no supporting configuration management processes, 100+ Unix environments – under utilised, replicating overnight process utilises huge CPU resource, lead to energy wastage, delays, resource misuse and environmental impact.

A number of options are available in this field and ever evolving technical solutions are coming forward, such as multi-core processing, virtualisation, data compression solutions, etc. These solutions not only reduce the environmental impact, they also reduce the financial impact. Here are certain testing approaches to reduce the impact.

• Pertaining to the project necessities, travel only when necessary, utilize communications technology; video-/tele-conferencing, minimize paper trails, use electronic media and define risk criteria.
• When handling Test Management, commission to sustainable, reusable environments, liaise closely with development community and be focused; perform impact analysis of change and direct testing accordingly.
• When dealing with the Test Process conduct an environmental analysis and design, order work methodically by maximising efficiencies, and adhere to structured testing methodology (Static Testing, Business Process Testing, V-Model and so on).
• To ensure environmental concerns the existing testing roles should be evolved. The roles should be captured during requirements gathering phase, they should be prioritized accordingly, alongside functionality and other business requirements, suitably tested (throughout development lifecycle), outstanding concerns are properly assessed, prioritised, risk managed and scheduled for future release.

Having a more energy efficient IT infrastructure and highly focused and driven people will minimize the organisations exposure to energy price increases and subsequent limitations imposed on the business.

Trends like Web based  applications, Service Oriented Architectures (SOA), Software as a Service (SaaS), Wireless Technologies, Mobile technologies, with the increasing emphasis on repeatability, reliability, re-use and robustness, has opened a new path to testing, emphasizing more on Analysis and Design phases and ensuring early engagement of testing in these areas.

Analysis and design have been people based activities with Business Analysts and Systems Architects producing a combination of natural language, some structured language techniques, models and diagramming techniques, hence leading to business requirements and functional designs which do not meet the four key criteria which make the resultant systems ‘testable’. And to make the quality of the application clear, assumption free, unambiguous and complete, with the limited time, we will need a more structured and more automated analysis and design technique, which can be attained with robotizing the Static Testing.

In this new world, the Professional Tester will need to further concentrate on adding value to the business and the software development lifecycle and will hence need to operate in a new and elevated position where the combination of their structured process driven approach, creativity, destructive inquisitiveness and ability to articulate and define quality and testing criteria into the models will be critical to the success of the business implementation of technology. 

The role of the Professional Tester will now become more interesting and more essential. And so will Testing and Quality Assurance, which will become more important and add more value as we move into architectures and technologies which support the business in their goals of bringing products and services to the market as rapidly as possible, with minimal risk. We will see a shift towards testing the transformed business operation, the business processes, the way people interact with the systems and processes and the information it provides, and therefore mitigating the risks and increasing the benefits of business change.