June 7th, 2007 — Testing Methodologies
Myself & jOHn (a.k.a “the pikey” cos he’s watched snatch one too many times n takes the software down in the first punch 
)were sitting down to our usual boxing match on software testing. They say tha’ testing is never complete. Well this is what we made of it tha’ day…..
Let’s start at the very beginning, a very good place to start. I have this code snippet that I have boxed into a wonder software. This is wha’ it does “Find the sum of 2 numbers”. Now if the world of software testers out there say that we could all sit and test it for a decent amount of time but still not ship it bug free, its time we hung up our boots
So this is the anthem, “Theoretically the number of bugs in an application/software is a finite number. The process of testing uncovers these bugs and their numbers go down. Hence mathematically there is a point in time (which does not tend to infinity) where there are actually 0 bugs in the software under test.”
The loudest outcry against this is that “time is your biggest enemy”. Most software development projects do not/cannot afford the time factor to reach that stage of zero defects. Most commercial testing is hence forced to stop at the light blue zone. The pink zone is classified as “phantom defects” and left to be.
Hence the job of software testing needs to re-classified from that finds bugs to one that also finds the time to find bugs. Let a so called “phantom bug” be just because we didnt get the time to do it is inexcusable (Sounds kinda like a developer saying that he didnt implement only 90% of the functionality cos that all the time he had).
This is where we require approaches and philosophies to improve test efficiency. Software testing needs to be a defect-time function. Below is a proposed methodology to increase your testing efficiency called the Weighted Graph Methodology.
Stream Lining Test Execution
Weighted Graph methodology
The normal process followed for execution of test cases manual or automation is to follow the sequential order of test case as presented in the overall test plan for the concerned module. The common order of cataloging test cases in the test plan for a particular module would be to group them by functionality or by the common factors of the various test scenarios involved. This is to ensure that the designer of the test case does not miss any valid test scenario. The purpose of the approach which is presented in the following sections addresses the following problem area which is typically encountered.
“Tester xyz is handling the testing of a particular module which encompasses 5 broad functionalities. The test plan he uses has test cases grouped into these 5 categories and presented sequentially in his test plan document. He is scheduled to complete testing of the module in 10 days. He starts off execution of test cases sequentially and ends up repeatedly executing test cases relevant to the first functionality to completion and so on.
Assumption: Of the 5 functionalities assume the first 3 are extremely bug free while the 4th and 5th are extremely buggy. Since tester xyz will spend the major part of the early period testing the first 3 modules sequentially he would take longer to zero in on the bugs hence delaying the reporting of bugs leading to delayed cycle times where the development team would be waiting on him during the early period.”
The weighted graph methodology modifies the process of test case execution as follows. The module to be tested is first represented in the format of a weighted graph for example:
Structure of the weighted graph:
Where the leaf node represent the different functionalities bundled in the module. The squares in blue represent the weights of each limb of the graph in the beginning (here 0). The cylinders show the subset of test cases that can be tagged to each of these functionalities.
Executing test cases the “weighted graph way”
To begin execution of test cases using weighted graphs; QA first develops a script which performs the following:
1. Pick a test case using a probabilistic approach depending on the weight of the limb from any one of the subset of test cases for all the functionalities.
2. The tester then executes that test case either manually of via automation scripts.
3. The result of the test case namely “pass” of “fail” is communicated to this master script.
4. If the test case fails then the script increases the weight of the limb for that functionality by a factor say 1.
5. In the next iteration of the above steps the probability of a test case from the limb with more weight being picked is higher.
6. The above steps repeat till all test cases have been covered.
The weighted graph approach is beneficial because the tester ends up executing more test cases that are tagged to a possibly buggy module hence the ratio of the number of defects identified to the number of test cases executed increases. Hence the defects in a module are identified at the early stages of testing. This translates to reduced cycle times and more throughput for the testing activity.
As for pikey…..well he now wants to take the software down with every punch
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
June 7th, 2007 — Testing Methodologies
Introduction
1. Critical components under QA Testable category
The most critical component where about 70-90% of QA efforts are concentrated in testing data intensive systems is a “data processing/ transformation engine” which can be identified as a “common critical component” of all these systems. This lion share of testing efforts is commonly concentrated around this engine which forms the core of such systems since all these systems are in place to process the streamed data into usable information.
From observations of such systems any “data processing/transformation engine” is governed by a set of data transformation rules whose complexity can range from low to extremely high depending on system design.
The common test automation approach for such projects would be to automate the verification and validation of the transformations that occur to the data since it forms a critical component of the system and huge testing efforts are concentrated around the same. A common yet wrong approach noticed to automate the same is to mimic the end to end transformation rules governing each module of the system using test scripts and performing a “parallel feed” of the candidate test data through the QA setup and test script and compare the outputs of the same.
QUESTION: If a QA personnel can mimic the end to end transformation rules governing a module for the purpose of testing the same and certify a bug free test script, cant the code be ported to be in-place within the application for a bug free rollout?
2. Proposed Approach
An alternative approach to automate the testing of the same; referred henceforth as the reverse tree parse methodology on a high level is described as an approach to invert the parsing of such transformation rules algorithms so that QA would be undertaking a bottom up parsing technique.
In this approach the relevant part of a transformation engine is converted into a flow diagram. The flow diagram is then reverse parsed to generate and codify an optimum number of reverse parse trees which will then be predictive parsed to validate the transformation scenario and arrive at certain subset of “base states” which denote a passed test case. In this approach QA would be adopting a “inverted logic flow” to validate the scenario. The plus point of this approach being that the test scripts would be able to not only predict the validity of a test case but also give pointers to what “base scenarios” would generate the current output.
3. Where do we start???
The following section attempts to do a step by step walkthrough of adopting the approach.
1. Modularize the transformation rules: Every transformation engine or the design specification governing them can be classified into modules such that for every module we have a finite set of rules governing the transformation of the data relevant to the module.
2. Define the finite set of relevant rules: Depending on the complexity of the transformation rules the optimum approach to maintain the modularity of the automation effort would be to identify and isolate every single rule governing the relevant module. This would form the superset of all rules within the transforming logic.
** Rules can be classified into the following broad categories:
Ø Rules that are independent of all other rules.
Ø Rules which act as the “invoker” of other rules.
Ø Rules which are “exclusively invoked” by other rules. They don’t have independent existence.
Ø Rules which act as “invoker” at times and as being “invoked” at other times.
Irrespective of the classification of the rule; every rule is then converted into a flow diagram.
4. Process of creating flow diagrams for codifying
Terminology:
The following are the major components of parse trees:
Nodes: These are the finite set of all states that the transforming process will reside in during the course of parsing the rule. Each node is associated with a set of data and a finite set of conditions which need to be verified to exist for the algorithm to move out of that node to the next node.
Start Node: This is the set of states from where a possible parse can start. The set of start nodes are dependant on the data which needs to be validated.
End Nodes: This is the finite set of nodes where the parse algorithm can reside; at the end of the parsing process. The test case is said to have passed when the algorithm resides in any of these set of nodes.
Process Arrow: This is a unit of processing that happens in the parse algorithm that will take the algorithm from on state to another. Every parse arrow exist attached to a node, hence they are in effect the conditions that help the algorithm to move from 1 valid node to another
** The algorithm can never reside at the following states:
Ø A state which is not pre-defined and whose conditions are not known.
Ø A state which does not have a valid outgoing process arrow attached to it.
If the process of executing takes the algorithm to a node other than an end node which has no valid process arrow which can be called; then it indicates a failed test case.
The process of creating the parse tree is to create a representation of the process flow of the rule in terms of start nodes, process arrows and end nodes. Since we reverse parse this tree for the purpose of validation the nodes that define the beginning of the transform rule will become the end node (the nodes that denote success when the tree is codified) and the leaf nodes defined will become the start nodes for the algorithm when the tree is coded. Refer to a sample transformation rule to explain the process of creating a parse tree.
“If modified date (MD) present and is greater than update date (UD) then output 1 else if modified date(MD) is not present or less that update date (UD) then output 0”
The following simple rule can be represented as a parse tree as follows:
In the above parse tree the nodes marked in gold will be the start nodes from where the algorithm can be initiated. The green node is the end node which signals a passed test case.
6. Coding for nodes
To convert the parse tree to an executable script each node in the tree is defined as an object of a custom class. The class definition for each node would contain the variables to hold the data that are relevant to the node e.g. variables MD and UD for node number 4, MD alone for node number 2 etc. The class definition would also contain functions to represent each process arrow and perform the validation check attached to that process arrow. Each process arrow would also pass relevant data to its target nodes if needed.
7. Building that “Parse Director”
The parse director module is a wrapper script which directs the calling of parse trees and performs the actual reverse parse of the transformation rule.
Need for parse director:
Every transformation will have a set of many rules. The parse director will perform the following:
- Will hold the structure that defines the rules that can be called after evaluating one rule in the reverse parse process.
- Direct the parsing of the tree by calling the appropriate parse tree to validate the test case.
8. Benefits of using parse tree oriented automation
- This automation framework when developed will enable the QA team to predict the finite set of permutations that are possible for the process flow in each module.
- This will enable them to finalize the minimum number of test cases required to ensure test coverage of all valid process paths.
- This can serve as the direct replacement for orthogonal arrays to enhance and streamline the testing of each module.
** note that this would serve to minimize test case to achieve coverage at a “function call level” and not a “code branching level”. This is justifiable for the black box level testing relevant to QA.
- The ratio of number of test cases execute to the number of defects identified will be minimized thus enhancing the productivity of the QA team and facilitate cut downs in QA schedules.
9. Requirements to adopt the approach
- In depth understanding of the transformation engine under test.
- QA tester with experience in building automation scripts using object oriented languages like JAVA/ RUBY etc.
- Testing teams with expertise in refining and analyzing test scripts developed in the above languages.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
May 29th, 2007 — QA Process
The creation of automation scripts for testing functionality/performance/load is a process that generates code in intself. Hence this is an activity that would require quality control processes tagged to it. Just like the core code that implements system functionality, the automation scripts form a supporting code base.
The variety of quality processes that a test scripts undergoes can be based on the concept of context based testing (relevance of context shall be expalined at the end of this post).
1. Peer code reviews
These can be done to leverage all the advantages of conventional code reviews and to implement relevant scripting standards and indetify static bugs within the script. The peer code review has an added advantage: since test scripts also need to be maintained constantly, familiarizing multiple team members with the test script can aid in script maintenance and de-risk the automations plan.
2. Having a skeletal requirements document for test scripts
The benefit of having a skeletal requirements document can aid in ensuring that the automation script has adequate functionality coverage. This can also be an part of the actual requirement traceablitiy matrix if one is maintained.
3. Performing a dry run sanity test using script
This is a case where a couple of test runs of the script is done and some relevant manual test cases are also executed to ensure that the results of both are in sync.
The relevance of using a context based approach in testing your test scripts is that based on the context of the system under test and its criticality the amount and type of testing done on test scripts can be increased or decreased. Hence doing an impact analysis of the possible failure of a test script is the most crucial part of this activity.
Ill fill in more details about these as and when i get around to stumbling on them 
Please leave a comment if you can think of something else.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
May 18th, 2007 — Testing Methodologies
The normal process followed for execution of test cases manual or automation is to follow the sequential order of test case as presented in the overall test plan for the concerned module. The common order of cataloging test cases in the test plan for a particular module would be to group them by functionality or by the common factors of the various test scenarios involved. This is to ensure that the designer of the test case does not miss any valid test scenario. The purpose of the approach which is presented in the following sections addresses the following problem area which is typically encountered.
“Tester xyz is handling the testing of a particular module which encompasses 5 broad functionalities. The test plan he uses has test cases grouped into these 5 categories and presented sequentially in his test plan document. He is scheduled to complete testing of the module in 10 days. He starts off execution of test cases sequentially and ends up repeatedly executing test cases relevant to the first functionality to completion and so on.
Assumption: Of the 5 functionalities assume the first 3 are extremely bug free while the 4th and 5th are extremely buggy. Since tester xyz will spend the major part of the early period testing the first 3 modules sequentially he would take longer to zero in on the bugs hence delaying the reporting of bugs leading to delayed cycle times where the development team would be waiting on him during the early period.”
The weighted graph methodology
The weighted graph methodology modifies the process of test case execution as follows. The module to be tested is first represented in the format of a weighted graph for example:
Structure of the weighted graph:
Where the leaf node represent the different functionalities bundled in the module. The squares in blue represent the weights of each limb of the graph in the beginning (here 0). The cylinders show the subset of test cases that can be tagged to each of these functionalities.
Executing test cases the “weighted graph way
To begin execution of test cases using weighted graphs; QA first develops a script which performs the following:
1. Pick a test case using a probabilistic approach depending on the weight of the limb from any one of the subset of test cases for all the functionalities.
2. The tester then executes that test case either manually of via automation scripts.
3. The result of the test case namely “pass” of “fail” is communicated to this master script.
4. If the test case fails then the script increases the weight of the limb for that functionality by a factor say 1.
5. In the next iteration of the above steps the probability of a test case from the limb with more weight being picked is higher.
6. The above steps repeat till all test cases have been covered.
Benefits of using weighted graph method
The weighted graph approach is beneficial because the tester ends up executing more test cases that are tagged to a possibly buggy module hence the ratio of the number of defects identified to the number of test cases executed increases. Hence the defects in a module are identified at the early stages of testing. This translates to reduced cycle times and more throughput for the testing activity.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
May 16th, 2007 — QA Process
Objective
I have outlined here some of the flawed practices adopted in the testing phase activities of projects and propose corrective measures to wrong/ absent practices and improvements to existing practices.
I have traced the incorrect practices down the entire testing lifecycle to elucidate the impacts of such practices and to showcase how the suggested improvements can streamline and eliminate the wrong practices followed in testing phase of projects.
This document is based on case studies and observations done on projects with respect to factors that impacted the following:
* Overall functioning of testing teams
* The quality of deliverables
* Root cause analysis of the causative factors for the deviations in estimated delivery/ roll out plan
* QA sign off activities.
Getting Started: Envisioning Phase
The typical envisioning phase QA activities are namely,
* Creation of QA estimates.
* Identification of QA planning/ execution challenges.
* Compatibility/ Feasibility study of scope of automation
* Identification of project execution risks/ dependencies
* Initial proposal of risk management/ mitigation strategies.
Identified wrong practices
The following are the improvement areas that have been identified with respect to envisioning phase activities:
Creation of QA estimates
Identified problem area: No QA estimate for Sanity testing
QA does not estimate for the sanity testing of builds deployed in QA. The fallout of this is that sanity testing is not identified as an essential part of any QA activities. There are no efforts dedicated to identifying a sanity test suite. Hence the QA team cannot establish baselines on which to accept/ reject builds.
Sanity testing (if any) when performed is an ad-hoc activity, and hence there is no traceability for this activity.
Recommendations:
Sanity testing of all builds expected to be delivered to QA has to be factored in the estimates and activities have to be allocated to this in the test planning and execution phase.
Compatibility/ Feasibility study of scope of automation
Identified problem area: No POC developed to validate compatibility/ feasibility of automation
QA teams do not do POC’s for projects where they do not have prior platform/ technology automation expertise to ascertain that there are no compatibility/ feasibility issues with the automation tool/ scripting language identified for use. The fallout of this is that the QA team is not able to proactively identify the drawbacks if any of the automation frameworks they have envisioned. Since the issues in the automation approach are not identified until the actual development of automation test scripts start no alternative tools/ scripts/ approaches can be envisioned.
Recommendations:
POC’s have to be undertaken to ascertain that the indentified automation framework is compatible with the platform/ technology area being addressed.
“How are we going to do it?”: Planning Phase
The typical planning phase activities involved in testing are:
* Creation of Testing strategy document
* Design of Test plans for the modules under test
* Development of automation test scripts
Identified wrong practices
The following are some of the identified practices that are incorrect or needs improvement segmented according to the planning phase activity they belong to:
Creation of T&E strategy
Identified problem area: Lack of a QA build management strategy
QA Testing strategy documents do not outline the strategy that will be adopted by QA to ensure the criteria to accept or reject QA builds. Hence the parameters with regard to “testability” which are mandatory are not aligned with either development team or the customer side stake holders. The fallout of this is that the process of accepting or rejecting builds in QA is not formalized and is an ad-hoc process. This scenario is aggravated when numerous builds get deployed into QA to rectify deployment issues. The end result being that there is no traceability with respect to build issues since they have been rectified in an ad-hoc manner. There are no QA side documents logging the number of builds rejected and for what reason. Hence the project team does not have a collated document citing the major issues that need to be tackled while doing final builds in production prior to “go-live”.
Recommendations
The Testing strategy document should outline the mandatory checkpoints to ensure testability of builds. This should be coupled with a template document that will be maintained and communicated when accepting and rejecting builds.
Identified problem area: The points of verification of each module are not explicitly mentioned.
The Testing strategy for each module should also identify what all would be the points of verification for that module. This is because of the n number of modules within the system; there will be a small set of points from where QA would extract the outputs of testing for verification. The relevance of this activity is that when a module is deployed into QA the corresponding points of verification have to be in place to ensure that testing teams are able to start testing. The fallout of not adhering to this practice is that the module would be deployed in a healthy build; but if the point of verification is not in place or is not functional; that would impact the testability of the build.
Recommendations
The test strategy for each module should have its points of verification explicitly stated. This is so that this can then serve as checkpoints to start testing. The development team would also be aware of the necessity of these supporting modules and can ensure that these are in-place.
Design of test plans
Identified problem area: No sanity test suite identified by QA.
Currently as part of the test planning activities QA does not identify the subset of sanity test cases for any module. The fallout of this absent practice is that there is no alignment with the customer side technology/ business stakeholders with regard to what is the critical subset of sanity test cases that have to pass for QA to start testing. A formal sanity test suite can establish the grounds on which QA would reject a build as un-testable. The process of sanity testing is ad-hoc in nature and in many cases the members of the QA team are themselves unsure of what are the highly critical test cases that have to pass to deem a build as testable. This has led to many person hours of effort being wasted in identifying a module as un-testable and communicating the same to the development team.
Recommendations
For all modules that have been planned for testing; QA team should identify and maintain a suite of sanity test cases. An alignment has to be reached with the customer regarding the completeness of the test suite. The status of these test cases can be used to substantiate the acceptance of a QA build. This activity will enhance the customer confidence levels in the process adherence of QA personnel to a great extent.
Identified problem area: “Seat of the pants” method of test automation.
This is a unique automation methodology that is being followed in many testing projects. In these cases though the QA team has estimated for the automated testing of a module, there lacks an absence in the mapping of the manual test cases designed to the automation scripts developed for the same. Hence the QA team is hazy with regard to the extent of automation that they have actually driven in the project. The test coverage via automation is also not specific. There are only rough estimates regarding the extent of automation planned. These estimates are never re-visited during the course of the project of taking stock of the extent and coverage of automation.
Recommendations
The QA team has to identify the subset of manual test cases that can be automated. This then has to be documented and revised constantly to ascertain what extent of automation has been planned and what has been achieved in the project executions. This is an area where the QA teams will have to conduct E-R gap analysis*.
* E-R gap analysis: Expectation – Reality gap.
“How are we doing it?”: Execution Phase
The following are the core set of activities undertaken by QA team during the execution phase of projects.
* Execution of manual test cases.
* Execution of automated test scripts
* Defect reporting/ tracking
* Re-testing of defect fixes
* Regression testing of modules
* UAT support (if applicable)
Identified wrong practices
Execution of manual test cases
Identified problem area: Sanity testing of builds not performed
Since the QA team does not have a pre-defined test plan for sanity testing; little or no sanity testing is done once a build has been deployed into QA for testing. The initial testing of such modules is an ad-hoc process.
Recommendations
The sanity test plan that has been designed as part of the planning phase activities have to be executed before testing commences on any deployment or re-deployment in QA. This will ensure faster turn around times in identifying unhealthy builds.
The process of build rejection will be more formalized by adopting this practice.
Conclusion
The above procedural flaws in testing practices have often led to sluggish turn around times for QA. Adopting the recommendations outlined herewith will ensure that QA teams do not incur wasted efforts identifying sick builds and that automation efforts are more traceable.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
May 15th, 2007 — Testing Methodologies
The content about xtreme testing that i have stumbled across focuses and emphasizes only on the software development process with little or no emphasis of how a testing team fits into an extreme programming project or how the process needs to be tailored to accommodate testing teams and the activities initiated by them to test software.I have also attempted to outline methodologies that can be adopted for testing teams to undertake the principles of extreme programming when the corresponding development is undertaken using conventional software development paradigms.
Extreme testing derives from the concepts outlined for extreme programming (XP) and distributed extreme programming (DXP) and tailored to be applicable to onshore-offshore model of software development, single & multi vendor model of software development. This approach retains the classical concepts of eXtreme Programming like simplicity, discipline, communication, quality, and flexibility
Introduction
Extreme testing (XT) derives from the XP and DXP agile development methodology and implements its core practices like communication, simplicity, feedback and courage. The following section outlines how most of the 12 practices of DXP can be applied to conventional software testing methodology namely,
- Planning Game
- Small Releases
- Metaphor
- Simple Design
- Testing,
- Refactoring
- Pair Programming
- Collective Ownership
- Continuous Integration
- 40-hour Week
- On-Site Customer
- Coding Standards
The principals from above will be adopted into extreme testing on a strict need to have basis and should be aggressively tailored to appreciate the fact that the process was initially designed for software development by keeping testing as a back end activity.
The methodologies of extreme testing (XT) which will be detailed in later sections will appreciate the core problem areas that current DXP methodology addresses namely,
quick and lean response to changing customer requirements from a testing perspective. How an extreme testing (XT) flavor can be added as a plug-in to a conventional development project.
Extreme Testing Practices
The following are the practices from eXtreme Programming that are adopted for eXtreme Testing.
1. Planning Game
The “planning game” concept of typical Distributed eXtreme Programming (DXP) address the following key decisions:
What will be accomplished by the due date?
What to do next?
The extreme testing (XT) approach will also answer these 2 questions. The tailored form of it being:
What deliverables will be tested by the due date?
What deliverables will be accepted into QA?
Conventional DXP works in iterations of 2 weeks. Where the planning game is re-envisioned and re-factored at the end of every 2 weeks. The 2 key planning steps involved in the planning game are:
1.QA Test planning:
What modules would be tested over the next 2 weeks? This forms a very critical part of steering of the QA activities since QA provides a customer commitment to provide complete test results of a named list of modules. The number of modules to be incorporated in a 2 week burst has to be kept at an optimum since the XT model of testing would aggressively test the modules in QA. Hence, the number of bugs reported for the module would be higher in the 2 week burst and if QA test more than an optimum number of modules the development team may be bogged down by defects reported and fail to make a defect fix delivery for all the modules.
* The core of this activity lies in the fact that
All modules that can be tested in parallel SHALL NOT be accepted into QA even if a build is available for deployment
The focus of eXtreme Testing(XT) is to aggressively test a select list of modules in a burst and NOT lukewarm testing of multiple modules
Advantages of this model of planning
The number of defects reported at any given point will cluster around a select list of modules.
The tracking of bugs is less tedious.
The development team can focus on the clean-up of a restricted number of modules.
The modules that will be tested are prioritized by the customer.
The customer perceived value addition in terms of a bug free roll-out of his high priority modules will be enhanced.
The completeness of testing is inherently transparent to the customer since the team is entirely focused on a select list of high priority modules.
2.Iteration Planning
This is the stage of planning where the QA team gets an alignment with customer regarding the next modules that needs to go into QA on a priority basis. Depending on the extent of bug fixes delivered and tested in the previous 2 week burst the QA team can factor what number of modules can be accepted into QA and still retain team focus on testing of the new modules and re-testing of builds.
The advantage of this type of planning from conventional test execution planning is that in conventional planning, periods and dates are tagged to modules to move into QA at the envisioning and planning phase. The is no room to accommodate scenarios where extensive cycle times are required or where the number of bug fixes are high which would require increased regression testing efforts. The end result of such planning is an overlap of the QA efforts in testing new modules and regression testing of old modules. This often results in mayhem with respect to tracking and closure.
In iteration planning of eXtreme Testing(XT) QA can take a call to ramp down the number of new modules hitting QA at the end of every 2 weeks. Thus they can retain their focus and efforts on select modules thus ensuring better delivery.
2.Simple Design
This feature of XP has been tailored to suit testing efforts by laying emphasis on the design of test plans for a module to be tested to be kept simple. This would be the guiding principle of the entire test planning phase where the inclusion of a test case into a test plan will be on a strict “need to” basis. Rather than design and document n number of test cases for a scenario (which does not aid testing anyway) the testing team would study and arrive at a complete list of all possible scenarios for a module (this could be a derivative of use cases if implemented for the project). This activity shall be the precursor to writing test cases. From the list of these possible scenarios, the team would then identify scenarios that that be clubbed and tested in one shot. The team would then proceed to write the minimum number of test cases for each of such “clubbed scenarios”.The alternative approach for simplifying test design phase of (XT) for project which neither implement use cases nor do scenario based testing is outlined below:
1.List out the comprehensive set of functionalities that can/ need to be tested as part of the module under test.
2.Begin writing the first test case.
3.Analyze if the test case written address some supporting functionality/ dependency/ requirement.
4.If yes, analyze if the test case completely address the supporting functionality/ dependency/ requirement.
5.IF it addresses it completely knock off that requirement from the set of functionalities that you need to test since it has already been covered and a further test case would be superfluous.
6.IF it addresses it only partially THEN attempt to re-work the scope of the written test case to encompass this requirement too. IF you succeed then execute step ‘e’. ELSE continue writing test cases.
7.After completion of writing all test cases re-analyze the test cases written in the light of steps ‘b’ to ‘f’.
3.Pair Testing
This concept has been derived from the classical XP concept of “pair programming”. Pair testing is however, not a mandatory requirement to implement eXtreme Testing (XT). The concept of pair testing can be adopted when the team is extremely focused on testing a very few modules in parallel, that too extremely aggressively. The test pair formed should then resort to classical testing concepts like “exploratory testing” and “intuitive testing” since 2 resources are deployed for testing 1 module the productivity of bugs reported to testing efforts should not be compromised.
This scenario is applicable to a highly mature team of tenured testers who have a comprehensive grasp of the system functionalities and can perform real time prioritizing of the test cases planned to be executed by spotting the defect density in a module and zeroing in on testing those functionalities early in the testing cycle.
The benefits of pair testing are:
Early reporting of a large number of defects.
Enhance the customer perception that the test team is highly efficient in spotting defects.
The paired team has to be dismantled as soon as the defect reporting stage reaches its “critical mass”. This methodology therefore assumes importance when fresh modules get deployed into QA. This would not be necessary for regression testing efforts where the number of bug fixes delivered is less.
4.Metaphor
This is an approach where the testing team evolves a common and complete understanding of what the system is intended to do at a macro level and what each piece of it coming into QA during a 2 week burst is supposed to address. The implementation of this concept is done through brain storming sessions prior to starting a 2 week testing burst where the team will review and strategize the activities planned for the week to arrive at a common understanding of the modules coming in and how it integrates to the already certified modules. This can help test teams evolve their understanding of the system and will help at advanced testing stages like end-to-end or clean pass testing.
5.Refactoring
This is a critical concept of eXtreme Testing (XT) where the initial test design documents, test strategy documents and test plans are treated as “live documents” and the team is free to enhance their test suite, augment the test coverage in real time during test execution or during the look ahead meeting they would have every 2 weeks to evolve the common metaphor (discussed in the above segment). These documents can then be updated on a “need to” basis. This flexibility would allow the testing teams to actively document their test efficiency improvements during the entire lifecycle of the project.
6.Collective Ownership
Collective ownership of all QA deliverables is a mandatory part of eXtreme Testing (XT). This means that all artifacts and plans can be reviewed and changes proposed at any point of time.
The benefits of implementing collective ownership:
The QA deliverables get the attention of the entire team.
Cross dependencies and redundancies of test strategies and test cases can be caught by any member of this team.
The probability of code segments and snippets of automation scripts developed being re-used is greatly enhanced.
The testing activities will hence become further lean and the bulk of test cases and LOC’s involved in automation scripts would be reduced because of extensive re-use or re-scoping them to be used for multiple scenarios. This involves test cases and automation scripts used across modules being re-scoped or re-used to produce a lean mode of operation.
Collective ownership will ensure that there is cross expertise in the team. This would help is de-risking the project by being heavily reliant on a single person to ensure delivery.
7.Continuous Integration
Continuous integration requests will be done for to the development team. This is so that instead of testing individual components till the fag end of testing or till end-to-end testing the QA team will be testing new modules by integrating them into the bug free modules already existing in QA. The benefit of this mode of operation is that integration does not become a one shot activity where multiple interfaces can fail OR integration leads to collateral damage on a large scale. This could make the task of defect analysis extremely tedious. Hence the focus in eXtreme Testing is on a continuous bottom up integration in QA. This would ensure that during end-to-end testing the entire system as a whole can be tested without being bogged down by minor interfacing issues being raised and fixed at the end of the testing effort.
8.40-Hour Week
This is a mandate of eXtreme Programming (XP) that is retained as is in eXtreme Testing (XT). Since teams adopting eXtreme Testing would endure long periods of concentrated efforts it is mandatory that they are not to work overtime. It is deemed in this paper that the necessity of adhering to this and the impact of not adhering to it need not be explained.
“XP teams are in it to win, not to die”
technorati tags: Quality Assurance software testing QA estimation extreme testing
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
