Entries from May 2007 ↓

Testing test scripts

May 29th, 2007 — QA Process

The creation of for functionality/performance/load is a that generates code in intself. Hence this is an activity that would require quality control processes tagged to it. Just like the core code that implements system functionality, the form a supporting code base.

The variety of quality processes that a test undergoes can be based on the concept of context based (relevance of context shall be expalined at the end of this post).

1. Peer code reviews

These can be done to leverage all the advantages of conventional code reviews and to implement relevant scripting standards and indetify static bugs within the script. The peer code review has an added advantage: since test also need to be maintained constantly, familiarizing multiple team members with the test script can aid in script maintenance and de-risk the automations plan.

2. Having a skeletal requirements document for test

The benefit of having a skeletal requirements document can aid in ensuring that the script has adequate functionality coverage. This can also be an part of the actual requirement traceablitiy matrix if one is maintained.

3. Performing a dry run sanity test using script

This is a case where a couple of test runs of the script is done and some relevant manual test cases are also executed to ensure that the results of both are in sync.

The relevance of using a context based approach in your test is that based on the context of the system under test and its criticality the amount and type of done on test can be increased or decreased. Hence doing an impact analysis of the possible failure of a test script is the most crucial part of this activity.

Ill fill in more details about these as and when i get around to stumbling on them ;-)

Please leave a comment if you can think of something else.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

No Comments

IMPROVE TEST EFFICIENCY - Weighted graph methodology

May 18th, 2007 — Testing Methodologies

Existing of test execution

The normal followed for execution of test cases manual or is to follow the sequential order of test case as presented in the overall test plan for the concerned module. The common order of cataloging test cases in the test plan for a particular module would be to group them by functionality or by the common factors of the various test scenarios involved. This is to ensure that the designer of the test case does not miss any valid test scenario. The purpose of the approach which is presented in the following sections addresses the following problem area which is typically encountered.

“Tester xyz is handling the of a particular module which encompasses 5 broad functionalities. The test plan he uses has test cases grouped into these 5 categories and presented sequentially in his test plan document. He is scheduled to complete of the module in 10 days. He starts off execution of test cases sequentially and ends up repeatedly executing test cases relevant to the first functionality to completion and so on.

Assumption: Of the 5 functionalities assume the first 3 are extremely bug free while the 4th and 5th are extremely buggy. Since tester xyz will spend the major part of the early period the first 3 modules sequentially he would take longer to zero in on the bugs hence delaying the reporting of bugs leading to delayed cycle times where the development team would be waiting on him during the early period.”

The

The modifies the of test case execution as follows. The module to be tested is first represented in the format of a for example:

Structure of the :


Where the leaf node represent the different functionalities bundled in the module. The squares in blue represent the weights of each limb of the graph in the beginning (here 0). The cylinders show the subset of test cases that can be tagged to each of these functionalities.

Executing test cases the “ way

To begin execution of test cases using weighted graphs; QA first develops a script which performs the following:

1. Pick a test case using a probabilistic approach depending on the weight of the limb from any one of the subset of test cases for all the functionalities.

2. The tester then executes that test case either manually of via .

3. The result of the test case namely “pass” of “fail” is communicated to this master script.

4. If the test case fails then the script increases the weight of the limb for that functionality by a factor say 1.

5. In the next iteration of the above steps the probability of a test case from the limb with more weight being picked is higher.

6. The above steps repeat till all test cases have been covered.

Benefits of using method

The approach is beneficial because the tester ends up executing more test cases that are tagged to a possibly buggy module hence the ratio of the number of defects identified to the number of test cases executed increases. Hence the defects in a module are identified at the early stages of . This translates to reduced cycle times and more throughput for the activity.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

6 Comments

SOFTWARE TESTING PROCESS IMPROVEMENTS

May 16th, 2007 — QA Process

Objective
I have outlined here some of the flawed practices adopted in the phase activities of projects and propose corrective measures to wrong/ absent practices and improvements to existing practices.

I have traced the incorrect practices down the entire lifecycle to elucidate the impacts of such practices and to showcase how the suggested improvements can streamline and eliminate the wrong practices followed in phase of projects.

This document is based on case studies and observations done on projects with respect to factors that impacted the following:

* Overall functioning of teams
* The quality of deliverables
* Root cause analysis of the causative factors for the deviations in estimated delivery/ roll out plan
* QA sign off activities.
Getting Started: Envisioning Phase

The typical envisioning phase QA activities are namely,

* Creation of QA estimates.
* Identification of QA planning/ execution challenges.
* Compatibility/ Feasibility study of scope of
* Identification of project execution risks/ dependencies
* Initial proposal of risk management/ mitigation strategies.

Identified wrong practices

The following are the improvement areas that have been identified with respect to envisioning phase activities:

Creation of QA estimates

Identified problem area: No QA estimate for Sanity

QA does not estimate for the sanity of builds deployed in QA. The fallout of this is that sanity is not identified as an essential part of any QA activities. There are no efforts dedicated to identifying a sanity test suite. Hence the QA team cannot establish baselines on which to accept/ reject builds.

Sanity (if any) when performed is an ad-hoc activity, and hence there is no traceability for this activity.

Recommendations:

Sanity of all builds expected to be delivered to QA has to be factored in the estimates and activities have to be allocated to this in the test planning and execution phase.

Compatibility/ Feasibility study of scope of

Identified problem area: No POC developed to validate compatibility/ feasibility of

QA teams do not do POC’s for projects where they do not have prior platform/ technology expertise to ascertain that there are no compatibility/ feasibility issues with the tool/ scripting language identified for use. The fallout of this is that the QA team is not able to proactively identify the drawbacks if any of the frameworks they have envisioned. Since the issues in the approach are not identified until the actual development of test start no alternative tools/ / approaches can be envisioned.

Recommendations:

POC’s have to be undertaken to ascertain that the indentified framework is compatible with the platform/ technology area being addressed.

“How are we going to do it?”: Planning Phase

The typical planning phase activities involved in are:

* Creation of strategy document
* Design of Test plans for the modules under test
* Development of test

Identified wrong practices

The following are some of the identified practices that are incorrect or needs improvement segmented according to the planning phase activity they belong to:

Creation of T&E strategy

Identified problem area: Lack of a QA build management strategy

QA strategy documents do not outline the strategy that will be adopted by QA to ensure the criteria to accept or reject QA builds. Hence the parameters with regard to “testability” which are mandatory are not aligned with either development team or the customer side stake holders. The fallout of this is that the of accepting or rejecting builds in QA is not formalized and is an ad-hoc . This scenario is aggravated when numerous builds get deployed into QA to rectify deployment issues. The end result being that there is no traceability with respect to build issues since they have been rectified in an ad-hoc manner. There are no QA side documents logging the number of builds rejected and for what reason. Hence the project team does not have a collated document citing the major issues that need to be tackled while doing final builds in production prior to “go-live”.

Recommendations

The strategy document should outline the mandatory checkpoints to ensure testability of builds. This should be coupled with a template document that will be maintained and communicated when accepting and rejecting builds.

Identified problem area: The points of verification of each module are not explicitly mentioned.

The strategy for each module should also identify what all would be the points of verification for that module. This is because of the n number of modules within the system; there will be a small set of points from where QA would extract the outputs of for verification. The relevance of this activity is that when a module is deployed into QA the corresponding points of verification have to be in place to ensure that teams are able to start . The fallout of not adhering to this practice is that the module would be deployed in a healthy build; but if the point of verification is not in place or is not functional; that would impact the testability of the build.

Recommendations

The test strategy for each module should have its points of verification explicitly stated. This is so that this can then serve as checkpoints to start . The development team would also be aware of the necessity of these supporting modules and can ensure that these are in-place.

Design of test plans

Identified problem area: No sanity test suite identified by QA.

Currently as part of the test planning activities QA does not identify the subset of sanity test cases for any module. The fallout of this absent practice is that there is no alignment with the customer side technology/ business stakeholders with regard to what is the critical subset of sanity test cases that have to pass for QA to start . A formal sanity test suite can establish the grounds on which QA would reject a build as un-testable. The of sanity is ad-hoc in nature and in many cases the members of the QA team are themselves unsure of what are the highly critical test cases that have to pass to deem a build as testable. This has led to many person hours of effort being wasted in identifying a module as un-testable and communicating the same to the development team.

Recommendations

For all modules that have been planned for ; QA team should identify and maintain a suite of sanity test cases. An alignment has to be reached with the customer regarding the completeness of the test suite. The status of these test cases can be used to substantiate the acceptance of a QA build. This activity will enhance the customer confidence levels in the adherence of QA personnel to a great extent.

Identified problem area: “Seat of the pants” method of test .

This is a unique that is being followed in many projects. In these cases though the QA team has estimated for the automated of a module, there lacks an absence in the mapping of the manual test cases designed to the developed for the same. Hence the QA team is hazy with regard to the extent of that they have actually driven in the project. The test coverage via is also not specific. There are only rough estimates regarding the extent of planned. These estimates are never re-visited during the course of the project of taking stock of the extent and coverage of .

Recommendations

The QA team has to identify the subset of manual test cases that can be automated. This then has to be documented and revised constantly to ascertain what extent of has been planned and what has been achieved in the project executions. This is an area where the QA teams will have to conduct E-R gap analysis*.

* E-R gap analysis: Expectation – Reality gap.
“How are we doing it?”: Execution Phase

The following are the core set of activities undertaken by QA team during the execution phase of projects.

* Execution of manual test cases.
* Execution of automated test
* Defect reporting/ tracking
* Re- of defect fixes
* Regression of modules
* UAT support (if applicable)

Identified wrong practices

Execution of manual test cases

Identified problem area: Sanity of builds not performed

Since the QA team does not have a pre-defined test plan for sanity ; little or no sanity is done once a build has been deployed into QA for . The initial of such modules is an ad-hoc .

Recommendations

The sanity test plan that has been designed as part of the planning phase activities have to be executed before commences on any deployment or re-deployment in QA. This will ensure faster turn around times in identifying unhealthy builds.

The of build rejection will be more formalized by adopting this practice.

Conclusion

The above procedural flaws in practices have often led to sluggish turn around times for QA. Adopting the recommendations outlined herewith will ensure that QA teams do not incur wasted efforts identifying sick builds and that efforts are more traceable.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

1 Comment

eXtreme Testing (XT)

May 15th, 2007 — Testing Methodologies

 
icon for podpress  Extreme_Testing: Play Now | Play in Popup | Download (45)

The content about xtreme that i have stumbled across focuses and emphasizes only on the development with little or no emphasis of how a team fits into an programming project or how the needs to be tailored to accommodate teams and the activities initiated by them to test .I have also attempted to outline methodologies that can be adopted for teams to undertake the principles of programming when the corresponding development is undertaken using conventional development paradigms.

derives from the concepts outlined for programming () and distributed programming () and tailored to be applicable to onshore-offshore model of development, single & multi vendor model of development. This approach retains the classical concepts of Programming like simplicity, discipline, communication, quality, and flexibility

Introduction

(XT) derives from the and agile development and implements its core practices like communication, simplicity, feedback and courage. The following section outlines how most of the 12 practices of can be applied to conventional namely,

  • Planning Game
  • Small Releases
  • Metaphor
  • Simple Design
  • ,
  • Refactoring
  • Collective Ownership
  • Continuous Integration
  • 40-hour Week
  • On-Site Customer
  • Coding Standards

The principals from above will be adopted into on a strict need to have basis and should be aggressively tailored to appreciate the fact that the was initially designed for development by keeping as a back end activity.

The methodologies of (XT) which will be detailed in later sections will appreciate the core problem areas that current addresses namely,

quick and lean response to changing customer requirements from a perspective. How an (XT) flavor can be added as a plug-in to a conventional development project.

Practices

The following are the practices from Programming that are adopted for .

1. Planning Game

The “planning game” concept of typical Distributed Programming () address the following key decisions:

What will be accomplished by the due date?

What to do next?

The (XT) approach will also answer these 2 questions. The tailored form of it being:

What deliverables will be tested by the due date?

What deliverables will be accepted into QA?

Conventional works in iterations of 2 weeks. Where the planning game is re-envisioned and re-factored at the end of every 2 weeks. The 2 key planning steps involved in the planning game are:

1.QA Test planning:

What modules would be tested over the next 2 weeks? This forms a very critical part of steering of the QA activities since QA provides a customer commitment to provide complete test results of a named list of modules. The number of modules to be incorporated in a 2 week burst has to be kept at an optimum since the XT model of would aggressively test the modules in QA. Hence, the number of bugs reported for the module would be higher in the 2 week burst and if QA test more than an optimum number of modules the development team may be bogged down by defects reported and fail to make a defect fix delivery for all the modules.

* The core of this activity lies in the fact that

All modules that can be tested in parallel SHALL NOT be accepted into QA even if a build is available for deployment

The focus of (XT) is to aggressively test a select list of modules in a burst and NOT lukewarm of multiple modules

    Advantages of this model of planning

The number of defects reported at any given point will cluster around a select list of modules.

The tracking of bugs is less tedious.

The development team can focus on the clean-up of a restricted number of modules.

The modules that will be tested are prioritized by the customer.

The customer perceived value addition in terms of a bug free roll-out of his high priority modules will be enhanced.

The completeness of is inherently transparent to the customer since the team is entirely focused on a select list of high priority modules.

2.Iteration Planning

This is the stage of planning where the QA team gets an alignment with customer regarding the next modules that needs to go into QA on a priority basis. Depending on the extent of bug fixes delivered and tested in the previous 2 week burst the QA team can factor what number of modules can be accepted into QA and still retain team focus on of the new modules and re- of builds.

The advantage of this type of planning from conventional test execution planning is that in conventional planning, periods and dates are tagged to modules to move into QA at the envisioning and planning phase. The is no room to accommodate scenarios where extensive cycle times are required or where the number of bug fixes are high which would require increased regression efforts. The end result of such planning is an overlap of the QA efforts in new modules and regression of old modules. This often results in mayhem with respect to tracking and closure.

In iteration planning of (XT) QA can take a call to ramp down the number of new modules hitting QA at the end of every 2 weeks. Thus they can retain their focus and efforts on select modules thus ensuring better delivery.

2.Simple Design

This feature of has been tailored to suit efforts by laying emphasis on the design of test plans for a module to be tested to be kept simple. This would be the guiding principle of the entire test planning phase where the inclusion of a test case into a test plan will be on a strict “need to” basis. Rather than design and document n number of test cases for a scenario (which does not aid anyway) the team would study and arrive at a complete list of all possible scenarios for a module (this could be a derivative of use cases if implemented for the project). This activity shall be the precursor to writing test cases. From the list of these possible scenarios, the team would then identify scenarios that that be clubbed and tested in one shot. The team would then proceed to write the minimum number of test cases for each of such “clubbed scenarios”.The alternative approach for simplifying test design phase of (XT) for project which neither implement use cases nor do scenario based is outlined below:

1.List out the comprehensive set of functionalities that can/ need to be tested as part of the module under test.

2.Begin writing the first test case.

3.Analyze if the test case written address some supporting functionality/ dependency/ requirement.

4.If yes, analyze if the test case completely address the supporting functionality/ dependency/ requirement.

5.IF it addresses it completely knock off that requirement from the set of functionalities that you need to test since it has already been covered and a further test case would be superfluous.

6.IF it addresses it only partially THEN attempt to re-work the scope of the written test case to encompass this requirement too. IF you succeed then execute step ‘e’. ELSE continue writing test cases.

7.After completion of writing all test cases re-analyze the test cases written in the light of steps ‘b’ to ‘f’.

3.Pair

This concept has been derived from the classical concept of “”. Pair is however, not a mandatory requirement to implement (XT). The concept of pair can be adopted when the team is extremely focused on a very few modules in parallel, that too extremely aggressively. The test pair formed should then resort to classical concepts like “exploratory ” and “intuitive ” since 2 resources are deployed for 1 module the productivity of bugs reported to efforts should not be compromised.

This scenario is applicable to a highly mature team of tenured testers who have a comprehensive grasp of the system functionalities and can perform real time prioritizing of the test cases planned to be executed by spotting the defect density in a module and zeroing in on those functionalities early in the cycle.

The benefits of pair are:

Early reporting of a large number of defects.

Enhance the customer perception that the test team is highly efficient in spotting defects.

The paired team has to be dismantled as soon as the defect reporting stage reaches its “critical mass”. This therefore assumes importance when fresh modules get deployed into QA. This would not be necessary for regression efforts where the number of bug fixes delivered is less.

4.Metaphor

This is an approach where the team evolves a common and complete understanding of what the system is intended to do at a macro level and what each piece of it coming into QA during a 2 week burst is supposed to address. The implementation of this concept is done through brain storming sessions prior to starting a 2 week burst where the team will review and strategize the activities planned for the week to arrive at a common understanding of the modules coming in and how it integrates to the already certified modules. This can help test teams evolve their understanding of the system and will help at advanced stages like end-to-end or clean pass .

5.Refactoring

This is a critical concept of (XT) where the initial test design documents, test strategy documents and test plans are treated as “live documents” and the team is free to enhance their test suite, augment the test coverage in real time during test execution or during the look ahead meeting they would have every 2 weeks to evolve the common metaphor (discussed in the above segment). These documents can then be updated on a “need to” basis. This flexibility would allow the teams to actively document their test efficiency improvements during the entire lifecycle of the project.

6.Collective Ownership

Collective ownership of all QA deliverables is a mandatory part of (XT). This means that all artifacts and plans can be reviewed and changes proposed at any point of time.

The benefits of implementing collective ownership:

The QA deliverables get the attention of the entire team.

Cross dependencies and redundancies of test strategies and test cases can be caught by any member of this team.

The probability of code segments and snippets of developed being re-used is greatly enhanced.

The activities will hence become further lean and the bulk of test cases and LOC’s involved in would be reduced because of extensive re-use or re-scoping them to be used for multiple scenarios. This involves test cases and used across modules being re-scoped or re-used to produce a lean mode of operation.

Collective ownership will ensure that there is cross expertise in the team. This would help is de-risking the project by being heavily reliant on a single person to ensure delivery.

7.Continuous Integration

Continuous integration requests will be done for to the development team. This is so that instead of individual components till the fag end of or till end-to-end the QA team will be new modules by integrating them into the bug free modules already existing in QA. The benefit of this mode of operation is that integration does not become a one shot activity where multiple interfaces can fail OR integration leads to collateral damage on a large scale. This could make the task of defect analysis extremely tedious. Hence the focus in is on a continuous bottom up integration in QA. This would ensure that during end-to-end the entire system as a whole can be tested without being bogged down by minor interfacing issues being raised and fixed at the end of the effort.

8.40-Hour Week

This is a mandate of Programming () that is retained as is in (XT). Since teams adopting would endure long periods of concentrated efforts it is mandatory that they are not to work overtime. It is deemed in this paper that the necessity of adhering to this and the impact of not adhering to it need not be explained.

“ teams are in it to win, not to die”

technorati tags:

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

5 Comments