Entries Tagged 'QA Process' ↓
February 14th, 2008 — QA Process
A question always asked is “How much should we test?” and a lot of schools of thought follow on with answers. However, there is another question which id like to pose.
“What level of documentation is required?”
The norm for the above question that we practice is “Document till good lord is squeezed out of your test case”. The reason why this question is prominent is that a tester may take 10 minutes to test a scenario and spend 20 minutes generating documentation and backing up log files. The base duty of a tester is to provide quality assurance. When a test case passes the quality is assured for that snippet. Everything beyond that point is what id call an overhead.
So the question is how much overhead are you willing to bear. The funny reason being that a testing document seems capture a very short attention span. For example, My team spends a fortnight preparing documentation of a testing activity. The purpose of it being that it gets send out to “N” stakeholders who are too busy to look into the nitty gritty’s there and the document gets a 1/2 hour window where it gets “reviewed” at a very high level.
Under the circumstances, to be realistic i believe that a stake holder is not interested in what message was logged when a test case failed. He would be more interested in metrics that say “100 defects were reported at Phase 1. 98 were fixed, retested and verified and 2 low sev defects were deferred.”
So the guiding principal in documentation would be “DO NOT document the nitty gritty’s that would lose importance beyond the realms of your desktop. Invest time in collating metrics.” If that be done we would not be sending out spread sheets of test reports that have 183493 rows of backup log messages and a green column indicating it has passed. The industry has to reach a level of maturity where they trust a tester who says “Test case # 2322 has passed. “
One point to make is that:
“NO ONE IS INTERESTED IN A PASSED TEST CASE. ITS ONLY THE DEFECTS THAT MATTER.”
Realistically if there is contention regarding the “pass” status of a test case, that should be re-verifiable by reverting to a pre-defined snapshot of the test environment and re-testing the scenario. Cos even if you have dumps and documentation for a contentious test case you would still have to do the above thingie….. So then whats the point in the whole idea.
Ideally testing is a “Defect targeted activity” AND NOT “Documentation generation activity”.
“For those who believe, no explanation is necessary. For those who do not believe, no explanation is possible.”
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
January 24th, 2008 — QA Process
Recently i was talking to my fiancé (another prolific tester) on her drive down to her office. The topic of conversation being the bad quality of development at her workplace. So these are a few thought that came outta it and we decided to jot it down 
There are many “maturity levels” that a development practice undergoes. Here is a skeletal framework of that:
Skeletal Development ( Writing & Compiling code)
This is the sort of code that should ideally be seen written by new grads from college or out of High School. Where a snippet of code is deemed to have the pinnacles of quality if it just compiles right without any errors. This is the first stage to being a good developer. Unfortunately this is a “coding standard” that i have found among a vast number of “experienced” developers in the so-called CMMi (deprecated IT standard) level 5 companies in India. This is best described as the infancy of a hacker.
Unit testing
This is when you take that first baby step towards QA. (Yes and we thought that is the height of Software Testing expected of a developer right??? Loooooong way to go…….. ) The practice is pretty prevalent now that the heading is self explanatory. Your write code and you write some unit test cases and then you execute them.
Coding Standards
This is where the real game starts ( big boys are found here)…… This is where your code is not just Unit tested but it adheres to the coding standards defined for the programming language used. This is important because one of the aspects of Quality code is not just that it performs well and correctly…. but also that it is highly maintainable.
The relevance of this derived from the fact that 70% of the IT world out there are involved not in developing fresh code but in maintenance and enhancement of legacy code. So lack of standards in legacy code makes them un-interpretable and the enhancement code tends to be more buggy.
Code refactoring
This is the stage of coding that is often left untouched. This stage comes into picture where your code has been developed, reviewed. Once the review is complete, the developer and reviewer has a sit-in to refactor the code. Now for those developers who are going “Huhh, whats refactoring” check out the following link below:
http://en.wikipedia.org/wiki/Refactoring
“halleluia to agile community for driving this practice…….”
Preemptive Functional testing
Preemptive functional testing is once of the final qualities of a legend class developer. Where he does all the above mentioned stuff and then sits down to do a bit of functional QA himself. He executes “black box” grade critical test cases and ensures that they pass.
The outcome of adopting the above standards are:
- Your build “will never break”
- Your QA will tear out the hair trying to find a bug
- A rookie out of college testing you app will come out with zero defects.
- Testing has to reach a new height to beat your code.
- A legend class developer is born.
For all those testers reading this send this to your developer friends out there……..
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
December 13th, 2007 — QA Process, Testing Methodologies
As a continuum of my post on test approach best fit vs first fit, thought id present Heuristic testing as a means of getting this done….
Question is “What is heuristics”
Well, a heuristic is a rule of thumb solution to a problem that may qualify as the solution but need not necessarily be so….. This a terminology where emphasis lies on the potential of a thought, approach or method to be come the solution for a problem.
How does this help testing…… Well QA and testing has reached very appreciable maturity levels the worldover (barring India ) that for any given problem posed you have a sizeable set of solutions as compared to what you would have 10 years back. Let say i want to do funtional testing. There are umpteen number of options at hand like the following:
- Sticking to age old manual testing.
- An automated solution ( just because a testing can be automated it need not be)
- The possibility of using my own custom scripts.
- Possibility of using COTS products ( high industry affinity)
Now heuristic testing is a methodology where the set of n possible solutions are all given equal importance initially. Then begins the probing stage where the validity of accepting any of these n solutions is questioned and a process of elimination gets kicked off. Darwin says “The fittest will survive”. Well if thats true for the cosmos then why should not hold true for Testing Methodologies ?
The biggest truth is “Winrunner need not be the best solutions for your functinal testing requirements just because your QA Consultant know only that !!!!! ”
Sorry if that came across too hard…… but then in many cases i have seen the following classic case “My carpenter knows how to use the only the hammer and i want him to put in a screw. So he bangs the bejesus out of the screw till it goes in….. End result i have a wall that looks like its survived world war II……… and we give him an award for ontime service delivery but failed to realize that the next day a painter and a mason will have to come in for damage control post deployment………” Now i be that sounds like a familiar story…….
This is where we do Heuristic testing where the primary focus is not on getting A single solution but to ascertain what options we have. Then we get around to finding our best option after careful analysis by questioning the authenticity of each of our options and let the fittest solutions emerge as the winner….
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
November 19th, 2007 — QA Process, Tech thoughts
Recently I was brainstorming with a highly competent tester based in Bangalore, India [a rare breed out here] and we stumbled on a startling fact. Requirement gathering and tracking methods have remained largely static for a long time. The delivery models in India with respect to QA and process are reminiscent of the late 90’s.
These are some of the facts that we ascertained:
- The most prolific and commonly used requirement tracking tool in the Indian IT service sector is…….. “Microsoft Word”.
- Every stakeholder/developer/tester in a project has a personal copy of the requirement document resident in his inbox.
- Requirement changes are often not centrally held/ distributed.
- Requirement discussions/ freeze is effected after multiple rounds of talks with the onsite customer/coordinator.
Software systems today have reached a stage of “critical complexity” where it is imperative that we hold requirements in a central repository. There core reason for no advances in this segment is because our internet non-savvy IT community is oblivious to the concepts of web2.0 and the “collaborative internet”.
The version of the internet as we have it today has moved towards distributed applications with added focus on communities,collaboration and harnessing collective intelligence.
The benefits this can provide to requirement tracking are the following:
- Use Web 2.0 principles to build requirement tracking tools on the “internet platform”.
- Provide features within the app like forums [to start discussion threads], chat enabling for real time gap resolution, wikis for public bulletin boards.
- The feature rich UI capabilities of internet based applications can provide “mind map” like interfaces for provide a structural organization for requirements.
- This would provide the benefit of greater usability and clarity compared to “referring to section 1.1.3 (a) or 1.2.4 (b)” to track requirements.
Requirements are the most critical pieces of information in the IT sector…. Information management is a science that Internet gurus are trying to perfect and they are the forerunners. So its only logical to adopt their practices in “managing” our information rather than stick to “Pubic library model of requirement tracking and analysis”.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
October 26th, 2007 — QA Process, Testing Methodologies
A picture is worth a thousand words…… Presenting the synopsis of what i mean
The nut bolt analogy would give u a small idea of how this is relevant to testing. There is always a set of N possible ways to test and application/product. This is the “superset of all solutions”. Out of this a small subset can be derived by using filters like
- Feasibility
- Compatibility
- Efficiency
- Capability
- Time
Once this subset has been derived, we reach the set of n possible testing solutions. Now in most delivery models that i have encountered, the focus of the testing group is to find out “a” solution rather than “the” solutions. Especially in the Indian IT scenario testers are highly susceptible to the “tunnel vision syndrome” where the first solutions that rings a bell is pursued.
What do I do about it?
Well for starts here are a few pointers:
- To start creating an overall strategy, have the best testers on-board work in silos.
- DO NOT rely on the collective intelligence of the team at this stage. That will come in later.
- DO NOT interfere with the independent thought process of the the “silo’ed testers”.
- Once individual approaches are derived, bring the team to the table.
- Battle mode: This is where “collective intelligence” begins. Discuss the various approaches on table.
- Survival of the fittest: Darwinian theory will do the rest for you. The best will emerge as none of the “silo approaches” but as the common maximum of the best approaches put forth by each tester.
This is how you can ascertain the “best fit” test approach and NOT pursue the first stray thought.
Testing principle: Excellence is always instilled never inbuilt.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
September 22nd, 2007 — QA Process
The concept of performing ACID testing on Databases have been theoretically emphasized over the years, but I have often noticed that this dimension is not an evident driver when it comes to strategizing testing of application.
Testers these days have failed to look beyond the conventional classroom wisdom imparted about software testing. The testing for ACID properties is highly essential for any system/application that has a database back end.
Conventional test strategies emphasize on achieving 100% test coverage. This will provide proof for functional correctness of the system. Post deployment when a system encounters real user/transactional volumes you may be surprised to find crashes, inconsistencies of data served. These can be traced to overlooking the testing of ACID properties.
For those of you who are wondering whats the concept this would be a good place to start looking: http://en.wikipedia.org/wiki/ACID
The crux of the matter lies in incorporating sections to include strategies to test for acidity and translate that to specific test cases to target these scenarios.
Post a reply to gain further insights into ACID testing…
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
June 12th, 2007 — QA Process
Estimating Testing Projects
Walkthrough on HowTo
The online content i have found regarding developing sound estimations for testing projects are found to be wanting in a lot of ways
- Articles start of promising and end up with “Software project estimation - Seat of the pants approach”.
- Articles packed with a lot of know-know but absolutely no how-to
- Articles that tell me how to keep doing what we are already doing.
Current Affairs
The current situation of software project estimation can be best described as CMM level 1 heroics. The nearest we conventionally come to it is a WAG (Wild Ass Guess). As to how this has become an acceptable practice in the software industry defies my comprehension!!?$!?!^@?!!! Sometimes people even go to the extent of categorizing this “exercise in futility” as a SWAG( Scientific Wild Ass Guess).
Bottomline: Doing a WAG or SWAG (Whatever you call it) is as good as doing no estimation at all.
What is an estimate?
An estimate serves as a masterplan for a software project covering all aspects like costing,staffing,timing. Hence basing this on pure guess work is a definite NO
Ok history apart lets get started……
Since this is a walkthrough on how to do an estimate we will start from scratch and move to the end…..
Please note that there are a lot of conventional things that we do which can remain as is. Lets concentrate on the flaws that haunt the practice.
1. Collectibles to start an estimate
Starting an estimation is back breaking work 
. There are the following elusive documents that one has to procure before sitting down to a sensible estimate.
1. Customer Requirements Specification
2. Request for proposal,
3. System specification/ Architecture.
4. Software Requirements Specification
I have often hear the following complaint, “Owwww but these don,t exist at that point”. The reply to that is that as per SDLC (Waterfall, Iterative, Jumbo circus whatever) models of software development these are the primary documents that need to be in place before we estimate for a project.
The necessity of having these document in hand is to prevent the wild ass from guessing about our project
2. Approaches to preparing QA estimates for a project
There are a lot of sound practices that people have been using to prepare development estimates like system lines of code (SLOC), use-case based analysis, function point analysis, object hierarchy trees etc.
In the following section i have proposed some of the techniques that can be used to prepare QA estimates for projects.
1. Begin estimation by conducting a comprehensive study of the system architecture, scope of work and the analyzing the complexity of work.
2. Determine what style of testing should be used in the strategy (you could chose from many options like use case based testing, scenario based testing, module hierarchy trees to name a few). This is a very important step which most QA personnel skip.
3. If you are adopting a module based testing, prepare a module hierarchy tree to visualize the inter-dependencies between modules.
4. Analyze and assign complexity to each node of this tree. Estimate the number of test cases in each module by analyzing the # of functionalities bundled in each module.
5. Ascertain by past experience or analysis, a realistic projection of QA productivity (no of test cases per person day). This is a metric which varies and WILL NOT FOLLOW ANY PRE-DEFINED ORGANIZATIONAL NORM.
6. Analyze each module to arrive at a preliminary idea of the extent of automation that is possible in each of these modules.
7. Estimate the strategy of automation in terms of how you will automate the testing, what will be the coverage of automation, what will be the complexity of developing automation scripts if any. ( a POC might be required for this at a later stage. This has to go into the estimate. Doing POC’s for software test automation is a severly neglected critical component of delivering QA processes.)
The above 7 steps have to necessarily completed and documented before you can start with the estimate.
3. Preparing the estimate
Identify the risks involved. These can range from technology risks, risks introduced by the delivery model that has been adopted and many other factors. Each risk identified has the potential of throwing your estimate haywire. Hence de-risking the project is an important part of estimation. In situation where the number of risks identified are high, it would be a good practice to prepare 2 separate estimates
- Conservative estimate: This will factor all the overruns in terms of effort, time and cost that would transpire if the risks realize themselves.
- Optimistic estimate: This will envision the delivery of the project if the risks identified do not materialize.
The benefit of having 2 versions of the estimate is that it would provide a clear cut picture of how bad things can go when risks materialize and what would be the losses incurred if the are let to remain.Conservative estimates can be projectional in nature where the increase in time, effort and cost can be show as a projection in relation to the risk.
The process of preparing the estimate start only once the above steps have been completed. All factors in the estimate have to be traceable to the documentation that has been prepared as part of section 2 of this document. The actual process of doing the estimate like assigning timelines and resource loading will be guided by the above section and help you arrive at a sound estimate.
The Golden rule of all estimations “DO NOT DO SOFTWARE PROJECT ESTIMATIONS WITHOUT ALL NECESSARY SUPPORT DOCUMENTATION IN PLACE.”
I have outlined only the measures to curb the flawed practices in the estimates. These could be incorporated to modify the existing process(if any) of preparing estimates within your organization. A complete walkthrough would expand the scope of this article exponentially 
For further insight into the implementation of such procedures at an organizational level you can contact me.
technorati tags: Quality Assurance software testing QA estimation extreme testing
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
May 29th, 2007 — QA Process
The creation of automation scripts for testing functionality/performance/load is a process that generates code in intself. Hence this is an activity that would require quality control processes tagged to it. Just like the core code that implements system functionality, the automation scripts form a supporting code base.
The variety of quality processes that a test scripts undergoes can be based on the concept of context based testing (relevance of context shall be expalined at the end of this post).
1. Peer code reviews
These can be done to leverage all the advantages of conventional code reviews and to implement relevant scripting standards and indetify static bugs within the script. The peer code review has an added advantage: since test scripts also need to be maintained constantly, familiarizing multiple team members with the test script can aid in script maintenance and de-risk the automations plan.
2. Having a skeletal requirements document for test scripts
The benefit of having a skeletal requirements document can aid in ensuring that the automation script has adequate functionality coverage. This can also be an part of the actual requirement traceablitiy matrix if one is maintained.
3. Performing a dry run sanity test using script
This is a case where a couple of test runs of the script is done and some relevant manual test cases are also executed to ensure that the results of both are in sync.
The relevance of using a context based approach in testing your test scripts is that based on the context of the system under test and its criticality the amount and type of testing done on test scripts can be increased or decreased. Hence doing an impact analysis of the possible failure of a test script is the most crucial part of this activity.
Ill fill in more details about these as and when i get around to stumbling on them
Please leave a comment if you can think of something else.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
May 16th, 2007 — QA Process
Objective
I have outlined here some of the flawed practices adopted in the testing phase activities of projects and propose corrective measures to wrong/ absent practices and improvements to existing practices.
I have traced the incorrect practices down the entire testing lifecycle to elucidate the impacts of such practices and to showcase how the suggested improvements can streamline and eliminate the wrong practices followed in testing phase of projects.
This document is based on case studies and observations done on projects with respect to factors that impacted the following:
* Overall functioning of testing teams
* The quality of deliverables
* Root cause analysis of the causative factors for the deviations in estimated delivery/ roll out plan
* QA sign off activities.
Getting Started: Envisioning Phase
The typical envisioning phase QA activities are namely,
* Creation of QA estimates.
* Identification of QA planning/ execution challenges.
* Compatibility/ Feasibility study of scope of automation
* Identification of project execution risks/ dependencies
* Initial proposal of risk management/ mitigation strategies.
Identified wrong practices
The following are the improvement areas that have been identified with respect to envisioning phase activities:
Creation of QA estimates
Identified problem area: No QA estimate for Sanity testing
QA does not estimate for the sanity testing of builds deployed in QA. The fallout of this is that sanity testing is not identified as an essential part of any QA activities. There are no efforts dedicated to identifying a sanity test suite. Hence the QA team cannot establish baselines on which to accept/ reject builds.
Sanity testing (if any) when performed is an ad-hoc activity, and hence there is no traceability for this activity.
Recommendations:
Sanity testing of all builds expected to be delivered to QA has to be factored in the estimates and activities have to be allocated to this in the test planning and execution phase.
Compatibility/ Feasibility study of scope of automation
Identified problem area: No POC developed to validate compatibility/ feasibility of automation
QA teams do not do POC’s for projects where they do not have prior platform/ technology automation expertise to ascertain that there are no compatibility/ feasibility issues with the automation tool/ scripting language identified for use. The fallout of this is that the QA team is not able to proactively identify the drawbacks if any of the automation frameworks they have envisioned. Since the issues in the automation approach are not identified until the actual development of automation test scripts start no alternative tools/ scripts/ approaches can be envisioned.
Recommendations:
POC’s have to be undertaken to ascertain that the indentified automation framework is compatible with the platform/ technology area being addressed.
“How are we going to do it?”: Planning Phase
The typical planning phase activities involved in testing are:
* Creation of Testing strategy document
* Design of Test plans for the modules under test
* Development of automation test scripts
Identified wrong practices
The following are some of the identified practices that are incorrect or needs improvement segmented according to the planning phase activity they belong to:
Creation of T&E strategy
Identified problem area: Lack of a QA build management strategy
QA Testing strategy documents do not outline the strategy that will be adopted by QA to ensure the criteria to accept or reject QA builds. Hence the parameters with regard to “testability” which are mandatory are not aligned with either development team or the customer side stake holders. The fallout of this is that the process of accepting or rejecting builds in QA is not formalized and is an ad-hoc process. This scenario is aggravated when numerous builds get deployed into QA to rectify deployment issues. The end result being that there is no traceability with respect to build issues since they have been rectified in an ad-hoc manner. There are no QA side documents logging the number of builds rejected and for what reason. Hence the project team does not have a collated document citing the major issues that need to be tackled while doing final builds in production prior to “go-live”.
Recommendations
The Testing strategy document should outline the mandatory checkpoints to ensure testability of builds. This should be coupled with a template document that will be maintained and communicated when accepting and rejecting builds.
Identified problem area: The points of verification of each module are not explicitly mentioned.
The Testing strategy for each module should also identify what all would be the points of verification for that module. This is because of the n number of modules within the system; there will be a small set of points from where QA would extract the outputs of testing for verification. The relevance of this activity is that when a module is deployed into QA the corresponding points of verification have to be in place to ensure that testing teams are able to start testing. The fallout of not adhering to this practice is that the module would be deployed in a healthy build; but if the point of verification is not in place or is not functional; that would impact the testability of the build.
Recommendations
The test strategy for each module should have its points of verification explicitly stated. This is so that this can then serve as checkpoints to start testing. The development team would also be aware of the necessity of these supporting modules and can ensure that these are in-place.
Design of test plans
Identified problem area: No sanity test suite identified by QA.
Currently as part of the test planning activities QA does not identify the subset of sanity test cases for any module. The fallout of this absent practice is that there is no alignment with the customer side technology/ business stakeholders with regard to what is the critical subset of sanity test cases that have to pass for QA to start testing. A formal sanity test suite can establish the grounds on which QA would reject a build as un-testable. The process of sanity testing is ad-hoc in nature and in many cases the members of the QA team are themselves unsure of what are the highly critical test cases that have to pass to deem a build as testable. This has led to many person hours of effort being wasted in identifying a module as un-testable and communicating the same to the development team.
Recommendations
For all modules that have been planned for testing; QA team should identify and maintain a suite of sanity test cases. An alignment has to be reached with the customer regarding the completeness of the test suite. The status of these test cases can be used to substantiate the acceptance of a QA build. This activity will enhance the customer confidence levels in the process adherence of QA personnel to a great extent.
Identified problem area: “Seat of the pants” method of test automation.
This is a unique automation methodology that is being followed in many testing projects. In these cases though the QA team has estimated for the automated testing of a module, there lacks an absence in the mapping of the manual test cases designed to the automation scripts developed for the same. Hence the QA team is hazy with regard to the extent of automation that they have actually driven in the project. The test coverage via automation is also not specific. There are only rough estimates regarding the extent of automation planned. These estimates are never re-visited during the course of the project of taking stock of the extent and coverage of automation.
Recommendations
The QA team has to identify the subset of manual test cases that can be automated. This then has to be documented and revised constantly to ascertain what extent of automation has been planned and what has been achieved in the project executions. This is an area where the QA teams will have to conduct E-R gap analysis*.
* E-R gap analysis: Expectation – Reality gap.
“How are we doing it?”: Execution Phase
The following are the core set of activities undertaken by QA team during the execution phase of projects.
* Execution of manual test cases.
* Execution of automated test scripts
* Defect reporting/ tracking
* Re-testing of defect fixes
* Regression testing of modules
* UAT support (if applicable)
Identified wrong practices
Execution of manual test cases
Identified problem area: Sanity testing of builds not performed
Since the QA team does not have a pre-defined test plan for sanity testing; little or no sanity testing is done once a build has been deployed into QA for testing. The initial testing of such modules is an ad-hoc process.
Recommendations
The sanity test plan that has been designed as part of the planning phase activities have to be executed before testing commences on any deployment or re-deployment in QA. This will ensure faster turn around times in identifying unhealthy builds.
The process of build rejection will be more formalized by adopting this practice.
Conclusion
The above procedural flaws in testing practices have often led to sluggish turn around times for QA. Adopting the recommendations outlined herewith will ensure that QA teams do not incur wasted efforts identifying sick builds and that automation efforts are more traceable.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts
