Entries Tagged 'QA Process' ↓

Trust the tester.

A question always asked is “How much should we test?” and a lot of schools of thought follow on with answers. However, there is another question which id like to pose.

“What level of documentation is required?”

The norm for the above question that we practice is “Document till good lord is squeezed out of your ”. The reason why this question is prominent is that a tester may take 10 minutes to test a scenario and spend 20 minutes generating documentation and backing up log files. The base duty of a tester is to provide quality assurance. When a passes the quality is assured for that snippet. Everything beyond that point is what id call an overhead.

So the question is how much overhead are you willing to bear. The funny reason being that a document seems capture a very short attention span. For example, My team spends a fortnight preparing documentation of a activity. The purpose of it being that it gets send out to “N” stakeholders who are too busy to look into the nitty gritty’s there and the document gets a 1/2 hour window where it gets “reviewed” at a very high level.

Under the circumstances, to be realistic i believe that a stake holder is not interested in what message was logged when a failed. He would be more interested in metrics that say “100 defects were reported at Phase 1. 98 were fixed, retested and verified and 2 low sev defects were deferred.”

So the guiding principal in documentation would be “DO NOT document the nitty gritty’s that would lose importance beyond the realms of your desktop. Invest time in collating metrics.” If that be done we would not be sending out spread sheets of test reports that have 183493 rows of backup log messages and a green column indicating it has passed. The industry has to reach a level of maturity where they trust a tester who says “ # 2322 has passed. “

One point to make is that:

“NO ONE IS INTERESTED IN A PASSED . ITS ONLY THE DEFECTS THAT MATTER.”

Realistically if there is contention regarding the “pass” status of a , that should be re-verifiable by reverting to a pre-defined snapshot of the test environment and re- the scenario. Cos even if you have dumps and documentation for a contentious you would still have to do the above thingie….. So then whats the point in the whole idea.

Ideally is a “Defect targeted activity” AND NOT “Documentation generation activity”.

“For those who believe, no explanation is necessary. For those who do not believe, no explanation is possible.”

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

Coding “IN” quality

Recently i was talking to my fiancé (another prolific tester) on her drive down to her office. The topic of conversation being the bad quality of development at her workplace. So these are a few thought that came outta it and we decided to jot it down :-)

There are many “maturity levels” that a development practice undergoes. Here is a skeletal framework of that:

Skeletal Development ( Writing & Compiling code)

This is the sort of code that should ideally be seen written by new grads from college or out of High School. Where a snippet of code is deemed to have the pinnacles of quality if it just compiles right without any errors. This is the first stage to being a good developer. Unfortunately this is a “coding standard” that i have found among a vast number of “experienced” developers in the so-called CMMi (deprecated IT standard) level 5 companies in India. This is best described as the infancy of a hacker.

Unit

This is when you take that first baby step towards QA. (Yes and we thought that is the height of expected of a developer right??? Loooooong way to go…….. ) The practice is pretty prevalent now that the heading is self explanatory. Your write code and you write some unit test cases and then you execute them.

This is where the real game starts ( big boys are found here)…… This is where your code is not just Unit tested but it adheres to the defined for the programming language used. This is important because one of the aspects of Quality code is not just that it performs well and correctly…. but also that it is highly maintainable.

The relevance of this derived from the fact that 70% of the IT world out there are involved not in developing fresh code but in maintenance and enhancement of legacy code. So lack of standards in legacy code makes them un-interpretable and the enhancement code tends to be more buggy.

This is the stage of coding that is often left untouched. This stage comes into picture where your code has been developed, reviewed. Once the review is complete, the developer and reviewer has a sit-in to refactor the code. Now for those developers who are going “Huhh, whats refactoring” check out the following link below:

http://en.wikipedia.org/wiki/Refactoring

“halleluia to agile community for driving this practice…….” :-)

Preemptive Functional

Preemptive functional is once of the final qualities of a legend class developer. Where he does all the above mentioned stuff and then sits down to do a bit of functional QA himself. He executes “black box” grade critical test cases and ensures that they pass.

The outcome of adopting the above standards are:

  1. Your build “will never break”
  2. Your QA will tear out the hair trying to find a bug
  3. A rookie out of college you app will come out with zero defects.
  4. has to reach a new height to beat your code.
  5. A legend class developer is born.

For all those testers reading this send this to your developer friends out there……..

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

Heuristic Testing

As a continuum of my post on best fit vs first fit, thought id present as a means of getting this done….

Question is “What is heuristics”

Well, a heuristic is a rule of thumb solution to a problem that may qualify as the solution but need not necessarily be so….. This a terminology where emphasis lies on the potential of a thought, approach or method to be come the solution for a problem.

How does this help …… Well QA and has reached very appreciable maturity levels the worldover (barring India :-) ) that for any given problem posed you have a sizeable set of solutions as compared to what you would have 10 years back. Let say i want to do funtional . There are umpteen number of options at hand like the following:

  • Sticking to age old manual .
  • An automated solution ( just because a can be automated it need not be)
  • The possibility of using my own custom .
  • Possibility of using COTS products ( high industry affinity)

Now is a methodology where the set of n possible solutions are all given equal importance initially. Then begins the probing stage where the validity of accepting any of these n solutions is questioned and a of elimination gets kicked off. Darwin says “The fittest will survive”. Well if thats true for the cosmos then why should not hold true for Methodologies ?

The biggest truth is “Winrunner need not be the best solutions for your functinal requirements just because your QA Consultant know only that !!!!! ”

Sorry if that came across too hard…… but then in many cases i have seen the following classic case “My carpenter knows how to use the only the hammer and i want him to put in a screw. So he bangs the bejesus out of the screw till it goes in….. End result i have a wall that looks like its survived world war II……… and we give him an award for ontime service delivery but failed to realize that the next day a painter and a mason will have to come in for damage control post deployment………” Now i be that sounds like a familiar story…….

This is where we do where the primary focus is not on getting A single solution but to ascertain what options we have. Then we get around to finding our best option after careful analysis by questioning the authenticity of each of our options and let the fittest solutions emerge as the winner….

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

Requirement tracking in India

Recently I was brainstorming with a highly competent tester based in Bangalore, India [a rare breed out here] and we stumbled on a startling fact. gathering and tracking methods have remained largely static for a long time. The delivery models in India with respect to QA and are reminiscent of the late 90’s.

These are some of the facts that we ascertained:

  1. The most prolific and commonly used in the Indian IT service sector is…….. “Microsoft Word”.
  2. Every stakeholder/developer/tester in a project has a personal copy of the document resident in his inbox.
  3. changes are often not centrally held/ distributed.
  4. discussions/ freeze is effected after multiple rounds of talks with the onsite customer/coordinator.

systems today have reached a stage of “critical complexity” where it is imperative that we hold requirements in a central repository. There core reason for no advances in this segment is because our internet non-savvy IT community is oblivious to the concepts of web2.0 and the “collaborative internet”.

The version of the internet as we have it today has moved towards distributed applications with added focus on communities,collaboration and harnessing .

The benefits this can provide to tracking are the following:

  • Use principles to build tracking tools on the “internet platform”.
  • Provide features within the app like forums [to start discussion threads], chat enabling for real time gap resolution, wikis for public bulletin boards.
  • The feature rich UI capabilities of internet based applications can provide “mind map” like interfaces for provide a structural organization for requirements.
  • This would provide the benefit of greater usability and clarity compared to “referring to section 1.1.3 (a) or 1.2.4 (b)” to track requirements.

Requirements are the most critical pieces of information in the IT sector…. Information management is a science that Internet gurus are trying to perfect and they are the forerunners. So its only logical to adopt their practices in “managing” our information rather than stick to “Pubic library model of tracking and analysis”.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

Test Approach: First fit vs Best Fit

A picture is worth a thousand words…… Presenting the synopsis of what i mean

nutbolt

The nut bolt analogy would give u a small idea of how this is relevant to . There is always a set of N possible ways to test and application/product. This is the “superset of all solutions”. Out of this a small subset can be derived by using filters like

  • Feasibility
  • Compatibility
  • Efficiency
  • Capability
  • Time

Once this subset has been derived, we reach the set of n possible solutions. Now in most delivery models that i have encountered, the focus of the group is to find out “a” solution rather than “the” solutions. Especially in the Indian IT scenario testers are highly susceptible to the “tunnel vision syndrome” where the first solutions that rings a bell is pursued.

What do I do about it?

Well for starts here are a few pointers:

  • To start creating an overall strategy, have the best testers on-board work in silos.
  • DO NOT rely on the of the team at this stage. That will come in later.
  • DO NOT interfere with the independent thought of the the “silo’ed testers”.
  • Once individual approaches are derived, bring the team to the table.
  • Battle mode: This is where “” begins. Discuss the various approaches on table.
  • Survival of the fittest: Darwinian theory will do the rest for you. The best will emerge as none of the “silo approaches” but as the common maximum of the best approaches put forth by each tester.

This is how you can ascertain the “best fit” and NOT pursue the first stray thought.

principle: Excellence is always instilled never inbuilt.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

Database Testing: Lookout for ACID-ity

The concept of performing ACID on Databases have been theoretically emphasized over the years, but I have often noticed that this dimension is not an evident driver when it comes to strategizing of application.

Testers these days have failed to look beyond the conventional classroom wisdom imparted about . The for ACID properties is highly essential for any system/application that has a back end.

Conventional test strategies emphasize on achieving 100% test coverage. This will provide proof for functional correctness of the system. Post deployment when a system encounters real user/transactional volumes you may be surprised to find crashes, inconsistencies of data served. These can be traced to overlooking the of ACID properties.

For those of you who are wondering whats the concept this would be a good place to start looking: http://en.wikipedia.org/wiki/ACID

The crux of the matter lies in incorporating sections to include strategies to test for acidity and translate that to specific test cases to target these scenarios.
Post a reply to gain further insights into ACID

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

QA Estimation

 
icon for podpress  QA_Estimations: Play Now | Play in Popup | Download (157)

Estimating Projects

Walkthrough on HowTo

The online content i have found regarding developing sound estimations for projects are found to be wanting in a lot of ways

  • Articles start of promising and end up with “ project estimation - Seat of the pants approach”. :-)
  • Articles packed with a lot of know-know but absolutely no how-to
  • Articles that tell me how to keep doing what we are already doing.

Current Affairs
The current situation of project estimation can be best described as CMM level 1 heroics. The nearest we conventionally come to it is a WAG (Wild Ass Guess). As to how this has become an acceptable practice in the industry defies my comprehension!!?$!?!^@?!!! Sometimes people even go to the extent of categorizing this “exercise in futility” as a SWAG( Scientific Wild Ass Guess).

Bottomline: Doing a WAG or SWAG (Whatever you call it) is as good as doing no estimation at all.

What is an estimate?
An estimate serves as a masterplan for a project covering all aspects like costing,staffing,timing. Hence basing this on pure guess work is a definite NO

Ok history apart lets get started……

Since this is a walkthrough on how to do an estimate we will start from scratch and move to the end…..
Please note that there are a lot of conventional things that we do which can remain as is. Lets concentrate on the flaws that haunt the practice.

1. Collectibles to start an estimate
Starting an estimation is back breaking work :-( . There are the following elusive documents that one has to procure before sitting down to a sensible estimate.

1. Customer Requirements Specification
2. Request for proposal,
3. System specification/ Architecture.
4. Requirements Specification

I have often hear the following complaint, “Owwww but these don,t exist at that point”. The reply to that is that as per SDLC (Waterfall, Iterative, Jumbo circus whatever) models of development these are the primary documents that need to be in place before we estimate for a project.
The necessity of having these document in hand is to prevent the wild ass from guessing about our project :-)

2. Approaches to preparing QA estimates for a project
There are a lot of sound practices that people have been using to prepare development estimates like system lines of code (SLOC), use-case based analysis, function point analysis, object hierarchy trees etc.
In the following section i have proposed some of the techniques that can be used to prepare QA estimates for projects.

1. Begin estimation by conducting a comprehensive study of the system architecture, scope of work and the analyzing the complexity of work.

2. Determine what style of should be used in the strategy (you could chose from many options like based , , module hierarchy trees to name a few). This is a very important step which most QA personnel skip.

3. If you are adopting a module based , prepare a module hierarchy tree to visualize the inter-dependencies between modules.

4. Analyze and assign complexity to each node of this tree. Estimate the number of test cases in each module by analyzing the # of functionalities bundled in each module.

5. Ascertain by past experience or analysis, a realistic projection of QA productivity (no of test cases per person day). This is a metric which varies and WILL NOT FOLLOW ANY PRE-DEFINED ORGANIZATIONAL NORM.

6. Analyze each module to arrive at a preliminary idea of the extent of that is possible in each of these modules.

7. Estimate the strategy of in terms of how you will automate the , what will be the coverage of , what will be the complexity of developing if any. ( a POC might be required for this at a later stage. This has to go into the estimate. Doing POC’s for test is a severly neglected critical component of delivering QA processes.)

The above 7 steps have to necessarily completed and documented before you can start with the estimate.

3. Preparing the estimate
Identify the risks involved. These can range from technology risks, risks introduced by the delivery model that has been adopted and many other factors. Each risk identified has the potential of throwing your estimate haywire. Hence de-risking the project is an important part of estimation. In situation where the number of risks identified are high, it would be a good practice to prepare 2 separate estimates

  • Conservative estimate: This will factor all the overruns in terms of effort, time and cost that would transpire if the risks realize themselves.
  • Optimistic estimate: This will envision the delivery of the project if the risks identified do not materialize.

The benefit of having 2 versions of the estimate is that it would provide a clear cut picture of how bad things can go when risks materialize and what would be the losses incurred if the are let to remain.Conservative estimates can be projectional in nature where the increase in time, effort and cost can be show as a projection in relation to the risk.

The of preparing the estimate start only once the above steps have been completed. All factors in the estimate have to be traceable to the documentation that has been prepared as part of section 2 of this document. The actual of doing the estimate like assigning timelines and resource loading will be guided by the above section and help you arrive at a sound estimate.

The Golden rule of all estimations “DO NOT DO PROJECT ESTIMATIONS WITHOUT ALL NECESSARY SUPPORT DOCUMENTATION IN PLACE.”

I have outlined only the measures to curb the flawed practices in the estimates. These could be incorporated to modify the existing (if any) of preparing estimates within your organization. A complete walkthrough would expand the scope of this article exponentially ;-)

For further insight into the implementation of such procedures at an organizational level you can contact me.

technorati tags:

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

Testing test scripts

The creation of for functionality/performance/load is a that generates code in intself. Hence this is an activity that would require quality control processes tagged to it. Just like the core code that implements system functionality, the form a supporting code base.

The variety of quality processes that a test undergoes can be based on the concept of context based (relevance of context shall be expalined at the end of this post).

1. Peer code reviews

These can be done to leverage all the advantages of conventional code reviews and to implement relevant scripting standards and indetify static bugs within the script. The peer code review has an added advantage: since test also need to be maintained constantly, familiarizing multiple team members with the test script can aid in script maintenance and de-risk the automations plan.

2. Having a skeletal requirements document for test

The benefit of having a skeletal requirements document can aid in ensuring that the script has adequate functionality coverage. This can also be an part of the actual traceablitiy matrix if one is maintained.

3. Performing a dry run sanity test using script

This is a case where a couple of test runs of the script is done and some relevant manual test cases are also executed to ensure that the results of both are in sync.

The relevance of using a context based approach in your test is that based on the context of the system under test and its criticality the amount and type of done on test can be increased or decreased. Hence doing an impact analysis of the possible failure of a test script is the most crucial part of this activity.

Ill fill in more details about these as and when i get around to stumbling on them ;-)

Please leave a comment if you can think of something else.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts

SOFTWARE TESTING PROCESS IMPROVEMENTS

Objective
I have outlined here some of the flawed practices adopted in the phase activities of projects and propose corrective measures to wrong/ absent practices and improvements to existing practices.

I have traced the incorrect practices down the entire lifecycle to elucidate the impacts of such practices and to showcase how the suggested improvements can streamline and eliminate the wrong practices followed in phase of projects.

This document is based on case studies and observations done on projects with respect to factors that impacted the following:

* Overall functioning of teams
* The quality of deliverables
* Root cause analysis of the causative factors for the deviations in estimated delivery/ roll out plan
* QA sign off activities.
Getting Started: Envisioning Phase

The typical envisioning phase QA activities are namely,

* Creation of QA estimates.
* Identification of QA planning/ execution challenges.
* Compatibility/ Feasibility study of scope of
* Identification of project execution risks/ dependencies
* Initial proposal of risk management/ mitigation strategies.

Identified wrong practices

The following are the improvement areas that have been identified with respect to envisioning phase activities:

Creation of QA estimates

Identified problem area: No QA estimate for Sanity

QA does not estimate for the sanity of builds deployed in QA. The fallout of this is that sanity is not identified as an essential part of any QA activities. There are no efforts dedicated to identifying a sanity test suite. Hence the QA team cannot establish baselines on which to accept/ reject builds.

Sanity (if any) when performed is an ad-hoc activity, and hence there is no traceability for this activity.

Recommendations:

Sanity of all builds expected to be delivered to QA has to be factored in the estimates and activities have to be allocated to this in the test planning and execution phase.

Compatibility/ Feasibility study of scope of

Identified problem area: No POC developed to validate compatibility/ feasibility of

QA teams do not do POC’s for projects where they do not have prior platform/ technology expertise to ascertain that there are no compatibility/ feasibility issues with the tool/ scripting language identified for use. The fallout of this is that the QA team is not able to proactively identify the drawbacks if any of the frameworks they have envisioned. Since the issues in the approach are not identified until the actual development of test start no alternative tools/ / approaches can be envisioned.

Recommendations:

POC’s have to be undertaken to ascertain that the indentified framework is compatible with the platform/ technology area being addressed.

“How are we going to do it?”: Planning Phase

The typical planning phase activities involved in are:

* Creation of strategy document
* Design of Test plans for the modules under test
* Development of test

Identified wrong practices

The following are some of the identified practices that are incorrect or needs improvement segmented according to the planning phase activity they belong to:

Creation of T&E strategy

Identified problem area: Lack of a QA build management strategy

QA strategy documents do not outline the strategy that will be adopted by QA to ensure the criteria to accept or reject QA builds. Hence the parameters with regard to “testability” which are mandatory are not aligned with either development team or the customer side stake holders. The fallout of this is that the of accepting or rejecting builds in QA is not formalized and is an ad-hoc . This scenario is aggravated when numerous builds get deployed into QA to rectify deployment issues. The end result being that there is no traceability with respect to build issues since they have been rectified in an ad-hoc manner. There are no QA side documents logging the number of builds rejected and for what reason. Hence the project team does not have a collated document citing the major issues that need to be tackled while doing final builds in production prior to “go-live”.

Recommendations

The strategy document should outline the mandatory checkpoints to ensure testability of builds. This should be coupled with a template document that will be maintained and communicated when accepting and rejecting builds.

Identified problem area: The points of verification of each module are not explicitly mentioned.

The strategy for each module should also identify what all would be the points of verification for that module. This is because of the n number of modules within the system; there will be a small set of points from where QA would extract the outputs of for verification. The relevance of this activity is that when a module is deployed into QA the corresponding points of verification have to be in place to ensure that teams are able to start . The fallout of not adhering to this practice is that the module would be deployed in a healthy build; but if the point of verification is not in place or is not functional; that would impact the testability of the build.

Recommendations

The test strategy for each module should have its points of verification explicitly stated. This is so that this can then serve as checkpoints to start . The development team would also be aware of the necessity of these supporting modules and can ensure that these are in-place.

Design of test plans

Identified problem area: No sanity test suite identified by QA.

Currently as part of the test planning activities QA does not identify the subset of sanity test cases for any module. The fallout of this absent practice is that there is no alignment with the customer side technology/ business stakeholders with regard to what is the critical subset of sanity test cases that have to pass for QA to start . A formal sanity test suite can establish the grounds on which QA would reject a build as un-testable. The of sanity is ad-hoc in nature and in many cases the members of the QA team are themselves unsure of what are the highly critical test cases that have to pass to deem a build as testable. This has led to many person hours of effort being wasted in identifying a module as un-testable and communicating the same to the development team.

Recommendations

For all modules that have been planned for ; QA team should identify and maintain a suite of sanity test cases. An alignment has to be reached with the customer regarding the completeness of the test suite. The status of these test cases can be used to substantiate the acceptance of a QA build. This activity will enhance the customer confidence levels in the adherence of QA personnel to a great extent.

Identified problem area: “Seat of the pants” method of test .

This is a unique methodology that is being followed in many projects. In these cases though the QA team has estimated for the automated of a module, there lacks an absence in the mapping of the manual test cases designed to the developed for the same. Hence the QA team is hazy with regard to the extent of that they have actually driven in the project. The test coverage via is also not specific. There are only rough estimates regarding the extent of planned. These estimates are never re-visited during the course of the project of taking stock of the extent and coverage of .

Recommendations

The QA team has to identify the subset of manual test cases that can be automated. This then has to be documented and revised constantly to ascertain what extent of has been planned and what has been achieved in the project executions. This is an area where the QA teams will have to conduct E-R gap analysis*.

* E-R gap analysis: Expectation – Reality gap.
“How are we doing it?”: Execution Phase

The following are the core set of activities undertaken by QA team during the execution phase of projects.

* Execution of manual test cases.
* Execution of automated test
* Defect reporting/ tracking
* Re- of defect fixes
* Regression of modules
* UAT support (if applicable)

Identified wrong practices

Execution of manual test cases

Identified problem area: Sanity of builds not performed

Since the QA team does not have a pre-defined test plan for sanity ; little or no sanity is done once a build has been deployed into QA for . The initial of such modules is an ad-hoc .

Recommendations

The sanity test plan that has been designed as part of the planning phase activities have to be executed before commences on any deployment or re-deployment in QA. This will ensure faster turn around times in identifying unhealthy builds.

The of build rejection will be more formalized by adopting this practice.

Conclusion

The above procedural flaws in practices have often led to sluggish turn around times for QA. Adopting the recommendations outlined herewith will ensure that QA teams do not incur wasted efforts identifying sick builds and that efforts are more traceable.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Related posts