Software Testing, QA

Pairwise testing


Let our program take a dozen parameters. To test all combinations is very difficult, so you should choose the most common and potentially affecting each other. Bugs that arise by a particular combination of all ten parameters are rare.
The most common are bugs that arise by a particular combination of two parameters. The more information about the mutual influence of the parameters (more precisely – of the mutual non-influence), the more combinations we can not test. In the absence of such information, as well as by complex algorithms of program behavior, you should apply the method of pairwise testing.
Thus, we can simplify our task and test all the possible values ​​for each pair of parameters.

Read the rest of this entry »

Advertisements

Comments on: "Pairwise testing" (10)

  1. montanha said:

    Very comprehensive – I’ll have to go through it more slowly so it can sink in. This knowledge is always good to have in your testing toolkit. I like your style of sharing the knowledge.

  2. “…you should choose the most common and potentially affecting each other. Bugs that arise by a particular combination of all ten parameters are rare.”

    I understand the concepts of pairwise testing, but the question is always – is it relevant for your particular application?

    How can you know what combinations are the most “common”?

    In order to determine which combinations affect each other, you need to read and understand the code, and that’s not what you are proposing here.

    How do you know – in your particular application with a dozen parameters – that bugs arising from a particular combination of parameters are rare? I suspect you don’t know that.

    Without knowing the code, you cannot conclude that pairwise testing is sufficient.

    • Svetlana Kislaya said:

      I have described the principle of pairwise testing in this article. Examples of use will be later.

      «How can you know what combinations are the most” common “? ”

      This information is available from the developers or independently read and understand the code:).
      If the information is not available and all combinations is not possible to verify not it be easier to refer to this method and be confident that we can test all the pairs combinations.

  3. Svetlana,

    Nice post on an important, but – in my view – extremely under-appreciated topic for software testers.

    For you readers interested in learning more about pairwise testing and more thorough combinatorial software testing methods, I would suggest the following sources:

    http://hexawise.com/combinatorial-and-pairwise-testing (which describes empirical evidence showing that this approach is an extremely effective way to design tests in the real world)

    http://www.combinatorialtesting.com/clear-introductions-1 (which contains a handful of very clear introductions with worked examples)

    and our Hexawise testing tool itself (which includes many sample example test plans in the tool). Free licenses are available to the Hexawise test design tool here: https://app.hexawise.com/signup-1-5

    The hardest thing for software testers just starting to use pairwise and/or combinatorial testing methods is understanding which parameters and values should be included in their test plans. The reading above and the sample test plans in Hexawise can be a good start.

    – Justin

  4. Svetlana, thank you for the article!

  5. Hi, Svetlana! Thank you for you article! I think we may implement some of the practicies 🙂

  6. James Bach argued In an interview at TestCast that by using a simple heuristic for choosing a number of random test-sets you can get very close to optimal Pairwise Testing.
    The mathematical reasoning behind it lies in the so called “Birthday Paradox” (http://en.wikipedia.org/wiki/Birthday_problem).
    I wrote a little Perl script to verify it, and indeed choosing randomly and using Pairwise converge very quickly.
    How does it work ? The idea is to know in advance how many sets of random parameters are needed, and this is what James suggests-
    Choose the two variables with the biggest number of discrete values, lets call those number of discrete values A and B. Multiply A by B and multiply again by a constant (2 is a good choice, 3 even better), let’s call the result C.
    Give each variable a random value, test and repeat C times.
    OTOH The benefit of using the Hexawise tools is that they allow you to add constraints and cross dependencies between variables.

  7. Balu S said:

    Hi,
    As of my knowledge Pairwise will not cover 100% testing , i know it is difficult to test each combination so it should be transparent to the all the stakeholders.

    Balu

  8. Avinash Magdum said:

    All Pairs is based on the principle that most defects are caused by either incorrect single values or by an interaction of pairs of values. It is very unlikely that a defect can occur only if three or more parameters have specific values. So if we create test cases based on the combinations provided by All Pairs, we could find most of the defects by executing this set of test cases. Since the set of combinations created by All Pairs is much smaller than set of all combinations, it gives best results (of testing) with minimum efforts. All Pairs testing is beneficial only in following situations,
    1. The scenario has 3 or more parameters (For example, if a scenario has only 2 parameters then it would generate the same set of combinations as all possible combinations. So there is no point in applying All Pairs)

    2. For every parameter there are at least two values such that at least one line of code is different for each value of the parameter. (We can say that a parameter has 2 values, only if the output changes for each value. For example, in a feature of calculating income tax, if the rules controlling calulations of income tax are same across all the States of a nation, like India, then State is not a parameter for calculating income tax, in India. But if any of the rules are different in different states of a nation, like USA, then State is a parameter, in USA)

    Typically All pairs is applied for every high level requirement (where the functionality is decided by a fix set of parameters) and the set of test cases created by applying All Pairs to each of such requirement (seperately) and adding all such set of test cases (for every requirement) to create complete set of test cases for the entire application.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: