preliminaries

Information about preliminaries

Published on March 5, 2008

Author: rupi

Source: authorstream.com

Content

Slide1:  CS/SE 4367 Software Testing, Verification, Validation and Quality Assurance Instructor and TA:  Instructor and TA Instructor: Joao Cangussu www.utdallas.edu/~cangussu Office ECS 3.901 [email protected] Office Hours: Tuesday/Thursday 4:00-5:30pm TA: Syed W. Haider (to be confirmed) Office ECS 3.216 [email protected] Office Hours: TBA Course Syllabus:  Course Syllabus Objective: This course will focus not only on the theory of software testing and quality assurance but also on how testing techniques can be applied in practice to help programmers and testers function more effectively and efficiently. Special topics on the impact of testing on debugging, program comprehension, performance profiling, and reliability estimation will also be covered. In addition, projects including the use of advanced testing techniques supported by industrial toolsuites are designed to help students learn the difference between state-of-art testing and state-of-practice testing. The overall objective is to teach students an integrated solution to reduce software development cost and also improve its productivity and quality. Main Topics:  Main Topics Preliminaries Black-box Testing White-box Testing GUI Test Formal Verification ABET Learning Objectives:  ABET Learning Objectives Ability to understand the goal and different types of software testing Ability to understand the concepts of verification and validation Ability to understand and apply functional testing Ability to understand and apply structural testing Ability to understand and apply mutation testing Ability to understand and apply GUI testing Ability to understand Robustness testing Ability to understand Reliability Assessment Ability to understand and apply Software Testing Tool Text Book:  Text Book Mathematical Theory of Computation* Zohar Manna Dover Publications, 2003 ISBN 0-486-43238-6 (originally published by McGraw-Hill in 1974) * Recommended (not required) Grades:  Grades More Information:  More Information Course Homepage (www.utdallas.edu/~cangussu) WebCT Dishonesty Acknowledgements Aditya P. Mathur Sudipto Ghosh Preliminaries:  Preliminaries Learning Objectives How and why does testing improve our confidence in program correctness? What is coverage and what role does it play in testing? What are the different types of testing? What is testing? How does it differ from verification? What are the formalisms for specification and design used as source for test and oracle generation? Testing: Preliminaries:  Testing: Preliminaries Gain confidence in the correctness of a part or a product. Check if there are any errors in a part or a product. What is testing? The act of checking if a part or a product performs as expected. Why test? What to test?:  What to test? During software lifecycle several products are generated. Examples: Requirements document Design document Software subsystems Software system Test all!:  Test all! Each of these products needs testing. Methods for testing various products are different. Examples: Test a requirements document using scenario construction and simulation Test a design document using simulation. Test a subsystem using functional testing. Verification, Validation, Testing:  Verification, Validation, Testing Verification: Process of determining whether a phase has been correctly carried out Are we building the product right? (Boehm) Validation: Intensive evaluation process that takes place just before the product is delivered to the client Are we building the right product? (Boehm) V & V denotes testing (Verification: mathematically prove program correctness) Problems with formal verification:  Problems with formal verification Adequate mathematical training Economic viability of proofs Proving hard for non-trivial products Correctness of the “prover” itself Finding input, output specifications, loop invariants Wrong specifications What is our focus?:  What is our focus? We focus on testing programs. Programs may be subsystems or complete systems. These are written in a formal programming language. There is a large collection of techniques and tools to test programs. Few basic terms:  Few basic terms Program: A collection of functions, as in C, or a collection of classes as in java. Specification Description of requirements for a program. This might be formal or informal. Few basic terms-continued:  Few basic terms-continued Test case or test input A set of values of input variables of a program. Values of environment variables are also included. Test set Set of test inputs Program execution Execution of a program on a test input. Few basic terms-continued:  Few basic terms-continued Oracle A function that determines whether or not the results of executing a program under test is as per the program’s specifications. Correctness:  Correctness Let P be a program (say, an integer sort program). Let S denote the specification for P. For sort let S be: Sample Specification:  Sample Specification P takes as input an integer N>0 and a sequence of N integers called elements of the sequence. Let K denote any element of this sequence, P sorts the input sequence in descending order and prints the sorted sequence. Correctness again:  Correctness again P is considered correct with respect to a specification S if and only if: For each valid input the output of P is in accordance with the specification S. Errors, defects, faults:  Errors, defects, faults Error: A mistake made by a programmer Example: Misunderstood the requirements. Defect/fault: Manifestation of an error in a program. Example: Incorrect code: if (a<b) {foo(a,b);} Correct code: if (a>b) {foo(a,b);} Failure:  Failure Incorrect program behavior due to a fault in the program. Failure can be determined only with respect to a set of requirement specifications. A necessary condition for a failure to occur is that execution of the program force the erroneous portion of the program to be executed. Errors and failure:  Errors and failure Program Inputs Error-revealing inputs cause failure Outputs Erroneous outputs indicate failure Debugging:  Debugging Suppose that a failure is detected during the testing of P. The process of finding and removing the cause of this failure is known as debugging. The word bug is slang for fault. Testing usually leads to debugging Testing and debugging usually happen in a cycle. Test-debug cycle:  Test-debug cycle Test Debug Done Failure? Testing complete? Yes Yes No No Software quality:  Software quality Extent to which the product satisfies its specifications. Get the software to function correctly. SQA group is in charge of ensuring correctness. Testing Develop various standards to which the software must conform Establish monitoring procedures for assuring compliance with those standards Nonexecution-Based Testing:  Nonexecution-Based Testing Person creating a product should not be the only one responsible for reviewing it. A document is checked by a team of software professionals with a range of skills. Increases chances of finding a fault. Types of reviews Walkthroughs Inspections Walkthroughs:  Walkthroughs TEAM Client representative Rep of team from next phase Rep from SQA group Rep from Specs team Manager from Specs team Reviewer prepares two lists: Items that the reviewer does not understand Items that the reviewer believes are incorrect Managing walkthroughs:  Managing walkthroughs Distribute material for walkthrough in advance. Includes senior technical staff. Chaired by the SQA representative. Task is to record fault for later correction Not much time to fix it during walkthrough Other individuals are trained to fix it better Cost of 1 team vs cost of 1 person Not all faults need to be fixed as not all “faults” flagged are incorrect Managing walkthroughs (contd.):  Managing walkthroughs (contd.) Two ways of doing walkthroughs Participant driven Present lists of unclear items and incorrect items Rep from specs team responds to each query Document driven Person responsible for document walks the participants through the document Reviewers interrupt with prepared comments or comments triggered by the presentation Interactive process Not to be used for the evaluation of participants Inspections:  Inspections Proposed by Fagan for testing designs code An inspection goes beyond a walkthrough Five formal stages Stage 1 - Overview Overview document (specs/design/code/plan) to be prepared by person responsible for producing the product. Document is distributed to participants. Inspections (contd.):  Inspections (contd.) Stage 2 - Preparation Understand the document in detail. List of fault types found in inspections ranked by frequency used for concentrating efforts. Stage 3 - Inspection Walk through the document and ensure that Each item is covered Every branch is taken at least once Find faults and document them (don’t correct) Leader (moderator) produces a written report Inspections (contd.):  Inspections (contd.) Stage 4 - Rework Resolve all faults and problems Stage 5 - Follow-up Moderator must ensure that every issue has been resolved in some way Team Moderator-manager, leader of inspection team Designer - team responsible for current phase Implementer - team responsible for next phase Tester - preferably from SQA team What to look for in inspections:  What to look for in inspections Is each item in a specs doc adequately and correctly addressed? Do actual and formal parameters match? Error handling mechanisms identified? Design compatible with hardware resources? What about with software resources? What to record in inspections:  What to record in inspections Record fault statistics Categorize by severity, fault type Compare # faults with average # faults in same stage of development Find disproportionate # in some modules, then begin checking other modules Too many faults => redesign the module Information on fault types will help in code inspection in the same module Pros and cons of inspection:  Pros and cons of inspection High number of faults found even before testing (design and code inspections) Higher programmer productivity, less time on module testing Fewer faults found in product that was inspected before If faults are detected early in the process there is a huge savings What if walkthroughs are used for performance appraisal? Comparison of inspection and walkthroughs:  Comparison of inspection and walkthroughs Inspection Checklist used to find faults Five step process Formalized procedure in each step Longer time Walkthrough No checklist used Two step process No formalized procedure in the steps Shorter time Strengths and weaknesses of reviews:  Strengths and weaknesses of reviews Effective way of detecting faults Faults detected early in development Effectiveness reduced if software process is inadequate Large scale software hard to review if it consists of smaller largely independent units (OO) Need access to complete, updated documents from previous phase Program testing:  Program testing Testing is a demonstration that failures are not present. Program testing can be an effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence. - Dijkstra Execution-based testing:  Execution-based testing Execution-based testing is a process of inferring certain behavioral properties of a product, based, in part, on the results of executing the product in a known environment with selected inputs. Depends on environment and inputs How well do we know the environment? How much control do we have over test inputs? Real time systems Utility:  Utility Extent to which a user’s needs are met when a correct product is used under conditions permitted by its specs. How easy is the product to use? Does it perform useful functions? Is it cost-effective? Reliability:  Reliability Probability of failure-free operation of a product for a given time duration How often does the product fail? mean time between failures How bad are the effects? How long does it take to repair it? mean time to repair Failure behavior controlled by Number of faults Operational profile of execution Robustness:  Robustness How well does the product behave with range of operating conditions possibility of unacceptable results with valid input? possibility of unacceptable results with invalid input? Should not crash even when not used under permissible conditions Performance:  Performance To what extent does the product meet its requirements with regard to response time space requirements What if there are too many clients/ processes, etc... Levels of testing:  Levels of testing Unit testing Integration testing System testing Acceptance testing Test plan:  Test plan Test unit specification Features to be tested Approach for testing Test deliverables Schedule Personnel allocation Testing and code inspection:  Testing and code inspection Code inspection is a technique whereby the source code is inspected for possible errors. Code inspection is generally considered complementary to testing. Neither is more important than the other! One is not likely to replace testing by code inspection or by verification. Testing for correctness?:  Testing for correctness? Identify the input domain of P. Execute P against each element of the input domain. For each execution of P, check if P generates the correct output as per its specification S. What is an input domain ? :  What is an input domain ? Input domain of a program P is the set of all valid inputs that P can expect. The size of an input domain is the number of elements in it. An input domain could be finite or infinite. Finite input domains might be very large! Identifying the input domain:  Identifying the input domain For the sort program: N: size of the sequence, K: each element of the sequence. Example: For N<3, e=3, some sequences in the input domain are: [ ]: An empty sequence (N=0). [0]: A sequence of size 1 (N=1) [2 1]: A sequence of size 2 (N=2). Size of an input domain :  Size of an input domain Suppose that The size of the input domain is the number of all sequences of size 0, 1, 2, and so on. The size can be computed as: Testing for correctness? :  Testing for correctness? To test for correctness P needs to be executed on all inputs. For N>100, it will take several years to execute a program on all inputs on the most powerful computers of today! Exhaustive Testing:  Exhaustive Testing This form of testing is also known as exhaustive testing as we execute P on all elements of the input domain. For most programs exhaustive testing is not feasible. What is the alternative? Verification:  Verification Verification for correctness is different from testing for correctness. There are techniques for program verification which we will discuss latter. Partition Testing:  Partition Testing In this form of testing the input domain is partitioned into a finite number of sub-domains. P is then executed on a few elements of each sub-domain. Let us go back to the sort program. Sub-domains:  Sub-domains Suppose that and e=3. The size of the partitions is : We can divide the input domain into three sub-domains as shown. Fewer test inputs:  Fewer test inputs Now sort can be tested on one element selected from each domain. For example, one set of three inputs is: [ ] Empty sequence from sub-domain 1. [2] Sequence from sub-domain 2. [2 0] Sequence from sub-domain 3. We have thus reduced the number of inputs used for testing from 13 to 3! Confidence in your program :  Confidence in your program Confidence is a measure of one’s belief in the correctness of the program. Correctness is not measured in binary terms: a correct or an incorrect program. Instead, it is measured as the probability of correct operation of a program when used in various scenarios. Measures of confidence:  Measures of confidence Reliability: Probability that a program will function correctly in a given environment over a certain number of executions. Test completeness: The extent to which a program has been tested and errors found have been removed. Example: Increase in Confidence:  Example: Increase in Confidence We consider a non-programming example to illustrate what is meant by “increase in confidence.” Example: A rectangular field has been prepared to certain specifications. One item in the specifications is: “There should be no stones remaining in the field .” Rectangular Field:  Rectangular Field Search for stones inside a rectangular field. Testing the rectangular field:  Testing the rectangular field The field has been prepared and our task is to test it to make sure that it has no stones. How should we organize our search? Partitioning the field:  Partitioning the field We divide the entire field into smaller search rectangles. The length and breadth of each search rectangle is one half that of the smallest stone. Partitioning into search rectangles:  Partitioning into search rectangles Input domain:  Input domain Input domain is the set of all possible inputs to the search process. In our example this is the set of all points in the field. Thus, the input domain is infinite! To reduce the size of the input domain we partition the field into finite size rectangles. Rectangle size:  Rectangle size The length and breadth of each search rectangle is one half that of the smallest stone. This ensures that each stone covers at least one rectangle. (Is this always true?) Constraints:  Constraints Testing must be completed in less than H hours. Any stone found during testing is removed. Upon completion of testing the probability of finding a stone must be less than p. Number of search rectangles:  Number of search rectangles Let L: Length of the field W: Width of the field l: Length of the smallest stone w: Width of the smallest stone Size of each rectangle: l/2 x w/2 Number of search rectangles (R)=(L/l)*(W/w)*4 Assume that L/l and W/w are integers. Time to test:  Time to test Let t be the time to look inside one search rectangle. No rectangle is examined more than once. Let o be the overhead in moving from one search rectangle to another. Total time to search (T)=R*t+(R-1)*o Testing with R rectangles is feasible only if T<H. Partitioning the input domain:  Partitioning the input domain This set consists of all search rectangles (R). Number of partitions of the input domain is finite (=R). However, if T>H then the number of partitions is too large and scanning each rectangle once is infeasible. What should we do in such a situation? Option 1: Do a limited search:  Option 1: Do a limited search Of the R search rectangles we examine only r where r is such that (t*r+o*(r-1)) < H. This limited search will satisfy the time constraint. Will it satisfy the probability constraint? Distribution of stones:  Distribution of stones To satisfy the probability constraint we must scan enough search rectangles so that the probability of finding a stone, after testing, remains less than p. Let us assume that there are stones remaining after i test cycles. Distribution of stones:  Distribution of stones There are search rectangles remaining after i test cycles. Stones are distributed uniformly over the field An estimate of the probability of finding a stone in a randomly selected remaining search rectangle is Probability constraint:  Probability constraint We will stop looking into rectangles if Can we really apply this test method in practice? Confidence:  Confidence Number of stones in the field is not known in advance. Hence we cannot compute the probability of finding a stone after a certain number of rectangles have been examined. The best we can do is to scan as many rectangles as we can and remove the stones found. Coverage:  Coverage After a rectangle has been scanned for a stone and any stone found has been removed, we say that the rectangle has been covered. Suppose that r rectangles have been scanned from a total of R. Then we say that the coverage is r/R. Coverage and confidence:  Coverage and confidence What happens when coverage increases? As coverage increases so does our confidence in a “stone-free” field. In this example, when the coverage reaches 100%, all stones have been found and removed. Can you think of a situation when this might not be true? Option 2: Reduce number of partitions:  Option 2: Reduce number of partitions If the number of rectangles to scan is too large, we can increase the size of a rectangle. This reduces the number of rectangles. Increasing the size of a rectangle also implies that there might be more than one stone within a rectangle. Is it good for a tester? Rectangle size:  Rectangle size As a stone may now be smaller than a rectangle, detecting a stone inside a rectangle is not guaranteed. Despite this fact our confidence in a “stone-free” field increases with coverage. However, when the coverage reaches100% we cannot guarantee a “stone-free” field. Coverage vs. Confidence:  Coverage vs. Confidence Rectangle size:  Rectangle size p=Probability of detecting a stone inside a rectangle, given that the stone is there. t=time to complete a test. Analogy:  Analogy Field: Program Stone: Error Scan a rectangle:Test program on one input Remove stone: Remove error Partition: Subset of input domain Size of stone: Size of an error Rectangle size: Size of a partition Analogy…continued:  Analogy…continued Size of an error is the number of inputs in the input domain each of which will cause a failure due to that error. Error 1 is larger than Error 2. Confidence and probability:  Confidence and probability Increase in coverage increases our confidence in a “stone-free” field. It might not increase the probability that the field is “stone-free”. Important: Increase in confidence is NOT justified if detected stones are not guaranteed to be removed! Types of testing:  Types of testing Basis for classification All of these methods can be applied here. Testing: based on source of test inputs:  Testing: based on source of test inputs Functional testing/specification testing/black-box testing/conformance testing: Clues for test input generation come from requirements. White-box testing/coverage testing/code-based testing Clues come from program text. Testing: based on source of test inputs:  Testing: based on source of test inputs Stress testing Clues come from “load” requirements. For example, a telephone system must be able to handle 1000 calls over any 1-minute interval. What happens when the system is loaded or overloaded? Testing: based on source of test inputs:  Testing: based on source of test inputs Performance testing Clues come from performance requirements. For example, each call must be processed in less than 5 seconds. Does the system process each call in less than 5 seconds? Fault- or error- based testing Clues come from the faults that are injected into the program text or are hypothesized to be in the program. Testing: based on source of test inputs:  Testing: based on source of test inputs Random testing Clues come from requirements. Test are generated randomly using these clues. Robustness testing Clues come from requirements. The goal is to test a program under scenarios not stipulated in the requirements. Testing: based on source of test inputs:  Testing: based on source of test inputs OO testing Clues come from the requirements and the design of an OO-program. Protocol testing Clues come from the specification of a protocol. As, for example, when testing for a communication protocol. Testing: based on item under test:  Testing: based on item under test Unit testing Testing of a program unit. A unit is the smallest testable piece of a program. One or more units form a subsystem. Subsystem testing Testing of a subsystem. A subsystem is a collection of units that cooperate to provide a part of system functionality Testing: based on item under test:  Testing: based on item under test Integration testing Testing of subsystems that are being integrated to form a larger subsystem or a complete system. System testing Testing of a complete system. Testing: based on item under test:  Testing: based on item under test Regression testing And the list goes on and on! Test a subsystem or a system on a subset of the set of existing test inputs to check if it continues to function correctly after changes have been made to an older version. Summary: Terms:  Summary: Terms Testing and debugging Specification Correctness Input domain Exhaustive testing Confidence Slide96:  Summary: Terms Reliability Coverage Error, defect, fault, failure Debugging, test-debug cycle Types of testing, basis for classification

Related presentations


Other presentations created by rupi

Funny, Weird - Just for a Laugh
26. 09. 2007
0 views

Funny, Weird - Just for a Laugh

Ver9
07. 11. 2011
0 views

Ver9

Tulips
05. 04. 2011
0 views

Tulips

PERFECT LAUNCH FOR CHANDRAYAAN1
21. 10. 2008
0 views

PERFECT LAUNCH FOR CHANDRAYAAN1

Salary Increment Survey
16. 07. 2008
0 views

Salary Increment Survey

bookoflife
07. 03. 2008
0 views

bookoflife

Career Day
06. 03. 2008
0 views

Career Day

Love
05. 03. 2008
0 views

Love

behindeverywoman
05. 03. 2008
0 views

behindeverywoman

To be Married
25. 09. 2007
0 views

To be Married

Gandhi
02. 10. 2007
0 views

Gandhi

clauses
01. 11. 2007
0 views

clauses

Silverlight Splashes
22. 11. 2007
0 views

Silverlight Splashes

spartans
15. 01. 2008
0 views

spartans

fragments
19. 02. 2008
0 views

fragments

Breath Taking Photo
19. 02. 2008
0 views

Breath Taking Photo

Zodiacsigns 1
04. 03. 2008
0 views

Zodiacsigns 1

Technical Writing Process
09. 05. 2007
0 views

Technical Writing Process

SmartData Company Information
05. 04. 2007
0 views

SmartData Company Information

WaheGuru
10. 04. 2007
0 views

WaheGuru

Arnold
14. 11. 2007
0 views

Arnold

3D4D present
20. 09. 2007
0 views

3D4D present