Software Testing Questions

Purdue CS408 Software Testing Midterm 2018

Why test generation is difficult to automate?

Unfortunately, automated test generation is infeasible in general because defining test oracle is mainly a manual process.

What is testing oracle?

test oracle is the procedure that determines if a test fails or passes.

Test Gen -> Test Run -> Debugging -> Back to Test Gen

1. Test gen
2. Test run
3. Stop Condition – # of bugs identified (graph should be reducing), coverage, Mutation Testing

Test case

id

important level 

L1 critical, if fail, can’t continue anything else

L2 functional

L3 usability

The “law of conservation of bugs”:

• The number of bugs remaining in a large system is proportional to the number of bugs already fixed.

Ripple Effect

There is a high probability that the efforts to remove the defects may have actually added new defects

• The maintainer tries to fix problems without fully understanding the ramifications of the changes
• The maintainer makes ordinary human errors
• The system regresses into a more and more failure-prone state

Failure, fault, error

A failure is an unacceptable behavior exhibited by a system

—The frequency of failures measures the reliability

—An important design objective is to achieve a very low failure rate and hence high reliability

A defect is a flaw in any aspect of the system that contributes, or may potentially contribute, to the occurrence of one or more failures

— could be in the requirements, the design and the code

— It might take several defects to cause a particular failure

An error is a slip-up or inappropriate decision by a software developer that leads to the introduction of a defect  

Chapter 1

Software testing is a process, or a series of processes, designed to make sure computer code does what it was designed to do and, conversely, that it does not do anything unintended.

C2

Testing

Testing is the process of executing a program with the intent of find- ing errors. 

A well-constructed and executed software test is successful when it finds errors that can be fixed. 

black-box testing (also known as data- driven or input/output-driven testing) 

exhaustive input testing 

white-box (or logic-driven) 

exhaustive path testing

Regression testing

Saving test cases and running them again after changes to other components of the program is known as regression testing. 

Testing is the process of executing a program with the intent of find- ing errors.
Testing is more successful when not performed by the developer(s). A good test case is one that has a high probability of detecting an undiscovered error. 

A successful test case is one that detects an undiscovered error. Successful testing includes carefully defining expected output as well as input.
Successful testing includes carefully studying test results. 

C3

C4

Decision coverage

In other words, the statement coverage criterion is so weak that it generally is useless. 

A stronger logic coverage criterion is known as decision coverage or branch coverage. 

Condition coverage

A criterion that is sometimes stronger than decision coverage is condition coverage. In this case, you write enough test cases to ensure that each condition in a decision takes on all possible outcomes at least once. 

Multiple-condition coverage. This criterion requires that you write sufficient test cases such that all possible combinations of condition outcomes in each decision, and all points of entry are invoked at least once.

Equivalent class partitioning

Notice that two types of equivalence classes are identified: valid equivalence classes represent valid inputs to the pro- gram, and invalid equivalence classes represent all other possible states of the condition (i.e., erroneous input values). 

Boundary Value Analysis 

Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not. Boundary conditions are those situations directly on, above, and beneath the edges of input equivalence classes and output equivalence classes. 

C5

Module testing (or unit testing) is a process of testing the individual sub- programs, subroutines, classes, or procedures in a program. 

Test-case Design

The test-case design procedure for a module test is the following: 

Analyze the module’s logic using one or more of the white-box methods, and then supplement these test cases by applying black-box methods to the module’s specification. 

Big Bang (non-incremental) testing, test a program by testing each module independently and then combining the modules to form a program.

Incremental testing

The alternative approach is incremental testing. Rather than testing each module in isolation, the next module to be tested is first combined with the set of modules that have been tested already. 

Driver

Test cases are first designed and then fed to module B by passing it input arguments from a driver module.

The driver module must also display, to the tester, the results produced by B.

Stub

A stub module, a special module given the name ‘‘E ’’ that must be coded to simulate the function of module E , accomplishes this. 

4 popular unit testing engines

JUnit, NUnit (C# .Net), TestNG(Java), PHPUnit

Module testing is largely white-box testing.

C6

High-order testing

Applying this definition, even if you could perform an absolutely perfect module test, you still couldn’t guarantee that you have found all software errors. To complete testing, then, some form of further testing is necessary. We call this new form higher-order testing. 

P116

The purpose of a module test is to find discrepancies between the program’s modules and their interface specifications.
The purpose of a function test is to show that a program does not match its external specifications. 

The purpose of a system test is to show that the product is inconsistent with its original objectives. 

System test

1. System testing is not limited to systems. If the product is a program, system testing is the process of attempting to demonstrate how the program, as a whole, fails to meet its objectives.

1. System testing, by definition, is impossible if there is no set of writ- ten, measurable objectives for the product.
   

Volume Testing

A second type of system testing is to subject the program to heavy volumes of data 

Stress Testing

A heavy stress is a peak volume of data, or activity, encountered over a short span of time. 

Security testing (code injection attacks, rop (c / C++ programs), (dos, mim (networking)), (SQL injection, XSS (web)), privilege escalation, information leak (mobile))

ROP is an acronym that stands for return oriented programming.

C7

What is the process of usability testing?

• Test plan – what is test user recall?
• Select Subjects – 3-5, hallway testing (random people in hallway) OR expert testing
• Document user experience – observer, videotaping + think aloud
• Questionnaire – T/F, free form comments, possible questions
• Analyze Results

For example, instead of asking ‘‘What is your opinion of the main menu system,’’ you might ask a series of questions that require an answer from 1 to 5, where 5 is totally agree and 1 is totally disagree: 

1. The main menu was easy to navigate. 
2. It was easy to find the proper software operation from the main
   menu. 

Usability or user-based testing basically is a black-box testing technique.

C9 Testing in Agile environment

Extreme programming

Another iterative and incremental approach that relies heavily on unit and acceptance testing. Probably the best known of the Agile methodologies. 

XP is a software process that helps developers create high- quality code, rapidly. 

XP focuses on: 

Implementing simple designs.
Communicating between developers and customers. Continually testing the code base.
Refactoring, to accommodate specification changes. Seeking customer feedback. 

C10 Testing web app

Tier 1, presentation tier.

The layer of an Internet application that provides the user interface (UI; or GUI).

1. Content testing. Overall aesthetics, fonts, colors, spelling, content ac- curacy, default values. 

2. Website architecture. Broken links or graphics.
3. User environment. Web browser versions and operating system configuration. 

Tier 2, business layer.

The layer that models your business processes, such as user authentication and transactions. 

Performance. Test to see whether the application meets documented performance specifications (generally specified in response times and throughput rates).
Data validity. Test to detect errors in data collected from customers. 

Transactions. Test to uncover errors in transaction processing, which may include credit card processing, e-mailing verifications, and calculating sales tax. 

Tier 3, data layer.

The layer that houses data used by the application or that is collected from the end user. 

Response time. Quantifying completion times for Structured Query Language (SQL) operations.
Data integrity. Verifying that the data are stored correctly and accurately. 

Fault tolerance and recoverability. Maximizing the MTBF and minimizing the MTTR.  Mean time between failure, mean time to repair.

Selenium is a portable software-testing framework for web applications. Selenium provides a playback (formerly also recording) tool for authoring tests without the need to learn a test scripting language (Selenium IDE). It also provides a test domain-specific language (Selenese) to write tests in a number of popular programming languages, including C#, Groovy, Java, Perl, PHP, Python, Ruby and Scala. The tests can then run against most modern web browsers. Selenium deploys on Windows, Linux, and OS X platforms. It is open-source software, released under the Apache 2.0 license: web developers can download and use it without charge.

Testing on Mobile device 

Challenges

1. Mobile Device Diversity, operating systems, browsers, application runtime environments, screen resolutions, user interfaces, ergonomics, screen size, and more.
2. Carrier Network Infrastructure. Testing your application on a carrier network sets up another challenge. This is especially true if you want to support multiple carriers. 
3. Scripting. An often-overlooked area of mobile applications testing is creating and running test scripts. Real devices do not allow you to load automated, repeatable scripts onto the device; test personnel manually execute all scripts. 
4. Usability. Usability testing presents challenges similar to those of test scripts. For instance, you will want to search for UI consistency issues between Apple’s products and the Android-based platforms.

Inspection team as:

• Moderator - leader
• Author - describe rationale for lines of code, answer questions
• Narrator - read the code, line by line
• Recorder - log the defects

Author is not Moderator, Reader, or Recorder