Is static code analysis a testing method or not?

Well, yes and no.

Among testing professionals [http://www.istqb.org], static testing techniques are treated as first class citizens just as any other (dynamic) technique.

Moreover, there are a lot of benefits with static techniques which make them important alternatives or, in fact, superior approaches in certain situations. A static technique is a testing technique that verifies a certain testing condition without the need for actually executing the system under test. This can include different kinds of activities within the development/testing process and can be applied at different testing levels, for instance reviewing requirements documents, checking designs against certain design flaws, code inspection or even manual verification of test case descriptions. A very important implication of static techniques is that they can indeed be used to verify the absence of defects in the system of a specific kind! We know of course that this is impossible in a general case with testing, but for a fixed set of defect types it is possible. On the other hand, with (dynamic) testing we can find only individual instances of defects, and the success rate of this activity (that is, how many defects of a certain kind are found) depends on the amount of testing and the quality of the testing process.

A specific kind of static testing technique is static code inspection. It can be performed at different levels of testing and with different levels of formality, but in general includes the analysis of the source code of the system to locate the defects without actually executing it. A manual review based on a checklist or simply on the expertise of the reviewer is a static code inspection technique. This kind of activity can consume, however, a significant amount of human resources, comparable to that of a comprehensive traditional testing. Fortunately we can save a lot of resources by leaving the mechanical work on the computer. Static code analysis tools can be programmed to detect a list of "suspicious" code which require further attention from the developer or tester. Moreover, there is a certain set of code constructs definitely containing a defect, that can be detected fully automatically.

But don't let ourselves be fooled: exhaustive testing (checking every possible execution of a system) is and will always be impossible, being it a static or dynamic technique. Exhaustive testing would be infinite, so both approaches limit the scope of their investigation. But what is the key difference between them? Static analysis employs a number of (theoretical) assumptions about the program and generalizes the findings for multiple (all possible) executions of the program. For example, it will assume that the compilation and execution will preserve the semantics of the source code text and that if certain conditions hold during a future execution the program (which may be the case with many different possible executions), it will eventually fail (for instance, if a variable x gets a value greater than 0 then pointer p will become a null which will in turn produce a runtime exception). With dynamic testing we also set certain assumptions about the program, but these are more about generalizing one specific case. When we design and execute a test case we assume that it is representative for a set of other possible test cases and because of that we don't execute the other test cases in this set. This is practically what is known as equivalence partitioning, and the effectiveness and efficiency of testing depend exactly on how much are we able to use the idea of generalization and at the same time spare with resources. So we can conclude that both static and dynamic testing limit the scope of the verification using some kind of generalization based on assumptions about the program and, in effect, can be real alternatives to each other in defect detection.

But what about their applicability? What are the costs for using static or dynamic testing? How can we decide which is more economical and how could we combine them? This is not easy to answer since it depends on many different factors like the kind of the development project, the type of the process, the criticality of the application, budget, composition of the team, availability of tools, etc. Generally speaking, dynamic testing is very simple in principle: we design a set of representative test cases and execute them one by one while recording the eventual failures observed. But this involves an extremely lot of practical problems which make testing a really tedious and expensive activity (in many projects it can be attributed about half of the total project costs). Some of the major problems testers face are the vast amount of different test cases required and the practical implementability of the execution of this large number of tests, often requiring serious infrastructure regardless of whether it is a manual test or an automated one. However, the good point is that after we have successfully performed all the tests and if we know that the tests have been designed reliably we can be quite confident about the system since it has been indeed executed as it supposed to be executed in live environment.

On the other hand, static analysis can be quite cheap as it does not require the whole system to be available readily deployed. Many analyses can be performed with relatively simple and cost effective tools addressing a decent lists of possible problems. These can be then fixed prior to the more costly testing. There is a wide range of problems which can be detected by relatively simple static analysis including coding style and compliance checking and the identification of potentially dangerous syntactic constructs (or the lack thereof). Furthermore, there is a whole range of other information that can be extracted from the source code which does not particularly fall into the "defect" category but are indicators of some other quality attributes. For instance, static analysis can be used to detect the most problematic parts of the system in terms of maintainability or reuse (code metrics like complexity, coupling or the detection of code duplication is an example).

The downsides are, however, that (1) the kinds of detectable defects are limited, and (2) the more sophisticated analyses are expensive (also in terms of costs and in terms of analysis complexity and time). A decent check for problems like null pointers or buffer overflow may require very deep analysis (like model checking) which unfortunately is usually very complex making it unsuitable for larger and more complex systems under test. There are of course solutions to check for the absence of specific more complex problems but they usually set a very narrow scope of analysis (in terms of program size and language features). Furthermore, many very important test objectives simply cannot be achieved by purely static analysis. These include: load testing, user interface testing, most parts of integration and system testing and, in general, final user acceptance testing or validation of the delivered systems and documents.

So the final word is that static analysis can help reduce the verification costs and can even provide further benefits for software quality control, but one must find the most appropriate places of static analysis in his/her process keeping in mind that every project is unique, so no universal set of techniques or tools is available but customization and adaptation is always required. In practice, it is a good idea to see static analysis as an integral part of testing (or more generally software quality) efforts. Testing practitioners and respected software quality managers give the following very practical advice:

• The first priority should always be the application of functional testing as it is closest to the final validation by the customer of the system.
• Functional testing can be augmented with different white box techniques to ensure its effectiveness, like measuring and controlling code coverage.
• According to risk-based priority static techniques can come after the above, but from practical point of view they can be started right at the beginning and can be used throughout the software life cycle, since usually they are cheap compared to the benefits they can provide.

The proverb "quality is free" (coined by Phil Crosby) is not new, but it especially fits to the place of static analysis within software quality. If we find defects and other problems early their fixation is cheap compared to finding them in later phases so, overall, investing in static analysis tools, methods and process restructuring will return in most cases.