The following table lists a side-by-side comparison of different application security testing approaches. Additional rating details are available when hovering over each column. In the following, each approach is introduced.
|Category||Automated Security Testing||Manual Security Testing|
|Approach||Static Application Security Testing (SAST) / Static Code Analysis||Interactive Application Security Testing (IAST)||Dynamic Application Security Testing (DAST) / Blackbox Tools||Whitebox / Code Audit||Blackbox / Pentest|
|Taint Analysis||Pattern Matching|
|Language Specific (RIPS)||Language Generic|
|Detect Complex Issues|
|Detect Logical Flaws|
Automated Application Security Testing
An automated security test of an application can be performed in two disparate ways. Either the source code files of the application that is written in a specific programming language are automatically scanned (static analysis), or the URL/IP of an already setup and running application is tested from remote (dynamic analysis).
Static Application Security Testing
Static analysis is performed solely on the source code of an application without executing it. This has the great advantage that the source code must not be running or be functional such that SAST tools can be directly integrated into the development process and detect security issues as early as possible, when the code is written. The available source code is scanned and all issues are pinpointed to the exact line of code for quick remediation. There are fundamentally different approaches regarding the complexity of a code analysis though and the amount of false positives.
> Taint Analysis
For taint analysis, the complete source code of the application is transformed into an abstract graph model that enables efficient data flow analysis. The data flow of user input (sources such as a GET and POST parameter) is traced throughout the complete code base, including the boundaries of files, functions, classes, and methods. Whenever user input flows into a security sensitive operation (e.g. a SQL query or a file access) an attacker could manipulate this operation and thus a security vulnerability is reported (e.g. a SQL injection or path traversal vulnerability). Input sanitization or validation is recognized to prevent false positives.
A taint analysis can only work as precise as its underlying abstract model is. Different programming languages have different features, behaviors, and pitfalls. RIPS tailors its awarded analysis algorithms specifically to each programming language. It simulates all language-specific features and characteristics in order to generate the most precise and efficient model possible. Specifically dynamic programming languages such as PHP are known for their typing issues and pitfall-prone built-in features that are precisely simulated. As a result, RIPS detects even complex and subtle security issues that generic solutions miss. Further, it can even generate a fixed code line to automatically remediate vulnerabilities.
Other SAST vendors use one generic model for multiple programming languages. While this can work for similar languages such as C and Java - the languages originally targeted by the vendors - this fails when dynamic scripting languages such as PHP are added later to the same model. The language-specific details which are often the root cause of modern vulnerabilities are lost in the generic abstraction layer used for fundamentally different languages. As a result, critical security vulnerabilities are missed and many false positives occur. Further, the analysis takes multiple hours or even days to complete which is impractical for continuous testing.
> Pattern Matching
There are static analysis tools that do not perform data flow analysis but only fingerprint for certain keywords or patterns in source code. For example, a tentative security report is issued whenever the function
eval() is found without actually verifying if an attacker could influence the evaluated code or not. While this works to find simple code quality issues, this approach fails to find real and exploitable security issues. A report for every
*query() to detect possible Cross-Site Scripting or SQL injection vulnerabilities leads to thousands of false positives while more complex vulnerability types remain undetected.
Dynamic Application Security Testing
DAST or blackbox tools perform a lightweight scan from the client-side of a given web application that is deployed and running. Multiple malicious input patterns for common web attacks are automatically send to the URL of the application while its responses are evaluated for abnormal behavior (e.g. SQL error messages or time delays) that could indicate a vulnerability. It is recommended to use an additional test setup to prevent interference with real user data. This fuzzing approach is very slow and only scratches on the surface of an application without crawling all features deep enough. For example, vulnerabilities are missed that require a specific combination of actions (e.g. login first, activate mode 1, use feature 5). As a result, blackbox tools have a limited code coverage, a lack of support for many vulnerability types, and miss many security issues. DAST tools are often used for assistance in manual penetration tests.
Interactive Application Security Testing
IAST checks for security vulnerabilities during the execution of an application when run on a modified language interpreter (agent). When the application is deployed and executed, the current execution path is tested for security vulnerabilities by following user input into security sensitive operations at runtime (dynamic taint tracking). In order to cover as many execution paths as possible, and thus parts of the source code, IAST is typically run during functional testing. As a result, the code coverage highly depends on the coverage and completeness of the hand-written test cases by developers. On the upside, false positives can be eliminated since all reported security vulnerabilities are already verified in practice.
When a security vulnerability is detected at runtime it can also be blocked. This approach is known as Runtime Application Security Protection (RASP). The problem with this approach is that a modified interpreter brings a performance overhead and can slow down business critical applications. Even worse, a false block due to misconfiguration or faulty detection leads to a denial of service for customers. Further, with IAST and RASP security vulnerabilities are detected rather late in the development process and are thus costly to remediate.
Manual Application Security Testing
Similar to automated application security testing, a manual test can be performed for two different scenarios: find as many bugs as possible in a given source code (audit) or simulate an attack against a running application (pentest).
Whitebox Approach / Code Review
The most thorough way of finding and eliminating all possible security vulnerabilities is to perform a manual review of the source code. A team of code auditors is hired which manually inspect relevant parts of the source code for security issues on-site or from remote. This enables skilled experts to find subtle security bugs that automated tools and developers missed, for example logical issues or complex crypto weaknesses. A final report will list all findings of the audit with remediation advices. This approach fits best when the code is highly business critical and not rapidly changing. However, for modern applications with thousands or millions of lines of code a manual code audit can be infeasable in a limited time frame or get very expensive. Static analysis tools can be used for assistance.
Blackbox Approach / Penetration Test
More commonly, web applications are tested from the outside (blackbox) and without the source code (whitebox). The goal is to simulate an attack and to get a snapshot of how successful an attacker could be. A team of penetration tester is hired that attacks a production or test setup of the application in a realistic scenario: only with access to the URL/IP and without further knowledge about the internals. A crucial factor is how much time is given to the testers. The final report can only list what was found in the limited time frame. This time frame should reflect the resources of a real attacker which varies from a few days for script kiddies to several weeks for motivated adversaries.
As a personal side note I would always recommend to hire a small boutique company with a strong team specialized in your technology stack, preferably with a recommendation or a list of renowned experts. There is a huge difference in what a team of skilled security experts can find manually in your site than a team that only uses automated blackbox tools.
In this post we looked at different approaches for application security testing. Each approach has its own advantages and disadvantages. There is no ultimate approach that fits to all setups, application stages and attacker models. For example, manual code audits are the most thorough approach to finding security issues in code but are expensive and don’t scale to agile development or large code bases. To continuously detect and remediate security bugs as early as possible during the inhouse development of applications, language-specific SAST clearly outperforms all other approaches. For this reason, top development teams from organizations like Google, Facebook or Vimeo are even building their own SAST tools.