5 Best Practices for your SAST Evaluation26 Feb 2019 by Dr. Johannes Dahse
Choosing the right solution for automated security testing is hard. A good way is to run a proof of concept (POC) of different vendors so you can verify marketing claims before adding another software to your stack. Our best practices can help to prepare an efficient and thorough evaluation so you can unmask snake oil from cutting-edge technology and make the best choice.
Static Application Security Testing (SAST) solutions analyze the source code of applications for vulnerabilities without running or deploying the code. In case you are not sure if SAST is the right approach for you or what different SAST approaches exist we recommend reading our previous blog post about a comparison of different testing approaches.
1. POC Time Plan
To make your evaluation as efficient as possible it helps to sketch a rough time plan. You can then estimate and allocate the minimum time and resources required and move on quickly. If you have a compeling event scheduled, like a security audit, take that date and plan backwards from there. By communicating your time plan early to each vendor you can also reserve their availability and support. Here is a good example recently shared with us:
|Jan 11||Product Demo with Q&A|
|Jan 18||POC Started (if selected)|
|Jan 18||Legal Process Started (if necessary)|
|Feb 1||Preliminary Quote|
|Feb 8||POC Complete|
|Feb 15||Finalize Quote|
2. Vendor Selection Criteria
Many vendors focus only on code quality and report warnings on easy to grep security weaknesses. Because they do not perform in-depth security analysis they are missing complex vulnerabilities. If you are looking for a security testing solution, choose a vendor with a strong security focus instead of a code quality checker with some security rules. Have a look at each vendor’s list of supported vulnerability types and the track record of published vulnerability reports in real-world applications. Testimonials of renowned security experts can also help to get an idea about a vendor’s reputation.
Next, create a checklist for your general requirements to use a SAST product in your software development life cycle. Here is a typical example of a requirement checklist:
|Software Installation||On-Premises, no internet connection|
|Language Support||PHP 7|
|Framework Coverage||Laravel, Symfony|
|CI/CD Integration||Jenkins Pipelines|
|Bug Tracker Synch||Jira|
|Compliance Standards||OWASP Top 10, PCI DSS|
|Reporting||Security Trend, PDF|
|Scan Configuration||Custom Analysis Rules, User Permissions|
In an initial product demo or call you can check for these requirements and find out how to do a product evaluation.
3. The Right Test Code
Since your product evaluation is likely limited in time and scans, you should pick your test code wisely. It should not only match to your expectations of results but also to your vendor’s expectation of code analysis.
Understand the Vendor’s Analysis
Before choosing your test code it helps to look at the analysis approach of the vendor and to roughly understand what their software does. Sure, it should detect security bugs, at best even deeply nested vulnerabilities and at least the bugs you can easily spot yourself in a second.
For example, the following code looks like it has an obvious SQL injection vulnerability that you might expect to get caught by any SAST solution.
The human reviewer quickly concludes that there is apparently a user-supplied request parameter
id in line 1 that is embedded unsanitized into a SQL query which causes a SQL injection vulnerability in line 2.
But what if
getParameter() does not actually receive user input? Or
sqlQuery() does not execute a database query? A SAST solution (with good accuracy) first needs to analyze the code of both functions to reason about what’s going there. That means you need to include the code of these functions into your scan as well. Without this additional code, neither the code example is exploitable nor a security analysis solution can report an issue with certainty.
The power of static analysis is that it can analyze incomplete code but for the most complete analysis results, we recommend to scan an application as it is run in production and available to attackers. That includes all classes and libraries you use. Often only the combination of multiple code parts leads to an exploitable vulnerability, while each component separately may not be exploitable.
Select Code with Known Vulnerabilities
The easy and lazy way is to use an existing benchmark suite but we strongly advise against it for many reasons. All popular suites comprise of very simple test cases that are used by SAST vendors themselves to train and test their products. You should expect every smart vendor to score a perfect 10 here. Further, we’ve seen many faulty test suites with exploitable vulnerabilities in supposedly secure code and vice versa. While test suites are nice to test-drive a solution’s interface features it will not reflect vulnerabilities in your real-world applications in any way.
Instead, we recommend using your own applications for testing. Choose applications that reflect your typical usage in terms of code size and frameworks. Ideally, choose an old version of your code that is known to have concrete security issues that should be picked up by the SAST product (see the previous section). To make sure there is actually something to catch in your code you can take a small application and add a few intentional vulnerabilities with different complexity to a copy of it. You can then test what each SAST solution is able to catch. Ask your developers and they surely help to add a few SQL injection or object injection issues.
4. Result Comparison
It’s time to scan your selected code with the different SAST products. This is a good time to discover the different user interfaces, usability and design.
Clock the Performance
Static code analysis, when done right, takes time. Your source code is typically transformed into a huge graph model where all possible combinations of data flow paths are analyzed for different types of vulnerabilities. Still, you will experience significantly different scan times by different vendors for the same code base.
While your focus should be on the quality of the results (covered in the next section), keep in mind that performance is key when it comes to the integration into your IDE or build process. Particularly if you want to scan your code base several times a day rather than only once a weekend then the scans should finish in minutes.
|Application||Code Size||RIPS||Vendor 1||Vendor 2|
|Admin Panel||400 KLOC||5m||6h||8h|
|REST API||1.0 MLOC||20m||9h||22h|
Quantity is not Quality
It sounds obvious but users are often fooled by a large number of critical findings in the initial scan summary. The more warnings pop up, the more valuable the software is, right? But later during remediation, it can turn out that a vast majority of these findings are false positives or non-sense. Do not make the mistake to compare only the overall number of reported vulnerabilities of the different solutions. Instead, focus on the issues that you know are real and present in your test code. Did the products catch your intentionally built-in vulnerabilities? Ask yourself: Do you want only relevant security issues or also many “could be” issues? Evaluate all findings carefully and verify that these are actual issues that indeed risk or weaken the security of your code.
Obviously, it looks beneficial to the vendor and to the user if a solution reports many items all the time. But keep in mind that you are buying software to save your time, and not to waste it. It’s not helpful, for example, if every single SQL query in your code is reported as a potential SQL injection. When there is no exploitable vulnerability in your code then there should be no report distracting you from your development tasks, or even worse, block your build. Besides, you don’t pay for a software solution to list all your SQL queries (your IDE can do that) but to find and report only real vulnerabilities.
Here are some awful examples of “vulnerabilities” that you can find hundredfold in the scan reports of major vendors.
Use Of Hardcoded Password
Hidden Form Value
CRITICAL: Dangerous File Inclusion
CRITICAL: System Information Leak: External
As shown in the very last example you should also check if the severity rating is realistic. There is nothing wrong in reporting debug code but when every low-severe code quality hint is reported as highly critical vulnerability it will be impossible to reasonably configure the thresholds of your CI/CD tool and to trust that your build fails only on real threats.
5. Integration Testing
If you are planning to integrate the SAST solution into your SDLC we recommend to include relevant integrations into your tests. This can get a little bit more tricky. The good thing is that you learn how easy the integrations can be setup if they can be configured to work the way you want them to, and how the support team reacts if you need help. You will also find out how feasible the analysis performance for your development process is and what the best applications are to begin with in your organization. In return, this will also help in choosing the right license type.
There is no ultimate security solution that can detect or prevent all possible security bugs in your code and every approach has its limits. But there are worlds in between the capabilities of different analysis solutions. With our five best practices you can prepare for your evaluation and find the right solution that fits your security analysis demands. Keep in mind that you are buying a security solution that you trust to detect real security issues in your code and to save your time.