Thanks - I think the article captures the challenges of modern appsec pretty well. But I'm afraid I'm not aligned with Nir on the solution. You've written about the "SaaSification of Code" in an earlier Cyber Why. Unfortunately, you can't do very much security analysis of the individual pieces of a complex distributed app. For example, an injection path might go through a web front end, a bunch of serverless functions and APIs, and eventually reach a sink. Moving static analysis (and SCA to a lesser extent) to the left loses context and results in far worse results (both FP and FN).
What we need to address the challenges Nir pointed out is better *context* -- and it doesn't really exist until the entire thing is built and running. To me, the "shift smart" strategy is to do security activities when they are fast, accurate, scalable, actionable, and cost-effective. And for most rules in modern apps, that means a runtime analysis of some sort. If you'd like, I'm happy to do a guest post explaining why the "shift left" strategy has failed and many companies are now "left" with huge backlogs of false positives and very little assurance that they've addressed the important issues.
Thank you both for publishing your interesting thoughts!
Respectfully, I think you both come at it from different angles which are complementary.
In his shift left article[1], Jeff rightfully makes the point that modern software doesn't live on an island, but instead is integrated with many "saas-ified" libraries. Accepting this as truth means you can't check for (all kinds of) vulnerabilities locally, you need to trace through the whole system. I think that's certainly true. Even more, I think putting 2 pieces of perfectly secure software together can introduce new vulnerabilities, because security breakdowns often happen between components.
So Jeff's point is you can't be in the IDE (shifting left) because you don't have the context. Which contrasts with Nir's point above, where he talks about "don't put work in the hands of a dev, have a 1 click solution". I think it's a very good idea to be in the IDE of a developer and have a squiggly line with auto-fix where said dev is not using the built-in method provided by the framework to access the database. Sure, it might be a false positive because the line above filters the input value, but when the cost of resolution is so low, why not just do it?
Which brings me to a point where I feel the "test the whole thing when it's running" is idea is weaker: coverage. I've seen many applications where the DAST tooling was unable to follow certain paths (ie because they need specific input), leading to large untested parts of the application.
I understand there are metrics about coverage now, but I'm not familiar with them. So coverage might be less important than I think.
All AST techniques must deal with coverage. As noted, DAST has many coverage challenges. Hook up a code coverage testing tool and get a surprise. SAST also has serious coverage problems. Even though it seem like it scans all the code, that’s not how they work. They build a model and attempt to trace paths from the entry points they can identify. Most don’t even explore library code!! Much less the rarely talked about but super important appserver and runtime platform code.
The problem is that SAST and DAST don’t disclose their coverage and give you no easy path to improve it. So you end up with a false sense of security.
IAST, on the other hand, has two key coverage advantages. First, it covers all the code including custom code, libraries, framework, appserver, and runtime platform. Second, IAST analyzes any code that runs, in dev, test, or even production. And it measures exactly which routes have and have not been exercised. This way, you can easily improve coverage to 100% by creating some simple end-to-end tests.
Across all our customers, the average route coverage is 83%. I believe that’s better than what you can get from traditional tools no matter how much you tube them. And I believe IAST offers the easiest path to 100% coverage, along with the speed, coverage, and accuracy advantages mentioned above.
Thanks - I think the article captures the challenges of modern appsec pretty well. But I'm afraid I'm not aligned with Nir on the solution. You've written about the "SaaSification of Code" in an earlier Cyber Why. Unfortunately, you can't do very much security analysis of the individual pieces of a complex distributed app. For example, an injection path might go through a web front end, a bunch of serverless functions and APIs, and eventually reach a sink. Moving static analysis (and SCA to a lesser extent) to the left loses context and results in far worse results (both FP and FN).
What we need to address the challenges Nir pointed out is better *context* -- and it doesn't really exist until the entire thing is built and running. To me, the "shift smart" strategy is to do security activities when they are fast, accurate, scalable, actionable, and cost-effective. And for most rules in modern apps, that means a runtime analysis of some sort. If you'd like, I'm happy to do a guest post explaining why the "shift left" strategy has failed and many companies are now "left" with huge backlogs of false positives and very little assurance that they've addressed the important issues.
Thank you both for publishing your interesting thoughts!
Respectfully, I think you both come at it from different angles which are complementary.
In his shift left article[1], Jeff rightfully makes the point that modern software doesn't live on an island, but instead is integrated with many "saas-ified" libraries. Accepting this as truth means you can't check for (all kinds of) vulnerabilities locally, you need to trace through the whole system. I think that's certainly true. Even more, I think putting 2 pieces of perfectly secure software together can introduce new vulnerabilities, because security breakdowns often happen between components.
So Jeff's point is you can't be in the IDE (shifting left) because you don't have the context. Which contrasts with Nir's point above, where he talks about "don't put work in the hands of a dev, have a 1 click solution". I think it's a very good idea to be in the IDE of a developer and have a squiggly line with auto-fix where said dev is not using the built-in method provided by the framework to access the database. Sure, it might be a false positive because the line above filters the input value, but when the cost of resolution is so low, why not just do it?
Which brings me to a point where I feel the "test the whole thing when it's running" is idea is weaker: coverage. I've seen many applications where the DAST tooling was unable to follow certain paths (ie because they need specific input), leading to large untested parts of the application.
I understand there are metrics about coverage now, but I'm not familiar with them. So coverage might be less important than I think.
[1] https://www.forbes.com/sites/forbestechcouncil/2023/05/08/application-security-part-1-shifting-left-or-shifting-smart/?sh=ddfc1ef45e97
Let’s explore coverage…
All AST techniques must deal with coverage. As noted, DAST has many coverage challenges. Hook up a code coverage testing tool and get a surprise. SAST also has serious coverage problems. Even though it seem like it scans all the code, that’s not how they work. They build a model and attempt to trace paths from the entry points they can identify. Most don’t even explore library code!! Much less the rarely talked about but super important appserver and runtime platform code.
The problem is that SAST and DAST don’t disclose their coverage and give you no easy path to improve it. So you end up with a false sense of security.
IAST, on the other hand, has two key coverage advantages. First, it covers all the code including custom code, libraries, framework, appserver, and runtime platform. Second, IAST analyzes any code that runs, in dev, test, or even production. And it measures exactly which routes have and have not been exercised. This way, you can easily improve coverage to 100% by creating some simple end-to-end tests.
Across all our customers, the average route coverage is 83%. I believe that’s better than what you can get from traditional tools no matter how much you tube them. And I believe IAST offers the easiest path to 100% coverage, along with the speed, coverage, and accuracy advantages mentioned above.