Is Automation the end of human pentesting?
By Alex Haynes, CISO, CDL
In the past few years, automation in many spheres of Cybersecurity has increased dramatically, but pentesting has remained stubbornly immune to this. While crowdsourced security has evolved as an alternative to pentesting in the past 10 years, it’s not based on automation but simply throwing more humans at a problem (and in the process, creating its own set of weaknesses). Recently though, automated pentesting tools have now surfaced to a point where they are used to automate pentesting under certain conditions. This begs the question, are human pentesters heading for redundancy? Can we replace them with these tools?
To answer this question, we need to understand how they work, and crucially, what they don’t do. While I’ve spent a great deal of the past year testing these tools and comparing them in like-for-like tests against a human pentester, the big caveat here is that these automation tools are improving at a phenomenal rate, so depending on when you read this, it may already be out of date.
First of all, the ‘delivery’ of the pentest is done by either an agent or a VM, which effectively simulates the pentester’s laptop and/or attack proxy plugging into your network. So far, so normal. The pentesting bot will then perform reconnaissance on its environment by performing identical scans to what a human would do – so where you often have human pentesters perform a vulnerability scan with their tool of choice or just a ports and services sweep with nmap or masscan. Once they’ve established where they sit within the environment they will filter through what they’ve found and this is where their similarities to vulnerability scanners end.
Vulnerability scanners will simply list a series of vulnerabilities and potential vulnerabilities that have been found with no context as to their exploitability and will simply regurgitate CVE references and CVSS scores. They will sometimes paste ‘proof’ that the system is vulnerable but don’t cater well to false positives. The automated pentesting tools will then choose out of this list of targets the ‘best’ system to take over, making decisions based on ease of exploit, noise, and other such factors. So for example, if it was presented with an windows machine that was vulnerable to eternalblue it may favour this over brute forcing an open SSH port that authenticates with a password as it’s a known quantity and much faster/easier exploit.
Once it gains a foothold, it will propagate itself through the network, mimicking the way a pentester or attacker would do it, but the only difference being it actually installs a version of its own agent on the exploited machine and continues its pivot from there (there are variations in how different vendors do this). It then starts the process again from scratch, but this time will also make sure it forensically investigates the machine it has landed on to give it more ammunition to continue it’s journey through your network. This is where it will dump password hashes if possible or look for hardcoded credentials or SSH keys. It will then add this to its repertoire for the next round of its expansion. So while previously it may have just repeated the scan/exploit/pivot this time it will try a pass the hash attack, or try connecting to an SSH port using the key it just pilfered. Then, it pivots again from here and so on and so forth.
If you notice a lot of similarities between how a human pentester behaves then you’re absolutely right – a lot of this is exactly how pentesters (and to a less extent) attackers behave. The toolsets are similar and the techniques and vectors used to pivot are identical in many ways. So what’s different?
Well first of all, the act of automation gives a few advantages over the ageing pentesting methodology (and equally chaotic crowdsourced methodology).
The speed of the test and reporting is many magnitudes faster, and the reports are actually surprisingly readable (after verifying with some QSA’s, they will also pass the various PCI-DSS pentesting requirements). No more waiting days or weeks for a report that has been drafted by humans hands and gone through a few rounds of QA before being delivered into your hands. This is one of the primary weaknesses of human pentests since the adoption of continuous delivery has caused many pentest reports to become out of date as soon as they are delivered since the environment on which the test was completed has been updated multiple times since, and therefore, had potential vulnerabilities and misconfigurations introduced that weren’t present at the time of the pentest. This is why traditional pentesting is more akin to a snapshot of your security posture at a particular point in time.
Automated pentesting tools get around this limitation by being able to run tests daily, or twice daily, or on every change, and have a report delivered almost instantly. This means you can potentially pentest your environment daily and detect changes in configuration on an exploitability level on a daily basis too rather than relying on a report delivered weeks later.
The 2nd advantage is the entry point. While with a human pentest you may typically give them a specific entry point into your network, with an automated pentest you can run the same pentest multiple times from different entry points to uncover vulnerable vectors within your network and monitor various impact scenarios depending on the entry point. While this is theoretically possible with a human it would require a huge budgetary investment due to having to pay each time for a different test.
So what are the downsides to all this? Well first off, automated pentesting tools don’t understand web applications – at all. While they will detect something like a web server at the ports/services level they won’t understand that you have an IDOR vulnerability in your internal API or a SSRF in an internal web page that you can use to pivot further. This is because the web stack today is complex, and to be fair even specialist scanners (like Web Application Scanners), have a hard time detecting vulnerabilities that aren’t low hanging fruit (such as XSS or SQLi). This leads to a secondary weakness in automated pentesting tools in that you can only use them ‘inside’ the network. As most exposed company infrastructure will be web based, and automated pentesting tools don’t understand these, you’ll still need to stick to a good ol’ fashioned human pentester for testing from the outside.
To conclude, the technology shows a lot of promise, but it’s early days and while they aren’t ready to make human pentesters redundant just yet, they do have a role in meeting today’s offensive security challenges that can’t be met without automation.
About the Author
Alex Haynes is a former pentester with a background in offensive security and is credited for discovering vulnerabilities in products by Microsoft, Adobe, Pinterest, Amazon Web Services and IBM. He is a former top 10 ranked researcher on Bugcrowd and a member of the Synack Red Team. He is currently CISO at CDL. Alex has contributed to United States Cyber Security Magazine, Cyber Defense Magazine, Infosecurity Magazine, and IAPP tech blog. He is also a regular speaker at security conferences on the topic of offensive security.
