[vlog] Vulnerability Assessments vs Penetration Tests

I occasionally see the terms Vulnerability Assessment and Penetration Test used interchangeably, or worse, phrases such as “Automated Penetration Test” – something that really pains me, as there are very distinct types of assessment. In this article I’d like to show the distinctions between the different types of assessment. Setting aside any argument of specific terminology, I aim to explain the different approaches that can be taken and the aims of each – regardless of what you choose to call them. I aim to assist companies engage with their security assessment providers to ensure that the service they’re getting is what they are expecting and so that they are aware of the alternatives.

I will however attribute a title to each of these throughout this article, but the article is about assessment types not terminology. Don’t get all “Oh I think you mean crackers!” whenever I use the word hacker; that’s not what we’re here for today, just like how I refer to Red Teaming – you might call that one Adversarial Emulation.

Partly the reason I’d like to avoid arguments of terminology is that I’d like to approach this space as a spectrum instead of individual assessment types. On the left we have assessments which are purely performed through automated tools and on the right we have a human-led assessment either working within some defined scope with a specific aspect of security under assessment, or a wider restriction-less engagement of all aspects of a companies defences.

Vulnerability Scanners

On the left of the spectrum we have a tool which you can supply a list of IP addresses or URLs, and potentially a set of administrative credentials. This tool will scan the system and look for vulnerabilities and then score these vulnerabilities in order to allow them to be prioritised. Scoring could be something bespoke for that scanner, or more likely something like the Common Vulnerability Scoring System. There are some well known tools in this space: SAINT, Nexpose, QualysGuard, Nessus to pick a few examples. These systems generally have two ways of determining if a system is vulnerable; signature based which generally work on a systems self-disclosed banner, or exploitation based where a more generic payload is supplied and a result is observed, a good candidate for detection in this way is command injection. However generally that is the point in which the line is drawn, prove the vulnerability and then move on to something else.

These systems may exploit some issues if safe to do so but for the most part do not, additionally these system do not chain vulnerabilities together for further access. They are entirely automated and aim to identify as many vulnerabilities as possible across the breadth of an organisation and present them individually in priority order. As breadth is the aim it stands to reason that a whitebox approach should be taken with contextual information and credentials given to the scanner wherever possible to ensure as many issues as possible can be discovered. The output is most likely a lengthy list of independent configurations issues and missing patches.

Penetration Tests

Penetration Tests are human led engagements that are goal-orientated, generally the intention is to chain together discovered vulnerabilities in a depth first assessment of the network. Where the line is drawn at a full system take over and therefore these assessments involve a degree of exploitation and vulnerability chaining.

These assessments may utilise automated tools to ensure efficient testing although the aim isn’t a complete list of all vulnerabilities on the system but to discover more realistically how far in to a system an attacker could go. With Vulnerability Scans a reduction in scope (or “sampling”) could be acceptable to ensure scans can be performed in a timely fashion wherever the is an expectation that systems are built in and updated in an even and standardised manner – although I would generally recommend against this wherever possible. However with Penetration Testing sampling of this nature is rarely useful and can often lead to artificially impeding the assessor – such as preventing an accurate indication of vulnerabilities (if a database is the target but the active directory authentication service it uses is out-of-scope) or where potential escalation route are out-of-scope preventing attacks such as token impersonation, such as by reducing the in-scope number of workstations.

Penetration Testing should be conducted with an informed defensive team and setup in such a way that systems can be efficiently attacked, such as supplying an assessor target IP addresses. This may be contrary to some peoples beliefs where the feel the defensive team should not be informed; however assessments should aim to accurately determine the effectiveness of a single aspect of security at a time. Are you assessing the security of the systems or the responsive capability of the defensive team? Further to this, a Penetration Test is generally best suited to a company who already perform vulnerability assessments and have managed most of the identified risks presented by automated tools. The output is most likely an example path or small number of paths an attacker could take to full compromise a specific aspect of a companies systems.

Red Teaming

If however, you are in fact looking to assess the responsive capability of the defensive team this type of assessment is better suited to Red Team Engagements. These are generally performed over a longer period of time, with a specific goal in mind, but with no specific tactic restriction in place. Meaning engagements will likely utilise a selection of attacks from exploitation of server vulnerabilities, exploitation of common client-side software and targeted social engineering. Therefore an assessor can utilise any tactic known to be effective to achieve the goal. Whereas a penetration test will likely assess one aspect of security at any one time, internal security, external security, social engineering, etc. Red Teams can take a diverse approach. These are therefore generally more time consuming, expensive assessments which are only suited to companies with a proven high level of overall security – but give a good indication of what could be achieved without any knowledge of the internal systems and under the noses of the defensive team (often called “blue team” in this context). Whereas a Penetration Test may be run from a corporate network connection to determine the level of vulnerability of internal-only systems, which could be targeted by attackers after a successful phishing engagement or by a malicious member of staff, with Red Teaming the assessor generally starts on the Internet and must initially successfully perform the aforementioned phishing engagement before pivoting in to the internal network. The output is most likely a report of a single real-world path of exploitation an attacker could take to bypass all security enforced by the affected company whilst evading defensive staff taking a the attacker from an unprivileged position on the internet to the point showing a specific, real risk to the affected company.

Overall: from the above it should now become evident that testing is a spectrum where there are various time, cost, tactic, and outcome expectation differences with each assessment type. It should also be seen that assessment types “further right” such as Red Teaming are only beneficial to companies who have already addressed risks more easily identified. Although these assessment types can be combined effectively such as quarterly vulnerability assessment and annual penetration testing. It should however be shown that these assessments are not interchangeable and that there are specific and critical difference.

Finally, it should be noted that Penetration Testing, and certainly Red Teaming, cannot be an automated assessment. Whereas there is no problem with a tester utilising scanning tools and testing tools to make an assessment as efficient as possible. Someone simply clicking “Go” on Metasploit Pro is not a Penetration Test. Penetration Tests are technique driven and not tool driven, with a requirement to adapt contextually to the systems as presented, have the ability to compromise bespoke and non-standard deployments. Therefore readers are to be wary of any assessor who informs you a Penetration Test can be driven purely through automated tools like Nessus, or a sales person trying to sell you “Automated Penetration Testing”.