Penetration Testing: A Comprehensive Guide

What is Penetration Testing?

In cyber-security, penetration testing, or pen testing, is a proactive and authorised form of ethical hacking whereby a qualified pentester (cyber-security expert and ethical hacker) is hired to actively seek out and exploit vulnerabilities in a computer system.

These vulnerabilities are then reported back to the client and promptly addressed, preventing potential exploitation by cybercriminals.

In physical security, penetration testing involves a highly trained pentester attempting to gain physical access to a restricted area, unlike cyber security, which focuses on digital systems and physical hardware assessments.

This guide will focus primarily on digital penetration testing. For more information on physical penetration testing, see our comprehensive guide (coming soon).

Why Is Penetration Testing Important?

Pentesting provides a comprehensive understanding of all your digital vulnerabilities, enabling you to manage risk and patch issues.

Many people falsely believe that the point of pentesting is to hack things. In reality, the point of pentesting is to assure companies that their systems are secure.

Organisations are often scared of pentests revealing significant flaws in their digital infrastructure, with fear of embarrassment being a common barrier.

So, it is important to remember that pentest service providers are here to help, and you’d certainly rather an authorised professional find the vulnerability than a cybercriminal.

What are the Different Types of Penetration Testing?

There are several types of pentest, each with its pros, cons, approach, and objective. Here is a quick overview of the most common types of pentest.

Black Box Penetration Testing

Black box pentesting, also called closed or opaque box pentesting, is a blind test whereby a pentester attempts to hack a digital infrastructure without prior knowledge of its composition and security systems. These tests are designed to mimic various aspects of a real-life cyberattack.

White Box Penetration Testing

White box pentesting, or open or glass box pentesting, is a test whereby a pentester surgically attempts to find vulnerabilities within a digital infrastructure with full knowledge of the systems composition and security systems. White box pentesting is the polar opposite of black box pentesting and is not designed to mimic real-life cyberattacks.

Grey Box Penetration Testing

As the name suggests, grey box pentesting, or translucent box pentesting, is a blend of white and black box pentesting where the pentester partially understands the digital infrastructure.

What about Red Teaming?

Red Teaming

Red teaming is a form of ethical hacking designed to simulate a real cyberattack as accurately as possible. Like black box pentesting, red teams start with no prior knowledge of the infrastructure and security systems and, with a clear objective in mind, must break in by any means necessary.

Red team pentests are usually carried out by external penetration testing specialists. To best simulate a real cyberattack, the organisation’s internal cybersecurity teams, commonly called the blue team, are not informed about the oncoming attack. This helps the organisation to develop a comprehensive measure of their cybersecurity.

Purple Teaming

Unlike red teaming, where the blue team is purposefully unaware of the oncoming attack, in purple teaming, the two teams collaborate, using their combined knowledge and ability to identify vulnerabilities and mitigate risks.

What can be Targeted in Penetration Testing?

The quick answer is anything digital, but since that’s hardly helpful, we’ve broken down some of the most common targets for penetration testing. It’s worth noting that while the target may change, the purpose of penetration testing does not – to find and report vulnerabilities for remediation.

Network Penetration Test

Network pentesting involves a pentester trying to find and exploit vulnerabilities in your network security using internal and external network tests. These weaknesses are then reported back so that they can be remediated before anyone can gain unauthorised access to your network(s).

Web Application (Web App) Penetration Test

Web app pentests involve a pentester trying to find and exploit vulnerabilities in your web application using various tools and techniques. Web app pentests are arguably the most common due to the large number of web apps available for public use and the frequency of cyber attacks targeting them.

Social Engineering Penetration Test

Social engineering pentests involve taking advantage of the human element of cybersecurity. Pentesters will attempt to gain access to data, networks, web apps, and much more by targeting humans within the business. Phishing is one of the most common social engineering attacks, so testing human vulnerabilities this way is extremely important.

Cloud Penetration Test

Cloud penetration testing involves a pentester attempting to find and exploit a cloud-based system to assess its strengths and weaknesses. This usually involves examining three areas: the cloud perimeter, the internal environments, and the on-premises cloud infrastructure.

Internet-of-Things (IoT) Penetration Test

During an IoT pentest, a pentester assesses and attempts to exploit the security vulnerabilities of IoT networks and devices before reporting them back for remediation.

What are the Five Stages of Penetration Testing?

As the title suggests, a standard penetration test has five unique stages: reconnaissance, scanning, vulnerability assessment, exploitation, and reporting. It is worth pointing out that this approach doesn’t quite work in some penetration tests, especially web and mobile app tests, but these are the key stages to understanding the basics of pentesting.

Reconnaissance Phase

This stage is all about information gathering. Like a spy, the pentester will attempt to gather as much information about the target system as possible, including IP addresses, network topology, operating systems and applications. This data is then used to develop an effective strategy for breaking into and exploiting the target system.

It’s worth mentioning that there are two forms of reconnaissance: active and passive. Active reconnaissance involves interacting with the target system, and passive reconnaissance uses publicly available and open-source information (OSINT).

Scanning or Discovery Phase

After reconnaissance, the pentester begins probing the target system using different tools and techniques to understand how it reacts to various scenarios and to identify open ports or attack points.

Many of these scanning tools can be automated. Still, as we’ve discussed previously in our oil & gas industry case study, automated pentesting lacks an understanding of context and can therefore miss obvious attack points.

In a white box pentest, the client will likely have already provided the pentester with open ports/attack points that it would like to test. In black or grey box pentesting and red teaming, it is unlikely that any of this information will have been shared, making the first two stages integral parts of penetration testing.

Vulnerability Assessment

This is the final planning stage, where the pentester uses all the knowledge acquired from the passive and active reconnaissance and the automated and manual scanning to identify the most vulnerable attack points and develop a strategy to exploit them.

When analysing which vulnerabilities to prioritise and target, pentesters can use resources like the National Vulnerability Database, or NVD, a “repository of vulnerability management data that analyses software vulnerabilities published in the Common Vulnerabilities and Exposures database (CVE).”

If you want to learn more about vulnerability assessments, check out our comprehensive guide.

Exploitation or Maintaining Access Phase

At this stage, the pentester attempts to exploit vulnerabilities highlighted in the previous phases, simulating a real-life cybersecurity attack. It’s important to note that at no point during a pentest is the purpose of damaging or disrupting. It is only to ascertain how vulnerable each attack point is.

Alongside the vulnerabilities highlighted during the previous phases, plenty of penetration testing methodologies online can be used as guidance for pentesters.

1. The Open Web Application Security Project (OWASP) Top 10 Penetration Testing Checklist is predominantly used for web app pentesting and, occasionally, IoT pentesting. Check our our guide to the OWASP Top Ten.
2. The Open Source Security Testing Methodology Manual (OSSTMM) is used for everything from network pentesting to social engineering.
3. The National Institute of Standards and Technology (NIST) is predominantly used for infrastructure penetration tests.

A pentester usually checks everything on these lists relevant to the target, even if they weren’t flagged as vulnerable in the previous phases.

Finally, this phase highlights how damaging a genuine cyberattack could be to an organisation, its digital infrastructure, reputation, employees, and customers.

Reporting Phase

Once the exploitation phase has been completed, the pentester will remove any trace of their presence in the system, leaving it exactly how they found it before reporting on their findings.

This report must document the vulnerabilities they found, often using the Common Vulnerability Scoring System (CVSS) to grade each vulnerability based on severity.

Once this report has been delivered to the client, it is up to them to choose which vulnerabilities they patch and which they decide are not severe enough to spend valuable resources on. While cyber security experts always strive for perfection, this is not always an option, and sometimes, non-perfect cybersecurity solutions are required.

What Tools and Techniques are Used in Penetration Testing?

The easy answer is that the tools and techniques used in a penetration test depend significantly on the type of pentest being carried out and the target. Every pentesting company will have its own set of open-source, commercial, and in-house tools, each with its own strengths and weaknesses.

That said, here are some very commonly used tools.

Nmap – Network Scanning

“Nmap, also known as Network Mapper, is a highly effective and adaptable network scanning tool. With Nmap, users can effortlessly discover hosts and services on a computer network, detect operating systems, identify open ports, and collect information about different network devices.”

There are 65,535 possible ports on an IP address, and checking all of them manually takes a very long time. So, pentesting companies often use active scanning tools, such as Nmap, to find open ports before cataloguing them depending on what sort of traffic they receive and transmit and what versions they are running, i.e., whether they’re up-to-date.

Nessus Expert – Vulnerability Scanning

“Nessus Expert is an effective tool to help you discover vulnerabilities across your attack surface. It supports scanning across a variety of asset types such as operating systems (MacOS, Windows, Linux), applications, network devices and more.”

Similar to Nmap, tools like Nessus can scan entire networks or work in tandem with tools like Nmap to scan specific ports. Using its database of known vulnerabilities, it can highlight specific vulnerabilities within the network.

Other vulnerability scanners commonly used by pentesting companies include Qualys and Nexpose.

Manual vs Automated Pentesting

As the names suggest, manual pentests are carried out manually by an expert penetration tester, and automated pentests are carried out by software applications designed to automatically scan systems, networks and applications to identify vulnerabilities. The common misconception is that manual pentesting is entirely manual.

The most critical distinction between manual and automated pentesting is the ability of manual pen testing to understand the context of a situation.

Manual pentests absolutely include the use of automated scanners, such as Nessus and Nmap. However, unlike automated scanners, manual pentesters can recognise irregularities and issues within a network based on their understanding of the situation’s context.

To see how this better understanding of context can make a difference, check out our oil and gas industry case study. In this particular case study, a manual pentest revealed a major security flaw within an international oil and gas company’s network that automated scanners had considered safe.

Or, check out automated vs. manual penetration testing comprehensive guide.

What are the Benefits of Penetration Testing?

Penetration testing has far too many benefits to list them all, so we’ve picked three of the most important ones.

Remediation of Vulnerabilities

While it’s pretty apparent, uncovering and remediating vulnerabilities within your environment is arguably the most important benefit of pentesting. Whether it’s highlighting infrastructural errors or employee habits that could be exploited, understanding your vulnerabilities and being able to remediate them is well worth the investment.

Compliance

Penetration testing is crucial for compliance as it ensures adherence to regulatory standards and the protection of sensitive data. Regular pentesting demonstrates due diligence and proactive risk management, often required by PCI-DSS, HIPAA, and GDPR compliance frameworks.

Maintain Trust

Finally, by carrying out frequent pen tests, you can ensure that the personal data of your clients and employees as well as any other sensitive data is safe. This will undoubtedly build trust in your organisation and help to maintain a positive brand image.

Common Misconceptions of Penetration Testing?

Given that pentesting is a very complex service, it’s completely normal that there is an abundance of misconceptions.

Pentesters Do Not Fix the Vulnerabilities

The first one is that some people falsely believe that once a pentester has found and reported a vulnerability, they will attempt to fix it.

While pentesters can be employed by a company to do this, this is not common practice. The role of the pentester is to locate the vulnerabilities and offer advice for remediation. Beyond that, everything is left up to the organisation that booked the pentest.

Additionally, many companies are likely to have in-house developers and I.T. teams who, thanks to their greater familiarity with the system, are better placed to act on the remediation advice offered by the pentesters. Employing a pentester to fix vulnerabilities is, therefore, more time-consuming and can be significantly more expensive.

That being said, if you lack an in-house team capable of fixing vulnerabilities, depending on the pentesting agency, they may be able to fix it for you. Just be aware that this is an additional service with its own costs and considerations.

Pentests are just External Infrastructure Tests

Put simply, external infrastructure tests are a type of pentest but are far from the only type, as discussed above. People also mistakenly believe that pentesting simply involves attempting to bypass a firewall, which isn’t true for the same reasons. Not to mention, thanks to the rise in remote working and cloud infrastructures, firewalls are becoming increasingly less common in business environments.

Nothing to Report is a Waste of Money

When organisations pay for a pentest, the initial hope is that very few vulnerabilities are found. Yet when they receive a report that concludes that there were very few vulnerabilities, they view the pentest as a waste of money.

At the end of the day, whether a pentest reveals major flaws or not, the benefits of a pentest are the peace of mind it provides, thanks to the remediation of vulnerabilities and compliance.

In reality, pentest reports that flag very few vulnerabilities are, in fact, the best kind of reports to receive.

How Often Should You Conduct a Penetration Test?

The simple answer to this is as often as possible, especially if digital changes are made regularly. This ensures you continuously check for and fix vulnerabilities, leaving you least exposed to cybersecurity threats.

That said, regular pentests may not be an option due to the high cost associated with manual penetration testing especially. So, it is recommended that companies always conduct, at the very least, an annual pentest on their digital infrastructure.

As we said previously, you’d much rather an authorised professional pentester finds vulnerabilities in your system than a cybercriminal, and it’s safe to say that while pentesting is expensive, it’s nothing compared to the impact of a genuine cyberattack.

How to Choose a Penetration Testing Provider?

Now that you understand what pentests are, why they’re important, their benefits, and the types of tests available, let’s explore how to choose the right pentest service provider for you and your organisation.

While you could easily search online and choose the first one, a much more reliable method is to look out for CREST-certified providers and CREST-qualified pentesters or use CHECK, a list of NCSC-approved (National Cyber Security Centre) penetration test providers.

“Companies providing CHECK services will do so using staff who hold NCSC approved qualifications and have suitable experience, and pen tests will be conducted using NCSC recognised methods.”

By using CREST-qualified or CHECK-verified suppliers, you significantly increase the trustworthiness and reliability of your pentest provider, which ultimately improves the quality of the test and the remediation advice.

Conclusion

In cyber-security, penetration testing, or pen testing, is a proactive and authorised form of ethical hacking whereby a qualified pentester (cyber-security expert and ethical hacker) is hired to actively seek out and exploit vulnerabilities in a computer system.

There are three different types of penetration tests: black box, white box, and grey box. Red and purple teaming are both very closely related to penetration testing, but they should not be considered the same.

Every company, regardless of size, should carry out penetration tests on their entire digital environment as often as possible, or annually, at a very minimum. If you cannot afford to get everything tested annually, the next best thing is to rotate what you test annually. Just be aware that this can potentially leave you vulnerable.

Finally, when choosing a penetration testing service provider, there are a few things to look out for that guarantee high-quality service and performance. Look for CREST-accredited providers or members of CHECK (the NCSC’s list of approved service providers).