Security testing: types, OSSTMM and where it fits in delivery

Types

5: static, dynamic, penetration, dependency and config review, threat modelling

OSSTMM channels

5: human, physical, wireless, telecommunications, data networks

OWASP Top Ten

10 risk categories, revised roughly every 4 years

Pipeline placement

static at commit, dynamic at deploy to test, pen test pre-release

Magazine issue

Issue 6, June 2009, security theme

Five types of security testing compared

The word security testing gets used for everything from a linter rule to a two-week red team engagement. Separating the types matters because they find different faults, cost different amounts and belong to different people. Static analysis reads code without running it and flags dangerous patterns such as string-built database queries. Dynamic analysis attacks the running application through its inputs. Penetration testing is a human, goal-driven attempt to break in, usually time-boxed. Dependency and configuration review checks third-party components for known vulnerabilities and servers for weak settings. Threat modelling happens before any of that, on a whiteboard, asking what an attacker would want and how they would get it.

OSSTMM in plain terms

The Open Source Security Testing Methodology Manual, published by ISECOM, is the methodology most often named when a team asks how to make security testing repeatable rather than heroic. Three of its ideas are useful even if you never run a full OSSTMM audit. First, it divides the target into channels: human, physical, wireless, telecommunications and data networks. A test that covers only the data network channel has left four doors unchecked, and saying so in those words is clearer than a list of tools. Second, it separates what is visible, what is accessible and what is trusted, the three ways an attacker gets a foothold, and asks the tester to enumerate each before attacking any. Third, it introduces the rav, a risk assessment value that scores the attack surface against the controls protecting it, so two audits of the same system a year apart produce comparable numbers instead of two different narratives.

Why does a methodology beat a checklist? A checklist tells you what to look for and stops when the list ends. A methodology tells you how to look, how to record what you found and did not find, and how to compute a result that another tester could reproduce. When the auditor asks what your security test covered, a rav and a channel map answer the question; a list of tool names does not. Issue 6 of the original magazine (June 2009) carried a security testing theme, including an article on the OSSTMM; the issue 6 page describes it.

The OWASP Top Ten as a risk vocabulary

The OWASP Top Ten is a ranked list of the most common web application risk categories, revised every few years from real vulnerability data. OWASP describes it as a shared vocabulary, and says it should not be used as a test plan. Broken access control, cryptographic failures, injection, insecure design, security misconfiguration, vulnerable and outdated components, identification and authentication failures, software and data integrity failures, logging and monitoring failures, and server-side request forgery. When a tester logs a defect as broken access control rather than user can see other user's invoice, the developer, the architect and the auditor all know the class of fault and its usual fix. Map each category to one of the five testing types and you have a coverage matrix: injection to static and dynamic analysis, access control to penetration and manual testing, outdated components to dependency review, insecure design to threat modelling.

Where security testing sits in a delivery pipeline

Place each type where its cost is lowest and its feedback fastest. Static analysis and secret scanning run on every commit and should finish in minutes; a finding at this stage costs a developer a few minutes. Dependency scanning runs on every build. Dynamic analysis runs against the deployed test environment, alongside the API and integration tests described in continuous testing, and takes 10 to 30 minutes. Penetration testing runs before a major release and on a schedule, because a human attacker is too slow and too expensive for every change. Threat modelling runs whenever a new trust boundary appears: a new integration, a new role, a new data store. The magazine's sister title Security Acts carried IT security coverage for the wider organization.

What a tester without a security background can start doing

You do not need to be a penetration tester to raise the security level of a product. Three of the five types are within reach of any functional tester in a quarter, and the other two become more effective when a tester prepares the ground.

1. Add abuse cases to every story. For each acceptance criterion, write the misuse version: what if the user changes the ID in the URL, submits the form twice, or uploads a 2 GB file. These are black-box tests with a hostile stance.
2. Own the dependency review. Read the scan output, triage by whether the vulnerable function is actually reachable, and track fixes like any other defect with a severity and an age.
3. Run the dynamic scanner and learn to read its report. Half the findings will be missing headers and cookie flags that a developer fixes in an hour; learning which half is the skill.
4. Facilitate threat modelling. A tester who asks what could go wrong for a living is the natural facilitator; use a simple prompt list (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege) and record the answers as test conditions.
5. Prepare the penetration test. Give the specialist a current architecture diagram, test accounts at every role, and the list of what your own testing already covers, so their days go on what only they can find. Rank the findings with the same risk-based approach you use for functional defects.

This guide is part of the software testing techniques hub. It is best read alongside performance testing and equivalence partitioning and boundary value analysis, which cover the neighbouring questions.

Work it out