How we test

Methodology

Security evaluations are different from common testing. Common testing normally defines a set of requirements, test plan and standard tools to automatically test features, protocols or certain communication interface. For security evaluations, we don’t have such fix framework, we often say ‘security is a moving target’. By definition, the designer of a product will try to find new measures to protect its product, and on our side we will try to find new methods to circumvent such measure. Therefore, there is a kind of constant race between designer and attacker to improve their skills. Hence, the scope and methods used during security evaluation also differ depending on the TOE (Target of Evaluation) , the approach we follow and the time.

White Box VS Black Box approach

White Box (WB) or Black Box (BB) refer to how much access we will get to the device when we start the evaluation. WB means that the vendor gives us access to the full product design documentation and source code. BB means that the vendor does not gives us access to such documentation or source code.
Both approach have pros and cons and should aim at different goals.

BB approach advantage is that it puts us in a real life situation. We have the same device and resources than an external attacker, hence our results should reflect what the vendor is exposed to ‘on the street’. However, because of this limited access, it is hard to predict what the outcome of the evaluation will be. If the vendor asks us to spend 4 weeks on such assessment, we have no possibility to anticipate if we can ‘brake’ the product or not. Hence, the outcome results is uncertain.

WB approach advantage is that having access to design documentation and source code allows us to first perform an overall review of it, then define the features we want to focus on and finally perform the tests. It allows us to go really deep in performing these attacks and we are able to give a better status at the end of the evaluation. The challenge of such WB evaluation is in extrapolating this level of access when finding a vulnerability. If we compromise an asset with such access, how long would it takes for an attacker without these resources.

BB approach advantage is that it puts us in a real life situation. We have the same device and resources than an external attacker, hence our results should reflect what the vendor is exposed to ‘on the street’. However, because of this limited access, it is hard to predict what the outcome of the evaluation will be. If the vendor asks us to spend 4 weeks on such assessment, we have no possibility to anticipate if we can ‘brake’ the product or not. Hence, the outcome results is uncertain.

WB approach advantage is that having access to design documentation and source code allows us to first perform an overall review of it, then define the features we want to focus on and finally perform the tests. It allows us to go really deep in performing these attacks and we are able to give a better status at the end of the evaluation. The challenge of such WB evaluation is in extrapolating this level of access when finding a vulnerability. If we compromise an asset with such access, how long would it takes for an attacker without these resources.

BB approach is more like a GO/NOGO evaluation limited in a certain execution time. WB approach is more about quantifying the level of robustness of a product.

Vulnerability Analysis & Penetration Testing

When executing a White Box approach evaluation, we have a fair level of access to design documentation and source code. This gives us the opportunity, before jumping into testing, to review the product features to better spot the ones we think deserves the most attention. It is called a VULNERABILITY ANALYSIS (VA).

A vulnerability will consist of three main steps:
- Documentation Review: the analyst reviews the product documentation to get familiar with it and start identifying the implemented functions.
- Code Review: the analysts now review the source code provided by the vendor, trying to map this code with the documented functions and also comparing the quality of the code with secure coding best practice.
- Vulnerability analysis: after all these reviews, the analyst now analyzes all results and define a penetration testing plan. This plan should summarize the attack paths identified which could lead to successful attack, what attack path should be tested or skipped, and the methodology (tests) to be executed. VA is a critical part of an evaluation report since it scopes the evaluation execution.

Penetration Testing (PT) is the phase following the VA. The analysts are now in charge of executing the tests according the vulnerability analysis. This is when we try to compromise the assets identified in the VA, following the attack paths. Where the VA should represent most of the work to be executed, it is common that some penetration testing results will modify our understanding of the product (discover counter-measures) and trigger new ideas. In such case, the analyst may decide to perform new penetration testing.

At the end of the penetration testing, the analyst shall write all findings, including the VA, in a report. If vulnerabilities are found, these vulnerabilities shall be rated to represent their criticality, taken into accounts, for example:
- WB approach and level of access
- Required expertize to execute the attack
- Required tools to execute the attack
- Exploitation impact of this attack
- Scalability of this attack
- Compensating mechanisms in place to mitigate the attack