A common feature at security conferences, especially those that demonstrate hacks, is the proof of concept. This typically involves a security researcher showing off an exploit against a vulnerable system. Often the result of these exploits is dramatic; the “cash machine jack pot” and various SCADA hacks spring to mind as examples of this kind of talk.

The aftermath of this kind of presentation can be dramatic, with IT security teams scrambling to patch vulnerabilities and, increasingly, senior policy makers developing approaches to cyber security issues based on these proofs of concept.

While these hacks are obviously important, what do they tell us about the vulnerability of similar devices that sit external to a lab environment? How useful are proof of concepts within the scope of a wider security program?

Simply because a phenomenon can be created and observed inside a laboratory environment, this does not automatically mean that it will occur outside of this environment under the same controlled conditions. This is the case across multiple academic disciplines as diverse as archaeology to engineering however, cyber security would appear to be a little late to the party in contextualizing what a lab based proof of concept tangibly signifies.

Returning the cash machine jack pot example, this exploit has been seen in the wild since the initial demonstration of the exploit, hence validating the underlying proposition behind the initial demonstration that there was a tangible cyber threat around ATM machines. What would have added to the impact of the initial demonstration of this exploit would have been an assessment of how many ATM machines within the wider ecosystem shared the same setup configuration of the test machine.

Including experimental parameters` as well as results is pretty standard within the wider scientific community. However, it is still relatively uncommon within cyber security and the effect of this can sometimes be acute. For example, the arrival of the Shodan search engine quickly highlighted the exposure of many industrial control system but it was left to other researchers to show that only a very small proportion of these systems were inadvertently exposed and unsecured.

Of course, proof of concept exploits are still valuable. However, these need to include the context in which they sit in order to have a tangible impact that amounts to more than hype.