Authors: Simon Hall, Isidoros Monogioudis

 

Here at Digital Shadows (now ReliaQuest) we perform regular purple team exercises to continually challenge and enhance our security posture. These exercises are a great chance to test out new tools and techniques from both an offensive and defensive perspective, allowing us to fine-tune processes and defensive solutions.

Purple teaming refers to cyber exercises where red and blue teams sit together and evaluate the effectiveness of detection controls. Red teams simulate the attacker/adversary’s activities while the blue team represents the defensive actions and controls that are expected to detect and/or prevent a cyber attack.

As always, our team likes to shake things up and try new approaches. For a recent exercise we used the “Vectr” platform from Security Risk Advisors. Vectr is a platform designed for tracking activities during a purple team exercise, allowing you to log both red and blue team activities and the success of each action.

In this blog we will demonstrate how we used Vectr in our assessments. We will also touch on how we use it alongside the MITRE ATT&CK framework to categorize attacker behavior, and Cobalt Strike to execute the tests.

 

Organizing Exercises into Assessment Groups

The foundation for tracking a purple team exercise with Vectr is the Assessment Group. Assessment Groups are parent containers that can house many associated campaigns. This structure is useful for organizing and referencing reoccurring assessments; having a single inventory list of quarterly assessments, for example, makes accessing historical information much simpler (Figure 1). Typically, you will use one Assessment Group per purple team exercise.

 

Figure 1 – Assessment Group overview

 

Configuring Custom Campaigns

A campaign in the world of Vectr is an organizational unit for grouping together the actions that will be carried out during the assessment. These are referred to as test cases. The campaign is fully configurable and can include actions from different phases and categories. In our case, for example, we split the exercise into different campaigns based on the tested systems’ platform such as MacOS, Windows, Linux (Figure 2).

 

Figure 2 – Assessment Group Campaign overview

 

There are multiple predefined campaigns in Vectr that are helpful for getting you up and running and can serve as a useful guide for creating custom campaigns targeted to your exact assessment requirements.

There is also a pre-configured ATT&CK group assessment based on the popular MITRE framework (Figure 3), where every ATTACK phase (tactic) is included along with the corresponding pre-defined test cases (techniques). This is a good starting point for any organization that wants to map its detection tests with the ATT&CK framework. Several other configurations are available to modify the entire assessment or each campaign according to your specific requirements.

 

Figure 3 – Custom Campaign creation

 

Test Cases to Match Your Exercise Scenario

The meat of Vectr is the “Test case” model. Test cases represent the campaign details and include information about the phase, the method, and the variant of an attack as well as the expected detection layers from blue team’s side.

A test case is prepared and created according to the exercise’s scenario, and each one should include a specific action – or group of actions – that mirror to a selected attack pattern. The blue team’s section needs to be filled with information about the specific detection/prevention controls that are involved in the attack pattern.

During execution, every test case’s status is changed to “in progress” and the Attack Start Time is captured. What is also captured is the toolset, commands, and any other attack activity in the corresponding fields as shown in Figure 4 below.

The blue side captures the outcome of the attack mainly as Blocked, Detected or NotDetected. It will also identify the detection controls (if any) that detected the attack along with any associated alerts triggered for this incident. Finally, there is the option to include more details for prevention and detection such as which rule prevented the incident.

Evidence files can be attached from both sides to support the test case execution outcome.

Figure 4 shows a Vectr user testing initial access to a system. One of the test cases might be sending a phishing email containing an attached Microsoft Office document with a malicious macro.

 

Figure 4 – Test case edit

 

Analyzing Results and Creating Reports

The organization of information and assessment details, along with the reporting function within Vectr make it an extremely useful tool not only for keeping track of red and blue actions during the exercise, but also for easily collating all the details gathered. The platform makes assessment wash-ups much simpler: no more trying to gather everyone’s notes from the assessment, which can be collected during the exercise and entered as you go by an assessment coordinator.

 

Campaign Timelines

These not only make great summary pieces, but can at a glance give you a better understanding of just how long it took for your detection layers to block, detect or alert on a given threat.

The execution timeline provides details for each test case and captures every change/progress until the action is stopped/completed.

As shown in Figure 5, we used Cobalt Strike’s flexible payload generation to drive our assessment work.

 

Figure 5 – Campaign timeline example

 

The Deliverables

When every test case for each of the campaigns is complete, the assessment is then considered completed too. An extensive range of reporting templates are then available to the user, allowing you to export every detail in the outcome of the assessment (Figure 6).

The level of detail provided for generating reports offers the flexibility to create reports for different target audiences, selecting the attributes that fit best for each one.

 

Figure 6 – Report Editor

 

As we can see in the following sample report (Figure 7), each campaign is analyzed in use cases where the reviewer can easily check some major elements such as outcome and detection tools, as well as other attack and defense details.

The report provides, in high-level detail, the information provided during the preparation and execution phase, so it is very important to invest enough time feeding the tool with proper data.

 

Figure 7  – Sample Report

 

If this is done well, the result will satisfy everyone, whether blue or red. Vectr can be a reusable tool used for tracking purple teaming exercises outcomes, and also testing the effectiveness of detection and prevention controls in your environment.

 

Vectr: The Verdict

Setting up Vectr is straightforward and it can be up and running in a short amount of time. The platform does have a bit of an initial learning curve, but once you get to grips with how campaigns and test cases fit together, everything starts to fall into place.

Fully-customizable test cases and campaigns make it possible to prepare your assessments in advance, saving time and confusion on D-day. You can also simulate adversary threats through default test cases – this is particularly useful if your team has limited experience or simply needs to setup a quick assessment with MITRE ATT&CK mapping.

Tracking red and blue activities during an assessment can be challenging at times, particularly with larger teams who have notes stored across multiple systems and individual styles of documenting tasks. Sometimes, red teaming can become a bit of a free for all. While these issues become easier to manage with experience, a platform such as Vectr certainly makes the process a lot smoother. For any aspiring purple teams out there, we would certainly recommend you check out Vectr.

 

To stay up to date with the latest digital risk and threat intelligence news, subscribe to our threat intelligence emails here.