Updated May 2021
DR solutions offer security tools that serve an integral part of an enterprise’s security posture and without them, an environment is left with many blind spots. That is why so many companies will invest in a reputable Endpoint Detection and Response (EDR) solution such as VMware Carbon Black, SentinelOne, or Crowdstrike. Unfortunately, these tools do not self-manage and are not plug and play solutions to protect your endpoints. Today, we’ll focus on Carbon Black specifically, a top EDR solution. These four tips will help you and your organization get the most value and enhanced security out of your Carbon Black instance.
1. Break down silos: Get to know your environment’s architecture and key personnel.
To be able to detect and proactively identify actions that are abnormal, you need to understand what normal looks like on an endpoint and how the endpoint should be communicating with other endpoints. One major challenge in threat hunting within an EDR security solution is being able to detect and respond to abnormal behaviors and actions on a host that indicate reconnaissance and attacks. To do this, you need to have a solid understanding of the architecture as a whole, as well as at a granular level to identify where potential vulnerabilities could be in the environment and how threat actors can exploit these vulnerabilities.
To gain an understanding of your network’s architecture, the analyst/threat hunter must have a detailed and clear idea of the technical aspects of the environment, such as the applications on hosts, network IoT devices in use, and systems that are currently in production.
In addition to knowing the technical aspects of a company’s environment, it is essential to develop relationships with key personnel inside and outside of the IT and cybersecurity departments to further understand what normal activity looks like as opposed to what anomalous activity looks like.
When threat hunting, it is likely you will come across activity that appears to be malicious; however, this activity could be due to the malpractice of devices, applications, or systems by the users in the network. Without working relationships with key personnel, threat hunters investigating or hardening the enterprise architecture using an EDR instance such as Carbon Black cannot make the needed security improvements in an EDR solution to exclude specific behaviors and binaries that may look and act malicious but are actually benign because they are specific to a group. For example, the development team could have a script created to automate a part of their job but when they run it, your VMware Carbon Black solution observes this file to be malicious. When you have a working relationship with the development team, you can get this information from them and approve the file name and process in a watchlist or approve the hash or file path in a policy so that you do not see alerts for that script in the future.
2. Do not settle for the repetitive false positive alerts – tune them.
Previously, we talked about the importance of understanding normalcy in your environment when threat hunting, to ensure you can discern normal from abnormal behavior. That is just as important when tuning your environment, whether you are modifying custom watchlists, enabling/disabling threat feeds in your EDR instance, or carefully crafting sensor groups and policies where your endpoints will reside.
With Carbon Black, you have the unique ability to exempt or block specific behaviors and binaries from triggering alerts using custom watchlists, threat feeds, and sensor policies. Understanding the normal activity occurring in your environment has a huge impact on the tuning capabilities an EDR analyst and administrator can implement to prevent false positive behavior from triggering alerts in Carbon Black.
It is imperative to encourage your security analysts to report false positive behavior/IOCs and to make educated decisions on what needs to be tuned out of future alerts. Then report that information to the necessary personnel to ensure those changes are enacted in your Carbon Black instance. Security is a team sport—you need all hands on deck to drive change and reduce the volume of false positive alerts.
At ReliaQuest, we use a 5-step tuning process to encourage team feedback, which has helped us to be able to efficiently and effectively update and implement new content daily. Implementing and modifying rule logic is incredibly important with EDR because these tend to be very noisy with false positive alerts, and a high volume of false positive alerts will reduce visibility into threats. The 5-step process is as follows:
- Event Identification: identify the actions or binary that is causing the alert to fire.
- Initial Analysis: analyze the action or binary and clearly state how and why the alert fired is a false positive.
- Incident Decision/Proposed tuning: develop the rule logic to exclude the behavior and/or binary from alerting the rule that you are editing.
- Tuning Complete: implement the change to the rule.
- Alert Change Documented: document this change as detailed as possible for future reference and tracking purposes.
With this 5-step process you should be able to effectively tune/edit rule logic around an alert that was generated due to a false positive event.
For more guidance on tuning false positives, Carbon Black offers free training for their customers (use your company email address) via VMware Carbon Black Technical Academy. They offer both self-paced and live classes and serve as valuable resources to help your analysts to tune out the false positives.
3. Utilize APIs for automation and tool health monitoring.
Carbon Black utilizes and integrates APIs for tool health and endpoint health, and has the ability to ban hashes, isolate hosts, and much more through their integrated API. Leverage these APIs to apply automation across the security lifecycle. APIs can be used to automate the patch lifecycle of the Carbon Black sensors that reside on each host to ensure that all endpoints have the most up to date software that Carbon Black provides. An important feature built-in to the Carbon Black sensor is that it will record the last check-in time of each sensor and report it back to your dashboard to ensure that there is always complete visibility on each host. The same can be done to automate a mass uninstall of sensors. For example, if a group of servers or a department of computers has reached end of life, the script can be configured to uninstall the sensors from the devices, contact the Carbon Black back-end, and remove that host from its respective group, thus returning the sensor license back to the account in order to be used on a different host.
One of the more popular ways to use APIs with Carbon Black is to continuously monitor the overall health of the Carbon Black instance(s) and the endpoints in the network. Using the API to create your own out-of-band monitoring is crucial to the success of your EDR implementation. Some examples of out-of-band health monitoring utilizing the API might include a baseline of the sensors that should report into the cloud. From there, you should be utilizing sensor APIs to consistently check for any deviations from that baseline, looking for any abnormalities in that data to identify potential misconfigurations or any other issue an endpoint might be experiencing causing it to not report in. If your team is not able to create their own out-of-band health monitoring, use what VMware Carbon Black has provided out-of-the-box for some of their products.
For instance, Carbon Black EDR (formerly CB Response) offers out-of-the-box sensor health monitoring, which will gather data metrics from the endpoint, ranging in severity from low to high, where a low health event could be a sign of error 879 (AgentStrongSSLUnavailable) which indicates an agent is unable to connect to server using Strong SSL, and a high severity health alert could be error 161 (AgentDatabaseIsCorrupt) indicating a corrupt database has been detected.
A common API your company should use to measure the overall health of the Carbon Black instance is monitoring the logging efficiency of the data that is coming in and out of your Carbon Black instance. Endpoints are constantly reporting back to the Carbon Black Predictive Security Cloud (PSE), a massive data and analytics based cloud platform that processes and analyzes customers’ unfiltered data for threats based on known malicious binaries and suspicious behavior from observed patterns, with unfiltered data. This data gets stored for a set amount of time and is monitored and made readily available to users who need to access that data. If the communication of data between one or all endpoints is broken, all visibility and data for the duration of the broken connection are lost. Endpoints are able to store a small, set amount of data in the agent cache on the host, but if the cache memory fills up on the hosts, then no more logging will be performed until the cache memory is pulled or the connection between endpoints and the Carbon Black backend is reconnected.
ReliaQuest GreyMatter, our SaaS security platform for delivering security confidence, utilizes our own out-of-band health monitoring to create baselines of your environment and track for any deviations from those established baselines that out-of-the-box health content would not catch, resulting in more endpoint uptime. GreyMatter uses APIs to monitor your endpoints’ health, which does not require anything additional to be deployed to your endpoints. We maximize your investment in EDR by applying automation across the security lifecycle through the use of APIs to gather and enrich data from your EDR, ban hashes, isolate hosts from the internal network, delete files on a host, and much more – providing analysts with all of the information they need, in one place, to detect, investigate, and respond all within the GreyMatter UI.
4. Dive into the Carbon Black User Exchange.
The Carbon Black User Exchange (CB UE) will provide useful information, whether you are an administrator troubleshooting an issue with your Carbon Black instance or an analyst investigating an alert that triggered on a host in your environment. In the user exchange, you will find a variety of knowledge pertaining to all areas of the Carbon Black toolset. Effectively utilizing the CB UE will be a big step in the right direction when striving to get the most out of your EDR tool.
The user exchange was built so that both Carbon Black employees and partners of Carbon Black who use or resell the tool, have a platform to gain new knowledge and share threat intelligence in one central location. With the user exchange, you have the opportunity to learn the best practices to secure your environment by learning from other security professionals who have walked in your shoes on the path to maximizing the potential of your EDR instance, all documented into organized knowledge base articles.
On the CB UE, there is information for every type of security professional. For EDR administrators, the CB UE offers step-by-step instructions for tasks like integrating your Carbon Black instance with an API to create a custom dashboard for your Carbon Black data. While the CB UE has a massive amount of information regarding implementing, fixing, and maintaining various types of configurations and different services Carbon Black uses, it also has updated information on enhancing your enterprise’s security posture. There is a wide range of behavioral threat feeds, custom watchlists, updated lists of known binaries for common malware families, and more to ensure you’re using your Carbon Black tool to its maximum potential.
The CB UE is not just a place for EDR administrators. Threat hunters will find the knowledge they need to conduct thorough and complete investigations by searching for updated search queries, investigations, and general information regarding a specific malware class or family. An example situation that a threat hunter may pivot into the User Exchange for is during an investigation of an alert where one internal host sent traffic to another internal host, and the traffic was analyzed by one of the IDS/IPS systems in the network as an Emotet infection attempting to infect another host. For this scenario, the alert did not generate in your Carbon Black instance; however, you can use the UE to gather custom search queries. In the User Exchange, there is a large knowledge base on what Emotet is, common binaries associated with it, common behavioral patterns, and custom search queries that were created to find indicators of compromise that are specific to an infection of Emotet. An example of a custom search query specific is:
ipaddr:10.141.153.49 OR ipaddr:10.131.91.65 cmdline:–* AND netconn_count:[2 TO *] AND modload:”c:\windows\syswow64\bcrypt.dll” AND digsig_result:”Untrusted Root”.
Lastly, the CB UE is a great resource when wanting to be proactive and read up on current malware trends so that you can begin to build your next custom watchlist tailored to what type of attack is most common among threat actors. Additionally, you can visit the User Exchange to stay up-to-date on recent updates to your sensors or threat feeds.
An account on the Carbon Black User Exchange can be created with your company email if they have an account with Carbon Black.
With these four tips, you should now have the knowledge to leverage different resources and processes to get more value out of a VMware Carbon Black EDR solution. To recap, start by:
- Building relationships with key personnel to understand normal and abnormal behaviors in your environment;
- Eradicating some of the redundancy in false positive alerts by creating and tuning custom watchlists in Carbon Black;
- Utilizing APIs to maximize time spent during investigations by automating tasks, and;
- Leveraging the CB User Exchange to troubleshoot issues in your CB environment or gain valuable information on search queries when conducting investigations on compromised hosts.
These tips paired with the integration of ReliaQuest GreyMatter will maximize the time spent on investigations and remediation of compromised hosts and will serve as a gateway to the information of your endpoints’ health.