Examine our research from the last year in the ReliaQuest 2024 Annual Cyber-Threat Report
Reduce Alert Noise and False Positives
Boost your team's productivity by cutting down alert noise and false positives.
Automate Security Operations
Boost efficiency, reduce burnout, and better manage risk through automation.
Dark Web Monitoring
Online protection tuned to the need of your business.
Maximize Existing Security Investments
Improve efficiencies from existing investments in security tools.
Beyond MDR
Move your security operations beyond the limitations of MDR.
Secure with Microsoft 365 E5
Boost the power of Microsoft 365 E5 security.
Secure Multi-Cloud Environments
Improve cloud security and overcome complexity across multi-cloud environments.
Secure Mergers and Acquisitions
Control cyber risk for business acquisitions and dispersed business units.
Operational Technology
Solve security operations challenges affecting critical operational technology (OT) infrastructure.
Force-Multiply Your Security Operations
Whether you’re just starting your security journey, need to up your game, or you’re not happy with an existing service, we can help you to achieve your security goals.
Detection Investigation Response
Modernize Detection, Investigation, Response with a Security Operations Platform.
Threat Hunting
Locate and eliminate lurking threats with ReliaQuest GreyMatter
Threat Intelligence
Find cyber threats that have evaded your defenses.
Model Index
Security metrics to manage and improve security operations.
Breach and Attack Simulation
GreyMatter Verify is ReliaQuest’s automated breach and attack simulation capability.
Digital Risk Protection
Continuous monitoring of open, deep, and dark web sources to identify threats.
Phishing Analyzer
GreyMatter Phishing Analyzer removes the abuse mailbox management by automating the DIR process for you.
Integration Partners
The GreyMatter cloud-native Open XDR platform integrates with a fast-growing number of market-leading technologies.
Unify and Optimize Your Security Operations
ReliaQuest GreyMatter is a security operations platform built on an open XDR architecture and designed to help security teams increase visibility, reduce complexity, and manage risk across their security tools, including on-premises, clouds, networks, and endpoints.
Blog
Company Blog
Case Studies
Brands of the world trust ReliaQuest to achieve their security goals.
Data Sheets
Learn how to achieve your security outcomes faster with ReliaQuest GreyMatter.
eBooks
The latest security trends and perspectives to help inform your security operations.
Industry Guides and Reports
The latest security research and industry reports.
Podcasts
Catch up on the latest cybersecurity podcasts, and mindset moments from our very own mental performance coaches.
Solution Briefs
A deep dive on how ReliaQuest GreyMatter addresses security challenges.
White Papers
The latest white papers focused on security operations strategy, technology & insight.
Videos
Current and future SOC trends presented by our security experts.
Events & Webinars
Explore all upcoming company events, in-person and on-demand webinars
ReliaQuest ResourceCenter
From prevention techniques to emerging security trends, our comprehensive library can arm you with the tools you need to improve your security posture.
Threat Research
Get the latest threat analysis from the ReliaQuest Threat Research Team. ReliaQuest ShadowTalk Weekly podcast featuring discussions on the latest cybersecurity news and threat research.
Shadow Talk
ReliaQuest's ShadowTalk is a weekly podcast featuring discussions on the latest cybersecurity news and threat research. ShadowTalk's hosts come from threat intelligence, threat hunting, security research, and leadership backgrounds providing practical perspectives on the week's top cybersecurity stories.
March 26, 2024
About ReliaQuest
We bring our best attitude, energy and effort to everything we do, every day, to make security possible.
Leadership
Security is a team sport.
No Show Dogs Podcast
Mental Performance Coaches Derin McMains and Dr. Nicole Detling interview world-class performers across multiple industries.
Make It Possible
Make It Possible reflects our focus on bringing cybersecurity awareness to our communities and enabling the next generation of cybersecurity professionals.
Careers
Join our world-class team.
Press and Media Coverage
ReliaQuest newsroom covering the latest press release and media coverage.
Become a Channel Partner
When you partner with ReliaQuest, you help deliver world-class cybersecurity solutions.
Contact Us
How can we help you?
A Mindset Like No Other in the Industry
Many companies tout their cultures; at ReliaQuest, we share a mindset. We focus on four values every day to make security possible: being accountable, helpful, adaptable, and focused. These values drive development of our platform, relationships with our customers and partners, and further the ReliaQuest promise of security confidence across our customers and our own teams.
More results...
Container orchestration tools have risen in popularity within the past few years, especially open source tools like Kubernetes, and have enabled organizations to more efficiently deploy and manage applications. However, these tools also come with their own security risks. All tools are susceptible to misconfigurations and insider abuse, in addition to more serious vulnerabilities that exploit the tools’ code itself. Since Kubernetes has control over the systems running critical business application development, the impact of a compromise is potentially massive.
In short, Kubernetes manages containers to ensure that resources scale with the application and that the overall state of the environment is healthy.
At the lowest level in the architecture is the Kubernetes node, which is the physical or virtual machine running all the services. The group of nodes help form the Kubernetes cluster. Each node runs at least one Kubernetes Pod, which houses the actual containers, such as a Docker container. Each node also runs an agent called kubelet, which manages the node and monitors the health of the containers in the Pods, automatically starting and stopping them to remediate issues.
Kubernetes architecture
The Kubernetes Control Plane manages the nodes and Pods across the cluster by sending commands through the kube-api-server. It also contains etcd, a key value store that contains all of the cluster data.
Just like with any application or tool introduced to the environment, it is important to not only patch and harden them to prevent attacks, but also enable logging to detect future attacks.
Kubernetes provides logging for many of its different components, but the most useful is from the API server. Logging can be configured by passing a configuration file to the API server as described in the Kubernetes Auditing documentation.
Logs are formatted in JSON. The following is a sample of the command “ls” executing on a pod:
Sample kube-api server log
While there have not been many attacks against Kubernetes observed in the wild, there is plenty of published research on various techniques to abuse and exploit the API. Most of these techniques rely on the assumption that the attacker already has access to the cluster API, usually by exploiting a public-facing service running in a Pod and getting a shell. Therefore, the risk to the Kubernetes cluster is highly dependent on the security of the applications it is hosting.
The following sections describe common attack vectors used to compromise a Kubernetes cluster and how to detect them.
In all versions except 1.5.1-1.5.x, Kubernetes will accept API calls from anonymous users by default. In a normal authenticated scenario, a user would need to authenticate to the API server with a token or password. However, if the user does not provide any token or password in the request, the API server will still accept it and assign the user a username of system:anonymous and a group of system:unauthenticated. This also applies to any API requests sent directly to the kubelet service.
It is strongly recommended to disable anonymous access by passing the flag –anonymous-auth=false to the API server and kubelet services. Another mitigation method is to enable Role-based Access Control (RBAC), which requires explicit authorization of anonymous requests.
This is the most severe type of misconfiguration, as it allows attackers, both internal and external, to send commands through the API directly to Kubernetes components and compromise the cluster.
Detection of this misconfiguration can be accomplished by checking the user object in the logs for the username of system:anonymous and the group of system:unauthenticated.
User object in anonymous authentication API server log
In Kubernetes, there are only two types of user accounts: service accounts and regular users. Normal users must be managed outside of Kubernetes, while service accounts are managed by Kubernetes itself. Kubernetes will automatically create service accounts as needed, but they can also be created manually. Service accounts are assigned not only to the main Kubernetes services, but also to every Pod as well.
For authentication, service accounts are assigned a “bearer” token that is passed to the API server when making requests. These tokens are stored securely in the etcd datastore, however for Pod service accounts, the tokens are also mounted to the Pod’s file system in /var/run/secrets/kubernetes.io/serviceaccount/token.
Once an attacker compromises a Pod, it is trivial for them to also compromise the Pod’s service account by reading the token file. The attacker can then authenticate to the API server as the service account and inherit the account’s permissions. Therefore, it is important to implement the principle of least privilege for Pod service accounts so they cannot be used by an attacker to further compromise the cluster.
While there is no universal detection signature for service account misuse, there are anomaly detection techniques that can be used to find potential abuse.
Since service accounts are used by automated services for a specific purpose, their activity should be defined and consistent. If an attacker were to compromise an account, it is expected that their abusive actions would be outside the normal scope for the service. By trending and observing changes in the logs from service accounts, it is possible to identify a compromise.
Every API log includes the user agent of the source that generated the request. For normal Kubernetes activity, the user agent starts with the name of the service, such as “kube-controller-manager” or “aws-k8s-agent”.
Sample user agent object from a Kubernetes API log
When a human user interacts with the API, they will be using a tool that will send the command to the API server. These can include native networking tools, such as curl, or the Kubernetes command line interface tool, kubectl. These tools use distinctive user agents which can be observed in the logs. In the original log sample above, the user agent field contains “kubectl”, which indicates the kubectl tool was used to send the command.
By alerting on deviations in observed user agents per service account, it is possible to detect when a Pod service account that historically uses the kubelet user agent is suddenly seen using curl instead.
The API logs also include the target resource and the action (verb) of the request. Since service accounts have a limited scope, they should need access to only a subset of resources and actions in order to function. If an attacker compromises a service account, it is likely they will execute commands outside the scope of the service account.
Trending the resources and verbs used by a service account can reveal when unexpected API calls are made and indicate a potential compromise. A Pod service account should probably not be requesting to delete, create, or execute commands on other Pods. If the service account is configured with least privilege, these requests may fail, however the compromise is still there.
Once an attacker compromises a Pod and service account, where do they go from there? While the answer is highly dependent on the environment and the attacker’s goal, it is likely they will try to pivot from their initial point of compromise and escalate privileges.
Kubernetes provides an easy method for remotely executing commands on Pods; there’s even a support article called “Get a Shell to a Running Container”. For an attacker, as long as their account has create rights to the pods/exec resource, they can execute commands. A common command is /bin/bash, which opens an interactive shell to the Pod. From there, an attacker can access the Pod’s file system, potentially installing a backdoor or looking for the Pod’s service account token, which may have more privileges than their current account.
An attacker can reuse this method, pivoting through the cluster and establishing persistence and escalating privileges along the way.
Pod command execution is logged by the API server and shows the API call targeting the resource pods with the subresource exec. Kubernetes treats the execution as “creating” the execution subresource, so the verb is create. The full command that was executed is logged in the requestURI object in the log. Note that the command is URL encoded.
Excerpts of Pod execution log
To open a remote shell, the kubectl program can be used:
To detect a user opening an interactive shell on a Pod, the following logic can be used:
All commands sent using exec will be logged by the API server. However, once the interactive shell is opened, any further commands sent inside the shell will not be logged by the API server.
If interactive shells are commonly used by employees, additional criteria can be applied to make the detection higher confidence, such as alerting on shells from service accounts or when the shell sources from other Pod IP addresses.
Pods are constantly being created and deleted as the cluster operates and scales, so they are not a stable persistence method for attackers. The underlying node that hosts the Pods, however, is a much more reliable option.
When creating a Pod, there is an option to mount a file system volume to the new Pod. Kubernetes supports a wide variety of volumes, such as AWS EBS, Microsoft Azure Data Disks, and more. One type is hostPath, which mounts a part of the node’s file system to the Pod. If an attacker has the create rights for the pod resource, they can create a new Pod that is configured to mount the node’s root directory.
Sample YAML Pod configuration file to mount the node’s root directory
The attacker can then install a backdoor on the node or scrape the file system for authentication tokens or SSH keys. As even the Kubernetes documentation notes, the hostPath option “offers a powerful escape hatch” from the containerized environment.
While Pod creations are logged by the API server, the details of the Pod’s configuration are not. If an attacker mounts the node’s file system, there is no indication in the API logs.
Any detection must instead rely on the circumstances surrounding the Pod’s creation.
If the attacker is running the command from a compromised Pod, the source IP in the log would show as the Pod’s IP. A Pod creating another Pod would be highly suspicious, as most Pod creations should come from a Kubernetes Admission Controller.
The API call could come from a user account that does not normally create Pods, such as a compromised service account, in which case anomaly detection techniques could also apply.
Secrets in Kubernetes are secure objects designed to store sensitive information, such as authentication tokens or passwords. Instead of storing credentials in files scattered across the cluster where they might be accidentally exposed, all secrets are stored in the etcd datastore and can only be accessed over secure communications. Secrets are created automatically by Kubernetes for service accounts but can also be created manually by users. When a Pod or application needs to access a secret, the secret can be mounted as a volume, such as for the Pod service account tokens, or accessed through an API call.
By default, secrets are not encrypted at rest. They are base64 encoded, so it is trivial to decode a secret once it is accessed. There is even a Kubernetes support article that explains how to decode the secrets.
Thankfully, access to secrets is restricted by the account’s permissions. First, the account must have the get or list permissions for the secrets resource to even view the secrets.
To read an individual secret, a user can run the kubectl command:
Instead of get secret, a user can run the get secrets command to read all the secrets in the namespace.Second, the account can only access secrets that are in the current namespace. Namespaces help organize resources into separate groups, similar to Active Directory domains.
However, if the account has enough permissions, the –all-namespaces flag can be used to read all secrets in all namespaces.
Attackers can abuse these permissions to access the secret credentials in the cluster, which will only provide them more opportunities for privilege escalation and persistence.
Secrets accessed through the API are logged in the API server logs. The verb is get if a single secret is accessed and list if there are multiple with the get secrets command. The log describes which secret was accessed and who accessed it.
Excerpt from secret access log
While there are no clear indicators in the secrets log that an attacker is abusing an account, anomaly detection can be used here again to determine if this access is normal for the account. Accessing secrets is a common occurrence in the cluster and required for its operation. Just like with service account activity, most of these accesses should be routine and expected.
By trending which secrets are normally accessed per account, it is possible to identify unexpected secret access that may be an attacker abusing the account to view out-of-scope secrets.
While detection may be difficult, there are several steps that can be taken to harden the cluster and prevent attackers from easily gaining persistence and pivoting.
The key to enabling all these attack vectors to succeed comes down to one thing: privileges. There have been very few exploitable vulnerabilities found in the Kubernetes’ code itself, so most attackers will have to rely on compromised credentials. And all the techniques above use built-in Kubernetes features, not bugs, so it is unlikely that these abuse vectors will be closed anytime soon. This puts the burden on Kubernetes administrators to be the first line of defense.
If or when a container gets compromised, it will be the privileges of the Pod accounts and others that determine how easily an attacker can take over the cluster and might be the difference between a simple Pod reimage and a cluster-wide compromise.
Detect these common threats faster with ReliaQuest GreyMatter’s custom content, continuously tuned to integrate into your environment. As threats continue to evolve and even specialize to target specific industries, vulnerabilities, and individuals, your threat detection cannot rely on generic threat content. Increase alert fidelity and environment visibility across SIEM, EDR, cloud services, and third-party apps with personalized content tuned to your environment.