A Technical Analysis of the Capital One Cloud Misconfiguration Breach

This article was originally published on Fugue's blog here. 

By Josh Stella, Co-founder & Chief Technology Officer, Fugue

This is a technical exploration of how the Capital One breach might have occurred, based on the evidence we have from the criminal complaint. I want to start by saying that I deeply respect the Capital One cloud team, and have friends on it. They’ve been leaders in cloud computing, and what happened to them could have happened to nearly anyone. I’m also not criticizing Amazon Web Services (AWS)⁠—my previous employer. They offer secure services and I have nothing but respect for them. The purpose of this post is to explore a combination firewall/IAM/S3 attack to illustrate some of the dangers of cloud misconfigurations that every organization on cloud should heed.

In order to write this, I analyzed the technical details of the FBI complaint, and then formed a hypothesis of how the attack might have taken place. I then simulated the attack in my development account, so that I could provide specific details in this post.

The facts as we know them

We have a smattering of information on the attack from the FBI complaint, and from social media posts by the alleged attacker. It appears that the attacker used Tor and IPredator in combination to obfuscate her identity, and through those services, discovered a misconfigured firewall in the Capital One AWS environment. What isn’t clear is whether this was a targeted attack on Capital One, or an opportunistic attack upon finding the misconfiguration.

Many articles have been written on the possible motivations and biography of the attacker, so I’ll leave those aside to focus on the technical specifics. There were four different elements to the attack that we know about:

  • Misconfigured firewall
  • Gaining access to an EC2 instance
  • Getting IAM role access to S3
  • S3 bucket discovery and duplication.

Each of the above are explored in some detail below.

How the hack could have happened

We have some details, but really not that many, and therefore I employed a fair amount of speculation and interpretation of the facts that we do have. I’ll clearly state when I’m speculating as we walk through the steps of the attack below. The diagram below shows my sandbox environment where I simulated some aspects of the Capital One breach.

The environment contains:

  1. a basic VPC network
  2. a Security Group that allows HTTP(S) and SSH
  3. a private S3 Bucket
  4. two IAM Roles – one with S3 access, and one without
  5. An EC2 Instance with a public IP address, with the IAM Role that doesn’t have S3 access

My goal was to go from shell access of the EC2 instance, to the ability to access and copy the S3 data in the private buckets.

Before we go into detail on the various steps, it’s worth pointing out that the FBI gained access via a tip to a GitHub hosted file that contained the IP address of a server, along with three commands:

“Capital One determined that the April 21 File contained code for three commands, as well as a list of more than 700 folders or buckets of data.” We’ll go into some detail on each of these commands, and what they might have been and the possible strategy used by the attacker.

Step one: misconfigured firewall

According to the FBI Complaint, “A firewall misconfiguration permitted commands to reach and be executed by that server, which enabled access to folders or buckets… (III.A.10)”

This suggests that the firewall was external to the server vs. a local firewall, though it isn’t explicit. While there are many firewall virtual appliances available on AWS, generally this would be a Security Group. It seems that a dangerous port was left open on whichever firewall type was in use, and this might have been the initial opening for the attack. It’s possible that an SSH port was opened for a maintenance window, or perhaps the server in question was left over from development efforts and no longer in use. Another possibility is that the server was running an application such as MongoDB or ElasticSearch that require an open port to function, but which should never have been exposed to the Internet via the firewall. Whatever the details, the attacker found a path into the CapitalOne cloud computing infrastructure.

Step two: gaining access to an EC2 instance

Based on the FBI complaint’s description of the compromised entity as a “server” and that the attacker was able to extract IAM credentials from it, it appears that an EC2 Instance was then breached. This might have been an application or OS vulnerability⁠—we simply don’t know. For the purposes of my simulation of the attack, I assumed the attacker gained shell access, but not root access to the instance, as all the remaining steps only require basic shell access to successfully complete.

Step three: getting IAM role access to S3

…”CapitalOne determined that the first command, when executed, obtained the security credentials for an account known as ***-WAF-Role that, in turn, enabled access to certain of Capital One’s folders at Cloud Computing Company. (III.A.11)”

Much of the “action” in this breach was via IAM role access to private S3 buckets, seemingly via AWS CLI commands from the compromised server. Much has been made of the name of the role in the press to date, but there is no solid evidence that this server was itself a WAF. It is common to “borrow” IAM roles in dynamic AWS environments (it’s not a great practice, but it is a common one), and also as will be shown below, policy associations can be changed and often have “Role” in the name, as shown in the example below. Perhaps this server was just something left around without tags to bring it up on management tooling dashboards. I’ve rarely seen a sizable AWS account without orphaned resources laying around one place or another.

If the server was itself a WAF, and intentionally had read/write access to buckets and objects in those buckets containing PII data, that was a naive defense strategy for the obvious reasons⁠—notably that a single firewall misconfiguration would be able to breach all architectural defenses to the sensitive data. This isn’t the only explanation for the breach however, and I suspect that there was more to it, using IAM and EC2 features designed for flexibility, but potentially misused to grant additional privileges.

Given that they are describing the “first command”, I think it’s reasonable to assume that they are referring to an AWS CLI script. If the EC2 instance already had access to S3, the attacker would have only needed to execute something like:

> curl http://169.254.169.254/latest/meta-data/iam/security-credentials/DemonstrationEC2Role

This command retrieves the current IAM temporary credentials from the role from the EC2 instance metadata. The output looks like this:

Where the access key and secret access key are replaced with *’s and the token corrupted to protect the innocent.

This is all that would be needed to access the S3 buckets and their content.

But there is another possibility as to what that “first command” did, and everyone who uses AWS needs to be aware of it. If the compromised server did not have access to the private S3 buckets, but did have permissions to list and attach IAM policies, the attacker could have used these capabilities to essentially go shopping for a set of credentials that would provide this access. Since IAM permissions often have no relationship to IP-level network access controls, and the AWS services connect via IAM, these roles and policies form a sort of alternate network that needs to be secured just as a traditional network does. IAM becomes a primary means of “lateral movement” within the cloud environment.

The command below, for example, replaces one set of IAM permissions with another:

> aws ec2 replace-iam-instance-profile-association –association-id iip-assoc-00facaf2870f3bcc9 –iam-instance-profile Name=DemonstrationEC2Role

which returns the following, showing that I’ve successfully replaced the existing IAM permissions for the ones in the DemonstrationEC2Role policy:

Other useful commands for such a fishing expedition include:

> aws iam list-roles

> aws iam list-policies

> aws iam list-instance-profiles

These do what it says on the tin⁠—enumerate the catalogue of available IAM resources an attacker might want to use once they have shell on an EC2 instance.

In this example, I replaced a limited IAM profile with one with additional permissions to S3. We cannot discount that the attacker executed a similar approach in the Capital One breach. Whether or not this was the case, it’s a critical capability to keep in mind when you configure IAM roles for your EC2 instances⁠—especially public facing ones—as IAM permissions can effectively “jump” to private resources in your environment.

In either scenario, the attacker gained access to the credentials needed to retrieve information from S3 and to duplicate that information.

Step four: S3 bucket discovery and duplication

“Capital One determined that the third command (the “Sync Command”), when executed, used the ***-WAF-Role to extract or copy data from those folders or buckets in Capital One’s storage space for which the ***-WAF-Role account had the requisite permissions. (III.A.11)”

This suggests the use of the AWS CLI command `aws s3 sync`, so I think it’s more evidence that the attacker gained shell access to the EC2 instance and used the AWS CLI to perform these commands. What isn’t clear is how the attacker knew which S3 buckets to perform the sync on, but if the IAM role used had adequate permissions, it’s pretty simple to list all the buckets accessible via that role. This highlights the danger of a single IAM role having this broad of S3 permissions, as even at this point, without the ability to discover the target buckets, the attacker would not have likely been able to capture much if any data.

Recommendations

  1. Constantly monitor for overly permissive Security Groups or any other access mechanism from 0.0.0.0/0. Checking on provisioning is necessary, but not nearly good enough. Cloud infrastructure is created and modified via API, which means that it is often drifting over time as different teams and people interact with it.
  2. Apply the principle of least privilege and ruthlessly limit IAM roles to what is absolutely necessary for the business function of the resource using that role. In the case of S3, you might consider different public end points for read and write operations, with different IAM roles that cannot perform the other function. Eliminate all production use cases where S3 bucket listing is enabled, in favor of shared secrets or other mechanisms.
  3. Don’t allow EC2 instances to have IAM roles that allow attaching or replacing role policies in any production environments.
  4. Ruthlessly clean up unused cloud resources (especially servers and S3 buckets) left over from prior development or production debugging efforts.
  5. Include cloud infrastructure misconfiguration in your pen testing efforts. Use outside pen testers and make sure they are knowledgeable about how to find and exploit cloud misconfigurations.

Read more industry insights from the team at Fugue here.

“Collection #1” Data Breach

By Paul Sullivan, Software Engineer, Bitglass

hacker in hoodie sitting in front of a laptop

News of the 773 million email data breach that Troy Hunt announced for Have I Been Pwned certainly got a lot of coverage a few months ago. Now that the dust has settled, let’s cut through some of the hype and see what this really means for enterprise security.

First, let’s clear some things up – the data itself is actually several years old, but it looks like the seller of the data has more recent material, as well. Also, this data did not come from a specific company, but was a composite of various sources that cybercriminals stitched together. It is unclear what these sources are, but some of them are likely to be breaches that have been widely known for some time. This is demonstrated by the fact that Have I Been Pwned has already seen about 82 percent of the compromised emails in previous breaches.

However, the above could also mean that individual emails have been breached multiple times across different services. Unfortunately, people commonly reuse passwords, which means if a cybercriminal gains access to one password or account, they can potentially gain access to various accounts on different websites.

This is important because this kind of data is used in credential stuffing attacks to automate trying to log in to various services with stolen data. Since passwords are often reused, criminals run all this data against other accounts (Spotify, Netflix, Amazon or other paid subscription accounts), hijack them, and resell them.

Unfortunately, this data is out there now and new breaches are happening all the time. Luckily there are ways both individuals and companies can mitigate the damage. For individuals, using a password manager to create strong unique passwords is definitely a good idea. For companies, password expiration is now arguably a bad idea, but IT teams can monitor services like HIBP and let employees know when to change passwords after a breach. Companies can also cut down on the number of passwords running around by using single sign on (SSO) for their cloud services, and by enabling multi-factor authentication to make it harder for credential stuffing attacks to work. A cloud access security broker (CASB) can also alert IT teams when a strange login occurs so they can take action to protect their data.  

For more information, download the Top CASB Use Cases.

The Many Benefits of a Cloud Access Security Broker

By Will Houcheime, Product Marketing Manager, Bitglass

server hallway leading to blue sky with clouds

Today, organizations are finding that storing and processing their data in the cloud brings countless benefits. However, without the right tools (such as cloud access security brokers (CASBs), they can put themselves at risk. Organizations’ IT departments understand how vital cybersecurity is, but must be equipped with modern tools in order to secure their data. CASBs protect against a wide range of security concerns that enterprises face when migrating to the cloud. Consequently, they have quickly increased in popularity and have become a one-stop-shop for countless enterprise security needs.   

BYOD, SaaS or IaaS

Depending on the industry in which an organization operates, it may need to focus on security for managed devices, or perhaps it might need more of a bring your own device (BYOD) solution. While major SaaS applications improve organizational productivity and flexibility, they can serve as entry points for malicious threats such as malware or be used to share sensitive data with unauthorized parties. In infrastructure-as-a-service platforms, even a simple misconfiguration can cause data leakage and jeopardize an organization’s wellbeing. Without a solution designed to address these modern security concerns, organizations can fall victim to these and other threats.

In recent years, cloud access security brokers have been used to prevent these types of unfortunate scenarios from happening to organizations. Whether it’s securing data on personal devices, limiting external sharing, stopping cloud malware, or other security needs, CASBs have been stepping in and protecting data whether it is in transit or at rest. In our latest white paper, Top CASB Use Cases, we go into detail about how organizations have used cloud access security brokers to embrace both the cloud and BYOD without compromising on security.

For information about how CASBs help secure data, download the Top CASB Use Cases.

Healthcare Breaches and the Rise of Hacking and IT Incidents

By Jacob Serpa, Product Marketing Manager, Bitglass

Healthcare breach report 2019

In the course of their day-to-day operations, healthcare organizations handle an extensive amount of highly sensitive data. From Social Security numbers to medical record numbers and beyond, it is imperative that these personal details are properly secured. 

Each year, Bitglass conducts an analysis and uncovers how well healthcare organizations are protecting their data. In 2019’s report, we detail the state of security in healthcare as well as shed light on recent breach trends in the vertical. Read on to learn more.

Bitglass’ 2019 Healthcare Breach Report analyzes data stored in the Department of Health and Human Services’ “Wall of Shame,” a database wherein details about healthcare breaches are stored. By scrutinizing this data set, Bitglass uncovered information related to the size of healthcare breaches, the causes of healthcare breaches, the states in which these breaches occur, and much more, over the last few years. A snapshot of some of this data is provided below.

The rise of hacking and IT incidents

Over the last few years, the threat landscape has been shifting in healthcare. It used to be that lost and stolen devices were the leading contributor to exposed data. However, each year since 2014, the number of breaches caused by lost and stolen devices has decreased. At the same time, hacking and IT incidents have enabled more and more breaches each year – in 2018, they were the leading cause of breaches in healthcare. 

The decreasing numbers of healthcare breaches

Despite the above, 2018 saw the number of healthcare breaches reach its lowest point in the last few years. Obviously, this is good news. While healthcare firms need to do something to address the growing number of hacking and IT incidents that are exposing their data, the fact that the overall breach number is down still bodes well for the industry’s progress in securing sensitive data. 

To learn more about the above findings as well as other interesting facts and figures, download the full 2019 Healthcare Breach Report.