Heartbleed Hype Left Enterprises Uninformed

By George Muldoon, Regional Director, Venafi

In early April, the vulnerability known simply as “Heartbleed” became the latest rage. During the first week after discovery, the mainstream media aggressively reported on Heartbleed, stirring up a tornado of fear, uncertainty, and doubt amongst all Internet users. Never thought I’d see “Fox and Friends” talking about OpenSSL, two-factor authentication, and digital certificates, but it happened daily only 7 short weeks ago.

This “Heartbleed Tornado” subsequently led to enterprise security professionals receiving email inbox loads of offers claiming to help you remediate. For many, especially those in the executive suites and board rooms, it was the first time they understood the true power and importance of private encryption keys and digital certificates, as well as the imperative need to protect them. Finally, I thought, the world is waking up and understanding the need to secure and protect its most valuable assets, which provide the backbone of a trustworthy Internet—encryption keys and digital certificates.

Unfortunately, as loud as the Heartbleed Tornado roared, the lions’ share of the remediation advice related to Heartbleed was simply the following:

  1. Check and see if websites you use are vulnerable (and have been patched), and
  2. Emphasize the importance of changing your passwords.

Patching OpenSSL and changing user-credential passwords are two of the steps to remediation. But the elephant in the room, the exposure of private encryption keys and certificates (and thus the need to revoke and reissue them ALL), was only consistently reported on by those media outlets and bloggers in the security space itself.

Any hot media story has a shelf life, and there’s only so many Heartbleed stories that will continue to draw readers in. So once the clicks died down, the mainstream all but forgot it. And those mainstream stories that remain, only touch upon the surface of the vulnerability, such as NBC’s cosmetic piece on “How Major Websites Rank on Password Security.”

But the important thing to realize is this: The threat against a trustworthy digital universe did not begin with Heartbleed. And it certainly does not end with it either. Heartbleed was simply the latest in a growing mountain of threats that continue to evolve against encryption keys and digital certificates, and thus trust online.

For more information on Heartbleed and how to remediate effectively, check out the Venafi Heartbleed Solution page.

Heartbleed Changed the Security Landscape, but Few Organizations Realize It

With the media no longer focusing on the Heartbleed vulnerability, most people think that organizations have adequately addressed the problem, and the threat has passed. Because most people don’t understand the full impact of Heartbleed, however, they don’t realize that the fallout from this one vulnerability is likely to continue, not just for weeks but possibly for months to come.

The problem is that most organizations responded to the Heartbleed vulnerability tactically, just as they would respond to any known vulnerability: they identified the systems using OpenSSL and patched them. These organizations did not understand that the Heartbleed vulnerability undermines the very trust on which every business and government relies to secure its data. It gives hackers privileges that they can use to compromise other, seemingly secure systems. Because most organizations didn’t understand the “big picture,” they failed to fully remediate the problem. They did not revoke and replace all of their digital certificates, leaving their systems vulnerable to ongoing trust-based attacks.

Unfortunately, I don’t believe the Heartbleed vulnerability is an isolated incident. Malicious attackers recognize the value of targeting digital assets, which is why trust-based attacks have significantly increased over the last several years. These malicious actors will continue to look for and target trust-based vulnerabilities, so organizations should not be wondering if another Heartbleed will occur; they should be preparing now to respond more quickly when the next event occurs.

Organizations that took a tactical approach to addressing the Heartbleed vulnerability (simply patching the systems they thought were affected) will be ill-prepared for the next trust-based crisis. Because these organizations don’t yet understand the danger of trust-based attacks, they will continue to focus on what they perceive is the greatest danger on the cyber-security landscape—Advanced Persistent Threats (APTs)—and rely solely on traditional security tools such as packet-inspection tools and Intrusion Detection System/Intrusion Protection System (IDS/IPS) solutions to protect their environment. All of which are inadequate against trust-based attacks. They will not realize that trust-based attacks are all too often the key component of APTs. Therefore, any security solution that does not detect and mitigate trust-based attacks is inadequate. Despite what some security vendors claim, detecting and remediating trust-based vulnerabilities such as Heartbleed requires more than just monitoring traffic and patching systems. Organizations must have a solution that inventories all certificates and digital keys in use on the network, detects anomalous usage, and helps administrators swiftly revoke and replace all certificates.

This is, of course, exactly what Venafi does best. In talking to our customers using Venafi TrustAuthority™ and TrustForce™, we found that these customers were able to respond quickly to Heartbleed, identifying susceptible systems, revoking and replacing all their certificates, as recommended by Gartner. When their Chief Executive Officers (CEOs), Chief Information Officers (CIOs), and even the Board of Directors asked, “What are you doing about this problem?” the Chief Information Security Officers (CISOs) at these organizations were able to say with complete confidence, “We have successfully remediated Heartbleed with Venafi. We have identified and patched all systems impacted, replaced private keys with new ones and issued new certificates.”

As more events such as the Heartbleed vulnerability occur, trust is going to become a top-of-mind issue for all CISOs. Protecting trust will quickly evolve from a nice-to-have to a must-have. Organizations are going to have to know where all the keys and certificates are in their environment, and they are going to have to have the agility to react to trust-based threats almost immediately. Organizations ignorant of the threat posed by trust-based attacks—organizations without a solution to combat these attacks—are going to struggle again and again.

However, CISOs who understand what hackers are looking for when they exploit a vulnerability like Heartbleed—those ever-so-critical keys and certificates—can rise above the struggle. When I meet with customers to discuss the challenges of trust-based attacks, I’ve often seen them experience a kind of “light bulb” moment, as they realize that they have to go beyond removing malware and beyond patching vulnerabilities. They have to restore the trust that the hackers compromised. I joined Venafi because I love being part of these “light bulb” moments. And I love being able to reply, when customers ask how they can possibly revoke and renew thousands or even tens of thousands of keys and certificates, that Venafi has a solution.


Tammy Moskites, Venafi CISO

Heartbleed Remediation: Replace ALL Keys and Certificates

Response is not complete until trust is re-established

By now most organizations have responded to the Heartbleed vulnerability by patching vulnerable systems. Good. The next step must be to replace ALL keys and certificates. Successful private key extraction exploitation in just hours shows keys and certificates must be assumed comprised. The urgency to replace keys and certificates is even more important as details emerge about exploits being used in the wild for months and possibly years. As a result, experts all agree — from Bruce Schneier, to Gartner’s Erik Heidt, to CloudFlare — the message is: replace keys and certificates.

Underestimating our adversaries and taking no action is not an option. Gartner’s advice to enterprise IT security teams is very clear:

“Because this attack enables the recovery of the private key itself, certificate rotation alone will not protect you! New private keys must be generated.

Following successful private key exploitation in its Heartbleed Challenge, CloudFlare turned from skeptical to genuinely concerned: “Our recommendation based on this finding is that everyone reissue and revoke their private keys.”

Unfortunately, I’ve observed some response teams either 1) assuming that patching is enough 2) patching and reissuing certificates without generating new keys. Unless private keys are replaced, attackers can still spoof websites and decrypt encrypted communication.

CISOs and CIOs should not report to their CEOs, board of directors, and customers that they are safe until they’ve replaced all keys and certificates. Doing so is ill advised as we learn more about new exploits and the likelihood that Heartbleed exploits occurred in 2013 and before.

Furthermore, enterprises must assume, just as they are with userid and passwords, that ALL keys and certificates are compromised, not just those that secured vulnerable Heartbleed systems. Kill Chain Analysis helps us understand that attackers will look to expand their attacks using similar methods and targets as their first intrusion. Further infiltration of networks means that SSL keys and certificate and SSH keys, even though not running vulnerable OpenSSL, should be assumed targets and compromised.

Respond & Remediate Now Before It’s Too Late

  • Know where all keys and certificates are located
  • Generate new keys and certificates
  • Replace new keys and certificates, revoke old ones
  • Validate remediation to ensure new key and certificates are in place

To help organizations respond, Venafi has prepared more guidance on remediation steps. Venafi customers have already remediated keys and certificates in hours. And in the last few days, we’ve been helping many new customers start to respond quickly. Please contact Venafi’s Incident Response and Remediation team to help your enterprise.

Related resources:


By Kevin Bocek
VP, Security Strategy & Threat Intelligence

The World is Failing to Remediate the Heartbleed Vulnerability

Time is running out to change keys and certificates or else…

The world appears to be failing to respond to the Heartbleed vulnerability. In fact well under 16% of vulnerable keys and certificates have been replaced. Experts Bruce Schneier, Gartner, Akamai, and CloudFlare all agree about what enterprises must assume and do: Enterprises must assume keys and certificates are compromised. As a result, organizations MUST change ALL keys and certificates to complete remediation: rekey, reissue, and revoke.

Remediation of ALL keys and certificates is important since the vulnerability may have exposed keys and certificates as a result of continued expansion of attacks beyond the initial infiltration and vulnerability.

Respected security researcher Dan Kaminsky explained the reason behind replacing ALL keys and certificates:

“Find anything moving SSL, particularly your SSL VPNs, prioritizing on open inbound, any TCP port. Cycle your certs if you have them, you’re going to lose them, you may have already, we don’t know.”

The EFF has confirmed Heartbleed exploits in November 2013 and other reports indicate possible exploits go back two years. Therefore, we must assume our adversaries were compromising keys and certificates for long time before Monday, April 7th when the Heartbleed bug was first publically announced.

Until key and certificate replacement remediation occurs, enterprises are vulnerable to spoofing and decryption. As of April 15th, over seven days since the first calls to change keys and certificates began, Netcraft reports that only 16% of certificates known to be used with publicly vulnerable webserver were only revoked.


Less than 16% remediation

And remediation on these systems is likely even lower than 16% of publicly vulnerable systems because new keys were not created in many cases. Some incident response teams are only reissuing certificates without changing keys. Gartner’s Erik Heidt accurately describes the situation in his Heartbleed remediation blog:

“Many organizations perform ‘lazy’ certificate rotations, and do not create new keys! This is a bad practice.“

Gartner concluded, “because this attack [Heartbleed] enables the recovery of the private key itself, certificate rotation alone will not protect you! New private keys must be generated.”

This is the same guidance that once-skeptical, now-converted CloudFlare gave after researchers proved SSL/TLS keys could be stolen using the Heartbleed vulnerability:

“Our recommendation based on this finding is that everyone reissue and revoke their private keys”

Furthermore, there are hundreds of applications from IBM, Juniper, Cisco, and many others that are vulnerable to Heartbleed and use keys and certificates. Many of these operate behind the firewall and some may, incorrectly, assume replacing keys and certificates on these systems is not important. Assuming this would be a terrible mistake since behind-the-firewall attackers would love nothing more than to be able to spoof services like VPNs, security systems, applications servers, and more and decrypt encrypted SSL/TLS traffic.

Take action now while you still can

CISOs should not and cannot tolerate this situation. Some IT security leaders may be told by incident response teams that a full-scale rekey, reissue, and revoke is not necessary. Others may be told that it’s too complicated or time consuming. And there has been a false assumption that patching is all that’s required. Some may be misinformed, possibly by websites that show remediation is complete, but have no awareness of changes to keys and certificates, only to basic patching.

Do CISOs and security teams believe that usernames and passwords should not be changed? No. Therefore, they should not, and cannot, live with a situation where all keys and certificates are not replaced.

Venafi customers are quickly remediating

Venafi customers are speeding through incident response. With Venafi TrustAuthority™, security teams have full visibility into all their keys and certificates, which applications use them, and who owns them. Combined with Venafi TrustForce™, remediation is only a click away: keys and certificates can be changed and securely distributed and installed. All without any intervention from an application owner or system administrator!

Whether you’re a Venafi customer or not, please change ALL of your keys AND certificates. Triage keys and certificates from public vulnerable systems, then internal vulnerable systems, and then the remaining keys and certificates throughout the enterprise. Remediation will be complete and your organization will be secure.

You can learn more about how Venafi can help you quickly respond and remediate to incidents like Heartbleed here.

By Kevin Bocek
VP, Security Strategy & Threat Intelligence


Remediating Heartbleed with Next-Generation Trust Protection

Heartbleed Impact

The Heartbleed vulnerability unequivocally demonstrates the impact a single vulnerability has on all organizations when keys and certificates are exposed. Cyber-criminals have unfettered access to the keys and certificates on vulnerable systems, without any trace. Researchers that identified the vulnerability sum up the impact simply, “any protection given by the encryption and the signatures in the X.509 certificates can be bypassed” (Heartbleed) You must assume all keys and certificates are compromised and immediately replace them to remediate. Unfortunately, most organizations cannot!

The vulnerability is not limited to webservers, it impacts any system running OpenSSL 1.0.1 – 1.0.1f. This includes mail servers, chat servers, VPN’s, network appliance, client software, VOIP phones and more. Hundreds of software applications from security vendors have already confirmed their software as being susceptible to the Heartbleed vulnerability.

Next-Generation Trust Protection for Next-Generation Threats

Venafi Trust Protection Platform provides holistic remediation from the Heartbleed vulnerability. Via TrustAuthority and TrustForce, organizations are able to quickly identify any system susceptible to the Heartbleed vulnerability, regardless if it is a publicly facing web server or on the internal network and remediate.

Venafi TrustAuthority can quickly identify systems impacted by the Heartbleed vulnerability, establish how many keys and certificates are in use, where they are used, and who is responsible for them. Once TrustAuthority defines a comprehensive inventory of all X.509 certificates, they need to be replaced.

Venafi TrustForce uses lightweight agent and agentless technologies to automate complex activities, including rekeying and recertification, for which manual processes might open vulnerabilities. With TrustForce, the remediation of keys and certificates is completely automated and secure.

The following step-by-step process outlines how organizations can automate remediation of the Heartbleed vulnerability using both TrustAuthority and TrustForce with the Vulnerability Remediation Plugin.

Step 1:

Using TrustAuthority, identify any server that may be susceptible to the Heartbleed vulnerability. This can be achieved by scanning both your internal and public networks.

Venafi Search

Once vulnerable systems have been identified, patch them by upgrading to OpenSSL 1.0.1g OR recompile the OpenSSL library with the OPENSSL_NO_HEARTBEATS flag

Step 2:

Identify keys and certificates that need to be fixed based on knowledge of vulnerable applications.

Venafi search results

As you review results from various search types, you can select certificates individually or in groups.

Step 3:

The generation of keys and X.509 certificates is automated via the Work Queue. However, prior to initiating a Work Queue, it is critical to make sure that a new private key is generated to remediate further compromise as a result of the private key being stolen via the Heartbleed vulnerability.

From within the Policy tree under a policy object or certificate object ensure that your certificate does not have the “Reuse Private key” option selected.

Venafi prive key edit

Step 4 – 5:

Using TrustAuthority and TrustForce together, the new private key generation, CSR, secure distribution, installation and revocation process for certificates is all performed automatically via the Work Queue. For organizations that only have TrustAuthority, the secure distribution and installation is manual.

Select work type

Step 6 – 8:

Once all publicly facing servers susceptible to the Heartbleed vulnerability are remediated by patching OpenSSL and replacing the private key and certificates, steps 1 – 5 should be repeated for all internal servers impacted by the vulnerability.

Step 9:

Validation of the Heartbleed remediation is critical to success. For this you should validate all keys and certificates are replaced, detect anomalies and alert the organization on any related security events at least every 24 hours.

Contact Venafi to help accelerate your Heartbleed remediation.

By Gavin Hill
Director, Product Marketing and Threat Intelligence


By Harold Byun, Senior director, Product Management, Skyhigh Networks
As we’ve reported, hundreds of cloud providers were vulnerable to the Heartbleed bug in OpenSSL even days after the vulnerability was widely publicized. Looking at the latest data pulled this morning, much progress has been made and there are only 42 cloud services that are vulnerable to Heartbleed.  For these services, user data, passwords, and private keys for these services can be stolen using a simple exploit.
However, more alarming today is the number of cloud services that have not fully addressed their past vulnerability. After patching SSL, the next step cloud providers must take is to reissue their certificates. As reported by CloudFlare, Heartbleed can be used by an attacker to access private keys and impersonate a website. Since Heartbleed exploits don’t leave a trace in server logs, cloud providers must assume their private keys have been compromised even if they don’t have any evidence of them being stolen.
Certificate updates trail Heartbleed patching
Most websites have patched SSL but they are reissuing and revoking certificates at a much slower pace. Netcraft reported that only 30,000 websites (out of more than 500,000) reissued new certificates by the end of last week, and evenfewer have revoked their certificates. While not completely eliminating the risk of a man-in-the-middle attack (MITM) this is a critical step in reducing the risk of these attacks.
Skyhigh is tracking certificate updates across cloud providers and as of this morning only 13.3% of cloud service providers affected by Heartbleed have updated their certificates. A smaller percentage have both reissued and revoked their certificates, making them vulnerable to impersonation in a phishing scam or man-in-the-middle attack. Most certificate authorities have agreed to replace certificates for free, but there are complaints they aren’t prepared for the volume of certificates that need to be reissued.
Already we’re seeing that Heartbleed has exposed not just a vulnerability in SSL but vulnerabilities in the way we approach security. According to security researcher Bruce Schneier:
“We’ve learned how hard the human aspects of a security system are to coordinate. We’re learning that we don’t have the infrastructure necessary to quickly revoke millions of certificates and issue new ones. We’re learning that some of our critical open-source software is maintained by volunteers who have busy lives, and that often no one else is evaluating that software’s security. We’re learning how complicated the process of disclosing a vulnerability of this magnitude is.”
Cleaning up and determining your exposure
Aside from critical infrastructure your company uses, corporate IT departments are being asked to quantify their exposure. With over 96% of companies using cloud services impacted by Heartbleed, the chances that your sensitive data was vulnerable is extremely high. Skyhigh has already provided our customers with the cloud services they use that were impacted, and we’re extending those audits to any company for free. Email us at [email protected]skyhighnetworks.com for more information.