CloudBleed – Cloudy With A Rain of Data

Last week, Tavis Ormandy from Google’s Project Zero discovered a huge problem in the security of CloudFlare‘s edge servers. He contacted the company and reported that corrupted web pages were returned by some HTTP requests run through the popular hosting provider. Below, we shed light on the problem as well as provide information on how to prevent Magento form the new bottleneck dubbed CloudBleed.

Uber, 1Password, FitBit, OKCupid, and hundreds of other internet giants have been affected. A huge users personal data leak occurred due to an error in the CloudFlare’s code. The error caused pieces of memory to dump into web pages. As a result such sensitive information as encryption keys, passwords, cookies, and HTTPS requests became available in public caches.

Cloudflare have been leaking customer HTTPS sessions for months. Uber, 1Password, FitBit, OKCupid, etc. https://t.co/wjwE4M3Pbk

— Tavis Ormandy (@taviso) 23 февраля 2017 г.

The reason of the problem has been already discovered and fixed – it was just a typo in the code. CloudFlare reports that it was an old code snippet with a latent security problem. The team was testing everything except the old software. The recently discovered mistake caused buffer overrun and further consequences. CloudFlare reports about the problem only now, because its main effect is already eliminated: the personal data is no longer available in search engine caches.

Although the code had been in production for years, you don’t have to panic – it was secure and stable until the problem caused recently. The greatest impact was observed from February 13 and February 18.

Usually, 3 months are necessary to deploy a fix such bug, but the CloudFlare team spent only 7 hours. Check the detailed timeline (UTC), if you don’t believe:

• 18 February 00:11 – Tavis Ormandy asks for CloudFlare contacts;
  • 00:32 – Google provides CloudFlare with bug details;
  • 00:40 – San Francisco cross functional team assembles;
  • 01:19 – CloudFlare disables Email Obfuscation;
  • 01:22 – London team enters the game;
  • 04:24 – CloudFlare disables Automatic HTTPS Rewrites;
  • 07:22 – A new patch is deployed worldwide, implementing kill switch for cf-html parser;
• 20 February 21:59 – SAFE_CHAR fix is deployed;
• 21 February 18:03 – previously disabled services are relaunched.

For further information on the problem, read the official article in the CloudFlare blog: Incident report on memory leak caused by Cloudflare parser bug.

What should you do now? First of all, change all your passwords, especially those on the affected websites. Next, rotate API keys & secrets. Even if you didn’t use any of the affected websites, remember that an affected website could have made an API request to a non-affected one, making your private data absolutely insecure.

What else? And what about Magento? It depends on how much you use CloudFlare. We recommend you to follow this discussion on StackExchange for more details: Magento and Cloudflare’s Cloudbleed traffic leak.

A list of websites possibly affected by the CloudBleed HTTPS traffic leak is available here.

Have any thoughts about the CloudBleed problem? Please, share them in comments.