ISMG Editors: The Complexity of Rackspace Zero-Day Attack

Anna Delaney: Hello, happy new year. I'm Anna Delaney and welcome to the first episode of the ISMG Editors' Panel of 2023. This is a weekly conversation among ISMG editors on the most recent InfoSec news and cybercrime trends. Joining me today Tom Field, senior vice president of editorial, Marianne Kolbasuk McGee, who leads our healthcare coverage, and Mathew Schwartz, executive editor of DataBreachToday and Europe. Good to see you all. Tom, a foreboding sky behind you. Tell us more.

Tom Field: It wasn't meant to be that way. It was a pretty one. It was the moon actually on New Year's Eve out at the local Cineplex. Took teenagers out to watch a movie and coming out at 11-12 o'clock at night. This is what it looked like. So I thought it was a lovely way to start the year.

Delaney: It is ... always a beautiful moon and actually it's a full moon this week at 12 night. Marianne, beautiful outdoors seen as always.

Marianne McGee: Thank you. This is a lake that's not far from where we live. And we were taking the dog for a walk during the end of the day. And I was like "this is pretty - I'll use this for background." So that's what I did.

Delaney: Good choice. Mathew, are we in Dundee?

Mathew Schwartz: Yeah, this is the McManus in the center of Dundee. It's an art gallery museum. And if you have a double decker bus, just in case anybody was wondering where in the world this might be. So just on a rainy day, there's a big puddle right behind me. You can't see it. But I was splashing about with the new year cheer.

Delaney: Very good. So the green buses in Dundee.

Schwartz: We've got green, we've got red, all sorts.

Delaney: We've got red. Okay, very good. Good to know. Well, I present you the English countryside. This was taken on a very long, fresh and muddy walk in between Christmas and New Year. Just what one needed after the insanity of Christmas.

Field: Big black dog coming up over the hill.

Delaney:There are a lot of sheep as well. Tom, you have been talking with the CISO of Zoom. Are we secure on this platform?

Field: I'm told we are! You are right, I did have the chance to speak last week with the new CISO. He just took over in the late summer, early fall. Michael Adams is his name. He's got a good distinguished history. He is the latest CISO at Zoom and we talked before the year end about his predictions for the New Year, which you can imagine revolve an awful lot around the cloud and around collaboration and software. But we spoke as well about the state of security generally. And guess what, we did speak about the state of security at Zoom, because you know, we aren't that far removed from the days we are all for tentatively trying this platform. And they remember Zoom bombing. So we talked a bit about initiatives. If you like, I wouldn't mind sharing a clip of our discussion about how Zoom is more secure today than it was just a year ago.

Michael Adams: It's a great and important question, because I think for us, we've really seen a strong evolution in the culture at Zoom, right? Security has become instilled in our culture. To me, the biggest advancement we've made on this front has been our investment in our security program and team really since 2020. We've done a lot of building out that program in a more comprehensive fashion. And then I think what we've pivoted to now is really an optimization paradigm where we're taking the foundation elements that we built, the growth we've had in teams and tools and really more sophisticated advancements. And we're dialing that in. And we're focused on the biggest risks, biggest impact areas. I'll say as a company, we've also kind of stepped up by continuing to grow the security features that we offer to our customers. At Zoomtopia, this past November, for example, we announced a series of new offerings and they include things like end-to-end encrypted feature for Zoom mail service, enterprise auto update. This is significant. In the last year, we rolled out automatic updates to our broader consumer base but last month, we've now introduced automatic updates for enterprise customers and we think that's a significant accomplishment. And then there are others that are not insignificant either such as advanced encryption for Zoom phone voicemail, so I'd say our program or people, and then also some of what we're offering through the technology to our customers themselves.

Field: Not insignificant. When you think about it, we all started using Zoom almost three years ago. That was at a time you could go up to the URL, essentially plug in a number and you could join anybody's meeting indiscriminately. Things have come far as Zoom is grown from something that we use to just sort of tide us over in the early stages of the pandemic to as natural a part of our business life today as a conference room used to be.

Delaney: Absolutely. It's amazing to see the security evolution at Zoom, and Tom, you mentioned a list of predictions that he offered and you've seen predictions come and go, or even stay. What surprised you this year? What's new? What's the new trend that you're watching that he mentioned?

Field: I don't know that there's anything particularly new. But here, look, I talked to lots of people over the course of the last quarter of the year leading up to 2023 about what their predictions were for security, spoke to researchers, spoke to vendors, spoke to people that have been CISOs. This was my first opportunity to speak to a CISO of an organization that we all use to be able to hear what he has to say. So I'll tease it only by saying the interview is on our sites right now. And I encourage people to take a look at it because it is one CISO's look at what this year ahead, how it shapes up. You find a lot of commonality there in terms of threats, in terms of adversaries, in terms of attack surface, but I think he's got a unique perspective.

McGee: Yeah, in fact, I think that what was buried in this $1.7 trillion omnibus spending bill that was signed into law at the end of last year, less than a week ago, by President Biden is one of the more interesting and significant U.S. legislative developments that I've seen in a long time having to do with health care cybersecurity. Buried in that bill are provisions that basically give the Food and Drug Administration more expanded authority over medical device cybersecurity. Under the new law, the medical device makers now are required to submit cybersecurity plans for their new products as part of their submissions to the FDA for market approval. That includes submitting to the FDA how their devices can be updated and patched to address vulnerabilities. The kinds of security controls that are contained in the devices, security testing information and so on. The medical device makers must also submit to the FDA a software bill of materials for their products. Now, the FDA, for the last several years has been urging medical device makers to address cybersecurity issues in the pre market of their products, including taking some of the steps I just mentioned. But until now, the FDA did not have legal authority to require medical device makers to include cybersecurity plans for their products in their submissions to the FDA for market approval. I spoke with Dr. Suzanne Schwartz of the FDA who heads up the FDA's medical device cybersecurity effort and for medical devices, specifically, but she thinks that this is a very significant development and she says the FDA is very optimistic about this law having a positive effect on cybersecurity in the overall health care ecosystem long term. That's because unlike many, if not most legacy medical devices that are unused today, upcoming generations of medical devices now will be required to address security concerns upfront, hopefully making those products more secure as those newer products also begin to age out. Now, since the legislation was only signed into law by President Biden last week, the FDA is still assessing the details of how it will implement the new law and enforce the law. While some of the larger, more established medical device makers today have already been doing many of the things that are called for under the legislation, including things such as coordinating vulnerability disclosures, designing security into their new products, these new requirements are likely to be more of a shock to some of the smaller and newer specialty device makers and less mature vendors for which cybersecurity has not been a very high priority. So it'll be interesting to see as the year plays out how the FDA regulations get fleshed out and how device makers respond.

Delaney: Very good. Well, we are encouraged for the year ahead. Marianne, there's a new law, new U.S. law, which pertains to the cybersecurity requirements of medical devices. So talk to us about this potential game changer. And Marianne, how does this new law combined with the 2021 year - the cybersecurity year launched by President Biden?

McGee: Well, Dr. Schwartz says that it meshes, for instance, you know, the software bill of materials that's called for under the executive order. You know, there's a bunch of other things that are kind of similar and, you know, again, the FDA has sort of been pushing for these things for a while. Some of these provisions were part of standalone bills that were introduced over the last year or so. That just never gained traction, but they surprisingly showed up in this budget bill of all things, and plus the FDA gets $5 million in spending funds to support these efforts. And that could include hiring new cyber experts to be involved with assessing the new products that these submissions involve.

Delaney: Yeah. Just to be clear, this law pertains to new devices. Not all, right?

McGee: Right.

Delaney: Great positive news indeed. Matt, back in December, we saw ransomware attacks against Rackspace' hosted Microsoft Exchange environment. You got more information to add to the story I believe.

Schwartz: I do. And we did see this big attack against, as you say, the hosted Exchange environment at Rackspace. So this is primarily hits small and mid-sized customers who were using Hosted Exchange services. And the TLDR there is they're no longer going to be using Hosted Exchange services. Rackspace is no longer going to provide it. They are moving everybody to what used to be known as, I guess, Office 365, Microsoft 365 now to get their email. There's a couple of other ways they can get their email as well. Rackspace offers its own email, they can go there if they want. But Microsoft 365 is what a lot of people are recommending, and possibly they should have been doing it already. I'll leave that debate open to others. But Rackspace says that the Play ransomware group is behind the hit against it. So there's a lot of detail to unpack here. I'm going to do it real quickly. There is a series of attacks that came along last September, which had been attributed to a nation state attack group with alleged ties to China. They were using some exploits to hit Exchange. And these two exploits are referred to as ProxyNotShell. Just to be confusing. Back in 2021, there was another hit on Exchange called ProxyShell, which used three exploits. This is different but looks similar. So they were able to use these two exploits to execute code remotely on Exchange servers. Bad news. Microsoft has shipped a patch for this problem for Exchange in November, and a lot of people started to put the patch in place. The patch helped protect organizations against this attack. Rackspace chose not to install the patch. Now before you rush to judgment, a lot of people didn't install the patch because it was causing problems with OWA - Outlook Web Access. People who installed the patch reported that they oftentimes or in some cases, at least, could no longer use OWA, which is a problem. So Rackspace and others decided to hold fire, they instead used workarounds, or mitigations that Microsoft has specified could be used instead, until you can get the patch in place. What those mitigations apparently didn't protect against was a different kind of attack, which used a certain exploit, not from ProxyNotShell in the first instance. And then in a second, activated the second ProxyNotShell vulnerability in order to ... execute code remotely in exchange environments. So attackers were able to accomplish the same thing using a slightly different attack chain. We know this because CrowdStrike was brought in to investigate at Rackspace. And on December 28, it issued a report into a series of attacks it said it had traced to the Play ransomware group, which appeared to be ProxyNotShell attacks, but which in fact, it found used this other vulnerability, a zero day exploit, had not previously been known. And then with that, change it together with the second part of ProxyNotShell, which even if you'd applied those mitigations in November, but not the patch, the mitigations did not protect you. CrowdStrike investigated at Rackspace. It didn't say Rackspace is one of the victims, but Rackspace said it was one of their victims. It came forward and said - a lot of people rush to judgment here and said, "oh, we didn't patch against ProxyNotShll mitigations." They said, "we did mitigations. Microsoft's mitigations didn't note that this other exploit was a risk we might face for not having patched." Like I said, there's a lot to unpack there. Did Microsoft know that there was this zero day flaw in the wild that can be used against Exchange? I've asked them. I haven't heard back. Probably, it didn't though or it probably would have put out the word to alert people. So interesting to hear some takeaways. Don't rush to judgment, I suppose, in some cases. On the patch and parish front. I mean darned if you do, darned if you don't, right? People who rushed to install the patch would have seen the inability to access OWA. This is a problem, especially if you're Rackspace and you got thousands of companies using your services. So they, like I said, held fire. But it turns out that one of the mitigations that you now need to use because there's no fix for the zero day flaw yet is to deactivate OWA if you haven't already done so, because that's how attackers broke into these organizations. And so, you can see why Rackspace may have decided just to ditch Hosted Exchange. They're still trying to get all of the emails recovered from the attack. December 2, there's still - some customers have got their emails back and migrated over to a different platform, but they're still in the process of restoration. Thousands of companies impacted. It's a really big mess. There's already been a class action lawsuit filed against Rackspace. But it's just fascinating. These additional details that have come to light. It looks like Rackspace tried to do the right thing. And somebody came up with a way to still get around those mitigations.

Delaney: Complex story. You've provided very helpful details there, Matt. So what do we know about this ransomware group Play in the tactics.

Schwartz: They're one of a number of ransomware groups. One of the things we don't know is if Play was able to exfiltrate data, were they able to steal Microsoft Exchange mailboxes. Rackspace hasn't commented on that yet, despite being asked. I suppose we could see data breach notifications if this did happen. But I think it has just concluded its investigation. And so additional detail will no doubt be coming to light. But there's a lot we don't know yet. Again, hopefully, we will see some additional detail come out. Kudos to CrowdStrike - difficult phrase there - especially in the new year for releasing this information. It's very actionable and helps other organizations in the same situation, meaning they use Hosted Exchange to protect themselves.

Delaney: Very good. Well, we await further details. Thank you, Matt. So finally, last week, last year, and last month, there's only one week of the year, I can say that. We discussed general predictions for 2023. And this week, I want you to don your Nostradamus hats again, and share one word or trend or a topic or a technology or even a ransomware group which you believe will dominate the industry headlines this year. What would that be?

Field: One word for you, Anna? Plastics. No, sorry, that was from the Graduate. One word though, close. Platforms.

Delaney: Okay. Another P. Marianne, your word?

McGee: I'm going to say vendors. Even the conversation we just had here today, you know, it's all about vendors, the security of vendors and how vulnerable their clients are to things that happened to them. Especially in health care, you know, a lot of the attackers, for instance, are kind of saying, "Okay, we're not going to go out for the hospital directly, but let's go over, let's go after a vendor that has many hospitals as their clients." So I think that's going to be a continuing theme.

Delaney: Mathew?

Schwartz: I'm going to say ransomware simply because we continue to see so much innovation with ransomware wielding groups. And while we see groups spinning off and trying other tactics, they are, I think, stoking the fires inside organizations, even if people aren't cybersecurity experts in terms of defense, and knowing they need to sharpen their game, and not knowing what's going to hit them next. So I think it is a useful forcing function for people to better understand cybersecurity. And I think it's an unfortunate cybercrime trend, and that they're going to keep hitting organizations and extorting hundreds of millions of dollars.

Field: And I feel bad that didn't qualify my answer: platforms. I say that because at a time when there are economic pressures on organizations, and they're looking to consolidate their tooling, and deal with fewer of Marianne's words: vendors, so they don't suffer more of Matt's word: ransomware, I see organizations shifting somewhere, at least, talking about shifting from point solutions to platforms.

Schwartz: Yeah, and a lot of platforms give you better security as well. I mean, they will give you more out of the box security with all of the right presets activated so that you have a harder time shooting yourself in the foot. Sorry to interrupt you.

Field: Spot on. So there you go, Anna. Your word?

Delaney: China. I think Russia dominated 2022. I think we'll be seeing a lot more action from China. We're already seeing some. Watch this space. We will compare these answers in December of this year.

Field: Will you put these in an envelope, put it up there on your safe deposit box?

Delaney: Have to remember. Well, Tom, Marianne and Matt, it's always a pleasure. Thank you very much for starting off the year with me today. Thank you.

Schwartz: Thanks for having us back.

Delaney: And thank you so much for watching. Until next time.