Transcript
This transcript has been edited and refined for clarity.
Anna Delaney: Hello, this is Proof of Concept; a talk show where we invite security leaders to discuss the cybersecurity and privacy challenges of today and tomorrow, and how we can potentially solve them. We are your hosts, I'm Anna Delaney, director of productions at ISMG.
Tom Field: I'm Tom Field; I'm senior vice president of editorial with Information Security Media Group.
Delaney: It was great to see you in New York for our Financial Services Summit. But, Tom, this is our second Proof of Concept episode on the topic of software supply chain security. In the first episode, we explored the state of the software supply chain, the MOVEit breaches, the role of SBOMs and transparency in software development. In this episode, we are tackling open-source governance, which while immensely valuable, also presents several challenges for organizations and software developers. We'll be diving into the benefits of OSS adoption, challenges around consumption and visibility, vulnerabilities on maintained software and so on. What are you hearing, Tom, right now, from security leaders on this subject?
Field: In just the past two weeks, I've hosted discussions in Washington D.C., as well as in Chicago on this very topic and spoke with government leaders. We talked about the progress they've made in the mandates of Biden's cybersecurity executive order, and it's like MFA? Check that box. Zero trust? Check that box. SBOM? Not so much there. When talking with prominent business leaders, in Chicago last week, this was one of the topics that were most vexing. It comes down to, there is so much open-source code within the applications that are in primary use, and security leaders don't know where it, necessarily, is. They don't know, necessarily, where it came from, and they're wrestling with this question that we're going to ask of our guests today. It is a prominent question in the industry now, which is: What's wrong with using free software that's posted by strangers on the internet? It's a tough one, and is a source of so many vulnerabilities today. It's become an area where the adversaries certainly are shifting left, as well as its defenders, in planting potential BOMs in the code that's being used in open the field as open-source components.
Delaney: But, you raised some really important points, and I look forward to hearing what our experts have to say. Let's welcome back our security leaders, Mike Baker and Chris Hughes, who joined us for the first part of this series. So take it away, Tom.
Field: Mike is VP and CISO with DXC Technology. Mike, I'm going to ask you the same question we, Anna and I, talked about here: What is wrong with using free software that's posted by strangers on the internet? I don't see anything wrong with that.
Mike Baker: I don't know if there's anything inherently wrong with it; I think, it's a governance question. There are a lot of advantages when you look at open-source software and the open-source software community. Number one, cost! Generally a lot of this OS is out there, you can use it, it solves good business problems, and there's large communities that are backing it up. You have transparency in the code, you can look at the code, and it is not a black box like off-the-shelf software. It prevents companies from getting in vendor lock in. Sometimes, when you get in these business partnerships, you get locked into that specific platform. There is a lot of flexibility options there. There are benefits to companies using open-source software. However, it is more of a right time, right place, and right governance situation because there are a lot of risks, and you're going to invest to mitigate those risks to make up for the potential savings or other opportunities you have using this sort of software. Look at the top three risks, just the vendor support and the updates. When you go out to an established company, they have a supported piece of software that they're looking at again, and there are problems with that too, with COTS software. It's not like it's exclusive to OSS. However, the consistency of that support and the updates and that community dependency is important to dive into as a released open-source software. The trustworthiness of the code, you have that transparency. People say that transparency automatically equals to the code being secure. However, you have to then have the processes to audit that code, to review that code, and make sure you're receiving that from a trusted source, because there's a lot of adversarial intent to get in the middle of that pipeline, introduce malicious software. Between the trusted support and the trusted source and auditing that code for your use; you have to do that to make it trusted. I think the last thing I want to point out was just, inherently, depending on the community and the piece of software, what's the documentation like? What's the education you have to do to your user and administrators to use that piece of software? That's something you may have to fill the gap or hole related to the use of that open-source or freeware software. There are a lot of benefits. However, when you look at the lack of consistent support, viewing the code, making sure you're getting from trusted sources/trusted communities, making sure the documentation is up-to-date, making sure users and your administrators are educated, there are a lot of risks that outweigh the benefits.
Field: I hear a lot here about visibility and tracking; what are some of the specific challenges that you face when it comes to consuming and maintaining open-source components?
Baker: We had talked about software bill of materials being the future. I'm invested in that; I think it's great thing for the community to bring everything up to a par level of transparency. However, I don't think we're there yet as a community. Number one, we're struggling with the adoption of SBOM. Second, we're struggling with the tools that are going to enable companies like ourselves or our customers to look at that, holistically, across a large enterprise. Third, just to keep it high-level is - the talent that's out there is hard to find and bring in, and keep at the companies. This is a unique and specific skill set. There's not much stuff out there; we do have a lot of people who can develop code. With software supply chain experts, it's still an emerging field, we have one of the top ones here on the panel. It is an emerging field, and it's a unique skill set to have here. When we look at the tracking, if you take SBOM out of it, we talked about the updates and how those can be dynamic and frequent and on different schedules and on different libraries. Where is the tooling to make sure that the software is up-to-date? I talked about that being one of the risks, but you're talking about updating different release schedules for different libraries and that software, different versions. It can get complex, quick. You're also talking about a lot of dependencies in that software; if you update this, does it break a dependency down the chain. There are specific and unique tools for doing that dependency analysis across a piece of open-source software that gets really challenging. That just goes back to that staffing thing. Are you ready to staff up the people processing the technology to do this the right way? That, I think, is a tough cost trade off question that each organization has to tackle. However, I want to kick it to Chris as well, when it comes to kind of the customers he's serving and the clients he's serving. What are you seeing out there in terms of consuming open-source or freeware software?
Chris Hughes: I think, in many organizations, like you talked about, it's not necessary that using open-source software is inherently bad. It's just that many organizations that are using it simply don't have a good inventory of what they're using. When we think about critical security controls, software asset inventory has been a critical control for decades. Many organizations don't have a good understanding of what open-source software they're using, whether internally developed software or that they're consuming from third parties. A lot of proprietary vendors are also using open-source software extensively as well. As we talked about, it's great to have that transparency factor. There's a saying that underneath enough eyeballs all bugs are shallow. That's kind of the pitch that you hear for open-source. The reality is, when you look at the maintenance of the open-source software ecosystem, some of the metrics are downright alarming. Some of the metrics I've been able to find are that 25% of projects in the open-source ecosystem have one single maintainer contributing code to it, and 94% of them have less than 10. When you're looking at eyeballs, that's not a lot of eyeballs to keep an eye on these projects. Then you have issues like in terms of getting timely updates. When we look at the software in terms of updates, when you have a proprietary piece of software, you have service-level agreements, and things around getting updates or vulnerability for mediating a certain timeline. With open-source software, that's not the case. These are not your suppliers, these are people doing this on their own free will and their own free time. They owe you nothing. It's free to use as is and you take it as is, for better or worse. Many organizations are simply struggling with that reality as they look across their environment, their ecosystem and see how widely they're using open-source software. The growth has been tremendous, but our approach to security hasn't simply scaled at the same time of our open-source software adoption has.
Baker: Chris brought up a great point that I wanted to bring up was, the licensing is so different, so varied, where it can be so unique. We talked about all the investments you need to do the stuff. However, when you're looking at the uniqueness of that licensing and the investment and the proper legal expertise to look at that licensing and understand if your usage is appropriate, even that is super huge in this world. Chris also brought up inventory, which is special. Inventory is hard enough, just seeing what you have, but when you think about using this level of software, those specialized skill sets and inventory tools to go that one layer down to what we talked about those dependencies within that software across the board. Transparency is great, but it adds a significant amount of complexity into tracking and maintaining that particularly at scale. You really scared me with the "community strength" comment, Chris. Saying that one person for x - 50% or whatever - of the projects. If you don't have the community strength behind these, this software, is it appropriate for enterprise use? I would hazard that the answer to that question is likely, no. How are you assessing that community? How are you assessing that developer? The backbone behind it? The supportability behind it? Huge questions that lead me away from using this software.
Hughes: Sonatype released their state of software supply chain report for 2023, recently. They found that 11% of the projects they assessed are actively maintained of the entire open-source software ecosystem - 11%! It's daunting to think how many projects are simply not being maintained out there but are still widely used. As he points out, vulnerabilities are great, but vulnerabilities are also a lagging indicator of risk. It tells you something's wrong, and we know what's wrong. However, if you know what's wrong, so do malicious actors, they know what's wrong as well. You need to be looking at other things like project health and the vitality of the project, what kind of protections they have in place, and so on.
Field: It's funny, you mentioned Sonatype. I like to speak with Brian Fox, the CTO and co-founder. Our ongoing joke is when I see him, as I say, "Brian, what's the state of software supply chain security today?" and, it ranges from embarrassing to unacceptable to whatever explanation he can put in there. However, very consistent with what you say, Chris, I appreciate your insight here. Michael, one last question for you. I'm going to turn this back to Anna and to Chris, we've talked about a lot in just a few minutes here. How are you at DXC addressing the issue of unmaintained open-source software and all the potential security risks we've discussed?
Baker: I'm not going to talk about how we're doing at our company specifically, but, in general, it goes back to policy and process. We've talked about the necessity to have an enterprise-wide view on what the software is, have that policy, communicate that policy, educate users, particularly, in a wide scale, in terms of what is the appropriate use of this? When you look at that, and you look at the process, are we going to have the process in place to vet this, to look at the communities we mentioned, to hire the people and the expertise needed to run that process effectively? But, I think it goes back to the basics. What's the usage of this? What do we what do we use them for? When you look at data sensitivity, business criticality, these are cornerstones of any third-party risk management program or software supply chain program. How are we doing the basics right? Do we know what data will be consumed or manipulated by this piece of software? Do we know the business criticality or the RTOs/RPOs? Are we asking those questions and then giving that to the people and the analysts that have the expertise to do open-source rather than just generic third-party risk management? This is way beyond the world of questionnaires being sent out. You can't send a questionnaire, nor are they obligated to answer it. We make sure we have to staff it. We have to get the vetted policy across the board, a good process that works, and do the basics right around business, criticality and data sensitivity.
Field: Mike, thank you so much, Anna, back to you and Chris.
Delaney: This is absolutely brilliant. Chris, I didn't do a formal introduction earlier. I'm going to slot it in now. You are Chris Hughes, you write prolifically on the topic of software supply chain security, co-founder and CISO at Aquia. Following on from Mike's last answer. What are some common misconceptions or expectations that organizations have when it comes to open-source software maintainers? How can these be better understood and managed?
Hughes: Some of the most common misperceptions is that it is a public good. People assume that they can just continue to use it and it's inexhaustible, but the reality is that these projects are being maintained by a small group of individuals that are doing it in their own free will. They don't owe you anything. If there's a vulnerability, they don't have to fix it; if there's a bug or a defect, they have no timeline that they have to adhere to, like a proprietary software vendor does, for example. So, you can't send them a security questionnaire, you can't make demands of them to fix defects or bugs or vulnerabilities. It doesn't mean they won't. There's lots thriving in the robust open-source software projects that have tremendous support from people in the community. They're responsive on par with some of the best technology leaders in the industry. On the other hand, there are a lot of open-source software projects and components that are simply not well-maintained and kept up. Another big misperception is that people don't have an understanding of what they're even using. They often look at their direct dependencies, but studies are starting to show that most - six out of seven - vulnerabilities are from transitive dependencies. We think about the supply chain we see how it's a supplier of target who got breached, who caused the target to get breached. It's a dependency of your dependency that gets compromised, that causes you to essentially be exploited or be vulnerable, for example, so many organizations just don't have a good understanding of the direct dependencies, let alone the transitive dependencies and kind of sprawl of how that spreads out and the vulnerabilities that can come back to impact them from that dependency tree.
Delaney: Given that six out of seven vulnerabilities come from transitive dependencies, what are some strategies, or even best practices organizations can implement to manage and secure transitive dependencies effectively?
Hughes: Starting to look at the ecosystem, we're seeing some innovative vendors that are coming out; there have been software composition analysis tools and things like that for quite a while. But, we're starting to see some more innovative vendors come out with more modernized tooling that can help to map out the dependency trees for you. Not just direct, but also transitive dependencies, and then provide critical insight. We've seen a lot of shift left, Tom used that term earlier, and we've seen the industry try to push for shift left. The problem is that a lot of these tools - SCA, SaaS, TaaS, etc. produce a lot of noise. It's one thing to know that I have a vulnerability, but is it reachable? Is it exploitable? Should I be concerned with it? If I send a developer a report with 678 vulnerability findings, and I tell them "This has to be addressed before you go to production." I'm not making friends, and I'm not breaking down silos, I'm building up resentment and they're not going to want to engage with me. We're starting to see some innovative SCA vendors, for example, provide things like reachability for these transitive dependencies. Is it vulnerable? And is it reachable on my code? For example, is it actually used? Is it called as part of the application? There's some great industry-led efforts that are being led that are available to anyone. A couple of things I want to call out is going to be CISA's Known Exploited Vulnerabilities Catalog that's going to show you not only is something vulnerable, but is it actually not going to be exploited in the wild? Should I be concerned with this? Are malicious actors targeting it right now? The EPSS - the exploit prediction scoring system – is another great resource that uses a machine learning model to come up with figures around if vulnerability – CVE – is likely to be exploited in the next 30 days? They've been incredibly accurate as they keep iterating on this model. It can help prioritize, if I show something to a developer, I want to show them, is it known to be exploited? Is it reachable? Is it likely to be exploited? Because vulnerability backlog some studies are showing are in the hundreds of thousands, even 1.1 million in some organizations and large enterprises? So, we need some prioritization strategy here, or we're just going to drown people in reports with too much information, and they can't act on it.
Baker: That brings up the issue of the ability to pivot if the community support goes away, or they're not doing something, how quickly can you pivot? Is that application used for something else? Or, can you plug that developer gap with people within your own organization? Are they trained enough on what this piece of software is? You talked about the continuing emerging nature of the vendors in this space, which does give me pause, someone who has to run the program. Chris, to me it's cool, it's innovative, and will we get there eventually? Will we get those dependencies chains cracked, at some point? Sure. However, that's a lot to put on something that's critical to a business or have any sort of scale. To your point, vulnerability management for off-the-shelf and supported software through traditional vendors is hard enough. You add this layer on top of it, you're going to talk about scaling the team and the cost associated with the program.
Hughes: No doubt about this. There are additional resource constraints. Tom made the joke what's wrong with using software from strangers on the internet? For a long time, that's the kind of perspective we've had, we've just kind of used it without asking any questions now we're seeing malicious actors target these, while they use open-source software projects and components. Now we're pulling the curtain back, and we have a lot of things we need to be concerned with and start paying attention to, which, as Mike points out, demand resources. We're already resource constrained when it comes to talent in the workforce.
Baker: At the same time, you have that philosophical question of the transparency of the software as opposed to the same supply chain threat, or risk that exists with off-the-shelf or enclosed software. We can go either way on it. Everything's a cost trade off, but it's a very interesting question to back up in terms of that risk. The actual support you need from a non-open-source vendor is starting to be the exact same support you need from an open-source vendor. You need to know the dependencies and the bill of materials, along with the existing supportability. It's just interesting how these worlds are merging together after things like SolarWinds and other issues we've seen lately.
Hughes : To add on to what Mike says, not that open-source software is inherently bad, or inherently more risky than proprietary software. This year, we've seen breaches from Okta and Microsoft, and some of the most leading capable technology vendors in the ecosystem. It's just that malicious actors are increasingly targeting software supply chain entities, whether it's proprietary software vendors, open-source software projects, nothing is infallible or protected from this threat.
Baker: We have to prepare now! We've logically got ourselves back to the same place. What I think any company at any point in time right now needs is to start staffing. A software supply chain or software supply chain risk, regardless of open-source or off-the-shelf, they need to double down in terms of their focus from a people and process side on this holistically across the board, because it's not going away.
Delaney: That was brilliant! Just to add, you mentioned, Chris, new technologies, new vendors, are there any specific tools that you recommend for organizations to enhance their open-source governance?
Hughes: I don't want necessarily name any vendors, because I'll come across biased. However, as I talked about, look at software composition analysis, tools, particularly newer, more innovative vendors that are adding these capabilities around reachability or known exploitation are likely to be exploited to vulnerabilities. Look at some of the emerging SBOM vendors who are just trying to help bring transparency. Like Mike said, we have a long way to go with SBOM, we're not there yet. Are your vendors even talking about this? Are they willing to provide these artifacts or provide that level of transparency? On the flip side, you have tools that can help you ingest and start to reason about some of the things that are in the software you're consuming and using. There are some open-source tools out there as well as proprietary tools. Definitely take a look at SBOM tooling, software composition analysis tooling, especially more newer, innovative ones with reachability analysis, you can start to peel back the curtain and get an idea of what you're up against and what your posture looks like.
Delaney: Let's all come together for a final question, and, hopefully, answers from you. Over to you, Tom.
Field: Mike, Chris, for both of you. Given everything we've talked about here over the past 20 minutes or so, how can organizations today strike a critical balance between leveraging the advantages of open-source software and mitigating the risks effectively? Mike?
Baker: I think we've talked about it. Going into it eyes wide open. There's advantages, like I mentioned, cost, flexibility, transparency, voting vendor lock in. However, there's other significant costs associated with setting up the right people, processes and technologies. Is it a risk acceptance sort of thing or, is it something that you're going to apply slowly across non-mission critical sort of applications or uses? Going into it eyes wide open and no, it's not like the panacea. This is something that organizations need to prioritize starting now. Right across all of their software, not just open-source, in terms of understanding what their third-party risk management program looks like. Is it accounting for software supply chain risk? How are they keeping up with the industry that's rapidly evolving?
Hughes: I'll jump in and just build on that comment from Mike. I like a few things he said there. One is that there's no panacea. There's no silver bullet. It is a journey. It's just a matter of understanding where we are right now, when it comes to software supply chain security, are we at least aware of the threat? Are we aware of what our software suppliers look like? What measures are we taking internally to put the people process and tooling in place to start to tackle this threat? Just getting a sense of where you stand and start to do a gap of where you want to be? As he talked about aligning with risk tolerance, it's not that we're trying to eliminate all risk and anything is inherently automatically bad, but just understanding what risks are we willing to tolerate, what protections and controls are we putting in place to mitigate the risks that we aren't willing to accept.
