You Can’t Automate The Difficult Decisions

00:02 — Jamie Parker
On the surface, security seems like an obvious candidate for automation. IT systems are complex conglomerations of components, most of which are potential vectors for attack. Proper configuration, constant vigilance, and rapid response are essential elements of an effective security system. Automation helps security teams be more effective with those and more. Yet, when security is an afterthought, or even just considered late in the development cycle, the odds of avoidable catastrophic breaches go up. But solutions have been difficult to implement.

00:41 — Jamie Parker
In this episode, we hear from a security advocate at Microsoft about her effort to better integrate security into development processes. Prioritizing security is a tough balancing act. How do you keep your system secure without excessively slowing down development? The struggle over cybersecurity is a constant back and forth, and automation is an absolutely essential component. Let's imagine you're leading the development of a new project.

01:12 — Joylynn Kirui
You've already mentioned to your stakeholders that you're supposed to deliver particular system by, let's say, 30th September. You can imagine what happens if by the 29th of September, you identify a critical security vulnerability, and then the security team goes ahead and tells the senior leadership team that they cannot do—this product cannot go live with this critical security vulnerability. What do you do? Do you sacrifice your profits because of security?

01:48 — Jamie Parker
What would you do? The knee-jerk reaction may be—to say—delay the project and secure your systems. But there's a lot that goes into that decision. How likely is a breach? How much leeway do you have with the deadline? Are there other solutions available? It's not an easy decision to make.

02:04 — Joylynn Kirui
We are still seeing 52% of companies actually still sacrifice cybersecurity for speed.

02:10 — Jamie Parker
That's a lot. It's higher than it probably should be. Joylynn Kirui is a Senior Cloud Security Advocate at Microsoft, and she's trying to turn that situation around. But let's flip that statistic. 48% of companies are prioritizing cybersecurity over speed. Given the frequency and severity of security breaches that reach the news, that's encouraging. While Joylynn is currently on the forefront of the effort to improve cybersecurity, that wasn't always the case.

02:42 — Joylynn Kirui
I actually got hacked. Prior to getting into cybersecurity I was a software developer. I used to develop several systems for some companies here in Kenya, and then actually, at that time, like very many developers, I wasn't considering security at all when I was developing.

03:03 — Jamie Parker
That was a turning point in Joylynn's career. She took it upon herself to learn about information security, or InfoSec, and incorporated into her own coding, eventually turning to security as her full-time focus. But one shudders to think of the extent to which security is not considered.

03:22 — Joylynn Kirui
44% of developers are still not yet trained to code security.

03:27 — Jamie Parker
Oh. Yeah, that's not great. It's not an easy problem to solve. Joylynn shared with us a common refrain in the InfoSec community. Security experts need to be lucky all the time. Hackers only need to get lucky once. Recruiting developers to help improve system security from the outset seems like a simple improvement, but reality is rarely that simple.

03:52 — Joylynn Kirui
You see, yes, you still get the pushback from developers because they're like, "Why should they care? We are paid to develop. Why are you adding more tasks to them?"

04:02 — Jamie Parker
Training takes time. Developers may feel like they already have enough to do to get their thing to work at all, without also having to consider how someone else is going to try and break the system and bring the whole thing crashing down. For a long time, and for too many organizations, the relationships between developers, operations, and security teams have been contentious.

04:26 — Joylynn Kirui
Traditionally, during when we're having waterfall teams or agile teams, what should happen is, the developers would create a minimum viable product, and then after they develop that, it goes to the security team. And then the security team does security reviews, generates a whole report, and then sends this back to the developers. There used to be a lot of back and forth between developers and security teams until all the security vulnerabilities are fixed.

04:55 — Jamie Parker
That back and forth takes time. Depending on the company's processes, and the severity of the issues discovered, that time going back and forth can cause frustration, resentment, and delays. If only there was a way to avoid the angry game of ping-pong.

05:14 — Joylynn Kirui
By implementing security automation within the developer's environment—so if we can identify 80% of those security vulnerabilities from the developer's IDE, we'll be narrowing down or reducing the time to delivery by reducing the back and forth between the developer team and the security team.

05:34 — Jamie Parker
Automation can help catch a lot of the easy issues early on. That way, developers can fix them quickly and on their own. It's a little more work up front, for a lot of time saved and frustration avoided later on. Joylynn and security teams in general aren't there to maliciously slow people down. It's quite the opposite. They're there to deter malice, but they can't do it alone.

06:00 — Joylynn Kirui
It's usually a lot of negotiation between the teams, like what are the safeguards can we put in place to help support the business? Because in my opinion, security is there to support the business, and we shouldn't be blockers in that way. We should see ways of being supportive. But if there are no ways of safeguarding, then yes, we'd have to delay the delivery of the product. But yes, so we're still seeing 57% of operations teams are still pushing back on security best practices.

06:36 — Jamie Parker
By now, most people have heard of DevOps, the merging of developer and operations teams and processes, breaking down the silos, promoting communication and cooperation, and the end result is everyone is able to do their jobs better. Wait. Everyone? No, not everyone. Not yet.

06:57 — Joylynn Kirui
You're always developing within a period of time, pushing to the market, and then always building on top of that product, and pushing a particular solution. That's a form of automation. But yes, we noticed that, yes, we are pushing many solutions very fast, but you are also pushing security incidents and vulnerabilities very fast, as well. That's why, it is very important for what we call DevSecOps, or shifting security left to come in place.

07:27 — Jamie Parker
Enter DevSecOps, the movement to include security earlier on in the development process, and build on the success of DevOps processes. In a project's timeline, this is shifting security to the left. Automation, like security plugins included in IDEs, plays a big role in making DevSecOps possible. Automation also helps security teams monitor production environments, flag any suspicious activity, and quickly coordinate with developers and operations team on addressing any issues, in theory anyways.

08:01 — Joylynn Kirui
It's not all benefits, for sure. We're still seeing teams still working in silos. The development team, they have their own agenda. They want to push as many features as possible, and then the security team is telling them that all these features need to be reviewed for security vulnerabilities. Yes, the whole aspect of DevSecOps is just so unique, it's just more of a culture change.

08:30 — Jamie Parker
Automations to make work easier and faster only go so far on their own. It's difficult to overcome entrenched culture, expectations, and goals, and that's not to say automation isn't a huge help to security teams. It makes penetration testing, for one, much more efficient.

08:47 — Joylynn Kirui
In the operations side, we try to automate as much as possible, because it might take many, many, many, many days to try systems, to identify the vulnerabilities. We also have automation to see if it identifies a particular vulnerability, it goes further to try and exploit it. If I do it manually, it'll take me a couple of days. But if we have some bit of automation, it may take a couple of hours.

09:18 — Jamie Parker
Going from days to hours to complete these tests is no small improvement. Penetration testing is mimicking what an intruder might attempt, and they likely have access to their own automation systems. The faster you can find those vulnerabilities, the better chance you have of addressing them, before someone else takes advantage.

09:37 — Joylynn Kirui
By using programmatic solutions, you can be able to identify the stress for our organization's security question, and it also helps in performing triading of these security attacks, and setting also a priority level on those security attacks. And then in some cases, it also responds to them in return, so they have visibility of all these logs in one place. Even now, they can be able to do incidents response, incidents handling, as well, to see what is the extent of damage of a particular attack towards your environment, towards the ecosystem.

10:16 — Jamie Parker
When we come back, we dig into the balancing act that security teams need to perform, when rolling out automations to help improve security. Remember that statistic from the start of the show? 52% of companies are sacrificing security for speed. Delaying the project may not be a viable option for many, but leaving your systems completely vulnerable to a critical security vulnerability isn't really either, even if that vulnerability might never be exploited, if you're lucky. There are thankfully other options available.

10:53 — Joylynn Kirui
There's many ways around that, as well. In some cases, you would think of ways of mitigating. By mitigating I don't mean fixing the security vulnerability at once. It's just more of putting more layers of security around that, in terms of blocking it from being exploited. By blocking it from being exploited, it can also be from the defensive side of security.

11:19 — Jamie Parker
Those layers of security, especially for the increasingly distributed cloud environments, rely on a host of available automations. As Joylynn mentioned earlier, those include IDE plugins that scan the code as it's being written.

11:33 — Joylynn Kirui
You can be able to even go in and just search for security, and then you'll see all the security plugins that are available for you to just install, and then it scans against your codebase for security vulnerabilities. And then we also have the pre-commit hook. It prevents you from committing your code. For example, if it detects that you have secrets within your code, if it identifies that you have security vulnerabilities as well, it'll prevent you from merging, or pushing your code, or commission your code until you fixed all those security vulnerabilities.

12:10 — Jamie Parker
Here's where things can get a little dicey. The goal of these automations is to address potential security issues early and avoid that angry game of ping-pong. When it works as intended, everyone is happy. But if those plugins are too aggressive or not tuned properly, they can end up creating problems that weren't there. One such problem is the possible prevalence of false positives.

12:33 — Joylynn Kirui
I've seen scenarios whereby they deployed a particular tool, it had very many false positives, so they were taking up a lot more time trying to fix those false positives than the time it took for them to actually build the code itself. So you can imagine, that's just wastage of resources.

12:55 — Jamie Parker
Imagine you're working on your code. You're getting ready to send it off to get deployed, and you get 100 errors indicating potential vulnerabilities. As you're going through the list, you realize about 30 of them aren't actually a problem, because the issue has already been addressed, and that's not a great experience.

13:14 — Joylynn Kirui
Because we're supposed to be there to make it easier for developers, we need to make sure that the tools that we're deploying to developers are geared to their needs, and they are not stopping their developer velocity as well.

13:33 — Jamie Parker
False positives are unfortunately fairly common, especially when first integrating security automations into your work processes. Tuning those systems is an expected process. Over time, their prevalence should decrease. Just be careful of overtuning and making it too lax, or you could risk allowing false negatives through.

13:53 — Joylynn Kirui
Security absolutely prefers no false negatives and more false positives.

13:59 — Jamie Parker
With false positives, your system flags an issue that's already been addressed. With false negatives, your system fails to flag an issue altogether. It may be addressed, but it may not be. If it's not and the code goes to production, you're vulnerable, and you don't know it. That's the scary scenario. How does that happen?

14:20 — Joylynn Kirui
It's probably something that went wrong with the automation, or the way the tool was automated, because remember, this one is trying to get all the security vulnerabilities by running particular rules, or by running particular—You have a particular database of security vulnerabilities, and then it's running against your codebase to see if it will actually match that behavior. In some cases, if a tool is not mature and not tested as much, then yes, you'll see a lot of false negatives as well as false positives.

14:56 — Jamie Parker
Too many false positives and your developers may perhaps rightfully revolt. Any false negatives and some hacker could get lucky and break through. It's not a situation with quick and easy solutions. Finding that balance will take time. Tuning your security automations is an expected part of the process. Off-the-shelf solutions are a template to adapt to your own system. First step, determine if it's a good fit in the first place. Do you need the features included? Is the system worth taking the time to adapt to yours?

15:32 — Joylynn Kirui
That's why, it's very important. Before you procure, or you decide on a particular product, it's very important for you to do a POC, a proof of concept, just for you to understand if it meets your requirements, if it has false positives, if it has false negatives, by running it against your repositories just to understand the behavior, and also reading what are people saying about a particular tool. It's very important for you to do your due diligence.

16:02 — Jamie Parker
Once you determine if it's the right tool for the job, you can start on the process of integrating it into your workflows.

16:09 — Joylynn Kirui
The security and development team, they have to do what you call DevSecOps maturity. In some cases, you get a tool off-the-shelf like that, and then you integrate within your environment and just use it as is. In these cases, you'll find a lot of false positives.

16:28 — Jamie Parker
How do we get those numbers down?

16:30 — Joylynn Kirui
You have to take some time to understand the behavior of the product, and tweak the scripts for it to work within your environment to reduce the number of false positives. Yes, you have to do a lot of research and go through the maturity to make sure that your adoption is well-matured.

16:50 — Jamie Parker
Got it. Research, understand, adjust, repeat. This is such a common process that there's a formal framework that's been developed to determine how mature your DevSecOps automation system is.

17:03 — Joylynn Kirui
Within DevSecOps like Open Source Foundation for Application Security (OWASP) have from, of course, the DevSecOps maturity model, it categorizes your DevSecOps maturity from level one to level four. By the time you reach DevSecOps maturity level four, you have very few false positives. The tool is very mature, and it's working within your environment. The scripts are running very well. Yes, you have to take time just to do your automation very well and making sure that it's meeting your requirements.

17:33 — Jamie Parker
We talk a lot on this show about how automation, at its most effective, is an ongoing process rather than one, or a collection of one-off projects. That's especially true in security. Even when you hit that high-maturity level, when the DevSecOps team is synergizing and firing on all cylinders, you can't rest on your laurels.

17:53 — Joylynn Kirui
Even if you're putting all these security measures, for sure, there's nothing that's 100% secure. If someone wants to get in, they will get in. It's just a matter of time.

18:04 — Jamie Parker
You'll never be 100% secure. But with a little help from automation, you can even the odds and avoid becoming an easy target. Next time on Code Comments, we hear from an automation expert at Discover who help them consolidate their automation efforts, all the while navigating internal politics and the constraints of operating in a financial institution. You can read more at redhat.com/code-comments-podcast, or visit redhat.com, to find out more about our automation solutions. Many thanks to Joylynn Kirui for being our guest, and thank you for joining us.

18:46 — Jamie Parker
This episode was produced by Johan Philippine, Kim Huang, Caroline Creaghead, and Brent Simoneaux. Our audio engineer is Elizabeth Hart. The audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Carrie DaSilva, Mira Cyril, Ocean Matthews, Paige Stroud, Alex Traboulsi, and Victoria Lawton. This has been Code Comments, an original podcast from Red Hat.