Companies are discovering the drawbacks of DevSecOps metrics

Measuring developer performance in securing code is an important part of getting started with DevSecOps, but too much focus on developer metrics can make it harder to mitigate business risk, according to business professionals.

DevSecOps refers to the practice of integrating application security into the early stages of the software development process, in which developers play a critical role. This practice has become common among companies affected by a growing wave of high-profile cyberattacks and security vulnerabilities, especially in the past two years. Given the pace of software releases demanded by Agile and DevOps and the scale of cloud-native distributed computing, modern applications require security to be built in from the start to have any hope of staying ahead of the game. threats, say many experts.

As companies like Komatsu Australia, a New South Wales-based manufacturing company, make this transition to DevSecOps, they rely on metrics like Mean Time To Detection (MTTD) and Mean Time To Resolution. (MTTR) vulnerabilities and security incidents. These metrics can help establish a foundation for the Agile practice of continuous improvement in the early stages of DevSecOps.

Eric Cheng

“We obviously have to start with a baseline and see where we are now, and then what makes sense for us to improve,” said Eric Cheng, digital architect at Komatsu. “The metrics will tell you, ‘Are you going in the right direction?'”

This benchmark will incorporate the DevSecOps Maturity Model from the standards organization Open Web Application Security Project (OWASP), which establishes a three-level process for adopting DevSecOps practices in several areas of application development and IT operations.

“It’s a well-known body,” Cheng said of OWASP. “For us, where we are in terms of travel, I think that’s a good place to start.”

For specific developer feedback, Komatsu uses static application security testing and software composition analysis dashboards from cybersecurity firm Snyk. In addition to MTTD and MTTR, Cheng shows developer trends in a Snyk metric called exposure window for each application, a measure of the total time elapsed between when an issue was detected and when it was resolved.

“We look at, ‘OK, you have some issues that are starting to show up in terms of the exposure window,’ or ‘You fixed those issues in a short time, keep up the great work,'” Cheng said.

It’s not the only tool in Komatsu’s DevSecOps toolkit – developer training, in part through Snyk Professional Services; ongoing collaboration with the company’s cybersecurity managers in the application design phase; and security automation in CI/CD pipelines are also important components, Cheng said. But the metrics and benchmarks will serve as a general guide for a multi-year endeavor to shift security to the left at Komatsu.

“It led to the bad behavior”

For teams further along the path to maturity, DevSecOps metrics initially served a similar purpose. But ultimately, IT organizations need to look beyond metrics-driven feedback to motivate developers to be security-conscious.

The limits to the effectiveness of DevSecOps metrics as an incentive for developers were reflected in the results of a May 2022 DevSecOps survey of 5,000 IT decision makers conducted by DevOps platform provider GitLab. Security was considered a performance metric for developers by 57% of organizations surveyed, but 56% said it was still difficult to get developers to prioritize fixing code vulnerabilities, and 59% said Application vulnerabilities were still the most likely to be found by security teams in production.

For retailer Target Corp., the early focus on DevSecOps metrics eventually led to unintended consequences, said Jennifer Czaplewski, senior director of cybersecurity, during a presentation at a DevOps Login co-located event with this year’s RSA conference.

Target calculates a product intelligence (PI) score for each of its 7,000 apps. This score takes into account DevSecOps metrics such as the percentage of endpoint vulnerabilities resolved according to company MTTR policies, developer use of internal security services, and whether an application or service was the cause of a breach. a security event.

In 2019, Target’s first DevSecOps dashboard showed whether each app’s PI score was in the top 10% of Target’s apps and services — or the bottom 20%.

We want to make sure that we don’t lead the teams to perfection. We’re not looking for perfect scores, we’re looking for good safety health.

Jennifer CzaplewskiSenior Director of Cybersecurity, Target

“It sparked a lot of friendly competition, but our teams got so good at understanding how to secure their apps that over 10% of all teams had a perfect score, so if you had a perfect score, you still couldn’t reach the top 10%,” Czaplewski said in his presentation. “People were frustrated, and that led to the bad behavior.”

Teams are now asked to achieve a minimum PI score, but benchmarking information is no longer part of Target’s DevSecOps interface, Czaplewski said.

“We want to make sure that we don’t lead the teams to perfection,” Czaplewski said. “We’re not looking for perfect scores, we’re looking for healthy safety.”

From DevSecOps Metrics to a Secure Culture

Target still expects software engineers to be security-aware, but has begun to cultivate what it calls Security Ninjas, developers who have become security experts embedded within software development teams. ‘apps. These experts, who represent about 5% of Target’s 5,000 developers, act as translators between security and developers. Security ninjas are also responsible for building two or three threat models each year to create a strategic view of business risk, rather than focusing on specific DevSecOps metrics.

“We’ve learned over the last three or four years that we really want to focus on what we should be doing to create a safe culture within a team, not what you need to know who, hopefully, will then turn into something,” Czaplewski said in his presentation. “If you’re a security ninja, the first responsibility is to have at least some basic security knowledge…but also to take steps to identify and resolve security issues.”

Ideally, a DevSecOps transformation should be accompanied by a culture of continuous learning that helps IT professionals identify and prioritize strategic risks, even amid rapid changes in cloud-native application security, a said Robert Slaughter, CEO of computer defense firm Defense Unicorns.

Measurements can be helpful, but they can make it easier to lose focus, Slaughter said.

“What we emphasize are daily habits and we focus on results,” he said. “Asking, ‘Are you getting more and more sure?’ and instilling that in the individual becomes a much better way to get the right outcome than any specific measure that may be hidden, obscured, or swept under a rug.”

A secure culture is ideal for Komatsu’s Cheng, but it will take time.

“Commercial teams and the developers they work with — and they might be vendors — have their own priorities,” he said. “They have this app that they have to develop, and they have to release it by a certain date, and they’ve had that mindset and that way of working for a long time.”

That’s starting to change, Cheng said, starting with product teams adopting tools like Snyk and asking the vendors they work with to also apply security scans to their products.

In the meantime, amid a widespread shortage of developers and cybersecurity skills, many organizations have begun using IT automation through self-service DevOps platforms to harden application security without slowing down developers. Komatsu’s use of Snyk’s automated testing tools is part of such an effort to automate DevSecOps workflows on behalf of developers.

Educating developers on how security awareness can save them time patching vulnerabilities in production is an important first step in the DevSecOps process, but “the second part is to make it as seamless and painless as possible,” Cheng said.

“It just becomes part of the development environment, whether it’s Eclipse or Visual Studio, and we can also integrate [security] in our CI/CD,” he said. “For them, it’s now just another tool that they’ve integrated.

Beth Pariseau, Senior Writer for TechTarget Editorial, is an award-winning veteran of computer journalism. She can be reached at [email protected] or on Twitter @PariseauTT.

Margie D. Carlisle