Measuring Success in DevSecOps: What Really Counts?

Welcome to ASSYST OnPoint xChange, where we bring insightful discussions on the technologies shaping government IT. Today, Padmaraj Viswanathan (Raj), Solutions Architect at ASSYST, joins us to discuss how to measure success in DevSecOps. Raj shares practical examples, tools, and metrics that help organizations achieve speed, security, and reliability in their workflows. He also manages ASSYST's Green Accelerator Program, which accelerates solution launches at the speed of business. Leading the conversation is Erin Slezak (Erin), Business Analyst at ASSYST, diving into the challenges and opportunities driving the next phase of DevSecOps.

Erin: Thank you for taking the time to speak with me, Raj. DevSecOps has transformed how teams deliver software, but success can be tricky to quantify. What do you think are the most meaningful metrics for measuring success?

Raj: That’s a great place to start, Erin. When I think about success in DevSecOps, it’s about balancing three things: speed, security, and reliability. Deployment frequency is a major indicator—it tells you how often teams deliver value. I worked with a team that used Tekton for CI and ArgoCD for automated deployments, moving from monthly to weekly deployments. That shift improved delivery times and built team confidence in the process.

Lead time for changes is another critical metric. For example, by optimizing their CI/CD pipelines with Tekton and introducing automated testing, the ASSYST team reduced lead time from two weeks to under three days. Automating repetitive tasks allowed them to focus on delivering features faster and with fewer errors.

Erin: Those sound like solid metrics, but what about when things go wrong? How do you measure and improve resilience?

Raj: Resilience is about how quickly you recover when something goes off track. Mean Time to Recovery (MTTR) is a key metric, and our team implemented automated rollback mechanisms using ArgoCD. For example, ArgoCD’s health checks detected deployment issues and triggered rollbacks automatically, reducing MTTR from eight hours to just over an hour.

Another useful metric is the change failure rate, which tracks how often deployments lead to issues. For one client, integrating chaos engineering tools alongside ArgoCD allowed them to test failure scenarios proactively. This helped identify weak points in their infrastructure and significantly reduced the failure rate.

Erin: Security is a cornerstone of DevSecOps. How do you measure its effectiveness without slowing teams down?

Raj: Automation plays a vital role here, and having a cohesive toolchain is essential for integrating security into every phase of the pipeline. Time to detect and remediate vulnerabilities is a crucial metric. Our team integrated static application security testing (SAST) into their Tekton CI pipeline. Vulnerabilities were flagged immediately after code commits, giving developers near-instant feedback. This not only improved security but also streamlined workflows by leveraging the toolchain's seamless integration.

Another important area is Infrastructure-as-Code (IaC) security. Using tools like OPA/Gatekeeper, Kubernetes configurations were continuously scanned for misconfigurations. For instance, before deployment, an improperly configured S3 bucket was identified in one project. The bucket had public-read permissions enabled, which could have exposed sensitive data. Gatekeeper flagged the issue during a pre-deployment scan, allowing the team to correct it without impacting production.

Similarly, OPA policies helped enforce standards for secure port configurations in Kubernetes pods. For example, open ports were automatically flagged as non-compliant, ensuring that only explicitly allowed ports were exposed. These proactive scans dramatically reduced the risk of security breaches, and metrics like "number of misconfigurations detected and resolved per pipeline" highlighted the effectiveness of these policies.

Erin: Automation sounds like a game-changer. What advice would you give to teams starting to automate or planning to scale their DevSecOps workflows?

Raj: Start small and focus on the areas with the biggest bottlenecks. We began automating the testing workflows using Tekton in one of our product development workstreams. Then, we applied it to a single application, which led to immediate results: fewer bugs in production and faster release cycles.

As confidence grew, our team expanded automation to include code scans and AI-driven anomaly detection for monitoring. For example, integrating AI tools into their Prometheus and Grafana observability stack provided actionable alerts and reduced the mean time to detect issues. The key is to build incrementally—automation should evolve naturally with your workflows.

Erin: Raj, observability is becoming a key focus in modern IT systems, and there’s growing talk about its connection to customer experience (CX). How does observability fit into DevSecOps, and is it meaningful for systems like citizen services?

Raj: Absolutely, Erin. Observability is essential for public-facing systems like citizen services, where reliability directly impacts user trust. Deploying Prometheus and Grafana in Kubernetes environments provides teams with end-to-end visibility into system health.

For instance, the ASSYST team worked with a regulatory submissions portal, where latency issues during peak hours were traced to a specific API through Prometheus metrics. With this insight, the team could prioritize and resolve the issue, ensuring users could complete transactions without disruptions. Additionally, by integrating observability with CX metrics, like transaction completion rates and feedback scores, teams shifted their focus from merely “keeping systems up” to “ensuring smooth user experiences.”

Erin: For teams managing large portfolios, how do you recommend tracking success across multiple teams and projects?

Raj: Consistency in metrics is key, but flexibility for team-specific needs is equally important. Metrics like deployment frequency and lead time for changes can be tracked across all teams, while individual teams might focus on their unique challenges, such as reducing MTTR for critical services.

Across the industry, organizations are using custom Grafana dashboards that pull data from Tekton and ArgoCD APIs. These dashboards provide leadership with a portfolio-wide view of metrics while allowing individual teams to drill down into their specific performance data. This visibility helps identify bottlenecks and promotes accountability.

Erin: Before we wrap up, what’s one piece of advice you’d give teams trying to measure and improve their DevSecOps practices?

Raj: Focus on progress, not perfection. Metrics are there to guide improvement, not to be the end goal themselves. Start by tracking key metrics, such as deployment frequency, lead time, and time to remediate vulnerabilities. Use those as a baseline and iterate incrementally. Remember, the ultimate goal is delivering secure, reliable value, not achieving perfect metrics.

Erin: Thanks, Raj. What you’re sharing has been incredibly insightful. It’s great to hear how practical metrics and real-world tools can drive meaningful improvements.

Raj: Thanks, Erin. It’s always exciting to see how teams evolve when they focus on the right things.

ASSYST Application Modernization and DevSecOps

ASSYST has consistently delivered exceptional DevSecOps capabilities across critical federal initiatives, including CMS’s Security Data Lake, EEOC’s Regulatory Application Modernization, HHS PSC Financial System Modernization, and DOE’s Application Modernization. Our approach to application modernization leverages DevSecOps to seamlessly integrate security, agility, and scalability into every phase of the application lifecycle. By embedding security practices directly into CI/CD pipelines and automating compliance checks, we ensure that modernization efforts align with evolving regulatory standards such as Zero Trust and CMMC.

This methodology enables agencies to transition from legacy systems to modern, cloud-native architectures while maintaining operational resilience and accelerating time-to-value for mission-critical applications. Through innovative tools and frameworks like the Green Accelerator, ASSYST facilitates the design, development, and deployment of secure, adaptive, and future-ready systems, empowering agencies to achieve their modernization and mission-critical goals.