Baking Security into Your Software Delivery Pipeline

Secured software delivery lifecycle (SDLC) is a C-level directive recently gaining steam. This is partially in response to Executive Order 14028, which aims to improve our nation’s cybersecurity practices. As an end-to-end software delivery platform provider, CloudBees seeks to empower developers to create great code to drive value for their end users while providing the management tools to ensure a streamlined, secure, and governed process. In this blog, based on a DevOps World 2022 session, we will dive into what it means to secure the SDLC and provide practical guidance organizations can take to ensure they take the appropriate preventative and remediation measures.

A secure software delivery lifecycle (SDLC) means integrating security activities and considerations into every stage of the software development process. The goal is to ensure that software is developed securely from the beginning, reducing the risk of vulnerabilities and minimizing their impact if discovered.

As more companies migrate to the cloud and adopt cloud-native development, developers become more responsible for the entire stack. This “shift-left” approach created a false reality for which management felt they solved the problem and delivered quality code. In many cases, things have gotten worse. Nearly 60% of companies use shift-left state in their development teams.

The solution requires more of a collective effort where we just don't focus on the tools but also on people and processes. Security must be incorporated throughout the entire software lifecycle, from ideation, development, delivery, and release to production. And cannot ignore post-production.

A release pipeline encompasses all the automated and manual steps that ensure users can access a reliable, secure version of a company’s software. The release pipeline is uniquely positioned to be the enabler of security end to end as it converges on three critical components of security: visibility, traceability, and compliance. 

• Visibility: The pipeline can be a centralized location for visibility by providing insight into your automated and manual process. It is essential to have visibility and awareness of all the tasks in the release process. Thus, visibility is the first step in baking security into your pipeline because you must first understand your processes before measuring and improving them. 

• Traceability: The pipeline helps log evidence across all tasks from ideation to production, making it seamless to generate audit reports. You want to ensure that at any point, you can audit your process and know precisely where defects are in the software delivery lifecycle, how they got there, what testing and security scans were done, and which developers were responsible for that code. And if there are any JIRA stories of what they were, too.

• Compliance: The pipeline helps enforce process compliance by proving that agreed-upon tasks performed as expected. Through this governance, you’re able to appease auditors on demand. 

Now, some teams have attempted to extend the CI build solution to construct monolithic pipelines, a single pipeline that does it all: dev, test, security, and operations. It's a grand effort but often results in frustration. You have many stakeholders: developers, QA engineers, release managers, and security, all trying to provide updates and enhancements to this pipeline. Each update can result in errors, affecting your data delivery and ultimately pin the responsibility on the development teams for not making the deadline. Since everything is sequential, if one step fails, the entire pipeline fails. The result is brittle pipelines.

To solve this issue, redefine your process and build pipelines that run in parallel. This means instead of having a single pipeline do everything, you can build sub-pipelines that run in parallel, handling single functions like security, compliance, or performance. Like other pipeline-related tasks, this must be automated and streamlined to ensure it doesn’t become a bottleneck. Then give your security, compliance, SRE, and whoever else to manage that pipeline.

In this section, we dive into some preventative measures you can take to ensure a secure release pipeline. 

Gating is elemental in securing your pipeline. But providing gating alone is not enough: you must ensure the gates work, as there’s no point in having gates if people can circumvent them. The following guidance can help ensure a successful implementation:

• When you begin to redesign your release process, make sure to include intelligence and end-to-end transparency. 

Your pipeline should be data-driven but also data-informed. An example would be to use “smart gates” that check the results to make a go or no-go decision and determine how those results were collected.  There's no point in looking at the results of a security scan if the scan itself was misconfigured and didn't meet the requirements.

A compromised pipeline can do all sorts of damage not only to your development but also to your overall business and brand. When designing your pipeline, securing your application is part of the solution. But securing access to your pipeline is instrumental.

You should go beyond RBAC to employ fine-grained access to your pipelines. Access can be granted down to the task level in that separation of duty. For example, you have a pipeline with gates that map to SLAs or compliance requirements. In other words, separate your pipeline definition, the configuration, and input data logs generated by every step, every integration, and every environment should be saved in a central location. This makes debugging and triaging much easier. 

The following guidance should be followed when securing access to your pipeline: 

• The pipeline should drive change control systems, such as ServiceNow. This way, the pipeline can create, update, and respond to tickets in ServiceNow. 

• The pipeline should be leveraged to limit the blast radius.

• The pipeline should be used as a single guarded access point to servers, tools, and other endpoints. If someone wants to run a script, run it through the pipeline. This way, it gets tracked and always gets audited.

Once the application gets deployed into production, monitoring its overall security is imperative.

A critical component of automatic remediation / automated rollback is artifact dependency tracking, which helps manage artifact dependencies when an issue is discovered with an artifact or a library. The artifact dependency tracking capabilities can quickly identify all applications, components, and environments which use a faulty version and, with a single click, can remediate them all. This only works if you're not tracking dependencies in spreadsheets, but your software delivery platform has that capability built in. 

It’s critical to stress the importance of having the ability to generate real-time audit reports on demand. This requires access to real-time evidence that is dynamic and granular. 

An audit trail will allow you to automatically create and assign tickets to development teams responsible for the bug, the security vulnerability, a complete audit trail of access to the audit logs, the results, and all the team members involved. The audit trail can pinpoint the problem area and help refine your process if there's a recurring issue.

The release pipeline is the meta orchestrator integrating and orchestrating through your entire toolchain to move artifacts across value streams. If done correctly, it should also push data to the analytics server allowing teams to build role-based dashboards with real-time data and offering trend analysis. 

Having access to real-time data is very important when dealing with security, as you want to avoid looking at stale data.  When done correctly, Platform Services teams should not have to context switch across multiple tools to determine what went wrong: the dashboards should just tell them!

Security is an ongoing concern, which should result in pipelines being viewed like a product: continue to improve it, add new features, maintain a backlog, test it, and open issues against it. 

Once you identify common patterns, you can begin to standardize the pipeline and value stream to make workflows reusable. This makes onboarding new applications and functionality much more manageable while reducing snowflake processes. 

Self-service processes should be created where all stakeholders (Developers, QA, Release Managers, etc.) can access better processes that comply with organizational policies. This way, everyone uses the same type of pipeline with predefined, mandatory tasks and gates with separation of duty and security embedded within. 

When shifting security, the solution isn't throwing more tools; it requires changes in culture and process. You don't want to build complex pipelines that break the separation of duty, disrupt developer experience, and become a management nightmare. Separating pipelines into essential functions and having them run parallel to your Release Pipelines with smart gates and guardrails helps ensure security and compliance throughout your software lifecycle. 

Integrations are easy, but you need to build intelligent workflows and dashboards capable of taking action when specific results are seen, or thresholds have passed, eventually leading to quicker resolution and avoiding exposure that can negatively affect your business. Ultimately, your goal is to have real-time end-to-end visibility throughout your entire value stream.

To watch this DevOps World session in its entirety, click here. For more on-demand DevOps World sessions, visit devopsworld.com.