The software delivery pipeline is the process of taking features from developers and getting them delivered to customers. The earliest tests should be the quickest and easiest to run, giving developers the fastest feedback. Successive rounds of testing should increase confidence that the code is a viable candidate for production and that more expensive tests—be it time, effort, cost—are justified. Manual testing should be performed toward the end of the pipeline, leaving computers to do as much work as possible before people get involved. Although it is tempting to arrange the delivery pipeline in phases (e.g., functional tests, then acceptance tests, then load and performance tests, then security tests), this can lead to problems progressing down the pipeline.

In this interactive workshop, we will discuss how to arrange your pipeline, automated or not, and so each round of tests provides just enough testing to give you confidence that the next set of tests is worth the investment. We'll explore how to get the right types of testing into your pipeline at the right points so that you can determine which builds are viable candidates for production.

DevOps needs to consider many different aspects of software quality, including security. The term DevSecOps was developed to highlight that security is a focus of the pipeline, not a second-class citizen.

Fortunately, we can define done for our pipeline so that it includes security. Continuous integration can invoke static analysis tools to test for security errors and check if we are using components with known vulnerabilities. Automated deployments and virtualization make dynamic environments available for testing in a production-like setting. Regression tests can drive traffic through proxies for security analysis. From the code to the systems where we deploy the software, the process can be designed to make sure that we follow security best practices, and not produce insecure software.

Participants will learn how to construct a definition of done that focuses on security in a DevOps pipeline. They will see how to define security practices that build confidence that they are doing DevSecOps, and how those practices and criteria might mature over time.

I often suggest to teams that they should be using all sorts of tools in their pipelines- from simple static analysis checks and automated builds to security scans and performance testing. I've done presentations and talks at conferences. I've lobbied to clients. I've commiserated with my colleagues. But I've never put together my dream pipeline in one of my own projects.

There are always reasons that some tests and tools get left out- our policies won't allow them, they will take too long to get approved, we don't have time, we have bigger problems to deal with, it just isn't what the client is looking for right now. And I usually think, if only I were in charge, I'd make sure we were using those...

In late 2017 I took over maintenance on an open-source project. Now I have no restrictions. The sky's the limit. No one is around to tell me what I can't do. So why don't I have my dream pipeline in place yet?

I'll talk about the trade-offs and compromises I made when building out the pipeline. Why I decided to focus on some tools and tests but skipped others, and what I need to do or change to make this delivery process the pipeline I've always dreamed about, now that I have no one else to blame.

Almost everyone has to deal with bad legacy code at some point. Not just legacy code that you inherited and obviously would have been better if you had written it, but legacy code so ugly and ill-conceived that it makes you want to hunt down the person responsible just so you can scream at them (or worse). And then replace it with a one-line library function that does the same thing.

We'll show some examples of the worst code I've seen, and we'll have a chuckle or a groan. The names, projects, and check-in comments have been changed to protect the guilty, but, unfortunately, these examples are all too real.

Not every continuous delivery initiative starts with someone saying "drop everything. Let's do DevOps." Sometimes you have grow your practice incrementally. And sometimes, you don’t set out to grow a practice at all-- you are just fixing problems with your process, trying to make things better.

I'll walk through a case study of how our team worked on an exemplar project for the Department of Defense to show that agile could work in a decidedly waterfall culture. I’ll also discuss techniques and tools we used to bring a DevOps mindset and continuous delivery practices into an environment that wasn't already Agile.

I'll talk about how we were able to start in development, where we had the most control, with a "let's starting being Agile" initiative and working on "why is continuous integration important?" From there, we tackled one problem after another, each time making the release a little easier and a little less risky. We incrementally brought our practices through other environments until the project was confidently delivering working, QA-tested, security-tested releases that were ready for production every two weeks. I’ll discuss the journey we took and the tools we used to get to build quality into our product, our releases, and our release process.

Developing a delivery pipeline means more than just adding automated deploys to the development cycle. To be successful, tests of all types must be incorporated throughout the process in order to be sure that problems aren’t slipping through. Most pipelines include unit tests, functional tests, and acceptance tests, but those aren’t always enough. I’ll present some types of testing you might not have considered, or at least might not have considered the importance of. Some types will address code quality, others code security, and some the health and security of the pipeline itself.

I’ll talk about specific tools we used to supplement our pipeline testing. I won’t get into how to use each tool-- this is more of a series of teasers to encourage people to look into the tools, and even letting them know what types of tools and testing opportunities are out there.

W. Edwards Deming noted that “people with targets and jobs dependent upon meeting them will probably meet the targets – even if they have to destroy the enterprise to do it.” While metrics can be a great tool for evaluating performance and software quality, becoming beholden to reaching metrics goals, especially the wrong ones, can be detrimental to the project. Each team needs to take care and understand what targets are appropriate for their project. They also need to consider the current and desired states of the source code and product and the capabilities and constraints of the team.

As one of the lead architects working with a huge codebase on a government project, I often have the opportunity to influence the teams around me into watching or ignoring various metrics. I will walk through some measures that are available to most projects and discuss what they really mean, various misconceptions about their meaning, the tools that can be used to collect them, and how you can use them to help your team. I’ll discuss experiences and lessons learned (often the hard way) about using the wrong metrics and the damage they can do.

Security is a lot like quality and performance. You can’t tack them on at the end of the development cycle and expect it to be effective. All of them have to be built in along the way, in every phase of every cycle. But even though many companies claim that security is a priority, that doesn’t always translate to supporting security initiatives in the software development process. Security code reviews are often overlooked or avoided, and when development schedules fall behind security testing is often dropped to help the team "catch up". And there is almost never any money in the budget for buying new tools.

So the first step of building secure applications has to be making security part of the regular development process, but at the same time there isn't time or budget to do so. Developers have to get some quick, easy wins with security without expending a lot of time, money, or effort. Like any agile practice, continuous and rapid feedback is critical.

Static analysis tools look at source code or compiled code, looking for common errors, unused variables, style and formatting variations, and similar items. Most modern languages have a selection of open-source static analysis tools that can scan source code looking for predictable problems. SQL injection, cross-site scripting, and hard-coded passwords are common vulnerabilities that can often be detected by static code analysis. Static analysis tools can also look at the third-party libraries that the source code depends on, identifying components that have known vulnerabilities.  

With a continuous integration practice, it is easy to run some of these static code analysis frequently, beginning early in the development cycle. We can make these tools part of the developer's normal routine, heading off potential security problems before it is too late. 

Gene talks about his experiences with using open-source static analysis tools to build security into the development process without spending much time or effort, but still adding plenty of security value.