Max will tell the story of how a healthcare company striving to get to continuous releases built up their automation to secure confidence in regular releases. Initially, as no test automation existed, Max was able to take a greenfield test automation opportunity, and in the span of 12 months, develop over 2000 test cases. A testing pipeline was created to verify the integrity of the automated test cases, and to build docker containers for simple execution of the tests. These containers could then be simply re-used by developers and the DevOps team to verify the application. Max will walk through the feedback loop created, which allowed verification of the application go from hours to minutes.

Max will discuss what processes and paths were taken to achieve continuous testing on this project. While he will cover the tools used and why they were chosen, the main focus will be on the HOW and WHY certain patterns and activities were performed. These choices were critical to achieving continuous testing, rather than just good testing coverage in CI or CD, even allowing a push left for performance and security. Additionally, some time will be spent on the organizational and culture changes that occured, and how he was able to accomplish this push for adoption in an organization that resisted automation, and had major quality problems.

 
8 favorite thumb_down thumb_up 4 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/Structure of the Case Study

  • What is continuous testing?
    • Continuous testing enablers
  • Case Study
    • Background
    • Developing automation framework
    • Building CI pipeline
    • Building test automation pipeline
    • Integrating test automation
    • Determining quality thresholds
    • Implementing quality gates
    • Setuping feedback loops

Learning Outcome

A case study looking at building an automated testing suite, and integrating into an evolving DevOps pipeline

Target Audience

DevOps and Test Practitioners

Prerequisites for Attendees

It's not a bad idea to understand the concepts of continuous testing, but they are briefly covered in topic

schedule Submitted 1 month ago

Public Feedback

comment Suggest improvements to the Speaker
  • George Dinwiddie
    By George Dinwiddie  ~  2 weeks ago
    reply Reply

    I like these aspects of the submission, and they should be retained:

    • continuous testing

    I think the submission could be improved by:

    • I'm not sure what is the focus. Is it on the particular tools (docker, CI, "framework," etc.) and the way they are used? What are the more general principles being highlighted?
    • Max Saperstone
      By Max Saperstone  ~  2 weeks ago
      reply Reply

      Thanks for the feedback George.

      Let me know if this helps some, and I'll modify the abstract some. This is essentially a case study about what processes and path was taken to achieve continuous on a project that initially had no automated testing done on it. I cover the tools we used and why, along with testing patterns, but a lot of the focus is on the HOW and WHY we did certain things, how we pushed for adoption in an organization that resisted automation, and had major quality problems.

      • George Dinwiddie
        By George Dinwiddie  ~  2 weeks ago
        reply Reply

        That helps a lot, Max. I like the fact that you generalize from the case study to the HOW and WHY.

        • Max Saperstone
          By Max Saperstone  ~  2 weeks ago
          reply Reply

          I made some updates, hopefully this helps capture the nature and goal of the talk better


  • Liked Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Building Confidence In Your Automated Tests

    45 Mins
    Keynote
    Beginner

    The growth of automation testing in today’s software development organizations is changing the the way we test applications. Software development practices have matured over the last 30 years, to include all forms of testing to verify software quality. In the last ten years, there has been a huge spike in the adoption of automated tests, effectively replacing some of these manual testing practices, and supplementing many traditional testing activities. Many parts of the software development industry, however, are wary of replacing manual testing with automated testing. Not only is there often a lack of confidence in the automation tests, many see automated testing as fragile, unmaintainable, and ultimately, something delivering a low return on investment. Max believes that by employing mature software development techniques, we can achieve robust, maintainable, tests, that deliver confidence of the application under test. In addition to discussing how to structure automated tests that are cleaner, more maintainable and efficient, developer testing, and deployment techniques can be used to programmatically verify test correctness. Drawing on his experiences building test automation, test frameworks and advising organizations to adopt test automation, Max will walk us through how to mature your test automation practices.

  • Liked Thomas Stiehm
    keyboard_arrow_down

    Thomas Stiehm - Continuous Build and other DevOps anti-patterns, and how to overcome them

    Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 3 weeks ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Continuous Build is an anti-pattern that I have often seen where a team will have what they call Continuous Integration (CI) in place but it only builds the code, there are no tests or static analysis run. Certainly, this is better not building but it leaves a lot of health check information on the table that is considered part of CI. Without this information, you can never really gain the confidence that your build is healthy. The whole goal of CI is to feel that your build is healthy so not tests and analysis means you aren’t going CI.

    Just like CI, other DevOps practices can be hard to understand, implement, and get right. Even with the best of intentions, we make mistakes or misinterpret the implementation of a technique. Learn how to spot common DevOps anti-patterns and how to correct them. These patterns include

    1. Continuous Build - CI without tests isn’t CI
    2. Turn the unit tests off to build the release
    3. Don’t automate that, it is my job
    4. Different build process for developers and high environments
    5. Different deployment process for developers, test environments and/or production
    6. Not having a production-like environment to test in before production
    7. Saving performance testing for the end of the release
    8. Saving security testing for the end of the release
    9. Never asking the users about the software
    10. Only automating build and deployment, not testing
    11. Not having retrospective
    12. Restricting retrospectives to only the development part of the process
    13. Running analysis and never looking at or acting on the findings
    14. Reduce coverage or static analysis gates to get a build to pass

    We have all experienced a time where we wanted to believe we could make an anti-pattern work but it never does. It is better to learn how to spot these and how to correct them than it is to try to keep tweaking a broken process hoping this time it will be better.

  • Liked Thomas Stiehm
    keyboard_arrow_down

    Thomas Stiehm - Nobody Cares about Security and What DevSecOps is doing about it

    Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 3 weeks ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Application security is the poster child for third-class citizens in the software development world (behind Quality Assurance). DevSecOps is trying to turn that around and get more people, teams, and companies to care about security as our online and real-world lives become more intertwined.

    Application security has a bad reputation with many people. It comes into the development process late and demands a lot. Who wants to deal with that? We have actual business value to get out the door. “Those things” won’t happen anyway. And when “those things” do happen, it will just become an exercise in finger-pointing and blame. Security is an ugly affair that no one wants a part of.

    DevSecOps is a movement within the DevOps and Security worlds to reverse this decades-long drama by getting the people creating and updating software to build security practices into their process from the beginning, even before the code is written. This allows security professionals to become the evangelist of security practices where they can help the teams adopt practices and teach them how to use the tools to resolve issues themselves. No longer dependent on the specialists the teams can address security findings as they are found and make the workload manageable by spreading it across their implementation cycles-- proactively, not reactively.

    Attendees will leave with an understanding of how to map security concepts onto a delivery pipeline, how to “sell” security concepts to stakeholders, and how automation makes it easier to gather security data and act upon it. Learn what is needed to get started with DevSecOps so that you can start creating secure software today.

  • Liked Thomas Stiehm
    keyboard_arrow_down

    Thomas Stiehm - Shifting Security Left - The Innovation of DevSecOps

    Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 3 weeks ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    DevSecOps uses application security practices that have been around for a while. The innovation of DevSecOps is incorporating security into the daily workflow of the team rather than leaving them to the end of a release like many legacy processes do. Shifting security left is made possible by the ability to automate many aspects of security testing and verification. DevSecOps leverages DevOps practices to make application security a first-class citizen in the practices of modern software product development. DevSecOps starts with a culture change mindset of cross-functional teams creating software through collaboration and fast feedback cycles.

    The security in DevSecOps starts before the code is written by using techniques like threat modeling and risk analysis to help figure out who might want to attack you and how they might do that. This often ignored security practice can be enabled by following the DevSecOps practices of having a cross-functional team involved in the process from the beginning, including security professionals.

    Next, DevSecOps maps application security practices into the build pipeline for a project in order to provide quick feedback about the security posture for any change made to the software. By using automation to allow the team to move quickly while maintaining confidence in the health of the code base, DevSecOps extends that health check to include application security checks. While automation can be used to make security data collection easier it is important to understand what security practices still require a human being.

    This talk focuses on how, when, and where practices should be incorporated into a build pipeline to get the most value out of your security practices through automation. It explores what manual security work still needs to be done by a person and how to maximize value while minimizing the effort of human beings.

  • Liked Glenn Buckholz
    keyboard_arrow_down

    Glenn Buckholz - Moving Your Pipeline to Kubernetes, Things I Wish People Had Told Me

    Glenn Buckholz
    Glenn Buckholz
    Technical Manager
    Coveros
    schedule 3 weeks ago
    Sold Out!
    45 Mins
    Case Study
    Intermediate

    Kubernetes married with a cloud provider elastic, highly available infrastructure. Many CI engines today (Jenkins, Bamboo, Gitlab, CircleCI), provide native integration with kubernetes so that your build and deploy workload can be elastically executed. This allows your pipeline to meet the needs of your schedule, be it the 4pm pile on to commit code before going home, the mad rush to get a hot fix to production, or the surge of an unexpected customer ask. Gone are the days of the build queue growing and you CI engine collapsing under the weight of a hundred build requests. In order for a pipeline to leverage this capacity changes must be made to the pipeline architecture. Tools must be dockerized, the ephemeral nature of running docker must be considered, kubernetes specifications or helm charts must be generated for the application, automated testing must be adapted to work in the new architecture, and then there is the database. Each one of these issues, plus many others I’ve missed contained unfortunate, unforeseen pitfalls that translated in schedule delays. Join Glenn as he helps you short circuit the pitfalls of migrating to kubernetes off of your static in-elastic virtual infrastructure.

  • Liked Joel Tosi
    keyboard_arrow_down

    Joel Tosi - Fostering the Third Way - Your DevOps Dojo

    Joel Tosi
    Joel Tosi
    Dojo & Co
    schedule 4 weeks ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    In the DevOps Handbook, Gene Kim introduces the Third Way - The Technical Practices of Continual Learning. Enter the DevOps Dojo - an immersive environment where whole teams come together to learn and practice their skills while solving real business problems.

    Joel Tosi and Dion Stewart say teams learn better in the immersive eco-system of Dojos than they do using traditional forms of training. They explain why and how Dojos help teams bond around product, foster rapid experimentation, and reframe small failures as learning.

    In this session, we will frame the need for dojos. From there we will walk attendees through the dojo format, including things they need to think about when creating their own. We wrap up with simple calls to actions for people to take to bring learning forward.

    Come to this session not only to learn about what works in creating a Dojo but also how Dojos help upskill your teams and support the cultural DevOps change.

  • Liked Jonathan Kauffman
    keyboard_arrow_down

    Jonathan Kauffman - Document Generation for Regulated Industries

    Jonathan Kauffman
    Jonathan Kauffman
    Consultant
    Coveros, Inc.
    schedule 1 month ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    One of the lines in the Agile Manifesto is "Working software over comprehensive documentation". This doesn't mean that no documentation is produced, but instead that only documentation that brings value to the team and the customer should be created. What do you do when you are working in a regulated industry and you need to produce extensive documentation to prove that the system works correctly? I recently worked with a company that produces FDA Class II medical devices and wanted to reduce the overhead of creating the documentation required by regulatory bodies. Their test team was spending a large amount of time each sprint on producing these documents, which took time away from verifying the system. We solved this problem by automating the generation of one type of document. We designed our solution such that all of the information that went into the Word document was pulled from JIRA -- this allowed the team to work within JIRA on a day-to-day basis and to use it as a source of truth. We describe the problem of document generation, how we approached solving the problem, and future work that we would like to do in this area.

  • Liked Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Managing BDD: Test Case Management for BDD Automation

    45 Mins
    Case Study
    Beginner

    Behavior-driven development (BDD) has been around for a while and is here to stay. However, the added abstraction levels pose a technical problem for writing and managing tests. While BDD does a great job of marrying the non-technical aspect of test writing to the technical flow of an application under test, keeping this information under source control becomes problematic. Frameworks such as JBehave, Cucumber, or Robot give subject matter experts that additional ability to write tests, but they are often restricted access from them; because people treat test cases as code, they get stored in source control repositories. Additionally, these given-when-then steps soon can grow to an extent where they are difficult to manage without an IDE, and nontechnical people lose interest. Using management tools, Max Saperstone shows how to manage these nontechnical steps and keep them in sync with the automaton in tools such as Git. He shows how to link tests to requirements, stories to development, and the traceability with continuous integration support. He also shares experiences of developing an open source product to help manage tests, with proven workflows at an enterprise level, for full team buy-in on both nontechnical and technical aspects of test case development.

  • Liked Richard Mills
    keyboard_arrow_down

    Richard Mills - DevSecOps: essential pipeline tooling to enable continuous security

    45 Mins
    Talk
    Intermediate

    As we embrace DevOps to optimize our Agility, we start pushing working code toward production releases more frequently. Whether we are doing true "Continuous Deployment" straight to production or not, we no longer have time for slow, manual, late-lifecycle security assessments to determine if our code is going to put us on the front page of the newspaper (for the wrong reasons). What we need is a way to know that our code is secure enough to pass muster every day. What we need is continuous security.

    The DevSecOps movement is about exactly that: shifting security assessment left and integrating it into the daily and sprint-ly cycles that DevOps has made popular. It means finding those touchpoints in our continuous integration/continuous delivery (CI/CD) pipeline where security tools can be inserted and run continuously against the software changes as they are made. It means using static code analysis, dynamic security testing, secure composition analysis of third party components, and platform vulnerability scanning to look at all aspects of security everyday. It means breaking builds and rejecting changes when developers introduce new security vulnerabilities.

    In this talk, I present my successes and challenges with integrating security into DevOps pipelines to provide continuous assessment of security posture. I focus on my latest experiences building delivery pipelines for a containerized microservice-based project where we integrated a broad set of open source and commercial tools to gather and present security data. Specifically, I highlight:

    • Touchpoints in your pipeline to asses security during build, deployment, and testing
    • Tool categories needed with examples of open-source and commercial options
    • Considerations to align tools with "security controls" for compliance
    • Data gathering, reporting, and dashboarding to get an easy view of security status
    • Team structures to encourage collaboration of security engineers with developers

    This talk is perfect for people struggling with ways to integrate application security assessment into their Agile development process.

  • Liked Richard Mills
    keyboard_arrow_down

    Richard Mills - DevOps Patterns to Enable Success with Microservices

    45 Mins
    Talk
    Intermediate

    DevOps can help you dig out of the problem you created for yourself: you spent your lunch period reading the interwebs, drank the kool-aid, and decided to embrace the utopia of microservices to solve all your fragile legacy monolithic code issues and allow you to release small independent changes into production. What you didn't realize is that you've translated an early-lifecycle code architecture problem into a late-lifecycle release management and quality assessment nightmare.

    This microservice thing has not provided the nirvana you expected. You ended up with:

    • a set of federated services that have hidden dependencies
    • independent applications maintained by teams that don't talk to each other
    • inability figure out which versions work together in your test environments, much less production
    • the need to test that your still-monolithic system works in pieces and as a whole

    You discover that this looks suspiciously like a DevOps problem and your pipeline is critical to your success.

    Someone once said to me "if you are building microservices without DevOps, you've already failed." I've learned that the integration problems created by independent microservices require a high level of automation with a pipeline that works independently of each service and can detect changes that break other services. The pipeline needs to facilitate communication between teams and assess which changes and versions work with each other.

    In this talk, I highlight the important things you need to succeed with microservices and avoid some of the common problems. Participants will leave with some new ideas on what they might be doing wrong in their current microservice-based project and/or anticipate what's going to go wrong if they are just getting started.

  • Liked Robin Foster
    keyboard_arrow_down

    Robin Foster - Maximizing Agile Benefits Through Understanding Learning Styles

    Robin Foster
    Robin Foster
    Consultant
    Coveros
    schedule 1 month ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    The Agile Manifesto says we value people and interactions over processes and tools. Yet when we talk about the agile methods, we usually end up talking about processes. Agilists use timeboxing as a forcing function to draw out resolutions and decisions. What if we just needed a better way to describe our needs and issues? What if we could actually be more effective with our communication? Join Robin Foster as he explores the relationships between learning, memory, communication, teaching, and the agile framework. Discover the cognitive science behind the process of learning new information, strategies for using different learning styles to grow your brain and its mastery of concepts, and how to map these strategies to phases in the agile cycle to benefit the team's cohesion and ability to self-organize. Agile is not meant to be rigid, so why should we be rigid in the way we share information with our team members?