DevOps Patterns to Enable Success with Microservices

schedule Sep 23rd 10:00 - 10:45 AM place Tiered Classroom people 8 Interested

DevOps can help you dig out of the problem you created for yourself: you spent your lunch period reading the interwebs, drank the kool-aid, and decided to embrace the utopia of microservices to solve all your fragile legacy monolithic code issues and allow you to release small independent changes into production. What you didn't realize is that you've translated an early-lifecycle code architecture problem into a late-lifecycle release management and quality assessment nightmare.

This microservice thing has not provided the nirvana you expected. You ended up with:

  • a set of federated services that have hidden dependencies
  • independent applications maintained by teams that don't talk to each other
  • inability figure out which versions work together in your test environments, much less production
  • the need to test that your still-monolithic system works in pieces and as a whole

You discover that this looks suspiciously like a DevOps problem and your pipeline is critical to your success.

Someone once said to me "if you are building microservices without DevOps, you've already failed." I've learned that the integration problems created by independent microservices require a high level of automation with a pipeline that works independently of each service and can detect changes that break other services. The pipeline needs to facilitate communication between teams and assess which changes and versions work with each other.

In this talk, I highlight the important things you need to succeed with microservices and avoid some of the common problems. Participants will leave with some new ideas on what they might be doing wrong in their current microservice-based project and/or anticipate what's going to go wrong if they are just getting started.

 
 

Outline/Structure of the Talk

My idea for this presentation came from some of the heavily attended microservices talks in the DevOps and Agile Dev tracks at Agile 2018 last year. It’s been accepted to be presented in August at Agile2019 so I'm hoping for some good feedback before AgileDC 2019.

In this talk, I highlight the important things you need to succeed with microservices and avoid some of the common problems:

  • Patterns for individual pipelines that combine to assess a working system
  • Using dynamic, ephemeral environments for early cross-service testing BEFORE people break things
  • Determining how appropriate versioning can simplify releases
  • Establishing a branch/merge process to minimize integration problems
  • Determining the value of using containers vs. virtual machines for microservices
  • Enumerating the types of testing needed at different stages of the pipeline

I'm drawing on my experiences implementing DevOps on my current project and the problems created having seven separate Agile teams building dozens of independent services. This, in turn, builds on my experiences implementing pipelines on many projects over the last 15 years that had similar integration and testing problems with federated, monolithic systems.

I like to organize my talks to give a high-level overview in the beginning and highlight the problems (microservices aren't all utopia, in this case). Then I dive into the details about the individual problems and my experiences solving them. I'll conclude with some takeaways and highlighting what people should have learned during the talk. I anticipate there will be a broad spectrum of listeners ranging high-level Agile coaches down to actual DevOps practitioners trying to solve real problems, so I try to move up and down through technical depth to make sure they each get something out of it.

Basic outline:

  • Problem - many things changing at different rates (fast vs. never), decoupled interfaces that are hard to verify independently
  • Solution - automated pipelines, lots of testing in dynamic test environment
  • Pipeline design and patterns
    • General pipeline design
    • Parallel pipelines
    • Test automation stages
    • Simultaneous branch and cross-service testing
    • Simplified versioning to avoid integration problems
    • Centralized jenkins libraries to avoid duplication
    • Full VM’s vs. streamlined containers
    • On-demand test environments
  • Importance of team structure and communication

Learning Outcome

  • Avoid common release management and integration problems caused by microservices
  • Define DevOps pipeline steps to independently deliver changes to a federated system
  • Design the right tests at the right stages for your pipeline
  • Use cross-service testing to enforce interfaces
  • Determine and promote cohesive releases of microservices

Target Audience

Developers, DevOps engineers, test automation engineers, and managers looking to improve their processes around microservices.

Prerequisites for Attendees

Attendees should have basic knowledge of microservices and DevOps automation, potentially employing them on their project already.

schedule Submitted 5 months ago

Public Feedback

comment Suggest improvements to the Speaker
  • George Dinwiddie
    By George Dinwiddie  ~  4 months ago
    reply Reply

    I like these aspects of the submission, and they should be retained:

    • Looks like a great topic.

    I think the submission could be improved by:

    • Tightening the abstract. This one is pretty long, and I suspect it could be reduced without losing important information that attracts the right people to the session.
    • Richard Mills
      By Richard Mills  ~  2 months ago
      reply Reply

      Hi George. I saw your original comment about "great topic" but not sure what your current intention is for accept/reject. 

      In case it's relevant, I was able to present this at Agile 2019 and received great response, questions, and feedback. I don't know if you were able to sit through it or not. I receive average of 5.0 (out of 5.0) scores (5.0 and 4.9 values) which topped the track averages in DevOps.

      The full slides are available on the Agile 2019 site if you want to look at them:

      https://agile2019.sched.com/event/70457dea732e1ee3ae371c7e488f0df0

       

    • Richard Mills
      By Richard Mills  ~  4 months ago
      reply Reply

      Good suggestion. I purposely wrote it long to give program committee a better idea of the talk. I think shortening it will make it easier for attendee to scan/evaluate while choosing talks to attend.

      • George Dinwiddie
        By George Dinwiddie  ~  4 months ago
        reply Reply
        I like that, but it’s better to put that info in the outline. 
        --Apple-Mail-76116452-2AF3-4430-B300-57609E3ADC0B--

    • Liked Max Saperstone
      keyboard_arrow_down

      Max Saperstone - Building Confidence In Your Automated Tests

      45 Mins
      Keynote
      Beginner

      The growth of automation testing in today’s software development organizations is changing the the way we test applications. Software development practices have matured over the last 30 years, to include all forms of testing to verify software quality. In the last ten years, there has been a huge spike in the adoption of automated tests, effectively replacing some of these manual testing practices, and supplementing many traditional testing activities. Many parts of the software development industry, however, are wary of replacing manual testing with automated testing. Not only is there often a lack of confidence in the automation tests, many see automated testing as fragile, unmaintainable, and ultimately, something delivering a low return on investment. Max believes that by employing mature software development techniques, we can achieve robust, maintainable, tests, that deliver confidence of the application under test. In addition to discussing how to structure automated tests that are cleaner, more maintainable and efficient, developer testing, and deployment techniques can be used to programmatically verify test correctness. Drawing on his experiences building test automation, test frameworks and advising organizations to adopt test automation, Max will walk us through how to mature your test automation practices.

    • Liked Gene Gotimer
      keyboard_arrow_down

      Gene Gotimer - Get to Green: How to safely refactor legacy code

      Gene Gotimer
      Gene Gotimer
      Principal Consultant
      Coveros, Inc.
      schedule 3 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      For many of us, legacy code is a fact of life. Code without tests -- no safe way to make changes, no safety net, no hope of untangling the web of accumulated ugliness, an incomplete understanding (or less) of how it really behaves. And your next set of changes is just going to add to the garbage pile and make it worse. You need tests so you can safely make changes, but you can't add tests without changing the code. It is a chicken-and-egg problem.

      So how do you turn legacy code into code you can change confidently? Slowly, one step at a time. Join Gene as he shares his experiences working with a monolithic codebase that was so bad it made national news. He'll go over the steps he and his team used to refactor the code safely by using mocking frameworks, mutation testing, and patience to build an understanding of how the code worked so that they could change it confidently.

      This talk is for anyone that has inherited legacy code that they aren't confident in and wants to make it something they can work on and improve. You'll leave with some tools and techniques that will help you change your legacy code into something maintainable.

    • Liked Thomas Stiehm
      keyboard_arrow_down

      Thomas Stiehm - Shifting Security Left - The Innovation of DevSecOps

      Thomas Stiehm
      Thomas Stiehm
      CTO
      Coveros, Inc.
      schedule 4 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      DevSecOps uses application security practices that have been around for a while. The innovation of DevSecOps is incorporating security into the daily workflow of the team rather than leaving them to the end of a release like many legacy processes do. Shifting security left is made possible by the ability to automate many aspects of security testing and verification. DevSecOps leverages DevOps practices to make application security a first-class citizen in the practices of modern software product development. DevSecOps starts with a culture change mindset of cross-functional teams creating software through collaboration and fast feedback cycles.

      The security in DevSecOps starts before the code is written by using techniques like threat modeling and risk analysis to help figure out who might want to attack you and how they might do that. This often ignored security practice can be enabled by following the DevSecOps practices of having a cross-functional team involved in the process from the beginning, including security professionals.

      Next, DevSecOps maps application security practices into the build pipeline for a project in order to provide quick feedback about the security posture for any change made to the software. By using automation to allow the team to move quickly while maintaining confidence in the health of the code base, DevSecOps extends that health check to include application security checks. While automation can be used to make security data collection easier it is important to understand what security practices still require a human being.

      This talk focuses on how, when, and where practices should be incorporated into a build pipeline to get the most value out of your security practices through automation. It explores what manual security work still needs to be done by a person and how to maximize value while minimizing the effort of human beings.

    • Liked Cherie Silas
      keyboard_arrow_down

      Cherie Silas - Forget Climbing the Ladder - Take the Escalator to Success!

      45 Mins
      Talk
      Beginner

      Ever wonder if climbing the ladder of success is really worth it? So much competition. So many people to out-perform. So many people to compete for to get the best jobs, the highest raise, the biggest bonus. And that frustrating bell curve! There's a better way. Forget the ladder! Take the Escalator instead.

      In this talk I will share my personal experience with the uselessness of climbing the ladder just to find out that you have reached the top only to be miserable and alone. What I have learned through countless experiences in my career is that ladder climbing is a futile way of progressing. There is a more powerful, effective, and satisfying way to achieve success. Take the escalator instead.

      The escalator turns what doesn't seem to make sense -- helping others succeed, helping your competitors win, giving away what others are selling - into your golden ticket to success. Join this talk to understand more about why the ladder method does not work long term and how the escalator model can bring more fulfillment, greater rewards, and financial security without the stress.

    • Liked Cherie Silas
      keyboard_arrow_down

      Cherie Silas / Chester Jackson - Coaching Change with Moving Motivators

      45 Mins
      Workshop/Game
      Beginner

      Presentation Overview: (What is the “message”? What key points will you make?)

      Change is hard – Staying motivated during change is even harder! But you can help your teams identify what things about the change are working for them and what they need to do to make small shifts that can keep them motivated.

      Is your team or team member facing a big change decision? Moving Motivators can help you identify the best choice by looking at how the factors in the change impact your long term motivation.

      Learn to discover and prioritize your motivators so you will understand how change today might impact you in the long run. See which choice is better when there are multiple options. Make conscious trade off decisions in a logical way with long term thinking in mind.

    • Liked Max Saperstone
      keyboard_arrow_down

      Max Saperstone - Getting to Continuous Testing

      45 Mins
      Case Study
      Beginner

      Max will tell the story of how a healthcare company striving to get to continuous releases built up their automation to secure confidence in regular releases. Initially, as no test automation existed, Max was able to take a greenfield test automation opportunity, and in the span of 12 months, develop over 2000 test cases. A testing pipeline was created to verify the integrity of the automated test cases, and to build docker containers for simple execution of the tests. These containers could then be simply re-used by developers and the DevOps team to verify the application. Max will walk through the feedback loop created, which allowed verification of the application go from hours to minutes.

      Max will discuss what processes and paths were taken to achieve continuous testing on this project. While he will cover the tools used and why they were chosen, the main focus will be on the HOW and WHY certain patterns and activities were performed. These choices were critical to achieving continuous testing, rather than just good testing coverage in CI or CD, even allowing a push left for performance and security. Additionally, some time will be spent on the organizational and culture changes that occured, and how he was able to accomplish this push for adoption in an organization that resisted automation, and had major quality problems.

    • Liked Glenn Buckholz
      keyboard_arrow_down

      Glenn Buckholz - Moving Your Pipeline to Kubernetes, Things I Wish People Had Told Me

      Glenn Buckholz
      Glenn Buckholz
      Technical Manager
      Coveros
      schedule 4 months ago
      Sold Out!
      45 Mins
      Case Study
      Intermediate

      Kubernetes married with a cloud provider elastic, highly available infrastructure. Many CI engines today (Jenkins, Bamboo, Gitlab, CircleCI), provide native integration with kubernetes so that your build and deploy workload can be elastically executed. This allows your pipeline to meet the needs of your schedule, be it the 4pm pile on to commit code before going home, the mad rush to get a hot fix to production, or the surge of an unexpected customer ask. Gone are the days of the build queue growing and you CI engine collapsing under the weight of a hundred build requests. In order for a pipeline to leverage this capacity changes must be made to the pipeline architecture. Tools must be dockerized, the ephemeral nature of running docker must be considered, kubernetes specifications or helm charts must be generated for the application, automated testing must be adapted to work in the new architecture, and then there is the database. Each one of these issues, plus many others I’ve missed contained unfortunate, unforeseen pitfalls that translated in schedule delays. Join Glenn as he helps you short circuit the pitfalls of migrating to kubernetes off of your static in-elastic virtual infrastructure.

    • Liked Thomas Stiehm
      keyboard_arrow_down

      Thomas Stiehm - Continuous Build and other DevOps anti-patterns, and how to overcome them

      Thomas Stiehm
      Thomas Stiehm
      CTO
      Coveros, Inc.
      schedule 4 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      Continuous Build is an anti-pattern that I have often seen where a team will have what they call Continuous Integration (CI) in place but it only builds the code, there are no tests or static analysis run. Certainly, this is better not building but it leaves a lot of health check information on the table that is considered part of CI. Without this information, you can never really gain the confidence that your build is healthy. The whole goal of CI is to feel that your build is healthy so not tests and analysis means you aren’t going CI.

      Just like CI, other DevOps practices can be hard to understand, implement, and get right. Even with the best of intentions, we make mistakes or misinterpret the implementation of a technique. Learn how to spot common DevOps anti-patterns and how to correct them. These patterns include

      1. Continuous Build - CI without tests isn’t CI
      2. Turn the unit tests off to build the release
      3. Don’t automate that, it is my job
      4. Different build process for developers and high environments
      5. Different deployment process for developers, test environments and/or production
      6. Not having a production-like environment to test in before production
      7. Saving performance testing for the end of the release
      8. Saving security testing for the end of the release
      9. Never asking the users about the software
      10. Only automating build and deployment, not testing
      11. Not having retrospective
      12. Restricting retrospectives to only the development part of the process
      13. Running analysis and never looking at or acting on the findings
      14. Reduce coverage or static analysis gates to get a build to pass

      We have all experienced a time where we wanted to believe we could make an anti-pattern work but it never does. It is better to learn how to spot these and how to correct them than it is to try to keep tweaking a broken process hoping this time it will be better.

    • Liked Thomas Stiehm
      keyboard_arrow_down

      Thomas Stiehm - Nobody Cares about Security and What DevSecOps is doing about it

      Thomas Stiehm
      Thomas Stiehm
      CTO
      Coveros, Inc.
      schedule 4 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      Application security is the poster child for third-class citizens in the software development world (behind Quality Assurance). DevSecOps is trying to turn that around and get more people, teams, and companies to care about security as our online and real-world lives become more intertwined.

      Application security has a bad reputation with many people. It comes into the development process late and demands a lot. Who wants to deal with that? We have actual business value to get out the door. “Those things” won’t happen anyway. And when “those things” do happen, it will just become an exercise in finger-pointing and blame. Security is an ugly affair that no one wants a part of.

      DevSecOps is a movement within the DevOps and Security worlds to reverse this decades-long drama by getting the people creating and updating software to build security practices into their process from the beginning, even before the code is written. This allows security professionals to become the evangelist of security practices where they can help the teams adopt practices and teach them how to use the tools to resolve issues themselves. No longer dependent on the specialists the teams can address security findings as they are found and make the workload manageable by spreading it across their implementation cycles-- proactively, not reactively.

      Attendees will leave with an understanding of how to map security concepts onto a delivery pipeline, how to “sell” security concepts to stakeholders, and how automation makes it easier to gather security data and act upon it. Learn what is needed to get started with DevSecOps so that you can start creating secure software today.

    • Liked Gene Gotimer
      keyboard_arrow_down

      Gene Gotimer - DevOps for Leadership

      Gene Gotimer
      Gene Gotimer
      Principal Consultant
      Coveros, Inc.
      schedule 3 months ago
      Sold Out!
      45 Mins
      Case Study
      Executive

      Organizations and leaders are often supportive of DevOps, but they don't always understand what DevOps is and what it will change. It isn't a one-size-fits-all issue; different environments need different benefits from a DevOps transformation. Join Gene Gotimer as he shows how to determine what parts of DevOps your organization needs to concentrate on first and how you should define success. This isn’t all strategic, high-level thinking- there is plenty of tactical, “what is the next step?” planning on how to shape your delivery pipeline to work for your situation and organization.

    • Liked Gene Gotimer
      keyboard_arrow_down

      Gene Gotimer - A practical approach to building security in

      Gene Gotimer
      Gene Gotimer
      Principal Consultant
      Coveros, Inc.
      schedule 3 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      The release date is a week away. Development is complete. The code works, and everything looks good. Marketing is ready with the media blitz. Our customers are waiting to get their hands on the new features and are sure to give us good feedback. The only step left is to get the security group to scan the application and give us the approval to release. Cross your fingers- let’s hope we get the green light! Otherwise, I don’t know what we are going to do.

      DevOps, and more importantly, DevSecOps, promises to do away with rolling the dice at the end and hoping we are allowed to release what we built. But how do we get to DevSecOps when we have a separate security sign-off, governance, regulations, and even corporate policies that say security gets the final word? How do we get from a classic security model to a DevOps-friendly process?

      Join Gene as he discusses practical steps for adding security to your existing development process. Attendees will learn tools and types of testing they can introduce to build security in from the beginning, eliminating the late surprises that the security team might find right before release.

    • Liked Jonathan Kauffman
      keyboard_arrow_down

      Jonathan Kauffman - Document Generation for Regulated Industries

      Jonathan Kauffman
      Jonathan Kauffman
      Consultant
      Coveros, Inc.
      schedule 5 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      One of the lines in the Agile Manifesto is "Working software over comprehensive documentation". This doesn't mean that no documentation is produced, but instead that only documentation that brings value to the team and the customer should be created. What do you do when you are working in a regulated industry and you need to produce extensive documentation to prove that the system works correctly? I recently worked with a company that produces FDA Class II medical devices and wanted to reduce the overhead of creating the documentation required by regulatory bodies. Their test team was spending a large amount of time each sprint on producing these documents, which took time away from verifying the system. We solved this problem by automating the generation of one type of document. We designed our solution such that all of the information that went into the Word document was pulled from JIRA -- this allowed the team to work within JIRA on a day-to-day basis and to use it as a source of truth. We describe the problem of document generation, how we approached solving the problem, and future work that we would like to do in this area.

    • Liked Max Saperstone
      keyboard_arrow_down

      Max Saperstone - Managing BDD: Test Case Management for BDD Automation

      45 Mins
      Case Study
      Beginner

      Behavior-driven development (BDD) has been around for a while and is here to stay. However, the added abstraction levels pose a technical problem for writing and managing tests. While BDD does a great job of marrying the non-technical aspect of test writing to the technical flow of an application under test, keeping this information under source control becomes problematic. Frameworks such as JBehave, Cucumber, or Robot give subject matter experts that additional ability to write tests, but they are often restricted access from them; because people treat test cases as code, they get stored in source control repositories. Additionally, these given-when-then steps soon can grow to an extent where they are difficult to manage without an IDE, and nontechnical people lose interest. Using management tools, Max Saperstone shows how to manage these nontechnical steps and keep them in sync with the automaton in tools such as Git. He shows how to link tests to requirements, stories to development, and the traceability with continuous integration support. He also shares experiences of developing an open source product to help manage tests, with proven workflows at an enterprise level, for full team buy-in on both nontechnical and technical aspects of test case development.

    • Liked Richard Mills
      keyboard_arrow_down

      Richard Mills - DevSecOps: essential pipeline tooling to enable continuous security

      Richard Mills
      Richard Mills
      DevOps Solution Lead
      Coveros, Inc.
      schedule 4 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      As we embrace DevOps to optimize our Agility, we start pushing working code toward production releases more frequently. Whether we are doing true "Continuous Deployment" straight to production or not, we no longer have time for slow, manual, late-lifecycle security assessments to determine if our code is going to put us on the front page of the newspaper (for the wrong reasons). What we need is a way to know that our code is secure enough to pass muster every day. What we need is continuous security.

      The DevSecOps movement is about exactly that: shifting security assessment left and integrating it into the daily and sprint-ly cycles that DevOps has made popular. It means finding those touchpoints in our continuous integration/continuous delivery (CI/CD) pipeline where security tools can be inserted and run continuously against the software changes as they are made. It means using static code analysis, dynamic security testing, secure composition analysis of third party components, and platform vulnerability scanning to look at all aspects of security everyday. It means breaking builds and rejecting changes when developers introduce new security vulnerabilities.

      In this talk, I present my successes and challenges with integrating security into DevOps pipelines to provide continuous assessment of security posture. I focus on my latest experiences building delivery pipelines for a containerized microservice-based project where we integrated a broad set of open source and commercial tools to gather and present security data. Specifically, I highlight:

      • Touchpoints in your pipeline to asses security during build, deployment, and testing
      • Tool categories needed with examples of open-source and commercial options
      • Considerations to align tools with "security controls" for compliance
      • Data gathering, reporting, and dashboarding to get an easy view of security status
      • Team structures to encourage collaboration of security engineers with developers

      This talk is perfect for people struggling with ways to integrate application security assessment into their Agile development process.

    • Liked Robin Foster
      keyboard_arrow_down

      Robin Foster - Maximizing Agile Benefits Through Understanding Learning Styles

      Robin Foster
      Robin Foster
      Consultant
      Coveros
      schedule 5 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      The Agile Manifesto says we value people and interactions over processes and tools. Yet when we talk about the agile methods, we usually end up talking about processes. Agilists use timeboxing as a forcing function to draw out resolutions and decisions. What if we just needed a better way to describe our needs and issues? What if we could actually be more effective with our communication? Join Robin Foster as he explores the relationships between learning, memory, communication, teaching, and the agile framework. Discover the cognitive science behind the process of learning new information, strategies for using different learning styles to grow your brain and its mastery of concepts, and how to map these strategies to phases in the agile cycle to benefit the team's cohesion and ability to self-organize. Agile is not meant to be rigid, so why should we be rigid in the way we share information with our team members?

    • Liked Saya Sone
      keyboard_arrow_down

      Saya Sone - Collaborate SAFe & Design For Better Product Performance

      Saya Sone
      Saya Sone
      Agile Coach
      OCTO Consulting
      schedule 4 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      While our program has grown up to have 16 teams, our Design Systems team still has the same amount of resources. We ran into a question - should we add more resources or find a better way to integrate SAFe (Scaling Agile Framework) along with the design process? Another concern is that the Design Systems team receives many ad hoc requests from different teams that are asking for the same prototype with a slightly different flavor. Our architects and designers are also not usually in sync of the UI/UX dynamics.


      This workshop will present a 4-D (Discover, Define, Design, and Deliver) model to help the Design Systems team integrate with all development teams to plan wisely to avoid duplicates or last-minute requests. The 4-D model aims to follow Agile and Lean principles to promote increased collaboration and interaction among individuals and teams. The model uses Design Thinking and Lean Startup concepts to design the right systems to gain customer values. Most importantly, our Design Systems team can improve their stakeholder engagement with shorter planning of iteration where our development team can produce software faster. The 4-D model proves a better product performance that requires more collaborative and shared design work with frequent agile delivery of customer-valued products. We anticipate UI/UX designers, architects, portfolio managers, product owners, project managers, scrum masters, and agile coaches who want to learn how to collaborate SAFe and design for better product performance join this workshop and take away the technique to improve their enterprise level of design processes.