Moving Your Pipeline to Kubernetes, Things I Wish People Had Told Me

Kubernetes married with a cloud provider elastic, highly available infrastructure. Many CI engines today (Jenkins, Bamboo, Gitlab, CircleCI), provide native integration with kubernetes so that your build and deploy workload can be elastically executed. This allows your pipeline to meet the needs of your schedule, be it the 4pm pile on to commit code before going home, the mad rush to get a hot fix to production, or the surge of an unexpected customer ask. Gone are the days of the build queue growing and you CI engine collapsing under the weight of a hundred build requests. In order for a pipeline to leverage this capacity changes must be made to the pipeline architecture. Tools must be dockerized, the ephemeral nature of running docker must be considered, kubernetes specifications or helm charts must be generated for the application, automated testing must be adapted to work in the new architecture, and then there is the database. Each one of these issues, plus many others I’ve missed contained unfortunate, unforeseen pitfalls that translated in schedule delays. Join Glenn as he helps you short circuit the pitfalls of migrating to kubernetes off of your static in-elastic virtual infrastructure.

 
7 favorite thumb_down thumb_up 1 comment visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/Structure of the Case Study

The presentation will be divided into several sections:

  • Quick overview of docker
  • Quick overview of kubernetes
  • Pros and cons of containerizing your build tools.
    • Cacheing solutions for maven/gradle/et al.
  • Approaches for executing your tests inside kubernetes
  • Discussion of ephemeral test environments
  • Persistance
  • Databases
  • Environment management inside kubernetes

Learning Outcome

For this session at the end you should have an approach to solve the following issues:

  • How to approach migrating your pipeline to kubernetes
  • A taxonimy of high level design mistakes to avoid
  • A laundry list of technical mistakes to avoid
    • What to do about persistence at the file system and database level
    • When to have databases inside and outside the cluster and what are the design considerations
    • An approach to managing your static environment into kubernetes and what it means for CD
    • How to avoid letting your build tools run afoul of dockers ephemeral nature.
    • How to partition build workload for faster build times and avoid build taking longer in kubernetes
    • How to ensure deployments to kubernetes are as efficient as deployments to the current virtual infrastructure.
  • How to migrate your infrastructure as code deployments to kubernetes

Target Audience

People who are looking to move their DevOps infrastructure to kubernetes or are looking for solutions to issues in their new kubernetes DevOps implementation.

Prerequisites for Attendees

Participants should know:

  • High level information about containers (docker)
  • What the purpose of a DevOps pipeline is and the typical inputs
  • What problems kubernetes is supposed to solve for containers
schedule Submitted 3 weeks ago

Public Feedback

comment Suggest improvements to the Speaker
  • George Dinwiddie
    By George Dinwiddie  ~  3 weeks ago
    reply Reply

    I like these aspects of the submission, and they should be retained:

    • Experienced advice on the details

    I think the submission could be improved by:

    • Writing the abstract to attract the right attendees. Right now, it's very hard to read. The first sentence isn't even a sentence.
    • Appeal to those who don't yet know they're going to use Kubernetes. What will it do for them?
    • For all attendees, why should they attend this talk? You want the abstract to help the right people self-select into the session.

  • Liked Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Building Confidence In Your Automated Tests

    45 Mins
    Keynote
    Beginner

    The growth of automation testing in today’s software development organizations is changing the the way we test applications. Software development practices have matured over the last 30 years, to include all forms of testing to verify software quality. In the last ten years, there has been a huge spike in the adoption of automated tests, effectively replacing some of these manual testing practices, and supplementing many traditional testing activities. Many parts of the software development industry, however, are wary of replacing manual testing with automated testing. Not only is there often a lack of confidence in the automation tests, many see automated testing as fragile, unmaintainable, and ultimately, something delivering a low return on investment. Max believes that by employing mature software development techniques, we can achieve robust, maintainable, tests, that deliver confidence of the application under test. In addition to discussing how to structure automated tests that are cleaner, more maintainable and efficient, developer testing, and deployment techniques can be used to programmatically verify test correctness. Drawing on his experiences building test automation, test frameworks and advising organizations to adopt test automation, Max will walk us through how to mature your test automation practices.

  • Liked Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Getting to Continuous Testing

    45 Mins
    Case Study
    Beginner

    Max will tell the story of how a healthcare company striving to get to continuous releases built up their automation to secure confidence in regular releases. Initially, as no test automation existed, Max was able to take a greenfield test automation opportunity, and in the span of 12 months, develop over 2000 test cases. A testing pipeline was created to verify the integrity of the automated test cases, and to build docker containers for simple execution of the tests. These containers could then be simply re-used by developers and the DevOps team to verify the application. Max will walk through the feedback loop created, which allowed verification of the application go from hours to minutes.

    Max will discuss what processes and paths were taken to achieve continuous testing on this project. While he will cover the tools used and why they were chosen, the main focus will be on the HOW and WHY certain patterns and activities were performed. These choices were critical to achieving continuous testing, rather than just good testing coverage in CI or CD, even allowing a push left for performance and security. Additionally, some time will be spent on the organizational and culture changes that occured, and how he was able to accomplish this push for adoption in an organization that resisted automation, and had major quality problems.

  • Liked Thomas Stiehm
    keyboard_arrow_down

    Thomas Stiehm - Continuous Build and other DevOps anti-patterns, and how to overcome them

    Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 3 weeks ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Continuous Build is an anti-pattern that I have often seen where a team will have what they call Continuous Integration (CI) in place but it only builds the code, there are no tests or static analysis run. Certainly, this is better not building but it leaves a lot of health check information on the table that is considered part of CI. Without this information, you can never really gain the confidence that your build is healthy. The whole goal of CI is to feel that your build is healthy so not tests and analysis means you aren’t going CI.

    Just like CI, other DevOps practices can be hard to understand, implement, and get right. Even with the best of intentions, we make mistakes or misinterpret the implementation of a technique. Learn how to spot common DevOps anti-patterns and how to correct them. These patterns include

    1. Continuous Build - CI without tests isn’t CI
    2. Turn the unit tests off to build the release
    3. Don’t automate that, it is my job
    4. Different build process for developers and high environments
    5. Different deployment process for developers, test environments and/or production
    6. Not having a production-like environment to test in before production
    7. Saving performance testing for the end of the release
    8. Saving security testing for the end of the release
    9. Never asking the users about the software
    10. Only automating build and deployment, not testing
    11. Not having retrospective
    12. Restricting retrospectives to only the development part of the process
    13. Running analysis and never looking at or acting on the findings
    14. Reduce coverage or static analysis gates to get a build to pass

    We have all experienced a time where we wanted to believe we could make an anti-pattern work but it never does. It is better to learn how to spot these and how to correct them than it is to try to keep tweaking a broken process hoping this time it will be better.

  • Liked Thomas Stiehm
    keyboard_arrow_down

    Thomas Stiehm - Nobody Cares about Security and What DevSecOps is doing about it

    Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 3 weeks ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Application security is the poster child for third-class citizens in the software development world (behind Quality Assurance). DevSecOps is trying to turn that around and get more people, teams, and companies to care about security as our online and real-world lives become more intertwined.

    Application security has a bad reputation with many people. It comes into the development process late and demands a lot. Who wants to deal with that? We have actual business value to get out the door. “Those things” won’t happen anyway. And when “those things” do happen, it will just become an exercise in finger-pointing and blame. Security is an ugly affair that no one wants a part of.

    DevSecOps is a movement within the DevOps and Security worlds to reverse this decades-long drama by getting the people creating and updating software to build security practices into their process from the beginning, even before the code is written. This allows security professionals to become the evangelist of security practices where they can help the teams adopt practices and teach them how to use the tools to resolve issues themselves. No longer dependent on the specialists the teams can address security findings as they are found and make the workload manageable by spreading it across their implementation cycles-- proactively, not reactively.

    Attendees will leave with an understanding of how to map security concepts onto a delivery pipeline, how to “sell” security concepts to stakeholders, and how automation makes it easier to gather security data and act upon it. Learn what is needed to get started with DevSecOps so that you can start creating secure software today.

  • Liked Thomas Stiehm
    keyboard_arrow_down

    Thomas Stiehm - Shifting Security Left - The Innovation of DevSecOps

    Thomas Stiehm
    Thomas Stiehm
    CTO
    Coveros, Inc.
    schedule 3 weeks ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    DevSecOps uses application security practices that have been around for a while. The innovation of DevSecOps is incorporating security into the daily workflow of the team rather than leaving them to the end of a release like many legacy processes do. Shifting security left is made possible by the ability to automate many aspects of security testing and verification. DevSecOps leverages DevOps practices to make application security a first-class citizen in the practices of modern software product development. DevSecOps starts with a culture change mindset of cross-functional teams creating software through collaboration and fast feedback cycles.

    The security in DevSecOps starts before the code is written by using techniques like threat modeling and risk analysis to help figure out who might want to attack you and how they might do that. This often ignored security practice can be enabled by following the DevSecOps practices of having a cross-functional team involved in the process from the beginning, including security professionals.

    Next, DevSecOps maps application security practices into the build pipeline for a project in order to provide quick feedback about the security posture for any change made to the software. By using automation to allow the team to move quickly while maintaining confidence in the health of the code base, DevSecOps extends that health check to include application security checks. While automation can be used to make security data collection easier it is important to understand what security practices still require a human being.

    This talk focuses on how, when, and where practices should be incorporated into a build pipeline to get the most value out of your security practices through automation. It explores what manual security work still needs to be done by a person and how to maximize value while minimizing the effort of human beings.

  • Liked Craeg K Strong
    keyboard_arrow_down

    Craeg K Strong - Kanban Antipatterns: What You Don’t Know Can Hurt You

    45 Mins
    Workshop/Game
    Beginner

    In this interactive workshop we will examine multiple examples of Antipatterns observed in real-world Kanban boards. In each case we will identify the issues and discuss ways to improve the situation. We will review a number of better alternatives and see how the improvements map to the core principles of Kanban such as visualization, managing flow, and making policies explicit. Brand new to Kanban? Learning by example is a great way to get started! A long-time Kanban veteran? Come to see how many antipatterns you recognize and help firm up our Kanban Antipattern taxonomy and nomenclature!

    Kanban is an extremely versatile and effective Agile method that has seen significant growth in popularity over recent years. Kanban’s flexibility has led to widespread adoption to manage business processes in disparate contexts such as HR, loan processing, drug discovery, and insurance underwriting, in addition to Information Technology. Like snowflakes, no two Kanban boards are alike. The downside to this flexibility is there is no well-known and easily accessible library of patterns for designing effective Kanban boards. Like Apollo engineers, teams are expected to design their board starting from first principles. Unfortunately, sometimes teams get stuck with board designs that may not provide the visibility and insight into their workflow they hope to see. Worse, some designs actually may serve only to obscure the situation. Working within the limitations of an electronic board can exacerbate the problem even further. Is all hope lost? Certainly not!

    Let’s learn more about effective Kanban system design by examining what to avoid and why. Learning by example is effective and fun!

  • Liked Claire Atwell
    keyboard_arrow_down

    Claire Atwell / Kerri Sutey - Lost in Transformation? Using the Integral Agile Model to Find Your Way out of the Maze

    45 Mins
    Workshop/Game
    Intermediate

    You think you have your transformation figured out, but then your efforts to address one issue, you create unintended consequences that become the next hurdle. The only way to find your way out of the transformation maze is to understand your complex ecosystem and plan for change. In this workshop, we will explore how leveraging both the Integral Agile Model and a framework, such as SAFe can lead to executing successful transformation strategies. Join us as we discuss how to pair the Integral Agile Model with your framework to create a holistic view of your entire eco-system of leaders, vendors, compliance needs, and policy. We will also explore how to bring organizational goals to life by creating a priorities-based roadmap. If you're feeling lost in transformation, this workshop is for you!

  • Liked Jonathan Kauffman
    keyboard_arrow_down

    Jonathan Kauffman - Document Generation for Regulated Industries

    Jonathan Kauffman
    Jonathan Kauffman
    Consultant
    Coveros, Inc.
    schedule 1 month ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    One of the lines in the Agile Manifesto is "Working software over comprehensive documentation". This doesn't mean that no documentation is produced, but instead that only documentation that brings value to the team and the customer should be created. What do you do when you are working in a regulated industry and you need to produce extensive documentation to prove that the system works correctly? I recently worked with a company that produces FDA Class II medical devices and wanted to reduce the overhead of creating the documentation required by regulatory bodies. Their test team was spending a large amount of time each sprint on producing these documents, which took time away from verifying the system. We solved this problem by automating the generation of one type of document. We designed our solution such that all of the information that went into the Word document was pulled from JIRA -- this allowed the team to work within JIRA on a day-to-day basis and to use it as a source of truth. We describe the problem of document generation, how we approached solving the problem, and future work that we would like to do in this area.

  • Liked Max Saperstone
    keyboard_arrow_down

    Max Saperstone - Managing BDD: Test Case Management for BDD Automation

    45 Mins
    Case Study
    Beginner

    Behavior-driven development (BDD) has been around for a while and is here to stay. However, the added abstraction levels pose a technical problem for writing and managing tests. While BDD does a great job of marrying the non-technical aspect of test writing to the technical flow of an application under test, keeping this information under source control becomes problematic. Frameworks such as JBehave, Cucumber, or Robot give subject matter experts that additional ability to write tests, but they are often restricted access from them; because people treat test cases as code, they get stored in source control repositories. Additionally, these given-when-then steps soon can grow to an extent where they are difficult to manage without an IDE, and nontechnical people lose interest. Using management tools, Max Saperstone shows how to manage these nontechnical steps and keep them in sync with the automaton in tools such as Git. He shows how to link tests to requirements, stories to development, and the traceability with continuous integration support. He also shares experiences of developing an open source product to help manage tests, with proven workflows at an enterprise level, for full team buy-in on both nontechnical and technical aspects of test case development.

  • Liked Archana dabbeeru
    keyboard_arrow_down

    Archana dabbeeru - Strategizing Complex Business Processes with Agile

    45 Mins
    Talk
    Intermediate
    1. Introduction
    The agile organization is a living organism.Organizations are rapidly seeing their projects shift from the ‘complicated’ to the ‘complex’ space and are finding the old ways of trying to predict plans and control change that no longer work. Strategic planning is a fluid process.Just as water naturally deviates its course around a boulder, an organization should aim to be agile enough to shift directions when obstacles arise.Today's world has become increasingly complex, unpredictable and prone to rapid change.There's little value in developing the perfect, rigid strategic plan. Instead, successful organizations are adopting a more dynamic approach to strategizing by embodying an agile decision-making culture.
    2. Explanation of benefits of building long-term Agile plan
    Agile development, in its simplest form, offers a lightweight framework for helping teams, given a constantly evolving functional and technical landscape, maintain a focus on the rapid delivery of business value.No matter what agile methodology we are using (kanban, scrum, or a combination) or where we are on our agile journey (just starting out or practicing agile at scale), there’s still a need to forecast over a long period of time, make date commitments, plan resources, and tie our work back to a strategic vision. This is the moment to pause, reflect on what we had originally set out to accomplish and ask ourself candidly if that vision still resonates.Creating a culture based on agile decision-making is fast becoming table stakes. Although it can be a challenging shift for well-established firms with legacy and bureaucracy issues, taking a more flexible approach to executing and communicating your strategic plan is vital.Harnessing the business process and the organization strategy to the agile roadmap will project the value of the agile transformation.
    3. Steps for building long term Agile Plan
    We will talk about the steps to build a long-term agile plan using the metaphor of building a house in 8 steps:
    3.1.Start with the big picture.
    3.2.Identify the big ticket items.
    3.3. Break it Down
    3.4. Get Estimating
    3.5. Create smart Releases
    3.6. Generate Roadmap
    3.7. Share with the team and validate
    3.8. Keep Improving

    4.Dialogue with the audience
    (re: examples of complex systems etc)
    5. FAQ
    Q&A
  • Liked Richard Mills
    keyboard_arrow_down

    Richard Mills - DevSecOps: essential pipeline tooling to enable continuous security

    45 Mins
    Talk
    Intermediate

    As we embrace DevOps to optimize our Agility, we start pushing working code toward production releases more frequently. Whether we are doing true "Continuous Deployment" straight to production or not, we no longer have time for slow, manual, late-lifecycle security assessments to determine if our code is going to put us on the front page of the newspaper (for the wrong reasons). What we need is a way to know that our code is secure enough to pass muster every day. What we need is continuous security.

    The DevSecOps movement is about exactly that: shifting security assessment left and integrating it into the daily and sprint-ly cycles that DevOps has made popular. It means finding those touchpoints in our continuous integration/continuous delivery (CI/CD) pipeline where security tools can be inserted and run continuously against the software changes as they are made. It means using static code analysis, dynamic security testing, secure composition analysis of third party components, and platform vulnerability scanning to look at all aspects of security everyday. It means breaking builds and rejecting changes when developers introduce new security vulnerabilities.

    In this talk, I present my successes and challenges with integrating security into DevOps pipelines to provide continuous assessment of security posture. I focus on my latest experiences building delivery pipelines for a containerized microservice-based project where we integrated a broad set of open source and commercial tools to gather and present security data. Specifically, I highlight:

    • Touchpoints in your pipeline to asses security during build, deployment, and testing
    • Tool categories needed with examples of open-source and commercial options
    • Considerations to align tools with "security controls" for compliance
    • Data gathering, reporting, and dashboarding to get an easy view of security status
    • Team structures to encourage collaboration of security engineers with developers

    This talk is perfect for people struggling with ways to integrate application security assessment into their Agile development process.

  • Liked Richard Mills
    keyboard_arrow_down

    Richard Mills - DevOps Patterns to Enable Success with Microservices

    45 Mins
    Talk
    Intermediate

    DevOps can help you dig out of the problem you created for yourself: you spent your lunch period reading the interwebs, drank the kool-aid, and decided to embrace the utopia of microservices to solve all your fragile legacy monolithic code issues and allow you to release small independent changes into production. What you didn't realize is that you've translated an early-lifecycle code architecture problem into a late-lifecycle release management and quality assessment nightmare.

    This microservice thing has not provided the nirvana you expected. You ended up with:

    • a set of federated services that have hidden dependencies
    • independent applications maintained by teams that don't talk to each other
    • inability figure out which versions work together in your test environments, much less production
    • the need to test that your still-monolithic system works in pieces and as a whole

    You discover that this looks suspiciously like a DevOps problem and your pipeline is critical to your success.

    Someone once said to me "if you are building microservices without DevOps, you've already failed." I've learned that the integration problems created by independent microservices require a high level of automation with a pipeline that works independently of each service and can detect changes that break other services. The pipeline needs to facilitate communication between teams and assess which changes and versions work with each other.

    In this talk, I highlight the important things you need to succeed with microservices and avoid some of the common problems. Participants will leave with some new ideas on what they might be doing wrong in their current microservice-based project and/or anticipate what's going to go wrong if they are just getting started.

  • Liked Robin Foster
    keyboard_arrow_down

    Robin Foster - Maximizing Agile Benefits Through Understanding Learning Styles

    Robin Foster
    Robin Foster
    Consultant
    Coveros
    schedule 1 month ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    The Agile Manifesto says we value people and interactions over processes and tools. Yet when we talk about the agile methods, we usually end up talking about processes. Agilists use timeboxing as a forcing function to draw out resolutions and decisions. What if we just needed a better way to describe our needs and issues? What if we could actually be more effective with our communication? Join Robin Foster as he explores the relationships between learning, memory, communication, teaching, and the agile framework. Discover the cognitive science behind the process of learning new information, strategies for using different learning styles to grow your brain and its mastery of concepts, and how to map these strategies to phases in the agile cycle to benefit the team's cohesion and ability to self-organize. Agile is not meant to be rigid, so why should we be rigid in the way we share information with our team members?