Security & Chaos Engineering: A Novel Approach to Crafting Secure and Resilient Distributed Systems

location_city Virtual schedule Oct 16th 05:00 - 05:45 PM place Online people 31 Interested

Modern systems pose a number of thorny challenges and securing the transformation from legacy monolithic systems to distributed systems demands a change in mindset and engineering toolkit. The security engineering toolkit is unfortunately out-of-style and outdated with today's approach to building, security and operating distributed systems. The speed, scale, and complex operations within microservice architectures make them tremendously difficult for humans to mentally model their behavior. If the latter is even remotely true how is it possible to adequately secure services that are not even fully comprehended by the engineering teams that built them. Security Chaos Engineering helps teams realign the actual state of operational security as well as build confidence that their security actually works the way the think it does. Chaos Engineering allows for security teams to proactively experiment on recurring incident patterns to derive new information about underlying factors that were previously unknown by reversing the postmortem and preparation phases. This is done by developing live fire exercises that can be measured, managed, and automated. It develops teams by building a learning culture around system failure to challenge engineering teams to proactively, safely discover system weakness before they disrupt business outcomes. In this session we will introduce a new concept known as Security Chaos Engineering and how it can be applied to create highly secure, performant, and resilient distributed systems.


Outline/Structure of the Talk

Example Outline

  • Title Slide
  • About the Speaker & Speaker Contact Info
  • Visual Overview of Example Modern Distributed Systems - Our systems have evolved beyond human ability to mentally model their behavior.
  • Review of the complexity of current software engineering models, techniques, and methods.
  • Highlight the ever-widening gap between those models and security as an engineering discipline.
  • Complexity vs. Simplicity
  • Software only increases in Complexity it never decreases.
  • Accidental vs. Essential Complexity
  • Complex Adaptive Systems – Dr. David Woods
    • Woods Theorem
  • The difficulty in understanding our own systems
  • How these concepts are impacting security
  • Incident, Outages and Breaches are happening more often and getting worse.
  • Teams spend too much time reacting to outages instead of building more resilient systems.
  • Chaos Engineering Defined
  • Chaos Engineering Origin Story – What, Why and How of Chaos Monkey and ChAP at Netflix
  • Chaos Engineering is not just Netflix – 1200+ companies now are adopting chaos engineering
  • Why is it so important to do Chaos Engineering?
  • The Normal Condition of a Human & Systems they Build is to fail
  • Bring Order to Chaos with Chaos Engineering
  • Ways it is used in Security – Validate runbooks, determine control effectiveness, learn new insights into system behavior, validate architectural patterns, and proactively identify system gaps and problems in system security before they impact customers.
  • How Security Chaos Engineering is different that Penetration Testing, Adversarial Testing, Red/Purple Teaming, etc.
  • A Shift in Mindset: Stop looking for better answers and start asking better questions.
    • What is the system actually doing?
    • Has it done this before?
    • Why is it behaving that way?
    • What is it supposed to do next?
    • How did it get into this state?
  • The 1 Open Source Security Chaos Engineering Tool – ChaoSlingr
    • Features
    • Function review
    • How experiments are constructed, deployed, and executed.
  • How it works
  • Where to get started
  • Speaker’s Reflection: The value learned by the speaker in applying these tools and techniques at the largest healthcare company in the world.
  • Review of more Example Security Chaos Experiments and some using Kubernetes
  • Takeaways
  • Q&A

Learning Outcome

1: Learn a new technique for uncovering system weaknesses in systems security.
2: Change incident response and security engineering team thinking.
3: Identify the hidden costs of security Incidents.

4: Discover new ways to proactively expose gaps in how we think our systems security works vs. the operational reality.

5: Learn about the importance of recalibration and understanding failure in distributed systems security

6: Learn about a new open source tool that they can use to do Security Chaos experiments

7: How to apply Chaos Engineering with Security to create a DevSecOps culture

8: The business value of Chaos Engineering with Security and how to get started

Target Audience

software engineers, software security engineers, security engineers, technology executives

Prerequisites for Attendees

Foundational knowledge of information security practices,

Basic knowledge of distributed systems

General knowledge of build systems at large scale

schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Author
  • Leena S N
    By Leena S N  ~  1 year ago
    reply Reply

    Hi Aaron,

    Thank you for the proposal. 

    I see a lot of content in the outline section. Wondering whether you will be able to cover all these in 45 minutes. Can you update it accordingly?

    Also I believe you will be presenting this as a case study or like an experience report. Can you confirm?



    • Aaron Rinehart
      By Aaron Rinehart  ~  1 year ago
      reply Reply

      Leena, Thx for reaching out!

      The outline is more or less a representation of key points I plan to deliver in the talk. 

      Many of these points/topics will be covered in logical groupings or covered together.

      My goal is to provide the audience with a solid foundation of key concepts and interactively lead them to the main body of content around Chaos Engineering and its applications to Cyber Security.

      The talk will have elements of an experience report and a case study. I will share my experiences in building ChaoSlingr, the first open source software application to demonstrates how to perform Security focused Chaos Engineering while at the largest Healthcare Company in the world(UnitedHealth Group). I will share the problems we faced and how we overcame them with Security Chaos Engineering.

      I have extensive experience in delivering this body of knowledge in both keynotes as well as an individual session to both Native English and Non-Native English audiences. I have a number of engaging moments with the audience in terms of audience interaction (questions, jokes, funny gifs) to keep them actively engaged and interested in the material.

      In terms of timing of the presentation I should wrap up the slides/content somewhere between 35-40 min.

      Feel free to reach out if you have any more questions or comments!