Security & Chaos Engineering: A Novel Approach to Crafting Secure and Resilient Distributed Systems
Modern systems pose a number of thorny challenges and securing the transformation from legacy monolithic systems to distributed systems demands a change in mindset and engineering toolkit. The security engineering toolkit is unfortunately out-of-style and outdated with today's approach to building, security and operating distributed systems. The speed, scale, and complex operations within microservice architectures make them tremendously difficult for humans to mentally model their behavior. If the latter is even remotely true how is it possible to adequately secure services that are not even fully comprehended by the engineering teams that built them. Security Chaos Engineering helps teams realign the actual state of operational security as well as build confidence that their security actually works the way the think it does. Chaos Engineering allows for security teams to proactively experiment on recurring incident patterns to derive new information about underlying factors that were previously unknown by reversing the postmortem and preparation phases. This is done by developing live fire exercises that can be measured, managed, and automated. It develops teams by building a learning culture around system failure to challenge engineering teams to proactively, safely discover system weakness before they disrupt business outcomes. In this session we will introduce a new concept known as Security Chaos Engineering and how it can be applied to create highly secure, performant, and resilient distributed systems.
Outline/Structure of the Talk
Example Outline
- Title Slide
- About the Speaker & Speaker Contact Info
- Visual Overview of Example Modern Distributed Systems - Our systems have evolved beyond human ability to mentally model their behavior.
- Review of the complexity of current software engineering models, techniques, and methods.
- Highlight the ever-widening gap between those models and security as an engineering discipline.
- Complexity vs. Simplicity
- Software only increases in Complexity it never decreases.
- Accidental vs. Essential Complexity
- Complex Adaptive Systems – Dr. David Woods
- Woods Theorem
- The difficulty in understanding our own systems
- How these concepts are impacting security
- Incident, Outages and Breaches are happening more often and getting worse.
- Teams spend too much time reacting to outages instead of building more resilient systems.
- Chaos Engineering Defined
- Chaos Engineering Origin Story – What, Why and How of Chaos Monkey and ChAP at Netflix
- Chaos Engineering is not just Netflix – 1200+ companies now are adopting chaos engineering
- Why is it so important to do Chaos Engineering?
- The Normal Condition of a Human & Systems they Build is to fail
- Bring Order to Chaos with Chaos Engineering
- Ways it is used in Security – Validate runbooks, determine control effectiveness, learn new insights into system behavior, validate architectural patterns, and proactively identify system gaps and problems in system security before they impact customers.
- How Security Chaos Engineering is different that Penetration Testing, Adversarial Testing, Red/Purple Teaming, etc.
- A Shift in Mindset: Stop looking for better answers and start asking better questions.
- What is the system actually doing?
- Has it done this before?
- Why is it behaving that way?
- What is it supposed to do next?
- How did it get into this state?
- The 1 Open Source Security Chaos Engineering Tool – ChaoSlingr
- Features
- Function review
- How experiments are constructed, deployed, and executed.
- How it works
- Where to get started
- Speaker’s Reflection: The value learned by the speaker in applying these tools and techniques at the largest healthcare company in the world.
- Review of more Example Security Chaos Experiments and some using Kubernetes
- Takeaways
- Q&A
Learning Outcome
1: Learn a new technique for uncovering system weaknesses in systems security.
2: Change incident response and security engineering team thinking.
3: Identify the hidden costs of security Incidents.
4: Discover new ways to proactively expose gaps in how we think our systems security works vs. the operational reality.
5: Learn about the importance of recalibration and understanding failure in distributed systems security
6: Learn about a new open source tool that they can use to do Security Chaos experiments
7: How to apply Chaos Engineering with Security to create a DevSecOps culture
8: The business value of Chaos Engineering with Security and how to get started
Target Audience
software engineers, software security engineers, security engineers, technology executives
Prerequisites for Attendees
Foundational knowledge of information security practices,
Basic knowledge of distributed systems
General knowledge of build systems at large scale
Links
Publications
Here is a sample of publications either written by me or written about the work I have done relating to the this talk
Injecting chaos experiments into security log pipelines - https://opensource.com/article/18/9/injecting-chaos-experiments-security-log-pipelines
Purple testing and chaos engineering in security experimentation
https://opensource.com/article/18/6/security-experimentation
Security Chaos Engineering: A new paradigm for cybersecurity
https://opensource.com/article/18/1/new-paradigm-cybersecurity
A new approach to security instrumentation
https://opensource.com/article/18/4/new-approach-security-instrumentation
InfoQ eMag: Chaos Engineering
https://www.infoq.com/minibooks/emag-chaos-engineering/#minibookDownload/
- Using Chaos Engineering to Secure Distributed Systems - Aaron Rinehart explores how chaos engineering can be applied to security testing in distributed systems, arguing that it differs from both red/purple-team security testing and penetration testing in its goals, purpose, and methodology.
What is security chaos engineering and why is it important?
https://hub.packtpub.com/what-is-security-chaos-engineering-and-why-is-it-important/
Information Security Media Group (ISMG) Healthcare Cybersecurity: “The New Strategy - Optum's Aaron Rinehart on Why It's Time for a New Approach”
https://www.databreachtoday.com/interviews/healthcare-cybersecurity-new-strategy-i-3377
Conference Talks & Podcasts
Sample Set of Conference Talks & Podcasts Relevant to this Talk
RSA 2018 - ChaoSlingr: Introducing Security-Based Chaos Testing
https://www.youtube.com/watch?v=wLlME4Ve1go
RSA 2019 - Security Precognition: Chaos Engineering in Incident Response
https://www.rsaconference.com/videos/security-precognition-chaos-engineering-in-incident-response
Craft Conference 2019 - Chaos Engineering in Security Incident Response w/ ChaoSlingr
https://craft-conf.com/speaker/AaronRinehart#
The Linux Foundation 2018 - Open Source Summit North America - OSS Security Chaos Engineering - Driving Transformation, Innovation, and Open Source with Giants - Aaron Rinehart & Kevin Nelson, UnitedHealth Group
SANS SecDevOps Summit 2018 - Total Chaos “How Experimentation Leads to Greater Control”
https://www.sans.org/cyber-security-summit/archives/file/summit-archive-1540327060.pdf
Hacked Podcast: Aaron Rinehart talks Chaos Engineering, ChaoSlinger, and objective monitoring of security components
https://www.owltail.com/podcasts/7568-hacked-into-the-minds-of-cybersecurity-leaders#top-jcpUf
The Modern Security Webinar Series ChaoSlingr: Introducing Security Based Chaos Testing
https://info.signalsciences.com/modern-security-series-chaoslingr-security-based-chaos-testing
The AppSec Podcast - Chaos Engineering and #AppSec (S04E11)
https://www.securityjourney.com/blog/chaos-engineering-and-appsec-s04e11/
Optional: Links to samples of prepared material or outlines ready.
Craft 2019 - Security Chaos Engineering - Security Precognition
https://www.slideshare.net/rinehartas/craft-2019-security-chaos-engineering-security-precognition
DevSecOps & Security Chaos Engineering
https://www.slideshare.net/rinehartas/gdsaustin-devsecops-security-chaos-engineering
AllDayDevOps : DevSecOps & Chaos Engineering: Knowing the Unknown
https://www.slideshare.net/rinehartas/alldaydevops-devsecops-chaos-engineering-knowing-the-unknown
ChaoSlingr: Introducing Security based Chaos Testing
https://www.slideshare.net/rinehartas/chaoslingr-introducing-security-based-chaos-testing
schedule Submitted 1 year ago
Recent Activities
- description Naresh Jain updated proposal Security & Chaos...schedule 4 months ago
- description Aaron Rinehart updated proposal Security & Chaos...schedule 4 months ago
- verified_user Program Committee accepted proposal - "Security & Chaos..." by Aaron Rinehartschedule 6 months ago
Public Feedback