Generative Testing Patterns
Tests are good. Generative tests are better. But we don't write them often enough. Why not?
Generative testing isn't new; John Hughes' QuickCheck is now implemented in 30 languages. Yet, it is still not an every day tool for most software teams. Why not?
I have seen that maintaining a generative, or simulation testing suite can require the full-time attention of a few senior engineers. And over time, the tests become complex, brittle, ineffective, and hard to diagnose.
But isn't generative testing supposed to make testing simpler, and more effective?
I believe that it requires a rather different paradigm of thought from what we're used to with traditional testing. And recognising this paradigm shift can help us leverage it effectively.
I'll discuss a holistic perspective of generative testing to elucidate this thought paradigm.
Using a practical scenario, I'll discuss the patterns that emerge from this thinking, and how they address the concerns of making generative tests more effective, and maintainable.
Outline/Structure of the Talk
- Basis of the talk [2m]
- Outline of a practical system, based on which the patterns can be understood [2m]
- Benefits and problems of the current state of generative tests [5m]
- Setting up the right terminology for the remainder of the talk [5m]
- Detailed patterns in [~25m]
- generating input
- executing the tests
- assertions, properties
- diagnosis
- Summary [2m]
Learning Outcome
I'll try to answer the following questions through the talk:
- How to minimise the regression in our tests as the source changes?
- How to model dependencies between actions, and generate meaningful flows to test?
- How to model state in tests, while testing complex stateful systems?
- What are some generic system invariants we can use broadly?
- How do we trawl through thousands of test results and diagnose them?
- How do we keep tests deterministic, and repeatable?
Target Audience
People who are interested in testing, generative, or property testing, etc.
Prerequisites for Attendees
- Know about quick check, and property based tests
- Have some experience in writing and running generative tests
Links
Strangeloop 2018 |
|
Dutch Clojure Days 2018 |
|
Fragments 2017 |
|
EuroClojure 2016 |
|
RootConf 2016 |
|
Functional Conf 2015 |
|
Functional Conf, 2015 |
schedule Submitted 2 years ago
People who liked this proposal, also liked:
-
keyboard_arrow_down
Abhinav Sarkar - Many Ways to Concur
45 Mins
Talk
Intermediate
Easy concurrency is one of the main prophesied benefits of the modern functional programming (FP) languages. But the implementation of concurrency differs widely between different FP languages. In this talk, we shall explore the methods and primitives of concurrency across three FP languages: Haskell, Erlang, and Clojure (with core.async).
We shall learn about and compare the trade-offs between
- the green threads and STM channels oriented concurrency of Haskell
- everything-is-a-process and message-passing actor pattern of Erlang
- macro-based CSP transformation of Clojure/core.async