Generating and Running 1,000,000 Selenium tests in 60 minutes
Writing (or recording) tests manually is a tedious, robotic, and dangerous practice. It combines the worst of human nature (it's easy to avoid, easy to forget about edge cases, and difficult to maintain) with the worst of computers (no creativity in trying to break the tests, no understanding of how the underlying system works). What if we could assert high-level properties about our product (in this case, a web site), and then teach the computer to generate tests in order to try to break those assertions?
Using generative, property-based testing combined with a setup carefully designed to handle concurrent tests (absolutely critical in scaling out coverage while maintaining a trustable build) and a whole lot of computing power, we'll look at how computers can actually be far better than developers at finding bugs in our systems (bug that our users would eventually hit!). We can use this as a first-line defense against regressions in our system.
We'll end by looking at some amazing extensions to this technique that aren't currently widespread (Concolic testing, predictive testing) but may have significant impact on testing practices in the coming years.
Outline/Structure of the Talk
- Challenges with traditional test writing/recording
- How many states are actually possible when using a SPA site?
- Examples of important but easily-missed real-world use cases
- Don't write tests, generate them!
- Look at work done in the Erlang, Haskell, and Clojure world (Simplecheck, Quickcheck, and test.check) and examples of success cases
- Show how shrinking can take and hour-long error example and automatically turn it into an 8 step-repro pre-written test cases
- Can we generate full web tests?
- What does it takes to teach of computer to autonomously navigate a site?
- What kind of properties can we assert about our site as the computers are navigating it?
- Challenges
- How do we make sure the right properties are tested?
- What kind of computing power do we need to pull this off on the server side?
- How can we get enough properly-prepared browsers?
- Demo time!
- Future areas (not possible yet, but at the very cutting edge of academic research)
- Concolic testing - what if we can combing generative (concrete) testing and symbolic execution to automatically (statically) examine our code, and generate test inputs that make sure to exercise every code path automatically? Mind-boggling stuff, that's what!
- Predictive testing - In addition to generating tests, what about recording actual user runs across a site (in the form of analytics even) and using that as seed to test variation of that? Run the last 10,000 sessions on your site against your new code before deploying it, and then run another 990,000 variations of it to make sure you're covering the way users use your site.
Learning Outcome
New techniques (and possibilities) in being effecting about testing:
- Declaring properties about a site and letting the computer try to break those properties
- Infrastructure and architectural requirements to achieve the above
- What might be possible with more sophisticated tooling
Target Audience
Developers, Q&A engineers, testing infrastructure engineers
Video
Links
This will be the first public talk on this topic, and it should be big! I can release some material beforehand, leading up to the conference if that would help however.
Anand Bagmar
Manoj Kumar
Dave Haeffner
Naresh Jain
nir
