The ‘science’ in data science refers to the underlying philosophy that you don’t know what works for your business until you make changes and rigorously measure impact. Rapid experimentation is a fundamental characteristic of high functioning data science teams. They experiment with models, business processes, user interfaces, marketing strategies, and anything else they can get their hands on. In this talk I will discuss what data platform tooling and organizational designs support rapid experimentation in data science teams.

Apache Spark is a general purpose distributed computing framework for distributed data processing. With MLlib, Spark's machine learning library, fitting a model to a huge data set becomes very easy. Similarly, Spark's general purpose functionality enables application of a model across a large collection of observations. We'll walk through fitting a model to a big data set using MLlib and applying a trained scikit-learn model to a large data set.