A machine learning project usually includes many moving parts, such as data processing, model training, model inference and model deployment. Because of the iterative and exploratory nature of developing a machine learning project, using a pipeline can make development faster and more effective.

Similar to the ETL process for data, machine learning projects can also have a development pipeline that pre-defines high-level building blocks. These building blocks work to establish a clear structure for the machine learning workflows. Through using pipelines, a large amount of the machine learning workflow can be automated, improvements to the models performance can be tracked and collaboration between engineers is simplified.

This talk will discuss some high-level building blocks in pipelines for ML projects: the concepts of DAG (Directed Acyclic Graph) and how it relates to machine learning pipelines; experience of building ML pipelines for our current project, with light-weight framework like `consecution` in Python and comprehensive framework like `Tensorflow Extended` with `Airflow`.