In this full-day workshop, you will learn applied big data solution patterns. most often, but not always using the public cloud. We’ll cover Amazon Web Services and Google Cloud Platform, and work with in small groups to design data pipeline architectures for common scenarios.

RESERVE YOUR SEAT NOW

Dr.Bauer and her team have been working to build genome-scale data pipelines that address the computational challenges and limits present in today’s cancer genomic (bioinformatics) data workflows.

Dr. Bauer and her team have built solutions which use modern architectures, such as serverless (AWS Lambda) and also customised machine learning on Apache Spark. AWS Community Hero and cloud architect Lynn Langit is also collaborating with the CSIRO team to push solutions at the cutting edge of bioinformatic research which best utilise advances in cloud technologies..

In this demo-filled session Lynn and Denis will discuss and demonstrate some of the latest cloud data pipeline work that they’ve been working together to build out for the bioinformatics community.

RESERVE YOUR SEAT NOW

In this full-day workshop, you will learn applied big data solution patterns. most often, but not always using the public cloud. We’ll cover Amazon Web Services and Google Cloud Platform, and work with in small groups to design data pipeline architectures for common scenarios.

WHAT WILL I LEARN

We’ll discuss database categories, products and understand actual cost to use one or more database types in your solution(s). Using combinations of Big Relational (for hot, cold or cold data), and one or more types from the NoSQL category (Key-value, Wide-column, Document or Graph) and sometimes even along with NewSQL, we’ll design implementable solutions. We’ll also cover what types of problems are best suited to the Hadoop ecosystem (including use of Spark and the associated libraries).

RESERVE YOUR SEAT NOW

Lynn Langit shares lessons learned and cloud data pipeline patterns via examples from work she’s doing with CSIRO Bioinformatics Australia. The team there, led by Dr. Denis Bauer, is analyzing a number of large genomic datasets.

First, Lynn examines real-time analysis with cloud-based solutions. Keeping runtime constant can be challenging for problems that vary in complexity, such as genome engineering. The CSIRO GT-Scan2 tool works by instantaneously recruiting additional Lambda functions as the complexity increases. It was built using a microservices pattern (serverless) using AWS services.

Next, Lynn will demo a Jupyter notebook which shows how genomic research can leverage Apache Spark to massively parallelize the generation of random forests to identify disease genes efficiently.She’ll discuss the pipeline’s use of an OSS library written by the team at CSIRO (VariantSpark).

VariantSpark can analyze 3,000 samples with 80 million features in under 30 minutes. This pipeline enables real-time diagnosis by finding similar patients. This platform is contributing to motor neuron disease research (publicized by the Ice Bucket Challenge) in Australia.

Lynn Langit shares lessons learned and cloud data pipeline patterns via examples from work she’s doing with CSIRO Bioinformatics Australia. The team there, led by Dr. Denis Bauer, is analyzing a number of large genomic datasets.

First, Lynn examines real-time analysis with cloud-based solutions. Keeping runtime constant can be challenging for problems that vary in complexity, such as genome engineering. The CSIRO GT-Scan2 tool works by instantaneously recruiting additional Lambda functions as the complexity increases. It was built using a microservices pattern (serverless) using AWS services.

Next, Lynn will demo a Jupyter notebook which shows how genomic research can leverage Apache Spark to massively parallelize the generation of random forests to identify disease genes efficiently.She’ll discuss the pipeline’s use of an OSS library written by the team at CSIRO (VariantSpark).

VariantSpark can analyze 3,000 samples with 80 million features in under 30 minutes. This pipeline enables real-time diagnosis by finding similar patients. This platform is contributing to motor neuron disease research (publicized by the Ice Bucket Challenge) in Australia.