Skip to Main Content

Session

Session

What’s in a Name? From ETL Mappings to Data Pipelines

We've witnessed staggering innovation in distributed data platforms powered by the Big Data and cloud revolutions. We've also seen a shift in the tools used to extract, transform, and load those platforms during these revolutions. This shift is perhaps most striking when we consider that the glossier name "data pipeline" has displaced the more muted name "ETL mapping" for the basic building block of the new data engineering discipline.

In this presentation, I'll make the case that the change in vocabulary is not merely semantic but fundamental to how modern data teams engineer their data. We'll explore two common patterns for building data pipelines: first, templated SQL, and second, DataFrame development. Finally, we'll consider the upsides and downsides of these patterns, but ultimately, I'll argue that either one is a bold and necessary evolution from our ETL tools of the past.
San Antonio 3
June 22nd 09:00 AM
June 22nd 10:00 AM
Analytics
Data Engineering
In-Depth Study
Data Engineer (8-10 years into career), Analytics Manager, Forward Thinker, Senior Data Analyst, Mid-level Data Analyst, Database Designer, Newer Data Engineer
Advanced

About the speaker:

Stewart Bryson
Co-founder & CCO
Qualytics

Stewart is a co-founder and chief customer officer at Qualytics, and a co-founder and former CEO of Red Pill Analytics. He is an internationally recognized writer, speaker, and podcast guest in the data and analytics community and holds numerous community advocacy titles including Snowflake Data Superhero, Oracle Groundbreaker Ambassador, and Oracle ACE Director. Stewart is a past winner of the ODTUG Innovation award and Best Speaker award for Big Data and Data Warehousing.

Release 1.0