This presentation was recorded at YOW! 2019. #GOTOcon #YOW
Tommy Hall - Theatre fan, occasional mountaineer, part time runner, thoroughly nice chap, available in fine bookstores everywhere
ABSTRACT
In all businesses, there is some kind of data pipeline, even if it’s powered by humans working off a shared drive somewhere. Lots of places are better than this - they have workflow systems, ETL pipelines, analytics teams, data scientists, etc - but can they say months later which version of which code is running on what data generated insights?
Can they be reproduced?
What if the algorithms change, do you go back and re-run everything?
Science itself has a reproducibility problem, but it’s worse in most companies, and mistakes can be expensive.
There is a useful subset of data pipelines, let’s call them “pure”, that only depend on the data flowing through them. For pure pipelines, we can use techniques from distributed build systems to allow us to know what code was used for each step, not lose any previous results as we improve our algorithms and avoid repeating work that has been done already.
This talk contains interesting theory but is resolutely practical and with concrete examples in several languages and distributed computation frameworks. [...]
RECOMMENDED BOOKS
Bas P. Harenslak & Julian Rutger de Ruiter • Data Pipelines with Apache Airflow •
James Densmore • Data Pipelines Pocket Reference •
Barr Moses, Lior Gavish & Molly Vorwerck • Data Quality Fundamentals •
Rishu Mehra • What is Data Observability •
Gerardus Blokdyk • Observability Services A Complete Guide •
#DataPipelines #DataPipeline #Prometheus #Grafana #Data #ETLPipelines #Backend #DevOps #Streams #Frontend #TommyHall #Programming #YOWcon
Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at
Sign up for updates and specials at
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
1 view
0
0
5 months ago 00:02:40 1
New features in RTSS beta 2: PresentMon latency analyzer, overlay layouts merging support
5 months ago 00:02:37 1
NVIDIA Omniverse Foundational Technology Montage I GTC Spring 2024 Edition
5 months ago 00:04:49 1
[60 fps] Views of Tokyo, Japan, 1913-1915
5 months ago 00:06:30 1
Sequencer Creation & Crowd Sim Pipeline Unreal 5 | Unreal Live Link 1.3 Tutorial
6 months ago 00:05:22 1
SUPERKIND (슈퍼카인드) ’MOODY’ Official MV
6 months ago 00:08:18 1
AI & Flare. The #1 AI token in the 2024/25 Bull Market?!? #flare #flarenetwork #ai
6 months ago 00:12:51 1
Create mind-blowing AI RENDERINGS of your 3D animations! [Free Blender + SDXL]
6 months ago 00:21:00 1
How do Video Game Graphics Work?
6 months ago 00:22:19 1
When Optimisations Work, But for the Wrong Reasons
6 months ago 00:12:17 1
The Dangerous Propaganda Of Techno-Optimist
6 months ago 01:27:46 1
NASA ARSET: Data Loaders for Training ML Models on Irregularly-Spaced Time-Series Imagery, Part 2/3
6 months ago 00:13:54 4
EV or Gas, What Pollutes More?
6 months ago 01:11:40 13
Экспертность этого видео заставит вас усомниться в...
6 months ago 00:10:53 1
НАУКА ЗА РАЗВОДОМ ЯРМАРОК - и как победить - YouTube
6 months ago 00:06:46 1
DTC: Deep Tracking Control
7 months ago 00:05:25 1
Getting Started with iClone Unreal Live Link 1.3 | Unreal Live Link 1.3 Tutorial
7 months ago 00:23:23 1
$13 Billion Trans-Saharan Gas Pipeline Can Change Africa Forever!
7 months ago 02:13:35 19
Let’s build the GPT Tokenizer
7 months ago 00:06:48 1
Production Ready Face Re Aging for Visual Effects
7 months ago 05:03:32 1
Coding Stable Diffusion from scratch in PyTorch
7 months ago 01:58:25 1
#FaangTalk 18 - Систем дизайн: Дата Пайплайны
7 months ago 00:21:24 1
Reinventing a Character Creation Pipeline Using Landmarking, Simulation, and Shared Character Data