Bringing Software Engineering Rigor to Data [Video]
Leverage software engineering practices for data engineering and learn to measure key performance metrics to help build more robust and reliable data pipelines.
This is a recording of a breakout session from AWS Heroes at re:Invent 2022, presented by AWS Hero Zainab Maleki. Posted with permission.
In software engineering, we've learned that building robust and stable applications has a direct correlation with overall organization performance. The data community is striving to incorporate the core concepts of engineering rigor found in software communities but still has further to go. This talk covers ways to leverage software engineering practices for data engineering and demonstrates how measuring key performance metrics could help build more robust and reliable data pipelines. This is achieved through practices like Infrastructure as Code for deployments, automated testing, application observability, and end-to-end application lifecycle ownership.
Opinions expressed by DZone contributors are their own.