Data Load Patterns 101: Full Refresh and Incremental
Explore the two fundamental methods of loading data (Full Refresh and Incremental). Learn about two common approaches for determining what data to load incrementally and how they compare.
Explore the two fundamental methods of loading data (Full Refresh and Incremental). Learn about two common approaches for determining what data to load incrementally and how they compare.
Although companies heavily rely on SQL / data pipelines to power applications and make decisions, testing is usually an afterthought. However, systematic testing of data is critical to creating trustworthy and reliable pipelines. This post highlights the three categories of tests for data: unit tests, table diffs, and audits.
Comparing development experience and runtime efficiency of dbt and SQLMesh.
Exploring what semantic understanding is and how it helps improve user experience when using SQL.
Writing efficient and correct incremental pipelines is challenging. While many folks do take on the challenge of writing incremental models, it is viewed as an advanced use case which could discourage teams from adopting incremental loads.
Categorizing changes in SQL queries to automatically establish data contracts between producers and consumers.
Creating highly efficient and scalable development environments for data.
Creating highly efficient and scalable development environments for data.
We are excited to share SQLMesh, an open-source DataOps framework that brings the benefits of DevOps to data teams.