The tremendous push towards streaming pipelines lately has led some to believe that the days of batch processing are either over or will soon be coming to an end. But is this actually true? In this post, I explore the reasons for keeping some of your batch pipelines.
Learn about what SQLMesh has to offer when it comes to evolving really large datasets.
The practice of augmenting SQL code with Jinja templates has gained significant traction and is consistently growing in popularity. However, is this truly the optimal approach? In this article, we examine the common challenges that arise when combining SQL and Jinja, while also suggesting a more effective method for metaprogramming in SQL.
Are you using dbt and have heard of SQLMesh, but aren't quite sure what it is? This post will teach you the basics of SQLMesh from a dbt user's perspective to help get you up to speed. Part 2 of 2.
Introduces the SQLMesh's open source CI/CD bot and how it can be used to automate the deployment of data pipelines.
Are you using dbt and have heard of SQLMesh, but aren't quite sure what it is? This post will teach you the basics of SQLMesh from a dbt user's perspective to help get you up to speed.
Explore the two fundamental methods of loading data (Full Refresh and Incremental). Learn about two common approaches for determining what data to load incrementally and how they compare.
Although companies heavily rely on SQL / data pipelines to power applications and make decisions, testing is usually an afterthought. However, systematic testing of data is critical to creating trustworthy and reliable pipelines. This post highlights the three categories of tests for data: unit tests, table diffs, and audits.
Comparing development experience and runtime efficiency of dbt and SQLMesh.
Exploring what semantic understanding is and how it helps improve user experience when using SQL.