Dreamhaven’s next-generation data pipelines using SQLMesh

When Aleks took on his new director role at Dreamhaven, he needed to develop more data pipeline engineering. He was lucky enough to have the opportunity to build his data pipelines from scratch for this new division called player analytics. When he and his team started setting up their Databricks warehouse to handle the complexity of their gaming data, they started running into some problems.

Challenges Dreamhaven faced

After trying several options, the team still encountered issues managing and scaling their data pipelines. In particular, their main three areas of challenge were:

1. Data streaming. They had issues around streaming because they needed to be able to change and modify the data, and you can’t stream from a table that has been modified.
2. Incrementality. The data needed to be loaded incrementally to manage the expected scale their pipelines would have to manage once their games were released.
3. Forward-only plans. Other competitors on the market, such as dbt, do a full reload every time you need to modify or update a pipeline, which, again, just wasn’t going to work with the amount of data they expected to have. These were a must because otherwise, it would frequently cause them to re-process very large tables, significantly adding to costs and SLAs.

But really, Aleks wanted to set this team up for success. Having been in the data space for 10 years and being well-versed in pipelines that needed to be handled better or were difficult to change, Aleks wanted to build something better from the beginning. You know, it's like, ‘hey, this is our ETL, and we don't even know what it looked like before, and there was no code, and there's no version control, and changes were done on production and we're just gonna drop the table and rebuild it’.

To keep Dreamhaven competitive, Aleks wanted to avoid repeating the mistakes he had seen in the past and construct a foundation that would scale smoothly. He wanted to test out and use the next generation of data tools to see how the space was evolving, and he was excited about trying out the paradigm of Data Ops.

Dreamhaven's switch to SQLMesh

Aleks realized that SQLMesh might be the solution they sought. SQLMesh's built-in incrementality and forward-only plans immediately solved two of their three major pain points.

Compared to dbt and dlt’s, SQLMesh immediately started saving them money on their cloud costs. Using Databricks dlt’s to get the same features as in SQLMesh, Aleks found: ...they’re about 3.5X the cost… So this is Databricks additional cost on top of compute. After switching to SQLMesh, they could run those same pipelines on their regular compute at the base price. So that's a pretty significant cost savings for our pipeline compute.

On top of the cost savings, Aleks and his team found SQLMesh easier to develop with than Databrick’s dlts. Since the dlts are written in notebook form, his team couldn't even test them. You just have to write the whole thing and then schedule it and see what happens, right? Which is very hard to iterate on, especially for somebody that doesn't have as much experience with it. With SQLMesh, Aleks’ team can make a change using notebooks, the SQLMesh UI, or directly from the CLI. Then, immediately run that plan in a test environment—no scheduling or waiting to see if those changes work.

Dreamhaven's Favourite SQLMesh features

Another feature that made Aleks’ job more manageable and empowered his team was SQLMesh’s Virtual Data Environments. Initially, they planned to have multiple Databricks environments, dev prod, and something in between. This came from their need to keep the data segregated; however, with SQLMesh’s architecture, they only needed one environment, allowing them to simplify their infrastructure and reduce the cost.

This feature further helped them because there's not a really good way to elevate code in Databricks from Dev to production. And that's something we experienced as we're testing… I don't want to bore you with the details, but it's not simple…. On the other hand, with SQLMesh, because it intelligently understands SQL and what is being changed, they could easily run just a SQLMesh plan in development locally and then promote it to their one Databricks environment. And there's no duplication of data… You don't have to store them twice at all. It just makes sense.

For Aleks and his team, using SQLMesh resulted in several significant bonuses: using SQLGlot in SQLMesh and its ability to parse SQL, the team's responsiveness, and SQLMesh’s unit tests.

SQLGlot allows you to seamlessly “translate” between different SQL syntaxes. This means that if Aleks and his team want to switch to a different data warehouse in the future, they can do so without a major migration because they can rely on SQLMesh to translate their SQL to the new syntax effectively.

The Tobiko Data team has a track record of efficient bug bashing and quick PR turnaround times. The level of responsiveness and support the Tobiko team provided surprised and impressed Dreamhavens' team. We didn't expect that… and it was like, ‘Wow, this is really nice and refreshing’, and ‘we found a bug’, and it's fixed within two hours, and they cut us a new build…. we will never have that support with another larger organization.

Finally, the Dreamhaven team's ability to create unit tests for data was a huge step forward. Data models rarely have checks and balances like true software coding, which can lead to issues if something fundamentally changes.

Unit tests— to me— that was like…. Never being a software developer it was just such a foreign thing to have unit tests and to see if your code will break some dependency. That's just yeah, it's crazy. Eye-opening that that could happen in our discipline.

Benefits of SQLMesh

Since implementing SQLMesh in their data pipelines, they have never experienced any major downtime caused by their data. They have never had to revert their changes post-implementation. Aleks credits this ...largely because every change is thoroughly tested before it gets into production.

With Dreamhaven’s current data and pipelines, they value SQLMesh at half a headcount. This team of three can do the work of three and a half people because of the increased productivity of the SQLMesh tool. Given the average Data Engineering salary, this saves them approximately $100,000 per year on the productivity side. Further, they expect this to grow as they scale their team and data and fully deploy SQLMesh into production.

Conclusion

SQLMesh provided Dreamhaven with a data engineering solution that was cost-effective, easy to develop, and supported a forward-looking approach to data operations. The implementation of SQLMesh has allowed Dreamhaven to avoid major downtimes and implement changes confidently, thanks to thorough testing. As Dreamhaven continues to grow and develop its games, SQLMesh remains a pivotal component in its data infrastructure, enabling them to focus on delivering quality gaming experiences without being hindered by data engineering complexities.