We at Tobiko Data love open source. We love using, contributing to, and creating open source projects. Our team manages SQLMesh and SQLGlot and want to share some thoughts on how we like to maintain them and how we plan to build a thriving open source community around them.
Responding quickly
Filing a well thought out issue takes effort, and it's common for issues to receive no response from a maintainer, which can be frustrating to the submitter. One of the things we look for when deciding whether or not to use an open source package is the responsiveness of projects and how many outstanding issues there are.
Most people run into bugs and move on. Only a small percentage of users will take the time and effort to create an issue. The percentage is further reduced if users are afraid that their time will be wasted by an ignored issue. It's not easy keeping the issues queue clear, but it builds trust in your project. Here are some of the things our team does in order to keep unresolved issues empty.
Having a clean issues queue can be very satisfying!
Avoid scope creep
We don't have the resources or time to implement every feature request. We've learned to say no (but in a polite way).
If it's something that we can see benefitting the project but have no plans on implementing, we close the issue and explain that it is not planned but that we would happily guide and accept a PR. This gives the reporter a clear resolution and also an option to contribute if the feature is very important to them.
Any response is better than nothing
Sometimes an issue is something you plan to do but can't be done immediately. In those cases, it's valuable to respond and explain the situation and encourage the reporter to contribute a PR (with assistance) if it's feasible. This way, the reporter will feel acknowledged and possibly empowered to fix the issue themselves.
Encouraging contribution
There's no greater gift in the open source world than a nicely crafted PR from a first time contributor. However, submitting a PR can be intimidating and potentially time consuming for both contributors and reviewers.
Although this process can be difficult, we believe it's ultimately worth it in the long run. Not only do we encourage PR's but we provide feedback and direction in order to foster rapid iteration.
If there's positive momentum but the iteration cycle is becoming tedious, we'll merge it as is, thank them for their contribution, and take it to the finish line. This way the contributor gets the credit for their contributions, the community gets the new features / fixes, and we maintain our coding style / standards.
There's no need to drag out a PR that's 90% there due to styling / nits.
Test coverage / Linting
Although automated testing and linting are important regardless of open source, it's especially important when your contributors may not be as committed to or familiar with the code base as you are.
Having robust testing lowers the bar for contribution and reduces the chance of something going wrong. Most contributors are not going to manually test and ensure that their changes have no issues, and automated tests allow us to accept their contributions without being bogged down by the effort to manually validate each one.
Staying motivated
Open source projects only stay alive if they are active and maintained. Burning out or losing interest in a project are real factors that lead to projects going stale.
By building an active community, you can recruit other maintainers to keep the project sustainable. Additionally, it helps if you have a project that is genuinely enjoyable to work on or find a way to get paid for it. Getting paid for open source is not easy but is possible. You can potentially get your employer to allow you to work on it part time, receive donations, or build a company around it.
Future
We at Tobiko have built our careers on top of open source tools like Spark, Airflow, and Flink, and owe a lot to the open source community. We're taking steps now to hopefully build both a healthy open source community and an economically sustainable business model.
For example, our initial enterprise offering will be an observability platform for SQLMesh. It includes an enhanced version of SQLMesh that logs additional metrics and a self hosting UI to consume and act on those metrics. With SQLMesh Observability, you'll be able to track run times, costs, and custom metrics. Additionally you'll be able to pinpoint exactly which change caused issues in systems, allowing you to quickly triage and fix problems with data deploys.
SQLMesh Observability is built to enhance open source.
We're just beginning our journey and know it's going to be a challenge. But we're inspired by companies like Confluent and Databricks, who have built successful businesses on top of amazing open source technologies.
We'd love to hear your thoughts on this subject. You can chat with us on our Slack community.