There was an awesome debate on DBT’s Slack last week discussing mainly two things:
- The state of open-source alternatives to Fivetran
- Whether an open-source (OSS) approach is more relevant than a commercial software approach in addressing the data integration problem.
If you’re already on DBT’s Slack, here is the thread’s URL. Even Fivetran’s CEO was involved in the debate.
In this article, we want to discuss the second point and go over the different points mentioned by each party. The first point will come in another article. It’s more relevant to discuss whether an OSS approach makes sense before drilling down into the different alternatives.
We’ll go over the main challenges that companies face and see which approach fits best. We’ll call “commercial companies” the ones with a commercial software product, and “OSS companies” the ones with an open-source approach.
1. Having a large number of high-quality well-maintained pre-built connectors
This might be the most challenging part for the open-source approach, but there are actually choices that can make an OSS approach even stronger than a commercial one on that matter.
Commercial approach
In this case, a company supports a limited number of connectors (the most used ones) and actively maintains them. They know when there is a schema change and when they need to update the connector, and can be pretty responsive on that, if the organization is responsive and scales well.
However, the more connectors, the more difficult it is for a commercial company to keep the same level of maintenance across all connectors. In an ideal world, the organization will grow linearly with the number of connectors. But most often, there are inefficient processes, so every organization will reach a limit. The more efficient they are, the higher the limit.
#open-source #data-integration #data #big-data #database #business-models #software-development #solving-data-integration
