Data Warehouse!

Data warehouses are systems that contain relational data from the past, where we perform data transformations or data cleaning with ETLs. Data warehouses commonly used to find answers to existing questions and to support decision processes.

As it is known, data warehouses have a long history in business intelligence application and this technology has been developing since the 1980s. These developments have created systems that can process larger data and MPP (Massively Parallel Processing) architectures. With this accumulated experience, we now know the needs and how to meet them.

However, although data warehouses are a good solution for structured data, many large enterprises now have to work with unstructured or semi-structured data. In particular, these are data with high speed and volume. As we know, big data is a world which Data Lakes and Hadoop come into play, and data are not structured. There are basic examples such as data flowing from IOT devices and social media platforms. In this case, data warehouses are not suitable both for use and in terms of cost.

Data Lake!

In this case, as companies began to collect big data, they began to design a single system to accommodate different analytics products and workloads. This created data lakes. However, the problem here was that data lakes do not support transaction flow, and they lack data quality and consistency. Moreover, the needs of the two worlds that we call Data Warehouse and Data Lake are of course different. While one of them is completely under control, the other is a field that requires experimental work.

For this reason, companies like Microsoft, which develop technology, have turned to developing flexible and high performance systems.

They started working on systems that will include all of the reporting, real-time data, data science and machine learning studies.

We can say that DataBricksis the pioneer of this architectural environment. Thus, Azure Synapse Analytics service integrated with Microsoft’s Azure DataBricks became a similar Lakehouse model.

#databricks #azure-synapse-analytics #data-lake #data lakehouse