Gordon  Matlala

Gordon Matlala

1670613420

Data Mesh Architecture and its Benefits

Introduction to Data Mesh

Data mesh builds a layer of connectivity that takes away the complexities of connecting, managing, and supporting data access. It is a way to fasten the data together that is held across multiple data silos. It combines the data distributed data across different locations and organizations. It provides data that is highly available, easily discoverable, and secure. It is beneficial in an organization where a team generates data from many data-driven use cases and access patterns in it.

We can use it, like when we need to connect cloud applications to sensitive data that lives in a customer's cloud environment. Also, when we need to create virtual data catalogs obtained from various data sources that can't be centralized. There is also a situation in which it is used, for instance, when we create virtual data warehouses or data lakes for analytics and ML training that can be done without consolidating data into a single repository.

What is Anthos Service Mesh?

It is a fully managed service mesh that is used for complex microservices architectures. It is a suite of tools that monitor and manage a reliable service mesh on-premises or Google Cloud. It's powered by Istio, which is a highly configurable and one of the powerful open-source service mesh platforms that have tools and features that enable industry best practices. It defines and manages configuration centrally at a higher level. It is deployed as a uniform layer across the full infrastructure. Service developers and operators can use a rich feature set without making a single change to the application code.

Anthos Service Mesh relies on Google Kubernetes Engine (GKE ) GKE On-Premise Observability features. Microservices architectures provide many benefits, but on the other hand, there are challenges like added complexity and fragmentation for different workloads. It solves the problem like it unburdens operations and development teams by simplifying service delivery across the board, from traffic management and mesh telemetry to securing communications between services.

What are the features of Anthos Service Mesh?

Here are some of the features of Anthos Service Mesh

Deep visibility built-in [beta]

Anthos Service Mesh is integrated with Cloud Logging, Cloud Monitoring, and Cloud Trace that provides many benefits, such as monitoring SLOs at a per-service level and setting targets for latency and availability.

Easy authentication, encryption

Anthos Service Mesh ensures easy authentication and encryption. It transport authentication through MTLS (Mutual Transport Layer Security) has never been more effortless. It secures service-to-service as well as end-user-to-service communications with just a one-click mTLS installation or incremental implementation.

Flexible authorization

It provides flexible authorization like we only need to specify the permissions after that grant access to them at the level that we choose, from namespace down to users.

Fine-grained traffic controls

Anthos Service Mesh opens up many traffic management features as it decouples traffic flow from infrastructure scaling and includes dynamic requests. Routing for A/B testing, canary deployments, and gradual rollout, and that also all outside of your application code.

Failure recovery out of the box

It provides many critical failure-recovery features out of the box, to configure dynamically at runtime.

What is Azure Service Fabric Mesh?

Azure Service Fabric Mesh helps the developers deploy microservices applications, and there is no need to manage virtual machines, storage, or networking. The applications hosted on Service Fabric Mesh run and scale without worrying about the infrastructure powering it. Service Fabric Mesh has clusters of many machines, and every one of these cluster operations is hidden from the developer.

You only need to upload the code and mention the resources we need, availability requirements, and resource limits. It automatically allocates the infrastructure and handles infrastructure failures as well, and we need to make sure the applications are highly available. We need to take care of the health and responsiveness of the application and not the infrastructure. Azure Service Fabric has three public offerings: Service Fabric Azure Cluster service, Service Fabric Standalone, and Azure Service Fabric Mesh service.

What is AWS App Mesh?

AWS App Mesh helps to run services by providing consistent visibility and network traffic controls. For services built across multiple computing infrastructure types. App Mesh abolishes the necessity to update the application code. To vary how monitoring data is collected or traffic is routed between services. It configures each service to export monitoring data and implements consistent communications control logic across your application. When any failure occurs or when code changes must be deployed, therein situation makes it easy. To pinpoint the precise location of errors quickly and automatically reroute network traffic.

What are the advantages of AWS App Mesh?

Following are the advantages of AWS App Mesh: Provides End-to-end visibility because it captures metrics, logs, and traces from all of your applications. We can combine and export this data to Amazon CloudWatch, AWS X-Ray, and community tools for monitoring, helping to quickly identify and isolate issues with any service to optimize your entire application.

Ensure High Availability

App Mesh gives controls to configure how traffic flows between your services. Implement easily custom traffic routing rules to ensure that every service is highly available during deployments, after failures, and as your application scales.

Streamline Operations

App Mesh configures and deploys a proxy that manages all communications traffic to and from your services. This removes the requirement to configure communication protocols for every service, write custom code, or implement libraries to control the application.

Enhance Any Application

Users can use App Mesh with services running on any compute services like AWS Fargate, Amazon EKS, Amazon ECS, and Amazon EC2. App Mesh can also monitor and control communications for monoliths running on EC2. Teams running containerized applications, orchestration systems, or VPCs as one application with no code changes.

Hybrid Deployments

To configure a service mesh for applications deployed on-premises, we can use AWS App Mesh on AWS Outposts. AWS Outposts could be a fully managed service that extends AWS infrastructure, AWS services, APIs, and tools to virtually any connected site. With AWS App Mesh on Outposts, you'll provide consistent communication control logic. For services across AWS Outposts and AWS cloud to simplify hybrid application networking.

Data Mesh vs Data Lake

Given below are the differences between Data Mesh and Data Lake.

  • The data lake is a storage repository. That holds a vast amount of raw data in its native format. The hierarchical data warehouse stores data in files or folders. Whereas the data lake uses a flat architecture to store data.
  • The advantage of the data lake is that it is a Centralized, singular, schema-less data store with raw (as-is) data as well as massaged data.
  • The Mechanism for fast ingestion of data with appropriate latency
  • It helps to map data across various sources and give visibility and security to users
  • Catalog to find and retrieve data
  • Costing model of centralized service
  • Ability to manage security, permissions, and data masking
  • The main difference between data mesh and data lake is that it is decentralized ownership in which domain teams usually consider their data a byproduct that they don't own because a data lake is centralized ownership of that raw data.

How is Data Mesh different from Data Fabric?

  • Data Fabric integrates data management across cloud and on-premises to accelerate digital transformation. It helps deliver consistent and integrated hybrid cloud data services that help data visibility and insights, data access and control, and data protection and security.
  • Data Fabric and its difference is that Data fabric allows clear access of data and sharing of data across distributed computing systems by means of a data management framework that is single, secured, and controlled.
  • But Data Mesh follows a metadata-driven approach and is a distributed data architecture supported by machine learning capabilities. It is a tailor-made distributed ecosystem with reusable data services, a centralized governance policy, and dynamic data pipelines.

What are the benefits of Data Mesh?

  1. It provides agility. In this, each node works independently. The node is containerized and can be deployed as soon as any changes are ready.
  2. Construct and deploy new nodes to the mesh, whenever new data arises. Many portals and teams can access the same node, allowing the organization to scale the data mesh. This way, it provides scalability.
  3. Use it under various circumstances, like connecting cloud applications to sensitive data that lives in a customer's on-premise or cloud environment. Use it while creating virtual data catalogs from various data sources. We need to create virtual data warehouses or data lakes for analytics and machine learning training without consolidating data into a single repository.

Conclusion

A data mesh allows the organization to escape the analytical and consumptive confines of monolithic data architectures and connects siloed data. To enable machine learning and automated analytics at scale. It allows the company to be data-driven and give up data lakes and data warehouses. It replaces them with the power of data access, control, and connectivity.

Original article source at: https://www.xenonstack.com/

#data #architecture 

What is GEEK

Buddha Community

Data Mesh Architecture and its Benefits
 iOS App Dev

iOS App Dev

1620466520

Your Data Architecture: Simple Best Practices for Your Data Strategy

If you accumulate data on which you base your decision-making as an organization, you should probably think about your data architecture and possible best practices.

If you accumulate data on which you base your decision-making as an organization, you most probably need to think about your data architecture and consider possible best practices. Gaining a competitive edge, remaining customer-centric to the greatest extent possible, and streamlining processes to get on-the-button outcomes can all be traced back to an organization’s capacity to build a future-ready data architecture.

In what follows, we offer a short overview of the overarching capabilities of data architecture. These include user-centricity, elasticity, robustness, and the capacity to ensure the seamless flow of data at all times. Added to these are automation enablement, plus security and data governance considerations. These points from our checklist for what we perceive to be an anticipatory analytics ecosystem.

#big data #data science #big data analytics #data analysis #data architecture #data transformation #data platform #data strategy #cloud data platform #data acquisition

Gerhard  Brink

Gerhard Brink

1620629020

Getting Started With Data Lakes

Frameworks for Efficient Enterprise Analytics

The opportunities big data offers also come with very real challenges that many organizations are facing today. Often, it’s finding the most cost-effective, scalable way to store and process boundless volumes of data in multiple formats that come from a growing number of sources. Then organizations need the analytical capabilities and flexibility to turn this data into insights that can meet their specific business objectives.

This Refcard dives into how a data lake helps tackle these challenges at both ends — from its enhanced architecture that’s designed for efficient data ingestion, storage, and management to its advanced analytics functionality and performance flexibility. You’ll also explore key benefits and common use cases.

Introduction

As technology continues to evolve with new data sources, such as IoT sensors and social media churning out large volumes of data, there has never been a better time to discuss the possibilities and challenges of managing such data for varying analytical insights. In this Refcard, we dig deep into how data lakes solve the problem of storing and processing enormous amounts of data. While doing so, we also explore the benefits of data lakes, their use cases, and how they differ from data warehouses (DWHs).


This is a preview of the Getting Started With Data Lakes Refcard. To read the entire Refcard, please download the PDF from the link above.

#big data #data analytics #data analysis #business analytics #data warehouse #data storage #data lake #data lake architecture #data lake governance #data lake management

Data Lake and Data Mesh Use Cases

As data mesh advocates come to suggest that the data mesh should replace the monolithic, centralized data lake, I wanted to check in with Dipti Borkar, co-founder and Chief Product Officer at Ahana. Dipti has been a tremendous resource for me over the years as she has held leadership positions at Couchbase, Kinetica, and Alluxio.

Definitions

  • A data lake is a concept consisting of a collection of storage instances of various data assets. These assets are stored in a near-exact, or even exact, copy of the resource format and in addition to the originating data stores.
  • A data mesh is a type of data platform architecture that embraces the ubiquity of data in the enterprise by leveraging a domain-oriented, self-serve design. Mesh is an abstraction layer that sits atop data sources and provides access.

According to Dipti, while data lakes and data mesh both have use cases they work well for, data mesh can’t replace the data lake unless all data sources are created equal — and for many, that’s not the case.

Data Sources

All data sources are not equal. There are different dimensions of data:

  • Amount of data being stored
  • Importance of the data
  • Type of data
  • Type of analysis to be supported
  • Longevity of the data being stored
  • Cost of managing and processing the data

Each data source has its purpose. Some are built for fast access for small amounts of data, some are meant for real transactions, some are meant for data that applications need, and some are meant for getting insights on large amounts of data.

AWS S3

Things changed when AWS commoditized the storage layer with the AWS S3 object-store 15 years ago. Given the ubiquity and affordability of S3 and other cloud storage, companies are moving most of this data to cloud object stores and building data lakes, where it can be analyzed in many different ways.

Because of the low cost, enterprises can store all of their data — enterprise, third-party, IoT, and streaming — into an S3 data lake. However, the data cannot be processed there. You need engines on top like Hive, Presto, and Spark to process it. Hadoop tried to do this with limited success. Presto and Spark have solved the SQL in S3 query problem.

#big data #big data analytics #data lake #data lake and data mesh #data lake #data mesh

Gage  Monahan

Gage Monahan

1618446720

Data Mesh concept - buzzzzz or not?

With the growing number of data sources and need for agility, a decentralized data architecture concept- Data Mesh can be explored to enforce data quality and governance adherence. Data Mesh achieves this via decentralizing the data responsibility to domain level and making high quality transformed data only available as a product.

Every year more data is produced globally. This holds also for companies: more details than ever are recorded from customers, partners, transactions, products and supply chain resulting in more data. According to IDC , “the global datasphere will grow from 45 zettabytes in 2019 to 175 by 2025”. This data forms the raw material from which organizations are drawing valuable, actionable insights. But the collection, integration and governance of this data is still one of the main challenges and inhibitors as established in recent research by Deloitte.

#data-mesh #data-governance #data-lake #data-mess-to-a-data-mesh #hackernoon-top-story #data-mesh-concept

Gerhard  Brink

Gerhard Brink

1621568700

Data Mesh – Rethinking Enterprise Data Architecture

In this age where self-service business intelligence rules the roost, almost every company seeks to position itself as a data-driven organization. Most businesses are acutely aware of the myriad benefits that can be leveraged by leverage to make intelligently empowered decisions. The ability to provide customers with a top-notch, hyper-personalized experience while reducing cost and capital being the most compelling.

However, organizations continue to face a range of complexities in transforming to a data-driven approach and leveraging its full potential. While migrating legacy systems, shunning legacy cultures, and prioritizing data management in a mix of ever-competing business demands are all valid constraints, the architectural structure of data platform initiatives also proves to be a major roadblock.

Siloed data warehouses and data lake architecture come with limited capabilities for real-time data streaming, and thus, undermine the organizations’ goals for scalability and democratization. Fortunately, Data Mesh – a new, transformative architecture paradigm that has created quite the stir – can give your data goals a new lease on life.

Let’s take a closer look at what Data Mesh is and how it can transform big data management.

#big data #data mesh #data mesh architecture #data platforms