Nebula Flink Connector: Implementation and Practices

Nebula Flink Connector: Implementation and Practices

This post introduces Nebula Flink Connector. Like the pre-defined Flink connectors, it enables Flink to read data from and write data to Nebula Graph. If you are still wondering about it then this article is for you.

This post introduces Nebula Flink Connector. Like the pre-defined Flink connectors, it enables Flink to read data from and write data to Nebula Graph.

In the scenarios of relational network analysis, relationship modeling, and real-time recommendation, using graph databases for background data is becoming popular, and some scenarios, such as recommendation systems and search engines, require high real-time graph data. To improve the real-time performance of data, stream processing is widely used for incremental processing of updated data in real-time. To support the stream processing of graph data, the Nebula Graph team developed Nebula Flink Connecter to empower Flink to operate stream processing of data in Nebula Graph.

Flink is a new generation of computing engines that can support both stream and batch processing of data. It reads data from a third-party storage engine, processes them, and then writes them to another storage engine. A Flink Connector works like a connector, connecting the Flink computing engine to an external storage system.

Flink can use four methods to exchange data with an external source:

  • The pre-defined API of Source and Sink.
  • The bundled connectors, such as JDBC connectors.
  • The Apache Bahir connectors. Apache Bahir was part of Apache Spark. It was intended to provide the implementation of extensions and/or plug-ins, connectors, and other pluggable components that are not limited to Spark.
  • Asynchronous I/O. In-stream processing, it is often necessary to interact with external storage systems, such as associating a table in MySQL. If synchronous I/O is used, a lot of time is consumed for waiting, which has an influence on throughput and latency. But in asynchronous I/O mode, multiple requests can be handled concurrently, so the throughput is improved and the latency is reduced.

This post introduces Nebula Flink Connector. Like the pre-defined Flink connectors, it enables Flink to read data from and write data to Nebula Graph.

database graph database databases nebula graph data exchange

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Which Database Is Right For You?Graph Database vs. Relational Database

Which Database Is Right For You?Graph Database vs. Relational Database. Learn about the main differences between graph and relational databases. What kind of use-cases are best suited for each type, their strengths, and weaknesses.

Data Migration From JanusGraph to Nebula Graph - Practice at 360 Finance

In this article, take a look at data migration from JanusGraph to Nebula Graph. Speaking of graph data processing, we have had experience in using various graph databases. In the beginning, we used the stand-alone edition of AgensGraph. Later, due to its performance limitations, we switched to JanusGraph, a distributed graph database.

Your Data Architecture: Simple Best Practices for Your Data Strategy

Your Data Architecture: Simple Best Practices for Your Data Strategy. Don't miss this helpful article.

Database Vs Data Warehouse Vs Data Lake: A Simple Explanation

A data lake is totally different from a data warehouse in terms of structure and function. Here is a truly quick explanation of "Data Lake vs Data Warehouse".

Benchmarking the Mainstream Open Source Distributed Graph Databases

The deep learning and knowledge graph technologies have been developing rapidly in recent years. Find out more about graph databases.