IndraDB — Rust Based Graph Database

IndraDB

A graph database written in rust.

IndraDB consists of a server and an underlying library. Most users would use the server, which is available via releases as pre-compiled binaries. But if you're a rust developer that wants to embed a graph database directly in your application, you can use the library.

IndraDB's original design is heavily inspired by TAO, facebook's graph datastore. In particular, IndraDB emphasizes simplicity of implementation and query semantics, and is similarly designed with the assumption that it may be representing a graph large enough that full graph processing is not possible. IndraDB departs from TAO (and most graph databases) in its support for properties.

For more details, see the homepage. See also a complete demo of IndraDB for browsing the wikipedia article link graph.

Features

Directed and typed graphs.
JSON-based properties tied to vertices and edges.
Queries with multiple hops, and queries on indexed properties.
Cross-language support via gRPC, or direct embedding as a library.
Pluggable underlying datastores, with several built-in datastores. Postgresql and sled are available separately.
Written in rust! High performance, no GC pauses, and a higher degree of safety.

Installation

Releases

We offer pre-compiled releases for linux and macOS.

Download the latest release for your platform.
Add the binaries to your PATH.
Start the server: indradb-server

This should start the default datastore.

From source

To build and install from source:

Install rust. IndraDB should work with any of the rust variants (stable, nightly, beta.)
Make sure you have gcc 5+ installed.
Clone the repo: git clone git@github.com:indradb/indradb.git.
Build/install it: cargo install.

Docker

If you want to run IndraDB in docker, follow the below instructions.

Server

Build the image for the server:

DOCKER_BUILDKIT=1 docker build --target server -t indradb-server .

Run the server:

docker run --network host --rm indradb-server -a 0.0.0.0:27615

Client

Build the image for the client:

DOCKER_BUILDKIT=1 docker build --target client -t indradb-client .

Run the client:

docker run --network host --rm indradb-client grpc://localhost:27615 ping

Datastores

IndraDB offers several different datastores with trade-offs in durability, transaction capabilities, and performance.

Memory

By default, IndraDB starts a datastore that stores all values in-memory. This is the fastest implementation, but there's no support for graphs larger than what can fit in-memory, and data is only persisted to disk when explicitly requested.

If you want to use the standard datastore without support for persistence, don't pass a subcommand; e.g.:

indradb-server [options]

If you want to use the standard datastore but persist to disk:

indradb-server memory --persist-path=[/path/to/memory/image.bincode]

You'll need to explicitly call Sync() when you want to save the graph.

RocksDB

If you want to use the rocksdb-backed datastore, use the rocksdb subcommand; e.g.:

indradb-server rocksdb [/path/to/rocksdb.rdb] [options]

Postgres, Sled, etc.

It's possible to develop other datastores implementations in separate crates, since the IndraDB exposes the necessary traits to implement:

Postgres is available through indradb-postgres.
Sled is available through indradb-sled.

Plugins

The IndraDB server includes support for plugins to extend functionality available to clients. Plugins are loaded via dynamically linked libraries.

See the hello world plugin and naive vertex plugin for demonstrations of how to author plugins.

To include plugins, see the --plugins argument for indradb-server, e.g. indradb-server --plugins=plugins/*.so. They are then callable via the gRPC ExecutePlugin function.

Testing

Unit tests

Use make test to run the test suite. Note that this will run the full test suite across the entire workspace, including tests for all datastore implementations. You can filter which tests run via the TEST_NAME environment variable. e.g. TEST_NAME=create_vertex make test will run tests with create_vertex in the name across all datastore implementations. All unit tests will run in CI.

Benchmarks

Microbenchmarks can be run via make bench.

Fuzzing

A fuzzer is available, ensuring the the RocksDB and in-memory datastores operate identically. Run it via make fuzz.

Checks

Lint and formatting checks can be run via make check. Equivalent checks will be run in CI.

Download Details:
Author: indradb
Source Code: https://github.com/indradb/indradb
License: MPL-2.0

#rust #database

What is GEEK

Buddha Community

Ruth Nabimanya

1620663480

Which Database Is Right For You?Graph Database vs. Relational Database

At the very beginning of most development endeavors lies an important question: What database do I choose? There is such an abundance of database technologies at this moment, it’s no wonder many developers don’t have the time or energy to research new ones. If you are one of those developers and you aren’t very familiar with graph databases in general, you’ve come to the right place!

In this article, you will learn about the main differences between a graph database and a relational database, what kind of use-cases are best suited for each database type, and what are their strengths and weaknesses.

How Does a Graph Database Differ from a Relational Database?

The Graph Data Model

The Relational Data Model

When to use a Graph Database?

When not to use a Graph Database

Is a Graph Database Worth it?

#graph-database #relational-database #graph-theory #graph-analysis #data-analytics #networks #data #database

Marielle Rippin

1603659600

Benchmarking the Mainstream Open Source Distributed Graph Databases

The deep learning and knowledge graph technologies have been developing rapidly in recent years. Compared with the “black box” of deep learning, knowledge graphs are highly interpretable, thus are widely adopted in such scenarios as search recommendations, intelligent customer support, and financial risk management.

Meituan has been digging deep in the connections buried in the huge amount of business data over the past few years and has gradually developed the knowledge graphs in nearly ten areas, including cuisine graphs, tourism graphs, and commodity graphs. The ultimate goal is to enhance the smart local life.

Compared with the traditional RDBMS, graph databases can store and query knowledge graphs more efficiently. It gains obvious performance advantage in multi-hop queries to select graph databases as the storage engine. Currently, there are dozens of graph database solutions out there on the market.

It is imperative for the Meituan team to select a graph database solution that can meet the business requirements and to use the solution as the basis of Meituan’s graph storage and graph learning platform. The team has outlined the basic requirements as below per our business status quo:

It should be an open-source project which is also business-friendly

By having control over the source code, the Meituan team can ensure data security and service availability.

It should support clustering and should be able to scale horizontally in terms of both storage and computation capabilities

The knowledge graph data size in Meituan can reach hundreds of billions of vertices and edges in total and the throughput can reach tens of thousands of QPS. With that being said, the single-node deployment cannot meet Meituan’s storage requirements.

It should work under OLTP scenarios with the capability of multi-hop queries at the millisecond level.

To ensure the best search experience for Meituan users, the team has strictly restricted the timeout value within all chains of paths. Therefore, it is unacceptable to respond to a query at the second level.

It should be able to import data in batch

The knowledge graph data is usually stored in data warehouses like Hive. The graph database should be equipped with the capability to quickly import data from such warehouses to the graph storage to ensure service effectiveness.

The Meituan team has tried the top 30 graph databases on DB-Engines and found that most well-known graph databases only support single-node deployment with their open-source edition, for example, Neo4j, ArangoDB, Virtuoso, TigerGraph, RedisGraph. This means that the storage service cannot scale horizontally and the requirement to store large-scale knowledge graph data cannot be met.

After thorough research and comparison, the team has selected the following graph databases for the final round: Nebula Graph (developed by a startup team who originally came from Alibaba), Dgraph (developed by a startup team who originally came from Google), and HugeGraph (developed by Baidu).

A Summary of The Testing Process

Hardware Configuration

Database instances: Docker containers running on different machines
Single instance resources: 32 Cores, 64 GB Memory, 1 TB SSD (Intel® Xeon® Gold 5218 CPU @ 2.30 GHz)
Number of instances: Three

#database #tutorial #graph database #database performance #nebula graph #dgraph #graph database adoption

Mikel Okuneva

1599897600

Data Migration From JanusGraph to Nebula Graph - Practice at 360 Finance

Speaking of graph data processing, we have had experience in using various graph databases. In the beginning, we used the stand-alone edition of AgensGraph. Later, due to its performance limitations, we switched to JanusGraph, a distributed graph database. I introduced details on how to migrate data in my article “Migrate tens of billions of graph data into JanusGraph (only in Chinese)”. As the data size and the number of business calls grew, a new problem appeared: Each query consumed too much time. In some business scenarios, a single query took up to 10 seconds, and with increase of the data size, a more complicated single query needed two or three seconds. These problems had seriously affected the performance of the entire business process and the development of related businesses.

The architecture design of JanusGraph determines that a single query is time-consuming. The core reason is that its storage depends on the external storage, and JanusGraph cannot control the external storage well. In our production environment, an HBase cluster is used, which makes it impossible for all queries to be pushed down to the storage layer for processing. Instead, data can only be queried from HBase to the JanusGraph Server memory and then filtered accordingly.

#database #tutorial #graph database #database performance #nebula graph #graph database adoption

Luna Mosciski

1595932020

Graph Therapy: The Year of the Graph Newsletter, June/May 2020

Parts of the world are still in lockdown, while others are returning to some semblance of normalcy. Either way, while the last few months have given some things pause, they have boosted others. It seems like developments in the world of Graphs are among those that have been boosted.

An abundance of educational material on all things graph has been prepared and delivered online, and is now freely accessible, with more on the way.

Graph databases have been making progress and announcements, repositioning themselves by a combination of releasing new features, securing additional funds, and entering strategic partnerships.

A key graph database technology, RDF*, which enables compatibility between RDF and property graph databases, is gaining momentum and tool support.

And more cutting edge research combining graph AI and knowledge graphs is seeing the light, too. Buckle up and enjoy some graph therapy.

Stanford’s series of online seminars featured some of the world’s leading experts on all things graph. If you missed it, or if you’d like to have an overview of what was said, you can find summaries for each lecture in this series of posts by Bob Kasenchak and Ahren Lehnert. Videos from the lectures are available here.

Stanford Knowledge Graph Course Not-Quite-Live-Blog

Stanford University’s computer science department is offering a free class on Knowledge Graphs available to the public. Stanford is also making recordings of the class available via the class website.

Another opportunity to get up to speed with educational material: The entire program of the course “Information Service Engineering” at KIT - Karlsruhe Institute of Technology, is delivered online and made freely available on YouTube. It includes topics such as ontology design, knowledge graph programming, basic graph theory, and more.

Information Service Engineering at KIT

Knowledge representation as a prerequisite for knowledge graphs. Learn about knowledge representation, ontologies, RDF(S), OWL, SPARQL, etc.

Ontology may sound like a formal term, while knowledge graph is a more approachable one. But the 2 are related, and so is ontology and AI. Without a consistent, thoughtful approach to developing, applying, evolving an ontology, AI systems lack underpinning that would allow them to be smart enough to make an impact.

The ontology is an investment that will continue to pay off, argue Seth Earley and Josh Bernoff in Harvard Business Review, making the case for how businesses may benefit from a knowldge-centric approach

Is Your Data Infrastructure Ready for AI?

Even after multiple generations of investments and billions of dollars of digital transformations, organizations struggle to use data to improve customer service, reduce costs, and speed the core processes that provide competitive advantage. AI was supposed to help with that.

Besides AI, knowledge graphs have a part to play in the Cloud, too. State is good, and lack of support for Stateful Cloud-native applications is a roadblock for many enterprise use-cases, writes Dave Duggal.

Graph knowledge bases are an old idea now being revisited to model complex, distributed domains. Combining high-level abstraction with Cloud-native design principles offers efficient “Context-as-a-Service” for hydrating stateless services. Graph knowledge-based systems can enable composition of Cloud-native services into event-driven dataflow processes.

Kubernetes also touches upon Organizational Knowledge, and that may be modeled as a Knowledge Graph.

Graph Knowledge Base for Stateful Cloud-Native Applications

Extending graph knowledge bases to model distributed systems creates a new kind of information system, one intentionally designed for today’s IT challenges.

The Enterprise Knowledge Graph Foundation was recently established to define best practices and mature the marketplace for EKG adoption, with a launch webinar on June the 23rd.

The Foundation defines its mission as including adopting semantic standards, developing best practices for accelerated EKG deployment, curating a repository of reusable models and resources, building a mechanism for engagement and shared knowledge, and advancing the business cases for EKG adoption.

Enterprise Knowledge Graph Maturity Model

The Enterprise Knowledge Graph Maturity Model (EKG/MM) is the industry-standard definition of the capabilities required for an enterprise knowledge graph. It establishes standard criteria for measuring progress and sets out the practical questions that all involved stakeholders ask to ensure trust, confidence and usage flexibility of data. Each capability area provides a business summary denoting its importance, a definition of the added value from semantic standards and scoring criteria based on five levels of defined maturity.

Enterprise Knowledge Graphs is what the Semantic Web Company (SWC) and Ontotext have been about for a long time, too. Two of the vendors in this space that have been around for the longer time just announced a strategic partnership: Ontotext, a graph database and platform provider, meets SWC, a management and added value layer that sits on top.

SWC and Ontotext CEOs emphasize how their portfolios are complementary, while the press release states that the companies have implemented a seamless integration of the PoolParty Semantic Suite™ v.8 with the GraphDB™ and Ontotext Platform, which offers benefits for many use cases.

#database #artificial intelligence #graph databases #rdf #graph analytics #knowledge graph #graph technology