Узнайте о сравнении баз данных NoSQL, управляемых в облаке

Узнайте о сравнении баз данных NoSQL, управляемых в облаке

Azure Cosmos DB — это глобально распределенная многомодельная служба базы данных, предоставляемая Microsoft Azure. Он предоставляет высокодоступную и масштабируемую базу данных NoSQL, которая может обрабатывать различные типы данных и рабочие нагрузки. Точно так же Google Cloud Platform (GCP) предлагает Cloud Firestore, Amazon Web Services (AWS) предлагает Amazon DynamoDB, а Oracle Cloud предлагает базу данных Oracle NoSQL в качестве эквивалентных управляемых служб базы данных NoSQL.

Сходства

• Все сервисы полностью управляемы и обеспечивают автоматическое масштабирование, высокую доступность и глобальное распространение.
• Все сервисы поддерживают модели данных NoSQL и предлагают гибкие структуры данных.
• Все сервисы предоставляют SDK и API для различных языков программирования для легкой интеграции и разработки.

Отличия

• Azure Cosmos DB поддерживает несколько API (MongoDB, Cassandra, SQL, Gremlin и Azure Table Storage), а GCP Cloud Firestore, AWS DynamoDB и Oracle NoSQL DB имеют собственные API.
• Cosmos DB поддерживает глобальное распространение данных в любом регионе Azure, а AWS DynamoDB поддерживает региональное распространение с репликами чтения в других регионах.
• Cosmos DB предлагает настраиваемые уровни согласованности, а GCP Firestore и AWS DynamoDB по умолчанию предлагают строгую согласованность.
• Oracle NoSQL DB поддерживает шифрование данных и интеграцию с Oracle Key Vault.

Варианты использования в реальном времени

• Microsoft использует Cosmos DB для поддержки таких служб Azure, как Azure IoT Hub, Azure Time Series Insights и Azure Cognitive Services.
• Snapchat использует Google Cloud Firestore для поддержки функции чата в реальном времени.
• Netflix использует AWS DynamoDB для хранения метаданных своего потокового сервиса.
• AT&T использует базу данных Oracle NoSQL для хранения и управления большими объемами клиентских данных.

Azure Cosmos DB, GCP Cloud Firestore, AWS DynamoDB и Oracle NoSQL Database — это управляемые службы баз данных NoSQL, которые предлагают масштабируемые, высокодоступные и глобально распределенные решения для современных приложений. Хотя между этими службами есть сходства, такие как гибкие модели данных и поддержка различных языков программирования, существуют различия в поддержке API, моделях согласованности и возможностях распространения. Примеры использования в реальном времени демонстрируют различные отрасли и приложения, использующие эти сервисы, включая IoT, чат в реальном времени, хранилище метаданных и управление данными клиентов. В конечном счете, организациям следует выбрать управляемую службу базы данных NoSQL, которая наилучшим образом соответствует их потребностям, исходя из их конкретных требований и вариантов использования.

Оригинальный источник статьи: https://www.c-sharpcorner.com

#nosql #databases #cloud 

找出比较云管理的 NoSQL 数据库

找出比较云管理的 NoSQL 数据库

Azure Cosmos DB 是 Microsoft Azure 提供的全球分布式多模型数据库服务。它提供了一个高度可用和可扩展的 NoSQL 数据库，可以处理各种数据类型和工作负载。同样，Google Cloud Platform (GCP) 提供 Cloud Firestore，Amazon Web Services (AWS) 提供 Amazon DynamoDB，而 Oracle Cloud 提供 Oracle NoSQL Database 作为其等效的托管 NoSQL 数据库服务。

相似之处

• 所有服务都是完全托管的，并提供自动扩展、高可用性和全球分布。
• 所有服务都支持 NoSQL 数据模型并提供灵活的数据结构。
• 所有服务均提供各种编程语言的SDK和API，方便集成开发。

差异

• Azure Cosmos DB 支持多种 API（MongoDB、Cassandra、SQL、Gremlin 和 Azure Table Storage），而 GCP Cloud Firestore、AWS DynamoDB 和 Oracle NoSQL DB 都有自己的 API。
• Cosmos DB 支持将数据全球分布到任何 Azure 区域，而 AWS DynamoDB 支持在其他区域使用只读副本进行区域分布。
• Cosmos DB 提供可调的一致性级别，而 GCP Firestore 和 AWS DynamoDB 默认提供强一致性。
• Oracle NoSQL DB 支持数据加密和 Oracle Key Vault 集成。

实时用例

• Microsoft 使用 Cosmos DB 来支持 Azure 服务，例如 Azure IoT Hub、Azure Time Series Insights 和 Azure Cognitive Services。
• Snapchat 使用 Google Cloud Firestore 为其实时聊天功能提供支持。
• Netflix 使用 AWS DynamoDB 为其流媒体服务存储元数据。
• AT&T 使用 Oracle NoSQL 数据库来存储和管理大量客户数据。

Azure Cosmos DB、GCP Cloud Firestore、AWS DynamoDB 和 Oracle NoSQL 数据库都是托管的 NoSQL 数据库服务，它们为现代应用程序提供可扩展、高可用性和全球分布的解决方案。虽然这些服务之间存在相似之处，例如灵活的数据模型和对各种编程语言的支持，但在 API 支持、一致性模型和分发功能方面存在差异。实时用例展示了利用这些服务的不同行业和应用程序，包括物联网、实时聊天、元数据存储和客户数据管理。最终，组织应根据其特定要求和用例选择最适合其需求的托管 NoSQL 数据库服务。

文章原文出处： https: //www.c-sharpcorner.com

#nosql #databases #cloud 

Find out Comparing Cloud Managed NoSQL Databases

Find out Comparing Cloud Managed NoSQL Databases

Azure Cosmos DB is a globally distributed, multi-model database service provided by Microsoft Azure. It provides a highly available and scalable NoSQL database that can handle a variety of data types and workloads. Similarly, Google Cloud Platform (GCP) offers Cloud Firestore, Amazon Web Services (AWS) offers Amazon DynamoDB, and Oracle Cloud offers Oracle NoSQL Database as their equivalent managed NoSQL database services.

Similarities

• All services are fully managed and provide automatic scaling, high availability, and global distribution.
• All services support NoSQL data models and offer flexible data structures.
• All services provide SDKs and APIs for various programming languages for easy integration and development.

Differences

• Azure Cosmos DB supports multiple APIs (MongoDB, Cassandra, SQL, Gremlin, and Azure Table Storage), while GCP Cloud Firestore, AWS DynamoDB, and Oracle NoSQL DB have their own APIs.
• Cosmos DB supports the global distribution of data to any Azure region, while AWS DynamoDB supports regional distribution with read replicas in other regions.
• Cosmos DB offers tunable consistency levels, while GCP Firestore and AWS DynamoDB offer strong consistency by default.
• Oracle NoSQL DB supports data encryption and Oracle Key Vault integration.

Real-time use cases

• Microsoft uses Cosmos DB to support Azure services such as Azure IoT Hub, Azure Time Series Insights, and Azure Cognitive Services.
• Snapchat uses Google Cloud Firestore to power its real-time chat feature.
• Netflix uses AWS DynamoDB to store metadata for its streaming service.
• AT&T uses Oracle NoSQL Database to store and manage large volumes of customer data.

Azure Cosmos DB, GCP Cloud Firestore, AWS DynamoDB, and Oracle NoSQL Database are all managed NoSQL database services that offer scalable, highly available, and globally distributed solutions for modern applications. While there are similarities between these services, such as flexible data models and support for various programming languages, there are differences in API support, consistency models, and distribution capabilities. Real-time use cases demonstrate the diverse industries and applications that utilize these services, including IoT, real-time chat, metadata storage, and customer data management. Ultimately, organizations should choose the managed NoSQL database service that best fits their needs based on their specific requirements and use cases.

Original article source at: https://www.c-sharpcorner.com

#nosql #databases #cloud 

Typeorm: TypeORM Module for Nest Framework (node.js)

Typeorm

A progressive Node.js framework for building efficient and scalable server-side applications.

Installation

$ npm i --save @nestjs/typeorm typeorm

Quick Start

Overview & Tutorial

Description

TypeORM module for Nest.

Support

Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please read more here.

Stay in touch

• Author - Kamil Myśliwiec
• Website - https://nestjs.com
• Twitter - @nestframework

Download Details:

Author: Nestjs
Source Code: https://github.com/nestjs/typeorm 
License: MIT license

#typescript #javascript #nodejs #sql #nosql 

Big Data & Hadoop for Beginners - Full Course in 12 Hours

This Big Data & Hadoop full course will help you understand and learn Hadoop concepts in detail. You'll learn: Introduction to Big Data, Hadoop Fundamentals, HDFS, MapReduce, Sqoop, Flume, Pig, Hive, NoSQL-HBase, Oozie, Hadoop Projects, Career in Big Data Domain, Big Data Hadoop Interview Q and A

Big Data & Hadoop Full Course In 12 Hours | BigData Hadoop Tutorial For Beginners 

This Edureka Big Data & Hadoop Full Course video will help you understand and learn Hadoop concepts in detail. This Big Data & Hadoop Tutorial is ideal for both beginners as well as professionals who want to master the Hadoop Ecosystem. Below are the topics covered in this Big Data Full Course:

• Introduction to Big Data 
• Hadoop Fundamentals
• HDFS
• MapReduce
• Sqoop
• Flume
• Pig
• Hive 
• NoSQL-HBase
• Oozie
• Hadoop Projects
• Career in Big Data Domain 
• Big Data Hadoop Interview Q and A

#bigdata #hadoop #pig #hive #nosql 

Dynamodb-toolbox: Single Table Designs Have Never Been This Easy!

DynamoDB Toolbox  

Single Table Designs have never been this easy!

The DynamoDB Toolbox is a set of tools that makes it easy to work with Amazon DynamoDB and the DocumentClient. It's designed with Single Tables in mind, but works just as well with multiple tables. It lets you define your Entities (with typings and aliases) and map them to your DynamoDB tables. You can then generate the API parameters to put, get, delete, update, query, scan, batchGet, and batchWrite data by passing in JavaScript objects. The DynamoDB Toolbox will map aliases, validate and coerce types, and even write complex UpdateExpressions for you. 😉

Why single table design?

Learn more about single table design in Alex Debrie's blog.

Version 0.6 🙌

Feedback is welcome and much appreciated! (Huge thanks to @ThomasAribart for all his work on this 🙌)

Docs & Community

• Full Documentation
• GitHub Discussions

Quick Start

:information_source: We're using aws-sdk v2 DynamoDB tools, the support for aws-sdk v3 is on its way. 
You can read more about the development here.

Using your favorite package manager, install DynamoDB Toolbox and aws-sdk v2 in your project by running one of the following commands:

# npm
npm i dynamodb-toolbox
npm install aws-sdk

# yarn
yarn add dynamodb-toolbox
yarn add aws-sdk

Require or import Table and Entity from dynamodb-toolbox:

import { Table, Entity } from 'dynamodb-toolbox'

Create a Table (with the DocumentClient using aws-sdk v2):

import DynamoDB from 'aws-sdk/clients/dynamodb'

const DocumentClient = new DynamoDB.DocumentClient({
  // Specify your client options as usual
  convertEmptyValues: false
})

// Instantiate a table
const MyTable = new Table({
  // Specify table name (used by DynamoDB)
  name: 'my-table',

  // Define partition and sort keys
  partitionKey: 'pk',
  sortKey: 'sk',

  // Add the DocumentClient
  DocumentClient
})

Create an Entity:

const Customer = new Entity({
  // Specify entity name
  name: 'Customer',

  // Define attributes
  attributes: {
    id: { partitionKey: true }, // flag as partitionKey
    sk: { hidden: true, sortKey: true }, // flag as sortKey and mark hidden
    age: { type: 'number' }, // set the attribute type
    name: { type: 'string', map: 'data' }, // map 'name' to table attribute 'data'
    emailVerified: { type: 'boolean', required: true }, // specify attribute as required
    co: { alias: 'company' }, // alias table attribute 'co' to 'company'
    status: ['sk', 0], // composite key mapping
    date_added: ['sk', 1] // composite key mapping
  },

  // Assign it to our table
  table: MyTable

  // In Typescript, the "as const" statement is needed for type inference
} as const)

Put an item:

// Create an item (using table attribute names or aliases)
const customer = {
  id: 123,
  age: 35,
  name: 'Jane Smith',
  emailVerified: true,
  company: 'ACME',
  status: 'active',
  date_added: '2020-04-24'
}

// Use the 'put' method of Customer:
await Customer.put(customer)

The item will be saved to DynamoDB like this:

{
  "pk": 123,
  "sk": "active#2020-04-24",
  "age": 35,
  "data": "Jane Smith",
  "emailVerified": true,
  "co": "ACME",
  // Attributes auto-generated by DynamoDB-Toolbox
  "_et": "customer", // Entity name (required for parsing)
  "_ct": "2021-01-01T00:00:00.000Z", // Item creation date (optional)
  "_md": "2021-01-01T00:00:00.000Z" // Item last modification date (optional)
}

You can then get the data:

// Specify primary key
const primaryKey = {
  id: 123,
  status: 'active',
  date_added: '2020-04-24'
}

// Use the 'get' method of Customer
const response = await Customer.get(primaryKey)

Since v0.4, the method inputs, options and response types are inferred from the Entity definition:

await Customer.put({
  id: 123,
  // ❌ Sort key is required ("sk" or both "status" and "date_added")
  age: 35,
  name: ['Jane', 'Smith'], // ❌ name should be a string
  emailVerified: undefined, // ❌ attribute is marked as required
  company: 'ACME'
})

const { Item: customer } = await Customer.get({
  id: 123,
  status: 'active',
  date_added: '2020-04-24' // ✅ Valid primary key
})
type Customer = typeof customer
// 🙌 Type is equal to:
type ExpectedCustomer =
  | {
      id: any
      age?: number | undefined
      name?: string | undefined
      emailVerified: boolean
      company?: any
      status: any
      date_added: any
      entity: string
      created: string
      modified: string
    }
  | undefined

See Type Inference in the documentation for more details.

Features

• Table Schemas and DynamoDB Typings: Define your Table and Entity data models using a simple JavaScript object structure, assign DynamoDB data types, and optionally set defaults.
• Magic UpdateExpressions: Writing complex UpdateExpression strings is a major pain, especially if the input data changes the underlying clauses or requires dynamic (or nested) attributes. This library handles everything from simple SET clauses, to complex list and set manipulations, to defaulting values with smartly applied if_not_exists() to avoid overwriting data.
• Bidirectional Mapping and Aliasing: When building a single table design, you can define multiple entities that map to the same table. Each entity can reuse fields (like pk andsk) and map them to different aliases depending on the item type. Your data is automatically mapped correctly when reading and writing data.
• Composite Key Generation and Field Mapping: Doing some fancy data modeling with composite keys? Like setting your sortKey to [country]#[region]#[state]#[county]#[city]#[neighborhood] model hierarchies? DynamoDB Toolbox lets you map data to these composite keys which will both autogenerate the value and parse them into fields for you.
• Type Coercion and Validation: Automatically coerce values to strings, numbers and booleans to ensure consistent data types in your DynamoDB tables. Validate list, map, and set types against your data. Oh yeah, and sets are automatically handled for you. 😉
• Powerful Query Builder: Specify a partitionKey, and then easily configure your sortKey conditions, filters, and attribute projections to query your primary or secondary indexes. This library can even handle pagination with a simple .next() method.
• Simple Table Scans: Scan through your table or secondary indexes and add filters, projections, parallel scans and more. And don't forget the pagination support with .next().
• Filter and Condition Expression Builder: Build complex Filter and Condition expressions using a standardized array and object notation. No more appending strings!
• Projection Builder: Specify which attributes and paths should be returned for each entity type, and automatically filter the results.
• Secondary Index Support: Map your secondary indexes (GSIs and LSIs) to your table, and dynamically link your entity attributes.
• Batch Operations: Full support for batch operations with a simpler interface to work with multiple entities and tables.
• Transactions: Full support for transaction with a simpler interface to work with multiple entities and tables.
• Default Value Dependency Graphs: Create dynamic attribute defaults by chaining other dynamic attribute defaults together.
• TypeScript Support: v0.4 of this library provides strong typing support AND type inference 😍. Inferred type can still overriden with Overlays. Some Utility Types are also exposed. Additional work is still required to support schema validation & typings.

Additional References

• DocumentClient SDK Reference
• Best Practices for DynamoDB
• DynamoDB, explained.
• The DynamoDB Book

Contributions and Feedback

Contributions, ideas and bug reports are welcome and greatly appreciated. Please add issues for suggestions and bug reports or create a pull request. You can also contact me on Twitter: @jeremy_daly.

Download Details:

Author: jeremydaly
Source Code: https://github.com/jeremydaly/dynamodb-toolbox 
License: MIT license

#aws #serverless #nosql #dynamodb 

SQL for Beginners - Full Course in 10 Hours

Learn SQL in this full course for beginners. You'll learn: What is SQL? SQL BASICS, SQL Operators, Normalization in SQL, Triggers in SQL, SQL Joins, SQL functions, SQL vs MySQL, SQL vs NoSQL, SQL Interview questions and answers, SQL for Data science, PostgreSQL, SQL server, SQL server interview questions and answer, and more

SQL Full Course In 10 Hours | SQL Tutorial For Beginners | SQL For Beginners 

This SQL Full Course video will cover all the topics of Structured Query Language (SQL) starting from scratch. This SQL tutorial for beginners is great for beginners who want to learn SQL and for professionals who want to brush up their SQL skills. The following topics are covered in this SQL Full Course Tutorial:

• What is SQL?
• SQL BASICS
• SQL Operators
• Normalization in SQL 
• Triggers in SQL
• SQL Joins
• SQL functions
• SQL vs MySQL
• SQL vs NoSQL
• SQL Interview questions and answers
• SQL for Data science
• PostgreSQL
• SQL server
• SQL server interview questions and answer

#sql #database #nosql #postgresql #mysql 

How to Playing with Semantic Data and MarkLogic

In this blog, I will discuss the Semantic Data Model and its support in one of the enterprise NoSQL databases with an example.

What is a Data Model

Data models always come into the picture when we talk about any database and what type of data storage and query utility it provides. The data model defines the structure, as well as the relationships of data elements. Some of the popular data models that everyone must have heard of are the relational data model. Where data is kept in form of tables or relations.

What is Semantic Data Model

A semantic data model is about storing semantics (real meaning between two entities) in form of a specified format that tells about the relations between two entities. Consider the below two statements as the semantics of entities Author, Employee, Book, etc.

• Clean Code is a Book.
• Uncle Bob is the Author of “Clean Code”
• Uncle Bob is an Employee of XCompany

If we represent it diagrammatically, It will look something like the below:

The magic of the Semantic model is that when we care about the semantics or the relations among the data, we are able to infer a few things which are not stored directly as information but true and inferred. For e.g. An employee can write a book, or an Employee has written a Book.

What has MarkLogic to do with Semantic Model

MarkLogic is one of the Database offering providing support to store and query the Semantic data. If you have not heard of this database before or installed it before, I found the documentation of MarkLogic very detailed which will give you a glimpse of introductory enlightenment of the Database. Likewise, If you want to install MarkLogic on Linux or ubuntu, you will find this blog very much helpful.

Starting with Experiment

I assume you have installed the MarkLogic and the server is running on your system. I am running a local cluster which will give you a start on localhost:8001/ and hopefully you would see an interface like the below:

MarkLogic Server Running on Local Cluster

Importing Semantic Data into MarkLogic

The data format here which I will be using for explanation is RDF triples. This is a format for Semantic Data Model. The data format will take the form of:

RDF Data Format.

For e.g. “Uncle Bob” has authored “Clean Code”.

For this blog, I will be using the same data set which is provided as part of a very detailed document from MarkLogic starting with Semantic Data. To check what is required before you can start hands-on, jump to the Pre-Requisites section.

To insert the triples on in the database, follow the simple steps below. I will be using

• Open localhost:8000 (Query console in your browser)
• Copy the below text into the query console and change the type of query to XQuery.

import module namespace sem = "http://marklogic.com/semantics"  at "/MarkLogic/semantics.xqy";   sem:rdf-insert(  (  sem:triple(    sem:iri("http://example.org/marklogic/people/Uncle_Bob"),    sem:iri("http://example.org/marklogic/predicate/authored"),    "Clean Code"    )  ,  sem:triple(    sem:iri("http://example.org/marklogic/people/Uncle_Bob"),    sem:iri("http://example.org/marklogic/predicate/livesOn"),    "Earth"    )  ) )

The above snippet, I have borrowed from the documentation as I mentioned before. Once it is written, hit the “Run” button and you would see a response there.

Inserting RDF Triples

Once the data is inserted, we can verify using the triple count. using the below function in another tab.

We have inserted 2 triples. In my case, It was three, you would see 2.

Querying RDF Data

As part of the above RDF data, we already established the below facts.

• Uncle_Bob lives on Earth
• Uncle_Bob authored Clean Code

Let us use the SPARQL query language to query the same question “Who lives on Earth”? . SPARQL is the query that is used to run on semantic data. Here you can find more about the SPARQL query structure in W3 documentation.

Original article source at: https://blog.knoldus.com/

#play #data #nosql #database 

How to Real-Time Location Changes with MongoDB Realm, Change Streams

When it comes to modern web applications, interactions often need to be done in real-time. This means that instead of periodically checking in for changes, watching or listening for changes often makes more sense.

Take the example of tracking something on a map. When it comes to package shipments, device tracking, or anything else where you need to know the real-time location, watching for those changes in location is great. Imagine needing to know where your fleet is so that you can dispatch them to a nearby incident?

When it comes to MongoDB, watching for changes can be done through change streams. These change streams can be used in any of the drivers, including front-end applications with MongoDB Realm.

In this tutorial, we’re going to leverage MongoDB Realm change streams. When the location data in our NoSQL documents change, we’re going to update the information on an interactive map powered by Mapbox.

Take the following animated image for example:

Rather than building an Internet of Things (IoT) device to track and submit GPS data, we’re going to simulate the experience by directly changing our documents in MongoDB. When the update operations are complete, the front-end application with the interactive map is watching for those changes and responding appropriately.

The Requirements

To be successful with this example, you’ll need to have a few things ready to go prior:

• A MongoDB Atlas cluster
• A MongoDB Realm application
• A Mapbox account

For this example, the data will exist in MongoDB Atlas. Since we’re planning on interacting with our data using a front-end application, we’ll be using MongoDB Realm. A Realm application should be created within the MongoDB Cloud and connected to the MongoDB Atlas cluster prior to exploring this tutorial.

Get started with MongoDB Atlas and Realm for FREE in the MongoDB Cloud.

Mapbox will be used as our interactive map. Since Mapbox is a service, you’ll need to have created an account and have access to your access token.

In the animated image, I’m using the MongoDB Visual Studio Code plugin for interacting with the documents in my collection. You can do the same or use another tool such as Compass, the CLI, or the data explorer within Atlas to get the job done.

Understanding the Document Model for the Location Tracking Example

Because we’re only planning on moving a marker around on a map, the data model that we use doesn’t need to be extravagant. For this example, the following is more than acceptable:

{
    "_id": "5ec44f70fa59d66ba0dd93ae",
    "coordinates": [
        -121.4252,
        37.7397
    ],
    "username": "nraboy"
}

In the above example, the coordinates array has the first item representing the longitude and the second item representing the latitude. We’re including a username to show that we are going to watch for changes based on a particular document field. In a polished application, all users probably wouldn’t be watching for changes for all documents. Instead they’d probably be watching for changes of documents that belong to them.

While we could put authorization rules in place for users to access certain documents, it is out of the scope of this example. Instead, we’re going to mock it.

Building a Real-Time Location Tracking Application with Mapbox and the Realm SDK

Now we’re going to build our client-facing application which consists of Mapbox, some basic HTML and JavaScript, and MongoDB Realm.

Let’s start by adding the following boilerplate code:

<!DOCTYPE html>
<head>
    <script src="https://api.mapbox.com/mapbox-gl-js/v1.10.0/mapbox-gl.js"></script>
    <link href="https://api.mapbox.com/mapbox-gl-js/v1.10.0/mapbox-gl.css" rel="stylesheet" />
    <script src="https://s3.amazonaws.com/stitch-sdks/js/bundles/4.6.0/stitch.js"></script>
</head>
<html>
    <body style="margin: 0">
        <div id="map" style="width: 100vw; height: 100vh"></div>
        <script>
            // Logic here ...
        </script>
    </body>
</html>

The above code sets us up by including the Mapbox and MongoDB Realm SDKs. When it comes to querying MongoDB and interacting with the map, we’re going to be doing that from within the <script> tags.

Within the <script> tags, let’s get started by adding the following:

const client = stitch.Stitch.initializeDefaultAppClient('REALM_APP_ID_HERE');
const db = client.getServiceClient(stitch.RemoteMongoClient.factory, "mongodb-atlas").db("location_services");
mapboxgl.accessToken = "MAPBOX_ACCESS_TOKEN_HERE";

var currentLocationMarker;

The above lines of code are useful for the initialization of our services. You’ll want to swap the app id with your actual Realm app id and the Mapbox access token with your actual Mapbox access token. Both of these can be found within each of the services dashboards.

For this example, I’m going to assume you’re using a location_services database within MongoDB Atlas and a tracking collection within that database.

The currentLocationMarker variable will represent our changing marker that moves around on the map as new data comes in from the MongoDB change stream.

Within the same <script> tags, we can initialize the map for displaying:

let map = new mapboxgl.Map({
    container: "map",
    style: "mapbox://styles/mapbox/streets-v11",
    center: [-121.4252, 37.7397],
    zoom: 9
});

Since the map tiles that compose the map come from a service, we need to wait until the map is ready before we start interacting with it. We can make use of the Mapbox load event to let us know the map is ready:

map.on("load", async () => {
    await client.auth.loginWithCredential(new stitch.AnonymousCredential());
    let currentLocation = (await db.collection("tracking").findOne({ "username": "nraboy" })).coordinates;
    currentLocationMarker = new mapboxgl.Marker().setLngLat(currentLocation).addTo(map);
    const stream = await db.collection("tracking").watch({
        "fullDocument.username": "nraboy"
    });
    stream.onNext(event => {
        currentLocationMarker.setLngLat(event.fullDocument.coordinates);
    });
});

Inside the load event, we are doing anonymous authentication with MongoDB Realm. Remember, we could very easily use a stronger authentication method and have authorization rules, but for this example it’s out of the scope.

Once we’re authenticated to Realm, we execute a findOne operation based on the mock username field on our document. The only data we care about is the coordinates, but we want to make sure it comes from the correct username field.

With the latitude and longitude coordinate information in hand, we can update the marker position.

Up until now, we are just setting the marker to the last known position. This is because when we start watching with a change stream, we won’t receive an initial document. This is why we are doing the findOne first.

This brings us to the change stream:

const stream = await db.collection("tracking").watch({
    "fullDocument.username": "nraboy"
});
stream.onNext(event => {
    currentLocationMarker.setLngLat(event.fullDocument.coordinates);
});

For this particular change stream, we are only watching for documents where the username field matches. In a more polished and realistic example, you could use this to watch for only your own documents. We’re mocking this by hard-coding the value.

When documents with the username field experience some kind of change within Atlas, the marker location will update. This is done without us having to constantly query for updates based on a timer.

Conclusion

You just saw how to use change streams in a client-facing application using the MongoDB Realm SDK. To make our example more attractive, we went with a location tracking scenario, whereas when latitude and longitude locations change in the database, the positions are updated in real-time on a map powered by Mapbox.

If you’d like to give MongoDB Atlas and MongoDB Realm a try, there’s a forever FREE tier available through the MongoDB Cloud.

When it comes to location data and MongoDB, there’s so much more you can do. If you’d like to learn how to create and manage geofences, check out my previous tutorial titled, Location Geofencing with MongoDB, Realm, and Mapbox.

This content first appeared on MongoDB.

Original article source at: https://developer.mongodb.com/

#mongodb #stream #mapbox #nosql 

How to Build an Autocomplete form Element with Atlas Search, JS

When you’re developing a web application, a quality user experience can make or break your application. A common application feature is to allow users to enter text into a search bar to find a specific piece of information. Rather than having the user enter information and hope it’s valid, you can help your users find what they are looking for by offering autocomplete suggestions as they type.

So what could go wrong?

If your users are like me, they’ll make multiple spelling mistakes for every one word of text. If you’re creating an autocomplete field using regular expressions on your data, programming to account for misspellings and fat fingers is tough!

In this tutorial, we’re going to see how to create a simple web application that surfaces autocomplete suggestions to the user. These suggestions can be easily created using the full-text search features available in Atlas Search.

To get a better idea of what we want to accomplish, have a look at the following animated image:

In the above image you’ll notice that I not only made spelling mistakes, but I also made use of a word that appeared anywhere within the field for any document in a collection.

The Requirements

We’ll skip the basics of configuring Node.js or MongoDB and assume that you already have a few things installed, configured, and ready to go:

• MongoDB Atlas with an M0 cluster or better, with user and network safe-list configurations established
• Node.js installed and configured
• A food database with a recipes collection established

We’ll be using Atlas Search within MongoDB Atlas. To follow this tutorial, the recipes collection (within the food database) will expect documents that look like this:

{
    "_id": "5e5421451c9d440000e7ca13",
    "name": "chocolate chip cookies",
    "ingredients": [
        "sugar",
        "flour",
        "chocolate"
    ]
}

Make sure to create many documents within your recipes collection, some of which with similar names. In my example, I used “grilled cheese”, “five cheese lasagna”, and “baked salmon”.

Configuring MongoDB Atlas with a Search Index

Before we start creating a frontend or backend, we need to prepare our collection for search by creating a special search index.

Within the Collections tab of your cluster, find the recipes collection and then choose the Search tab.

You probably won’t have an Atlas Search index created yet, so we’ll need to create one. By default, Atlas Search dynamically maps every field in a collection. That means every field in our document will be checked against our search terms. This is great for growing collections where the schema may evolve, and you want to search through many different fields. However it can be resource intensive, as well. For our app, we actually just want to search by one particular field, the “name” field in our recipe documents. To do that, choose “Create Search Index” and change the code to the following:

{
    "mappings": {
        "dynamic": false,
        "fields": {
            "name": [
                {
                    "foldDiacritics": false,
                    "maxGrams": 7,
                    "minGrams": 3,
                    "tokenization": "edgeGram",
                    "type": "autocomplete"
                }
            ]
        }
    }
}

In the above example, we’re creating an index on the name field within our documents using an autocomplete index. Any fields that aren’t explicitly mapped, like the ingredients array, will not be searched.

Now, click “Create Index”. That’s it! Just give MongoDB Atlas a few minutes to create your search index.

If you want to learn more about Atlas Search autocomplete indexes and the various tokenization strategies that can be used, you can find information in the official documentation.

Create a Node.js with MongoDB Backend for Atlas Search

At this point in time, we should have our data collection of recipes, as well as an Atlas Search index created on that data for the name field. We’re now ready to create a backend that will interact with our data using the MongoDB Node.js driver.

We’re only going to brush over the getting started with MongoDB aspect of this backend application. If you want to read something more in-depth, check out Lauren Schaefer’s tutorial series on the subject.

On your computer, create a new project directory with a main.js file within that directory. Using the command line, execute the following commands:

npm init -y
npm install mongodb express body-parser cors --save

The above commands will initialize the package.json file and install each of our project dependencies for creating a RESTful API that interacts with MongoDB.

Within the main.js file, add the following code:

const { MongoClient, ObjectID } = require("mongodb");
const Express = require("express");
const Cors = require("cors");
const BodyParser = require("body-parser");
const { request } = require("express");

const client = new MongoClient(process.env["ATLAS_URI"]);
const server = Express();

server.use(BodyParser.json());
server.use(BodyParser.urlencoded({ extended: true }));
server.use(Cors());

var collection;

server.get("/search", async (request, response) => {});
server.get("/get/:id", async (request, response) => {});

server.listen("3000", async () => {
    try {
        await client.connect();
        collection = client.db("food").collection("recipes");
    } catch (e) {
        console.error(e);
    }
});

Remember when I said I’d be brushing over the getting started with MongoDB stuff? I meant it, but, if you’re copying and pasting the above code, make sure you replace the following line in your code:

const client = new MongoClient(process.env["ATLAS_URI"]);

I store my MongoDB Atlas information in an environment variable rather than hard-coding it into the application. If you wish to do the same, create an environment variable on your computer called ATLAS_URI and set it to your MongoDB connection string. This connection string will look something like this:

mongodb+srv://<username>:<password>@cluster0-yyarb.mongodb.net/<dbname>?retryWrites=true&w=majority

If you need help obtaining it, circle back to that tutorial by Lauren Schaefer that I had suggested.

What we’re interested in for this example are the /search and /get/:id endpoints for the RESTful API. The first endpoint will leverage Atlas Search, while the second endpoint will get the document based on its _id value. This is useful in case you want to search for documents and then get all the information about the selected document.

So let’s expand upon the endpoint for searching:

server.get("/search", async (request, response) => {
    try {
        let result = await collection.aggregate([
            {
                "$search": {
                    "autocomplete": {
                        "query": `${request.query.query}`,
                        "path": "name",
                        "fuzzy": {
                            "maxEdits": 2,
                            "prefixLength": 3
                        }
                    }
                }
            }
        ]).toArray();
        response.send(result);
    } catch (e) {
        response.status(500).send({ message: e.message });
    }
});

In the above code, we are creating an aggregation pipeline with a single $search stage, which will be powered by our Atlas Search index. It will use the user-provided data as the query and the autocomplete operator to give them that type-ahead experience. In a production scenario we might want to do further validation on the user provided data, but it’s fine for this example. Also note that when using Atlas Search, an aggregation pipeline is required, and the $search operator must be the first stage of that pipeline.

The path field of name represents the field within our documents that we want to search in. Remember, name is also the field we defined in our index.

This is where the fun stuff comes in!

We’re doing a fuzzy search. This means we’re finding strings which are similar, but not necessarily exactly the same, to the search term. Remember when I misspelled cheese by entering chease instead? The maxEdits field represents how many consecutive characters must match. In my example there was only one, but what if I misspelled it as cheaze where the az characters are not correct?

The prefixLength field indicates the number of characters at the beginning of each term in the result that must match exactly. In our example, three characters at the beginning of each term must match.

This is all very powerful considering what kind of mess your code would look like by using regular expressions or $text instead.

You can find more information on what can be used with the autocomplete operator in the documentation.

So let’s take care of our other endpoint:

server.get("/get/:id", async (request, response) => {
    try {
        let result = await collection.findOne({ "_id": ObjectID(request.params.id) });
        response.send(result);
    } catch (e) {
        response.status(500).send({ message: e.message });
    }
});

The above code is nothing fancy. We’re taking an id hash that the user provides, converting it into a proper object id, and then finding a single document.

You can test this application by first serving it with node main.js and then using a tool like Postman against the http://localhost:3000/search?query= or http://localhost:3000/get/ urls.

Develop a JavaScript with jQuery Frontend with Autocomplete Form Elements

Now that we have a backend to work with, we can take care of the frontend that improves the overall user experience. There are plenty of ways to create an autocomplete form, but for this example jQuery will be doing the heavy lifting.

Create a new project with an index.html file in it. Open that file and include the following:

<!DOCTYPE html>
<html>
    <head>
        <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
        <script src="//code.jquery.com/jquery-1.12.4.js"></script>
        <script src="//code.jquery.com/ui/1.12.1/jquery-ui.js"></script>
    </head>
    <body>
        <div class="ui-widget">
            <label for="recipe">Recipe:</label><br />
            <input id="recipe">
            <ul id="ingredients"></ul>
        </div>
        <script>
            $(document).ready(function () {});
        </script>
    </body>
</html>

The above markup doesn’t do much of anything. We’re just importing the jQuery dependencies and defining the form element that will show the autocomplete. After clicking the autocomplete element, the data will populate our list of ingredients in the list.

Within the <script> tag we can add the following:

<script>
    $(document).ready(function () {
        $("#recipe").autocomplete({
            source: async function(request, response) {
                let data = await fetch(`http://localhost:3000/search?query=${request.term}`)
                    .then(results => results.json())
                    .then(results => results.map(result => {
                        return { label: result.name, value: result.name, id: result._id };
                    }));
                response(data);
            },
            minLength: 2,
            select: function(event, ui) {
                fetch(`http://localhost:3000/get/${ui.item.id}`)
                    .then(result => result.json())
                    .then(result => {
                        $("#ingredients").empty();
                        result.ingredients.forEach(ingredient => {
                            $("#ingredients").append(`<li>${ingredient}</li>`);
                        });
                    });
            }
        });
    });
</script>

A few things are happening in the above code.

Within the autocomplete function, we define a source for where our data comes from and a select for what happens when we select something from our list. We also define a minLength so that we aren’t hammering our backend and database with every keystroke.

If we take a closer look at the source function, we have the following:

source: async function(request, response) {
    let data = await fetch(`http://localhost:3000/search?query=${request.term}`)
        .then(results => results.json())
        .then(results => results.map(result => {
            return { label: result.name, value: result.name, id: result._id };
        }));
    response(data);
},

We’re making a fetch against our backend, and then formatting the results into something the jQuery plugin recognizes. If you want to learn more about making HTTP requests with JavaScript, you can check out a previous tutorial I wrote titled Execute HTTP Requests in JavaScript Applications.

In the select function, we can further analyze what’s happening:

select: function(event, ui) {
    fetch(`http://localhost:3000/get/${ui.item.id}`)
        .then(result => result.json())
        .then(result => {
            $("#ingredients").empty();
            result.ingredients.forEach(ingredient => {
                $("#ingredients").append(`<li>${ingredient}</li>`);
            });
        });
}

We are making a second request to our other API endpoint. We are then flushing the list of ingredients on our page and repopulating them with the new ingredients.

When running this application, make sure the backend is running as well, otherwise your frontend will have nothing to communicate to.

To serve the frontend application, there are a few options not limited to what’s below:

• Host the code on MongoDB Realm or another hosting service.
• Use the serve package with Node.js.
• Use Python to create a SimpleHTTPServer.

My personal favorite is to use the serve package. If you install it, you can execute serve from your command line within the working directory of your project.

With the project serving with serve, it should be accessible at http://localhost:5000 in your web browser.

Conclusion

You just saw how to leverage MongoDB Atlas Search to suggest data to users in an autocomplete form. Atlas Search is great because it uses natural language to search within document fields to spare you from having to write long and complicated regular expressions or application logic.

Don’t forget that we did our search by using the $search operator within an aggregation pipeline. This means you could add other stages to your pipeline to do some really extravagant things. For example, after the $search pipeline stage, you could use an $in on the ingredients array to limit the results to only chocolate recipes. Also, you can make use of other neat operators within the $search stage, beyond the autocomplete operator. For example, you could make use of the near operator for numerical and geospatial search, or operators such as compound and wildcard for other tasks. More information on these operators can be found in the documentation.

This content first appeared on MongoDB.

Original article source at: https://www.mongodb.com/

#javascript #mongodb #nosql 

How to Choose SQL vs. NoSQL Databases for Your Project

NoSQL databases promise flexibility and performance benefits for certain use cases. But is it worth leaving behind the relational database comfort zone that guarantees consistency and organization? Read on and find the answers to the NoSQL vs. SQL database debate.

We probably all agree on the most basic definition of what a database is: a repository of information organized in such a way as to make it easy to update, search, and read.

The habit of working with relational databases instinctively leads us to think of databases as sets of interrelated tables composed of rows and columns. But the definition of a database does not talk about tables, rows, or columns.

When we put those concepts aside, we open up a range of possibilities for working with databases of unconventional design that are more suited to the modern world of Cloud computing, IoT, Big Data, real-time web applications, and unstructured information.

Welcome, NoSQL

The heterogeneous set of data models that don’t fit the traditional definition of a database is collectively known as NoSQL. This is because they don’t support standard Structured Query Language (SQL) – although many interpret NoSQL as “not only SQL”, indicating that other methods of data manipulation can be employed in addition to SQL.

The success that some NoSQL database models have achieved in recent times is due to their goal of providing more flexibility, more performance, more scalability, and more versatility than the good old relational model.

You should also note that, since it was coined in 2009,  the term NoSQL has become a buzzword that has proliferated in books, blogs, courses and CVs – to the point that many IT departments have allocated resources to the adoption of NoSQL technologies without knowing precisely what goal they were pursuing.

NoSQL models offer many advantages, but it is important to understand the:

• Differences between SQL and NoSQL databases.
• Situations in which NoSQL benefits can be reaped.

In this article, I will give you clues to decide properly between NoSQL and SQL databases.

Relational Databases and the Relational Model

To begin our analysis, let’s briefly examine relational databases and the relational model.

Relational databases organize information in tables consisting of rows and columns. All rows in a table have the same columns; each row is identified by a key consisting of one or more columns. The same key value cannot be repeated in different rows.

In relational databases, you can establish relationships between pairs of tables. These relationships are implemented as constraints that force the values of one or more columns of a table to be equal to the values of the key fields for some row of a related table.

Unlike other database models, the relational model ensures that logical structures are separate from physical storage structures. While not all database engines maintain this independence, a relational database should – in theory – be able to keep its logical structure intact even if its storage architecture is replaced. MySQL, for example, uses an architecture of pluggable storage engines that adapt to different forms of storage: clustered storage, replicated storage, in-memory storage, etc.

The separation between logical and physical structures allows application developers to access information without worrying about how that information is physically stored.

If you want to learn more about relational terminology, you can read its theory and history and FIND OUT WHY RELATIONAL DATABASES ARE CALLED “RELATIONAL''.

ERDs and SQL in Relational Databases

The relational standard was adopted by THE MOST POPULAR DATABASE MANAGEMENT SYSTEMS (DBMSs). These also use the SQL query language to write data retrieval and manipulation scripts in declarative form. Declarative form uses statements that describe the whole operation to be performed; this is instead of stating procedures that indicate, step by step, what to do with each individual piece of data.

Entity-relationship diagrams (ERDs) are the preferred tools for relational database design. ERDs allow us to visually define all the elements that form the structure of a database; they can be used to generate physical and operational databases. For more info, check out this 10-MINUTE READ ON THE BASICS OF DATA MODELING.

Entity-relationship diagrams allow to visually define all the elements that form the structure of a database.

Another important aspect of relational databases is that they impose a separation between data and metadata. Metadata is a separate repository from the actual data in which table definitions, views, constraints, indexes, etc. are stored. The database engine needs to query the metadata repository first to know how to access and manipulate the data itself.

The Shortcomings of the Relational Model

The drawback of relational databases is that the table structures do not allow for exceptions. If you want to insert a row with an attribute that the other rows in the table do not have, the relational model forces you to add that attribute to the table structure so that all rows then have it – even if they do not need it.

Similarly, it is impossible for an attribute to have values of different types in different rows of the same table. If a table has a field of type varchar(50), you will get an error if you try to store a 51-character string in it. You will have to extend the capacity of that field in the table structure if you want to store even one larger string. This extension will affect all the rows of the table, even if no other row needs more than 50 characters.

This rigidity of the relational model makes it unsuitable for the creation of repositories where the information lacks predefined structure. And today, many use cases require the handling of large volumes of unstructured information, such as information from massive web applications or real-time information collected by sensors or IoT devices.

Another shortcoming of the relational model is its difficult scalability. Relational databases use a scale-up architecture: to scale up, you need to add more storage, more memory, and more processing power to the database server. This form of growth is often expensive and sometimes simply not feasible.

Make Way for NoSQL

One of the differences between relational and NoSQL databases is that NoSQL was born alongside Cloud computing; it is perfectly suited to scale-out scalability architectures, where growth in capacity is achieved by spreading storage and data processing across large clusters of servers. When you need more capacity, you just add new servers to the cluster. So scalability is one of the main factors to consider when weighing NoSQL vs. SQL databases.

In addition to the advantages of scalability, the lack of predefined structures provides the flexibility to easily adapt to changes in information format or content. The fact that they allow handling unstructured information makes the term database less fitting forNoSQL repositories; many refer to them simply as data stores.

In most NoSQL data stores, schemas are flexible. Developers can define them as needed, which allows the data store to easily adapt to new types of information. This makes data stores suitable for Agile development methodologies, since changes in data definitions do not require a schema redesign.

However, it is not advisable to rely too much on the ease of working without designing schemas, thinking that it makes development more agile. WHAT I LIKE ABOUT DATABASE MODELING is the  BENEFITS it brings to keeping data organization under control.

Types of NoSQL Data Stores

Although NoSQL has a great diversity of database models, four types are the most popular:

• Document: Data is stored as documents with variable type information, usually encoded as JSON (although other encodings, such as XML, can also be used).
• Key-value: Data is stored in the form of key-value pairs, where a key is a unique identifier and has a single value associated with it.
• Graph: Data is stored in a graph structure, using nodes to contain information entities and vertices to define the relationships between nodes.
• Wide-table and columnar: Data is stored in flexible columns. When grouped, the information can be viewed as tables, but – unlike the relational model – the names and format of the columns can vary from row to row.

Some of the NoSQL data store models are ideal for very specific use cases but are not suitable for generality. For example, graph databases are the most suitable model where the relationships between data points are more relevant than the data itself. This situation applies in cases like fraud detection, identity management, or access control.

Likewise, wide-table and columnar databases are the preferred choice when large data sets need to be managed and distributed across multiple database nodes, and where the columns are not the same for each row. Log information, IoT sensor information, and real-time analytics are application examples for this type of database.

For these particular niches where NoSQL database models stand out for their effectiveness, comparisons with relational databases become unnecessary. However, document and key-value database models can be said to “invade” the terrain traditionally dominated by relational databases; in these cases, it makes sense to balance the advantages and disadvantages and compare relational SQL to NoSQL data models.

Modeling for SQL vs. NoSQL Databases

We have seen that the ideal way to model a relational database is using ERDs. This is a universally accepted fact. In the NoSQL world, however, there is no universal consensus on modeling techniques. It is even said that these databases are “schema-less”, suggesting that there is no need to design a schema prior to manipulating the data. This may be true when it comes to prototyping or exploratory development. However, the absence of data models becomes a problem when information repositories grow in complexity or in the number of data entities or relationships between them; in such cases, working with data models is inevitable.

In NoSQL data stores, the modeling technique depends on the type of database used. Moreover, as these technologies are still evolving, there are currently no universally accepted procedures for creating models that can then be automatically derived in operational NoSQL data stores.

For the time being, conceptual and logical ERDs are the tools of choice for creating document or key-value NoSQL databases, which you can model using an intelligent database design platform such as VERTABELO.

We will see below how to use ERDs with these two NoSQL models.

ERDs for Document Stores

Document NoSQL data stores are the most easily assimilated to ERD modeling because we can define an equivalence between relational and document database concepts. This is shown in the following table:

Equivalence between relational and NoSQL document model concepts.

Based on these equivalences, it is relatively easy to use a logical ERD to create a NoSQL data store of JSON documents. Let’s see how to adapt the simple invoicing model from above to a document store model. This logical schema can be automatically converted into a relational physical model. You can also see that it is normalized to avoid data redundancies.

Using the equivalences of relational objects and document objects, we could derive this JSON document store with just one document for each entity:

JSON Nested Documents

Unlike tables in the relational model, JSON documents support object nesting. There are cases where one entity is subordinate to another (as in the previous example, where InvoiceItem is subordinate to Invoice). When you want to derive a JSON document store from the logical model in these cases, it is best to denormalize the model to take advantage of the nesting feature.

Thus, the InvoiceItem entity can be contained within the Invoice entity, which in a document store simplifies data manipulation. For this, you should denormalize the logical diagram in all those cases in which you can take advantage of the JSON document nesting feature.

There are specific design tools for document stores that extend the entity-relationship diagram with the possibility of nesting entities, allowing to directly derive the models in JSON databases. But if you use a standard ERD design tool, you can also design nested entities using a notation that allows you to define nested objects. For example, you can prefix the attributes of the nested object with the object name:

To allow an ERD to model nested objects, you need to adopt some kind of notation, such as prefixing the attribute names of nested objects with the entity name.

Since the subordinate entity is contained within its parent entity, there is no necessity to include the foreign key (InvoiceNo). Translating this diagram to a JSON document store, it would look like this:

ERDs for Key-Value Data Stores

Key-value data stores excel in applications that require very high volumes of read/write operations, support for large numbers of users, high scalability, and redundancy to provide fault tolerance. On top of that, these data stores provide optimal performance. Some examples of applications that take advantage of key-value stores include:

• Applications that require fast scalability to adapt to seasonal usage peaks (such as e-commerce sites at Christmas time).
• Delivery of personalized digital ads on web pages or apps.
• E-commerce platform product recommendations.
• Multiplayer online games’ management of user sessions.
• Big Data research.

Key-value stores can be thought of as a single large table with only two fields: a key and a value. That large table can be sorted by the key field for optimized searching and data retrieval.

In general, key-value stores do not have a query language to facilitate data retrieval. They only support basic operations such as put, get, and delete. This implies that queries and transactions must be entirely handled at the application level.

Modeling Entities for a Key-Value Data Store

To model a key-value store with multiple entities, a logical ERD can be used without having to take into account the denormalization issues for document databases. The trick to handle multiple entities in a key-value data store is to establish a unique mechanism for translating the data elements of a relational table into key-value pairs.

To identify each data element in a relational model, we need at least a table name, a primary key value, an attribute name, and a value for the attribute. Any relational database can be translated to a list of these four elements – if you don’t mind leaving aside all the metadata, such as column type definitions. Then, if you can transform these four elements into key-value pairs with unique keys, we can (theoretically) turn any relational data model into a key-value store.

The most obvious way to do this is to concatenate the first three elements – entity name, primary key value, and attribute name – to form the key. The very definition of the relational model means we know this combination does not admit repetitions. The fourth element simply becomes the value corresponding to the composite key.

To concatenate the elements that form the key, we will need a separator element. This must necessarily be a character that cannot be part of any of the other elements, e.g. a pipe character (|). Following this criterion, we can represent a schema for customer information with an email addresses table as follows:

Relational model for our Customers info schema.

Sample customer data.

The customer schema translated to a key-value data store.

With this way of translating relational schemas to key-value stores we can map – in theory – any relational model to a key-value store. The main reason to do so would be to take advantage of the great performance and scalability of key-value data stores. However, the actual exploitation of that performance will depend on how you implement the mapping between the two models, which undoubtedly requires a very large programming effort. Depending on how this programming work is done, the performance of the key-value data store can be either very well or very poorly exploited.

NoSQL or SQL: Which Database Should You Choose?

When comparing NoSQL vs. SQL databases, we have seen that each NoSQL database model has a set of needs for which it is more suitable and effective. The relational model, on the other hand, aims to provide databases for any need, although with possibly less flexibility, scalability, or performance. But for highly complex data models with a large number of entities, the relational model is the most suitable to keep the data organized and its structures under control.

So, before jumping into the NoSQL wagon, take into account that there are still no universally accepted standards for query or data manipulation languages in that wagon, nor for the documentation or model presentation. For this reason, whoever chooses to work with NoSQL databases must be very sure to do so for a truly relevant reason. Is the greater scalability, flexibility, performance, or un-structuring critical to an application? Are those needs decidedly outside of what a relational database can provide? Unless you can answer yes to both questions, the good old relational model will undoubtedly be the safest choice.

Original article source at: https://www.vertabelo.com/

#sql #nosql #databases 

A Scalable, Distributed, Collaborative, Document-graph Database

Surrealdb is the ultimate cloud database for tomorrow's applications

Develop easier.   Build faster.   Scale quicker.

What is SurrealDB?

SurrealDB is an end-to-end cloud native database for web, mobile, serverless, jamstack, backend, and traditional applications. SurrealDB reduces the development time of modern applications by simplifying your database and API stack, removing the need for most server-side components, allowing you to build secure, performant apps quicker and cheaper. SurrealDB acts as both a database and a modern, realtime, collaborative API backend layer. SurrealDB can run as a single server or in a highly-available, highly-scalable distributed mode - with support for SQL querying from client devices, GraphQL, ACID transactions, WebSocket connections, structured and unstructured data, graph querying, full-text indexing, geospatial querying, and row-by-row permissions-based access.

View the features, the latest releases, the product roadmap, and documentation.

Features

•  Database server, or embedded library
•  Multi-row, multi-table ACID transactions
•  Single-node, or highly-scalable distributed mode
•  Record links and directed typed graph connections
•  Store structured and unstructured data
•  Incrementally computed views for pre-computed advanced analytics
•  Realtime-api layer, and security permissions built in
•  Store and model data in any way with tables, documents, and graph
•  Simple schema definition for frontend and backend development
•  Connect and query directly from web-browsers and client devices
•  Use embedded JavaScript functions for custom advanced functionality

Documentation

For guidance on installation, development, deployment, and administration, see our documentation.

Installation

SurrealDB is designed to be simple to install and simple to run - using just one command from your terminal. In addition to traditional installation, SurrealDB can be installed and run with HomeBrew, Docker, or using any other container orchestration tool such as Docker Compose, Docker Swarm, Rancher, or in Kubernetes.

Install on macOS

The quickest way to get going with SurrealDB on macOS is to use Homebrew. This will install both the command-line tools, and the SurrealDB server as a single executable. If you don't use Homebrew, follow the instructions for Linux below to install SurrealDB.

brew install surrealdb/tap/surreal

Install on Linux

The easiest and preferred way to get going with SurrealDB on Unix operating systems is to install and use the SurrealDB command-line tool. Run the following command in your terminal and follow the on-screen instructions.

curl --proto '=https' --tlsv1.2 -sSf https://install.surrealdb.com | sh

If you want a binary newer than what's currently released, you can install the nightly one.

curl --proto '=https' --tlsv1.2 -sSf https://install.surrealdb.com | sh -s -- --nightly

Install on Windows

The easiest and preferred way to get going with SurrealDB on Windows is to install and use the SurrealDB command-line tool. Run the following command in your terminal and follow the on-screen instructions.

iwr https://windows.surrealdb.com -useb | iex

Run using Docker

Docker can be used to manage and run SurrealDB database instances without the need to install any command-line tools. The SurrealDB docker container contains the full command-line tools for importing and exporting data from a running server, or for running a server itself.

docker run --rm --name surrealdb -p 8000:8000 surrealdb/surrealdb:latest start

For just getting started with a demonstration server running in memory, you can pass the container a basic initialization to set the user and password as root and enable logging and limit access to localhost (do not run this in production!)

docker run --rm --name surrealdb -p 127.0.0.1:8000:8000 surrealdb/surrealdb:latest start --log trace --user root --pass root memory

To update the image to the latest version:

docker pull surrealdb/surrealdb:latest

Getting started

Getting started with SurrealDB is as easy as starting up the SurrealDB database server, choosing your platform, and integrating its SDK into your code. You can easily get started with your platform of choice by reading one of our tutorials.

Client side apps

•  Getting started with Javascript
•  Getting started with WebAssembly
•  Getting started with Ember.js
•  Getting started with React.js
•  Getting started with Angular.js
•  Getting started with Vue.js
•  Getting started with Apollo GraphQL

Server side code

•  Getting started with Javascript
•  Getting started with Node.js
•  Getting started with Golang
•  Getting started with Rust
•  Getting started with Deno
•  Getting started with Python (coming soon)
•  Getting started with C (coming soon)
•  Getting started with Java (coming soon)
•  Getting started with Ruby (coming soon)
•  Getting started with PHP (coming soon)
•  Getting started with Swift (coming soon)
•  Getting started with R (coming soon)

Quick look

With strongly-typed data types, data can be fully modelled right in the database.

UPDATE person SET
    waist = <int> "34.59",
    height = <float> 201,
    score = <decimal> 0.3 + 0.3 + 0.3 + 0.1
;

Store dynamically computed fields which are calculated when retrieved.

CREATE person SET
    birthday = "2007-06-22",
    can_drive = <future> { time::now() > birthday + 18y }
;

Easily work with unstructured or structured data, in schema-less or schema-full mode.

-- Create a schemafull table
DEFINE TABLE user SCHEMAFULL;

-- Specify fields on the user table
DEFINE FIELD name ON TABLE user TYPE object;
DEFINE FIELD name.first ON TABLE user TYPE string;
DEFINE FIELD name.last ON TABLE user TYPE string;
DEFINE FIELD email ON TABLE user TYPE string ASSERT is::email($value);

-- Add a unique index on the email field preventing duplicate values
DEFINE INDEX email ON TABLE user COLUMNS email UNIQUE;

-- Create a new event whenever a user changes their email address
DEFINE EVENT email ON TABLE user WHEN $before.email != $after.email THEN (
    CREATE event SET user = $this, time = time::now(), value = $after.email, action = 'email_changed'
);

Connect records together with fully directed graph edge connections.

-- Add a graph edge between user:tobie and article:surreal
RELATE user:tobie->write->article:surreal
    SET time.written = time::now()
;

-- Add a graph edge between specific users and developers
LET $from = (SELECT users FROM company:surrealdb);
LET $devs = (SELECT * FROM user WHERE tags CONTAINS 'developer');
RELATE $from->like->$devs UNIQUE
    SET time.connected = time::now()
;

Query data flexibly with advanced expressions and graph queries.

-- Select a nested array, and filter based on an attribute
SELECT emails[WHERE active = true] FROM person;

-- Select all 1st, 2nd, and 3rd level people who this specific person record knows, or likes, as separate outputs
SELECT ->knows->(? AS f1)->knows->(? AS f2)->(knows, likes AS e3 WHERE influencer = true)->(? AS f3) FROM person:tobie;

-- Select all person records (and their recipients), who have sent more than 5 emails
SELECT *, ->sent->email->to->person FROM person WHERE count(->sent->email) > 5;

-- Select other products purchased by people who purchased this laptop
SELECT <-purchased<-person->purchased->product FROM product:laptop;

-- Select products purchased by people in the last 3 weeks who have purchased the same products that we purchased
SELECT ->purchased->product<-purchased<-person->(purchased WHERE created_at > time::now() - 3w)->product FROM person:tobie;

Store GeoJSON geographical data types, including points, lines and polygons.

UPDATE city:london SET
    centre = (-0.118092, 51.509865),
    boundary = {
        type: "Polygon",
        coordinates: [[
            [-0.38314819, 51.37692386], [0.1785278, 51.37692386],
            [0.1785278, 51.61460570], [-0.38314819, 51.61460570],
            [-0.38314819, 51.37692386]
        ]]
    }
;

Write custom embedded logic using JavaScript functions.

CREATE film SET
    ratings = [
        { rating: 6, user: user:bt8e39uh1ouhfm8ko8s0 },
        { rating: 8, user: user:bsilfhu88j04rgs0ga70 },
    ],
    featured = function() {
        return this.ratings.filter(r => {
            return r.rating >= 7;
        }).map(r => {
            return { ...r, rating: r.rating * 10 };
        });
    }
;

Specify granular access permissions for client and application access.

-- Specify access permissions for the 'post' table
DEFINE TABLE post SCHEMALESS
    PERMISSIONS
        FOR select
            -- Published posts can be selected
            WHERE published = true
            -- A user can select all their own posts
            OR user = $auth.id
        FOR create, update
            -- A user can create or update their own posts
            WHERE user = $auth.id
        FOR delete
            -- A user can delete their own posts
            WHERE user = $auth.id
            -- Or an admin can delete any posts
            OR $auth.admin = true
;

Why SurrealDB?

Database, API, and permissions

SurrealDB combines the database layer, the querying layer, and the API and authentication layer into one platform. Advanced table-based and row-based customisable access permissions allow for granular data access patterns for different types of users. There's no need for custom backend code and security rules with complicated database development.

Tables, documents, and graph

As a multi-model database, SurrealDB enables developers to use multiple techniques to store and model data, without having to choose a method in advance. With the use of tables, SurrealDB has similarities with relational databases, but with the added functionality and flexibility of advanced nested fields and arrays. Inter-document record links allow for simple to understand and highly-performant related queries without the use of JOINs, eliminating the N+1 query problem.

Advanced inter-document relations and analysis. No JOINs. No pain.

With full graph database functionality SurrealDB enables more advanced querying and analysis. Records (or vertices) can be connected to one another with edges, each with its own record properties and metadata. Simple extensions to traditional SQL queries allow for multi-table, multi-depth document retrieval, efficiently in the database, without the use of complicated JOINs and without bringing the data down to the client.

Simple schema definition for frontend and backend development

With SurrealDB, specify your database and API schema in one place, and define column rules and constraints just once. Once a schema is defined, database access is automatically granted to the relevant users. No more custom API code, and no more GraphQL integration. Simple, flexible, and ready for production in minutes not months.

Connect and query directly from web-browsers and client devices

Connect directly to SurrealDB from any end-user client device. Run SurrealQL queries directly within web-browsers, ensuring that users can only view or modify the data that they are allowed to access. Highly-performant WebSocket connections allow for efficient bi-directional queries, responses and notifications.

Query the database with the tools you want

Your data, your choice. SurrealDB is designed to be flexible to use, with support for SurrealQL, GraphQL (coming soon), CRUD support over REST, and JSON-RPC querying and modification over WebSockets. With direct-to-client connection with in-built permissions, SurrealDB speeds up the development process, and fits in seamlessly into any tech stack.

Realtime live queries and data changes direct to application

SurrealDB keeps every client device in-sync with data modifications pushed in realtime to the clients, applications, end-user devices, and server-side libraries. Live SQL queries allow for advanced filtering of the changes to which a client subscribes, and efficient data formats, including DIFFing and PATCHing enable highly-performant web-based data syncing.

Scale effortlessly to hundreds of nodes for high-availability and scalability

SurrealDB can be run as a single in-memory node, or as part of a distributed cluster - offering highly-available and highly-scalable system characteristics. Designed from the ground up to run in a distributed environment, SurrealDB makes use of special techniques when handling multi-table transactions, and document record IDs - with no use of table or row locks.

Extend your database with JavaScript functions

Embedded JavaScript functions allow for advanced, custom functionality, with computation logic being moved to the data layer. This improves upon the traditional approach of moving data to the client devices before applying any computation logic, ensuring that only the necessary data is transferred remotely. These advanced JavaScript functions, with support for the ES2020 standard, allow any developer to analyse the data in ever more simple-yet-advanced ways.

Designed to be embedded or to run distributed in the cloud

Built entirely in Rust as a single library, SurrealDB is designed to be used as both an embedded database library with advanced querying functionality, and as a database server which can operate in a distributed cluster. With low memory usage and cpu requirements, the system requirements have been specifically thought through for running in all types of environment.

Community

Join our growing community around the world, for help, ideas, and discussions regarding SurrealDB.

• View our official Blog
• Chat live with us on Discord
• Follow us on Twitter
• Connect with us on LinkedIn
• Visit us on YouTube
• Join our Dev community
• Questions tagged #surrealdb on Stack Overflow

Contributing

We would    for you to get involved with SurrealDB development! If you wish to help, you can learn more about how you can contribute to this project in the contribution guide.

Security

For security issues, view our vulnerability policy, view our security policy, and kindly email us at security@surrealdb.com instead of posting a public issue on GitHub.

Download Details:

Author: Surrealdb
Source Code: https://github.com/surrealdb/surrealdb 
License: View license

#serverless #sql #database #web #nosql 

NoSQL Databases Explained

What Is NoSQL?

📍What is NoSQL Database?

📌Oracle NoSQL Database Cloud Service is a fully managed database cloud service built to handle massive amounts of data at high velocities.

📌NoSQL, often known as "not only SQL" or "non-SQL," is a database design approach that allows for the storage and querying of data outside of the standard structures found in relational databases.

This video will cover: 
00:00 - What is NoSQL Database
01:43 - NoSQL Eco-system
02:46 - Why NoSQL
06:49 - NoSQL vs SQL Databases 
08:01 - Oracle NoSQL Database Cloud Service
09:15 - NoSQL Security 
09:54 - NoSQL Elasticity 
11:20 - Developer Overview 
13:24 - NoSQL Tables
14:09 - NoSQL throughput Provisioning
15:25 - How to register for FREE CLASS

Uses of NoSQL Database

It can store the data of any size and type : 
🔹Structured data 
🔹Semi-structured data
🔹Unstructured data

NoSQL database is basically meant for the unstructured data type.

NoSQL Tutorial: What is, Types of NoSQL Databases & Example

What is NoSQL?

NoSQL Database is a non-relational Data Management System, that does not require a fixed schema. It avoids joins, and is easy to scale. The major purpose of using a NoSQL database is for distributed data stores with humongous data storage needs. NoSQL is used for Big data and real-time web apps. For example, companies like Twitter, Facebook and Google collect terabytes of user data every single day.

NoSQL database stands for “Not Only SQL” or “Not SQL.” Though a better term would be “NoREL”, NoSQL caught on. Carl Strozz introduced the NoSQL concept in 1998.

Traditional RDBMS uses SQL syntax to store and retrieve data for further insights. Instead, a NoSQL database system encompasses a wide range of database technologies that can store structured, semi-structured, unstructured and polymorphic data. Let’s understand about NoSQL with a diagram in this NoSQL database tutorial:

In this NoSQL tutorial for beginners, you will learn NoSQL basics like:

• What is NoSQL?
• Why NoSQL?
• Brief History of NoSQL Databases
• Features of NoSQL
• Types of NoSQL Databases
• Query Mechanism tools for NoSQL
• What is the CAP Theorem?
• Eventual Consistency
• Advantages of NoSQL
• Disadvantages of NoSQL

Why NoSQL?

The concept of NoSQL databases became popular with Internet giants like Google, Facebook, Amazon, etc. who deal with huge volumes of data. The system response time becomes slow when you use RDBMS for massive volumes of data.

To resolve this problem, we could “scale up” our systems by upgrading our existing hardware. This process is expensive.

The alternative for this issue is to distribute database load on multiple hosts whenever the load increases. This method is known as “scaling out.”

NoSQL database is non-relational, so it scales out better than relational databases as they are designed with web applications in mind.

Brief History of NoSQL Databases

• 1998- Carlo Strozzi use the term NoSQL for his lightweight, open-source relational database
• 2000- Graph database Neo4j is launched
• 2004- Google BigTable is launched
• 2005- CouchDB is launched
• 2007- The research paper on Amazon Dynamo is released
• 2008- Facebooks open sources the Cassandra project
• 2009- The term NoSQL was reintroduced

Features of NoSQL

Non-relational

• NoSQL databases never follow the relational model
• Never provide tables with flat fixed-column records
• Work with self-contained aggregates or BLOBs
• Doesn’t require object-relational mapping and data normalization
• No complex features like query languages, query planners,referential integrity joins, ACID

Schema-free

• NoSQL databases are either schema-free or have relaxed schemas
• Do not require any sort of definition of the schema of the data
• Offers heterogeneous structures of data in the same domain

NoSQL is Schema-Free

Simple API

• Offers easy to use interfaces for storage and querying data provided
• APIs allow low-level data manipulation & selection methods
• Text-based protocols mostly used with HTTP REST with JSON
• Mostly used no standard based NoSQL query language
• Web-enabled databases running as internet-facing services

Distributed

• Multiple NoSQL databases can be executed in a distributed fashion
• Offers auto-scaling and fail-over capabilities
• Often ACID concept can be sacrificed for scalability and throughput
• Mostly no synchronous replication between distributed nodes Asynchronous Multi-Master Replication, peer-to-peer, HDFS Replication
• Only providing eventual consistency
• Shared Nothing Architecture. This enables less coordination and higher distribution.

NoSQL is Shared Nothing.

Types of NoSQL Databases

NoSQL Databases are mainly categorized into four types: Key-value pair, Column-oriented, Graph-based and Document-oriented. Every category has its unique attributes and limitations. None of the above-specified database is better to solve all the problems. Users should select the database based on their product needs.

Types of NoSQL Databases:

• Key-value Pair Based
• Column-oriented Graph
• Graphs based
• Document-oriented

Key Value Pair Based

Data is stored in key/value pairs. It is designed in such a way to handle lots of data and heavy load.

Key-value pair storage databases store data as a hash table where each key is unique, and the value can be a JSON, BLOB(Binary Large Objects), string, etc.

For example, a key-value pair may contain a key like “Website” associated with a value like “Guru99”.

It is one of the most basic NoSQL database example. This kind of NoSQL database is used as a collection, dictionaries, associative arrays, etc. Key value stores help the developer to store schema-less data. They work best for shopping cart contents.

Redis, Dynamo, Riak are some NoSQL examples of key-value store DataBases. They are all based on Amazon’s Dynamo paper.

Column-based

Column-oriented databases work on columns and are based on BigTable paper by Google. Every column is treated separately. Values of single column databases are stored contiguously.

Column based NoSQL database

They deliver high performance on aggregation queries like SUM, COUNT, AVG, MIN etc. as the data is readily available in a column.

Column-based NoSQL databases are widely used to manage data warehouses, business intelligence, CRM, Library card catalogs,

HBase, Cassandra, HBase, Hypertable are NoSQL query examples of column based database.

Document-Oriented:

Document-Oriented NoSQL DB stores and retrieves data as a key value pair but the value part is stored as a document. The document is stored in JSON or XML formats. The value is understood by the DB and can be queried.

Relational Vs. Document

In this diagram on your left you can see we have rows and columns, and in the right, we have a document database which has a similar structure to JSON. Now for the relational database, you have to know what columns you have and so on. However, for a document database, you have data store like JSON object. You do not require to define which make it flexible.

The document type is mostly used for CMS systems, blogging platforms, real-time analytics & e-commerce applications. It should not use for complex transactions which require multiple operations or queries against varying aggregate structures.

Amazon SimpleDB, CouchDB, MongoDB, Riak, Lotus Notes, MongoDB, are popular Document originated DBMS systems.

Graph-Based

A graph type database stores entities as well the relations amongst those entities. The entity is stored as a node with the relationship as edges. An edge gives a relationship between nodes. Every node and edge has a unique identifier.

Compared to a relational database where tables are loosely connected, a Graph database is a multi-relational in nature. Traversing relationship is fast as they are already captured into the DB, and there is no need to calculate them.

Graph base database mostly used for social networks, logistics, spatial data.

Neo4J, Infinite Graph, OrientDB, FlockDB are some popular graph-based databases.

Query Mechanism tools for NoSQL

The most common data retrieval mechanism is the REST-based retrieval of a value based on its key/ID with GET resource

Document store Database offers more difficult queries as they understand the value in a key-value pair. For example, CouchDB allows defining views with MapReduce

What is the CAP Theorem?

CAP theorem is also called brewer’s theorem. It states that is impossible for a distributed data store to offer more than two out of three guarantees

1. Consistency
2. Availability
3. Partition Tolerance

Consistency:

The data should remain consistent even after the execution of an operation. This means once data is written, any future read request should contain that data. For example, after updating the order status, all the clients should be able to see the same data.

Availability:

The database should always be available and responsive. It should not have any downtime.

Partition Tolerance:

Partition Tolerance means that the system should continue to function even if the communication among the servers is not stable. For example, the servers can be partitioned into multiple groups which may not communicate with each other. Here, if part of the database is unavailable, other parts are always unaffected.

Eventual Consistency

The term “eventual consistency” means to have copies of data on multiple machines to get high availability and scalability. Thus, changes made to any data item on one machine has to be propagated to other replicas.

Data replication may not be instantaneous as some copies will be updated immediately while others in due course of time. These copies may be mutually, but in due course of time, they become consistent. Hence, the name eventual consistency.

BASE: Basically Available, Soft state, Eventual consistency

• Basically, available means DB is available all the time as per CAP theorem
• Soft state means even without an input; the system state may change
• Eventual consistency means that the system will become consistent over time

Advantages of NoSQL

• Can be used as Primary or Analytic Data Source
• Big Data Capability
• No Single Point of Failure
• Easy Replication
• No Need for Separate Caching Layer
• It provides fast performance and horizontal scalability.
• Can handle structured, semi-structured, and unstructured data with equal effect
• Object-oriented programming which is easy to use and flexible
• NoSQL databases don’t need a dedicated high-performance server
• Support Key Developer Languages and Platforms
• Simple to implement than using RDBMS
• It can serve as the primary data source for online applications.
• Handles big data which manages data velocity, variety, volume, and complexity
• Excels at distributed database and multi-data center operations
• Eliminates the need for a specific caching layer to store data
• Offers a flexible schema design which can easily be altered without downtime or service disruption

Disadvantages of NoSQL

• No standardization rules
• Limited query capabilities
• RDBMS databases and tools are comparatively mature
• It does not offer any traditional database capabilities, like consistency when multiple transactions are performed simultaneously.
• When the volume of data increases it is difficult to maintain unique values as keys become difficult
• Doesn’t work as well with relational data
• The learning curve is stiff for new developers
• Open source options so not so popular for enterprises.

Summary

• NoSQL is a non-relational DMS, that does not require a fixed schema, avoids joins, and is easy to scale
• The concept of NoSQL databases became popular with Internet giants like Google, Facebook, Amazon, etc. who deal with huge volumes of data
• In the year 1998- Carlo Strozzi use the term NoSQL for his lightweight, open-source relational database
• NoSQL databases never follow the relational model it is either schema-free or has relaxed schemas
• Four types of NoSQL Database are 1). Key-value Pair Based 2). Column-oriented Graph 3). Graphs based 4). Document-oriented
• NOSQL can handle structured, semi-structured, and unstructured data with equal effect
• CAP theorem consists of three words Consistency, Availability, and Partition Tolerance
• BASE stands for Basically Available, Soft state, Eventual consistency
• The term “eventual consistency” means to have copies of data on multiple machines to get high availability and scalability
• NOSQL offer limited query capabilities

#nosql #sql #database #mongodb #mysql

A Complete Guide to NoSQL Databases

This tutorial on NoSQL Databases will help you understand what a NoSQL databes is, how is it different from traditional databases like MySQL and what are the advantages of using NoSQL database over SQL database.

NoSQL Tutorial: What is, Types of NoSQL Databases & Example

What is NoSQL?

NoSQL Database is a non-relational Data Management System, that does not require a fixed schema. It avoids joins, and is easy to scale. The major purpose of using a NoSQL database is for distributed data stores with humongous data storage needs. NoSQL is used for Big data and real-time web apps. For example, companies like Twitter, Facebook and Google collect terabytes of user data every single day.

NoSQL database stands for “Not Only SQL” or “Not SQL.” Though a better term would be “NoREL”, NoSQL caught on. Carl Strozz introduced the NoSQL concept in 1998.

Traditional RDBMS uses SQL syntax to store and retrieve data for further insights. Instead, a NoSQL database system encompasses a wide range of database technologies that can store structured, semi-structured, unstructured and polymorphic data. Let’s understand about NoSQL with a diagram in this NoSQL database tutorial:

In this NoSQL tutorial for beginners, you will learn NoSQL basics like:

• What is NoSQL?
• Why NoSQL?
• Brief History of NoSQL Databases
• Features of NoSQL
• Types of NoSQL Databases
• Query Mechanism tools for NoSQL
• What is the CAP Theorem?
• Eventual Consistency
• Advantages of NoSQL
• Disadvantages of NoSQL

Why NoSQL?

The concept of NoSQL databases became popular with Internet giants like Google, Facebook, Amazon, etc. who deal with huge volumes of data. The system response time becomes slow when you use RDBMS for massive volumes of data.

To resolve this problem, we could “scale up” our systems by upgrading our existing hardware. This process is expensive.

The alternative for this issue is to distribute database load on multiple hosts whenever the load increases. This method is known as “scaling out.”

NoSQL database is non-relational, so it scales out better than relational databases as they are designed with web applications in mind.

Brief History of NoSQL Databases

• 1998- Carlo Strozzi use the term NoSQL for his lightweight, open-source relational database
• 2000- Graph database Neo4j is launched
• 2004- Google BigTable is launched
• 2005- CouchDB is launched
• 2007- The research paper on Amazon Dynamo is released
• 2008- Facebooks open sources the Cassandra project
• 2009- The term NoSQL was reintroduced

Features of NoSQL

Non-relational

• NoSQL databases never follow the relational model
• Never provide tables with flat fixed-column records
• Work with self-contained aggregates or BLOBs
• Doesn’t require object-relational mapping and data normalization
• No complex features like query languages, query planners,referential integrity joins, ACID

Schema-free

• NoSQL databases are either schema-free or have relaxed schemas
• Do not require any sort of definition of the schema of the data
• Offers heterogeneous structures of data in the same domain

NoSQL is Schema-Free

Simple API

• Offers easy to use interfaces for storage and querying data provided
• APIs allow low-level data manipulation & selection methods
• Text-based protocols mostly used with HTTP REST with JSON
• Mostly used no standard based NoSQL query language
• Web-enabled databases running as internet-facing services

Distributed

• Multiple NoSQL databases can be executed in a distributed fashion
• Offers auto-scaling and fail-over capabilities
• Often ACID concept can be sacrificed for scalability and throughput
• Mostly no synchronous replication between distributed nodes Asynchronous Multi-Master Replication, peer-to-peer, HDFS Replication
• Only providing eventual consistency
• Shared Nothing Architecture. This enables less coordination and higher distribution.

NoSQL is Shared Nothing.

Types of NoSQL Databases

NoSQL Databases are mainly categorized into four types: Key-value pair, Column-oriented, Graph-based and Document-oriented. Every category has its unique attributes and limitations. None of the above-specified database is better to solve all the problems. Users should select the database based on their product needs.

Types of NoSQL Databases:

• Key-value Pair Based
• Column-oriented Graph
• Graphs based
• Document-oriented

Key Value Pair Based

Data is stored in key/value pairs. It is designed in such a way to handle lots of data and heavy load.

Key-value pair storage databases store data as a hash table where each key is unique, and the value can be a JSON, BLOB(Binary Large Objects), string, etc.

For example, a key-value pair may contain a key like “Website” associated with a value like “Guru99”.

It is one of the most basic NoSQL database example. This kind of NoSQL database is used as a collection, dictionaries, associative arrays, etc. Key value stores help the developer to store schema-less data. They work best for shopping cart contents.

Redis, Dynamo, Riak are some NoSQL examples of key-value store DataBases. They are all based on Amazon’s Dynamo paper.

Column-based

Column-oriented databases work on columns and are based on BigTable paper by Google. Every column is treated separately. Values of single column databases are stored contiguously.

Column based NoSQL database

They deliver high performance on aggregation queries like SUM, COUNT, AVG, MIN etc. as the data is readily available in a column.

Column-based NoSQL databases are widely used to manage data warehouses, business intelligence, CRM, Library card catalogs,

HBase, Cassandra, HBase, Hypertable are NoSQL query examples of column based database.

Document-Oriented:

Document-Oriented NoSQL DB stores and retrieves data as a key value pair but the value part is stored as a document. The document is stored in JSON or XML formats. The value is understood by the DB and can be queried.

Relational Vs. Document

In this diagram on your left you can see we have rows and columns, and in the right, we have a document database which has a similar structure to JSON. Now for the relational database, you have to know what columns you have and so on. However, for a document database, you have data store like JSON object. You do not require to define which make it flexible.

The document type is mostly used for CMS systems, blogging platforms, real-time analytics & e-commerce applications. It should not use for complex transactions which require multiple operations or queries against varying aggregate structures.

Amazon SimpleDB, CouchDB, MongoDB, Riak, Lotus Notes, MongoDB, are popular Document originated DBMS systems.

Graph-Based

A graph type database stores entities as well the relations amongst those entities. The entity is stored as a node with the relationship as edges. An edge gives a relationship between nodes. Every node and edge has a unique identifier.

Compared to a relational database where tables are loosely connected, a Graph database is a multi-relational in nature. Traversing relationship is fast as they are already captured into the DB, and there is no need to calculate them.

Graph base database mostly used for social networks, logistics, spatial data.

Neo4J, Infinite Graph, OrientDB, FlockDB are some popular graph-based databases.

Query Mechanism tools for NoSQL

The most common data retrieval mechanism is the REST-based retrieval of a value based on its key/ID with GET resource

Document store Database offers more difficult queries as they understand the value in a key-value pair. For example, CouchDB allows defining views with MapReduce

What is the CAP Theorem?

CAP theorem is also called brewer’s theorem. It states that is impossible for a distributed data store to offer more than two out of three guarantees

1. Consistency
2. Availability
3. Partition Tolerance

Consistency:

The data should remain consistent even after the execution of an operation. This means once data is written, any future read request should contain that data. For example, after updating the order status, all the clients should be able to see the same data.

Availability:

The database should always be available and responsive. It should not have any downtime.

Partition Tolerance:

Partition Tolerance means that the system should continue to function even if the communication among the servers is not stable. For example, the servers can be partitioned into multiple groups which may not communicate with each other. Here, if part of the database is unavailable, other parts are always unaffected.

Eventual Consistency

The term “eventual consistency” means to have copies of data on multiple machines to get high availability and scalability. Thus, changes made to any data item on one machine has to be propagated to other replicas.

Data replication may not be instantaneous as some copies will be updated immediately while others in due course of time. These copies may be mutually, but in due course of time, they become consistent. Hence, the name eventual consistency.

BASE: Basically Available, Soft state, Eventual consistency

• Basically, available means DB is available all the time as per CAP theorem
• Soft state means even without an input; the system state may change
• Eventual consistency means that the system will become consistent over time

Advantages of NoSQL

• Can be used as Primary or Analytic Data Source
• Big Data Capability
• No Single Point of Failure
• Easy Replication
• No Need for Separate Caching Layer
• It provides fast performance and horizontal scalability.
• Can handle structured, semi-structured, and unstructured data with equal effect
• Object-oriented programming which is easy to use and flexible
• NoSQL databases don’t need a dedicated high-performance server
• Support Key Developer Languages and Platforms
• Simple to implement than using RDBMS
• It can serve as the primary data source for online applications.
• Handles big data which manages data velocity, variety, volume, and complexity
• Excels at distributed database and multi-data center operations
• Eliminates the need for a specific caching layer to store data
• Offers a flexible schema design which can easily be altered without downtime or service disruption

Disadvantages of NoSQL

• No standardization rules
• Limited query capabilities
• RDBMS databases and tools are comparatively mature
• It does not offer any traditional database capabilities, like consistency when multiple transactions are performed simultaneously.
• When the volume of data increases it is difficult to maintain unique values as keys become difficult
• Doesn’t work as well with relational data
• The learning curve is stiff for new developers
• Open source options so not so popular for enterprises.

Summary

• NoSQL is a non-relational DMS, that does not require a fixed schema, avoids joins, and is easy to scale
• The concept of NoSQL databases became popular with Internet giants like Google, Facebook, Amazon, etc. who deal with huge volumes of data
• In the year 1998- Carlo Strozzi use the term NoSQL for his lightweight, open-source relational database
• NoSQL databases never follow the relational model it is either schema-free or has relaxed schemas
• Four types of NoSQL Database are 1). Key-value Pair Based 2). Column-oriented Graph 3). Graphs based 4). Document-oriented
• NOSQL can handle structured, semi-structured, and unstructured data with equal effect
• CAP theorem consists of three words Consistency, Availability, and Partition Tolerance
• BASE stands for Basically Available, Soft state, Eventual consistency
• The term “eventual consistency” means to have copies of data on multiple machines to get high availability and scalability
• NOSQL offer limited query capabilities

#nosql #database 

What is NoSQL Database?

What Is NoSQL Database | NoSQL vs SQL Difference | NoSQL Tutorial For Beginners

This video is based on NoSQL tutorial. The topics covered in this NoSQL tutorial video are essential for mastering NoSQL. In this tutorial we will cover what is NoSQL, difference between RDBMS and NoSQL databases, benefits of NoSQL and types of NoSQL. NoSQL (commonly referred to as "Not Only SQL") represents a completely different framework of databases that allows for high-performance, agile processing of information at massive scale.                                                             

What is NoSQL?

In this article, you'll learn what a NoSQL database is, why (and when!) you should use one, and how to get started.

NoSQL databases (aka "not only SQL") are non-tabular databases and store data differently than relational tables. NoSQL databases come in a variety of types based on their data model. The main types are document, key-value, wide-column, and graph. They provide flexible schemas and scale easily with large amounts of data and high user loads.

What is a NoSQL database?

When people use the term “NoSQL database,” they typically use it to refer to any non-relational database. Some say the term “NoSQL” stands for “non SQL” while others say it stands for “not only SQL.” Either way, most agree that NoSQL databases are databases that store data in a format other than relational tables.

Brief history of NoSQL databases

NoSQL databases emerged in the late 2000s as the cost of storage dramatically decreased. Gone were the days of needing to create a complex, difficult-to-manage data model in order to avoid data duplication. Developers (rather than storage) were becoming the primary cost of software development, so NoSQL databases optimized for developer productivity.

As storage costs rapidly decreased, the amount of data that applications needed to store and query increased. This data came in all shapes and sizes — structured, semi-structured, and polymorphic — and defining the schema in advance became nearly impossible. NoSQL databases allow developers to store huge amounts of unstructured data, giving them a lot of flexibility.

Additionally, the Agile Manifesto was rising in popularity, and software engineers were rethinking the way they developed software. They were recognizing the need to rapidly adapt to changing requirements. They needed the ability to iterate quickly and make changes throughout their software stack — all the way down to the database. NoSQL databases gave them this flexibility.

Cloud computing also rose in popularity, and developers began using public clouds to host their applications and data. They wanted the ability to distribute data across multiple servers and regions to make their applications resilient, to scale out instead of scale up, and to intelligently geo-place their data. Some NoSQL databases like MongoDB provide these capabilities.

NoSQL database features

Each NoSQL database has its own unique features. At a high level, many NoSQL databases have the following features:

• Flexible schemas
• Horizontal scaling
• Fast queries due to the data model
• Ease of use for developers

Types of NoSQL databases

Over time, four major types of NoSQL databases emerged: document databases, key-value databases, wide-column stores, and graph databases.

• Document databases store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects.
• Key-value databases are a simpler type of database where each item contains keys and values.
• Wide-column stores store data in tables, rows, and dynamic columns.
• Graph databases store data in nodes and edges. Nodes typically store information about people, places, and things, while edges store information about the relationships between the nodes.

Difference between RDBMS and NoSQL databases

While a variety of differences exist between relational database management systems (RDBMS) and NoSQL databases, one of the key differences is the way the data is modeled in the database. In this section, we'll work through an example of modeling the same data in a relational database and a NoSQL database. Then, we'll highlight some of the other key differences between relational databases and NoSQL databases.

RDBMS vs NoSQL: Data Modeling Example

Let's consider an example of storing information about a user and their hobbies. We need to store a user's first name, last name, cell phone number, city, and hobbies.

In a relational database, we'd likely create two tables: one for Users and one for Hobbies.

Users

Hobbies

In order to retrieve all of the information about a user and their hobbies, information from the Users table and Hobbies table will need to be joined together.

The data model we design for a NoSQL database will depend on the type of NoSQL database we choose. Let's consider how to store the same information about a user and their hobbies in a document database like MongoDB.

{
   "_id": 1,
   "first_name": "Leslie",
   "last_name": "Yepp",
   "cell": "8125552344",
   "city": "Pawnee",
   "hobbies": ["scrapbooking", "eating waffles", "working"]
}

In order to retrieve all of the information about a user and their hobbies, a single document can be retrieved from the database. No joins are required, resulting in faster queries.

Other differences between RDBMS and relational databases

While the example above highlights the differences in data models between relational databases and NoSQL databases, many other important differences exist, including:

• Flexibility of the schema
• Scaling technique
• Support for transactions
• Reliance on data to object mapping

Why NoSQL?

NoSQL databases are used in nearly every industry. Use cases range from the highly critical (e.g., storing financial data and healthcare records) to the more fun and frivolous (e.g., storing IoT readings from a smart kitty litter box).

In the following sections, we'll explore when you should choose to use a NoSQL database and common misconceptions about NoSQL databases.

When should NoSQL be used?

When deciding which database to use, decision-makers typically find one or more of the following factors lead them to selecting a NoSQL database:

• Fast-paced Agile development
• Storage of structured and semi-structured data
• Huge volumes of data
• Requirements for scale-out architecture
• Modern application paradigms like microservices and real-time streaming

NoSQL database misconceptions

Over the years, many misconceptions about NoSQL databases have spread throughout the developer community. In this section, we'll discuss two of the most common misconceptions:

• Relationship data is best suited for relational databases.
• NoSQL databases don't support ACID transactions.

Misconception: relationship data is best suited for relational databases

A common misconception is that NoSQL databases or non-relational databases don’t store relationship data well. NoSQL databases can store relationship data — they just store it differently than relational databases do.

In fact, when compared with relational databases, many find modeling relationship data in NoSQL databases to be easier than in relational databases, because related data doesn’t have to be split between tables. NoSQL data models allow related data to be nested within a single data structure.

Misconception: NoSQL databases don't support ACID transactions

Another common misconception is that NoSQL databases don't support ACID transactions. Some NoSQL databases like MongoDB do, in fact, support ACID transactions.

Note that the way data is modeled in NoSQL databases can eliminate the need for multi-record transactions in many use cases. Consider the earlier example where we stored information about a user and their hobbies in both a relational database and a document database. In order to ensure information about a user and their hobbies was updated together in a relational database, we'd need to use a transaction to update records in two tables. In order to do the same in a document database, we could update a single document — no multi-record transaction required.

NoSQL query tutorial

A variety of NoSQL databases exist. Today, we'll be trying MongoDB, the world's most popular NoSQL database according to DB-Engines.

In this tutorial, you'll load a sample database and learn to query it — all without installing anything on your computer or paying anything.

Authenticate to MongoDB Atlas

The easiest way to get started with MongoDB is MongoDB Atlas. Atlas is MongoDB's fully managed database-as-a-service. Atlas has a forever free tier, which is what you'll be using today.

1. Navigate to Atlas.
2. Create an account if you haven't already.
3. Log into Atlas.
4. Create an Atlas organization and project.

Create a cluster and a database

A cluster is a place where you can store your MongoDB databases. In this section, you'll create a free cluster.

Once you have a cluster, you can begin storing data in Atlas. You could choose to manually create a database in the Atlas Data Explorer, in the MongoDB Shell, in MongoDB Compass, or using your favorite programming language. Instead, in this example, you will import Atlas's sample dataset.

1. Create a free cluster by following the steps in the official MongoDB documentation.
2. Load the sample dataset by following the instructions in the official MongoDB documentation.

Loading the sample dataset will take several minutes.

While we don't need to think about database design for this tutorial, note that database design and data modeling are major factors in MongoDB performance.

Query the database

Now that you have data in your cluster, let's query it! Just like you had multiple ways to create a database, you have multiple options for querying a database: in the Atlas Data Explorer, in the MongoDB Shell, in MongoDB Compass, or using your favorite programming language.

In this section, you’ll query the database using the Atlas Data Explorer. This is a good way to get started querying, as it requires zero setup.

Navigate to the Data Explorer (the Collections tab), if you are not already there. See the official MongoDB documentation for information on how to navigate to the Data Explorer.

The left panel of the Data Explorer displays a list of databases and collections in the current cluster. The right panel of the Data Explorer displays a list of documents in the current collection.

The Data Explorer displays a list of documents in the listingsAndReviews collection.

Expand the sample_mflix database in the left panel. A list of the database's collections is displayed.

Select the movies collection. The Find View is displayed in the right panel. The first twenty documents of the results are displayed.

You are now ready to query the movies collection. Let's query for the movie Pride and Prejudice. In the query bar, input { title: "Pride and Prejudice"} in the query bar and click Apply.

Two documents with the title “Pride and Prejudice” are returned.

The results for querying for movies with the title "Pride and Prejudice".

Congrats! You've successfully queried a NoSQL database!

Continue exploring your data

In this tutorial, we only scratched the surface of what you can do in MongoDB and Atlas.
Continue interacting with your data by using the Data Explorer to insert new documents, edit existing documents, and delete documents.

When you are ready to try more advanced queries that aggregate your data, create an aggregation pipeline. The aggregation framework is an incredibly powerful tool for analyzing your data. To learn more, take the free MongoDB University Course M121 The MongoDB Aggregation Framework.

When you want to visualize your data, check out MongoDB Charts. Charts is the easiest way to visualize data stored in Atlas and Atlas Data Lake. Charts allows you to create dashboards that are filled with visualizations of your data.

Summary

NoSQL databases provide a variety of benefits including flexible data models, horizontal scaling, lightning fast queries, and ease of use for developers. NoSQL databases come in a variety of types including document databases, key-values databases, wide-column stores, and graph databases.

#database #nosql #sql #mongodb #mysql