Create a Marvel Database with SQL and Javascript, the easy way

Originally published by Henry Heberle, PhD at towardsdatascience.com

This is a post for who doesn’t have time to learn the complex syntax of the modified sql used in the Wikidata system and still would like to play with their data.

The original reason for me to have a comics database was to create this visualization, which you will see again at the end of this article:

Legend of Marvel & DC visualization: https://heberleh.github.io/comics-universe/

The pipeline

The idea here is to go through the following steps to get a flexible database:

1. Query tables from Wikidata and export as json files

2. Create an Alasql database and insert the data from the json files

3. Use the created database performing SQL queries

1. Extracting data from Wikipedia

Wikipedia has a great database and ontology behind it. You can access the query system by clicking here: https://w.wiki/4iP

In the above link you will find something similar to this (and you will find more information about it there):

SELECT ?name ?gender ?genderLabel
  WHERE{
      ?name wdt:P31 wd:Q1114461;
          wdt:P1080 wd:Q931597.
      ?name wdt:P21 ?gender.
      SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}

In this code I am selecting entities that are instances of comic character and are in the fictional universe Marvel Universe. It may seem strange at first, but each of those codes are part of the Wikipedia Ontology. It is how they deal with redundancy and can organize and make everything work! But how can you create your own queries?

First, will help you. Try typing wdt:<Ctrl+Space>. You notice that a helper tooltip pops up for you. Try typing instance of. You see that the code for ‘instance of’ is wdt:P31. Using the same approach, now for a value, you will type wdt:<Ctrl+Space> (no ‘t’ this time!) and comic character, enter, and voilà, you got the code wd:Q1114461.

<Ctrl+Space> opens the helper

Okay, Henry, but I have no idea about what we can use, I don’t have time to learn the Wikipedia language… and I understand you! So, to overcome this we will open this page here Juggernaut. This is the Wikipedia item code Q841372, the Juggernaut! Now look and check what is the information available for a comic character. Put your mouse pointer over the items in the Statements Section. For example, if you put your mouse over “instance of” you will get the P31 code! Since it is not a value, use wdt for this one.

By browsing that page you get a good idea about what are the properties you can use. Pause: If you want to create a database but don’t know yet about what it will be, try the Random article function to get insights. We can do the same thing for any Wikipedia page. Go to the official Wikipedia website and search for anything, let’s say… Aluminium. Now check the left-side bar and look for Wikidata item link.

Click, and voilà, it shows all the attributes for this item. We see we get the mass by using wdt:P2067, and many other attributes that may apply for other types of metal.

Each Wikidata entry has many properties. The tooltip tells you the property ID.

Now that we have the Wikidata, we will:

1. Create a query for an attribute of interest

2. Download the result as .json

3. Always track the IDs (Wikidata item URL) and Labels

For instance, I created one query for gender. One query for occupation, and so on. The result is a folder with many .json files I will load in my database formed by many tables. You could perform a single query with all the information. In my case, I want it to be malleable, so I can perform JOIN, LEFT JOIN , and other SELECT commands to get different tables.

In fact, instead of getting only Marvel Universe, I discarded that filter and got a full database of instances of Fictional Characters (not only Comics Characters — for some reason this one would not query Captain America). So, in my json files, you find Marvel, DC, and others. The resulting jsons are found in my GitHub page. Next, I will show how I used the json files with an easy SQL DB in Javascript.

As I mentioned in the last paragraph, my query is actually about the class Fictional Characters, and its sub-classes. I also queried partners and spouses, so for this I used the command | to define two wdt:

SELECT DISTINCT ?char ?partner  ?partnerLabel
WHERE
{
	?char wdt:P31/wdt:P279* wd:Q95074. # instande of Fictional Character and any sub-class
    ?char wdt:P451|wdt:P26 ?partner  # partner|spouse
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}

Note that in the above code I am not filtering for Marvel anymore. For more complex queries or handling lists, please refer to the Wikidata website. Another good option is the Help button at the top of the Wikidata Query Service where you can find examples and support.

2. Creating an SQL Database in Javascript

Now that we have the json files, let’s insert them into our SQL Database. For this, we will use a tool named Alasql. With Alasql, you can load data and create tables in different manners (read more here).

Here, since we have the jsons from Wikidata, we will apply a SELECT on each json (file) and insert into a new table in our database for easy access. For simplification, my code will contain only two tables; you can find the complete updated version here. It looks like this:

// Partial code - check my github to get the complete one
import alasql from 'alasql'
const character = require('../dataset/character.json');
const abilities = require('../dataset/abilities.json');
const fictionalUniverse = require('../dataset/fictionalUniverse.json');
function MarvelDB(){
// Our jsons and table names
let tables = [
    {name: 'abilities', data: abilities},
    {name: 'character', data: character}
]

// Create database
let db = new alasql.Database()

// Only characters registered in the Marvel Universe
db.exec('CREATE TABLE fictionalUniverse')
db.exec('INSERT INTO fictionalUniverse\
            SELECT char FROM ? \
            WHERE fictionalUniverseLabel LIKE "Marvel Universe"', [fictionalUniverse])

// For each Json in 'tables', create table
tables.forEach(table =&gt; {
    // Creates temporary table (Should not be necessary, 
    // but because of bugs caused by nesting INSERT INTO with SELECT WHERE IN,)
    // I kept it like this:
    db.exec(`CREATE TABLE aux_${table.name}`);
    db.exec(`INSERT INTO aux_${table.name} SELECT * FROM ?`, [table.data]);
    let filtered_data = db.exec(`SELECT * FROM aux_${table.name}\
                                    WHERE aux_${table.name}.char IN\
                                    (SELECT char FROM fictionalUniverse)`);

    // Then when can create our table
    db.exec(`CREATE TABLE ${table.name}`);
    db.exec(`INSERT INTO ${table.name} SELECT * FROM ?`, [filtered_data]);

    // and delete the temporary one
    db.exec(`DROP TABLE  aux_${table.name}`);
})

// Drop fictionalUniverse table -&gt; character table contains the same elements
db.exec('DROP TABLE fictionalUniverse')

return db

}

We are set. We have a database and can use it by calling let mydb = MarvelDB() and apply SQL commands like this: mydb.exec(‘SELECT DISTINCT abilityLabel from abilities’).

3. Final code and usage

I defined some tests to make sure everything was working. It exemplifies the usage of our database. If you are not used to testing, just consider the let result = this.db.exec(…) and console.table(result) in the code below and ignore the other parts.

// Unit test

describe(‘Select Queries’, function() {
beforeEach(() => this.db = marvelDB())
it(‘Name of the only agender with registered abilities’, () => {

let result = this.db.exec(

‘SELECT DISTINCT character.charLabel AS name

FROM character JOIN gender ON character.char = gender.char

JOIN abilities ON character.char = abilities.char

WHERE gender.genderLabel LIKE “agender”’);

//console.table(result);

expect(result[0].name).to.equal(“Phoenix Force”)

})

})

The above SELECT returns the column charLabel and renames to name. It also joins the tables character, gender and abilities, making sure the entries have the same character id (char). Finally, it filters by a specific gender named agender. For more examples on this database, please go to the GitHub page - don’t forget to star it :)

Originally published by Henry Heberle, PhD at towardsdatascience.com

==============================================================================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter

Learn More

☞ The Ultimate MySQL Bootcamp: Go from SQL Beginner to Expert

☞ Learn MySQL - For Beginners

☞ SQL - MySQL for Data Analytics and Business Intelligence

☞ Learn Database Design with MySQL

☞ SQL for Beginners: Learn SQL using MySQL and Database Design

☞ The Complete MySQL Developer Course

☞ SQL Tutorial: Learn SQL with MySQL Database -Beginner2Expert

☞ MySQL Database: SQL- Learn MySQL & MySQL Database Management

#data-science #database #javascript

What is GEEK

Buddha Community

Create a Marvel Database with SQL and Javascript, the easy way

Cayla Erdman

1594369800

Introduction to Structured Query Language SQL pdf

SQL stands for Structured Query Language. SQL is a scripting language expected to store, control, and inquiry information put away in social databases. The main manifestation of SQL showed up in 1974, when a gathering in IBM built up the principal model of a social database. The primary business social database was discharged by Relational Software later turning out to be Oracle.

Models for SQL exist. In any case, the SQL that can be utilized on every last one of the major RDBMS today is in various flavors. This is because of two reasons:

1. The SQL order standard is genuinely intricate, and it isn’t handy to actualize the whole standard.

2. Every database seller needs an approach to separate its item from others.

Right now, contrasts are noted where fitting.

#programming books #beginning sql pdf #commands sql #download free sql full book pdf #introduction to sql pdf #introduction to sql ppt #introduction to sql #practical sql pdf #sql commands pdf with examples free download #sql commands #sql free bool download #sql guide #sql language #sql pdf #sql ppt #sql programming language #sql tutorial for beginners #sql tutorial pdf #sql #structured query language pdf #structured query language ppt #structured query language

Ruth Nabimanya

1621850444

List of Available Database for Current User In SQL Server

Introduction

When working in the SQL Server, we may have to check some other databases other than the current one which we are working. In that scenario we may not be sure that does we have access to those Databases?. In this article we discuss the list of databases that are available for the current logged user in SQL Server

Get the list of database

Conclusion

#sql server #available databases for current user #check database has access #list of available database #sql #sql query #sql server database #sql tips #sql tips and tricks #tips

Ruth Nabimanya

1620633584

System Databases in SQL Server

Introduction

In SSMS, we many of may noticed System Databases under the Database Folder. But how many of us knows its purpose?. In this article lets discuss about the System Databases in SQL Server.

System Database

Fig. 1 System Databases

There are five system databases, these databases are created while installing SQL Server.

Master
Model
MSDB
Tempdb
Resource

Master

This database contains all the System level Information in SQL Server. The Information in form of Meta data.
Because of this master database, we are able to access the SQL Server (On premise SQL Server)

Model

This database is used as a template for new databases.
Whenever a new database is created, initially a copy of model database is what created as new database.

MSDB

This database is where a service called SQL Server Agent stores its data.
SQL server Agent is in charge of automation, which includes entities such as jobs, schedules, and alerts.

TempDB

The Tempdb is where SQL Server stores temporary data such as work tables, sort space, row versioning information and etc.
User can create their own version of temporary tables and those are stored in Tempdb.
But this database is destroyed and recreated every time when we restart the instance of SQL Server.

Resource

The resource database is a hidden, read only database that holds the definitions of all system objects.
When we query system object in a database, they appear to reside in the sys schema of the local database, but in actually their definitions reside in the resource db.

#sql server #master system database #model system database #msdb system database #sql server system databases #ssms #system database #system databases in sql server #tempdb system database

Cayla Erdman

1597214640

AlaSQL in Action: The JavaScript SQL Database

What’s AlaSQL?

For starters, this open-source project garners 11k weekly downloads off npm and has over 5K stars on GitHub.

I was surprised to see that there aren’t more posts about this popular lightweight client-side in-memory SQL database online apart from this awesome article I found. However, AlaSQL’s website gets straight to the point:

JavaScript SQL database for browser and Node.js.
Handles both traditional relational tables and nested JSON data (NoSQL).
Export, store, and import data from localStorage, IndexedDB, or Excel.

What’s not to love? AlaSQL was designed to work in browser and Node.js, is fast, and super easy to use. Some say that when you need to find data fast and want a better alternative to a full database or Redis, check out AlaSQL.

We Love AlaSQL, too!

HarperDB has had the pleasure of using AlaSQL for its backend SQL functionality. We chose to use AlaSQL because it has extensive language support, is well supported and extensible, can execute SQL against data sets (JSON or Arrays), and the ability to generate reusable functions. AlaSQL enables us to convert SQL language into an Abstract Syntax Tree (AST) that we can programmatically interpret into our data model. We can feed data into a processor that can perform the calculations native to SQL, using the functions defined in the library throughout our project as APIs and not just in the context of a SQL call.

AlaSQL is extensible and allows us to create custom functions. For example, we created a custom function called SEARCH_JSON that allows HarperDB users to search and transform nested documents. This function wraps a popular npm package called JSONata. With AlaSQL we were able to embed another open-source package as a simple function call. Our implementation was as easy as defining the function (Note this is sample code):

Java

1

const jsonata = require('jsonata');

2

/**

3

 * wrapper function that implements the JSONata library, which performs searches, transforms, etc... on JSON

4

 * @param {String} jsonata_expression - the JSONata expression to execute

5

 * @param {any} data - data which will be evaluated

6

 * @returns {any}

7

*/

8

function searchJSON(jsonata_expression, data){

9

    if(typeof jsonata_expression !== 'string' || jsonata_expression.length === 0){

10

        throw new Error('search json expression must be a non-empty string');

11

12
13

    let alias = '__' + jsonata_expression + '__';

14
15

    if(hdb_utils.isEmpty(this.__ala__.res)){

16

        this.__ala__.res = {};

17

18
19

    if(hdb_utils.isEmpty(this.__ala__.res[alias])) {

20

        let expression = jsonata(jsonata_expression);

21

        this.__ala__.res[alias] = expression;

22

23

    return this.__ala__.res[alias].evaluate(data);

24

25

//Then define a custom function in AlaSQL:

26

const alasql = require('alasql');

27

alasql.fn.search_json = alasql.fn.SEARCH_JSON = searchJSON;

We are happy with our decision to use AlaSQL to interpret SQL into our data model and run performant queries against as much SQL as possible. That’s why we hosted the creators of AlaSQL on June 16th for a showcase to get an inside look at how AlaSQL was created, how it grew in popularity, and real-world use cases and products. It was awesome to hear from AlaSQL creators Mathias Rangel Wulff and Andrey Gershun. After Q&A on AlaSQL, our CTO Kyle Bernhardy shared more about using AlaSQL as HarperDB’s engine to interpret and parse complex SQL into our data model and perform simple to complex SQL CRUD operations, as well as exposing other libraries like Turf.js, JSONata and more.

Looking for more resources on this innovative client-side in-memory SQL database? Check out this JavaScript library designed for:

Fast in-memory SQL data processing for BI and ERP applications on fat clients
Easy ETL and options for persistence by data import/manipulation/export of several formats
All major browsers, Node.js, and mobile applications

#database #sql #databases #sql (structured query language) #database applications #code activation #sql aggregation

Brain Crist

1600347600

SCHEMAS in SQL Server -MS SQL Server – Zero to Hero Query Master

Introduction

This is part 3 of “MS SQL Server- Zero to Hero” and in this article, we will be discussing about the SCHEMAS in SQL SERVER. Before getting into this article, please consider to visit previous articles in this series from below,

A glimpse of previous articles

Part 1

In part one, we learned the basics of data, database, database management system, and types of DBMS and SQL.

Part 2

We learned to create a database and maintain it using SQL statements.
Best practice methods were also mentioned.

#sql server #benefits of schemas #create schema in sql #database schemas #how to create schema in sql server #schemas #schemas in sql server #sql server schemas #what is schema in sql server

Create a Marvel Database with SQL and Javascript, the easy way

The pipeline

1. Extracting data from Wikipedia

2. Creating an SQL Database in Javascript

3. Final code and usage

﻿Learn More

What is GEEK

Buddha Community

Introduction to Structured Query Language SQL pdf

List of Available Database for Current User In SQL Server

Introduction

Get the list of database

Conclusion

System Databases in SQL Server

Introduction

System Database

Master

Model

MSDB

TempDB

Resource

AlaSQL in Action: The JavaScript SQL Database

What’s AlaSQL?

We Love AlaSQL, too!

SCHEMAS in SQL Server -MS SQL Server – Zero to Hero Query Master

Introduction

A glimpse of previous articles

Part 1

Part 2

Learn More