LevelDB.jl: Julia interface to Google's LevelDB Key Value Database

LevelDB 

LevelDB is Google's open source on-disk key-value storage library that provides an ordered mapping from string keys to binary values. In many applications where only key based accesses are needed, it tends to be a faster alternative than databases. LevelDB was written in C++ with a C calling API included. This module provides a Julia interface to LevelDB using Julia's ccall mechanism.

Install LevelDB

You can build LevelDB from its source code at https://github.com/google/leveldb. Please install the final dynamic library into a system directory such as /usr/lib or make sure libleveldb.so is in one of your LD_LIBRARY_PATH directories. If libleveldb.so is not installed, Julia will try to download and build it automatically.

Run Testing Code

(v1.1) pkg> test LevelDB

This will exercise batched and non-batched writes and reads for string and float array values.

Create/Open/Close a LevelDB database

julia> db = LevelDB.DB(file_path; create_if_missing = false, error_if_exists = false)

Here file_path is the full path to a directory that hosts a LevelDB database. create_if_missing is a boolean flag when true the database will be created if it does not exist. error_if_exists is a boolean flag when true an error will be thrown if the database already exists. The return value is a database object for passing to read/write calls.

julia> close(db)

Close a database, db is the object returned from a LevelDB.DB call. A directory can only be opened by a single LevelDB.DB at a time.

Read and Write Operations

julia> db[key] = value

key and value are Array{UInt8}.

julia> db[key]

Return value is an Array{UInt8}, one can use the reinterpret function to cast it into the right array type (see test code).

julia> delete!(db, key)

Delete a key from db.

Batched Write

LevelDB supports grouping a number of put operations into a write batch, the batch will either succeed as a whole or fail altogether, behaving like an atomic update.

julia> db[keys] = values

keys and values must behave like iterators returning Array{UInt8}. Creates a write batch internally which is then commited to db.

Iterate

julia> for (key, value) in db
           #do something with the key value pair
       end

Iterate over all key => value pairs in a LevelDB.DB.

julia> for (key, value) in LevelDB.RangeView(db, key1, key2)
           #do something with the key value pair
       end

Iterate over a range between key1 and key2 (inclusive)

Authors

  • Jerry Zhenlei Cai ( jpenguin at gmail dot com )
  • Guido Kraemer

additional contributions by

  • @huwenshuo
  • @tmlbl

Download Details:

Author: jerryzhenleicai
Source Code: https://github.com/jerryzhenleicai/LevelDB.jl 
License: View license

#julia #leveldb #interface #database 

LevelDB.jl: Julia interface to Google's LevelDB Key Value Database
Reid  Rohan

Reid Rohan

1658174820

Http Request Handlers for Building json Web Services on top Of Leveldb

PlumbDB

HTTP request handlers for building web services on top of LevelDB

npm install plumbdb

To see a web server written using this check out Plummet

Author: Maxogden
Source Code: https://github.com/maxogden/plumbdb 
License: MIT license

#javascript #node #leveldb 

Http Request Handlers for Building json Web Services on top Of Leveldb
Reid  Rohan

Reid Rohan

1658163552

Level-map-index: Create indices for Leveldb

level-map-index

Another indexing solution for leveldb. Adapted from map-reduce. The API is unstable because this module is a by-product of refactoring an ugly ORM.

examples

// create your database
var sublevel = require('level-sublevel')
  , level = require('levelup')
  , db = sublevel(level('./data'), {valueEncoding: 'json'})

// install
require('level-map-index')(db)

// create indices
db.index('title')
db.index(['author.name', 'price'])

// a custom index (should return an array)
db.index('custom', function map(key, value){
  return [value.title.indexOf('f')]
})

var author = { name: 'bob' }
var book   = { title: 'foo', price: 10, author: author }

db.put('foo', book, search)

function search() {
  // streams will wait for indexing to complete

  // every book by bob, ordered by price
  db.streamBy(['author.name', 'price'], 'bob').pipe(somewhere)

  // every book by bob with a price of 10
  db.streamBy(['author.name', 'price'], ['bob', 10]).pipe(somewhere)

  // ordered by title
  db.streamBy('title').pipe(somewhere)

  // use sublevel hooks
  db.index('custom').post(function(op){
    // op.key is the array we returned from our `map` fn above
    if (op.type=='del' && op.key[0]===0) {
      console.log('no more titles starting with "f" in our db')
    }
  })

  // this will (eventually) trigger the above post hook
  db.del('a')
}

api

index (db, [opts || fn])

Install the plugin.

db.index(props, [opts || fn])

Create an index.

db.streamBy(props, [range, opts])

Likely to change, no docs yet.

db.by(props, [range, opts], cb)

db.getBy(props, [range, opts], cb)

db.hasIndex(props)

index.start()

Manually trigger a rebuild. A complete event fires when it's done.

install

With npm do:

npm install level-map-index

Author: vweevers
Source Code: https://github.com/vweevers/level-map-index 
License: View license

#javascript #node #map #leveldb 

Level-map-index: Create indices for Leveldb
Reid  Rohan

Reid Rohan

1658156064

Leveldown-prebuilt: Pure C++ Node.js LevelDB Binding Serving

UPDATE

leveldown (>= 1.2.2) now supports prebuilds so you shouldn't have to use this fork anymore

This is an experimental fork of leveldown using prebuild binaries to avoid the compile step when installing from npm. To use this fork do

npm install leveldown-prebuilt

Or to install the hyper fork

npm install leveldown-hyper-prebuilt

For a list of supported prebuilt platform binaries check out https://github.com/mafintosh/node-leveldown/releases

LevelDOWN

A Low-level Node.js LevelDB binding

LevelDOWN was extracted from LevelUP and now serves as a stand-alone binding for LevelDB.

It is strongly recommended that you use LevelUP in preference to LevelDOWN unless you have measurable performance reasons to do so. LevelUP is optimised for usability and safety. Although we are working to improve the safety of the LevelDOWN interface it is still easy to crash your Node process if you don't do things in just the right way.

See the section on safety below for details of known unsafe operations with LevelDOWN.

Tested & supported platforms


  • Linux (including ARM platforms such as Raspberry Pi and Kindle!)
  • Mac OS
  • Solaris (SmartOS & Nodejitsu)
  • FreeBSD
  • Windows
    • Node 0.10 and above only, see issue #5 for more info
    • See installation instructions for node-gyp dependencies here, you'll need these (free) components from Microsoft to compile and run any native Node add-on in Windows.

API


leveldown(location)

leveldown() returns a new LevelDOWN instance. location is a String pointing to the LevelDB location to be opened.


leveldown#open([options, ]callback)

open() is an instance method on an existing database object.

The callback function will be called with no arguments when the database has been successfully opened, or with a single error argument if the open operation failed for any reason.

options

The optional options argument may contain:

'createIfMissing' (boolean, default: true): If true, will initialise an empty database at the specified location if one doesn't already exist. If false and a database doesn't exist you will receive an error in your open() callback and your database won't open.

'errorIfExists' (boolean, default: false): If true, you will receive an error in your open() callback if the database exists at the specified location.

'compression' (boolean, default: true): If true, all compressible data will be run through the Snappy compression algorithm before being stored. Snappy is very fast and shouldn't gain much speed by disabling so leave this on unless you have good reason to turn it off.

'cacheSize' (number, default: 8 * 1024 * 1024 = 8MB): The size (in bytes) of the in-memory LRU cache with frequently used uncompressed block contents.

Advanced options

The following options are for advanced performance tuning. Modify them only if you can prove actual benefit for your particular application.

  • 'writeBufferSize' (number, default: 4 * 1024 * 1024 = 4MB): The maximum size (in bytes) of the log (in memory and stored in the .log file on disk). Beyond this size, LevelDB will convert the log data to the first level of sorted table files. From the LevelDB documentation:

Larger values increase performance, especially during bulk loads. Up to two write buffers may be held in memory at the same time, so you may wish to adjust this parameter to control memory usage. Also, a larger write buffer will result in a longer recovery time the next time the database is opened.

'blockSize' (number, default 4096 = 4K): The approximate size of the blocks that make up the table files. The size related to uncompressed data (hence "approximate"). Blocks are indexed in the table file and entry-lookups involve reading an entire block and parsing to discover the required entry.

'maxOpenFiles' (number, default: 1000): The maximum number of files that LevelDB is allowed to have open at a time. If your data store is likely to have a large working set, you may increase this value to prevent file descriptor churn. To calculate the number of files required for your working set, divide your total data by 2MB, as each table file is a maximum of 2MB.

'blockRestartInterval' (number, default: 16): The number of entries before restarting the "delta encoding" of keys within blocks. Each "restart" point stores the full key for the entry, between restarts, the common prefix of the keys for those entries is omitted. Restarts are similar to the concept of keyframs in video encoding and are used to minimise the amount of space required to store keys. This is particularly helpful when using deep namespacing / prefixing in your keys.


leveldown#close(callback)

close() is an instance method on an existing database object. The underlying LevelDB database will be closed and the callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.


leveldown#put(key, value[, options], callback)

put() is an instance method on an existing database object, used to store new entries, or overwrite existing entries in the LevelDB store.

The key and value objects may either be Strings or Node.js Buffer objects. Other object types are converted to JavaScript Strings with the toString() method. Keys may not be null or undefined and objects converted with toString() should not result in an empty-string. Values of null, undefined, '', [] and new Buffer(0) (and any object resulting in a toString() of one of these) will be stored as a zero-length character array and will therefore be retrieved as either '' or new Buffer(0) depending on the type requested.

A richer set of data-types are catered for in LevelUP.

options

The only property currently available on the options object is 'sync' (boolean, default: false). If you provide a 'sync' value of true in your options object, LevelDB will perform a synchronous write of the data; although the operation will be asynchronous as far as Node is concerned. Normally, LevelDB passes the data to the operating system for writing and returns immediately, however a synchronous write will use fsync() or equivalent so your callback won't be triggered until the data is actually on disk. Synchronous filesystem writes are significantly slower than asynchronous writes but if you want to be absolutely sure that the data is flushed then you can use 'sync': true.

The callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.


leveldown#get(key[, options], callback)

get() is an instance method on an existing database object, used to fetch individual entries from the LevelDB store.

The key object may either be a String or a Node.js Buffer object and cannot be undefined or null. Other object types are converted to JavaScript Strings with the toString() method and the resulting String may not be a zero-length. A richer set of data-types are catered for in LevelUP.

Values fetched via get() that are stored as zero-length character arrays (null, undefined, '', [], new Buffer(0)) will return as empty-String ('') or new Buffer(0) when fetched with asBuffer: true (see below).

options

The optional options object may contain:

'fillCache' (boolean, default: true): LevelDB will by default fill the in-memory LRU Cache with data from a call to get. Disabling this is done by setting fillCache to false.

'asBuffer' (boolean, default: true): Used to determine whether to return the value of the entry as a String or a Node.js Buffer object. Note that converting from a Buffer to a String incurs a cost so if you need a String (and the value can legitimately become a UFT8 string) then you should fetch it as one with asBuffer: true and you'll avoid this conversion cost.

The callback function will be called with a single error if the operation failed for any reason. If successful the first argument will be null and the second argument will be the value as a String or Buffer depending on the asBuffer option.


leveldown#del(key[, options], callback)

del() is an instance method on an existing database object, used to delete entries from the LevelDB store.

The key object may either be a String or a Node.js Buffer object and cannot be undefined or null. Other object types are converted to JavaScript Strings with the toString() method and the resulting String may not be a zero-length. A richer set of data-types are catered for in LevelUP.

options

The only property currently available on the options object is 'sync' (boolean, default: false). See leveldown#put() for details about this option.

The callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.


leveldown#batch(operations[, options], callback)

batch() is an instance method on an existing database object. Used for very fast bulk-write operations (both put and delete). The operations argument should be an Array containing a list of operations to be executed sequentially, although as a whole they are performed as an atomic operation inside LevelDB. Each operation is contained in an object having the following properties: type, key, value, where the type is either 'put' or 'del'. In the case of 'del' the 'value' property is ignored. Any entries with a 'key' of null or undefined will cause an error to be returned on the callback. Any entries where the type is 'put' that have a 'value' of undefined, null, [], '' or new Buffer(0) will be stored as a zero-length character array and therefore be fetched during reads as either '' or new Buffer(0) depending on how they are requested.

See LevelUP for full documentation on how this works in practice.

options

The only property currently available on the options object is 'sync' (boolean, default: false). See leveldown#put() for details about this option.

The callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.


leveldown#approximateSize(start, end, callback)

approximateSize() is an instance method on an existing database object. Used to get the approximate number of bytes of file system space used by the range [start..end). The result may not include recently written data.

The start and end parameters may be either String or Node.js Buffer objects representing keys in the LevelDB store.

The callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.


leveldown#getProperty(property)

getProperty can be used to get internal details from LevelDB. When issued with a valid property string, a readable string will be returned (this method is synchronous).

Currently, the only valid properties are:

'leveldb.num-files-at-levelN': return the number of files at level N, where N is an integer representing a valid level (e.g. "0").

'leveldb.stats': returns a multi-line string describing statistics about LevelDB's internal operation.

'leveldb.sstables': returns a multi-line string describing all of the sstables that make up contents of the current database.


leveldown#iterator([options])

iterator() is an instance method on an existing database object. It returns a new Iterator instance.

options

The optional options object may contain:

'gt' (greater than), 'gte' (greater than or equal) define the lower bound of the values to be fetched and will determine the starting point where 'reverse' is not true. Only records where the key is greater than (or equal to) this option will be included in the range. When 'reverse' is 'true` the order will be reversed, but the records returned will be the same.

'lt' (less than), 'lte' (less than or equal) define the higher bound of the range to be fetched and will determine the starting poitn where 'reverse' is not true. Only key / value pairs where the key is less than (or equal to) this option will be included in the range. When 'reverse' is true the order will be reversed, but the records returned will be the same.

'start', 'end' legacy ranges - instead use 'gte', 'lte'

'reverse' (boolean, default: false): a boolean, set to true if you want the stream to go in reverse order. Beware that due to the way LevelDB works, a reverse seek will be slower than a forward seek.

'keys' (boolean, default: true): whether the callback to the next() method should receive a non-null key. There is a small efficiency gain if you ultimately don't care what the keys are as they don't need to be converted and copied into JavaScript.

'values' (boolean, default: true): whether the callback to the next() method should receive a non-null value. There is a small efficiency gain if you ultimately don't care what the values are as they don't need to be converted and copied into JavaScript.

'limit' (number, default: -1): limit the number of results collected by this iterator. This number represents a maximum number of results and may not be reached if you get to the end of the store or your 'end' value first. A value of -1 means there is no limit.

'fillCache' (boolean, default: false): wheather LevelDB's LRU-cache should be filled with data read.

'keyAsBuffer' (boolean, default: true): Used to determine whether to return the key of each entry as a String or a Node.js Buffer object. Note that converting from a Buffer to a String incurs a cost so if you need a String (and the value can legitimately become a UFT8 string) then you should fetch it as one.

'valueAsBuffer' (boolean, default: true): Used to determine whether to return the value of each entry as a String or a Node.js Buffer object.


iterator#next(callback)

next() is an instance method on an existing iterator object, used to increment the underlying LevelDB iterator and return the entry at that location.

the callback function will be called with no arguments in any of the following situations:

  • the iterator comes to the end of the store
  • the end key has been reached; or
  • the limit has been reached

Otherwise, the callback function will be called with the following 3 arguments:

  • error - any error that occurs while incrementing the iterator.
  • key - either a String or a Node.js Buffer object depending on the keyAsBuffer argument when the iterator() was called.
  • value - either a String or a Node.js Buffer object depending on the valueAsBuffer argument when the iterator() was called.

iterator#end(callback)

end() is an instance method on an existing iterator object. The underlying LevelDB iterator will be deleted and the callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.


leveldown.destroy(location, callback)

destroy() is used to completely remove an existing LevelDB database directory. You can use this function in place of a full directory rm if you want to be sure to only remove LevelDB-related files. If the directory only contains LevelDB files, the directory itself will be removed as well. If there are additional, non-LevelDB files in the directory, those files, and the directory, will be left alone.

The callback will be called when the destroy operation is complete, with a possible error argument.

leveldown.repair(location, callback)

repair() can be used to attempt a restoration of a damaged LevelDB store. From the LevelDB documentation:

If a DB cannot be opened, you may attempt to call this method to resurrect as much of the contents of the database as possible. Some data may be lost, so be careful when calling this function on a database that contains important information.

You will find information on the repair operation in the LOG file inside the store directory.

A repair() can also be used to perform a compaction of the LevelDB log into table files.

The callback will be called when the repair operation is complete, with a possible error argument.

Safety


Database state

Currently LevelDOWN does not track the state of the underlying LevelDB instance. This means that calling open() on an already open database may result in an error. Likewise, calling any other operation on a non-open database may result in an error.

LevelUP currently tracks and manages state and will prevent out-of-state operations from being send to LevelDOWN. If you use LevelDOWN directly then you must track and manage state for yourself.

Snapshots


LevelDOWN exposes a feature of LevelDB called snapshots. This means that when you do e.g. createReadStream and createWriteStream at the same time, any data modified by the write stream will not affect data emitted from the read stream. In other words, a LevelDB Snapshot captures the latest state at the time the snapshot was created, enabling the snapshot to iterate or read the data without seeing any subsequent writes. Any read not performed on a snapshot will implicitly use the latest state.

Getting support


There are multiple ways you can find help in using LevelDB in Node.js:

  • IRC: you'll find an active group of LevelUP users in the ##leveldb channel on Freenode, including most of the contributors to this project.
  • Mailing list: there is an active Node.js LevelDB Google Group.
  • GitHub: you're welcome to open an issue here on this GitHub repository if you have a question.

Contributing


LevelDOWN is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the CONTRIBUTING.md file for more details.

Contributors

LevelDOWN is only possible due to the excellent work of the following contributors:

Rod VaggGitHub/rvaggTwitter/@rvagg
John ChesleyGitHub/cheslesTwitter/@chesles
Jake VerbatenGitHub/raynosTwitter/@raynos2
Dominic TarrGitHub/dominictarrTwitter/@dominictarr
Max OgdenGitHub/maxogdenTwitter/@maxogden
Lars-Magnus SkogGitHub/ralphtheninjaTwitter/@ralphtheninja
David BjörklundGitHub/keslaTwitter/@david_bjorklund
Julian GruberGitHub/juliangruberTwitter/@juliangruber
Paolo FragomeniGitHub/hij1nxTwitter/@hij1nx
Anton WhalleyGitHub/No9Twitter/@antonwhalley
Matteo CollinaGitHub/mcollinaTwitter/@matteocollina
Pedro TeixeiraGitHub/pgteTwitter/@pgte
James HallidayGitHub/substackTwitter/@substack

Windows

A large portion of the Windows support comes from code by Krzysztof Kowalczyk @kjk, see his Windows LevelDB port here. If you're using LevelUP on Windows, you should give him your thanks!


Author: mafintosh
Source Code: https://github.com/mafintosh/leveldown-prebuilt 
License: MIT license

#javascript #cpluplus #node #leveldb 

Leveldown-prebuilt: Pure C++ Node.js LevelDB Binding Serving
Reid  Rohan

Reid Rohan

1658140800

Level-search: index Every Property In Leveldb

level-search

index every property in leveldb

example

First, install the indexer, and then fill your leveldb with data.

var level  = require('level')
var sub    = require('level-sublevel')
var search = require('level-search')

var db = sub(level(pathToLevelDir, {valueEncoding: 'json'}))
var index = search(db, 'search') //by default the sublevel has the same name as the module.

//then put loads of JSONdata into the database...
streamOfJSON //with {key:..., value:...}
.pipe(db.createWriteStream())

//then query the database like this!

//retrive all the modules someone wrote...
index.createSearchStream(['maintainers', true, 'name', username])
  .on('data', console.log)

you can also request ranges!

index.createSearchStream(['date', {min: new Date('2009-0-0'), max: Date.now()}])
  .on('data', console.log)

methods

var search = require('level-search')

var index = search(db, indexName)

Create a new search index inside db the sublevel string name indexName.

index.createSearchStream(keys, opts)

Return a readable stream from an array search path keys. opts are passed to the underlying db.createReadStream() call.

keys works like JSONStream.parse() where each item describes a key more deeply nested inside the document. At leaf nodes, equality or regex test is used. At non-leaf nodes, keys are traversed or searched for as the first matching regex key name.

Each item in keys can be a string, boolean, or regex. The boolean true means always match and false means never match.

rebuilding the index

If want to add search to old data, or have been messing with stuff, you'll need to rebuild indexes.

index.rebuild(function (err) {
  //the search index is ready to be used again
})

If you insert keys while the rebuild, strange things could happen, so if you do that, you need to rebuild again.

Author: Dominictarr
Source Code: https://github.com/dominictarr/level-search 
License: MIT license

#javascript #search #leveldb #node 

Level-search: index Every Property In Leveldb
Reid  Rohan

Reid Rohan

1658133265

Level-prebuilt: Fast, Simple Storage - A Node.js-style LevelDB Wrapper

level-prebuilt

This is the level module but using leveldown-prebuilt to avoid leveldb compilation at install time

Fast & simple storage - a Node.js-style LevelDB wrapper

This is a convenience package that bundles the current release of LevelUP and LevelDOWN and exposes LevelUP on its export.

Use this package to avoid having to explicitly install LevelDOWN when you just want plain old LevelDB from LevelUP.

var level = require('level')

// 1) Create our database, supply location and options.
//    This will create or open the underlying LevelDB store.
var db = level('./mydb')

// 2) put a key & value
db.put('name', 'Level', function (err) {
  if (err) return console.log('Ooops!', err) // some kind of I/O error

  // 3) fetch by key
  db.get('name', function (err, value) {
    if (err) return console.log('Ooops!', err) // likely the key was not found

    // ta da!
    console.log('name=' + value)
  })
})

See LevelUP and LevelDOWN for more details.

Contributing


Level is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the CONTRIBUTING.md file for more details.

Contributors

Level, including LevelUP & LevelDOWN, is only possible due to the excellent work of the following contributors:

Rod VaggGitHub/rvaggTwitter/@rvagg
John ChesleyGitHub/cheslesTwitter/@chesles
Jake VerbatenGitHub/raynosTwitter/@raynos2
Dominic TarrGitHub/dominictarrTwitter/@dominictarr
Max OgdenGitHub/maxogdenTwitter/@maxogden
Lars-Magnus SkogGitHub/ralphtheninjaTwitter/@ralphtheninja
David BjörklundGitHub/keslaTwitter/@david_bjorklund
Julian GruberGitHub/juliangruberTwitter/@juliangruber
Paolo FragomeniGitHub/hij1nxTwitter/@hij1nx
Anton WhalleyGitHub/No9Twitter/@antonwhalley
Matteo CollinaGitHub/mcollinaTwitter/@matteocollina
Pedro TeixeiraGitHub/pgteTwitter/@pgte
James HallidayGitHub/substackTwitter/@substack

Author: jhermsmeier
Source Code: https://github.com/jhermsmeier/level-prebuilt 
License: MIT license

#javascript #leveldb #node 

Level-prebuilt: Fast, Simple Storage - A Node.js-style LevelDB Wrapper
Reid  Rohan

Reid Rohan

1658132591

Level-peek: Peek The First Or Last Record in A Leveldb Range

level-peek

peek the first or last record in a leveldb range.

example

var levelup = require('levelup')
var db = levelup(PATH_TO_DB)

var peek = require('level-peek')

//get the first value, 'a' or after
peek.first(db, {start: 'a'}, function (err, key, value) {
  console.log(key, value)
})

//get last value, 'z' or before.
peek.last(db, {end: 'z'}, function (err, key, value) {
  console.log(key, value)
})

see also, level-fix-range and https://github.com/rvagg/node-levelup/issues/110

Author: Dominictarr
Source Code: https://github.com/dominictarr/level-peek 
License: MIT license

#javascript #node #leveldb 

Level-peek: Peek The First Or Last Record in A Leveldb Range
Reid  Rohan

Reid Rohan

1658124962

Leveldb with Bytewise Key Encodings & Bytewise Friendly Sublevels

level-bytewise

leveldb with bytewise key encodings and bytewise friendly sublevels.

Ie. a "batteries included" distribution of levelup that has friendly key sorting and sublevels that work with bytewise encodings.

Installation

This module is installed via npm:

$ npm install level-bytewise

Example Usage (put and get)

var levelBytewise = require('level-bytewise');
var db = levelBytewise('/tmp/path-to-db')
var users = db.sublevel('users');
users.put(['eugene'], { color: 'black' }, function (err) {
  if (err) throw err;
  users.get(['eugene'], function (err, data) {
    if (err) throw err;
    console.log(data);
    // { color: 'black }
  });
});

Example Usage (build and index on color)

var levelBytewise = require('level-bytewise'),
    timestamp = require('monotonic-timestamp');
var db = levelBytewise('/tmp/path-to-db')
var users = db.sublevel('users');
var colors = db.sublevel('colors');
users.pre(function (change, add, batch) {
  // build an index on `color` and store it in the `colors` sublevel
  add({
    key: [change.value.color, timestamp()],
    value: change.key,
    prefix: colors
  });
});
users.put(['eugene'], { color: 'black' }, function (err) {
  if (err) throw err;
  colors.createReadStream()
    .on('data', console.log);
  // { key: ['black', 1461480614762], value: ['eugene'] }
});

Author: Eugeneware
Source Code: https://github.com/eugeneware/level-bytewise 
License: View license

#javascript #key #leveldb #node 

Leveldb with Bytewise Key Encodings & Bytewise Friendly Sublevels
Gordon  Taylor

Gordon Taylor

1658038500

A Very Fast & Persistent Web Server Session Manager Backed By LevelDB

Level Session 

A framework-agnostic, LevelDB-backed session manager for Node.js web servers. Provides very fast session data storage that persists across server restarts.

Compatible with Connect / Express middleware.

Backed by Generic Session, the flexible session manager, Level Session gives you simple and speedy entry-level session management that allows you to scale when ready by switching the storage back-end of Generic Session to a more appropriate solution such as Redis.

A LevelDB store can only be accessed by one process at a time so Level Session is not ideal for multi-process deployments unless you're passing in a multilevel instance as your 'db'.

Level Session uses LevelUP for LevelDB access in Node.js, you must either install levelup or level from npm for Level Session to work; it is not loaded as a dependency.

Example

// server.js
var http           = require('http')
  , levelSession   = require('level-session')('/tmp/level_session_example.db')
  , port           = 8080

http.createServer(function (req, res) {
  levelSession(req, res, function () {
    var m

    res.writeHead(200)

    if (m = req.url.match(/^\/get\/(.+)/)) {
      return req.session.get(m[1], function (err, data) {
        res.end(JSON.stringify(data))
      })
    } else if (m = req.url.match(/^\/set\/([^\/]+)\/(.+)/)) {
      return req.session.set(m[1], m[2], function () {
        res.end(JSON.stringify(m[2]))
      })
    }

    res.end('ERROR')
  })
}).listen(port)
// client.js
var request = require('request')
  , jar     = request.jar()
  , port    = 8080

  , req     = function(url, cb) {
      request({
          url: 'http://localhost:' + port + '/' + url
        , jar: jar
        , json: true
      }, cb)
    }

req('set/foo/bar', function () {
  console.log('Set foo = bar')
  req('get/foo', function (e, res, body) {
    console.log('Fetched foo =', body)
  })
})

Running the two processes, we'll get:

$ node server.js &
$ node client.js
Set foo = bar
Fetched foo = bar

This example is available in the examples/ directory.

API

levelSession(options | location)

Will create a new LevelSession instance, including an open LevelDB instance. You must provide a location for the LevelDB store, either as a String or on an options object with the property 'location'. Alternatively you can provide an existing (open), LevelUP instance with the 'db' property. Level Session can coexist with other uses of the same LevelUP by using the same technique as level-sublevel to operate in a "session" namespace.

The returned object can be used as a stand-alone filter or as a Connect / Express middleware.

Any additional options you provide on an options object will be passed on to Generic Session, these options include:

Options:

  • keys (optional): either an Array of strings constituting your signing secret keys to be passed to a new instance of Keygrip, or you can pass in an instance of **Keygrip directly. If you pass in a keys instance, Keygrip must be installed.
  • cookies (optional): provide an instance of Cookies or a compatible cookie manager to use to manage cookies.
  • expire (optional, default: 2 weeks): number of seconds to set for the session cookie expiry.
  • cookieName (optional, default: 's'): the name of the session cookie.

A LevelSession instance can be used as a filter / middleware in a Node.js server, invoke it as a function with the arguments: HTTP server request, HTTP server response and a next callback function to be called when LevelSession is finished. You will get a session object attached both request and response.

close()

Each LevelSession instance has a close() method that can be used to finalise and close all resources.


levelSession.LevelStore(options | location)

Use this to create a LevelStore instance that can be used directly with Generic Session as the store property. This provides the flexibility to invoke the session manager in the most appropriate way for your application.

Session API

session.get(key[, callback])

Get the object stored as key for the current session in the session store. Automatically updates the expires time for this session.

If the callback is not provided then it will simply perform an expiry time update. If you just provide a callback and no key then it operates as an alias for getAll().


session.getAll(callback)

Get all objects stored for the current session in the session store. Automatically updates the expires time for this session.

If the callback is not provided then it will simply perform an expiry time update.


session.set(key[, value][, callback])

Sets value as key for the current session in the session store. Automatically updates the expires time for this session.

If value is omitted then key will be set to null. callback may be omitted and the operation will still be performed.


session.del(key[, callback])

Deletes the property key from the current session in the session store. Automatically updates the expires time for this session.

If key is omitted then it operates as an alias for delAll() (beware!). callback may be omitted and the operation will still be performed.


session.delAll([callback])

Deletes all data for the current session in the session store.

callback may be omitted and the operation will still be performed.


session.destroy([callback])

Deletes all data for the current session in the session store and will also remove the session cookie from the client.

callback may be omitted and the operation will still be performed.


Author: rvagg
Source Code: https://github.com/rvagg/archived-node-level-session 
License: View license

#javascript #node #leveldb 

A Very Fast & Persistent Web Server Session Manager Backed By LevelDB
Gordon  Taylor

Gordon Taylor

1658032260

Gun-level: LevelDB Storage Plugin for GunDB

gun-level

LevelDB is awesome. It's awesomer with gun.

indexedDB NOW SUPPORTED IN MAIN GUN REPO - THIS REPO NO LONGER MAINTAINED

The main GUN repo now has lib/rindexed that can be used like so, now built on the RAD storage engine for GUN.

If you still need NodeJS LevelDB bindings, consider writing a RAD adapter, or use this project - please contribute and maintain it for others, as it is now deprecated.

Overview

GunDB is a graph database engine with real-time sync and offline-editing. Although gun comes with storage and sync out of the box, it's design is pluggable, so you can still use your favorite storage backend or transport layer by using an adapter.

LevelDB operates on a similar paradigm. It ships with an interface called LevelUP that gives all the methods you'd expect from a storage engine (like get, put, batch, etc.) and forwards those to a lower-level database (it uses LevelDOWN by default).

Arguably the most valuable aspect of level is it's ecosystem. There are tons are plugins, backends, dashboards, and utilities made specifically for level. It's kinda like a database buffet.

Here, check out their list of modules!

If you're feeling stressed, don't worry. LevelDB comes out of the box as a quite capable database, and if you don't want to dabble with plugins and configs, there's no need.

So what happens when you combine Level with GunDB? You get the power and control of level right alongside the ease of gun.

That's what this library does.

Usage

To get started, you'll first need to install gun-level.

If you're unfamiliar with npm, you can get started here

$ npm install --save gun-level gun

Now require them from your node project.

// Import the `Gun` library
const Gun = require('gun')

// Imported for side effects, adds level adapters.
// MUST be required after Gun to work
require('gun-level')

Once they're imported you can create a new Gun database:

const gun = new Gun({
    // We'll put options here in a moment.
})

Sweet, you're set up! However, gun-level won't do anything unless you pass it a levelDB instance through the Gun constructor. For that, you'll need to download the Node levelup package.

There are some differences between levelup v1 and v2 so be sure to note which version you're using.

levelup v1

$ npm install --save levelup leveldown

If you get a build error at this step, replace all examples of leveldown with jsondown.

// Import the two libraries
const levelup = require('levelup')
const leveldown = require('leveldown')

// Create a new level instance which saves
// to the `data/` folder.
const levelDB = levelup('data', {
    db: leveldown,
})

levelup >v2

Note that Levelup v2 only supports Node >6.

$ npm install --save levelup leveldown encoding-down
// Import the required libraries
const levelup = require('levelup')
const encode = require('encoding-down')
const leveldown = require('leveldown')

// Create a new level instance which saves
// to the `data/` folder.
const levelDB = levelup(
    encode(
        leveldown('data'),
        { valueEncoding: 'json' }
    )
)

Instantiating Gun with Level

Now we can pass our new levelDB instance to the Gun constructor.

const Gun = require('gun')
require('gun-level')

// ... instantiate LevelDB per above

// Pass LevelDB instance into Gun
const gun = new Gun({
    level: levelDB
})

Done! Now your gun database will store it's graph in your Level DB!

Let's try a few things...

const bob = gun.get('bob').put({ name: 'Bob' })
const dave = gun.get('dave').put({ name: 'Dave' })

// Write a fun circular reference.
bob.get('friend').put(dave)
dave.get('friend').put(bob)

// Print the data!
bob.get('friend').val(friend => {
    console.log(friend.name) // Dave
});

// Now with a circular reference
bob.get('friend').get('friend').val(friend => {
    console.log(friend.name) // Bob
});

That's pretty much all there is to the gun-level API. If you're unfamiliar with gun's API, here's a good reference.

Advanced Levelry

You've seen the basics, but it's not enough. You crave more power.

To exchange backends with level, like Riak, Mongo, IndexedDB, etc., you can find the official list of storage backends here. Usually it's just a matter of passing the module as the db option to levelup.

Here's an example with MongoDB using mongodown:

const levelup = require('levelup')
const mongoDown = require('mongodown')

// Initialize Level
const levelDB = levelup('localhost:27017/YOUR_COLLECTION_NAME', {
    db: mongoDown,
})

// Initialize Gun
const gun = new Gun({
    level: levelDB,
    file: false,
})

Here's another example with IndexedDB using level-js:

const levelup = require('levelup')
const leveldown = require('level-js')
const encode = require('encoding-down')

// Initialize Level
const levelDB = levelup(encode(leveldown('my-big-db'), { valueEncoding: 'json' }))

// Initialize Gun
const gun = Gun({
    level: levelDB,
    localStorage: false,
})

Even if you're content with the default levelDB setup, I really recommend you scan this list of plugins. It's inspiring what the community has built.

gun-level will try to read and write values as json. If you're having trouble getting a plugin to work, or keep seeing "[object Object]", make sure it's using the json value encoding.

Getting Support

If you're running into problems, feel free to either post an issue on GitHub or chat with us humans in the Gitter channel.

Installing from Source

Clone the repo and install the dependencies:

$ git clone https://github.com/PsychoLlama/gun-level.git
$ cd gun-level
$ npm install

Running Tests

# In directory `gun-level`
$ npm test

Building

$ npm run build
# Compiles to folder `dist/`

Contributors

  • Project owner @PsychoLlama
  • Code contributor @swhgoon
  • The friendly @greenkeeperio-bot

Sponsored by the fabulous people at GunDB.

Author: gundb
Source Code: https://github.com/gundb/gun-level 
License: 

#javascript #leveldb #node 

Gun-level: LevelDB Storage Plugin for GunDB
Gordon  Taylor

Gordon Taylor

1658017260

Level-scuttlebutt: Leveldb Persistence for Scuttlebutts

level-scuttlebutt

Plugin to add persistence and querying scuttlebutt documents stored in leveldb.

Instead of representing an object as a single document, scuttlebutt represents a document as a series of immutable transactions. The 'document' is modified by appending a new transaction. Old transactions that are no longer relevant can be cleaned up, but you can never modify a transaction in place. As it turns out, leveldb (a log-structured merge tree) is optimized for exactly this sort of data.

Must be used with level-sublevel

Example

var levelup = require("levelup")
var level_scuttlebutt = require("level-scuttlebutt")
var SubLevel = require('level-sublevel')


//create a leveldb instance...
//levelup must be extended with SubLevel!
var db = SubLevel(levelup(DB_FILE))



//a scuttlebutt model.
var Model = require('scuttlebutt/model')

//level-scuttlebutt needs to have an unique identifier of the current instance
var udid = require('udid')('app-name')


//patch it with level-scuttlebutt.
var sbDb = db.sublevel('scuttlebutt') //add a scuttlebutt 'table'

//k
level_scuttlebutt(sbDb, udid, function (name) {
  //create a scuttlebutt instance given a name.
  //the key will match the start of the name.
  return new Model()
  //now is a good time to customize the scuttlebutt instance.
})

//open a scuttlebutt instance by name.
sbDb.open(name, function (err, model) {
  model.on('change:key', console.log) //...
  model.set('key', value)
  
  // when you're done get rid of it
  model.dispose()
})

//the toJSON values are stored in the db,
//so you can just use any other map reduce library on it!
sbDb.views['all'] =
  mapReduce(sbDb, 'all', 
    function (key, json, emit) { 
      return emit(key.split('!'), 1)
    },
    function (acc, item) {
      return '' + (Number(acc) + Number(item))
    },
    '0'
  )

Initialization

Add level-scuttlebutt plugin to the db object var level_scuttlebutt = require('level-scuttlebutt'); level_scuttlebutt(db, ID, schema)

ID is a unique string that identifies the node (the machine) and should be tied to the leveldb instance. I suggest using udid.

schema should be a function that takes a string (the name of the scuttlebutt instance) and returns and empty scuttlebutt instance. You can use scuttlebutt-schema.

Queries

Use some other level-* plugin for queries!

map-reduce, level-map-merge

Example

get the 10 last edited documents!

sbDb.views['latest10']
  = 
  MapReduce(sdb, 'latest10',
  function (key, json) {
    var name = key
    var obj = JSON.parse(json)
    //emit 0-many group-value pairs.
    //value must be a string or a buffer.
    this.emit([], JSON.stringify({name: name, time: Date.now(), length: obj.text.length}))
  },
  //merge the latest value into the accumulator.
  function (acc, value) {
    var all = JSON.parse(acc).concat(JSON.parse(value))
    //sort by time, decending.
    all.sort(function (a, b) {
      return b.time - a.time
    })
    //top ten most recent
    var all = all.slice(0, 10)
    return JSON.stringify(all)
  },
  //the first value for the accumulator.
  //since we are parsing it, it needs to be valid JSON.
  '[]'
})

Author: Dominictarr
Source Code: https://github.com/dominictarr/level-scuttlebutt 
License: MIT license

#javascript #leveldb #node 

Level-scuttlebutt: Leveldb Persistence for Scuttlebutts
Gordon  Taylor

Gordon Taylor

1658009640

Level-object: Store Objects in Leveldb

level-object

Store objects in leveldb.

It's recommended to use level-pathwise instead since it's a lot more flexible and the api isn't more complicated. store.set('userid', 'name', 'fritz') becomes store.put(['userid', 'name'], 'fritz').

Rationale

The idea is that storing JSON in leveldb sucks, because it's hard to update without risiking inconsistencies.

Say you have {foo:'bar', bar:'baz'} stored in a leveldb and want to add a field:

db.get('obj', function (err, obj) {
  obj.aNew = 'field';
  db.set('obj', obj);
});

Now at the same time some other code wants to e.g. modify the value of one field:

db.get('obj', function (err, obj) {
  obj.foo = 'bar, but changed';
  db.set('obj', obj);
});

And you will either get

{
  "foo": "bar",
  "bar": "baz",
  "aNew": "field"
}

or

{
  "foo": "bar, but changed",
  "bar": "baz"
}

but not

{
  "foo": "bar, but changed",
  "bar": "baz",
  "aNew": "field"
}

There needs to be a mechanism that merges changes together, that still allows you to retrieve the whole object.

Usage

var levelObj = require('level-object');
var level = require('level');
var db = level(__dirname + '/db');

var store = levelObj(db);
store.set('name', 'key', 'value', function (err) {
  store.toJSON('name', function (err, json) {
    console.log(json);
    // => { "key": "value" }
  });
});

API

Obj(db)

Create a new store on the passed db object.

Obj#get(name, key, cb)

Call cb with (err, value).

Obj#set(name, key, value[, cb])

Set name's key to value and call cb, possibly with an error object.

Obj#del(name, key[, cb])

Delete name's key and call cb, possibly with an error object.

Obj#patch(ops, cb)

Modify multiple values and objects at once. ops is an array containing objects.

To write / modify a value, add { type: 'set', object: name, key: key, value: value }.

To delete a value, add { type: 'del', object: name, key: key }.

Obj#keys(name, cb)

Call cb with err and an array of all set keys, like Object.keys.

Obj#toJSON(name, cb)

Call cb with err and a JavaScript representation of name.

Installation

With npm do

$ npm install level-object

Author: juliangruber
Source Code: https://github.com/juliangruber/level-object 
License: 

#javascript #object #node #leveldb 

Level-object: Store Objects in Leveldb
Lawrence  Lesch

Lawrence Lesch

1658004900

Prerender-level-cache: Prerender Plugin for Level Caching

prerender-level-cache

Prerender plugin for level caching, to be used with the prerender node application

How it works

This plugin will store all prerendered pages into LevelDB.

How to use

In your local prerender project run:

$ npm install prerender-level-cache

Then in the server.js that initializes the prerender:

server.use(require('prerender-level-cache'));

Features

Ttl

It uses a default ttl of 2 months, but that can be overridden by setting a different value in prerender server.options.ttl.

Refresh

You can force a refresh from another plugin by setting req.refresh = true

Exemple:

// in another plugin placed before prerender-level-cache
module.exports = {
  requestReceived: (req, res, next) => {
    req.refresh = true
    next()
  }
}

Alternatively, you can delete the cache directly in the LevelDB database, using a tool such as lev2.

Level-party

The plugin uses the level-party module to get over the level multi-process restrictions.

Author: Maxlath
Source Code: https://github.com/maxlath/prerender-level-cache 
License: MIT license

#javascript #cache #node #leveldb 

Prerender-level-cache: Prerender Plugin for Level Caching
Lawrence  Lesch

Lawrence Lesch

1657967040

Level-todo: Todo List in Leveldb

level-todo

A todo list using leveldb

example

To add a new list

level-todo add "this is a name" "this is a value"

or

level-todo add read https://www.npmjs.com

Install

npm install -g level-todo

Usage

usage:

  level-todo add name value
  Add a new list.

  level-todo get name
  Get a list

  level-todo rm/done name
  Mark a list as done and remove it by the given name.

  level-todo list
  Show all the todo list

  level-todo destroy
  Clear all the list.

Author: Nickleefly
Source Code: https://github.com/nickleefly/level-todo 
License: MIT license

#javascript #leveldb #todo #node 

Level-todo: Todo List in Leveldb
Lawrence  Lesch

Lawrence Lesch

1657902780

Electron-demo: Demo App Loading LevelDB into an Electron Context

electron-demo

Demo app loading LevelDB into an Electron context.   

Get started

git clone https://github.com/Level/electron-demo.git
cd electron-demo
npm install
npm start

You're now ready to use LevelDB. Try running await db.put('key', 2) in the devtools console of the Electron window, followed by await db.get('key')!

Architecture

The main process (main.js) opens a LevelDB database using level and exposes it to renderer processes (renderer.js) using many-level. The processes communicate using Electron IPC. This approach is a secure default that allows renderer processes to be sandboxed. As a consequence, require() is not available, so we use browserify to bundle the JavaScript into a single file.

Alternatively you can enable the potentially insecure nodeIntegration option and do the following in a renderer process:

const { Level } = require('level')
const db = new Level('./db')

But then only one process can open the database at the same time. Another alternative is to use browser-level in renderer processes, if you want each process to have its own database backed by IndexedDB.

As this is a demo, the implementation here (specifically the IPC) is not optimized and does not handle errors. The demo also does not include scripts to package up the Electron app for production (e.g. using electron-builder) which will require a few additional steps (e.g. asarUnpack in the case of electron-builder) due to the use of Node.js native addons.

Contributing

Level/electron-demo is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Author: Level
Source Code: https://github.com/Level/electron-demo 
License: MIT license

#javascript #electron #node #leveldb 

Electron-demo: Demo App Loading LevelDB into an Electron Context