Level-mongo: Mongo interface to Leveldb

level-mongo    

A basic mongo query interface for leveldb backend, future version will try to allow more advanced querying and partial & full updates

Install

$ npm install level-mongo --save

Usage

Valid options for constructor object exposed by module are shown below

options:

• collections - object with keys being name of collection (sublevel-level) to create
  • key - unique identifier for each json record within collection
• location - string required to define base directory where leveldb is persisted on disk
• config - object with the following keys
  • keyEncoding - string, valid options are one of 'hex', 'utf8', 'ascii', 'binary', 'base64', 'ucs2', 'utf16le'
  • createIfMissing - boolean, default true. Create db if missing
  • errorIfExists - boolean, default false. Error if db already exists
  • compression - boolean, default true. Set to true to use snappy compression end for leveldb files
  • cacheSize - number, default 8 x 1024 x 1024. LRU cache for leveldb

'use strict';

const LevelMongo = require('level-mongo');

const db = new LevelMongo({
   config: {
       keyEncoding: 'ascii',
       compression: false,
       cacheSize: 4 * 1024 * 1024
   },
   collections: {
       users: {
           key: '_id'
       }
   },
   location: './db'
});

db.open((err) => {

   db.collections.users.findOne({ _id: 'abcd'}, (err, doc) => {

       console.log(err, doc);

       db.close((err) => {

           console.log(err);

       });
   });

});

Mongo methods partially implemented on each collection object

findOne - find record by key

find - return all records within collection

insertOne - insert record, unique key must be embedded within object being inserted

insertMany - batch insert operation

deleteOne - find and delete one record by key

deleteMany - deletes all records within collection

updateOne - updates one record by merging modifier object into original document

count - return number of records in collection

getKeys - returns all keys stored within collection

TODO

• improve indexing - in memory only? B-Tree?
• add schema parameter and validation option to define data shape
• findOne
• findMany
• insertOne
• insertMany
• updateOne
• updatedMany
• count
• allow querying by document properties as opposed to just be id
• abstract shared logic better
• add update operators
  • $set - partially implemented
  • $unset - partially implemented
  • $inc
  • $mul
  • $rename
  • $min
  • $max
  • $currentDate
• add array operators
• add find projection
  • Return fields
  • Exclude fields

Author: Simon-p-r
Source Code: https://github.com/simon-p-r/level-mongo 
License: 

#javascript #query #leveldb #mongodb 

What is GEEK

Buddha Community

Level-mongo: Mongo interface to Leveldb

level-mongo    

A basic mongo query interface for leveldb backend, future version will try to allow more advanced querying and partial & full updates

Install

$ npm install level-mongo --save

Usage

Valid options for constructor object exposed by module are shown below

options:

• collections - object with keys being name of collection (sublevel-level) to create
  • key - unique identifier for each json record within collection
• location - string required to define base directory where leveldb is persisted on disk
• config - object with the following keys
  • keyEncoding - string, valid options are one of 'hex', 'utf8', 'ascii', 'binary', 'base64', 'ucs2', 'utf16le'
  • createIfMissing - boolean, default true. Create db if missing
  • errorIfExists - boolean, default false. Error if db already exists
  • compression - boolean, default true. Set to true to use snappy compression end for leveldb files
  • cacheSize - number, default 8 x 1024 x 1024. LRU cache for leveldb

'use strict';

const LevelMongo = require('level-mongo');

const db = new LevelMongo({
   config: {
       keyEncoding: 'ascii',
       compression: false,
       cacheSize: 4 * 1024 * 1024
   },
   collections: {
       users: {
           key: '_id'
       }
   },
   location: './db'
});

db.open((err) => {

   db.collections.users.findOne({ _id: 'abcd'}, (err, doc) => {

       console.log(err, doc);

       db.close((err) => {

           console.log(err);

       });
   });

});

Mongo methods partially implemented on each collection object

findOne - find record by key

find - return all records within collection

insertOne - insert record, unique key must be embedded within object being inserted

insertMany - batch insert operation

deleteOne - find and delete one record by key

deleteMany - deletes all records within collection

updateOne - updates one record by merging modifier object into original document

count - return number of records in collection

getKeys - returns all keys stored within collection

TODO

• improve indexing - in memory only? B-Tree?
• add schema parameter and validation option to define data shape
• findOne
• findMany
• insertOne
• insertMany
• updateOne
• updatedMany
• count
• allow querying by document properties as opposed to just be id
• abstract shared logic better
• add update operators
  • $set - partially implemented
  • $unset - partially implemented
  • $inc
  • $mul
  • $rename
  • $min
  • $max
  • $currentDate
• add array operators
• add find projection
  • Return fields
  • Exclude fields

Author: Simon-p-r
Source Code: https://github.com/simon-p-r/level-mongo 
License: 

#javascript #query #leveldb #mongodb 

LevelDB.jl: Julia interface to Google's LevelDB Key Value Database

LevelDB 

LevelDB is Google's open source on-disk key-value storage library that provides an ordered mapping from string keys to binary values. In many applications where only key based accesses are needed, it tends to be a faster alternative than databases. LevelDB was written in C++ with a C calling API included. This module provides a Julia interface to LevelDB using Julia's ccall mechanism.

Install LevelDB

You can build LevelDB from its source code at https://github.com/google/leveldb. Please install the final dynamic library into a system directory such as /usr/lib or make sure libleveldb.so is in one of your LD_LIBRARY_PATH directories. If libleveldb.so is not installed, Julia will try to download and build it automatically.

Run Testing Code

(v1.1) pkg> test LevelDB

This will exercise batched and non-batched writes and reads for string and float array values.

Create/Open/Close a LevelDB database

julia> db = LevelDB.DB(file_path; create_if_missing = false, error_if_exists = false)

Here file_path is the full path to a directory that hosts a LevelDB database. create_if_missing is a boolean flag when true the database will be created if it does not exist. error_if_exists is a boolean flag when true an error will be thrown if the database already exists. The return value is a database object for passing to read/write calls.

julia> close(db)

Close a database, db is the object returned from a LevelDB.DB call. A directory can only be opened by a single LevelDB.DB at a time.

Read and Write Operations

julia> db[key] = value

key and value are Array{UInt8}.

julia> db[key]

Return value is an Array{UInt8}, one can use the reinterpret function to cast it into the right array type (see test code).

julia> delete!(db, key)

Delete a key from db.

Batched Write

LevelDB supports grouping a number of put operations into a write batch, the batch will either succeed as a whole or fail altogether, behaving like an atomic update.

julia> db[keys] = values

keys and values must behave like iterators returning Array{UInt8}. Creates a write batch internally which is then commited to db.

Iterate

julia> for (key, value) in db
           #do something with the key value pair
       end

Iterate over all key => value pairs in a LevelDB.DB.

julia> for (key, value) in LevelDB.RangeView(db, key1, key2)
           #do something with the key value pair
       end

Iterate over a range between key1 and key2 (inclusive)

Authors

• Jerry Zhenlei Cai ( jpenguin at gmail dot com )
• Guido Kraemer

additional contributions by

• @huwenshuo
• @tmlbl

Download Details:

Author: jerryzhenleicai
Source Code: https://github.com/jerryzhenleicai/LevelDB.jl 
License: View license

#julia #leveldb #interface #database 

Pdf2gerb: Perl Script Converts PDF Files to Gerber format

pdf2gerb

Perl script converts PDF files to Gerber format

Pdf2Gerb generates Gerber 274X photoplotting and Excellon drill files from PDFs of a PCB. Up to three PDFs are used: the top copper layer, the bottom copper layer (for 2-sided PCBs), and an optional silk screen layer. The PDFs can be created directly from any PDF drawing software, or a PDF print driver can be used to capture the Print output if the drawing software does not directly support output to PDF.

The general workflow is as follows:

1. Design the PCB using your favorite CAD or drawing software.
2. Print the top and bottom copper and top silk screen layers to a PDF file.
3. Run Pdf2Gerb on the PDFs to create Gerber and Excellon files.
4. Use a Gerber viewer to double-check the output against the original PCB design.
5. Make adjustments as needed.
6. Submit the files to a PCB manufacturer.

Please note that Pdf2Gerb does NOT perform DRC (Design Rule Checks), as these will vary according to individual PCB manufacturer conventions and capabilities. Also note that Pdf2Gerb is not perfect, so the output files must always be checked before submitting them. As of version 1.6, Pdf2Gerb supports most PCB elements, such as round and square pads, round holes, traces, SMD pads, ground planes, no-fill areas, and panelization. However, because it interprets the graphical output of a Print function, there are limitations in what it can recognize (or there may be bugs).

See docs/Pdf2Gerb.pdf for install/setup, config, usage, and other info.

---

pdf2gerb_cfg.pm

#Pdf2Gerb config settings:
#Put this file in same folder/directory as pdf2gerb.pl itself (global settings),
#or copy to another folder/directory with PDFs if you want PCB-specific settings.
#There is only one user of this file, so we don't need a custom package or namespace.
#NOTE: all constants defined in here will be added to main namespace.
#package pdf2gerb_cfg;

use strict; #trap undef vars (easier debug)
use warnings; #other useful info (easier debug)


##############################################################################################
#configurable settings:
#change values here instead of in main pfg2gerb.pl file

use constant WANT_COLORS => ($^O !~ m/Win/); #ANSI colors no worky on Windows? this must be set < first DebugPrint() call

#just a little warning; set realistic expectations:
#DebugPrint("${\(CYAN)}Pdf2Gerb.pl ${\(VERSION)}, $^O O/S\n${\(YELLOW)}${\(BOLD)}${\(ITALIC)}This is EXPERIMENTAL software.  \nGerber files MAY CONTAIN ERRORS.  Please CHECK them before fabrication!${\(RESET)}", 0); #if WANT_DEBUG

use constant METRIC => FALSE; #set to TRUE for metric units (only affect final numbers in output files, not internal arithmetic)
use constant APERTURE_LIMIT => 0; #34; #max #apertures to use; generate warnings if too many apertures are used (0 to not check)
use constant DRILL_FMT => '2.4'; #'2.3'; #'2.4' is the default for PCB fab; change to '2.3' for CNC

use constant WANT_DEBUG => 0; #10; #level of debug wanted; higher == more, lower == less, 0 == none
use constant GERBER_DEBUG => 0; #level of debug to include in Gerber file; DON'T USE FOR FABRICATION
use constant WANT_STREAMS => FALSE; #TRUE; #save decompressed streams to files (for debug)
use constant WANT_ALLINPUT => FALSE; #TRUE; #save entire input stream (for debug ONLY)

#DebugPrint(sprintf("${\(CYAN)}DEBUG: stdout %d, gerber %d, want streams? %d, all input? %d, O/S: $^O, Perl: $]${\(RESET)}\n", WANT_DEBUG, GERBER_DEBUG, WANT_STREAMS, WANT_ALLINPUT), 1);
#DebugPrint(sprintf("max int = %d, min int = %d\n", MAXINT, MININT), 1); 

#define standard trace and pad sizes to reduce scaling or PDF rendering errors:
#This avoids weird aperture settings and replaces them with more standardized values.
#(I'm not sure how photoplotters handle strange sizes).
#Fewer choices here gives more accurate mapping in the final Gerber files.
#units are in inches
use constant TOOL_SIZES => #add more as desired
(
#round or square pads (> 0) and drills (< 0):
    .010, -.001,  #tiny pads for SMD; dummy drill size (too small for practical use, but needed so StandardTool will use this entry)
    .031, -.014,  #used for vias
    .041, -.020,  #smallest non-filled plated hole
    .051, -.025,
    .056, -.029,  #useful for IC pins
    .070, -.033,
    .075, -.040,  #heavier leads
#    .090, -.043,  #NOTE: 600 dpi is not high enough resolution to reliably distinguish between .043" and .046", so choose 1 of the 2 here
    .100, -.046,
    .115, -.052,
    .130, -.061,
    .140, -.067,
    .150, -.079,
    .175, -.088,
    .190, -.093,
    .200, -.100,
    .220, -.110,
    .160, -.125,  #useful for mounting holes
#some additional pad sizes without holes (repeat a previous hole size if you just want the pad size):
    .090, -.040,  #want a .090 pad option, but use dummy hole size
    .065, -.040, #.065 x .065 rect pad
    .035, -.040, #.035 x .065 rect pad
#traces:
    .001,  #too thin for real traces; use only for board outlines
    .006,  #minimum real trace width; mainly used for text
    .008,  #mainly used for mid-sized text, not traces
    .010,  #minimum recommended trace width for low-current signals
    .012,
    .015,  #moderate low-voltage current
    .020,  #heavier trace for power, ground (even if a lighter one is adequate)
    .025,
    .030,  #heavy-current traces; be careful with these ones!
    .040,
    .050,
    .060,
    .080,
    .100,
    .120,
);
#Areas larger than the values below will be filled with parallel lines:
#This cuts down on the number of aperture sizes used.
#Set to 0 to always use an aperture or drill, regardless of size.
use constant { MAX_APERTURE => max((TOOL_SIZES)) + .004, MAX_DRILL => -min((TOOL_SIZES)) + .004 }; #max aperture and drill sizes (plus a little tolerance)
#DebugPrint(sprintf("using %d standard tool sizes: %s, max aper %.3f, max drill %.3f\n", scalar((TOOL_SIZES)), join(", ", (TOOL_SIZES)), MAX_APERTURE, MAX_DRILL), 1);

#NOTE: Compare the PDF to the original CAD file to check the accuracy of the PDF rendering and parsing!
#for example, the CAD software I used generated the following circles for holes:
#CAD hole size:   parsed PDF diameter:      error:
#  .014                .016                +.002
#  .020                .02267              +.00267
#  .025                .026                +.001
#  .029                .03167              +.00267
#  .033                .036                +.003
#  .040                .04267              +.00267
#This was usually ~ .002" - .003" too big compared to the hole as displayed in the CAD software.
#To compensate for PDF rendering errors (either during CAD Print function or PDF parsing logic), adjust the values below as needed.
#units are pixels; for example, a value of 2.4 at 600 dpi = .0004 inch, 2 at 600 dpi = .0033"
use constant
{
    HOLE_ADJUST => -0.004 * 600, #-2.6, #holes seemed to be slightly oversized (by .002" - .004"), so shrink them a little
    RNDPAD_ADJUST => -0.003 * 600, #-2, #-2.4, #round pads seemed to be slightly oversized, so shrink them a little
    SQRPAD_ADJUST => +0.001 * 600, #+.5, #square pads are sometimes too small by .00067, so bump them up a little
    RECTPAD_ADJUST => 0, #(pixels) rectangular pads seem to be okay? (not tested much)
    TRACE_ADJUST => 0, #(pixels) traces seemed to be okay?
    REDUCE_TOLERANCE => .001, #(inches) allow this much variation when reducing circles and rects
};

#Also, my CAD's Print function or the PDF print driver I used was a little off for circles, so define some additional adjustment values here:
#Values are added to X/Y coordinates; units are pixels; for example, a value of 1 at 600 dpi would be ~= .002 inch
use constant
{
    CIRCLE_ADJUST_MINX => 0,
    CIRCLE_ADJUST_MINY => -0.001 * 600, #-1, #circles were a little too high, so nudge them a little lower
    CIRCLE_ADJUST_MAXX => +0.001 * 600, #+1, #circles were a little too far to the left, so nudge them a little to the right
    CIRCLE_ADJUST_MAXY => 0,
    SUBST_CIRCLE_CLIPRECT => FALSE, #generate circle and substitute for clip rects (to compensate for the way some CAD software draws circles)
    WANT_CLIPRECT => TRUE, #FALSE, #AI doesn't need clip rect at all? should be on normally?
    RECT_COMPLETION => FALSE, #TRUE, #fill in 4th side of rect when 3 sides found
};

#allow .012 clearance around pads for solder mask:
#This value effectively adjusts pad sizes in the TOOL_SIZES list above (only for solder mask layers).
use constant SOLDER_MARGIN => +.012; #units are inches

#line join/cap styles:
use constant
{
    CAP_NONE => 0, #butt (none); line is exact length
    CAP_ROUND => 1, #round cap/join; line overhangs by a semi-circle at either end
    CAP_SQUARE => 2, #square cap/join; line overhangs by a half square on either end
    CAP_OVERRIDE => FALSE, #cap style overrides drawing logic
};
    
#number of elements in each shape type:
use constant
{
    RECT_SHAPELEN => 6, #x0, y0, x1, y1, count, "rect" (start, end corners)
    LINE_SHAPELEN => 6, #x0, y0, x1, y1, count, "line" (line seg)
    CURVE_SHAPELEN => 10, #xstart, ystart, x0, y0, x1, y1, xend, yend, count, "curve" (bezier 2 points)
    CIRCLE_SHAPELEN => 5, #x, y, 5, count, "circle" (center + radius)
};
#const my %SHAPELEN =
#Readonly my %SHAPELEN =>
our %SHAPELEN =
(
    rect => RECT_SHAPELEN,
    line => LINE_SHAPELEN,
    curve => CURVE_SHAPELEN,
    circle => CIRCLE_SHAPELEN,
);

#panelization:
#This will repeat the entire body the number of times indicated along the X or Y axes (files grow accordingly).
#Display elements that overhang PCB boundary can be squashed or left as-is (typically text or other silk screen markings).
#Set "overhangs" TRUE to allow overhangs, FALSE to truncate them.
#xpad and ypad allow margins to be added around outer edge of panelized PCB.
use constant PANELIZE => {'x' => 1, 'y' => 1, 'xpad' => 0, 'ypad' => 0, 'overhangs' => TRUE}; #number of times to repeat in X and Y directions

# Set this to 1 if you need TurboCAD support.
#$turboCAD = FALSE; #is this still needed as an option?

#CIRCAD pad generation uses an appropriate aperture, then moves it (stroke) "a little" - we use this to find pads and distinguish them from PCB holes. 
use constant PAD_STROKE => 0.3; #0.0005 * 600; #units are pixels
#convert very short traces to pads or holes:
use constant TRACE_MINLEN => .001; #units are inches
#use constant ALWAYS_XY => TRUE; #FALSE; #force XY even if X or Y doesn't change; NOTE: needs to be TRUE for all pads to show in FlatCAM and ViewPlot
use constant REMOVE_POLARITY => FALSE; #TRUE; #set to remove subtractive (negative) polarity; NOTE: must be FALSE for ground planes

#PDF uses "points", each point = 1/72 inch
#combined with a PDF scale factor of .12, this gives 600 dpi resolution (1/72 * .12 = 600 dpi)
use constant INCHES_PER_POINT => 1/72; #0.0138888889; #multiply point-size by this to get inches

# The precision used when computing a bezier curve. Higher numbers are more precise but slower (and generate larger files).
#$bezierPrecision = 100;
use constant BEZIER_PRECISION => 36; #100; #use const; reduced for faster rendering (mainly used for silk screen and thermal pads)

# Ground planes and silk screen or larger copper rectangles or circles are filled line-by-line using this resolution.
use constant FILL_WIDTH => .01; #fill at most 0.01 inch at a time

# The max number of characters to read into memory
use constant MAX_BYTES => 10 * M; #bumped up to 10 MB, use const

use constant DUP_DRILL1 => TRUE; #FALSE; #kludge: ViewPlot doesn't load drill files that are too small so duplicate first tool

my $runtime = time(); #Time::HiRes::gettimeofday(); #measure my execution time

print STDERR "Loaded config settings from '${\(__FILE__)}'.\n";
1; #last value must be truthful to indicate successful load


#############################################################################################
#junk/experiment:

#use Package::Constants;
#use Exporter qw(import); #https://perldoc.perl.org/Exporter.html

#my $caller = "pdf2gerb::";

#sub cfg
#{
#    my $proto = shift;
#    my $class = ref($proto) || $proto;
#    my $settings =
#    {
#        $WANT_DEBUG => 990, #10; #level of debug wanted; higher == more, lower == less, 0 == none
#    };
#    bless($settings, $class);
#    return $settings;
#}

#use constant HELLO => "hi there2"; #"main::HELLO" => "hi there";
#use constant GOODBYE => 14; #"main::GOODBYE" => 12;

#print STDERR "read cfg file\n";

#our @EXPORT_OK = Package::Constants->list(__PACKAGE__); #https://www.perlmonks.org/?node_id=1072691; NOTE: "_OK" skips short/common names

#print STDERR scalar(@EXPORT_OK) . " consts exported:\n";
#foreach(@EXPORT_OK) { print STDERR "$_\n"; }
#my $val = main::thing("xyz");
#print STDERR "caller gave me $val\n";
#foreach my $arg (@ARGV) { print STDERR "arg $arg\n"; }

---

Download Details:

Author: swannman
Source Code: https://github.com/swannman/pdf2gerb

License: GPL-3.0 license

#perl 

Level-cli: Command Line interface for Leveldb

LevelDB command line interface

Inspect and alter your leveldb records with the command line.

Similar to lev

Setup

npm i -g level-cli

Usage

Usage: lev [command] [options]

Commands:                         Alias:

  keys                            k
  values                          v
  list                            l
  get <key>                       g
  put <key> <value>               p
  del <key>                       d
      
Options:

  -V, --version                   output the version number
  -p, --path <path>               Path to leveldb (default: . )
  -n, --limit <limit>             Stop reading after "limit" entries
  -f, --from <from>               Read records starting at "from"
  -t, --to <to>                   Read records until "to"
  -r, --reverse                   Reverse order
  -k, --keyEncoding <encoding>    key encoding [utf8, ascii, json, hex] (default: utf8)
  -v, --valueEncoding <encoding>  value encoding [utf8, ascii, json, hex] (default: utf8)
  -h, --help                      output usage information

Example

lev keys -n 10

Troubleshooting

Some shells treat the ! symbol specifically. Thus, if you encounter sth. like that:

lev put a!foo bar

=> -bash: !foo: event not found

It means you're using history substitution which you can turn off / on using set +H / set -H.

Author: Mablay
Source Code: https://github.com/mablay/level-cli 
License: 

#javascript #node #cli #leveldb 

Level-over-http: Provides an interface to Level Db Over Http

level-over-http

Serves and stores items to a level database over http. level-over-http can live stream a level database and accept the same options defined in the LevelDOWN API.

Here is an example of serving 'test.db' over http://localhost:3000/test

var level       = require('level');
var http        = require('http');
var levelHttp   = require('level-over-http');

var db   = level('test.db');
var server = http.createServer(levelHttp.serve('/test', db)).listen(3000);

To push something to the leveldb, make a POST request.

    var options = {
        host : 'localhost',
        port : 3000,
        path : '/test',
        method : 'POST'
    }

    var req = http.request(options, function(res) {
        res.pipe(process.stdout);
    });

    req.on('error', function(e) {
        console.log('problem with request: ' + e.message);
    })

    req.end('{"value":"hello wisconsin"}\n');

The following response is streamed to stdout.

{"result":"success","key":"0001427765362253.000000000"}

To live stream something from the leveldb, make a GET request.

    var options = {
        host : 'localhost',
        port : 3000,
        path : '/test',
        headers : { gte : '0001427765362253.000000000' }
    }

    var req = http.request(options, onResponse);

    req.end();

Notice the headers object can contain the same options as defined in the LevelDOWN API.

{"key":"0001427765362229.000000000","value":"hello wisconsin"}

level-over-http uses lexicographic-timestamp for key generation.

Author: lakowske 
Source Code: https://github.com/lakowske/level-over-http 
License: 

#javascript #http #leveldb