1657048080
Leveldb: A Fast Key-value Storage Library Written At Google
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
Features
- Keys and values are arbitrary byte arrays.
- Data is stored sorted by key.
- Callers can provide a custom comparison function to override the sort order.
- The basic operations are
Put(key,value),Get(key),Delete(key). - Multiple changes can be made in one atomic batch.
- Users can create a transient snapshot to get a consistent view of data.
- Forward and backward iteration is supported over the data.
- Data is automatically compressed using the Snappy compression library.
- External activity (file system operations etc.) is relayed through a virtual interface so users can customize the operating system interactions.
Limitations
- This is not a SQL database. It does not have a relational data model, it does not support SQL queries, and it has no support for indexes.
- Only a single process (possibly multi-threaded) can access a particular database at a time.
- There is no client-server support builtin to the library. An application that needs such support will have to wrap their own server around the library.
Getting the Source
git clone --recurse-submodules https://github.com/google/leveldb.git
Building
This project supports CMake out of the box.
Build for POSIX
Quick start:
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release .. && cmake --build .
Building for Windows
First generate the Visual Studio 2017 project/solution files:
mkdir build
cd build
cmake -G "Visual Studio 15" ..
The default default will build for x86. For 64-bit run:
cmake -G "Visual Studio 15 Win64" ..
To compile the Windows solution from the command-line:
devenv /build Debug leveldb.sln
or open leveldb.sln in Visual Studio and build from within.
Please see the CMake documentation and CMakeLists.txt for more advanced usage.
Contributing to the leveldb Project
The leveldb project welcomes contributions. leveldb's primary goal is to be a reliable and fast key/value store. Changes that are in line with the features/limitations outlined above, and meet the requirements below, will be considered.
Contribution requirements:
Tested platforms only. We generally will only accept changes for platforms that are compiled and tested. This means POSIX (for Linux and macOS) or Windows. Very small changes will sometimes be accepted, but consider that more of an exception than the rule.
Stable API. We strive very hard to maintain a stable API. Changes that require changes for projects using leveldb might be rejected without sufficient benefit to the project.
Tests: All changes must be accompanied by a new (or changed) test, or a sufficient explanation as to why a new (or changed) test is not required.
Consistent Style: This project conforms to the Google C++ Style Guide. To ensure your changes are properly formatted please run:
clang-format -i --style=file <file>We are unlikely to accept contributions to the build configuration files, such as CMakeLists.txt. We are focused on maintaining a build configuration that allows us to test that the project works in a few supported configurations inside Google. We are not currently interested in supporting other requirements, such as different operating systems, compilers, or build systems.
Submitting a Pull Request
Before any pull request will be accepted the author must first sign a Contributor License Agreement (CLA) at https://cla.developers.google.com/.
In order to keep the commit timeline linear squash your changes down to a single commit and rebase on google/leveldb/main. This keeps the commit timeline linear and more easily sync'ed with the internal repository at Google. More information at GitHub's About Git rebase page.
Performance
Here is a performance report (with explanations) from the run of the included db_bench program. The results are somewhat noisy, but should be enough to get a ballpark performance estimate.
Setup
We use a database with a million entries. Each entry has a 16 byte key, and a 100 byte value. Values used by the benchmark compress to about half their original size.
LevelDB: version 1.1
Date: Sun May 1 12:11:26 2011
CPU: 4 x Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
CPUCache: 4096 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
Raw Size: 110.6 MB (estimated)
File Size: 62.9 MB (estimated)
Write performance
The "fill" benchmarks create a brand new database, in either sequential, or random order. The "fillsync" benchmark flushes data from the operating system to the disk after every operation; the other write operations leave the data sitting in the operating system buffer cache for a while. The "overwrite" benchmark does random writes that update existing keys in the database.
fillseq : 1.765 micros/op; 62.7 MB/s
fillsync : 268.409 micros/op; 0.4 MB/s (10000 ops)
fillrandom : 2.460 micros/op; 45.0 MB/s
overwrite : 2.380 micros/op; 46.5 MB/s
Each "op" above corresponds to a write of a single key/value pair. I.e., a random write benchmark goes at approximately 400,000 writes per second.
Each "fillsync" operation costs much less (0.3 millisecond) than a disk seek (typically 10 milliseconds). We suspect that this is because the hard disk itself is buffering the update in its memory and responding before the data has been written to the platter. This may or may not be safe based on whether or not the hard disk has enough power to save its memory in the event of a power failure.
Read performance
We list the performance of reading sequentially in both the forward and reverse direction, and also the performance of a random lookup. Note that the database created by the benchmark is quite small. Therefore the report characterizes the performance of leveldb when the working set fits in memory. The cost of reading a piece of data that is not present in the operating system buffer cache will be dominated by the one or two disk seeks needed to fetch the data from disk. Write performance will be mostly unaffected by whether or not the working set fits in memory.
readrandom : 16.677 micros/op; (approximately 60,000 reads per second)
readseq : 0.476 micros/op; 232.3 MB/s
readreverse : 0.724 micros/op; 152.9 MB/s
LevelDB compacts its underlying storage data in the background to improve read performance. The results listed above were done immediately after a lot of random writes. The results after compactions (which are usually triggered automatically) are better.
readrandom : 11.602 micros/op; (approximately 85,000 reads per second)
readseq : 0.423 micros/op; 261.8 MB/s
readreverse : 0.663 micros/op; 166.9 MB/s
Some of the high cost of reads comes from repeated decompression of blocks read from disk. If we supply enough cache to the leveldb so it can hold the uncompressed blocks in memory, the read performance improves again:
readrandom : 9.775 micros/op; (approximately 100,000 reads per second before compaction)
readrandom : 5.215 micros/op; (approximately 190,000 reads per second after compaction)
Repository contents
See doc/index.md for more explanation. See doc/impl.md for a brief overview of the implementation.
The public interface is in include/leveldb/*.h. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.
Guide to header files:
include/leveldb/db.h: Main interface to the DB: Start here.
include/leveldb/options.h: Control over the behavior of an entire database, and also control over the behavior of individual reads and writes.
include/leveldb/comparator.h: Abstraction for user-specified comparison function. If you want just bytewise comparison of keys, you can use the default comparator, but clients can write their own comparator implementations if they want custom ordering (e.g. to handle different character encodings, etc.).
include/leveldb/iterator.h: Interface for iterating over data. You can get an iterator from a DB object.
include/leveldb/write_batch.h: Interface for atomically applying multiple updates to a database.
include/leveldb/slice.h: A simple module for maintaining a pointer and a length into some other byte array.
include/leveldb/status.h: Status is returned from many of the public interfaces and is used to report success and various kinds of errors.
include/leveldb/env.h: Abstraction of the OS environment. A posix implementation of this interface is in util/env_posix.cc.
include/leveldb/table.h, include/leveldb/table_builder.h: Lower-level modules that most clients probably won't use directly.
Documentation
LevelDB library documentation is online and bundled with the source code.
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
Author: Google
Source Code: https://github.com/google/leveldb
License: BSD-3-Clause license
What is GEEK
Buddha Community
1657048080
Leveldb: A Fast Key-value Storage Library Written At Google
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
Features
- Keys and values are arbitrary byte arrays.
- Data is stored sorted by key.
- Callers can provide a custom comparison function to override the sort order.
- The basic operations are
Put(key,value),Get(key),Delete(key). - Multiple changes can be made in one atomic batch.
- Users can create a transient snapshot to get a consistent view of data.
- Forward and backward iteration is supported over the data.
- Data is automatically compressed using the Snappy compression library.
- External activity (file system operations etc.) is relayed through a virtual interface so users can customize the operating system interactions.
Limitations
- This is not a SQL database. It does not have a relational data model, it does not support SQL queries, and it has no support for indexes.
- Only a single process (possibly multi-threaded) can access a particular database at a time.
- There is no client-server support builtin to the library. An application that needs such support will have to wrap their own server around the library.
Getting the Source
git clone --recurse-submodules https://github.com/google/leveldb.git
Building
This project supports CMake out of the box.
Build for POSIX
Quick start:
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release .. && cmake --build .
Building for Windows
First generate the Visual Studio 2017 project/solution files:
mkdir build
cd build
cmake -G "Visual Studio 15" ..
The default default will build for x86. For 64-bit run:
cmake -G "Visual Studio 15 Win64" ..
To compile the Windows solution from the command-line:
devenv /build Debug leveldb.sln
or open leveldb.sln in Visual Studio and build from within.
Please see the CMake documentation and CMakeLists.txt for more advanced usage.
Contributing to the leveldb Project
The leveldb project welcomes contributions. leveldb's primary goal is to be a reliable and fast key/value store. Changes that are in line with the features/limitations outlined above, and meet the requirements below, will be considered.
Contribution requirements:
Tested platforms only. We generally will only accept changes for platforms that are compiled and tested. This means POSIX (for Linux and macOS) or Windows. Very small changes will sometimes be accepted, but consider that more of an exception than the rule.
Stable API. We strive very hard to maintain a stable API. Changes that require changes for projects using leveldb might be rejected without sufficient benefit to the project.
Tests: All changes must be accompanied by a new (or changed) test, or a sufficient explanation as to why a new (or changed) test is not required.
Consistent Style: This project conforms to the Google C++ Style Guide. To ensure your changes are properly formatted please run:
clang-format -i --style=file <file>We are unlikely to accept contributions to the build configuration files, such as CMakeLists.txt. We are focused on maintaining a build configuration that allows us to test that the project works in a few supported configurations inside Google. We are not currently interested in supporting other requirements, such as different operating systems, compilers, or build systems.
Submitting a Pull Request
Before any pull request will be accepted the author must first sign a Contributor License Agreement (CLA) at https://cla.developers.google.com/.
In order to keep the commit timeline linear squash your changes down to a single commit and rebase on google/leveldb/main. This keeps the commit timeline linear and more easily sync'ed with the internal repository at Google. More information at GitHub's About Git rebase page.
Performance
Here is a performance report (with explanations) from the run of the included db_bench program. The results are somewhat noisy, but should be enough to get a ballpark performance estimate.
Setup
We use a database with a million entries. Each entry has a 16 byte key, and a 100 byte value. Values used by the benchmark compress to about half their original size.
LevelDB: version 1.1
Date: Sun May 1 12:11:26 2011
CPU: 4 x Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
CPUCache: 4096 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
Raw Size: 110.6 MB (estimated)
File Size: 62.9 MB (estimated)
Write performance
The "fill" benchmarks create a brand new database, in either sequential, or random order. The "fillsync" benchmark flushes data from the operating system to the disk after every operation; the other write operations leave the data sitting in the operating system buffer cache for a while. The "overwrite" benchmark does random writes that update existing keys in the database.
fillseq : 1.765 micros/op; 62.7 MB/s
fillsync : 268.409 micros/op; 0.4 MB/s (10000 ops)
fillrandom : 2.460 micros/op; 45.0 MB/s
overwrite : 2.380 micros/op; 46.5 MB/s
Each "op" above corresponds to a write of a single key/value pair. I.e., a random write benchmark goes at approximately 400,000 writes per second.
Each "fillsync" operation costs much less (0.3 millisecond) than a disk seek (typically 10 milliseconds). We suspect that this is because the hard disk itself is buffering the update in its memory and responding before the data has been written to the platter. This may or may not be safe based on whether or not the hard disk has enough power to save its memory in the event of a power failure.
Read performance
We list the performance of reading sequentially in both the forward and reverse direction, and also the performance of a random lookup. Note that the database created by the benchmark is quite small. Therefore the report characterizes the performance of leveldb when the working set fits in memory. The cost of reading a piece of data that is not present in the operating system buffer cache will be dominated by the one or two disk seeks needed to fetch the data from disk. Write performance will be mostly unaffected by whether or not the working set fits in memory.
readrandom : 16.677 micros/op; (approximately 60,000 reads per second)
readseq : 0.476 micros/op; 232.3 MB/s
readreverse : 0.724 micros/op; 152.9 MB/s
LevelDB compacts its underlying storage data in the background to improve read performance. The results listed above were done immediately after a lot of random writes. The results after compactions (which are usually triggered automatically) are better.
readrandom : 11.602 micros/op; (approximately 85,000 reads per second)
readseq : 0.423 micros/op; 261.8 MB/s
readreverse : 0.663 micros/op; 166.9 MB/s
Some of the high cost of reads comes from repeated decompression of blocks read from disk. If we supply enough cache to the leveldb so it can hold the uncompressed blocks in memory, the read performance improves again:
readrandom : 9.775 micros/op; (approximately 100,000 reads per second before compaction)
readrandom : 5.215 micros/op; (approximately 190,000 reads per second after compaction)
Repository contents
See doc/index.md for more explanation. See doc/impl.md for a brief overview of the implementation.
The public interface is in include/leveldb/*.h. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.
Guide to header files:
include/leveldb/db.h: Main interface to the DB: Start here.
include/leveldb/options.h: Control over the behavior of an entire database, and also control over the behavior of individual reads and writes.
include/leveldb/comparator.h: Abstraction for user-specified comparison function. If you want just bytewise comparison of keys, you can use the default comparator, but clients can write their own comparator implementations if they want custom ordering (e.g. to handle different character encodings, etc.).
include/leveldb/iterator.h: Interface for iterating over data. You can get an iterator from a DB object.
include/leveldb/write_batch.h: Interface for atomically applying multiple updates to a database.
include/leveldb/slice.h: A simple module for maintaining a pointer and a length into some other byte array.
include/leveldb/status.h: Status is returned from many of the public interfaces and is used to report success and various kinds of errors.
include/leveldb/env.h: Abstraction of the OS environment. A posix implementation of this interface is in util/env_posix.cc.
include/leveldb/table.h, include/leveldb/table_builder.h: Lower-level modules that most clients probably won't use directly.
Documentation
LevelDB library documentation is online and bundled with the source code.
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
Author: Google
Source Code: https://github.com/google/leveldb
License: BSD-3-Clause license
1619247660
Google's TPU's being primed for the Quantum Jump
The liquid-cooled Tensor Processing Units, built to slot into server racks, can deliver up to 100 petaflops of compute.
The liquid-cooled Tensor Processing Units, built to slot into server racks, can deliver up to 100 petaflops of compute.
As the world is gearing towards more automation and AI, the need for quantum computing has also grown exponentially. Quantum computing lies at the intersection of quantum physics and high-end computer technology, and in more than one way, hold the key to our AI-driven future.
Quantum computing requires state-of-the-art tools to perform high-end computing. This is where TPUs come in handy. TPUs or Tensor Processing Units are custom-built ASICs (Application Specific Integrated Circuits) to execute machine learning tasks efficiently. TPUs are specific hardware developed by Google for neural network machine learning, specially customised to Google’s Machine Learning software, Tensorflow.
The liquid-cooled Tensor Processing units, built to slot into server racks, can deliver up to 100 petaflops of compute. It powers Google products like Google Search, Gmail, Google Photos and Google Cloud AI APIs.
#opinions #alphabet #asics #floq #google #google alphabet #google quantum computing #google tensorflow #google tensorflow quantum #google tpu #google tpus #machine learning #quantum computer #quantum computing #quantum computing programming #quantum leap #sandbox #secret development #tensorflow #tpu #tpus
1624575120
Java: How to Get Keys and Values from a Map
Introduction
Key-value stores are essential and often used, especially in operations that require fast and frequent lookups. They allow an object - the key - to be mapped to another object, the value. This way, the values can easily be retrieved, by looking up the key.
In Java, the most popular Map implementation is the HashMap class. Aside from key-value mapping, it’s used in code that requires frequest insertions, updates and lookups. The insert and lookup time is a constant O(1).
In this tutorial, we’ll go over how to get the Keys and Values of a map in Java.
#java #java: how to get keys and values from a map #keys #map #values #how to get keys and values from a map
1661805960
Julia Bindings to KyotoCabinet Library (key Value Storage)
Julia binding for KyotoCabinet
This package provides bindings for KyotoCabinet key-value storage.
Installation
Pkg.add("KyotoCabinet")
Generic interface
using KyotoCabinet
To open database, use open method:
db = open("db.kch", "r")
# db::Dict{Array{Uint8,1},Array{Uint8,1}}
close(db)
There is also bracketed version:
open(Db{K,V}(), "db.kch", "w+") do db
# db::Dict{K,V}
# do stuff...
end
Db object implements basic collections and Dict methods.
open(Db{String,String}(), "db.kch", "w+") do db
# Basic getindex, setindex! methods
db["a"] = "1"
println(db["a"])
# Dict methods also implemented:
# haskey, getkey, get, get!, delete!, pop!
if (!haskey(db, "x"))
x = get(db, "x", "default")
y = get!(db, "y", "set_value_if_non_exists")
end
end
Support iteration over records, keys and values:
for (k, v) = db
println("k=$k v=$v")
end
for k = keys(db)
println("k=$k")
end
Serialization/Deserialization
KyotoCabinet treats keys and values as byte arrays. To make it work with arbitrary types, one needs to define pack/unpack methods.
immutable K
x::Int
end
immutable V
a::Int
b::String
end
function KyotoCabinet.pack(k::K)
io = IOBuffer()
write(io, int32(k.x))
takebuf_array(io)
end
function KyotoCabinet.unpack(T::Type{K}, buf::Array{Uint8,1})
io = IOBuffer(buf)
x = read(io, Int32)
K(int(x))
end
function KyotoCabinet.pack(v::V)
io = IOBuffer()
write(io, int32(v.a))
write(io, int32(length(v.b)))
write(io, v.b)
takebuf_array(io)
end
function KyotoCabinet.unpack(T::Type{V}, buf::Array{Uint8,1})
io = IOBuffer(buf)
a = read(io, Int32)
l = read(io, Int32)
b = bytestring(read(io, Uint8, l))
V(int(a), b)
end
After that these types can be used as keys/values:
open(Db{K, V}(), "db.kch", "w+") do db
db[K(1)] = V(1, "a")
db[K(1999999999)] = V(2, repeat("b",100))
end
k = K(1)
println(db[k])
KyotoCabinet specific
There are also KyotoCabinet specific methods.
Database info
# Get the path of the database file
p = path(db)
Compare-and-swap
cas(db::Db, key, old, new)
Compare-and-swap method. Update the value only if it's in the expected state. Returns true if value have been updated.
cas(db, "k", "old", "new") # update only if db["k"] == "old"
cas(db, "k", "old", ()) # remove record, only if db["k"] == "old"
cas(db, "k", (), "new") # add record, only if "k" not in db
Bulk operations
# Updates records in one operation, atomically if needed.
bulkset!(db, ["a" => "1", "b" => "2"], true)
# Removes records in one operation, atomically if needed.
bulkdelete!(db, ["a", "b"], true)Download Details:
Author: Tuzzeg
Source Code: https://github.com/tuzzeg/kyotocabinet.jl
License: View license
1598383290
What Are Google Compute Engine ? - Explained
What Are Google Compute Engine ? - Explained
The Google computer engine exchanges a large number of scalable virtual machines to serve as clusters used for that purpose. GCE can be managed through a RESTful API, command line interface, or web console. The computing engine is serviced for a minimum of 10-minutes per use. There is no up or front fee or time commitment. GCE competes with Amazon’s Elastic Compute Cloud (EC2) and Microsoft Azure.
https://www.mrdeluofficial.com/2020/08/what-are-google-compute-engine-explained.html
#google compute engine #google compute engine tutorial #google app engine #google cloud console #google cloud storage #google compute engine documentation