Sexpr.jl: Julia <3 Clojure + Macroexpansion

S-Julia - s-expression to julia convertor.  

Quickstart

> Pkg.clone("https://github.com/vshesh/Sexpr.jl.git")

$ julia -e 'import Sexpr; Sexpr.main()' --
usage: Sexpr.jl [-i] [-c] [-l LINES] [-o OUTPUT] [-e EXTENSION] [-h]
               [files...]
A program to port clojure-like s-expression syntax to and from
julia. By default, this program takes clojure syntax and outputs
the julia version. Use -i to flip direction.

positional arguments:
  files                 If given one file and no output directory,
                        will dump to stdout. If given a directory or
                        multiple files, eg "sjulia file1 dir file2",
                        an output directory must be specified with
                        -o/--output where the files will go.
optional arguments:
  -i, --invert          take julia code and print out s-expression
                        code instead
  -c, --cat             cat all the input from STDIN rather than read
                        from file. Ignores all positional args to the
                        program.
  -l, --lines LINES     how many blank lines should exist between top
                        level forms, default 1 (type: Int64, default:
                        1)
  -o, --output OUTPUT   where to write out files if there are multiple
                        positional arguments to the file. If this is
                        empty, and there are >1 argument, the program
                        will throw an error.
  -e, --extension EXTENSION
                        add an extension that qualifies as a lisp file
                        (can use multiple times). Defaults: clj, cljs,
                        cl, lisp, wisp, hy.
  -h, --help            show this help message and exit
$ julia -e 'import Sexpr; Sexpr.main()' -- -o test/output/ test/programs/
# will transpile all .clj files in test/programs and dump them into test/output.

Overview

This project aims to make s-expression syntax interoperable with julia's own Expr objects.

If you've seen LispSyntax.jl, it's a similar idea, but IMHO this project does a bit more, such as allow you to transpile file->file rather than just read in a program, and also transpile back, so you can convert your julia files (minus a few special forms that aren't supported yet) into clojure syntax. This makes it possible to go from julia to python (again, not that anyone needed another route b/c pycall) via Hylang, or to JS via WispJS. The benefit here is that the awkward macro syntax in both of those languages is avoided (Hy necessitates wrapping everything in HyModel objects yourself, which is ridiculous, and WispJS's module system is broken, because it is Javascript, so resolving variable names is not working properly).

The final goal is to use interoperability to do a macroexpand operation on the input clj syntax. So you would be able to give a folder of clj files, and a temp folder with jl files would be created, then each file would be read in and macroexpanded, converted back to clj syntax, and written out to a third folder. Unfortunately, it's necessary to write the jl files out as an intermediary step, because they need to be able to find each other to resolve imports. Alternatively, you could write the clj files as jl files with the macro @clj_str, but that makes your whole file a string, which breaks most syntax highlighters, which can be annoying.

I know that you're probably thinking "why?" and it was mostly a project for me to learn Julia and muck around with its internals. I learned quite a bit, so mission accomplished! CLJS has self-hosting now, which means that they will hopefully have a js-only package soon. However, dealing with google closure compiler and leiningen's java/jvm dependencies are a larger problem to be solved, and until then, I still consider it unwieldy, so there's still some practical use to be had here.

Effectively, this is just the reader portion of implementing a lisp - Julia does everything else using its inbuilt mechanisms.

Syntax Overview

Atoms

  • nil translates to julia's nothing. They work exactly the same.
  • true -> true and false -> false. No surprises at all there.
  • number constants compile to either Int64 or Float64 types.
    • rational constants also supported, so 3/5 -> 3//5 in Julia.
  • character any atom starting with a \ is a character.
    • \newline, \space, \tab, \formfeed, \backspace, \return for escapes
    • unicode/octal support still needs to be handled.
    • non-strict, giving a longer literal silently just gives you the first character right now. This is probably not the best long-term strategy. Eg, \xyz -> \x
  • string is any sequence of characters inside double quotes.
    • multiline strings are allowed, but padding is not subtracted (yet).
  • keyword basically a symbol that starts with a :. In julia, these are confusingly called symbols, and symbols are called variables.
    • keywords cannot have a / or . in them anywhere.
    • in clojure keywords with two colons are resolved in the current namespace, that behavior is not the same here. Everything just compiles to a normal symbol in julia, so no namespacing. There are probably issues here, I just don't know what they are.
  • symbol which is any identifier for a variable.
    • any / or . character is converted to a . in julia. Eg, module/function becomes module.function as an identifier. This should be relatively consistent with clojure semantics.
    • clojure is more lenient about symbol characters than julia. In order to get around this limitation, the default is to output unicode characters where regular ones won't suffice. so *+?!-_':>< are all allowed inside a symbol.
      • TODO make the option to use escaped ascii-only names available. (ugly, but avoids having to use unicode, which is a pain depending on how your unicode extension is defined).
      • :: in a symbol identifier compiles to a type. eg, x::Int compiles to (:: x Int)
      • ::T1::T2 compiles to a union like x::T1::T2 -> (:: x T1 T2)

Collections

  • '(a b c) a list - if not quoted, it's evaluated and transpiled.
    • quoted lists evaluate to tuples, as of now.
  • [a b c] a vector - transpiles to a julia array.
  • {a b c d} a map - transpiles to a julia Dict(a => b, c=> d) form.
  • TODO #{a b c} a set, which can map to Set() in julia.

Julia Special Forms

  • Short Circuit
    • and/&& (what you expect this to be) - needs to be a special form because of short circuiting. Julia defines the and and or forms this way on purpose.
    • or/|| (again, what you expect), see above.
  • x[i] family (getting/setting/slicing arrays)
    • (aget x 1) -> Expr(:ref, :x, 1) -> x[1].
    • (aget x 1 2 4 5) -> Expr(:ref, :x, 1, 2, 4, 5) -> x[1, 2, 4, 5]
    • (aget x (: 1 3)) -> x[1:3]
    • (aget x (: 6)) -> x[6:end] (preferred)
    • (aget x (: 6 :end)) -> x[6:end] (not preferred)
  • Typing
    • (:: x Int) -> x::Int The :: form defines types.
      • (:: x Int In64) -> x::Union{Int, Int64} there's auto-union if many types are defined.
      • only useful for function and type defintions.
    • (curly Array Int64) -> Array{Int64} will allow parameterized types.
    • (.b a x y) -> a.b(x,y) is the dot call form.
    • (. a b c d) -> a.b.c.d is the dot access form.
      • note that ((. a b) x y) is equivalent to (.b a x y).
  • Modules and Import
    • (module M ... end) creates a module. This is visually annoying since you indent your whole file by two spaces just for this call to module, however I haven't figured out any better way to do this - the other option is to make #module M a special hash dispatch that wraps the whole file but... meh, I don't consider this a high enough priority.
    • (import|using X y z a b) contrary to my expectations, this will give you import X.y.a.b. There will be a separate import statement for each function/file you want to use.
      • TODO make this cartesian productable, so (import X [y z a]) will expand to import X.y; import X.z; import X.a instead. This should shorten the writing. Ideally should make this a system macro (in a system.clj file that I define) and call it import* or something.
    • (export a b c) -> export a, b, c. It makes sense from julia's point of view, since modules are flat things, and you only ever have one level of definitions to export.

Special Forms

()/'() or empty list.

  • For now, this compiles to an empty tuple. In some lisps this is equivalent to nil (eg Common Lisp) but in Clojure it's not, so I'm following that convention.

(do exprs...) does each expression and returns the results of the last one.

(if test true-case false-case?) standard if, evaluates form #2 and branches.

(let [var1 value1 var2 value2...] exprs...) binds pairs of variables to their values, then evaluates exprs in an implicit do.

(fn name? [params...] exprs...) defines a function.

  • a function with no name and only one expr in the body will be converted to a -> form. Eg: (fn [x] x) -> (x) -> x.

(defn name docstring? [params...] exprs...) named defined function.

  • docstrings are ignored right now.

(def var expr) defines a variable.

throw is a function already in julia, so there's no special form dedicated to it.

include is a function already in julia, so there's no dedicated special form for it.

TODOS

  • loop/recur (this doesn't have a julia equivalent),
  • try/catch/finally
  • for vars in expr do... (useful for lazy iterators)
  • destructuring and rest param like (fn [& rest])
  • defmulti and related (does this even mean anything given julia's multiple-dispatch?)
  • deftype -> type in Julia.

Macro Forms

  • (@m x y z) how to call a macro - prepend it's name with @. There is unfortunately no way around this, since julia requires this distinction and for me to resolve what things are macros without it would involve writing an entire compiler. To keep it simple, I'm leaving this requirement in place.
    • TODO Since @x means deref in clojure, I might choose to use a different symbol to denote macrocall in the future. maybe μ or something. Another idea is abusing # dispatch so `#macro (html [:div "helloworld"])`` calls the next form as a macro rather than a regular function. The hash dispatch one seems worse, though.
  • defmacro defines a macro, as expected.
    • The way that macros work right now is that the macro definition is passed a clojure s-expression to work with. This is not the same as being passed a julia equivalent.
    • the macro output should again be a clojure expression, which has to be translated by the reader into a julia expression. This means that whatever program you write has to include the reader module of this project in order to produce the desired output.
    • Every macro will end in a call to Sexpr.rehydrate() which will translate the expression back to julia's native AST.
  • quote or ' gives a literal list of the following expression.
    • The quote form doesn't properly escape symbols yet. Eg, 'x is equal to :x in Julia, but in order to stop the gensym pass from running you actually have to do esc(:x) to get the equivalent. I'm unclear as of yet how the translation should work to get the desired results, so right now quote and syntax-quote do the same thing, which needs to be changed.
    • you can get around this yourself by putting esc calls in the right places, it will compile down to a function call in the code.
  • syntax-quote or backtick character. the :() quoting form in julia is actually a syntax quote. It also has an auto-gensym (which can be a pain to get around if you want to return the original name without obfuscation).
  • unquote or ~ is $ in julia inside expressions. It should evaluate the variable that's given to the macro and use the evaluated value.
  • unquote-splice or ~@ unquotes, and also expands the form by one layer into the form that's being returned. Ie, (f ~@x) is the same as :(f($x...)) in julia.

Download Details:

Author: VShesh
Source Code: https://github.com/vshesh/Sexpr.jl 

#julia  #clojure 

What is GEEK

Buddha Community

Sexpr.jl: Julia <3 Clojure + Macroexpansion
Veronica  Roob

Veronica Roob

1653475560

A Pure PHP Implementation Of The MessagePack Serialization Format

msgpack.php

A pure PHP implementation of the MessagePack serialization format.

Features

Installation

The recommended way to install the library is through Composer:

composer require rybakit/msgpack

Usage

Packing

To pack values you can either use an instance of a Packer:

$packer = new Packer();
$packed = $packer->pack($value);

or call a static method on the MessagePack class:

$packed = MessagePack::pack($value);

In the examples above, the method pack automatically packs a value depending on its type. However, not all PHP types can be uniquely translated to MessagePack types. For example, the MessagePack format defines map and array types, which are represented by a single array type in PHP. By default, the packer will pack a PHP array as a MessagePack array if it has sequential numeric keys, starting from 0 and as a MessagePack map otherwise:

$mpArr1 = $packer->pack([1, 2]);               // MP array [1, 2]
$mpArr2 = $packer->pack([0 => 1, 1 => 2]);     // MP array [1, 2]
$mpMap1 = $packer->pack([0 => 1, 2 => 3]);     // MP map {0: 1, 2: 3}
$mpMap2 = $packer->pack([1 => 2, 2 => 3]);     // MP map {1: 2, 2: 3}
$mpMap3 = $packer->pack(['a' => 1, 'b' => 2]); // MP map {a: 1, b: 2}

However, sometimes you need to pack a sequential array as a MessagePack map. To do this, use the packMap method:

$mpMap = $packer->packMap([1, 2]); // {0: 1, 1: 2}

Here is a list of type-specific packing methods:

$packer->packNil();           // MP nil
$packer->packBool(true);      // MP bool
$packer->packInt(42);         // MP int
$packer->packFloat(M_PI);     // MP float (32 or 64)
$packer->packFloat32(M_PI);   // MP float 32
$packer->packFloat64(M_PI);   // MP float 64
$packer->packStr('foo');      // MP str
$packer->packBin("\x80");     // MP bin
$packer->packArray([1, 2]);   // MP array
$packer->packMap(['a' => 1]); // MP map
$packer->packExt(1, "\xaa");  // MP ext

Check the "Custom types" section below on how to pack custom types.

Packing options

The Packer object supports a number of bitmask-based options for fine-tuning the packing process (defaults are in bold):

NameDescription
FORCE_STRForces PHP strings to be packed as MessagePack UTF-8 strings
FORCE_BINForces PHP strings to be packed as MessagePack binary data
DETECT_STR_BINDetects MessagePack str/bin type automatically
  
FORCE_ARRForces PHP arrays to be packed as MessagePack arrays
FORCE_MAPForces PHP arrays to be packed as MessagePack maps
DETECT_ARR_MAPDetects MessagePack array/map type automatically
  
FORCE_FLOAT32Forces PHP floats to be packed as 32-bits MessagePack floats
FORCE_FLOAT64Forces PHP floats to be packed as 64-bits MessagePack floats

The type detection mode (DETECT_STR_BIN/DETECT_ARR_MAP) adds some overhead which can be noticed when you pack large (16- and 32-bit) arrays or strings. However, if you know the value type in advance (for example, you only work with UTF-8 strings or/and associative arrays), you can eliminate this overhead by forcing the packer to use the appropriate type, which will save it from running the auto-detection routine. Another option is to explicitly specify the value type. The library provides 2 auxiliary classes for this, Map and Bin. Check the "Custom types" section below for details.

Examples:

// detect str/bin type and pack PHP 64-bit floats (doubles) to MP 32-bit floats
$packer = new Packer(PackOptions::DETECT_STR_BIN | PackOptions::FORCE_FLOAT32);

// these will throw MessagePack\Exception\InvalidOptionException
$packer = new Packer(PackOptions::FORCE_STR | PackOptions::FORCE_BIN);
$packer = new Packer(PackOptions::FORCE_FLOAT32 | PackOptions::FORCE_FLOAT64);

Unpacking

To unpack data you can either use an instance of a BufferUnpacker:

$unpacker = new BufferUnpacker();

$unpacker->reset($packed);
$value = $unpacker->unpack();

or call a static method on the MessagePack class:

$value = MessagePack::unpack($packed);

If the packed data is received in chunks (e.g. when reading from a stream), use the tryUnpack method, which attempts to unpack data and returns an array of unpacked messages (if any) instead of throwing an InsufficientDataException:

while ($chunk = ...) {
    $unpacker->append($chunk);
    if ($messages = $unpacker->tryUnpack()) {
        return $messages;
    }
}

If you want to unpack from a specific position in a buffer, use seek:

$unpacker->seek(42); // set position equal to 42 bytes
$unpacker->seek(-8); // set position to 8 bytes before the end of the buffer

To skip bytes from the current position, use skip:

$unpacker->skip(10); // set position to 10 bytes ahead of the current position

To get the number of remaining (unread) bytes in the buffer:

$unreadBytesCount = $unpacker->getRemainingCount();

To check whether the buffer has unread data:

$hasUnreadBytes = $unpacker->hasRemaining();

If needed, you can remove already read data from the buffer by calling:

$releasedBytesCount = $unpacker->release();

With the read method you can read raw (packed) data:

$packedData = $unpacker->read(2); // read 2 bytes

Besides the above methods BufferUnpacker provides type-specific unpacking methods, namely:

$unpacker->unpackNil();   // PHP null
$unpacker->unpackBool();  // PHP bool
$unpacker->unpackInt();   // PHP int
$unpacker->unpackFloat(); // PHP float
$unpacker->unpackStr();   // PHP UTF-8 string
$unpacker->unpackBin();   // PHP binary string
$unpacker->unpackArray(); // PHP sequential array
$unpacker->unpackMap();   // PHP associative array
$unpacker->unpackExt();   // PHP MessagePack\Type\Ext object

Unpacking options

The BufferUnpacker object supports a number of bitmask-based options for fine-tuning the unpacking process (defaults are in bold):

NameDescription
BIGINT_AS_STRConverts overflowed integers to strings [1]
BIGINT_AS_GMPConverts overflowed integers to GMP objects [2]
BIGINT_AS_DECConverts overflowed integers to Decimal\Decimal objects [3]

1. The binary MessagePack format has unsigned 64-bit as its largest integer data type, but PHP does not support such integers, which means that an overflow can occur during unpacking.

2. Make sure the GMP extension is enabled.

3. Make sure the Decimal extension is enabled.

Examples:

$packedUint64 = "\xcf"."\xff\xff\xff\xff"."\xff\xff\xff\xff";

$unpacker = new BufferUnpacker($packedUint64);
var_dump($unpacker->unpack()); // string(20) "18446744073709551615"

$unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_GMP);
var_dump($unpacker->unpack()); // object(GMP) {...}

$unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_DEC);
var_dump($unpacker->unpack()); // object(Decimal\Decimal) {...}

Custom types

In addition to the basic types, the library provides functionality to serialize and deserialize arbitrary types. This can be done in several ways, depending on your use case. Let's take a look at them.

Type objects

If you need to serialize an instance of one of your classes into one of the basic MessagePack types, the best way to do this is to implement the CanBePacked interface in the class. A good example of such a class is the Map type class that comes with the library. This type is useful when you want to explicitly specify that a given PHP array should be packed as a MessagePack map without triggering an automatic type detection routine:

$packer = new Packer();

$packedMap = $packer->pack(new Map([1, 2, 3]));
$packedArray = $packer->pack([1, 2, 3]);

More type examples can be found in the src/Type directory.

Type transformers

As with type objects, type transformers are only responsible for serializing values. They should be used when you need to serialize a value that does not implement the CanBePacked interface. Examples of such values could be instances of built-in or third-party classes that you don't own, or non-objects such as resources.

A transformer class must implement the CanPack interface. To use a transformer, it must first be registered in the packer. Here is an example of how to serialize PHP streams into the MessagePack bin format type using one of the supplied transformers, StreamTransformer:

$packer = new Packer(null, [new StreamTransformer()]);

$packedBin = $packer->pack(fopen('/path/to/file', 'r+'));

More type transformer examples can be found in the src/TypeTransformer directory.

Extensions

In contrast to the cases described above, extensions are intended to handle extension types and are responsible for both serialization and deserialization of values (types).

An extension class must implement the Extension interface. To use an extension, it must first be registered in the packer and the unpacker.

The MessagePack specification divides extension types into two groups: predefined and application-specific. Currently, there is only one predefined type in the specification, Timestamp.

Timestamp

The Timestamp extension type is a predefined type. Support for this type in the library is done through the TimestampExtension class. This class is responsible for handling Timestamp objects, which represent the number of seconds and optional adjustment in nanoseconds:

$timestampExtension = new TimestampExtension();

$packer = new Packer();
$packer = $packer->extendWith($timestampExtension);

$unpacker = new BufferUnpacker();
$unpacker = $unpacker->extendWith($timestampExtension);

$packedTimestamp = $packer->pack(Timestamp::now());
$timestamp = $unpacker->reset($packedTimestamp)->unpack();

$seconds = $timestamp->getSeconds();
$nanoseconds = $timestamp->getNanoseconds();

When using the MessagePack class, the Timestamp extension is already registered:

$packedTimestamp = MessagePack::pack(Timestamp::now());
$timestamp = MessagePack::unpack($packedTimestamp);

Application-specific extensions

In addition, the format can be extended with your own types. For example, to make the built-in PHP DateTime objects first-class citizens in your code, you can create a corresponding extension, as shown in the example. Please note, that custom extensions have to be registered with a unique extension ID (an integer from 0 to 127).

More extension examples can be found in the examples/MessagePack directory.

To learn more about how extension types can be useful, check out this article.

Exceptions

If an error occurs during packing/unpacking, a PackingFailedException or an UnpackingFailedException will be thrown, respectively. In addition, an InsufficientDataException can be thrown during unpacking.

An InvalidOptionException will be thrown in case an invalid option (or a combination of mutually exclusive options) is used.

Tests

Run tests as follows:

vendor/bin/phpunit

Also, if you already have Docker installed, you can run the tests in a docker container. First, create a container:

./dockerfile.sh | docker build -t msgpack -

The command above will create a container named msgpack with PHP 8.1 runtime. You may change the default runtime by defining the PHP_IMAGE environment variable:

PHP_IMAGE='php:8.0-cli' ./dockerfile.sh | docker build -t msgpack -

See a list of various images here.

Then run the unit tests:

docker run --rm -v $PWD:/msgpack -w /msgpack msgpack

Fuzzing

To ensure that the unpacking works correctly with malformed/semi-malformed data, you can use a testing technique called Fuzzing. The library ships with a help file (target) for PHP-Fuzzer and can be used as follows:

php-fuzzer fuzz tests/fuzz_buffer_unpacker.php

Performance

To check performance, run:

php -n -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

=============================================
Test/Target            Packer  BufferUnpacker
---------------------------------------------
nil .................. 0.0030 ........ 0.0139
false ................ 0.0037 ........ 0.0144
true ................. 0.0040 ........ 0.0137
7-bit uint #1 ........ 0.0052 ........ 0.0120
7-bit uint #2 ........ 0.0059 ........ 0.0114
7-bit uint #3 ........ 0.0061 ........ 0.0119
5-bit sint #1 ........ 0.0067 ........ 0.0126
5-bit sint #2 ........ 0.0064 ........ 0.0132
5-bit sint #3 ........ 0.0066 ........ 0.0135
8-bit uint #1 ........ 0.0078 ........ 0.0200
8-bit uint #2 ........ 0.0077 ........ 0.0212
8-bit uint #3 ........ 0.0086 ........ 0.0203
16-bit uint #1 ....... 0.0111 ........ 0.0271
16-bit uint #2 ....... 0.0115 ........ 0.0260
16-bit uint #3 ....... 0.0103 ........ 0.0273
32-bit uint #1 ....... 0.0116 ........ 0.0326
32-bit uint #2 ....... 0.0118 ........ 0.0332
32-bit uint #3 ....... 0.0127 ........ 0.0325
64-bit uint #1 ....... 0.0140 ........ 0.0277
64-bit uint #2 ....... 0.0134 ........ 0.0294
64-bit uint #3 ....... 0.0134 ........ 0.0281
8-bit int #1 ......... 0.0086 ........ 0.0241
8-bit int #2 ......... 0.0089 ........ 0.0225
8-bit int #3 ......... 0.0085 ........ 0.0229
16-bit int #1 ........ 0.0118 ........ 0.0280
16-bit int #2 ........ 0.0121 ........ 0.0270
16-bit int #3 ........ 0.0109 ........ 0.0274
32-bit int #1 ........ 0.0128 ........ 0.0346
32-bit int #2 ........ 0.0118 ........ 0.0339
32-bit int #3 ........ 0.0135 ........ 0.0368
64-bit int #1 ........ 0.0138 ........ 0.0276
64-bit int #2 ........ 0.0132 ........ 0.0286
64-bit int #3 ........ 0.0137 ........ 0.0274
64-bit int #4 ........ 0.0180 ........ 0.0285
64-bit float #1 ...... 0.0134 ........ 0.0284
64-bit float #2 ...... 0.0125 ........ 0.0275
64-bit float #3 ...... 0.0126 ........ 0.0283
fix string #1 ........ 0.0035 ........ 0.0133
fix string #2 ........ 0.0094 ........ 0.0216
fix string #3 ........ 0.0094 ........ 0.0222
fix string #4 ........ 0.0091 ........ 0.0241
8-bit string #1 ...... 0.0122 ........ 0.0301
8-bit string #2 ...... 0.0118 ........ 0.0304
8-bit string #3 ...... 0.0119 ........ 0.0315
16-bit string #1 ..... 0.0150 ........ 0.0388
16-bit string #2 ..... 0.1545 ........ 0.1665
32-bit string ........ 0.1570 ........ 0.1756
wide char string #1 .. 0.0091 ........ 0.0236
wide char string #2 .. 0.0122 ........ 0.0313
8-bit binary #1 ...... 0.0100 ........ 0.0302
8-bit binary #2 ...... 0.0123 ........ 0.0324
8-bit binary #3 ...... 0.0126 ........ 0.0327
16-bit binary ........ 0.0168 ........ 0.0372
32-bit binary ........ 0.1588 ........ 0.1754
fix array #1 ......... 0.0042 ........ 0.0131
fix array #2 ......... 0.0294 ........ 0.0367
fix array #3 ......... 0.0412 ........ 0.0472
16-bit array #1 ...... 0.1378 ........ 0.1596
16-bit array #2 ........... S ............. S
32-bit array .............. S ............. S
complex array ........ 0.1865 ........ 0.2283
fix map #1 ........... 0.0725 ........ 0.1048
fix map #2 ........... 0.0319 ........ 0.0405
fix map #3 ........... 0.0356 ........ 0.0665
fix map #4 ........... 0.0465 ........ 0.0497
16-bit map #1 ........ 0.2540 ........ 0.3028
16-bit map #2 ............. S ............. S
32-bit map ................ S ............. S
complex map .......... 0.2372 ........ 0.2710
fixext 1 ............. 0.0283 ........ 0.0358
fixext 2 ............. 0.0291 ........ 0.0371
fixext 4 ............. 0.0302 ........ 0.0355
fixext 8 ............. 0.0288 ........ 0.0384
fixext 16 ............ 0.0293 ........ 0.0359
8-bit ext ............ 0.0302 ........ 0.0439
16-bit ext ........... 0.0334 ........ 0.0499
32-bit ext ........... 0.1845 ........ 0.1888
32-bit timestamp #1 .. 0.0337 ........ 0.0547
32-bit timestamp #2 .. 0.0335 ........ 0.0560
64-bit timestamp #1 .. 0.0371 ........ 0.0575
64-bit timestamp #2 .. 0.0374 ........ 0.0542
64-bit timestamp #3 .. 0.0356 ........ 0.0533
96-bit timestamp #1 .. 0.0362 ........ 0.0699
96-bit timestamp #2 .. 0.0381 ........ 0.0701
96-bit timestamp #3 .. 0.0367 ........ 0.0687
=============================================
Total                  2.7618          4.0820
Skipped                     4               4
Failed                      0               0
Ignored                     0               0

With JIT:

php -n -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.jit_buffer_size=64M -dopcache.jit=tracing -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

=============================================
Test/Target            Packer  BufferUnpacker
---------------------------------------------
nil .................. 0.0005 ........ 0.0054
false ................ 0.0004 ........ 0.0059
true ................. 0.0004 ........ 0.0059
7-bit uint #1 ........ 0.0010 ........ 0.0047
7-bit uint #2 ........ 0.0010 ........ 0.0046
7-bit uint #3 ........ 0.0010 ........ 0.0046
5-bit sint #1 ........ 0.0025 ........ 0.0046
5-bit sint #2 ........ 0.0023 ........ 0.0046
5-bit sint #3 ........ 0.0024 ........ 0.0045
8-bit uint #1 ........ 0.0043 ........ 0.0081
8-bit uint #2 ........ 0.0043 ........ 0.0079
8-bit uint #3 ........ 0.0041 ........ 0.0080
16-bit uint #1 ....... 0.0064 ........ 0.0095
16-bit uint #2 ....... 0.0064 ........ 0.0091
16-bit uint #3 ....... 0.0064 ........ 0.0094
32-bit uint #1 ....... 0.0085 ........ 0.0114
32-bit uint #2 ....... 0.0077 ........ 0.0122
32-bit uint #3 ....... 0.0077 ........ 0.0120
64-bit uint #1 ....... 0.0085 ........ 0.0159
64-bit uint #2 ....... 0.0086 ........ 0.0157
64-bit uint #3 ....... 0.0086 ........ 0.0158
8-bit int #1 ......... 0.0042 ........ 0.0080
8-bit int #2 ......... 0.0042 ........ 0.0080
8-bit int #3 ......... 0.0042 ........ 0.0081
16-bit int #1 ........ 0.0065 ........ 0.0095
16-bit int #2 ........ 0.0065 ........ 0.0090
16-bit int #3 ........ 0.0056 ........ 0.0085
32-bit int #1 ........ 0.0067 ........ 0.0107
32-bit int #2 ........ 0.0066 ........ 0.0106
32-bit int #3 ........ 0.0063 ........ 0.0104
64-bit int #1 ........ 0.0072 ........ 0.0162
64-bit int #2 ........ 0.0073 ........ 0.0174
64-bit int #3 ........ 0.0072 ........ 0.0164
64-bit int #4 ........ 0.0077 ........ 0.0161
64-bit float #1 ...... 0.0053 ........ 0.0135
64-bit float #2 ...... 0.0053 ........ 0.0135
64-bit float #3 ...... 0.0052 ........ 0.0135
fix string #1 ....... -0.0002 ........ 0.0044
fix string #2 ........ 0.0035 ........ 0.0067
fix string #3 ........ 0.0035 ........ 0.0077
fix string #4 ........ 0.0033 ........ 0.0078
8-bit string #1 ...... 0.0059 ........ 0.0110
8-bit string #2 ...... 0.0063 ........ 0.0121
8-bit string #3 ...... 0.0064 ........ 0.0124
16-bit string #1 ..... 0.0099 ........ 0.0146
16-bit string #2 ..... 0.1522 ........ 0.1474
32-bit string ........ 0.1511 ........ 0.1483
wide char string #1 .. 0.0039 ........ 0.0084
wide char string #2 .. 0.0073 ........ 0.0123
8-bit binary #1 ...... 0.0040 ........ 0.0112
8-bit binary #2 ...... 0.0075 ........ 0.0123
8-bit binary #3 ...... 0.0077 ........ 0.0129
16-bit binary ........ 0.0096 ........ 0.0145
32-bit binary ........ 0.1535 ........ 0.1479
fix array #1 ......... 0.0008 ........ 0.0061
fix array #2 ......... 0.0121 ........ 0.0165
fix array #3 ......... 0.0193 ........ 0.0222
16-bit array #1 ...... 0.0607 ........ 0.0479
16-bit array #2 ........... S ............. S
32-bit array .............. S ............. S
complex array ........ 0.0749 ........ 0.0824
fix map #1 ........... 0.0329 ........ 0.0431
fix map #2 ........... 0.0161 ........ 0.0189
fix map #3 ........... 0.0205 ........ 0.0262
fix map #4 ........... 0.0252 ........ 0.0205
16-bit map #1 ........ 0.1016 ........ 0.0927
16-bit map #2 ............. S ............. S
32-bit map ................ S ............. S
complex map .......... 0.1096 ........ 0.1030
fixext 1 ............. 0.0157 ........ 0.0161
fixext 2 ............. 0.0175 ........ 0.0183
fixext 4 ............. 0.0156 ........ 0.0185
fixext 8 ............. 0.0163 ........ 0.0184
fixext 16 ............ 0.0164 ........ 0.0182
8-bit ext ............ 0.0158 ........ 0.0207
16-bit ext ........... 0.0203 ........ 0.0219
32-bit ext ........... 0.1614 ........ 0.1539
32-bit timestamp #1 .. 0.0195 ........ 0.0249
32-bit timestamp #2 .. 0.0188 ........ 0.0260
64-bit timestamp #1 .. 0.0207 ........ 0.0281
64-bit timestamp #2 .. 0.0212 ........ 0.0291
64-bit timestamp #3 .. 0.0207 ........ 0.0295
96-bit timestamp #1 .. 0.0222 ........ 0.0358
96-bit timestamp #2 .. 0.0228 ........ 0.0353
96-bit timestamp #3 .. 0.0210 ........ 0.0319
=============================================
Total                  1.6432          1.9674
Skipped                     4               4
Failed                      0               0
Ignored                     0               0

You may change default benchmark settings by defining the following environment variables:

NameDefault
MP_BENCH_TARGETSpure_p,pure_u, see a list of available targets
MP_BENCH_ITERATIONS100_000
MP_BENCH_DURATIONnot set
MP_BENCH_ROUNDS3
MP_BENCH_TESTS-@slow, see a list of available tests

For example:

export MP_BENCH_TARGETS=pure_p
export MP_BENCH_ITERATIONS=1000000
export MP_BENCH_ROUNDS=5
# a comma separated list of test names
export MP_BENCH_TESTS='complex array, complex map'
# or a group name
# export MP_BENCH_TESTS='-@slow' // @pecl_comp
# or a regexp
# export MP_BENCH_TESTS='/complex (array|map)/'

Another example, benchmarking both the library and the PECL extension:

MP_BENCH_TARGETS=pure_p,pure_u,pecl_p,pecl_u \
php -n -dextension=msgpack.so -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

===========================================================================
Test/Target            Packer  BufferUnpacker  msgpack_pack  msgpack_unpack
---------------------------------------------------------------------------
nil .................. 0.0031 ........ 0.0141 ...... 0.0055 ........ 0.0064
false ................ 0.0039 ........ 0.0154 ...... 0.0056 ........ 0.0053
true ................. 0.0038 ........ 0.0139 ...... 0.0056 ........ 0.0044
7-bit uint #1 ........ 0.0061 ........ 0.0110 ...... 0.0059 ........ 0.0046
7-bit uint #2 ........ 0.0065 ........ 0.0119 ...... 0.0042 ........ 0.0029
7-bit uint #3 ........ 0.0054 ........ 0.0117 ...... 0.0045 ........ 0.0025
5-bit sint #1 ........ 0.0047 ........ 0.0103 ...... 0.0038 ........ 0.0022
5-bit sint #2 ........ 0.0048 ........ 0.0117 ...... 0.0038 ........ 0.0022
5-bit sint #3 ........ 0.0046 ........ 0.0102 ...... 0.0038 ........ 0.0023
8-bit uint #1 ........ 0.0063 ........ 0.0174 ...... 0.0039 ........ 0.0031
8-bit uint #2 ........ 0.0063 ........ 0.0167 ...... 0.0040 ........ 0.0029
8-bit uint #3 ........ 0.0063 ........ 0.0168 ...... 0.0039 ........ 0.0030
16-bit uint #1 ....... 0.0092 ........ 0.0222 ...... 0.0049 ........ 0.0030
16-bit uint #2 ....... 0.0096 ........ 0.0227 ...... 0.0042 ........ 0.0046
16-bit uint #3 ....... 0.0123 ........ 0.0274 ...... 0.0059 ........ 0.0051
32-bit uint #1 ....... 0.0136 ........ 0.0331 ...... 0.0060 ........ 0.0048
32-bit uint #2 ....... 0.0130 ........ 0.0336 ...... 0.0070 ........ 0.0048
32-bit uint #3 ....... 0.0127 ........ 0.0329 ...... 0.0051 ........ 0.0048
64-bit uint #1 ....... 0.0126 ........ 0.0268 ...... 0.0055 ........ 0.0049
64-bit uint #2 ....... 0.0135 ........ 0.0281 ...... 0.0052 ........ 0.0046
64-bit uint #3 ....... 0.0131 ........ 0.0274 ...... 0.0069 ........ 0.0044
8-bit int #1 ......... 0.0077 ........ 0.0236 ...... 0.0058 ........ 0.0044
8-bit int #2 ......... 0.0087 ........ 0.0244 ...... 0.0058 ........ 0.0048
8-bit int #3 ......... 0.0084 ........ 0.0241 ...... 0.0055 ........ 0.0049
16-bit int #1 ........ 0.0112 ........ 0.0271 ...... 0.0048 ........ 0.0045
16-bit int #2 ........ 0.0124 ........ 0.0292 ...... 0.0057 ........ 0.0049
16-bit int #3 ........ 0.0118 ........ 0.0270 ...... 0.0058 ........ 0.0050
32-bit int #1 ........ 0.0137 ........ 0.0366 ...... 0.0058 ........ 0.0051
32-bit int #2 ........ 0.0133 ........ 0.0366 ...... 0.0056 ........ 0.0049
32-bit int #3 ........ 0.0129 ........ 0.0350 ...... 0.0052 ........ 0.0048
64-bit int #1 ........ 0.0145 ........ 0.0254 ...... 0.0034 ........ 0.0025
64-bit int #2 ........ 0.0097 ........ 0.0214 ...... 0.0034 ........ 0.0025
64-bit int #3 ........ 0.0096 ........ 0.0287 ...... 0.0059 ........ 0.0050
64-bit int #4 ........ 0.0143 ........ 0.0277 ...... 0.0059 ........ 0.0046
64-bit float #1 ...... 0.0134 ........ 0.0281 ...... 0.0057 ........ 0.0052
64-bit float #2 ...... 0.0141 ........ 0.0281 ...... 0.0057 ........ 0.0050
64-bit float #3 ...... 0.0144 ........ 0.0282 ...... 0.0057 ........ 0.0050
fix string #1 ........ 0.0036 ........ 0.0143 ...... 0.0066 ........ 0.0053
fix string #2 ........ 0.0107 ........ 0.0222 ...... 0.0065 ........ 0.0068
fix string #3 ........ 0.0116 ........ 0.0245 ...... 0.0063 ........ 0.0069
fix string #4 ........ 0.0105 ........ 0.0253 ...... 0.0083 ........ 0.0077
8-bit string #1 ...... 0.0126 ........ 0.0318 ...... 0.0075 ........ 0.0088
8-bit string #2 ...... 0.0121 ........ 0.0295 ...... 0.0076 ........ 0.0086
8-bit string #3 ...... 0.0125 ........ 0.0293 ...... 0.0130 ........ 0.0093
16-bit string #1 ..... 0.0159 ........ 0.0368 ...... 0.0117 ........ 0.0086
16-bit string #2 ..... 0.1547 ........ 0.1686 ...... 0.1516 ........ 0.1373
32-bit string ........ 0.1558 ........ 0.1729 ...... 0.1511 ........ 0.1396
wide char string #1 .. 0.0098 ........ 0.0237 ...... 0.0066 ........ 0.0065
wide char string #2 .. 0.0128 ........ 0.0291 ...... 0.0061 ........ 0.0082
8-bit binary #1 ........... I ............. I ........... F ............. I
8-bit binary #2 ........... I ............. I ........... F ............. I
8-bit binary #3 ........... I ............. I ........... F ............. I
16-bit binary ............. I ............. I ........... F ............. I
32-bit binary ............. I ............. I ........... F ............. I
fix array #1 ......... 0.0040 ........ 0.0129 ...... 0.0120 ........ 0.0058
fix array #2 ......... 0.0279 ........ 0.0390 ...... 0.0143 ........ 0.0165
fix array #3 ......... 0.0415 ........ 0.0463 ...... 0.0162 ........ 0.0187
16-bit array #1 ...... 0.1349 ........ 0.1628 ...... 0.0334 ........ 0.0341
16-bit array #2 ........... S ............. S ........... S ............. S
32-bit array .............. S ............. S ........... S ............. S
complex array ............. I ............. I ........... F ............. F
fix map #1 ................ I ............. I ........... F ............. I
fix map #2 ........... 0.0345 ........ 0.0391 ...... 0.0143 ........ 0.0168
fix map #3 ................ I ............. I ........... F ............. I
fix map #4 ........... 0.0459 ........ 0.0473 ...... 0.0151 ........ 0.0163
16-bit map #1 ........ 0.2518 ........ 0.2962 ...... 0.0400 ........ 0.0490
16-bit map #2 ............. S ............. S ........... S ............. S
32-bit map ................ S ............. S ........... S ............. S
complex map .......... 0.2380 ........ 0.2682 ...... 0.0545 ........ 0.0579
fixext 1 .................. I ............. I ........... F ............. F
fixext 2 .................. I ............. I ........... F ............. F
fixext 4 .................. I ............. I ........... F ............. F
fixext 8 .................. I ............. I ........... F ............. F
fixext 16 ................. I ............. I ........... F ............. F
8-bit ext ................. I ............. I ........... F ............. F
16-bit ext ................ I ............. I ........... F ............. F
32-bit ext ................ I ............. I ........... F ............. F
32-bit timestamp #1 ....... I ............. I ........... F ............. F
32-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #1 ....... I ............. I ........... F ............. F
64-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #3 ....... I ............. I ........... F ............. F
96-bit timestamp #1 ....... I ............. I ........... F ............. F
96-bit timestamp #2 ....... I ............. I ........... F ............. F
96-bit timestamp #3 ....... I ............. I ........... F ............. F
===========================================================================
Total                  1.5625          2.3866        0.7735          0.7243
Skipped                     4               4             4               4
Failed                      0               0            24              17
Ignored                    24              24             0               7

With JIT:

MP_BENCH_TARGETS=pure_p,pure_u,pecl_p,pecl_u \
php -n -dextension=msgpack.so -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.jit_buffer_size=64M -dopcache.jit=tracing -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

===========================================================================
Test/Target            Packer  BufferUnpacker  msgpack_pack  msgpack_unpack
---------------------------------------------------------------------------
nil .................. 0.0001 ........ 0.0052 ...... 0.0053 ........ 0.0042
false ................ 0.0007 ........ 0.0060 ...... 0.0057 ........ 0.0043
true ................. 0.0008 ........ 0.0060 ...... 0.0056 ........ 0.0041
7-bit uint #1 ........ 0.0031 ........ 0.0046 ...... 0.0062 ........ 0.0041
7-bit uint #2 ........ 0.0021 ........ 0.0043 ...... 0.0062 ........ 0.0041
7-bit uint #3 ........ 0.0022 ........ 0.0044 ...... 0.0061 ........ 0.0040
5-bit sint #1 ........ 0.0030 ........ 0.0048 ...... 0.0062 ........ 0.0040
5-bit sint #2 ........ 0.0032 ........ 0.0046 ...... 0.0062 ........ 0.0040
5-bit sint #3 ........ 0.0031 ........ 0.0046 ...... 0.0062 ........ 0.0040
8-bit uint #1 ........ 0.0054 ........ 0.0079 ...... 0.0062 ........ 0.0050
8-bit uint #2 ........ 0.0051 ........ 0.0079 ...... 0.0064 ........ 0.0044
8-bit uint #3 ........ 0.0051 ........ 0.0082 ...... 0.0062 ........ 0.0044
16-bit uint #1 ....... 0.0077 ........ 0.0094 ...... 0.0065 ........ 0.0045
16-bit uint #2 ....... 0.0077 ........ 0.0094 ...... 0.0063 ........ 0.0045
16-bit uint #3 ....... 0.0077 ........ 0.0095 ...... 0.0064 ........ 0.0047
32-bit uint #1 ....... 0.0088 ........ 0.0119 ...... 0.0063 ........ 0.0043
32-bit uint #2 ....... 0.0089 ........ 0.0117 ...... 0.0062 ........ 0.0039
32-bit uint #3 ....... 0.0089 ........ 0.0118 ...... 0.0063 ........ 0.0044
64-bit uint #1 ....... 0.0097 ........ 0.0155 ...... 0.0063 ........ 0.0045
64-bit uint #2 ....... 0.0095 ........ 0.0153 ...... 0.0061 ........ 0.0045
64-bit uint #3 ....... 0.0096 ........ 0.0156 ...... 0.0063 ........ 0.0047
8-bit int #1 ......... 0.0053 ........ 0.0083 ...... 0.0062 ........ 0.0044
8-bit int #2 ......... 0.0052 ........ 0.0080 ...... 0.0062 ........ 0.0044
8-bit int #3 ......... 0.0052 ........ 0.0080 ...... 0.0062 ........ 0.0043
16-bit int #1 ........ 0.0089 ........ 0.0097 ...... 0.0069 ........ 0.0046
16-bit int #2 ........ 0.0075 ........ 0.0093 ...... 0.0063 ........ 0.0043
16-bit int #3 ........ 0.0075 ........ 0.0094 ...... 0.0062 ........ 0.0046
32-bit int #1 ........ 0.0086 ........ 0.0122 ...... 0.0063 ........ 0.0044
32-bit int #2 ........ 0.0087 ........ 0.0120 ...... 0.0066 ........ 0.0046
32-bit int #3 ........ 0.0086 ........ 0.0121 ...... 0.0060 ........ 0.0044
64-bit int #1 ........ 0.0096 ........ 0.0149 ...... 0.0060 ........ 0.0045
64-bit int #2 ........ 0.0096 ........ 0.0157 ...... 0.0062 ........ 0.0044
64-bit int #3 ........ 0.0096 ........ 0.0160 ...... 0.0063 ........ 0.0046
64-bit int #4 ........ 0.0097 ........ 0.0157 ...... 0.0061 ........ 0.0044
64-bit float #1 ...... 0.0079 ........ 0.0153 ...... 0.0056 ........ 0.0044
64-bit float #2 ...... 0.0079 ........ 0.0152 ...... 0.0057 ........ 0.0045
64-bit float #3 ...... 0.0079 ........ 0.0155 ...... 0.0057 ........ 0.0044
fix string #1 ........ 0.0010 ........ 0.0045 ...... 0.0071 ........ 0.0044
fix string #2 ........ 0.0048 ........ 0.0075 ...... 0.0070 ........ 0.0060
fix string #3 ........ 0.0048 ........ 0.0086 ...... 0.0068 ........ 0.0060
fix string #4 ........ 0.0050 ........ 0.0088 ...... 0.0070 ........ 0.0059
8-bit string #1 ...... 0.0081 ........ 0.0129 ...... 0.0069 ........ 0.0062
8-bit string #2 ...... 0.0086 ........ 0.0128 ...... 0.0069 ........ 0.0065
8-bit string #3 ...... 0.0086 ........ 0.0126 ...... 0.0115 ........ 0.0065
16-bit string #1 ..... 0.0105 ........ 0.0137 ...... 0.0128 ........ 0.0068
16-bit string #2 ..... 0.1510 ........ 0.1486 ...... 0.1526 ........ 0.1391
32-bit string ........ 0.1517 ........ 0.1475 ...... 0.1504 ........ 0.1370
wide char string #1 .. 0.0044 ........ 0.0085 ...... 0.0067 ........ 0.0057
wide char string #2 .. 0.0081 ........ 0.0125 ...... 0.0069 ........ 0.0063
8-bit binary #1 ........... I ............. I ........... F ............. I
8-bit binary #2 ........... I ............. I ........... F ............. I
8-bit binary #3 ........... I ............. I ........... F ............. I
16-bit binary ............. I ............. I ........... F ............. I
32-bit binary ............. I ............. I ........... F ............. I
fix array #1 ......... 0.0014 ........ 0.0059 ...... 0.0132 ........ 0.0055
fix array #2 ......... 0.0146 ........ 0.0156 ...... 0.0155 ........ 0.0148
fix array #3 ......... 0.0211 ........ 0.0229 ...... 0.0179 ........ 0.0180
16-bit array #1 ...... 0.0673 ........ 0.0498 ...... 0.0343 ........ 0.0388
16-bit array #2 ........... S ............. S ........... S ............. S
32-bit array .............. S ............. S ........... S ............. S
complex array ............. I ............. I ........... F ............. F
fix map #1 ................ I ............. I ........... F ............. I
fix map #2 ........... 0.0148 ........ 0.0180 ...... 0.0156 ........ 0.0179
fix map #3 ................ I ............. I ........... F ............. I
fix map #4 ........... 0.0252 ........ 0.0201 ...... 0.0214 ........ 0.0167
16-bit map #1 ........ 0.1027 ........ 0.0836 ...... 0.0388 ........ 0.0510
16-bit map #2 ............. S ............. S ........... S ............. S
32-bit map ................ S ............. S ........... S ............. S
complex map .......... 0.1104 ........ 0.1010 ...... 0.0556 ........ 0.0602
fixext 1 .................. I ............. I ........... F ............. F
fixext 2 .................. I ............. I ........... F ............. F
fixext 4 .................. I ............. I ........... F ............. F
fixext 8 .................. I ............. I ........... F ............. F
fixext 16 ................. I ............. I ........... F ............. F
8-bit ext ................. I ............. I ........... F ............. F
16-bit ext ................ I ............. I ........... F ............. F
32-bit ext ................ I ............. I ........... F ............. F
32-bit timestamp #1 ....... I ............. I ........... F ............. F
32-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #1 ....... I ............. I ........... F ............. F
64-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #3 ....... I ............. I ........... F ............. F
96-bit timestamp #1 ....... I ............. I ........... F ............. F
96-bit timestamp #2 ....... I ............. I ........... F ............. F
96-bit timestamp #3 ....... I ............. I ........... F ............. F
===========================================================================
Total                  0.9642          1.0909        0.8224          0.7213
Skipped                     4               4             4               4
Failed                      0               0            24              17
Ignored                    24              24             0               7

Note that the msgpack extension (v2.1.2) doesn't support ext, bin and UTF-8 str types.

License

The library is released under the MIT License. See the bundled LICENSE file for details.

Author: rybakit
Source Code: https://github.com/rybakit/msgpack.php
License: MIT License

#php 

A Wrapper for Sembast and SQFlite to Enable Easy

FHIR_DB

This is really just a wrapper around Sembast_SQFLite - so all of the heavy lifting was done by Alex Tekartik. I highly recommend that if you have any questions about working with this package that you take a look at Sembast. He's also just a super nice guy, and even answered a question for me when I was deciding which sembast version to use. As usual, ResoCoder also has a good tutorial.

I have an interest in low-resource settings and thus a specific reason to be able to store data offline. To encourage this use, there are a number of other packages I have created based around the data format FHIR. FHIR® is the registered trademark of HL7 and is used with the permission of HL7. Use of the FHIR trademark does not constitute endorsement of this product by HL7.

Using the Db

So, while not absolutely necessary, I highly recommend that you use some sort of interface class. This adds the benefit of more easily handling errors, plus if you change to a different database in the future, you don't have to change the rest of your app, just the interface.

I've used something like this in my projects:

class IFhirDb {
  IFhirDb();
  final ResourceDao resourceDao = ResourceDao();

  Future<Either<DbFailure, Resource>> save(Resource resource) async {
    Resource resultResource;
    try {
      resultResource = await resourceDao.save(resource);
    } catch (error) {
      return left(DbFailure.unableToSave(error: error.toString()));
    }
    return right(resultResource);
  }

  Future<Either<DbFailure, List<Resource>>> returnListOfSingleResourceType(
      String resourceType) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.getAllSortedById(resourceType: resourceType);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }

  Future<Either<DbFailure, List<Resource>>> searchFunction(
      String resourceType, String searchString, String reference) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.searchFor(resourceType, searchString, reference);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }
}

I like this because in case there's an i/o error or something, it won't crash your app. Then, you can call this interface in your app like the following:

final patient = Patient(
    resourceType: 'Patient',
    name: [HumanName(text: 'New Patient Name')],
    birthDate: Date(DateTime.now()),
);

final saveResult = await IFhirDb().save(patient);

This will save your newly created patient to the locally embedded database.

IMPORTANT: this database will expect that all previously created resources have an id. When you save a resource, it will check to see if that resource type has already been stored. (Each resource type is saved in it's own store in the database). It will then check if there is an ID. If there's no ID, it will create a new one for that resource (along with metadata on version number and creation time). It will save it, and return the resource. If it already has an ID, it will copy the the old version of the resource into a _history store. It will then update the metadata of the new resource and save that version into the appropriate store for that resource. If, for instance, we have a previously created patient:

{
    "resourceType": "Patient",
    "id": "fhirfli-294057507-6811107",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2020-10-16T19:41:28.054369Z"
    },
    "name": [
        {
            "given": ["New"],
            "family": "Patient"
        }
    ],
    "birthDate": "2020-10-16"
}

And we update the last name to 'Provider'. The above version of the patient will be kept in _history, while in the 'Patient' store in the db, we will have the updated version:

{
    "resourceType": "Patient",
    "id": "fhirfli-294057507-6811107",
    "meta": {
        "versionId": "2",
        "lastUpdated": "2020-10-16T19:45:07.316698Z"
    },
    "name": [
        {
            "given": ["New"],
            "family": "Provider"
        }
    ],
    "birthDate": "2020-10-16"
}

This way we can keep track of all previous version of all resources (which is obviously important in medicine).

For most of the interactions (saving, deleting, etc), they work the way you'd expect. The only difference is search. Because Sembast is NoSQL, we can search on any of the fields in a resource. If in our interface class, we have the following function:

  Future<Either<DbFailure, List<Resource>>> searchFunction(
      String resourceType, String searchString, String reference) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.searchFor(resourceType, searchString, reference);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }

You can search for all immunizations of a certain patient:

searchFunction(
        'Immunization', 'patient.reference', 'Patient/$patientId');

This function will search through all entries in the 'Immunization' store. It will look at all 'patient.reference' fields, and return any that match 'Patient/$patientId'.

The last thing I'll mention is that this is a password protected db, using AES-256 encryption (although it can also use Salsa20). Anytime you use the db, you have the option of using a password for encryption/decryption. Remember, if you setup the database using encryption, you will only be able to access it using that same password. When you're ready to change the password, you will need to call the update password function. If we again assume we created a change password method in our interface, it might look something like this:

class IFhirDb {
  IFhirDb();
  final ResourceDao resourceDao = ResourceDao();
  ...
    Future<Either<DbFailure, Unit>> updatePassword(String oldPassword, String newPassword) async {
    try {
      await resourceDao.updatePw(oldPassword, newPassword);
    } catch (error) {
      return left(DbFailure.unableToUpdatePassword(error: error.toString()));
    }
    return right(Unit);
  }

You don't have to use a password, and in that case, it will save the db file as plain text. If you want to add a password later, it will encrypt it at that time.

General Store

After using this for a while in an app, I've realized that it needs to be able to store data apart from just FHIR resources, at least on occasion. For this, I've added a second class for all versions of the database called GeneralDao. This is similar to the ResourceDao, but fewer options. So, in order to save something, it would look like this:

await GeneralDao().save('password', {'new':'map'});
await GeneralDao().save('password', {'new':'map'}, 'key');

The difference between these two options is that the first one will generate a key for the map being stored, while the second will store the map using the key provided. Both will return the key after successfully storing the map.

Other functions available include:

// deletes everything in the general store
await GeneralDao().deleteAllGeneral('password'); 

// delete specific entry
await GeneralDao().delete('password','key'); 

// returns map with that key
await GeneralDao().find('password', 'key'); 

FHIR® is a registered trademark of Health Level Seven International (HL7) and its use does not constitute an endorsement of products by HL7®

Use this package as a library

Depend on it

Run this command:

With Flutter:

 $ flutter pub add fhir_db

This will add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):

dependencies:
  fhir_db: ^0.4.3

Alternatively, your editor might support or flutter pub get. Check the docs for your editor to learn more.

Import it

Now in your Dart code, you can use:

import 'package:fhir_db/dstu2.dart';
import 'package:fhir_db/dstu2/fhir_db.dart';
import 'package:fhir_db/dstu2/general_dao.dart';
import 'package:fhir_db/dstu2/resource_dao.dart';
import 'package:fhir_db/encrypt/aes.dart';
import 'package:fhir_db/encrypt/salsa.dart';
import 'package:fhir_db/r4.dart';
import 'package:fhir_db/r4/fhir_db.dart';
import 'package:fhir_db/r4/general_dao.dart';
import 'package:fhir_db/r4/resource_dao.dart';
import 'package:fhir_db/r5.dart';
import 'package:fhir_db/r5/fhir_db.dart';
import 'package:fhir_db/r5/general_dao.dart';
import 'package:fhir_db/r5/resource_dao.dart';
import 'package:fhir_db/stu3.dart';
import 'package:fhir_db/stu3/fhir_db.dart';
import 'package:fhir_db/stu3/general_dao.dart';
import 'package:fhir_db/stu3/resource_dao.dart'; 

example/lib/main.dart

import 'package:fhir/r4.dart';
import 'package:fhir_db/r4.dart';
import 'package:flutter/material.dart';
import 'package:test/test.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();

  final resourceDao = ResourceDao();

  // await resourceDao.updatePw('newPw', null);
  await resourceDao.deleteAllResources(null);

  group('Playing with passwords', () {
    test('Playing with Passwords', () async {
      final patient = Patient(id: Id('1'));

      final saved = await resourceDao.save(null, patient);

      await resourceDao.updatePw(null, 'newPw');
      final search1 = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search1[0]);

      await resourceDao.updatePw('newPw', 'newerPw');
      final search2 = await resourceDao.find('newerPw',
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search2[0]);

      await resourceDao.updatePw('newerPw', null);
      final search3 = await resourceDao.find(null,
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search3[0]);

      await resourceDao.deleteAllResources(null);
    });
  });

  final id = Id('12345');
  group('Saving Things:', () {
    test('Save Patient', () async {
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);
      final patient = Patient(id: id, name: [humanName]);
      final saved = await resourceDao.save(null, patient);

      expect(saved.id, id);

      expect((saved as Patient).name?[0], humanName);
    });

    test('Save Organization', () async {
      final organization = Organization(id: id, name: 'FhirFli');
      final saved = await resourceDao.save(null, organization);

      expect(saved.id, id);

      expect((saved as Organization).name, 'FhirFli');
    });

    test('Save Observation1', () async {
      final observation1 = Observation(
        id: Id('obs1'),
        code: CodeableConcept(text: 'Observation #1'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1');
    });

    test('Save Observation1 Again', () async {
      final observation1 = Observation(
          id: Id('obs1'),
          code: CodeableConcept(text: 'Observation #1 - Updated'));
      final saved = await resourceDao.save(null, observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1 - Updated');

      expect(saved.meta?.versionId, Id('2'));
    });

    test('Save Observation2', () async {
      final observation2 = Observation(
        id: Id('obs2'),
        code: CodeableConcept(text: 'Observation #2'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation2);

      expect(saved.id, Id('obs2'));

      expect((saved as Observation).code.text, 'Observation #2');
    });

    test('Save Observation3', () async {
      final observation3 = Observation(
        id: Id('obs3'),
        code: CodeableConcept(text: 'Observation #3'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation3);

      expect(saved.id, Id('obs3'));

      expect((saved as Observation).code.text, 'Observation #3');
    });
  });

  group('Finding Things:', () {
    test('Find 1st Patient', () async {
      final search = await resourceDao.find(null,
          resourceType: R4ResourceType.Patient, id: id);
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);

      expect(search.length, 1);

      expect((search[0] as Patient).name?[0], humanName);
    });

    test('Find 3rd Observation', () async {
      final search = await resourceDao.find(null,
          resourceType: R4ResourceType.Observation, id: Id('obs3'));

      expect(search.length, 1);

      expect(search[0].id, Id('obs3'));

      expect((search[0] as Observation).code.text, 'Observation #3');
    });

    test('Find All Observations', () async {
      final search = await resourceDao.getResourceType(
        null,
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 3);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), true);

      expect(idList.contains('obs3'), true);
    });

    test('Find All (non-historical) Resources', () async {
      final search = await resourceDao.getAll(null);

      expect(search.length, 5);
      final patList = search.toList();
      final orgList = search.toList();
      final obsList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);
      obsList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Observation);

      expect(patList.length, 1);

      expect(orgList.length, 1);

      expect(obsList.length, 3);
    });
  });

  group('Deleting Things:', () {
    test('Delete 2nd Observation', () async {
      await resourceDao.delete(
          null, null, R4ResourceType.Observation, Id('obs2'), null, null);

      final search = await resourceDao.getResourceType(
        null,
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 2);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), false);

      expect(idList.contains('obs3'), true);
    });

    test('Delete All Observations', () async {
      await resourceDao.deleteSingleType(null,
          resourceType: R4ResourceType.Observation);

      final search = await resourceDao.getAll(null);

      expect(search.length, 2);

      final patList = search.toList();
      final orgList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);

      expect(patList.length, 1);

      expect(patList.length, 1);
    });

    test('Delete All Resources', () async {
      await resourceDao.deleteAllResources(null);

      final search = await resourceDao.getAll(null);

      expect(search.length, 0);
    });
  });

  group('Password - Saving Things:', () {
    test('Save Patient', () async {
      await resourceDao.updatePw(null, 'newPw');
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);
      final patient = Patient(id: id, name: [humanName]);
      final saved = await resourceDao.save('newPw', patient);

      expect(saved.id, id);

      expect((saved as Patient).name?[0], humanName);
    });

    test('Save Organization', () async {
      final organization = Organization(id: id, name: 'FhirFli');
      final saved = await resourceDao.save('newPw', organization);

      expect(saved.id, id);

      expect((saved as Organization).name, 'FhirFli');
    });

    test('Save Observation1', () async {
      final observation1 = Observation(
        id: Id('obs1'),
        code: CodeableConcept(text: 'Observation #1'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1');
    });

    test('Save Observation1 Again', () async {
      final observation1 = Observation(
          id: Id('obs1'),
          code: CodeableConcept(text: 'Observation #1 - Updated'));
      final saved = await resourceDao.save('newPw', observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1 - Updated');

      expect(saved.meta?.versionId, Id('2'));
    });

    test('Save Observation2', () async {
      final observation2 = Observation(
        id: Id('obs2'),
        code: CodeableConcept(text: 'Observation #2'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation2);

      expect(saved.id, Id('obs2'));

      expect((saved as Observation).code.text, 'Observation #2');
    });

    test('Save Observation3', () async {
      final observation3 = Observation(
        id: Id('obs3'),
        code: CodeableConcept(text: 'Observation #3'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation3);

      expect(saved.id, Id('obs3'));

      expect((saved as Observation).code.text, 'Observation #3');
    });
  });

  group('Password - Finding Things:', () {
    test('Find 1st Patient', () async {
      final search = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Patient, id: id);
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);

      expect(search.length, 1);

      expect((search[0] as Patient).name?[0], humanName);
    });

    test('Find 3rd Observation', () async {
      final search = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Observation, id: Id('obs3'));

      expect(search.length, 1);

      expect(search[0].id, Id('obs3'));

      expect((search[0] as Observation).code.text, 'Observation #3');
    });

    test('Find All Observations', () async {
      final search = await resourceDao.getResourceType(
        'newPw',
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 3);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), true);

      expect(idList.contains('obs3'), true);
    });

    test('Find All (non-historical) Resources', () async {
      final search = await resourceDao.getAll('newPw');

      expect(search.length, 5);
      final patList = search.toList();
      final orgList = search.toList();
      final obsList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);
      obsList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Observation);

      expect(patList.length, 1);

      expect(orgList.length, 1);

      expect(obsList.length, 3);
    });
  });

  group('Password - Deleting Things:', () {
    test('Delete 2nd Observation', () async {
      await resourceDao.delete(
          'newPw', null, R4ResourceType.Observation, Id('obs2'), null, null);

      final search = await resourceDao.getResourceType(
        'newPw',
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 2);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), false);

      expect(idList.contains('obs3'), true);
    });

    test('Delete All Observations', () async {
      await resourceDao.deleteSingleType('newPw',
          resourceType: R4ResourceType.Observation);

      final search = await resourceDao.getAll('newPw');

      expect(search.length, 2);

      final patList = search.toList();
      final orgList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);

      expect(patList.length, 1);

      expect(patList.length, 1);
    });

    test('Delete All Resources', () async {
      await resourceDao.deleteAllResources('newPw');

      final search = await resourceDao.getAll('newPw');

      expect(search.length, 0);

      await resourceDao.updatePw('newPw', null);
    });
  });
} 

Download Details:

Author: MayJuun

Source Code: https://github.com/MayJuun/fhir/tree/main/fhir_db

#sqflite  #dart  #flutter 

Garry Taylor

Garry Taylor

1669952228

Dijkstra's Algorithm Explained with Examples

In this tutorial, you'll learn: What is Dijkstra's Algorithm and how Dijkstra's algorithm works with the help of visual guides.

You can use algorithms in programming to solve specific problems through a set of precise instructions or procedures.

Dijkstra's algorithm is one of many graph algorithms you'll come across. It is used to find the shortest path from a fixed node to all other nodes in a graph.

There are different representations of Dijkstra's algorithm. You can either find the shortest path between two nodes, or the shortest path from a fixed node to the rest of the nodes in a graph.

In this article, you'll learn how Dijkstra's algorithm works with the help of visual guides.

How Does Dijkstra’s Algorithm Work?

Before we dive into more detailed visual examples, you need to understand how Dijkstra's algorithm works.

Although the theoretical explanation may seem a bit abstract, it'll help you understand the practical aspect better.

In a given graph containing different nodes, we are required to get the shortest path from a given node to the rest of the nodes.

These nodes can represent any object like the names of cities, letters, and so on.

Between each node is a number denoting the distance between two nodes, as you can see in the image below:

nodes-1

We usually work with two arrays – one for visited nodes, and another for unvisited nodes. You'll learn more about the arrays in the next section.

When a node is visited, the algorithm calculates how long it took to get to the node and stores the distance. If a shorter path to a node is found, the initial value assigned for the distance is updated.

Note that a node cannot be visited twice.

The algorithm runs recursively until all the nodes have been visited.

Dijkstra's Algorithm Example

In this section, we'll take a look at a practical example that shows how Dijkstra's algorithm works.

Here's the graph we'll be working with:

nodes

We'll use the table below to put down the visited nodes and their distance from the fixed node:

NODESHORTEST DISTANCE FROM FIXED NODE
A
B
C
D
E

Visited nodes = []
Unvisited nodes = [A,B,C,D,E]

Above, we have a table showing each node and the shortest distance from the that node to the fixed node. We are yet to choose the fixed node.

Note that the distance for each node in the table is currently denoted as infinity (∞). This is because we don't know the shortest distance yet.

We also have two arrays – visited and unvisited. Whenever a node is visited, it is added to the visited nodes array.

Let's get started!

To simplify things, I'll break the process down into iterations. You'll see what happens in each step with the aid of diagrams.

Iteration #1

The first iteration might seem confusing, but that's totally fine. Once we start repeating the process in each iteration, you'll have a clearer picture of how the algorithm works.

Step #1 - Pick an unvisited node

We'll choose A as the fixed node. So we'll find the shortest distance from A to every other node in the graph.

node1-1

We're going to give A a distance of 0 because it is the initial node. So the table would look like this:

NODESHORTEST DISTANCE FROM FIXED NODE
A0
B
C
D
E

Step #2 - Find the distance from current nodenode1a-3

The next thing to do after choosing a node is to find the distance from it to the unvisited nodes around it.

The two unvisited nodes directly linked to A are B and C.

To get the distance from A to B:

0 + 4 = 4

0 being the value of the current node (A), and 4 being the distance between A and B in the graph.

To get the distance from A to C:

0 + 2 = 2

Step #3 - Update table with known distances

In the last step, we got 4 and 2 as the values of B and C respectively. So we'll update the table with those values:

NODESHORTEST DISTANCE FROM FIXED NODE
A0
B4
C2
D
E

Step #4 - Update arrays

At this point, the first iteration is complete. We'll move node A to the visited nodes array:

Visited nodes = [A]
Unvisited nodes = [B,C,D,E]

Before we proceed to the next iteration, you should know the following:

  • Once a node has been visited, it cannot be linked to the current node. Refer to step #2 in the iteration above and step #2 in the next iteration.
  • A node cannot be visited twice.
  • You can only update the shortest known distance if you get a value smaller than the recorded distance.

Iteration #2

Step #1 - Pick an unvisited node

We have four unvisited nodes — [B,C,D,E]. So how do you know which node to pick for the next iteration?

Well, we pick the node with the smallest known distance recorded in the table. Here's the table:

NODESHORTEST DISTANCE FROM FIXED NODE
A0
B4
C2
D
E

So we're going with node C.

node2-2

Step #2 - Find the distance from current node

To find the distance from the current node to the fixed node, we have to consider the nodes linked to the current node.

The nodes linked to the current node are A and B.

But A has been visited in the previous iteration so it will not be linked to the current node. That is:

node2a-1

From the diagram above,

  • The green color denotes the current node.
  • The blue color denotes the visited nodes. We cannot link to them or visit them again.
  • The red color shows the link from the unvisited nodes to the current node.

To find the distance from C to B:

2 + 1 = 3

2 above is recorded distance for node C while 1 is the distance between C and B in the graph.

Step #3 - Update table with known distances

In the last step, we got the value of B to be 3. In the first iteration, it was 4.

We're going to update the distance in the table to 3.

NODESHORTEST DISTANCE FROM FIXED NODE
A0
B3
C2
D
E

So, A --> B = 4 (First iteration).

A --> C --> B = 3 (Second iteration).

The algorithm has helped us find the shortest path to B from A.

Step #4 - Update arrays

We're done with the last visited node. Let's add it to the visited nodes array:

Visited nodes = [A,C]
Unvisited nodes = [B,D,E]

Iteration #3

Step #1 - Pick an unvisited node

We're down to three unvisited nodes — [B,D,E]. From the array, B has the shortest known distance.

node3-2

To restate what is going on in the diagram above:

  • The green color denotes the current node.
  • The blue color denotes the visited nodes. We cannot link to them or visit them again.
  • The red color shows the link from the unvisited nodes to the current node.

Step #2 - Find the distance from current node

The nodes linked to the current node are D and E.

B (the current node) has a value of 3. Therefore,

For node D, 3 + 3 = 6.

For node E, 3 + 2 = 5.

Step #3 - Update table with known distances

NODESHORTEST DISTANCE FROM FIXED NODE
A0
B3
C2
D6
E5

Step #4 - Update arrays

Visited nodes = [A,C,B]
Unvisited nodes = [D,E]

Iteration #4

Step #1 - Pick an unvisited node

Like other iterations, we'll go with the unvisited node with the shortest known distance. That is E.

node4-1

Step #2 - Find the distance from current node

According to our table, E has a value of 5.

For D in the current iteration,

5 + 5 = 10.

The value gotten for D here is 10, which is greater than the recorded value of 6 in the previous iteration. For this reason, we'll not update the table.

Step #3 - Update table with known distances

Our table remains the same:

NODESHORTEST DISTANCE FROM FIXED NODE
A0
B3
C2
D6
E5

Step #4 - Update arrays

Visited nodes = [A,C,B,E]
Unvisited nodes = [D]

Iteration #5

Step #1 - Pick an unvisited node

We're currently left with one node in the unvisited array — D.

node5-1

Step #2 - Find the distance from current node

The algorithm has gotten to the last iteration. This is because all nodes linked to the current node have been visited already so we can't link to them.

Step #3 - Update table with known distances

Our table remains the same:

NODESHORTEST DISTANCE FROM FIXED NODE
A0
B3
C2
D6
E5

At this point, we have updated the table with the shortest distance from the fixed node to every other node in the graph.

Step #4 - Update arrays

Visited nodes = [A,C,B,E,D]
Unvisited nodes = []

As can be seen above, we have no nodes left to visit. Using Dijkstra's algorithm, we've found the shortest distance from the fixed node to others nodes in the graph.

Dijkstra's Algorithm Pseudocode Example

The pseudocode example in this section was gotten from Wikipedia. Here it is:

 1  function Dijkstra(Graph, source):
 2      
 3      for each vertex v in Graph.Vertices:
 4          dist[v] ← INFINITY
 5          prev[v] ← UNDEFINED
 6          add v to Q
 7      dist[source] ← 0
 8      
 9      while Q is not empty:
10          u ← vertex in Q with min dist[u]
11          remove u from Q
12          
13          for each neighbor v of u still in Q:
14              alt ← dist[u] + Graph.Edges(u, v)
15              if alt < dist[v]:
16                  dist[v] ← alt
17                  prev[v] ← u
18
19      return dist[], prev[]

Applications of Dijkstra's Algorithm

Here are some of the common applications of Dijkstra's algorithm:

  • In maps to get the shortest distance between locations. An example is Google Maps.
  • In telecommunications to determine transmission rate.
  • In robotic design to determine shortest path for automated robots.

Summary

In this article, we talked about Dijkstra's algorithm. It is used to find the shortest distance from a fixed node to all other nodes in a graph.

We started by giving a brief summary of how the algorithm works.

We then had a look at an example that further explained Dijkstra's algorithm in steps using visual guides.

We concluded with a pseudocode example and some of the applications of Dijkstra's algorithm.

Happy coding!

Original article source at https://www.freecodecamp.org

#algorithm #datastructures

Sexpr.jl: Julia <3 Clojure + Macroexpansion

S-Julia - s-expression to julia convertor.  

Quickstart

> Pkg.clone("https://github.com/vshesh/Sexpr.jl.git")

$ julia -e 'import Sexpr; Sexpr.main()' --
usage: Sexpr.jl [-i] [-c] [-l LINES] [-o OUTPUT] [-e EXTENSION] [-h]
               [files...]
A program to port clojure-like s-expression syntax to and from
julia. By default, this program takes clojure syntax and outputs
the julia version. Use -i to flip direction.

positional arguments:
  files                 If given one file and no output directory,
                        will dump to stdout. If given a directory or
                        multiple files, eg "sjulia file1 dir file2",
                        an output directory must be specified with
                        -o/--output where the files will go.
optional arguments:
  -i, --invert          take julia code and print out s-expression
                        code instead
  -c, --cat             cat all the input from STDIN rather than read
                        from file. Ignores all positional args to the
                        program.
  -l, --lines LINES     how many blank lines should exist between top
                        level forms, default 1 (type: Int64, default:
                        1)
  -o, --output OUTPUT   where to write out files if there are multiple
                        positional arguments to the file. If this is
                        empty, and there are >1 argument, the program
                        will throw an error.
  -e, --extension EXTENSION
                        add an extension that qualifies as a lisp file
                        (can use multiple times). Defaults: clj, cljs,
                        cl, lisp, wisp, hy.
  -h, --help            show this help message and exit
$ julia -e 'import Sexpr; Sexpr.main()' -- -o test/output/ test/programs/
# will transpile all .clj files in test/programs and dump them into test/output.

Overview

This project aims to make s-expression syntax interoperable with julia's own Expr objects.

If you've seen LispSyntax.jl, it's a similar idea, but IMHO this project does a bit more, such as allow you to transpile file->file rather than just read in a program, and also transpile back, so you can convert your julia files (minus a few special forms that aren't supported yet) into clojure syntax. This makes it possible to go from julia to python (again, not that anyone needed another route b/c pycall) via Hylang, or to JS via WispJS. The benefit here is that the awkward macro syntax in both of those languages is avoided (Hy necessitates wrapping everything in HyModel objects yourself, which is ridiculous, and WispJS's module system is broken, because it is Javascript, so resolving variable names is not working properly).

The final goal is to use interoperability to do a macroexpand operation on the input clj syntax. So you would be able to give a folder of clj files, and a temp folder with jl files would be created, then each file would be read in and macroexpanded, converted back to clj syntax, and written out to a third folder. Unfortunately, it's necessary to write the jl files out as an intermediary step, because they need to be able to find each other to resolve imports. Alternatively, you could write the clj files as jl files with the macro @clj_str, but that makes your whole file a string, which breaks most syntax highlighters, which can be annoying.

I know that you're probably thinking "why?" and it was mostly a project for me to learn Julia and muck around with its internals. I learned quite a bit, so mission accomplished! CLJS has self-hosting now, which means that they will hopefully have a js-only package soon. However, dealing with google closure compiler and leiningen's java/jvm dependencies are a larger problem to be solved, and until then, I still consider it unwieldy, so there's still some practical use to be had here.

Effectively, this is just the reader portion of implementing a lisp - Julia does everything else using its inbuilt mechanisms.

Syntax Overview

Atoms

  • nil translates to julia's nothing. They work exactly the same.
  • true -> true and false -> false. No surprises at all there.
  • number constants compile to either Int64 or Float64 types.
    • rational constants also supported, so 3/5 -> 3//5 in Julia.
  • character any atom starting with a \ is a character.
    • \newline, \space, \tab, \formfeed, \backspace, \return for escapes
    • unicode/octal support still needs to be handled.
    • non-strict, giving a longer literal silently just gives you the first character right now. This is probably not the best long-term strategy. Eg, \xyz -> \x
  • string is any sequence of characters inside double quotes.
    • multiline strings are allowed, but padding is not subtracted (yet).
  • keyword basically a symbol that starts with a :. In julia, these are confusingly called symbols, and symbols are called variables.
    • keywords cannot have a / or . in them anywhere.
    • in clojure keywords with two colons are resolved in the current namespace, that behavior is not the same here. Everything just compiles to a normal symbol in julia, so no namespacing. There are probably issues here, I just don't know what they are.
  • symbol which is any identifier for a variable.
    • any / or . character is converted to a . in julia. Eg, module/function becomes module.function as an identifier. This should be relatively consistent with clojure semantics.
    • clojure is more lenient about symbol characters than julia. In order to get around this limitation, the default is to output unicode characters where regular ones won't suffice. so *+?!-_':>< are all allowed inside a symbol.
      • TODO make the option to use escaped ascii-only names available. (ugly, but avoids having to use unicode, which is a pain depending on how your unicode extension is defined).
      • :: in a symbol identifier compiles to a type. eg, x::Int compiles to (:: x Int)
      • ::T1::T2 compiles to a union like x::T1::T2 -> (:: x T1 T2)

Collections

  • '(a b c) a list - if not quoted, it's evaluated and transpiled.
    • quoted lists evaluate to tuples, as of now.
  • [a b c] a vector - transpiles to a julia array.
  • {a b c d} a map - transpiles to a julia Dict(a => b, c=> d) form.
  • TODO #{a b c} a set, which can map to Set() in julia.

Julia Special Forms

  • Short Circuit
    • and/&& (what you expect this to be) - needs to be a special form because of short circuiting. Julia defines the and and or forms this way on purpose.
    • or/|| (again, what you expect), see above.
  • x[i] family (getting/setting/slicing arrays)
    • (aget x 1) -> Expr(:ref, :x, 1) -> x[1].
    • (aget x 1 2 4 5) -> Expr(:ref, :x, 1, 2, 4, 5) -> x[1, 2, 4, 5]
    • (aget x (: 1 3)) -> x[1:3]
    • (aget x (: 6)) -> x[6:end] (preferred)
    • (aget x (: 6 :end)) -> x[6:end] (not preferred)
  • Typing
    • (:: x Int) -> x::Int The :: form defines types.
      • (:: x Int In64) -> x::Union{Int, Int64} there's auto-union if many types are defined.
      • only useful for function and type defintions.
    • (curly Array Int64) -> Array{Int64} will allow parameterized types.
    • (.b a x y) -> a.b(x,y) is the dot call form.
    • (. a b c d) -> a.b.c.d is the dot access form.
      • note that ((. a b) x y) is equivalent to (.b a x y).
  • Modules and Import
    • (module M ... end) creates a module. This is visually annoying since you indent your whole file by two spaces just for this call to module, however I haven't figured out any better way to do this - the other option is to make #module M a special hash dispatch that wraps the whole file but... meh, I don't consider this a high enough priority.
    • (import|using X y z a b) contrary to my expectations, this will give you import X.y.a.b. There will be a separate import statement for each function/file you want to use.
      • TODO make this cartesian productable, so (import X [y z a]) will expand to import X.y; import X.z; import X.a instead. This should shorten the writing. Ideally should make this a system macro (in a system.clj file that I define) and call it import* or something.
    • (export a b c) -> export a, b, c. It makes sense from julia's point of view, since modules are flat things, and you only ever have one level of definitions to export.

Special Forms

()/'() or empty list.

  • For now, this compiles to an empty tuple. In some lisps this is equivalent to nil (eg Common Lisp) but in Clojure it's not, so I'm following that convention.

(do exprs...) does each expression and returns the results of the last one.

(if test true-case false-case?) standard if, evaluates form #2 and branches.

(let [var1 value1 var2 value2...] exprs...) binds pairs of variables to their values, then evaluates exprs in an implicit do.

(fn name? [params...] exprs...) defines a function.

  • a function with no name and only one expr in the body will be converted to a -> form. Eg: (fn [x] x) -> (x) -> x.

(defn name docstring? [params...] exprs...) named defined function.

  • docstrings are ignored right now.

(def var expr) defines a variable.

throw is a function already in julia, so there's no special form dedicated to it.

include is a function already in julia, so there's no dedicated special form for it.

TODOS

  • loop/recur (this doesn't have a julia equivalent),
  • try/catch/finally
  • for vars in expr do... (useful for lazy iterators)
  • destructuring and rest param like (fn [& rest])
  • defmulti and related (does this even mean anything given julia's multiple-dispatch?)
  • deftype -> type in Julia.

Macro Forms

  • (@m x y z) how to call a macro - prepend it's name with @. There is unfortunately no way around this, since julia requires this distinction and for me to resolve what things are macros without it would involve writing an entire compiler. To keep it simple, I'm leaving this requirement in place.
    • TODO Since @x means deref in clojure, I might choose to use a different symbol to denote macrocall in the future. maybe μ or something. Another idea is abusing # dispatch so `#macro (html [:div "helloworld"])`` calls the next form as a macro rather than a regular function. The hash dispatch one seems worse, though.
  • defmacro defines a macro, as expected.
    • The way that macros work right now is that the macro definition is passed a clojure s-expression to work with. This is not the same as being passed a julia equivalent.
    • the macro output should again be a clojure expression, which has to be translated by the reader into a julia expression. This means that whatever program you write has to include the reader module of this project in order to produce the desired output.
    • Every macro will end in a call to Sexpr.rehydrate() which will translate the expression back to julia's native AST.
  • quote or ' gives a literal list of the following expression.
    • The quote form doesn't properly escape symbols yet. Eg, 'x is equal to :x in Julia, but in order to stop the gensym pass from running you actually have to do esc(:x) to get the equivalent. I'm unclear as of yet how the translation should work to get the desired results, so right now quote and syntax-quote do the same thing, which needs to be changed.
    • you can get around this yourself by putting esc calls in the right places, it will compile down to a function call in the code.
  • syntax-quote or backtick character. the :() quoting form in julia is actually a syntax quote. It also has an auto-gensym (which can be a pain to get around if you want to return the original name without obfuscation).
  • unquote or ~ is $ in julia inside expressions. It should evaluate the variable that's given to the macro and use the evaluated value.
  • unquote-splice or ~@ unquotes, and also expands the form by one layer into the form that's being returned. Ie, (f ~@x) is the same as :(f($x...)) in julia.

Download Details:

Author: VShesh
Source Code: https://github.com/vshesh/Sexpr.jl 

#julia  #clojure 

Chatgpt Api: Node.js Client for The Unofficial ChatGPT API

On December 11th, OpenAI added Cloudflare protections that make it more difficult to access the unofficial API.

To circumvent these protections, we've added a fully automated browser-based solution, which uses Puppeteer and CAPTCHA solvers under the hood. 🔥

import { ChatGPTAPIBrowser } from 'chatgpt'

const api = new ChatGPTAPIBrowser({
  email: process.env.OPENAI_EMAIL,
  password: process.env.OPENAI_PASSWORD
})
await api.initSession()

const result = await api.sendMessage('Hello World!')
console.log(result.response)

This solution is not lightweight, but it does work a lot more consistently than the previous REST API-based approach. For example, I'm currently using this approach to automate 10 concurrent OpenAI accounts for my Twitter bot. 😂

To use the updated version, make sure you're using the latest version of this package and Node.js >= 18. Then update your code following the examples below, paying special attention to the sections on Authentication, Restrictions, and CAPTCHAs.

We recently added support for CAPTCHA automation using either nopecha or 2captcha. Keep in mind that this package will be updated to use the official API as soon as it's released, so things should get much easier over time. 💪

Lastly, please consider starring this repo and following me on twitter  to help support the project.

Thanks && cheers, Travis


ChatGPT API

Node.js client for the unofficial ChatGPT API.

Intro

This package is a Node.js wrapper around ChatGPT by OpenAI. TS batteries included. ✨

You can use it to start building projects powered by ChatGPT like chatbots, websites, etc...

Install

npm install chatgpt puppeteer

puppeteer is an optional peer dependency used to automate bypassing the Cloudflare protections via getOpenAIAuth. The main API wrapper uses fetch directly.

Usage

import { ChatGPTAPIBrowser } from 'chatgpt'

async function example() {
  // use puppeteer to bypass cloudflare (headful because of captchas)
  const api = new ChatGPTAPIBrowser({
    email: process.env.OPENAI_EMAIL,
    password: process.env.OPENAI_PASSWORD
  })

  await api.initSession()

  const result = await api.sendMessage('Hello World!')
  console.log(result.response)
}

Or, if you want to use the REST-based version:

import { ChatGPTAPI, getOpenAIAuth } from 'chatgpt'

async function example() {
  // use puppeteer to bypass cloudflare (headful because of captchas)
  const openAIAuth = await getOpenAIAuth({
    email: process.env.OPENAI_EMAIL,
    password: process.env.OPENAI_PASSWORD
  })

  const api = new ChatGPTAPI({ ...openAIAuth })
  await api.initSession()

  // send a message and wait for the response
  const result = await api.sendMessage('Write a python version of bubble sort.')

  // result.response is a markdown-formatted string
  console.log(result.response)
}

ChatGPT responses are formatted as markdown by default. If you want to work with plaintext instead, you can use:

const api = new ChatGPTAPIBrowser({ email, password, markdown: false })

If you want to track the conversation, use the conversationId and messageId in the result object, and pass them to sendMessage as conversationId and parentMessageId respectively.

const api = new ChatGPTAPIBrowser({ email, password })
await api.initSession()

// send a message and wait for the response
let res = await api.sendMessage('What is OpenAI?')
console.log(res.response)

// send a follow-up
res = await api.sendMessage('Can you expand on that?', {
  conversationId: res.conversationId,
  parentMessageId: res.messageId
})
console.log(res.response)

// send another follow-up
// send a follow-up
res = await api.sendMessage('What were we talking about?', {
  conversationId: res.conversationId,
  parentMessageId: res.messageId
})
console.log(res.response)

Sometimes, ChatGPT will hang for an extended period of time before beginning to respond. This may be due to rate limiting or it may be due to OpenAI's servers being overloaded.

To mitigate these issues, you can add a timeout like this:

// timeout after 2 minutes (which will also abort the underlying HTTP request)
const response = await api.sendMessage('this is a timeout test', {
  timeoutMs: 2 * 60 * 1000
})

Usage in CommonJS (Dynamic import)

async function example() {
  // To use ESM in CommonJS, you can use a dynamic import
  const { ChatGPTAPI, getOpenAIAuth } = await import('chatgpt')

  const openAIAuth = await getOpenAIAuth({
    email: process.env.OPENAI_EMAIL,
    password: process.env.OPENAI_PASSWORD
  })

  const api = new ChatGPTAPI({ ...openAIAuth })
  await api.initSession()

  const result = await api.sendMessage('Hello World!')
  console.log(result)
}

Docs

See the auto-generated docs for more info on methods and parameters. Here are the docs for the browser-based version.

Demos

To run the included demos:

  1. clone repo
  2. install node deps
  3. set OPENAI_EMAIL and OPENAI_PASSWORD in .env

A basic demo is included for testing purposes:

npx tsx demos/demo.ts

A conversation demo is also included:

npx tsx demos/demo-conversation.ts

A browser-based conversation demo is also included:

npx tsx demos/demo-conversation-browser.ts

Authentication

The authentication section relates to the REST-based version (using getOpenAIAuth + ChatGPTAPI). The browser-based solution, ChatGPTAPIBrowser, takes care of all the authentication for you.

On December 11, 2022, OpenAI added some additional Cloudflare protections which make it more difficult to access the unofficial API.

You'll need a valid OpenAI "session token" and Cloudflare "clearance token" in order to use the API.

We've provided an automated, Puppeteer-based solution getOpenAIAuth to fetch these for you, but you may still run into cases where you have to manually pass the CAPTCHA. We're working on a solution to automate this further.

You can also get these tokens manually, but keep in mind that the clearanceToken only lasts for max 2 hours.

Getting tokens manually

To get session token manually:

  1. Go to https://chat.openai.com/chat and log in or sign up.
  2. Open dev tools.
  3. Open Application > Cookies. ChatGPT cookies
  4. Copy the value for __Secure-next-auth.session-token and save it to your environment. This will be your sessionToken.
  5. Copy the value for cf_clearance and save it to your environment. This will be your clearanceToken.
  6. Copy the value of the user-agent header from any request in your Network tab. This will be your userAgent.

Pass sessionToken, clearanceToken, and userAgent to the ChatGPTAPI constructor.

Note This package will switch to using the official API once it's released, which will make this process much simpler.

CAPTCHAs

The browser portions of this package use Puppeteer to automate as much as possible, including solving all CAPTCHAs. 🔥

Basic Cloudflare CAPTCHAs are handled by default, but if you want to automate the email + password Recaptchas, you'll need to sign up for one of these paid providers:

  • nopecha - Uses AI to solve CAPTCHAS
    • Faster and cheaper
    • Set the NOPECHA_KEY env var to your nopecha API key
    • Demo video of nopecha solving the login Recaptcha (41 seconds)
  • 2captcha - Uses real people to solve CAPTCHAS
    • More well-known solution that's been around longer
    • Set the CAPTCHA_TOKEN env var to your 2captcha API token

Alternatively, if your OpenAI account uses Google Auth, you shouldn't encounter any of the more complicated Recaptchas — and can avoid using these third-party providers. To use Google auth, make sure your OpenAI account is using Google and then set isGoogleLogin to true whenever you're passing your email and password. For example:

const api = new ChatGPTAPIBrowser({
  email: process.env.OPENAI_EMAIL,
  password: process.env.OPENAI_PASSWORD,
  isGoogleLogin: true
})

Using Proxies

The browser implementation supports setting a proxy server. This is useful if you're running into rate limiting issues or if you want to use a proxy to hide your IP address.

To use a proxy, pass the proxyServer option to the ChatGPTAPIBrowser constructor, or simply set the PROXY_SERVER env var. For more information on the format, see here.

const api = new ChatGPTAPIBrowser({
  email: process.env.OPENAI_EMAIL,
  password: process.env.OPENAI_PASSWORD,
  proxyServer: '<ip>:<port>'
})

You can also set the PROXY_VALIDATE_IP env var to your proxy's IP address. This will be used to validate that the proxy is working correctly, and will throw an error if it's not.

Restrictions

These restrictions are for the getOpenAIAuth + ChatGPTAPI solution, which uses the unofficial API. The browser-based solution, ChatGPTAPIBrowser, generally doesn't have any of these restrictions.

Please read carefully

  • You must use node >= 18 at the moment. I'm using v19.2.0 in my testing.
  • Cloudflare cf_clearance tokens expire after 2 hours, so right now we recommend that you refresh your cf_clearance token every hour or so.
  • Your user-agent and IP address must match from the real browser window you're logged in with to the one you're using for ChatGPTAPI.
    • This means that you currently can't log in with your laptop and then run the bot on a server or proxy somewhere.
  • Cloudflare will still sometimes ask you to complete a CAPTCHA, so you may need to keep an eye on it and manually resolve the CAPTCHA.
  • You should not be using this account while the bot is using it, because that browser window may refresh one of your tokens and invalidate the bot's session.

Note Prior to v1.0.0, this package used a headless browser via Playwright to automate the web UI. Here are the docs for the initial browser version.

Projects

All of these awesome projects are built using the chatgpt package. 🤯

If you create a cool integration, feel free to open a PR and add it to the list.

Compatibility

This package is ESM-only. It supports:

  • Node.js >= 18
    • Node.js 17, 16, and 14 were supported in earlier versions, but OpenAI's Cloudflare update caused a bug with undici on v17 and v16 that needs investigation. So for now, use node >= 18
  • We recommend against using chatgpt from client-side browser code because it would expose your private session token
  • If you want to build a website using chatgpt, we recommend using it only from your backend API

Credits


Download Details:

Author: transitive-bullshit
Source Code: https://github.com/transitive-bullshit/chatgpt-api

License: MIT license

#chartgpt