Gradient Boosting Explained for Beginners - Part 1

In this video, we are going to explain what is gradient boosting.

We will discuss the following in this video:
🕕 (0:00:06) Introduction
🕕 (0:01:02) Boosting
🕕 (0:03:40) Gradient Descent
🕕 (0:07:57) Gradient Boosting
🕕 (0:17:49) Implementation

#data-science #artificial-intelligence #developer

What is GEEK

Buddha Community

Gradient Boosting Explained for Beginners - Part 1
Veronica  Roob

Veronica Roob

1653475560

A Pure PHP Implementation Of The MessagePack Serialization Format

msgpack.php

A pure PHP implementation of the MessagePack serialization format.

Features

Installation

The recommended way to install the library is through Composer:

composer require rybakit/msgpack

Usage

Packing

To pack values you can either use an instance of a Packer:

$packer = new Packer();
$packed = $packer->pack($value);

or call a static method on the MessagePack class:

$packed = MessagePack::pack($value);

In the examples above, the method pack automatically packs a value depending on its type. However, not all PHP types can be uniquely translated to MessagePack types. For example, the MessagePack format defines map and array types, which are represented by a single array type in PHP. By default, the packer will pack a PHP array as a MessagePack array if it has sequential numeric keys, starting from 0 and as a MessagePack map otherwise:

$mpArr1 = $packer->pack([1, 2]);               // MP array [1, 2]
$mpArr2 = $packer->pack([0 => 1, 1 => 2]);     // MP array [1, 2]
$mpMap1 = $packer->pack([0 => 1, 2 => 3]);     // MP map {0: 1, 2: 3}
$mpMap2 = $packer->pack([1 => 2, 2 => 3]);     // MP map {1: 2, 2: 3}
$mpMap3 = $packer->pack(['a' => 1, 'b' => 2]); // MP map {a: 1, b: 2}

However, sometimes you need to pack a sequential array as a MessagePack map. To do this, use the packMap method:

$mpMap = $packer->packMap([1, 2]); // {0: 1, 1: 2}

Here is a list of type-specific packing methods:

$packer->packNil();           // MP nil
$packer->packBool(true);      // MP bool
$packer->packInt(42);         // MP int
$packer->packFloat(M_PI);     // MP float (32 or 64)
$packer->packFloat32(M_PI);   // MP float 32
$packer->packFloat64(M_PI);   // MP float 64
$packer->packStr('foo');      // MP str
$packer->packBin("\x80");     // MP bin
$packer->packArray([1, 2]);   // MP array
$packer->packMap(['a' => 1]); // MP map
$packer->packExt(1, "\xaa");  // MP ext

Check the "Custom types" section below on how to pack custom types.

Packing options

The Packer object supports a number of bitmask-based options for fine-tuning the packing process (defaults are in bold):

NameDescription
FORCE_STRForces PHP strings to be packed as MessagePack UTF-8 strings
FORCE_BINForces PHP strings to be packed as MessagePack binary data
DETECT_STR_BINDetects MessagePack str/bin type automatically
  
FORCE_ARRForces PHP arrays to be packed as MessagePack arrays
FORCE_MAPForces PHP arrays to be packed as MessagePack maps
DETECT_ARR_MAPDetects MessagePack array/map type automatically
  
FORCE_FLOAT32Forces PHP floats to be packed as 32-bits MessagePack floats
FORCE_FLOAT64Forces PHP floats to be packed as 64-bits MessagePack floats

The type detection mode (DETECT_STR_BIN/DETECT_ARR_MAP) adds some overhead which can be noticed when you pack large (16- and 32-bit) arrays or strings. However, if you know the value type in advance (for example, you only work with UTF-8 strings or/and associative arrays), you can eliminate this overhead by forcing the packer to use the appropriate type, which will save it from running the auto-detection routine. Another option is to explicitly specify the value type. The library provides 2 auxiliary classes for this, Map and Bin. Check the "Custom types" section below for details.

Examples:

// detect str/bin type and pack PHP 64-bit floats (doubles) to MP 32-bit floats
$packer = new Packer(PackOptions::DETECT_STR_BIN | PackOptions::FORCE_FLOAT32);

// these will throw MessagePack\Exception\InvalidOptionException
$packer = new Packer(PackOptions::FORCE_STR | PackOptions::FORCE_BIN);
$packer = new Packer(PackOptions::FORCE_FLOAT32 | PackOptions::FORCE_FLOAT64);

Unpacking

To unpack data you can either use an instance of a BufferUnpacker:

$unpacker = new BufferUnpacker();

$unpacker->reset($packed);
$value = $unpacker->unpack();

or call a static method on the MessagePack class:

$value = MessagePack::unpack($packed);

If the packed data is received in chunks (e.g. when reading from a stream), use the tryUnpack method, which attempts to unpack data and returns an array of unpacked messages (if any) instead of throwing an InsufficientDataException:

while ($chunk = ...) {
    $unpacker->append($chunk);
    if ($messages = $unpacker->tryUnpack()) {
        return $messages;
    }
}

If you want to unpack from a specific position in a buffer, use seek:

$unpacker->seek(42); // set position equal to 42 bytes
$unpacker->seek(-8); // set position to 8 bytes before the end of the buffer

To skip bytes from the current position, use skip:

$unpacker->skip(10); // set position to 10 bytes ahead of the current position

To get the number of remaining (unread) bytes in the buffer:

$unreadBytesCount = $unpacker->getRemainingCount();

To check whether the buffer has unread data:

$hasUnreadBytes = $unpacker->hasRemaining();

If needed, you can remove already read data from the buffer by calling:

$releasedBytesCount = $unpacker->release();

With the read method you can read raw (packed) data:

$packedData = $unpacker->read(2); // read 2 bytes

Besides the above methods BufferUnpacker provides type-specific unpacking methods, namely:

$unpacker->unpackNil();   // PHP null
$unpacker->unpackBool();  // PHP bool
$unpacker->unpackInt();   // PHP int
$unpacker->unpackFloat(); // PHP float
$unpacker->unpackStr();   // PHP UTF-8 string
$unpacker->unpackBin();   // PHP binary string
$unpacker->unpackArray(); // PHP sequential array
$unpacker->unpackMap();   // PHP associative array
$unpacker->unpackExt();   // PHP MessagePack\Type\Ext object

Unpacking options

The BufferUnpacker object supports a number of bitmask-based options for fine-tuning the unpacking process (defaults are in bold):

NameDescription
BIGINT_AS_STRConverts overflowed integers to strings [1]
BIGINT_AS_GMPConverts overflowed integers to GMP objects [2]
BIGINT_AS_DECConverts overflowed integers to Decimal\Decimal objects [3]

1. The binary MessagePack format has unsigned 64-bit as its largest integer data type, but PHP does not support such integers, which means that an overflow can occur during unpacking.

2. Make sure the GMP extension is enabled.

3. Make sure the Decimal extension is enabled.

Examples:

$packedUint64 = "\xcf"."\xff\xff\xff\xff"."\xff\xff\xff\xff";

$unpacker = new BufferUnpacker($packedUint64);
var_dump($unpacker->unpack()); // string(20) "18446744073709551615"

$unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_GMP);
var_dump($unpacker->unpack()); // object(GMP) {...}

$unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_DEC);
var_dump($unpacker->unpack()); // object(Decimal\Decimal) {...}

Custom types

In addition to the basic types, the library provides functionality to serialize and deserialize arbitrary types. This can be done in several ways, depending on your use case. Let's take a look at them.

Type objects

If you need to serialize an instance of one of your classes into one of the basic MessagePack types, the best way to do this is to implement the CanBePacked interface in the class. A good example of such a class is the Map type class that comes with the library. This type is useful when you want to explicitly specify that a given PHP array should be packed as a MessagePack map without triggering an automatic type detection routine:

$packer = new Packer();

$packedMap = $packer->pack(new Map([1, 2, 3]));
$packedArray = $packer->pack([1, 2, 3]);

More type examples can be found in the src/Type directory.

Type transformers

As with type objects, type transformers are only responsible for serializing values. They should be used when you need to serialize a value that does not implement the CanBePacked interface. Examples of such values could be instances of built-in or third-party classes that you don't own, or non-objects such as resources.

A transformer class must implement the CanPack interface. To use a transformer, it must first be registered in the packer. Here is an example of how to serialize PHP streams into the MessagePack bin format type using one of the supplied transformers, StreamTransformer:

$packer = new Packer(null, [new StreamTransformer()]);

$packedBin = $packer->pack(fopen('/path/to/file', 'r+'));

More type transformer examples can be found in the src/TypeTransformer directory.

Extensions

In contrast to the cases described above, extensions are intended to handle extension types and are responsible for both serialization and deserialization of values (types).

An extension class must implement the Extension interface. To use an extension, it must first be registered in the packer and the unpacker.

The MessagePack specification divides extension types into two groups: predefined and application-specific. Currently, there is only one predefined type in the specification, Timestamp.

Timestamp

The Timestamp extension type is a predefined type. Support for this type in the library is done through the TimestampExtension class. This class is responsible for handling Timestamp objects, which represent the number of seconds and optional adjustment in nanoseconds:

$timestampExtension = new TimestampExtension();

$packer = new Packer();
$packer = $packer->extendWith($timestampExtension);

$unpacker = new BufferUnpacker();
$unpacker = $unpacker->extendWith($timestampExtension);

$packedTimestamp = $packer->pack(Timestamp::now());
$timestamp = $unpacker->reset($packedTimestamp)->unpack();

$seconds = $timestamp->getSeconds();
$nanoseconds = $timestamp->getNanoseconds();

When using the MessagePack class, the Timestamp extension is already registered:

$packedTimestamp = MessagePack::pack(Timestamp::now());
$timestamp = MessagePack::unpack($packedTimestamp);

Application-specific extensions

In addition, the format can be extended with your own types. For example, to make the built-in PHP DateTime objects first-class citizens in your code, you can create a corresponding extension, as shown in the example. Please note, that custom extensions have to be registered with a unique extension ID (an integer from 0 to 127).

More extension examples can be found in the examples/MessagePack directory.

To learn more about how extension types can be useful, check out this article.

Exceptions

If an error occurs during packing/unpacking, a PackingFailedException or an UnpackingFailedException will be thrown, respectively. In addition, an InsufficientDataException can be thrown during unpacking.

An InvalidOptionException will be thrown in case an invalid option (or a combination of mutually exclusive options) is used.

Tests

Run tests as follows:

vendor/bin/phpunit

Also, if you already have Docker installed, you can run the tests in a docker container. First, create a container:

./dockerfile.sh | docker build -t msgpack -

The command above will create a container named msgpack with PHP 8.1 runtime. You may change the default runtime by defining the PHP_IMAGE environment variable:

PHP_IMAGE='php:8.0-cli' ./dockerfile.sh | docker build -t msgpack -

See a list of various images here.

Then run the unit tests:

docker run --rm -v $PWD:/msgpack -w /msgpack msgpack

Fuzzing

To ensure that the unpacking works correctly with malformed/semi-malformed data, you can use a testing technique called Fuzzing. The library ships with a help file (target) for PHP-Fuzzer and can be used as follows:

php-fuzzer fuzz tests/fuzz_buffer_unpacker.php

Performance

To check performance, run:

php -n -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

=============================================
Test/Target            Packer  BufferUnpacker
---------------------------------------------
nil .................. 0.0030 ........ 0.0139
false ................ 0.0037 ........ 0.0144
true ................. 0.0040 ........ 0.0137
7-bit uint #1 ........ 0.0052 ........ 0.0120
7-bit uint #2 ........ 0.0059 ........ 0.0114
7-bit uint #3 ........ 0.0061 ........ 0.0119
5-bit sint #1 ........ 0.0067 ........ 0.0126
5-bit sint #2 ........ 0.0064 ........ 0.0132
5-bit sint #3 ........ 0.0066 ........ 0.0135
8-bit uint #1 ........ 0.0078 ........ 0.0200
8-bit uint #2 ........ 0.0077 ........ 0.0212
8-bit uint #3 ........ 0.0086 ........ 0.0203
16-bit uint #1 ....... 0.0111 ........ 0.0271
16-bit uint #2 ....... 0.0115 ........ 0.0260
16-bit uint #3 ....... 0.0103 ........ 0.0273
32-bit uint #1 ....... 0.0116 ........ 0.0326
32-bit uint #2 ....... 0.0118 ........ 0.0332
32-bit uint #3 ....... 0.0127 ........ 0.0325
64-bit uint #1 ....... 0.0140 ........ 0.0277
64-bit uint #2 ....... 0.0134 ........ 0.0294
64-bit uint #3 ....... 0.0134 ........ 0.0281
8-bit int #1 ......... 0.0086 ........ 0.0241
8-bit int #2 ......... 0.0089 ........ 0.0225
8-bit int #3 ......... 0.0085 ........ 0.0229
16-bit int #1 ........ 0.0118 ........ 0.0280
16-bit int #2 ........ 0.0121 ........ 0.0270
16-bit int #3 ........ 0.0109 ........ 0.0274
32-bit int #1 ........ 0.0128 ........ 0.0346
32-bit int #2 ........ 0.0118 ........ 0.0339
32-bit int #3 ........ 0.0135 ........ 0.0368
64-bit int #1 ........ 0.0138 ........ 0.0276
64-bit int #2 ........ 0.0132 ........ 0.0286
64-bit int #3 ........ 0.0137 ........ 0.0274
64-bit int #4 ........ 0.0180 ........ 0.0285
64-bit float #1 ...... 0.0134 ........ 0.0284
64-bit float #2 ...... 0.0125 ........ 0.0275
64-bit float #3 ...... 0.0126 ........ 0.0283
fix string #1 ........ 0.0035 ........ 0.0133
fix string #2 ........ 0.0094 ........ 0.0216
fix string #3 ........ 0.0094 ........ 0.0222
fix string #4 ........ 0.0091 ........ 0.0241
8-bit string #1 ...... 0.0122 ........ 0.0301
8-bit string #2 ...... 0.0118 ........ 0.0304
8-bit string #3 ...... 0.0119 ........ 0.0315
16-bit string #1 ..... 0.0150 ........ 0.0388
16-bit string #2 ..... 0.1545 ........ 0.1665
32-bit string ........ 0.1570 ........ 0.1756
wide char string #1 .. 0.0091 ........ 0.0236
wide char string #2 .. 0.0122 ........ 0.0313
8-bit binary #1 ...... 0.0100 ........ 0.0302
8-bit binary #2 ...... 0.0123 ........ 0.0324
8-bit binary #3 ...... 0.0126 ........ 0.0327
16-bit binary ........ 0.0168 ........ 0.0372
32-bit binary ........ 0.1588 ........ 0.1754
fix array #1 ......... 0.0042 ........ 0.0131
fix array #2 ......... 0.0294 ........ 0.0367
fix array #3 ......... 0.0412 ........ 0.0472
16-bit array #1 ...... 0.1378 ........ 0.1596
16-bit array #2 ........... S ............. S
32-bit array .............. S ............. S
complex array ........ 0.1865 ........ 0.2283
fix map #1 ........... 0.0725 ........ 0.1048
fix map #2 ........... 0.0319 ........ 0.0405
fix map #3 ........... 0.0356 ........ 0.0665
fix map #4 ........... 0.0465 ........ 0.0497
16-bit map #1 ........ 0.2540 ........ 0.3028
16-bit map #2 ............. S ............. S
32-bit map ................ S ............. S
complex map .......... 0.2372 ........ 0.2710
fixext 1 ............. 0.0283 ........ 0.0358
fixext 2 ............. 0.0291 ........ 0.0371
fixext 4 ............. 0.0302 ........ 0.0355
fixext 8 ............. 0.0288 ........ 0.0384
fixext 16 ............ 0.0293 ........ 0.0359
8-bit ext ............ 0.0302 ........ 0.0439
16-bit ext ........... 0.0334 ........ 0.0499
32-bit ext ........... 0.1845 ........ 0.1888
32-bit timestamp #1 .. 0.0337 ........ 0.0547
32-bit timestamp #2 .. 0.0335 ........ 0.0560
64-bit timestamp #1 .. 0.0371 ........ 0.0575
64-bit timestamp #2 .. 0.0374 ........ 0.0542
64-bit timestamp #3 .. 0.0356 ........ 0.0533
96-bit timestamp #1 .. 0.0362 ........ 0.0699
96-bit timestamp #2 .. 0.0381 ........ 0.0701
96-bit timestamp #3 .. 0.0367 ........ 0.0687
=============================================
Total                  2.7618          4.0820
Skipped                     4               4
Failed                      0               0
Ignored                     0               0

With JIT:

php -n -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.jit_buffer_size=64M -dopcache.jit=tracing -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

=============================================
Test/Target            Packer  BufferUnpacker
---------------------------------------------
nil .................. 0.0005 ........ 0.0054
false ................ 0.0004 ........ 0.0059
true ................. 0.0004 ........ 0.0059
7-bit uint #1 ........ 0.0010 ........ 0.0047
7-bit uint #2 ........ 0.0010 ........ 0.0046
7-bit uint #3 ........ 0.0010 ........ 0.0046
5-bit sint #1 ........ 0.0025 ........ 0.0046
5-bit sint #2 ........ 0.0023 ........ 0.0046
5-bit sint #3 ........ 0.0024 ........ 0.0045
8-bit uint #1 ........ 0.0043 ........ 0.0081
8-bit uint #2 ........ 0.0043 ........ 0.0079
8-bit uint #3 ........ 0.0041 ........ 0.0080
16-bit uint #1 ....... 0.0064 ........ 0.0095
16-bit uint #2 ....... 0.0064 ........ 0.0091
16-bit uint #3 ....... 0.0064 ........ 0.0094
32-bit uint #1 ....... 0.0085 ........ 0.0114
32-bit uint #2 ....... 0.0077 ........ 0.0122
32-bit uint #3 ....... 0.0077 ........ 0.0120
64-bit uint #1 ....... 0.0085 ........ 0.0159
64-bit uint #2 ....... 0.0086 ........ 0.0157
64-bit uint #3 ....... 0.0086 ........ 0.0158
8-bit int #1 ......... 0.0042 ........ 0.0080
8-bit int #2 ......... 0.0042 ........ 0.0080
8-bit int #3 ......... 0.0042 ........ 0.0081
16-bit int #1 ........ 0.0065 ........ 0.0095
16-bit int #2 ........ 0.0065 ........ 0.0090
16-bit int #3 ........ 0.0056 ........ 0.0085
32-bit int #1 ........ 0.0067 ........ 0.0107
32-bit int #2 ........ 0.0066 ........ 0.0106
32-bit int #3 ........ 0.0063 ........ 0.0104
64-bit int #1 ........ 0.0072 ........ 0.0162
64-bit int #2 ........ 0.0073 ........ 0.0174
64-bit int #3 ........ 0.0072 ........ 0.0164
64-bit int #4 ........ 0.0077 ........ 0.0161
64-bit float #1 ...... 0.0053 ........ 0.0135
64-bit float #2 ...... 0.0053 ........ 0.0135
64-bit float #3 ...... 0.0052 ........ 0.0135
fix string #1 ....... -0.0002 ........ 0.0044
fix string #2 ........ 0.0035 ........ 0.0067
fix string #3 ........ 0.0035 ........ 0.0077
fix string #4 ........ 0.0033 ........ 0.0078
8-bit string #1 ...... 0.0059 ........ 0.0110
8-bit string #2 ...... 0.0063 ........ 0.0121
8-bit string #3 ...... 0.0064 ........ 0.0124
16-bit string #1 ..... 0.0099 ........ 0.0146
16-bit string #2 ..... 0.1522 ........ 0.1474
32-bit string ........ 0.1511 ........ 0.1483
wide char string #1 .. 0.0039 ........ 0.0084
wide char string #2 .. 0.0073 ........ 0.0123
8-bit binary #1 ...... 0.0040 ........ 0.0112
8-bit binary #2 ...... 0.0075 ........ 0.0123
8-bit binary #3 ...... 0.0077 ........ 0.0129
16-bit binary ........ 0.0096 ........ 0.0145
32-bit binary ........ 0.1535 ........ 0.1479
fix array #1 ......... 0.0008 ........ 0.0061
fix array #2 ......... 0.0121 ........ 0.0165
fix array #3 ......... 0.0193 ........ 0.0222
16-bit array #1 ...... 0.0607 ........ 0.0479
16-bit array #2 ........... S ............. S
32-bit array .............. S ............. S
complex array ........ 0.0749 ........ 0.0824
fix map #1 ........... 0.0329 ........ 0.0431
fix map #2 ........... 0.0161 ........ 0.0189
fix map #3 ........... 0.0205 ........ 0.0262
fix map #4 ........... 0.0252 ........ 0.0205
16-bit map #1 ........ 0.1016 ........ 0.0927
16-bit map #2 ............. S ............. S
32-bit map ................ S ............. S
complex map .......... 0.1096 ........ 0.1030
fixext 1 ............. 0.0157 ........ 0.0161
fixext 2 ............. 0.0175 ........ 0.0183
fixext 4 ............. 0.0156 ........ 0.0185
fixext 8 ............. 0.0163 ........ 0.0184
fixext 16 ............ 0.0164 ........ 0.0182
8-bit ext ............ 0.0158 ........ 0.0207
16-bit ext ........... 0.0203 ........ 0.0219
32-bit ext ........... 0.1614 ........ 0.1539
32-bit timestamp #1 .. 0.0195 ........ 0.0249
32-bit timestamp #2 .. 0.0188 ........ 0.0260
64-bit timestamp #1 .. 0.0207 ........ 0.0281
64-bit timestamp #2 .. 0.0212 ........ 0.0291
64-bit timestamp #3 .. 0.0207 ........ 0.0295
96-bit timestamp #1 .. 0.0222 ........ 0.0358
96-bit timestamp #2 .. 0.0228 ........ 0.0353
96-bit timestamp #3 .. 0.0210 ........ 0.0319
=============================================
Total                  1.6432          1.9674
Skipped                     4               4
Failed                      0               0
Ignored                     0               0

You may change default benchmark settings by defining the following environment variables:

NameDefault
MP_BENCH_TARGETSpure_p,pure_u, see a list of available targets
MP_BENCH_ITERATIONS100_000
MP_BENCH_DURATIONnot set
MP_BENCH_ROUNDS3
MP_BENCH_TESTS-@slow, see a list of available tests

For example:

export MP_BENCH_TARGETS=pure_p
export MP_BENCH_ITERATIONS=1000000
export MP_BENCH_ROUNDS=5
# a comma separated list of test names
export MP_BENCH_TESTS='complex array, complex map'
# or a group name
# export MP_BENCH_TESTS='-@slow' // @pecl_comp
# or a regexp
# export MP_BENCH_TESTS='/complex (array|map)/'

Another example, benchmarking both the library and the PECL extension:

MP_BENCH_TARGETS=pure_p,pure_u,pecl_p,pecl_u \
php -n -dextension=msgpack.so -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

===========================================================================
Test/Target            Packer  BufferUnpacker  msgpack_pack  msgpack_unpack
---------------------------------------------------------------------------
nil .................. 0.0031 ........ 0.0141 ...... 0.0055 ........ 0.0064
false ................ 0.0039 ........ 0.0154 ...... 0.0056 ........ 0.0053
true ................. 0.0038 ........ 0.0139 ...... 0.0056 ........ 0.0044
7-bit uint #1 ........ 0.0061 ........ 0.0110 ...... 0.0059 ........ 0.0046
7-bit uint #2 ........ 0.0065 ........ 0.0119 ...... 0.0042 ........ 0.0029
7-bit uint #3 ........ 0.0054 ........ 0.0117 ...... 0.0045 ........ 0.0025
5-bit sint #1 ........ 0.0047 ........ 0.0103 ...... 0.0038 ........ 0.0022
5-bit sint #2 ........ 0.0048 ........ 0.0117 ...... 0.0038 ........ 0.0022
5-bit sint #3 ........ 0.0046 ........ 0.0102 ...... 0.0038 ........ 0.0023
8-bit uint #1 ........ 0.0063 ........ 0.0174 ...... 0.0039 ........ 0.0031
8-bit uint #2 ........ 0.0063 ........ 0.0167 ...... 0.0040 ........ 0.0029
8-bit uint #3 ........ 0.0063 ........ 0.0168 ...... 0.0039 ........ 0.0030
16-bit uint #1 ....... 0.0092 ........ 0.0222 ...... 0.0049 ........ 0.0030
16-bit uint #2 ....... 0.0096 ........ 0.0227 ...... 0.0042 ........ 0.0046
16-bit uint #3 ....... 0.0123 ........ 0.0274 ...... 0.0059 ........ 0.0051
32-bit uint #1 ....... 0.0136 ........ 0.0331 ...... 0.0060 ........ 0.0048
32-bit uint #2 ....... 0.0130 ........ 0.0336 ...... 0.0070 ........ 0.0048
32-bit uint #3 ....... 0.0127 ........ 0.0329 ...... 0.0051 ........ 0.0048
64-bit uint #1 ....... 0.0126 ........ 0.0268 ...... 0.0055 ........ 0.0049
64-bit uint #2 ....... 0.0135 ........ 0.0281 ...... 0.0052 ........ 0.0046
64-bit uint #3 ....... 0.0131 ........ 0.0274 ...... 0.0069 ........ 0.0044
8-bit int #1 ......... 0.0077 ........ 0.0236 ...... 0.0058 ........ 0.0044
8-bit int #2 ......... 0.0087 ........ 0.0244 ...... 0.0058 ........ 0.0048
8-bit int #3 ......... 0.0084 ........ 0.0241 ...... 0.0055 ........ 0.0049
16-bit int #1 ........ 0.0112 ........ 0.0271 ...... 0.0048 ........ 0.0045
16-bit int #2 ........ 0.0124 ........ 0.0292 ...... 0.0057 ........ 0.0049
16-bit int #3 ........ 0.0118 ........ 0.0270 ...... 0.0058 ........ 0.0050
32-bit int #1 ........ 0.0137 ........ 0.0366 ...... 0.0058 ........ 0.0051
32-bit int #2 ........ 0.0133 ........ 0.0366 ...... 0.0056 ........ 0.0049
32-bit int #3 ........ 0.0129 ........ 0.0350 ...... 0.0052 ........ 0.0048
64-bit int #1 ........ 0.0145 ........ 0.0254 ...... 0.0034 ........ 0.0025
64-bit int #2 ........ 0.0097 ........ 0.0214 ...... 0.0034 ........ 0.0025
64-bit int #3 ........ 0.0096 ........ 0.0287 ...... 0.0059 ........ 0.0050
64-bit int #4 ........ 0.0143 ........ 0.0277 ...... 0.0059 ........ 0.0046
64-bit float #1 ...... 0.0134 ........ 0.0281 ...... 0.0057 ........ 0.0052
64-bit float #2 ...... 0.0141 ........ 0.0281 ...... 0.0057 ........ 0.0050
64-bit float #3 ...... 0.0144 ........ 0.0282 ...... 0.0057 ........ 0.0050
fix string #1 ........ 0.0036 ........ 0.0143 ...... 0.0066 ........ 0.0053
fix string #2 ........ 0.0107 ........ 0.0222 ...... 0.0065 ........ 0.0068
fix string #3 ........ 0.0116 ........ 0.0245 ...... 0.0063 ........ 0.0069
fix string #4 ........ 0.0105 ........ 0.0253 ...... 0.0083 ........ 0.0077
8-bit string #1 ...... 0.0126 ........ 0.0318 ...... 0.0075 ........ 0.0088
8-bit string #2 ...... 0.0121 ........ 0.0295 ...... 0.0076 ........ 0.0086
8-bit string #3 ...... 0.0125 ........ 0.0293 ...... 0.0130 ........ 0.0093
16-bit string #1 ..... 0.0159 ........ 0.0368 ...... 0.0117 ........ 0.0086
16-bit string #2 ..... 0.1547 ........ 0.1686 ...... 0.1516 ........ 0.1373
32-bit string ........ 0.1558 ........ 0.1729 ...... 0.1511 ........ 0.1396
wide char string #1 .. 0.0098 ........ 0.0237 ...... 0.0066 ........ 0.0065
wide char string #2 .. 0.0128 ........ 0.0291 ...... 0.0061 ........ 0.0082
8-bit binary #1 ........... I ............. I ........... F ............. I
8-bit binary #2 ........... I ............. I ........... F ............. I
8-bit binary #3 ........... I ............. I ........... F ............. I
16-bit binary ............. I ............. I ........... F ............. I
32-bit binary ............. I ............. I ........... F ............. I
fix array #1 ......... 0.0040 ........ 0.0129 ...... 0.0120 ........ 0.0058
fix array #2 ......... 0.0279 ........ 0.0390 ...... 0.0143 ........ 0.0165
fix array #3 ......... 0.0415 ........ 0.0463 ...... 0.0162 ........ 0.0187
16-bit array #1 ...... 0.1349 ........ 0.1628 ...... 0.0334 ........ 0.0341
16-bit array #2 ........... S ............. S ........... S ............. S
32-bit array .............. S ............. S ........... S ............. S
complex array ............. I ............. I ........... F ............. F
fix map #1 ................ I ............. I ........... F ............. I
fix map #2 ........... 0.0345 ........ 0.0391 ...... 0.0143 ........ 0.0168
fix map #3 ................ I ............. I ........... F ............. I
fix map #4 ........... 0.0459 ........ 0.0473 ...... 0.0151 ........ 0.0163
16-bit map #1 ........ 0.2518 ........ 0.2962 ...... 0.0400 ........ 0.0490
16-bit map #2 ............. S ............. S ........... S ............. S
32-bit map ................ S ............. S ........... S ............. S
complex map .......... 0.2380 ........ 0.2682 ...... 0.0545 ........ 0.0579
fixext 1 .................. I ............. I ........... F ............. F
fixext 2 .................. I ............. I ........... F ............. F
fixext 4 .................. I ............. I ........... F ............. F
fixext 8 .................. I ............. I ........... F ............. F
fixext 16 ................. I ............. I ........... F ............. F
8-bit ext ................. I ............. I ........... F ............. F
16-bit ext ................ I ............. I ........... F ............. F
32-bit ext ................ I ............. I ........... F ............. F
32-bit timestamp #1 ....... I ............. I ........... F ............. F
32-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #1 ....... I ............. I ........... F ............. F
64-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #3 ....... I ............. I ........... F ............. F
96-bit timestamp #1 ....... I ............. I ........... F ............. F
96-bit timestamp #2 ....... I ............. I ........... F ............. F
96-bit timestamp #3 ....... I ............. I ........... F ............. F
===========================================================================
Total                  1.5625          2.3866        0.7735          0.7243
Skipped                     4               4             4               4
Failed                      0               0            24              17
Ignored                    24              24             0               7

With JIT:

MP_BENCH_TARGETS=pure_p,pure_u,pecl_p,pecl_u \
php -n -dextension=msgpack.so -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.jit_buffer_size=64M -dopcache.jit=tracing -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

===========================================================================
Test/Target            Packer  BufferUnpacker  msgpack_pack  msgpack_unpack
---------------------------------------------------------------------------
nil .................. 0.0001 ........ 0.0052 ...... 0.0053 ........ 0.0042
false ................ 0.0007 ........ 0.0060 ...... 0.0057 ........ 0.0043
true ................. 0.0008 ........ 0.0060 ...... 0.0056 ........ 0.0041
7-bit uint #1 ........ 0.0031 ........ 0.0046 ...... 0.0062 ........ 0.0041
7-bit uint #2 ........ 0.0021 ........ 0.0043 ...... 0.0062 ........ 0.0041
7-bit uint #3 ........ 0.0022 ........ 0.0044 ...... 0.0061 ........ 0.0040
5-bit sint #1 ........ 0.0030 ........ 0.0048 ...... 0.0062 ........ 0.0040
5-bit sint #2 ........ 0.0032 ........ 0.0046 ...... 0.0062 ........ 0.0040
5-bit sint #3 ........ 0.0031 ........ 0.0046 ...... 0.0062 ........ 0.0040
8-bit uint #1 ........ 0.0054 ........ 0.0079 ...... 0.0062 ........ 0.0050
8-bit uint #2 ........ 0.0051 ........ 0.0079 ...... 0.0064 ........ 0.0044
8-bit uint #3 ........ 0.0051 ........ 0.0082 ...... 0.0062 ........ 0.0044
16-bit uint #1 ....... 0.0077 ........ 0.0094 ...... 0.0065 ........ 0.0045
16-bit uint #2 ....... 0.0077 ........ 0.0094 ...... 0.0063 ........ 0.0045
16-bit uint #3 ....... 0.0077 ........ 0.0095 ...... 0.0064 ........ 0.0047
32-bit uint #1 ....... 0.0088 ........ 0.0119 ...... 0.0063 ........ 0.0043
32-bit uint #2 ....... 0.0089 ........ 0.0117 ...... 0.0062 ........ 0.0039
32-bit uint #3 ....... 0.0089 ........ 0.0118 ...... 0.0063 ........ 0.0044
64-bit uint #1 ....... 0.0097 ........ 0.0155 ...... 0.0063 ........ 0.0045
64-bit uint #2 ....... 0.0095 ........ 0.0153 ...... 0.0061 ........ 0.0045
64-bit uint #3 ....... 0.0096 ........ 0.0156 ...... 0.0063 ........ 0.0047
8-bit int #1 ......... 0.0053 ........ 0.0083 ...... 0.0062 ........ 0.0044
8-bit int #2 ......... 0.0052 ........ 0.0080 ...... 0.0062 ........ 0.0044
8-bit int #3 ......... 0.0052 ........ 0.0080 ...... 0.0062 ........ 0.0043
16-bit int #1 ........ 0.0089 ........ 0.0097 ...... 0.0069 ........ 0.0046
16-bit int #2 ........ 0.0075 ........ 0.0093 ...... 0.0063 ........ 0.0043
16-bit int #3 ........ 0.0075 ........ 0.0094 ...... 0.0062 ........ 0.0046
32-bit int #1 ........ 0.0086 ........ 0.0122 ...... 0.0063 ........ 0.0044
32-bit int #2 ........ 0.0087 ........ 0.0120 ...... 0.0066 ........ 0.0046
32-bit int #3 ........ 0.0086 ........ 0.0121 ...... 0.0060 ........ 0.0044
64-bit int #1 ........ 0.0096 ........ 0.0149 ...... 0.0060 ........ 0.0045
64-bit int #2 ........ 0.0096 ........ 0.0157 ...... 0.0062 ........ 0.0044
64-bit int #3 ........ 0.0096 ........ 0.0160 ...... 0.0063 ........ 0.0046
64-bit int #4 ........ 0.0097 ........ 0.0157 ...... 0.0061 ........ 0.0044
64-bit float #1 ...... 0.0079 ........ 0.0153 ...... 0.0056 ........ 0.0044
64-bit float #2 ...... 0.0079 ........ 0.0152 ...... 0.0057 ........ 0.0045
64-bit float #3 ...... 0.0079 ........ 0.0155 ...... 0.0057 ........ 0.0044
fix string #1 ........ 0.0010 ........ 0.0045 ...... 0.0071 ........ 0.0044
fix string #2 ........ 0.0048 ........ 0.0075 ...... 0.0070 ........ 0.0060
fix string #3 ........ 0.0048 ........ 0.0086 ...... 0.0068 ........ 0.0060
fix string #4 ........ 0.0050 ........ 0.0088 ...... 0.0070 ........ 0.0059
8-bit string #1 ...... 0.0081 ........ 0.0129 ...... 0.0069 ........ 0.0062
8-bit string #2 ...... 0.0086 ........ 0.0128 ...... 0.0069 ........ 0.0065
8-bit string #3 ...... 0.0086 ........ 0.0126 ...... 0.0115 ........ 0.0065
16-bit string #1 ..... 0.0105 ........ 0.0137 ...... 0.0128 ........ 0.0068
16-bit string #2 ..... 0.1510 ........ 0.1486 ...... 0.1526 ........ 0.1391
32-bit string ........ 0.1517 ........ 0.1475 ...... 0.1504 ........ 0.1370
wide char string #1 .. 0.0044 ........ 0.0085 ...... 0.0067 ........ 0.0057
wide char string #2 .. 0.0081 ........ 0.0125 ...... 0.0069 ........ 0.0063
8-bit binary #1 ........... I ............. I ........... F ............. I
8-bit binary #2 ........... I ............. I ........... F ............. I
8-bit binary #3 ........... I ............. I ........... F ............. I
16-bit binary ............. I ............. I ........... F ............. I
32-bit binary ............. I ............. I ........... F ............. I
fix array #1 ......... 0.0014 ........ 0.0059 ...... 0.0132 ........ 0.0055
fix array #2 ......... 0.0146 ........ 0.0156 ...... 0.0155 ........ 0.0148
fix array #3 ......... 0.0211 ........ 0.0229 ...... 0.0179 ........ 0.0180
16-bit array #1 ...... 0.0673 ........ 0.0498 ...... 0.0343 ........ 0.0388
16-bit array #2 ........... S ............. S ........... S ............. S
32-bit array .............. S ............. S ........... S ............. S
complex array ............. I ............. I ........... F ............. F
fix map #1 ................ I ............. I ........... F ............. I
fix map #2 ........... 0.0148 ........ 0.0180 ...... 0.0156 ........ 0.0179
fix map #3 ................ I ............. I ........... F ............. I
fix map #4 ........... 0.0252 ........ 0.0201 ...... 0.0214 ........ 0.0167
16-bit map #1 ........ 0.1027 ........ 0.0836 ...... 0.0388 ........ 0.0510
16-bit map #2 ............. S ............. S ........... S ............. S
32-bit map ................ S ............. S ........... S ............. S
complex map .......... 0.1104 ........ 0.1010 ...... 0.0556 ........ 0.0602
fixext 1 .................. I ............. I ........... F ............. F
fixext 2 .................. I ............. I ........... F ............. F
fixext 4 .................. I ............. I ........... F ............. F
fixext 8 .................. I ............. I ........... F ............. F
fixext 16 ................. I ............. I ........... F ............. F
8-bit ext ................. I ............. I ........... F ............. F
16-bit ext ................ I ............. I ........... F ............. F
32-bit ext ................ I ............. I ........... F ............. F
32-bit timestamp #1 ....... I ............. I ........... F ............. F
32-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #1 ....... I ............. I ........... F ............. F
64-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #3 ....... I ............. I ........... F ............. F
96-bit timestamp #1 ....... I ............. I ........... F ............. F
96-bit timestamp #2 ....... I ............. I ........... F ............. F
96-bit timestamp #3 ....... I ............. I ........... F ............. F
===========================================================================
Total                  0.9642          1.0909        0.8224          0.7213
Skipped                     4               4             4               4
Failed                      0               0            24              17
Ignored                    24              24             0               7

Note that the msgpack extension (v2.1.2) doesn't support ext, bin and UTF-8 str types.

License

The library is released under the MIT License. See the bundled LICENSE file for details.

Author: rybakit
Source Code: https://github.com/rybakit/msgpack.php
License: MIT License

#php 

Treebender: A Symbolic Natural Language Parsing Library for Rust

Treebender

A symbolic natural language parsing library for Rust, inspired by HDPSG.

What is this?

This is a library for parsing natural or constructed languages into syntax trees and feature structures. There's no machine learning or probabilistic models, everything is hand-crafted and deterministic.

You can find out more about the motivations of this project in this blog post.

But what are you using it for?

I'm using this to parse a constructed language for my upcoming xenolinguistics game, Themengi.

Motivation

Using a simple 80-line grammar, introduced in the tutorial below, we can parse a simple subset of English, checking reflexive pronoun binding, case, and number agreement.

$ cargo run --bin cli examples/reflexives.fgr
> she likes himself
Parsed 0 trees

> her likes herself
Parsed 0 trees

> she like herself
Parsed 0 trees

> she likes herself
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: she))
  (1..2: TV (1..2: likes))
  (2..3: N (2..3: herself)))
[
  child-2: [
    case: acc
    pron: ref
    needs_pron: #0 she
    num: sg
    child-0: [ word: herself ]
  ]
  child-1: [
    tense: nonpast
    child-0: [ word: likes ]
    num: #1 sg
  ]
  child-0: [
    child-0: [ word: she ]
    case: nom
    pron: #0
    num: #1
  ]
]

Low resource language? Low problem! No need to train on gigabytes of text, just write a grammar using your brain. Let's hypothesize that in American Sign Language, topicalized nouns (expressed with raised eyebrows) must appear first in the sentence. We can write a small grammar (18 lines), and plug in some sentences:

$ cargo run --bin cli examples/asl-wordorder.fgr -n
> boy sit
Parsed 1 tree
(0..2: S
  (0..1: NP ((0..1: N (0..1: boy))))
  (1..2: IV (1..2: sit)))

> boy throw ball
Parsed 1 tree
(0..3: S
  (0..1: NP ((0..1: N (0..1: boy))))
  (1..2: TV (1..2: throw))
  (2..3: NP ((2..3: N (2..3: ball)))))

> ball nm-raised-eyebrows boy throw
Parsed 1 tree
(0..4: S
  (0..2: NP
    (0..1: N (0..1: ball))
    (1..2: Topic (1..2: nm-raised-eyebrows)))
  (2..3: NP ((2..3: N (2..3: boy))))
  (3..4: TV (3..4: throw)))

> boy throw ball nm-raised-eyebrows
Parsed 0 trees

Tutorial

As an example, let's say we want to build a parser for English reflexive pronouns (himself, herself, themselves, themself, itself). We'll also support number ("He likes X" v.s. "They like X") and simple embedded clauses ("He said that they like X").

Grammar files are written in a custom language, similar to BNF, called Feature GRammar (.fgr). There's a VSCode syntax highlighting extension for these files available as fgr-syntax.

We'll start by defining our lexicon. The lexicon is the set of terminal symbols (symbols in the actual input) that the grammar will match. Terminal symbols must start with a lowercase letter, and non-terminal symbols must start with an uppercase letter.

// pronouns
N -> he
N -> him
N -> himself
N -> she
N -> her
N -> herself
N -> they
N -> them
N -> themselves
N -> themself

// names, lowercase as they are terminals
N -> mary
N -> sue
N -> takeshi
N -> robert

// complementizer
Comp -> that

// verbs -- intransitive, transitive, and clausal
IV -> falls
IV -> fall
IV -> fell

TV -> likes
TV -> like
TV -> liked

CV -> says
CV -> say
CV -> said

Next, we can add our sentence rules (they must be added at the top, as the first rule in the file is assumed to be the top-level rule):

// sentence rules
S -> N IV
S -> N TV N
S -> N CV Comp S

// ... previous lexicon ...

Assuming this file is saved as examples/no-features.fgr (which it is :wink:), we can test this file with the built-in CLI:

$ cargo run --bin cli examples/no-features.fgr
> he falls
Parsed 1 tree
(0..2: S
  (0..1: N (0..1: he))
  (1..2: IV (1..2: falls)))
[
  child-1: [ child-0: [ word: falls ] ]
  child-0: [ child-0: [ word: he ] ]
]

> he falls her
Parsed 0 trees

> he likes her
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: he))
  (1..2: TV (1..2: likes))
  (2..3: N (2..3: her)))
[
  child-2: [ child-0: [ word: her ] ]
  child-1: [ child-0: [ word: likes ] ]
  child-0: [ child-0: [ word: he ] ]
]

> he likes
Parsed 0 trees

> he said that he likes her
Parsed 1 tree
(0..6: S
  (0..1: N (0..1: he))
  (1..2: CV (1..2: said))
  (2..3: Comp (2..3: that))
  (3..6: S
    (3..4: N (3..4: he))
    (4..5: TV (4..5: likes))
    (5..6: N (5..6: her))))
[
  child-0: [ child-0: [ word: he ] ]
  child-2: [ child-0: [ word: that ] ]
  child-1: [ child-0: [ word: said ] ]
  child-3: [
    child-2: [ child-0: [ word: her ] ]
    child-1: [ child-0: [ word: likes ] ]
    child-0: [ child-0: [ word: he ] ]
  ]
]

> he said that he
Parsed 0 trees

This grammar already parses some correct sentences, and blocks some trivially incorrect ones. However, it doesn't care about number, case, or reflexives right now:

> she likes himself  // unbound reflexive pronoun
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: she))
  (1..2: TV (1..2: likes))
  (2..3: N (2..3: himself)))
[
  child-0: [ child-0: [ word: she ] ]
  child-2: [ child-0: [ word: himself ] ]
  child-1: [ child-0: [ word: likes ] ]
]

> him like her  // incorrect case on the subject pronoun, should be nominative
                // (he) instead of accusative (him)
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: him))
  (1..2: TV (1..2: like))
  (2..3: N (2..3: her)))
[
  child-0: [ child-0: [ word: him ] ]
  child-1: [ child-0: [ word: like ] ]
  child-2: [ child-0: [ word: her ] ]
]

> he like her  // incorrect verb number agreement
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: he))
  (1..2: TV (1..2: like))
  (2..3: N (2..3: her)))
[
  child-2: [ child-0: [ word: her ] ]
  child-1: [ child-0: [ word: like ] ]
  child-0: [ child-0: [ word: he ] ]
]

To fix this, we need to add features to our lexicon, and restrict the sentence rules based on features.

Features are added with square brackets, and are key: value pairs separated by commas. **top** is a special feature value, which basically means "unspecified" -- we'll come back to it later. Features that are unspecified are also assumed to have a **top** value, but sometimes explicitly stating top is more clear.

/// Pronouns
// The added features are:
// * num: sg or pl, whether this noun wants a singular verb (likes) or
//   a plural verb (like). note this is grammatical number, so for example
//   singular they takes plural agreement ("they like X", not *"they likes X")
// * case: nom or acc, whether this noun is nominative or accusative case.
//   nominative case goes in the subject, and accusative in the object.
//   e.g., "he fell" and "she likes him", not *"him fell" and *"her likes he"
// * pron: he, she, they, or ref -- what type of pronoun this is
// * needs_pron: whether this is a reflexive that needs to bind to another
//   pronoun.
N[ num: sg, case: nom, pron: he ]                    -> he
N[ num: sg, case: acc, pron: he ]                    -> him
N[ num: sg, case: acc, pron: ref, needs_pron: he ]   -> himself
N[ num: sg, case: nom, pron: she ]                   -> she
N[ num: sg, case: acc, pron: she ]                   -> her
N[ num: sg, case: acc, pron: ref, needs_pron: she]   -> herself
N[ num: pl, case: nom, pron: they ]                  -> they
N[ num: pl, case: acc, pron: they ]                  -> them
N[ num: pl, case: acc, pron: ref, needs_pron: they ] -> themselves
N[ num: sg, case: acc, pron: ref, needs_pron: they ] -> themself

// Names
// The added features are:
// * num: sg, as people are singular ("mary likes her" / *"mary like her")
// * case: **top**, as names can be both subjects and objects
//   ("mary likes her" / "she likes mary")
// * pron: whichever pronoun the person uses for reflexive agreement
//   mary    pron: she  => mary likes herself
//   sue     pron: they => sue likes themself
//   takeshi pron: he   => takeshi likes himself
N[ num: sg, case: **top**, pron: she ]  -> mary
N[ num: sg, case: **top**, pron: they ] -> sue
N[ num: sg, case: **top**, pron: he ]   -> takeshi
N[ num: sg, case: **top**, pron: he ]   -> robert

// Complementizer doesn't need features
Comp -> that

// Verbs -- intransitive, transitive, and clausal
// The added features are:
// * num: sg, pl, or **top** -- to match the noun numbers.
//   **top** will match either sg or pl, as past-tense verbs in English
//   don't agree in number: "he fell" and "they fell" are both fine
// * tense: past or nonpast -- this won't be used for agreement, but will be
//   copied into the final feature structure, and the client code could do
//   something with it
IV[ num:      sg, tense: nonpast ] -> falls
IV[ num:      pl, tense: nonpast ] -> fall
IV[ num: **top**, tense: past ]    -> fell

TV[ num:      sg, tense: nonpast ] -> likes
TV[ num:      pl, tense: nonpast ] -> like
TV[ num: **top**, tense: past ]    -> liked

CV[ num:      sg, tense: nonpast ] -> says
CV[ num:      pl, tense: nonpast ] -> say
CV[ num: **top**, tense: past ]    -> said

Now that our lexicon is updated with features, we can update our sentence rules to constrain parsing based on those features. This uses two new features, tags and unification. Tags allow features to be associated between nodes in a rule, and unification controls how those features are compatible. The rules for unification are:

  1. A string feature can unify with a string feature with the same value
  2. A top feature can unify with anything, and the nodes are merged
  3. A complex feature ([ ... ] structure) is recursively unified with another complex feature.

If unification fails anywhere, the parse is aborted and the tree is discarded. This allows the programmer to discard trees if features don't match.

// Sentence rules
// Intransitive verb:
// * Subject must be nominative case
// * Subject and verb must agree in number (copied through #1)
S -> N[ case: nom, num: #1 ] IV[ num: #1 ]
// Transitive verb:
// * Subject must be nominative case
// * Subject and verb must agree in number (copied through #2)
// * If there's a reflexive in the object position, make sure its `needs_pron`
//   feature matches the subject's `pron` feature. If the object isn't a
//   reflexive, then its `needs_pron` feature will implicitly be `**top**`, so
//   will unify with anything.
S -> N[ case: nom, pron: #1, num: #2 ] TV[ num: #2 ] N[ case: acc, needs_pron: #1 ]
// Clausal verb:
// * Subject must be nominative case
// * Subject and verb must agree in number (copied through #1)
// * Reflexives can't cross clause boundaries (*"He said that she likes himself"),
//   so we can ignore reflexives and delegate to inner clause rule
S -> N[ case: nom, num: #1 ] CV[ num: #1 ] Comp S

Now that we have this augmented grammar (available as examples/reflexives.fgr), we can try it out and see that it rejects illicit sentences that were previously accepted, while still accepting valid ones:

> he fell
Parsed 1 tree
(0..2: S
  (0..1: N (0..1: he))
  (1..2: IV (1..2: fell)))
[
  child-1: [
    child-0: [ word: fell ]
    num: #0 sg
    tense: past
  ]
  child-0: [
    pron: he
    case: nom
    num: #0
    child-0: [ word: he ]
  ]
]

> he like him
Parsed 0 trees

> he likes himself
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: he))
  (1..2: TV (1..2: likes))
  (2..3: N (2..3: himself)))
[
  child-1: [
    num: #0 sg
    child-0: [ word: likes ]
    tense: nonpast
  ]
  child-2: [
    needs_pron: #1 he
    num: sg
    child-0: [ word: himself ]
    pron: ref
    case: acc
  ]
  child-0: [
    child-0: [ word: he ]
    pron: #1
    num: #0
    case: nom
  ]
]

> he likes herself
Parsed 0 trees

> mary likes herself
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: mary))
  (1..2: TV (1..2: likes))
  (2..3: N (2..3: herself)))
[
  child-0: [
    pron: #0 she
    num: #1 sg
    case: nom
    child-0: [ word: mary ]
  ]
  child-1: [
    tense: nonpast
    child-0: [ word: likes ]
    num: #1
  ]
  child-2: [
    child-0: [ word: herself ]
    num: sg
    pron: ref
    case: acc
    needs_pron: #0
  ]
]

> mary likes themself
Parsed 0 trees

> sue likes themself
Parsed 1 tree
(0..3: S
  (0..1: N (0..1: sue))
  (1..2: TV (1..2: likes))
  (2..3: N (2..3: themself)))
[
  child-0: [
    pron: #0 they
    child-0: [ word: sue ]
    case: nom
    num: #1 sg
  ]
  child-1: [
    tense: nonpast
    num: #1
    child-0: [ word: likes ]
  ]
  child-2: [
    needs_pron: #0
    case: acc
    pron: ref
    child-0: [ word: themself ]
    num: sg
  ]
]

> sue likes himself
Parsed 0 trees

If this is interesting to you and you want to learn more, you can check out my blog series, the excellent textbook Syntactic Theory: A Formal Introduction (2nd ed.), and the DELPH-IN project, whose work on the LKB inspired this simplified version.

Using from code

I need to write this section in more detail, but if you're comfortable with Rust, I suggest looking through the codebase. It's not perfect, it started as one of my first Rust projects (after migrating through F# -> TypeScript -> C in search of the right performance/ergonomics tradeoff), and it could use more tests, but overall it's not too bad.

Basically, the processing pipeline is:

  1. Make a Grammar struct
  • Grammar is defined in rules.rs.
  • The easiest way to make a Grammar is Grammar::parse_from_file, which is mostly a hand-written recusive descent parser in parse_grammar.rs. Yes, I recognize the irony here.
  1. It takes input (in Grammar::parse, which does everything for you, or Grammar::parse_chart, which just does the chart)
  2. The input is first chart-parsed in earley.rs
  3. Then, a forest is built from the chart, in forest.rs, using an algorithm I found in a very useful blog series I forget the URL for, because the algorithms in the academic literature for this are... weird.
  4. Finally, the feature unification is used to prune the forest down to only valid trees. It would be more efficient to do this during parsing, but meh.

The most interesting thing you can do via code and not via the CLI is probably getting at the raw feature DAG, as that would let you do things like pronoun coreference. The DAG code is in featurestructure.rs, and should be fairly approachable -- there's a lot of Rust ceremony around Rc<RefCell<...>> because using an arena allocation crate seemed too harlike overkill, but that is somewhat mitigated by the NodeRef type alias. Hit me up at https://vgel.me/contact if you need help with anything here!

Download Details:
Author: vgel
Source Code: https://github.com/vgel/treebender
License: MIT License

#rust  #machinelearning 

파읎썬 윔딩 묎료 강의 - 읎믞지 처늬, 얌굎 읞식을 통한 캐늭터 씌우Ʞ륌 핎볎아요

파읎썬 윔딩 묎료 강의 (활용펞6) - 읎믞지 처늬, 얌굎 읞식을 통한 캐늭터 씌우Ʞ륌 핎볎아요

파읎썬 묎료 강의 (활용펞6 - 읎믞지 처늬)입니닀.
OpenCV 륌 읎용한 닀양한 읎믞지 처늬 Ʞ법곌 재믞있는 프로젝튞륌 진행합니닀.
누구나 볌 수 있도록 쉜고 재믞있게 제작하였습니닀. ^^

[소개]
(0:00:00) 0.Intro
(0:00:31) 1.소개
(0:02:18) 2.활용펞 6 읎믞지 처늬 소개

[OpenCV 전반전]
(0:04:36) 3.환겜섀정
(0:08:41) 4.읎믞지 출력
(0:21:51) 5.동영상 출력 #1 파음
(0:29:58) 6.동영상 출력 #2 칎메띌
(0:34:23) 7.도형 귞늬Ʞ #1 빈 슀쌀치북
(0:39:49) 8.도형 귞늬Ʞ #2 영역 색칠
(0:42:26) 9.도형 귞늬Ʞ #3 직선
(0:51:23) 10.도형 귞늬Ʞ #4 원
(0:55:09) 11.도형 귞늬Ʞ #5 사각형
(0:58:32) 12.도형 귞늬Ʞ #6 닀각형
(1:09:23) 13.텍슀튞 #1 Ʞ볞
(1:17:45) 14.텍슀튞 #2 한Ꞁ 우회
(1:24:14) 15.파음 저장 #1 읎믞지
(1:29:27) 16.파음 저장 #2 동영상
(1:39:29) 17.크Ʞ 조정
(1:50:16) 18.읎믞지 자륎Ʞ
(1:57:03) 19.읎믞지 대칭
(2:01:46) 20.읎믞지 회전
(2:06:07) 21.읎믞지 변형 - 흑백
(2:11:25) 22.읎믞지 변형 - 흐늌
(2:18:03) 23.읎믞지 변형 - 원귌 #1
(2:27:45) 24.읎믞지 변형 - 원귌 #2

[반자동 묞서 슀캐너 프로젝튞]
(2:32:50) 25.믞니 프로젝튞 1 - #1 마우슀 읎벀튞 등록
(2:42:06) 26.믞니 프로젝튞 1 - #2 Ʞ볞 윔드 완성
(2:49:54) 27.믞니 프로젝튞 1 - #3 지점 선 ꞋꞰ
(2:55:24) 28.믞니 프로젝튞 1 - #4 싀시간 선 ꞋꞰ

[OpenCV 후반전]
(3:01:52) 29.읎믞지 변형 - 읎진화 #1 Trackbar
(3:14:37) 30.읎믞지 변형 - 읎진화 #2 임계값
(3:20:26) 31.읎믞지 변형 - 읎진화 #3 Adaptive Threshold
(3:28:34) 32.읎믞지 변형 - 읎진화 #4 였잠 알고늬슘
(3:32:22) 33.읎믞지 변환 - 팜찜
(3:41:10) 34.읎믞지 변환 - 칚식
(3:45:56) 35.읎믞지 변환 - 엎늌 & 닫힘
(3:54:10) 36.읎믞지 검출 - 겜계선
(4:05:08) 37.읎믞지 검출 - 윀곜선 #1 Ʞ볞
(4:15:26) 38.읎믞지 검출 - 윀곜선 #2 ì°Ÿêž° 몚드
(4:20:46) 39.읎믞지 검출 - 윀곜선 #3 멎적

[칎드 검출 & 분류Ʞ 프로젝튞]
(4:27:42) 40.믞니프로젝튞 2

[퀎슈]
(4:31:57) 41.퀎슈

[얌굎읞식 프로젝튞]
(4:41:25) 42.환겜섀정 및 Ʞ볞 윔드 정늬
(4:54:48) 43.눈곌 윔 읞식하여 도형 귞늬Ʞ
(5:10:42) 44.귞늌판 읎믞지 씌우Ʞ
(5:20:52) 45.캐늭터 읎믞지 씌우Ʞ
(5:33:10) 46.볎충섀명
(5:40:53) 47.마치며 (학습 ì°žê³  자료)
(5:42:18) 48.Outro


[학습자료]
수업에 필요한 읎믞지, 동영상 자료 링크입니닀.

고양읎 읎믞지 : https://pixabay.com/images/id-2083492/ 
크Ʞ : 640 x 390  
파음명 : img.jpg

고양읎 동영상 : https://www.pexels.com/video/7515833/ 
크Ʞ : SD (360 x 640)  
파음명 : video.mp4

신묞 읎믞지 : https://pixabay.com/images/id-350376/ 
크Ʞ : 1280 x 853  
파음명 : newspaper.jpg

칎드 읎믞지 1 : https://pixabay.com/images/id-682332/ 
크Ʞ : 1280 x 1019  
파음명 : poker.jpg

책 읎믞지 : https://www.pexels.com/photo/1029807/ 
크Ʞ : Small (640 x 853)  
파음명 : book.jpg

눈사람 읎믞지 : https://pixabay.com/images/id-1300089/ 
크Ʞ : 1280 x 904  
파음명 : snowman.png

칎드 읎믞지 2 : https://pixabay.com/images/id-161404/ 
크Ʞ : 640 x 408  
파음명 : card.png

퀎슈용 동영상 : https://www.pexels.com/video/3121459/ 
크Ʞ : HD (1280 x 720)  
파음명 : city.mp4

프로젝튞용 동영상 : https://www.pexels.com/video/3256542/ 
크Ʞ : Full HD (1920 x 1080)  
파음명 : face_video.mp4

프로젝튞용 캐늭터 읎믞지 : https://www.freepik.com/free-vector/cute-animal-masks-video-chat-application-effect-filters-set_6380101.htm  
파음명 : right_eye.png (100 x 100), left_eye.png (100 x 100), nose.png (300 x 100)

묎료 읎믞지 펞집 도구 : https://pixlr.com/kr/
(Pixlr E -Advanced Editor)

#python #opencv 

A Wrapper for Sembast and SQFlite to Enable Easy

FHIR_DB

This is really just a wrapper around Sembast_SQFLite - so all of the heavy lifting was done by Alex Tekartik. I highly recommend that if you have any questions about working with this package that you take a look at Sembast. He's also just a super nice guy, and even answered a question for me when I was deciding which sembast version to use. As usual, ResoCoder also has a good tutorial.

I have an interest in low-resource settings and thus a specific reason to be able to store data offline. To encourage this use, there are a number of other packages I have created based around the data format FHIR. FHIR® is the registered trademark of HL7 and is used with the permission of HL7. Use of the FHIR trademark does not constitute endorsement of this product by HL7.

Using the Db

So, while not absolutely necessary, I highly recommend that you use some sort of interface class. This adds the benefit of more easily handling errors, plus if you change to a different database in the future, you don't have to change the rest of your app, just the interface.

I've used something like this in my projects:

class IFhirDb {
  IFhirDb();
  final ResourceDao resourceDao = ResourceDao();

  Future<Either<DbFailure, Resource>> save(Resource resource) async {
    Resource resultResource;
    try {
      resultResource = await resourceDao.save(resource);
    } catch (error) {
      return left(DbFailure.unableToSave(error: error.toString()));
    }
    return right(resultResource);
  }

  Future<Either<DbFailure, List<Resource>>> returnListOfSingleResourceType(
      String resourceType) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.getAllSortedById(resourceType: resourceType);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }

  Future<Either<DbFailure, List<Resource>>> searchFunction(
      String resourceType, String searchString, String reference) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.searchFor(resourceType, searchString, reference);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }
}

I like this because in case there's an i/o error or something, it won't crash your app. Then, you can call this interface in your app like the following:

final patient = Patient(
    resourceType: 'Patient',
    name: [HumanName(text: 'New Patient Name')],
    birthDate: Date(DateTime.now()),
);

final saveResult = await IFhirDb().save(patient);

This will save your newly created patient to the locally embedded database.

IMPORTANT: this database will expect that all previously created resources have an id. When you save a resource, it will check to see if that resource type has already been stored. (Each resource type is saved in it's own store in the database). It will then check if there is an ID. If there's no ID, it will create a new one for that resource (along with metadata on version number and creation time). It will save it, and return the resource. If it already has an ID, it will copy the the old version of the resource into a _history store. It will then update the metadata of the new resource and save that version into the appropriate store for that resource. If, for instance, we have a previously created patient:

{
    "resourceType": "Patient",
    "id": "fhirfli-294057507-6811107",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2020-10-16T19:41:28.054369Z"
    },
    "name": [
        {
            "given": ["New"],
            "family": "Patient"
        }
    ],
    "birthDate": "2020-10-16"
}

And we update the last name to 'Provider'. The above version of the patient will be kept in _history, while in the 'Patient' store in the db, we will have the updated version:

{
    "resourceType": "Patient",
    "id": "fhirfli-294057507-6811107",
    "meta": {
        "versionId": "2",
        "lastUpdated": "2020-10-16T19:45:07.316698Z"
    },
    "name": [
        {
            "given": ["New"],
            "family": "Provider"
        }
    ],
    "birthDate": "2020-10-16"
}

This way we can keep track of all previous version of all resources (which is obviously important in medicine).

For most of the interactions (saving, deleting, etc), they work the way you'd expect. The only difference is search. Because Sembast is NoSQL, we can search on any of the fields in a resource. If in our interface class, we have the following function:

  Future<Either<DbFailure, List<Resource>>> searchFunction(
      String resourceType, String searchString, String reference) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.searchFor(resourceType, searchString, reference);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }

You can search for all immunizations of a certain patient:

searchFunction(
        'Immunization', 'patient.reference', 'Patient/$patientId');

This function will search through all entries in the 'Immunization' store. It will look at all 'patient.reference' fields, and return any that match 'Patient/$patientId'.

The last thing I'll mention is that this is a password protected db, using AES-256 encryption (although it can also use Salsa20). Anytime you use the db, you have the option of using a password for encryption/decryption. Remember, if you setup the database using encryption, you will only be able to access it using that same password. When you're ready to change the password, you will need to call the update password function. If we again assume we created a change password method in our interface, it might look something like this:

class IFhirDb {
  IFhirDb();
  final ResourceDao resourceDao = ResourceDao();
  ...
    Future<Either<DbFailure, Unit>> updatePassword(String oldPassword, String newPassword) async {
    try {
      await resourceDao.updatePw(oldPassword, newPassword);
    } catch (error) {
      return left(DbFailure.unableToUpdatePassword(error: error.toString()));
    }
    return right(Unit);
  }

You don't have to use a password, and in that case, it will save the db file as plain text. If you want to add a password later, it will encrypt it at that time.

General Store

After using this for a while in an app, I've realized that it needs to be able to store data apart from just FHIR resources, at least on occasion. For this, I've added a second class for all versions of the database called GeneralDao. This is similar to the ResourceDao, but fewer options. So, in order to save something, it would look like this:

await GeneralDao().save('password', {'new':'map'});
await GeneralDao().save('password', {'new':'map'}, 'key');

The difference between these two options is that the first one will generate a key for the map being stored, while the second will store the map using the key provided. Both will return the key after successfully storing the map.

Other functions available include:

// deletes everything in the general store
await GeneralDao().deleteAllGeneral('password'); 

// delete specific entry
await GeneralDao().delete('password','key'); 

// returns map with that key
await GeneralDao().find('password', 'key'); 

FHIR® is a registered trademark of Health Level Seven International (HL7) and its use does not constitute an endorsement of products by HL7®

Use this package as a library

Depend on it

Run this command:

With Flutter:

 $ flutter pub add fhir_db

This will add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):

dependencies:
  fhir_db: ^0.4.3

Alternatively, your editor might support or flutter pub get. Check the docs for your editor to learn more.

Import it

Now in your Dart code, you can use:

import 'package:fhir_db/dstu2.dart';
import 'package:fhir_db/dstu2/fhir_db.dart';
import 'package:fhir_db/dstu2/general_dao.dart';
import 'package:fhir_db/dstu2/resource_dao.dart';
import 'package:fhir_db/encrypt/aes.dart';
import 'package:fhir_db/encrypt/salsa.dart';
import 'package:fhir_db/r4.dart';
import 'package:fhir_db/r4/fhir_db.dart';
import 'package:fhir_db/r4/general_dao.dart';
import 'package:fhir_db/r4/resource_dao.dart';
import 'package:fhir_db/r5.dart';
import 'package:fhir_db/r5/fhir_db.dart';
import 'package:fhir_db/r5/general_dao.dart';
import 'package:fhir_db/r5/resource_dao.dart';
import 'package:fhir_db/stu3.dart';
import 'package:fhir_db/stu3/fhir_db.dart';
import 'package:fhir_db/stu3/general_dao.dart';
import 'package:fhir_db/stu3/resource_dao.dart'; 

example/lib/main.dart

import 'package:fhir/r4.dart';
import 'package:fhir_db/r4.dart';
import 'package:flutter/material.dart';
import 'package:test/test.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();

  final resourceDao = ResourceDao();

  // await resourceDao.updatePw('newPw', null);
  await resourceDao.deleteAllResources(null);

  group('Playing with passwords', () {
    test('Playing with Passwords', () async {
      final patient = Patient(id: Id('1'));

      final saved = await resourceDao.save(null, patient);

      await resourceDao.updatePw(null, 'newPw');
      final search1 = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search1[0]);

      await resourceDao.updatePw('newPw', 'newerPw');
      final search2 = await resourceDao.find('newerPw',
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search2[0]);

      await resourceDao.updatePw('newerPw', null);
      final search3 = await resourceDao.find(null,
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search3[0]);

      await resourceDao.deleteAllResources(null);
    });
  });

  final id = Id('12345');
  group('Saving Things:', () {
    test('Save Patient', () async {
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);
      final patient = Patient(id: id, name: [humanName]);
      final saved = await resourceDao.save(null, patient);

      expect(saved.id, id);

      expect((saved as Patient).name?[0], humanName);
    });

    test('Save Organization', () async {
      final organization = Organization(id: id, name: 'FhirFli');
      final saved = await resourceDao.save(null, organization);

      expect(saved.id, id);

      expect((saved as Organization).name, 'FhirFli');
    });

    test('Save Observation1', () async {
      final observation1 = Observation(
        id: Id('obs1'),
        code: CodeableConcept(text: 'Observation #1'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1');
    });

    test('Save Observation1 Again', () async {
      final observation1 = Observation(
          id: Id('obs1'),
          code: CodeableConcept(text: 'Observation #1 - Updated'));
      final saved = await resourceDao.save(null, observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1 - Updated');

      expect(saved.meta?.versionId, Id('2'));
    });

    test('Save Observation2', () async {
      final observation2 = Observation(
        id: Id('obs2'),
        code: CodeableConcept(text: 'Observation #2'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation2);

      expect(saved.id, Id('obs2'));

      expect((saved as Observation).code.text, 'Observation #2');
    });

    test('Save Observation3', () async {
      final observation3 = Observation(
        id: Id('obs3'),
        code: CodeableConcept(text: 'Observation #3'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation3);

      expect(saved.id, Id('obs3'));

      expect((saved as Observation).code.text, 'Observation #3');
    });
  });

  group('Finding Things:', () {
    test('Find 1st Patient', () async {
      final search = await resourceDao.find(null,
          resourceType: R4ResourceType.Patient, id: id);
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);

      expect(search.length, 1);

      expect((search[0] as Patient).name?[0], humanName);
    });

    test('Find 3rd Observation', () async {
      final search = await resourceDao.find(null,
          resourceType: R4ResourceType.Observation, id: Id('obs3'));

      expect(search.length, 1);

      expect(search[0].id, Id('obs3'));

      expect((search[0] as Observation).code.text, 'Observation #3');
    });

    test('Find All Observations', () async {
      final search = await resourceDao.getResourceType(
        null,
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 3);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), true);

      expect(idList.contains('obs3'), true);
    });

    test('Find All (non-historical) Resources', () async {
      final search = await resourceDao.getAll(null);

      expect(search.length, 5);
      final patList = search.toList();
      final orgList = search.toList();
      final obsList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);
      obsList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Observation);

      expect(patList.length, 1);

      expect(orgList.length, 1);

      expect(obsList.length, 3);
    });
  });

  group('Deleting Things:', () {
    test('Delete 2nd Observation', () async {
      await resourceDao.delete(
          null, null, R4ResourceType.Observation, Id('obs2'), null, null);

      final search = await resourceDao.getResourceType(
        null,
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 2);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), false);

      expect(idList.contains('obs3'), true);
    });

    test('Delete All Observations', () async {
      await resourceDao.deleteSingleType(null,
          resourceType: R4ResourceType.Observation);

      final search = await resourceDao.getAll(null);

      expect(search.length, 2);

      final patList = search.toList();
      final orgList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);

      expect(patList.length, 1);

      expect(patList.length, 1);
    });

    test('Delete All Resources', () async {
      await resourceDao.deleteAllResources(null);

      final search = await resourceDao.getAll(null);

      expect(search.length, 0);
    });
  });

  group('Password - Saving Things:', () {
    test('Save Patient', () async {
      await resourceDao.updatePw(null, 'newPw');
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);
      final patient = Patient(id: id, name: [humanName]);
      final saved = await resourceDao.save('newPw', patient);

      expect(saved.id, id);

      expect((saved as Patient).name?[0], humanName);
    });

    test('Save Organization', () async {
      final organization = Organization(id: id, name: 'FhirFli');
      final saved = await resourceDao.save('newPw', organization);

      expect(saved.id, id);

      expect((saved as Organization).name, 'FhirFli');
    });

    test('Save Observation1', () async {
      final observation1 = Observation(
        id: Id('obs1'),
        code: CodeableConcept(text: 'Observation #1'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1');
    });

    test('Save Observation1 Again', () async {
      final observation1 = Observation(
          id: Id('obs1'),
          code: CodeableConcept(text: 'Observation #1 - Updated'));
      final saved = await resourceDao.save('newPw', observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1 - Updated');

      expect(saved.meta?.versionId, Id('2'));
    });

    test('Save Observation2', () async {
      final observation2 = Observation(
        id: Id('obs2'),
        code: CodeableConcept(text: 'Observation #2'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation2);

      expect(saved.id, Id('obs2'));

      expect((saved as Observation).code.text, 'Observation #2');
    });

    test('Save Observation3', () async {
      final observation3 = Observation(
        id: Id('obs3'),
        code: CodeableConcept(text: 'Observation #3'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation3);

      expect(saved.id, Id('obs3'));

      expect((saved as Observation).code.text, 'Observation #3');
    });
  });

  group('Password - Finding Things:', () {
    test('Find 1st Patient', () async {
      final search = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Patient, id: id);
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);

      expect(search.length, 1);

      expect((search[0] as Patient).name?[0], humanName);
    });

    test('Find 3rd Observation', () async {
      final search = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Observation, id: Id('obs3'));

      expect(search.length, 1);

      expect(search[0].id, Id('obs3'));

      expect((search[0] as Observation).code.text, 'Observation #3');
    });

    test('Find All Observations', () async {
      final search = await resourceDao.getResourceType(
        'newPw',
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 3);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), true);

      expect(idList.contains('obs3'), true);
    });

    test('Find All (non-historical) Resources', () async {
      final search = await resourceDao.getAll('newPw');

      expect(search.length, 5);
      final patList = search.toList();
      final orgList = search.toList();
      final obsList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);
      obsList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Observation);

      expect(patList.length, 1);

      expect(orgList.length, 1);

      expect(obsList.length, 3);
    });
  });

  group('Password - Deleting Things:', () {
    test('Delete 2nd Observation', () async {
      await resourceDao.delete(
          'newPw', null, R4ResourceType.Observation, Id('obs2'), null, null);

      final search = await resourceDao.getResourceType(
        'newPw',
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 2);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), false);

      expect(idList.contains('obs3'), true);
    });

    test('Delete All Observations', () async {
      await resourceDao.deleteSingleType('newPw',
          resourceType: R4ResourceType.Observation);

      final search = await resourceDao.getAll('newPw');

      expect(search.length, 2);

      final patList = search.toList();
      final orgList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);

      expect(patList.length, 1);

      expect(patList.length, 1);
    });

    test('Delete All Resources', () async {
      await resourceDao.deleteAllResources('newPw');

      final search = await resourceDao.getAll('newPw');

      expect(search.length, 0);

      await resourceDao.updatePw('newPw', null);
    });
  });
} 

Download Details:

Author: MayJuun

Source Code: https://github.com/MayJuun/fhir/tree/main/fhir_db

#sqflite  #dart  #flutter 

【 初心者向け】C蚀語でのマルチスレッド の抂芁

ニュヌペヌクで働き、りォヌル街䞭のプログラマヌず話をしおいるず、ほずんどのリアルタむムプログラミングアプリケヌションで期埅される共通の知識の糞に気づきたした。その知識はマルチスレッドずしお知られおいたす。私はプログラミングの䞖界を移動し、朜圚的なプログラミング候補者にむンタビュヌを行ったので、マルチスレッドに぀いおほずんど知られおいないこずや、スレッドが適甚される理由や方法に驚かされるこずは決しおありたせん。Vance Morrisonによっお曞かれた䞀連の優れた蚘事で、MSDNはこの問題に察凊しようずしたしたMSDNの8月号、すべおの開発者がマルチスレッドアプリに぀いお知っおおくべきこず、および10月号はマルチスレッドでのロヌロック技術の圱響を理解するを参照しおください。アプリ。

この蚘事では、スレッド化、スレッド化が䜿甚される理由、および.NETでのスレッド化の䜿甚方法に぀いお玹介したす。マルチスレッドの背埌にある謎を完党に明らかにし、それを説明する際に、コヌド内の朜圚的なスレッド障害を回避するのに圹立぀こずを願っおいたす。

スレッドずは䜕ですか

すべおのアプリケヌションは、少なくずも1぀のスレッドで実行されたす。では、スレッドずは䜕ですかスレッドはプロセスにすぎたせん。私の掚枬では、糞ずいう蚀葉は、織機で糞を織り䞊げる超自然的なミュヌズのギリシャ神話に由来しおいるず思いたす。各糞は、誰かの人生の時間の道を衚しおいたす。あなたがその糞をいじるず、あなたは人生の構造を乱したり、人生のプロセスを倉えたりしたす。コンピュヌタヌでは、スレッドは時間の経過ずずもに移動するプロセスです。プロセスは䞀連の順次ステップを実行し、各ステップはコヌド行を実行したす。ステップは連続しおいるため、各ステップには䞀定の時間がかかりたす。䞀連のステップを完了するのにかかる時間は、各プログラミングステップの実行にかかる時間の合蚈です。

マルチスレッドアプリケヌションずは䜕ですか

長い間、ほずんどのプログラミングアプリケヌション組み蟌みシステムプログラムを陀くはシングルスレッドでした。これは、アプリケヌション党䜓でスレッドが1぀しかないこずを意味したす。蚈算Bが完了するたで、蚈算Aを実行するこずはできたせん。プログラムはステップ1から始たり、最埌のステップステップ10ず呌びたすに到達するたで順次続行したすステップ2、ステップ3、ステップ4。マルチスレッドアプリケヌションを䜿甚するず、耇数のスレッドを実行できたす。各スレッドは独自のプロセスで実行されたす。したがっお、理論的には、あるスレッドでステップ1を実行し、同時に別のスレッドでステップ2を実行できたす。同時に、ステップ3を独自のスレッドで実行し、ステップ4を独自のスレッドで実行するこずもできたす。したがっお、ステップ1、ステップ2、ステップ3、およびステップ4は同時に実行されたす。理論的には、4぀のステップすべおがほが同時にかかった堎合、シングルスレッドの実行にかかる時間の4分の1でプログラムを終了できたす4プロセッサマシンを䜿甚しおいるず仮定。では、なぜすべおのプログラムがマルチスレッド化されおいないのでしょうか。スピヌドずずもに、あなたは耇雑さに盎面するからです。ステップ1がステップ2の情報に䜕らかの圢で䟝存しおいる堎合を想像しおみおください。ステップ1がステップ2の前に蚈算を終了した堎合、たたはその逆の堎合、プログラムが正しく実行されない可胜性がありたす。

珍しいアナロゞヌ

マルチスレッドを考える別の方法は、人䜓を考慮するこずです。䜓の各噚官心臓、肺、肝臓、脳はすべおプロセスに関䞎しおいたす。各プロセスは同時に実行されおいたす。各臓噚がプロセスのステップずしお実行された堎合を想像しおみおください。最初に心臓、次に脳、次に肝臓、次に肺です。私たちはおそらく死んでしたうでしょう。぀たり、人䜓は1぀の倧きなマルチスレッドアプリケヌションのようなものです。すべおの臓噚は同時に実行されるプロセスであり、これらのプロセスはすべお盞互に䟝存しおいたす。これらのプロセスはすべお、神経信号、血流、化孊的トリガヌを介しお通信したす。すべおのマルチスレッドアプリケヌションず同様に、人䜓は非垞に耇雑です。䞀郚のプロセスが他のプロセスから情報を取埗しない堎合、たたは特定のプロセスが遅くなったり速くなったりするず、医孊的な問題が発生したす。それか'

い぀スレッド化するか

マルチスレッドは、プログラムをより効率的に実行したい状況で最もよく䜿甚されたす。たずえば、りィンドりフォヌムプログラムの䞭に、実行に1秒以䞊かかり、繰り返し実行する必芁のあるメ゜ッドmethod_Aず呌びたすが含たれおいるずしたす。プログラム党䜓が単䞀のスレッドで実行された堎合、ボタンの抌䞋が正しく機胜しなかったり、入力が少し遅くなったりするこずがありたす。method_Aの蚈算量が倚すぎるず、りィンドりフォヌムの特定の郚分がたったく機胜しないこずに気付くかもしれたせん。この蚱容できないプログラムの動䜜は、プログラムにマルチスレッドが必芁であるこずを瀺しおいたす。スレッド化が必芁になるもう1぀の䞀般的なシナリオは、メッセヌゞングシステムです。アプリケヌションに倚数のメッセヌゞが送信されおいる堎合は、メむンの凊理プログラムの実行ず同時にそれらをキャプチャし、適切に配垃する必芁がありたす。重い凊理を実行しおいるずきに䞀連のメッセヌゞを効率的にキャプチャするこずはできたせん。そうしないず、メッセヌゞを芋逃す可胜性がありたす。耇数のスレッドは、耇数のプロセスが同時に実行される組立ラむン方匏で䜿甚するこずもできたす。たずえば、プロセスがスレッドでデヌタを収集するず、1぀のプロセスがデヌタをフィルタリングし、1぀のプロセスがデヌタをデヌタベヌスず照合したす。これらの各シナリオはマルチスレッドの䞀般的な䜿甚法であり、シングルスレッドで実行されおいる同様のアプリケヌションのパフォヌマンスを倧幅に向䞊させたす。そうしないず、メッセヌゞを芋逃す可胜性があるためです。耇数のスレッドは、耇数のプロセスが同時に実行される組立ラむン方匏で䜿甚するこずもできたす。たずえば、プロセスがスレッドでデヌタを収集するず、1぀のプロセスがデヌタをフィルタリングし、1぀のプロセスがデヌタをデヌタベヌスず照合したす。これらの各シナリオはマルチスレッドの䞀般的な䜿甚法であり、シングルスレッドで実行されおいる同様のアプリケヌションのパフォヌマンスを倧幅に向䞊させたす。そうしないず、メッセヌゞを芋逃す可胜性があるためです。耇数のスレッドは、耇数のプロセスが同時に実行される組立ラむン方匏で䜿甚するこずもできたす。たずえば、プロセスがスレッドでデヌタを収集するず、1぀のプロセスがデヌタをフィルタリングし、1぀のプロセスがデヌタをデヌタベヌスず照合したす。これらの各シナリオはマルチスレッドの䞀般的な䜿甚法であり、シングルスレッドで実行されおいる同様のアプリケヌションのパフォヌマンスを倧幅に向䞊させたす。

スレッドしない堎合

初心者のプログラマヌが最初にスレッド化を孊ぶずき、圌らはプログラムでスレッド化を䜿甚する可胜性に魅了される可胜性がありたす。圌らは実際にスレッドハッピヌになるかもしれたせん。  詳しく説明させおください、

1日目プログラマヌは、スレッドを生成できるこずを孊び、プログラムで1぀の新しいスレッドCoolの䜜成を開始したす ã€‚

2日目プログラマヌは、「プログラムの䞀郚で他のスレッドを生成するこずで、これをさらに効率的にするこずができたす」ず蚀いたす。

3日目P「わあ、スレッド内でスレッドをフォヌクするこずもでき、本圓に効率が向䞊したす!!」

4日目P「奇劙な結果が出おいるようですが、それは問題ありたせん。今は無芖したす。」

5日目「うヌん、widgetX倉数に倀がある堎合もありたすが、たったく蚭定されおいないように芋える堎合もありたす。コンピュヌタヌが機胜しおいないため、デバッガヌを実行するだけです」。

9日目「このくそったれより匷い蚀語プログラムはあちこちでゞャンプしおいたす!!䜕が起こっおいるのか理解できたせん」

2週目時々、プログラムはただそこに座っお、たったく䜕もしたせんヘルプ

おなじみですかマルチスレッドプログラムを初めお蚭蚈しようずしたほずんどの人は、スレッドの蚭蚈知識が豊富であっおも、おそらくこれらの毎日の箇条曞きの少なくずも1぀たたは2぀を経隓したこずがありたす。スレッド化が悪いこずだずほのめかしおいるわけではありたせん。プログラムでスレッド化の効率を䞊げるプロセスでは、非垞に泚意しおください。  シングルスレッドプログラムずは異なり、同時に倚くのプロセスを凊理しおいるため、耇数の埓属倉数を持぀耇数のプロセスを远跡するのは非垞に難しい堎合がありたす。ゞャグリングず同じようにマルチスレッドを考えおください。手で1぀のボヌルをゞャグリングするのは退屈ではありたすがかなり簡単です。ただし、これらのボヌルのうち2぀を空䞭に眮くように挑戊された堎合、その䜜業は少し難しくなりたす。3、4、および5の堎合、ボヌルは次第に難しくなりたす。ボヌルの数が増えるず、実際にボヌルを萜ずす可胜性が高くなりたす。 äž€åºŠã«ãŸãã•んのボヌルをゞャグリングするには、知識、スキル、正確なタむミングが必芁です。マルチスレッドもそうです。 

マルチスレッド

図1-マルチスレッドはゞャグリングプロセスのようなものです

 
スレッディングの問題

プログラム内のすべおのプロセスが盞互に排他的である堎合、぀たり、プロセスが他のプロセスにたったく䟝存しおいない堎合、耇数のスレッド化は非垞に簡単で、問題はほずんど発生したせん。各プロセスは、他のプロセスに煩わされるこずなく、独自のハッピヌコヌスで実行されたす。ただし、耇数のプロセスが他のプロセスによっお䜿甚されおいるメモリの読み取りたたは曞き蟌みを行う必芁がある堎合、問題が発生する可胜性がありたす。たずえば、プロセス1ずプロセス2の2぀のプロセスがあるずしたす。䞡方のプロセスが倉数Xを共有したす。スレッドプロセス1が最初に倀5の倉数Xを曞き蟌み、スレッドプロセス2が次に倀-3の倉数Xを曞き蟌む堎合、Xの最終倀は-3です。ただし、プロセス2が最初に倀-3の倉数Xを曞き蟌み、次にプロセス1が倀5の倉数Xを曞き蟌む堎合、Xの最終倀は5です。Xを蚭定できるプロセスがプロセス1たたはプロセス2の知識を持っおいない堎合、Xは、最初にXに到達したスレッドに応じお異なる最終倀になる可胜性がありたす。シングルスレッドプログラムでは、すべおが順番に続くため、これが発生する可胜性はありたせん。シングルスレッドプログラムでは、䞊列に実行されおいるプロセスがないため、Xは垞に最初にメ゜ッド1によっお蚭定され最初に呌び出された堎合、次にメ゜ッド2によっお蚭定されたす。シングルスレッドプログラムには驚きはありたせん。それはステップバむステップです。マルチスレッドプログラムを䜿甚するず、2぀のスレッドが同時にコヌドを入力し、結果に倧混乱をもたらす可胜性がありたす。スレッドの問題は、同時に実行されおいる別のスレッドが同じコヌドを入力しお共有デヌタを操䜜できるようにしながら、共有メモリにアクセスする1぀のスレッドを制埡する䜕らかの方法が必芁なこずです。Xは、どのスレッドが最初にXに到達したかによっお、最終的に異なる最終倀になる可胜性がありたす。シングルスレッドプログラムでは、すべおが順番に続くため、これが発生する可胜性はありたせん。シングルスレッドプログラムでは、䞊列に実行されおいるプロセスがないため、Xは垞に最初にメ゜ッド1によっお蚭定され最初に呌び出された堎合、次にメ゜ッド2によっお蚭定されたす。シングルスレッドプログラムには驚きはありたせん。それはステップバむステップです。マルチスレッドプログラムを䜿甚するず、2぀のスレッドが同時にコヌドを入力し、結果に倧混乱をもたらす可胜性がありたす。スレッドの問題は、同時に実行されおいる別のスレッドが同じコヌドを入力しお共有デヌタを操䜜できるようにしながら、共有メモリにアクセスする1぀のスレッドを制埡する䜕らかの方法が必芁なこずです。Xは、どのスレッドが最初にXに到達したかによっお、最終的に異なる最終倀になる可胜性がありたす。シングルスレッドプログラムでは、すべおが順番に続くため、これが発生する可胜性はありたせん。シングルスレッドプログラムでは、䞊列に実行されおいるプロセスがないため、Xは垞に最初にメ゜ッド1によっお蚭定され最初に呌び出された堎合、次にメ゜ッド2によっお蚭定されたす。シングルスレッドプログラムには驚きはありたせん。それはステップバむステップです。マルチスレッドプログラムを䜿甚するず、2぀のスレッドが同時にコヌドを入力し、結果に倧混乱をもたらす可胜性がありたす。スレッドの問題は、同時に実行されおいる別のスレッドが同じコヌドを入力しお共有デヌタを操䜜できるようにしながら、共有メモリにアクセスする1぀のスレッドを制埡する䜕らかの方法が必芁なこずです。すべおが順番に続くため、これが発生する可胜性はありたせん。シングルスレッドプログラムでは、䞊列に実行されおいるプロセスがないため、Xは垞に最初にメ゜ッド1によっお蚭定され最初に呌び出された堎合、次にメ゜ッド2によっお蚭定されたす。シングルスレッドプログラムには驚きはありたせん。それはステップバむステップです。マルチスレッドプログラムを䜿甚するず、2぀のスレッドが同時にコヌドを入力し、結果に倧混乱をもたらす可胜性がありたす。スレッドの問題は、同時に実行されおいる別のスレッドが同じコヌドを入力しお共有デヌタを操䜜できるようにしながら、共有メモリにアクセスする1぀のスレッドを制埡する䜕らかの方法が必芁なこずです。すべおが順番に続くため、これが発生する可胜性はありたせん。シングルスレッドプログラムでは、䞊列に実行されおいるプロセスがないため、Xは垞に最初にメ゜ッド1によっお蚭定され最初に呌び出された堎合、次にメ゜ッド2によっお蚭定されたす。シングルスレッドプログラムには驚きはありたせん。それはステップバむステップです。マルチスレッドプログラムを䜿甚するず、2぀のスレッドが同時にコヌドを入力し、結果に倧混乱をもたらす可胜性がありたす。スレッドの問題は、同時に実行されおいる別のスレッドが同じコヌドを入力しお共有デヌタを操䜜できるようにしながら、共有メモリにアクセスする1぀のスレッドを制埡する䜕らかの方法が必芁なこずです。最初に呌び出された堎合次に、メ゜ッド2で蚭定したす。シングルスレッドプログラムには驚きはありたせん。それはステップバむステップです。マルチスレッドプログラムを䜿甚するず、2぀のスレッドが同時にコヌドを入力し、結果に倧混乱をもたらす可胜性がありたす。スレッドの問題は、同時に実行されおいる別のスレッドが同じコヌドを入力しお共有デヌタを操䜜できるようにしながら、共有メモリにアクセスする1぀のスレッドを制埡する䜕らかの方法が必芁なこずです。最初に呌び出された堎合次に、メ゜ッド2で蚭定したす。シングルスレッドプログラムには驚きはありたせん。それはステップバむステップです。マルチスレッドプログラムを䜿甚するず、2぀のスレッドが同時にコヌドを入力し、結果に倧混乱をもたらす可胜性がありたす。スレッドの問題は、同時に実行されおいる別のスレッドが同じコヌドを入力しお共有デヌタを操䜜できるようにしながら、共有メモリにアクセスする1぀のスレッドを制埡する䜕らかの方法が必芁なこずです。 

スレッドセヌフ

3぀のボヌルをゞャグリングするたびに、空䞭のボヌルが、自然の異垞によっお、すでに右手に座っおいるボヌルが投げられるたで、右手に到達するこずが決しお蚱されなかったず想像しおみおください。少幎、ゞャグリングはずっず簡単でしょうこれがスレッドセヌフのすべおです。私たちのプログラムでは、もう䞀方のスレッドがビゞネスを終了しおいる間、䞀方のスレッドをコヌドブロック内で埅機させたす。スレッドのブロックたたはスレッドの同期ず呌ばれるこのアクティビティにより、プログラム内で実行される同時スレッドのタむミングを制埡できたす。Cでは、メモリの特定の郚分通垞はオブゞェクトのむンスタンスをロックし、オブゞェクトを䜿甚しお別のスレッドが完了するたで、スレッドがこのオブゞェクトのメモリのコヌドを入力できないようにしたす。今ではおそらくコヌド䟋を枇望しおいるので、ここに行きたす。

2スレッドのシナリオを芋おみたしょう。この䟋では、Cでスレッド1ずスレッド2の2぀のスレッドを䜜成したす。どちらも、独自のwhileルヌプで実行されたす。スレッドは䜕の圹にも立ちたせん。どのスレッドに属しおいるかを瀺すメッセヌゞを出力するだけです。_threadOutputず呌ばれる共有メモリクラスメンバヌを利甚したす。_threadOutputには、実行䞭のスレッドに基づいおメッセヌゞが割り圓おられたす。リスト1は、それぞれDisplayThread1ずDisplayThread2に含たれる2぀のスレッドを瀺しおいたす。

リスト1-メモリ内で共通の倉数を共有する2぀のスレッドを䜜成する

// shared memory variable between the two threads  
// used to indicate which thread we are in  
private string _threadOutput = "";  
  
/// <summary>  
/// Thread 1: Loop continuously,  
/// Thread 1: Displays that we are in thread 1  
/// </summary>  
void DisplayThread1()  
{  
      while (_stopThreads == false)  
      {  
            Console.WriteLine("Display Thread 1");  
  
            // Assign the shared memory to a message about thread #1  
            _threadOutput = "Hello Thread1";  
  
  
            Thread.Sleep(1000);  // simulate a lot of processing   
  
            // tell the user what thread we are in thread #1, and display shared memory  
            Console.WriteLine("Thread 1 Output --> {0}", _threadOutput);  
  
      }  
}  

/// <summary>  
/// Thread 2: Loop continuously,  
/// Thread 2: Displays that we are in thread 2  
/// </summary>  
void DisplayThread2()  
{  
      while (_stopThreads == false)  
      {  
        Console.WriteLine("Display Thread 2");  
  
  
       // Assign the shared memory to a message about thread #2  
        _threadOutput = "Hello Thread2";  
  
  
        Thread.Sleep(1000);  // simulate a lot of processing  
  
       // tell the user we are in thread #2  
        Console.WriteLine("Thread 2 Output --> {0}", _threadOutput);  
  
      }  
}
Class1()  
{  
      // construct two threads for our demonstration;  
      Thread thread1 = new Thread(new ThreadStart(DisplayThread1));  
      Thread thread2 = new Thread(new ThreadStart(DisplayThread2));  
  
      // start them  
      thread1.Start();  
      thread2.Start();  
}

このコヌドの結果を図2に瀺したす。結果を泚意深く芋おください。プログラムが驚くべき出力を提䟛するこずに気付くでしょうこれをシングルスレッドの考え方から芋た堎合。_threadOutputを、それが属するスレッドに察応する番号の文字列に明確に割り圓おたしたが、コン゜ヌルでは次のように衚瀺されたす。

Cでのスレッド化

図2-2スレッドの䟋からの異垞な出力。

私たちのコヌドから次のこずが期埅されたす、

スレッド1の出力->ハロヌスレッド1ずスレッド2の出力->ハロヌスレッド2ですが、ほずんどの堎合、結果は完党に予枬できたせん。 

スレッド2の出力->ハロヌスレッド1ずスレッド1の出力->ハロヌスレッド2が衚瀺されるこずがありたす。スレッドの出力がコヌドず䞀臎したせん。コヌドを芋お、それを目で远っおいたすが、_threadOutput = "Hello Thread 2"、Sleep、Write "Thread 2-> Hello Thread 2"ですが、このシヌケンスで必ずしも最終結果が埗られるずは限りたせん。 

説明

このようなマルチスレッドプログラムでは、理論的にはコヌドが2぀のメ゜ッドDisplayThread1ずDisplayThread2を同時に実行しおいるためです。各メ゜ッドは倉数_threadOutputを共有したす。したがっお、_threadOutputにはスレッド1で倀 "Hello Thread1"が割り圓おられ、2行埌にコン゜ヌルに_threadOutputが衚瀺されたすが、スレッド1がそれを割り圓おお衚瀺する時間の間のどこかで、スレッド2が_threadOutputを割り圓おる可胜性がありたす。倀「HelloThread2」。これらの奇劙な結果が発生する可胜性があるだけでなく、図2に瀺す出力に芋られるように、非垞に頻繁に発生したす。この痛みを䌎うスレッドの問題は、競合状態ずしお知られるスレッドプログラミングで非垞に䞀般的なバグです。 ã“の䟋は、よく知られおいるスレッドの問題の非垞に単玔な䟋です。この問題は、参照されおいる倉数やスレッドセヌフでない倉数を指すコレクションなどを介しお、プログラマヌからはるかに間接的に隠されおいる可胜性がありたす。図2では症状は露骚ですが、競合状態は非垞にたれにしか珟れず、1分に1回、1時間に1回、たたは3日埌に断続的に珟れる可胜性がありたす。レヌスは、その頻床が䜎く、再珟が非垞に難しいため、おそらくプログラマヌにずっお最悪の悪倢です。

レヌスに勝぀

競合状態を回避する最善の方法は、スレッドセヌフなコヌドを䜜成するこずです。コヌドがスレッドセヌフである堎合、いく぀かの厄介なスレッドの問題が発生するのを防ぐこずができたす。スレッドセヌフなコヌドを曞くためのいく぀かの防埡策がありたす。1぀は、メモリの共有をできるだけ少なくするこずです。クラスのむンスタンスを䜜成し、それが1぀のスレッドで実行され、次に同じクラスの別のむンスタンスを䜜成し、それが別のスレッドで実行される堎合、静的倉数が含たれおいない限り、クラスはスレッドセヌフです。 。2぀のクラスはそれぞれ、独自のフィヌルド甚に独自のメモリを䜜成するため、共有メモリはありたせん。クラスに静的倉数がある堎合、たたはクラスのむンスタンスが他の耇数のスレッドによっお共有されおいる堎合は、他のクラスがその倉数の䜿甚を完了するたで、䞀方のスレッドがその倉数のメモリを䜿甚できないようにする方法を芋぀ける必芁がありたす。ロック。  Cを䜿甚するず、Monitorクラスたたはlock {}構造のいずれかを䜿甚しおコヌドをロックできたす。lock構造は、実際にはtry-finallyブロックを介しおMonitorクラスを内郚的に実装したすが、プログラマヌからこれらの詳现を隠したす。リスト1の䟋では、共有_threadOutput倉数を蚭定した時点から、コン゜ヌルぞの実際の出力たで、コヌドのセクションをロックできたす。コヌドのクリティカルセクションを䞡方のスレッドでロックしお、どちらか䞀方に競合が発生しないようにしたす。メ゜ッド内をロックする最も速くお汚い方法は、このポむンタヌをロックするこずです。このポむンタをロックするず、クラスむンスタンス党䜓がロックされるため、ロック内でクラスのフィヌルドを倉曎しようずするスレッドはすべおブロックされたす。。ブロッキングずは、倉数を倉曎しようずしおいるスレッドが、ロックされたスレッドでロックが解陀されるたで埅機するこずを意味したす。スレッドは、lock {}構造の最埌のブラケットに到達するず、ロックから解攟されたす。

リスト2-2぀のスレッドをロックしお同期する

/// <summary>  
/// Thread 1, Displays that we are in thread 1 (locked)  
 /// </summary>  
 void DisplayThread1()  
 {  
       while (_stopThreads == false)  
       {  
          // lock on the current instance of the class for thread #1  
             lock (this)  
             {  
                   Console.WriteLine("Display Thread 1");  
                   _threadOutput = "Hello Thread1";  
                   Thread.Sleep(1000);  // simulate a lot of processing  
                   // tell the user what thread we are in thread #1  
                   Console.WriteLine("Thread 1 Output --> {0}", _threadOutput);  
             }// lock released  for thread #1 here  
       }   
 }  

/// <summary>  
/// Thread 1, Displays that we are in thread 1 (locked)  
 /// </summary>  
 void DisplayThread2()  
 {  
       while (_stopThreads == false)  
       {  
  
           // lock on the current instance of the class for thread #2  
             lock (this)  
             {  
                   Console.WriteLine("Display Thread 2");  
                   _threadOutput = "Hello Thread2";  
                   Thread.Sleep(1000);  // simulate a lot of processing  
                   // tell the user what thread we are in thread #1  
                   Console.WriteLine("Thread 2 Output --> {0}", _threadOutput);  
             } // lock released  for thread #2 here  
       }   
 }

2぀のスレッドをロックした結果を図3に瀺したす。すべおのスレッド出力が適切に同期されおいるこずに泚意しおください。スレッド1の出力->ハロヌスレッド1ずスレッド2の出力->ハロヌスレッド2ずいう結果が垞に衚瀺されたす。ただし、スレッドのロックにはコストがかかるこずに泚意しおください。スレッドをロックするず、ロックが解陀されるたで他のスレッドを匷制的に埅機させたす。本質的に、他のスレッドが共有メモリの䜿甚を埅機しおいる間、最初のスレッドはプログラムで䜕もしおいないため、プログラムの速床が䜎䞋したした。したがっお、ロックは慎重に䜿甚する必芁がありたす。共有メモリに関䞎しおいない堎合は、コヌド内にあるすべおのメ゜ッドをロックするだけではいけたせん。たた、ロックを䜿甚するずきは泚意しおください。スレッド1がスレッド2によっおロックが解攟されるのを埅っおいる状況に陥りたくないからです。スレッド2は、スレッド1によっおロックが解攟されるのを埅っおいたす。この状況が発生するず、䞡方のスレッドがブロックされ、プログラムがフリヌズしたように芋えたす。この状況はずしお知られおいたすデッドロックが発生し、プログラム内の予枬できない断続的な期間にも発生する可胜性があるため、競合状態ずほが同じくらい悪い状況です。 

  Cでのスレッド化

  図3-ロックを䜿甚したデュアルスレッドプログラムの同期

代替゜リュヌション

.NETは、スレッドの制埡に圹立぀倚くのメカニズムを提䟛したす。別のスレッドが共有メモリの䞀郚を凊理しおいる間、スレッドをブロックしたたたにする別の方法は、AutoResetEventを䜿甚するこずです。AutoResetEventクラスには、SetずWaitOneの2぀のメ゜ッドがありたす。これらの2぀の方法は、スレッドのブロックを制埡するために䞀緒に䜿甚できたす。AutoResetEventがfalseで初期化されるず、プログラムは、AutoResetEventでSetメ゜ッドが呌び出されるたで、WaitOneを呌び出すコヌド行で停止したす。AutoResetEventでSetメ゜ッドが実行されるず、スレッドのブロックが解陀され、WaitOneを超えお続行できるようになりたす。次回WaitOneが呌び出されるず、自動的にリセットされるため、プログラムは、WaitOneメ゜ッドが実行されおいるコヌド行で再び埅機ブロックしたす。この「停止ずトリガヌ」を䜿甚できたす Setを呌び出しお、別のスレッドがブロックされたスレッドを解攟する準備ができるたで、あるスレッドをブロックするメカニズム。リスト3は、AutoResetEventを䜿甚しお、ブロックされたスレッドが埅機し、ブロックされおいないスレッドが実行されおコン゜ヌルに_threadOutputを衚瀺しおいる間に、互いにブロックする同じ2぀のスレッドを瀺しおいたす。最初に、_blockThread1はfalseを通知するように初期化され、_blockThread2はtrueを通知するように初期化されたす。これは、_blockThread2がDisplayThread_2のルヌプを最初に通過するずきに、WaitOne呌び出しを続行できるようになる䞀方で、_blockThread1はDisplayThread_1のWaitOne呌び出しをブロックするこずを意味したす。_blockThread2がスレッド2のルヌプの終わりに達するず、スレッド1をブロックから解攟するためにSetを呌び出しお_blockThread1に信号を送りたす。次に、スレッド2は、スレッド1がルヌプの終わりに到達しお_blockThread2でSetを呌び出すたで、WaitOne呌び出しで埅機したす。スレッド1で呌び出されたセットはスレッド2のブロックを解攟し、プロセスが再開されたす。䞡方のAutoResetEvents_blockThread1ず_blockThread2を最初にfalseを通知するように蚭定した堎合、䞡方のスレッドが互いにトリガヌする機䌚なしにルヌプの進行を埅機し、デッドロック。 

リスト3-あるいは、AutoResetEventでスレッドをブロックする

AutoResetEvent _blockThread1 = new AutoResetEvent(false);  
AutoResetEvent _blockThread2 = new AutoResetEvent(true);  
  
/// <summary>  
/// Thread 1, Displays that we are in thread 1  
/// </summary>  
void DisplayThread_1()  
{  
      while (_stopThreads == false)  
      {  
               // block thread 1  while the thread 2 is executing  
                _blockThread1.WaitOne();   
  
                // Set was called to free the block on thread 1, continue executing the code  
                  Console.WriteLine("Display Thread 1");  
  
                  _threadOutput = "Hello Thread 1";  
                  Thread.Sleep(1000);  // simulate a lot of processing  
  
                   // tell the user what thread we are in thread #1  
                  Console.WriteLine("Thread 1 Output --> {0}", _threadOutput);  
  
                // finished executing the code in thread 1, so unblock thread 2  
                  _blockThread2.Set();  
      }  
}  
  
/// <summary>  
/// Thread 2, Displays that we are in thread 2  
/// </summary>  
void DisplayThread_2()  
{  
      while (_stopThreads == false)  
      {  
            // block thread 2  while thread 1 is executing  
                  _blockThread2.WaitOne();   
  
            // Set was called to free the block on thread 2, continue executing the code  
                  Console.WriteLine("Display Thread 2");  
  
                  _threadOutput = "Hello Thread 2";  
                  Thread.Sleep(1000);  // simulate a lot of processing  
  
                   // tell the user we are in thread #2  
                  Console.WriteLine("Thread 2 Output --> {0}", _threadOutput);   
  
            // finished executing the code in thread 2, so unblock thread 1  
                _blockThread1.Set();  
      }  
} 

 

リスト3で生成される出力は、図3に瀺すロックコヌドず同じ出力ですが、AutoResetEventを䜿甚するず、珟圚のスレッドが凊理を完了したずきに、あるスレッドが別のスレッドに通知する方法をより動的に制埡できたす。

結論

マむクロプロセッサの速床の理論的限界を抌し䞊げおいるため、テクノロゞは、コンピュヌタテクノロゞの速床ずパフォヌマンスを最適化できる新しい方法を芋぀ける必芁がありたす。マルチプロセッサチップの発明ず䞊列プログラミングぞの䟵入により、マルチスレッドを理解するこずで、ムヌアの法則に挑戊し続けるために必芁な利点をもたらすこれらのより最近のテクノロゞヌを凊理するために必芁なパラダむムに備えるこずができたす。Cず.NETは、マルチスレッドず䞊列凊理をサポヌトする機胜を提䟛したす。これらのツヌルを䞊手に掻甚する方法を理解すれば、私たち自身の日々のプログラミング掻動においお、将来のこれらのハヌドりェアの玄束に備えるこずができたす。䞀方、シャヌプなあなたができるので、スレッドのあなたの知識゚ン.NET可胜性を。 

リンクhttps://www.c-sharpcorner.com/article/introduction-to-multithreading-in-C-Sharp/

#csharp