Connor Mills

Connor Mills

1668238299

YAML vs JSON vs XML: Which One to Choose?

In this tutorial, you will learn how YAML compares to XML and JSON - two languages also used for creating configuration files. XML VS JSON VS YAML - What's The Difference?

YAML is one of the most popular languages for writing configuration files.

In this article, you will learn how YAML compares to XML and JSON - two languages also used for creating configuration files.

You will also learn some of the rules and features of the language, along with its basic syntax.

Here is what we will cover:

  1. What is YAML?
  2. XML VS JSON VS YAML - What's The Difference?
    1. XML
    2. JSON
    3. YAML
  3. Features and basic rules of YAML
    1. How to create a YAML file
    2. Multi-document support in YAML
    3. Indentation in YAML
    4. Tabs in YAML
    5. Whitespace in YAML
    6. Explicit data types in YAML
  4. An introduction to YAML syntax
    1. Scalars
    2. Collections

What is YAML?

YAML stands for YAML Ain't Markup Language, but it originally stood for Yet Another Markup Language.

YAML is a human-readable data serialization language, just like XML and JSON.

Serialization is a process where one application or service that has different data structures and is written in a different set of technologies can transfer data to another application using a standard format.

In other words, serialization is about translating, converting, and wrapping up a data structure in another format.

The data in the new format can be stored in a file or transmitted to another application or service over a network.

YAML is a widely used format for writing configuration files for different DevOps tools, programs, and applications because of its human-readable and intuitive syntax.

XML VS JSON VS YAML - What's The Difference?

XML, JSON, and YAML are all used for creating configuration files and transferring data between applications.

Each language has its advantages and disadvantages.

Now, let's see some of the characteristics of the three languages. You will also see an example of how the same code is written in each language to demonstrate the high-level differences in their syntax.

XML

XML, which stands for Extensible Markup Language, was first introduced in 1996 and was designed for general-purpose use.

XML is a generalized markup language. It offers a structured yet flexible syntax and a defined document schema. This makes it a good choice when working with complex configurations that require a structured format and finer control over schema validation to ensure configurations always have the correct format.

With that said, XML's syntax can be verbose, redundant, and harder to read in comparison with other serialization languages.

<Employees>
    <Employee> 
        <name> John Doe </name>
        <department> Engineering </department>
        <country> USA </country>
    </Employee>
     <Employee> 
        <name> Kate Kateson </name>
        <department> IT Support </department>
        <country> United Kingdom </country>
    </Employee>
</Employees>

JSON

JSON stands for JavaScript Object Notation and has been around since the early 2000s.

JSON was initially inspired by the JavaScript programming language, but it is not tied to only one language. Instead, it is a language-independent format.

Most modern programming languages have libraries for parsing and generating JSON data.

JSON offers a much more readable, human-friendly, compact, and simple syntax compared to XML. It makes for a great format for storing and transferring information between web applications and servers over a network.

With that said, it may not offer the best support for complex configurations.

{
	"Employees": [
    {
			"name": "John Doe",
			"department": "Engineering",
			"country": "USA"
		},

		{
			"name": "Kate Kateson",
			"department": "IT support",
			"country": "United Kingdom"
		}
	]
}

YAML

YAML, originally known as Yet Another Markup Language, was created in 2001 but now stands for YAML Ain't Markup Language.

YAML is an official strict superset of JSON despite looking very different from JSON.

YAML can do everything that JSON can and more. A valid YAML file can contain JSON, and JSON can transform into YAML.

YAML has the most human-readable, intuitive, and compact syntax for defining configurations compared to XML and JSON.

YAML uses indentation to define structure in the file, which is helpful if you are used to writing Python code and are familiar with the indentation style the language uses.

With that said, if you don't get the indentation and format right, it can lead to validation errors, making it not the friendliest for beginners.

Employees:
- name: John Doe
  department: Engineering
  country: USA
- name: Kate Kateson
  department: IT support
  country: United Kingdom

Features and Basic Rules of YAML

Now, let's go over some of the basic rules and features of the language.

How to Create a YAML File

To create a YAML file, use either the .yaml or .yml file extension.

Multi-Document Support in YAML

Before writing any YAML code, you can add three dashes (---) at the start of the file:

---
Employees:
- name: John Doe
  department: Engineering
  country: USA
- name: Kate Kateson
  department: IT support
  country: United Kingdom

YAML allows you to have multiple YAML documents in a singe YAML file, making file organization much easier.

Separate each document with three dashes (---):

---
Employees:
- name: John Doe
  department: Engineering
  country: USA
- name: Kate Kateson
  department: IT support
  country: United Kingdom
---
Fruit:
 - Oranges
 - Pears
 - Apples

You can also use three dots (...) to mark the end of the document:

---
Employees:
- name: John Doe
  department: Engineering
  country: USA
- name: Kate Kateson
  department: IT support
  country: United Kingdom
...

Indentation in YAML

In YAML, there is an emphasis on indentation and line separation to denote levels and structure in data. The indentation system is quite similar to the one Python uses.

YAML doesn't use symbols such as curly braces, square brackets, or opening or closing tags - just indentation.

Tabs in YAML

YAML doesn't allow you to use any tabs when creating indentation - use spaces instead.

Whitespace in YAML

Whitespace doesn't matter as long as child elements are indented inside the parent element.

How to Write A Comment in YAML

To add a comment to comment out a line of code, use the # character:

---
# Employees in my company
Employees:
- name: John Doe
  department: Engineering
  country: USA
- name: Kate Kateson
  department: IT support
  country: United Kingdom

Explicit Data Types in YAML

Although YAML auto-detects the data types in a file, you can specify the type of data you want to use.

To explicitly specify the type of data, use the !! symbol and the name of the data type before the value:

# parse this value as a string
date: !!str 2022-11-11

## parse this value as a float (it will be 1.0 instead of 1)
fave_number: !!float 1

An Introduction to YAML Syntax

Scalars

Scalars in YAML are the data on the page - strings, numbers, booleans, and nulls.

Let's see some examples of how to use each one.

In YAML, strings in some cases can be left unquoted, but you can also wrap them in single (' ') or double (" ") quotation marks:

A string in YAML!

'A string in YAML!'

"A string in YAML!"

If you want to write a string that spans across multiple lines and you want to preserve the line breaks, use the pipe symbol (|):

|
 I am message that spans multiple lines
 I go on and on across lines
 and lines
 and more lines

Make sure that the message is indented!

Alternatively, if you have a string in a YAML file that spans across multiple lines for readability, but you want the parser to interpret it as a single line string, you can use the > character, which will replace each line break with a space:

>
 I am message that spans
 multiple lines
 but I will be parsed
 on one line

Again, make sure you don't forget to indent the message!

Numbers express numerical data, and in YAML, these include integers (whole numbers), floats (numbers with a decimal point), exponentials, octals, and hexadecimals:

# integer
19

# float 
8.7

# exponential
4.5e+13

# octal 
0o23

# hexadecimal
0xFF

Booleans in YAML, and other programming languages, have one of two states and are expressed with either true or false.

Words like true and false are keywords in YAML, so don't surround them with quotation marks if you want them interpreted as booleans.

Lastly, Null values are expressed with the keyword null or the tilde character, ~.

Collections

More often than not, you will not be writing simple scalars in your YAML files - you will be using collections instead.

Collections in YAML can be:

  • Sequences (lists/arrays)
  • Mappings (dictionaries/hashes)

To write a sequence, use a dash (-) followed by a space:

- HTML
- CSS
- JavaScript

Each item in the sequence (list) is placed on a separate line, with a dash in front of the value.

And each item in the list is on the same level.

That said, you can create a nested sequence (remember, use spaces - not tabs - to create the levels of indentation):

- HTML
- CSS
- JavaScript
 - React
 - Angular
 - Vue

In the sequence above, React, Angular and Vue are sub-items of the item JavaScript.

Mappings allow you to list keys with values. Key/value pairs are the building blocks of YAML documents.

Use a colon (:) followed by a space to create key/value pairs:

Employees:
 name: John Doe
 age: 23
 country: USA

In the example above, a name gets assigned to a specific value.

The value John Doe gets mapped (or assigned) to the name key, the value 23 gets mapped to the age key, and the value USA gets mapped to the country key. Altogether, these create an object.

You can also use a mapping with a sequence.

For example, taking the example sequence from earlier on, here is how you would build a list of frontend_languages:

frontend_languages:
 - HTML
 - CSS
 - JavaScript
  - React
  - Angular
  - Vue

In the example above, I created a list of frontend_languages, where there are multiple values under the same key, frontend_languages.

Similarly, you can create a list of objects:

Employees:
- name: John Doe
  department: Engineering
  country: USA
- name: Kate Kateson
  department: IT support
  country: United Kingdom

Conclusion

Hopefully, this article was helpful and gave you insight into what YAML is, what the syntax of the language looks like, and how it differs from XML and JSON.

Thank you for reading, and happy coding!

Original article source at https://www.freecodecamp.org

#yaml #json #xml 

YAML vs JSON vs XML: Which One to Choose?

Fq: Jq for Binary Formats

fq

Tool, language and decoders for working with binary data.

fq is inspired by the well known jq tool and language and allows you to work with binary formats the same way you would using jq. In addition it can present data like a hex viewer, transform, slice and concatenate binary data. It also supports nested formats and has an interactive REPL with auto-completion.

It was originally designed to query, inspect and debug media codecs and containers like mp4, flac, mp3, jpeg. Since then it has been extended to support a variety of formats like executables, packet captures (including TCP reassembly) and serialization formats like JSON, YAML, XML, ASN1 BER, Avro, CBOR, protobuf. In addition it also has functions to work with URL:s, convert to/from hex, number bases, search for things etc.

In summary it aims to be jq, hexdump, dd and gdb for files combined into one.

NOTE: fq is still early in development so things might change, be broken or do not make sense. That also means that there is a great opportunity to help out!

fq demo

Goals

  • Make binaries accessible, queryable and sliceable.
  • Nested formats and bit-oriented decoding.
  • Quick and comfortable CLI tool.
  • Bits and bytes transformations.

Hopes

  • Make it useful enough that people want to help improve it.
  • Inspire people to create similar tools.

Supported formats

aac_frame, adts, adts_frame, amf0, apev2, ar, asn1_ber, av1_ccr, av1_frame, av1_obu, avc_annexb, avc_au, avc_dcr, avc_nalu, avc_pps, avc_sei, avc_sps, avro_ocf, bencode, bitcoin_blkdat, bitcoin_block, bitcoin_script, bitcoin_transaction, bplist, bsd_loopback_frame, bson, bzip2, cbor, csv, dns, dns_tcp, elf, ether8023_frame, exif, fairplay_spc, flac, flac_frame, flac_metadatablock, flac_metadatablocks, flac_picture, flac_streaminfo, gif, gzip, hevc_annexb, hevc_au, hevc_dcr, hevc_nalu, hevc_pps, hevc_sps, hevc_vps, html, icc_profile, icmp, icmpv6, id3v1, id3v11, id3v2, ipv4_packet, ipv6_packet, jpeg, json, jsonl, macho, macho_fat, markdown, matroska, mp3, mp3_frame, mp4, mpeg_asc, mpeg_es, mpeg_pes, mpeg_pes_packet, mpeg_spu, mpeg_ts, msgpack, ogg, ogg_page, opus_packet, pcap, pcapng, png, prores_frame, protobuf, protobuf_widevine, pssh_playready, raw, rtmp, sll2_packet, sll_packet, tar, tcp_segment, tiff, toml, udp_datagram, vorbis_comment, vorbis_packet, vp8_frame, vp9_cfm, vp9_frame, vpx_ccr, wasm, wav, webp, xing, xml, yaml, zip

It can also work with some common text formats like URL:s, hex, base64, PEM etc and for some serialization formats like XML, YAML etc it can transform both from and to jq values.

For details see formats.md and usage.md.

Usage

Basic usage is fq . file.

For details see usage.md

Presentations

Install

Use one of the methods listed below or download release for your platform. Unarchive it and move the executable to PATH etc.

On macOS if you don't install using a method below you might have to manually allow the binary to run. This can be done by trying to run the binary, ignore the warning and then go into security preference and allow it. Or you can run this command:

xattr -d com.apple.quarantine fq && spctl --add fq

Homebrew

# install latest release
brew install wader/tap/fq

MacPorts

On macOS, fq can also be installed via MacPorts. More details here.

sudo port install fq

Windows

fq can be installed via scoop.

scoop install fq

Arch Linux

fq can be installed from the community repository using pacman:

pacman -S fq

You can also build and install the development (VCS) package using an AUR helper:

paru -S fq-git

Nix

nix-shell -p fq

FreeBSD

Use the fq port.

Alpine

Currently in edge testing but should work fine in stable also.

apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing fq

Build from source

Make sure you have go 1.18 or later installed.

To install directly from git repository do:

# build and install latest release
go install github.com/wader/fq@latest

# or build and install latest master
go install github.com/wader/fq@master

# copy binary to $PATH if needed
cp "$(go env GOPATH)/bin/fq" /usr/local/bin

To run and run tests from source directory:

# run all tests and build binary
make test fq
# it's also possible to use go run
go run fq.go

TODO and ideas

See TODO.md

Development and adding a new decoder

See dev.md

Thanks and related projects

This project would not have been possible without itchyny's jq implementation gojq. I also want to thank HexFiend for inspiration and ideas and stedolan for inventing the jq language.

Similar or related projects

  • HexFiend Hex editor for macOS with format template support.
  • binspector Binary format analysis tool with query langauge and REPL.
  • kaitai Declarative binary format parsing.
  • Wireshark Decodes network traffic (tip: tshark -T json).
  • MediaInfo Analyze media files (tip mediainfo --Output=JSON and mediainfo --Details=1).
  • GNU poke The extensible editor for structured binary data.
  • ffmpeg/ffprobe Powerful media libraries and tools.
  • hexdump Hex viewer tool.
  • hex Interactive hex viewer with format support via lua.
  • hachoir General python library for working binary data.
  • scapy Decode/Encode formats, focus on network protocols.

Download Details:

Author: Wader
Source Code: https://github.com/wader/fq 
License: View license

#go #golang #cli #yaml #json 

Fq: Jq for Binary Formats

Countries: World Countries in JSON, CSV, XML and Yaml

World countries in JSON, CSV, XML and YAML.

Countries data

This repository contains a list of world countries, as defined by ISO Standard 3166-1, in JSON, CSV, XML and YAML. Warning: not all entities in this project are independent countries; refer to the independent property to know if the country is considered a sovereign state.

Each line contains the country:

  • name
    • common - common name in english
    • official - official name in english
    • native - list of all native names
      • key: three-letter ISO 639-3 language code
      • value: name object
        • key: official - official name translation
        • key: common - common name translation
  • country code top-level domain (tld)
  • code ISO 3166-1 alpha-2 (cca2)
  • code ISO 3166-1 numeric (ccn3)
  • code ISO 3166-1 alpha-3 (cca3)
  • code International Olympic Committee (cioc)
  • ISO 3166-1 independence status (independent) (denotes the country is considered a sovereign state)
  • ISO 3166-1 assignment status (status)
  • UN Member status (unMember)
  • currencies - list of all currencies
    • key: ISO 4217 currency code
    • value: currency object
      • key: name name of the currency
      • key: symbol symbol of the currency
  • International Direct Dialing info (idd)
    • root - the root geographical code prefix. e.g. +6 for New Zealand, +4 for UK.
    • suffixes - list of all suffixes assigned to this country. 4 for NZ, 809, 829, and 849 for Dominican Republic.
  • capital city(ies) (capital)
  • alternative spellings (altSpellings)
  • region
  • subregion
  • list of official languages (languages)
    • key: three-letter ISO 639-3 language code
    • value: name of the language in english
  • list of name translations (translations)
    • key: three-letter ISO 639-3 language code
    • value: name object
      • key: official - official name translation
      • key: common - common name translation
  • latitude and longitude (latlng)
  • demonyms - name of residents, translated & genderized
    • key: three-letter ISO 639-3 language code
    • value: genderized demonym object
      • key: f (female) or m (male)
      • value: genderized demonym translation
  • landlocked status (landlocked)
  • land borders (borders)
  • land area in km² (area)
  • Emoji flag (flag)
  • calling codes (callingCodes)

Additional data

The data folder contains additional data such as the countries GeoJSON outlines and flags in SVG format.

Examples

JSON

{
    "name": {
        "common": "Austria",
        "official": "Republic of Austria",
        "native": {
            "bar": {
                "official": "Republik Österreich",
                "common": "Österreich"
            }
        }
    },
    "tld": [".at"],
    "cca2": "AT",
    "ccn3": "040",
    "cca3": "AUT",
    "cioc": "AUT",
    "independent": true,
    "status": "officially-assigned",
    "unMember": true,
    "currencies": {"EUR": {"name": "Euro", "symbol": "\u20ac"}},
    "idd": {
        "root": "+4",
        "suffixes": ["3"]
    },
    "capital": ["Vienna"],
    "altSpellings": ["AT", "Osterreich", "Oesterreich"],
    "region": "Europe",
    "subregion": "Western Europe",
    "languages": {
        "bar": "Austro-Bavarian German"
    },
    "translations": {
        "cym": {"official": "Republic of Austria", "common": "Awstria"},
        "deu": {"official": "Republik Österreich", "common": "Österreich"},
        "fra": {"official": "République d'Autriche", "common": "Autriche"},
        "hrv": {"official": "Republika Austrija", "common": "Austrija"},
        "ita": {"official": "Repubblica d'Austria", "common": "Austria"},
        "jpn": {"official": "オーストリア共和国", "common": "オーストリア"},
        "nld": {"official": "Republiek Oostenrijk", "common": "Oostenrijk"},
        "por": {"official": "República da Áustria", "common": "Áustria"},
        "rus": {"official": "Австрийская Республика", "common": "Австрия"},
        "spa": {"official": "República de Austria", "common": "Austria"}
    },
    "latlng": [47.33333333, 13.33333333],
    "demonyms": {
        "fra": {
            "f": "Autrichienne",
            "m": "Autrichien"
        },
        "spa": {
            "f": "Austriaco",
            "m": "Austriaca"
        }
    },
    "landlocked": true,
    "borders": ["CZE", "DEU", "HUN", "ITA", "LIE", "SVK", "SVN", "CHE"],
    "area": 83871,
    "callingCodes": ["+43"]
    "flag": "\ud83c\udde6\ud83c\uddf9"
}

GeoJSON and TopoJSON outlines

See an example for Germany: GeoJSON or TopoJSON.

CSV

"name";"tld";"cca2";"ccn3";"cca3";"cioc";"currency";"idd";"capital";"altSpellings";"region";"subregion";"languages";"translations";"latlng";"demonym";"landlocked";"borders";"area"
⋮
"Aruba,Aruba,Aruba,Aruba,Aruba,Aruba";".aw";"AW";"533";"ABW";"ARU";"AWG";"+2,97";"Oranjestad";"AW";"Americas";"Caribbean";"Dutch,Papiamento";"Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,アルバ,アルバ,Aruba,Aruba,Aruba,Aruba,Аруба,Аруба,Aruba,Aruba";"12.5,-69.96666666";"Aruban";"";"";"180"
"Afghanistan,Islamic Republic of Afghanistan,جمهوری اسلامی افغانستان,افغانستان,د افغانستان اسلامي جمهوریت,افغانستان,Owganystan Yslam Respublikasy,Owganystan";".af";"AF";"004";"AFG";"AFG";"AFN";"+9,3";"Kabul";"AF,Afġānistān";"Asia";"Southern Asia";"Dari,Pashto,Turkmen";"Islamic Republic of Afghanistan,Affganistan,Islamischen Republik Afghanistan,Afghanistan,Afganistanin islamilainen tasavalta,Afganistan,République islamique d'Afghanistan,Afghanistan,Islamska Republika Afganistan,Afganistan,Repubblica islamica dell'Afghanistan,Afghanistan,アフガニスタン·イスラム共和国,アフガニスタン,Islamitische Republiek Afghanistan,Afghanistan,República Islâmica do Afeganistão,Afeganistão,Исламская Республика Афганистан,Афганистан,República Islámica de Afganistán,Afganistán";"33,65";"Afghan";"1";"IRN,PAK,TKM,UZB,TJK,CHN";"652230"
"Angola,Republic of Angola,República de Angola,Angola";".ao";"AO";"024";"AGO";"ANG";"AOA";"+2,44";"Luanda";"AO,República de Angola,ʁɛpublika de an'ɡɔla";"Africa";"Middle Africa";"Portuguese";"Republic of Angola,Angola,Republik Angola,Angola,Angolan tasavalta,Angola,République d'Angola,Angola,Republika Angola,Angola,Repubblica dell'Angola,Angola,アンゴラ共和国,アンゴラ,Republiek Angola,Angola,República de Angola,Angola,Республика Ангола,Ангола,República de Angola,Angola";"-12.5,18.5";"Angolan";"";"COG,COD,ZMB,NAM";"1246700"
⋮

XML

<?xml version="1.0" encoding="UTF-8"?>
<countries>
  <country name="Aruba,Aruba,Aruba,Aruba,Aruba,Aruba" tld=".aw" cca2="AW" ccn3="533" cca3="ABW" cioc="ARU" currency="AWG" idd="+2,97" capital="Oranjestad" altSpellings="AW" region="Americas" subregion="Caribbean" languages="Dutch,Papiamento" translations="Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,Aruba,アルバ,アルバ,Aruba,Aruba,Aruba,Aruba,Аруба,Аруба,Aruba,Aruba" latlng="12.5,-69.96666666" demonym="Aruban" landlocked="" borders="" area="180"/>
  <country name="Afghanistan,Islamic Republic of Afghanistan,جمهوری اسلامی افغانستان,افغانستان,د افغانستان اسلامي جمهوریت,افغانستان,Owganystan Yslam Respublikasy,Owganystan" tld=".af" cca2="AF" ccn3="004" cca3="AFG" cioc="AFG" currency="AFN" idd="+9,3" capital="Kabul" altSpellings="AF,Afġānistān" region="Asia" subregion="Southern Asia" languages="Dari,Pashto,Turkmen" translations="Islamic Republic of Afghanistan,Affganistan,Islamischen Republik Afghanistan,Afghanistan,Afganistanin islamilainen tasavalta,Afganistan,République islamique d'Afghanistan,Afghanistan,Islamska Republika Afganistan,Afganistan,Repubblica islamica dell'Afghanistan,Afghanistan,アフガニスタン·イスラム共和国,アフガニスタン,Islamitische Republiek Afghanistan,Afghanistan,República Islâmica do Afeganistão,Afeganistão,Исламская Республика Афганистан,Афганистан,República Islámica de Afganistán,Afganistán" latlng="33,65" demonym="Afghan" landlocked="1" borders="IRN,PAK,TKM,UZB,TJK,CHN" area="652230"/>
  <country name="Angola,Republic of Angola,República de Angola,Angola" tld=".ao" cca2="AO" ccn3="024" cca3="AGO" cioc="ANG" currency="AOA" idd="+2,44" capital="Luanda" altSpellings="AO,República de Angola,ʁɛpublika de an'ɡɔla" region="Africa" subregion="Middle Africa" languages="Portuguese" translations="Republic of Angola,Angola,Republik Angola,Angola,Angolan tasavalta,Angola,République d'Angola,Angola,Republika Angola,Angola,Repubblica dell'Angola,Angola,アンゴラ共和国,アンゴラ,Republiek Angola,Angola,República de Angola,Angola,Республика Ангола,Ангола,República de Angola,Angola" latlng="-12.5,18.5" demonym="Angolan" landlocked="" borders="COG,COD,ZMB,NAM" area="1246700"/>
⋮
<countries>

YAML

- { name: { common: Aruba, official: Aruba, native: { nld: { official: Aruba, common: Aruba }, pap: { official: Aruba, common: Aruba } } }, tld: [.aw], cca2: AW, ccn3: '533', cca3: ABW, cioc: ARU, currency: [AWG], idd: { root: '+2', suffixes: ['97'] }, capital: Oranjestad, altSpellings: [AW], region: Americas, subregion: Caribbean, languages: { nld: Dutch, pap: Papiamento }, translations: { deu: { official: Aruba, common: Aruba }, fin: { official: Aruba, common: Aruba }, fra: { official: Aruba, common: Aruba }, hrv: { official: Aruba, common: Aruba }, ita: { official: Aruba, common: Aruba }, jpn: { official: アルバ, common: アルバ }, nld: { official: Aruba, common: Aruba }, por: { official: Aruba, common: Aruba }, rus: { official: Аруба, common: Аруба }, spa: { official: Aruba, common: Aruba } }, latlng: [12.5, -69.96666666], demonym: Aruban, landlocked: false, borders: {  }, area: 180 }
- { name: { common: Afghanistan, official: 'Islamic Republic of Afghanistan', native: { prs: { official: 'جمهوری اسلامی افغانستان', common: افغانستان }, pus: { official: 'د افغانستان اسلامي جمهوریت', common: افغانستان }, tuk: { official: 'Owganystan Yslam Respublikasy', common: Owganystan } } }, tld: [.af], cca2: AF, ccn3: '004', cca3: AFG, cioc: AFG, currency: [AFN], idd: { root: '+9', suffixes: ['3'] }, capital: Kabul, altSpellings: [AF, Afġānistān], region: Asia, subregion: 'Southern Asia', languages: { prs: Dari, pus: Pashto, tuk: Turkmen }, translations: { cym: { official: 'Islamic Republic of Afghanistan', common: Affganistan }, deu: { official: 'Islamischen Republik Afghanistan', common: Afghanistan }, fin: { official: 'Afganistanin islamilainen tasavalta', common: Afganistan }, fra: { official: 'République islamique d''Afghanistan', common: Afghanistan }, hrv: { official: 'Islamska Republika Afganistan', common: Afganistan }, ita: { official: 'Repubblica islamica dell''Afghanistan', common: Afghanistan }, jpn: { official: アフガニスタン·イスラム共和国, common: アフガニスタン }, nld: { official: 'Islamitische Republiek Afghanistan', common: Afghanistan }, por: { official: 'República Islâmica do Afeganistão', common: Afeganistão }, rus: { official: 'Исламская Республика Афганистан', common: Афганистан }, spa: { official: 'República Islámica de Afganistán', common: Afganistán } }, latlng: [33, 65], demonym: Afghan, landlocked: true, borders: [IRN, PAK, TKM, UZB, TJK, CHN], area: 652230 }
- { name: { common: Angola, official: 'Republic of Angola', native: { por: { official: 'República de Angola', common: Angola } } }, tld: [.ao], cca2: AO, ccn3: '024', cca3: AGO, cioc: ANG, currency: [AOA], idd: { root: '+2', suffixes: ['44'] }, capital: Luanda, altSpellings: [AO, 'República de Angola', 'ʁɛpublika de an''ɡɔla'], region: Africa, subregion: 'Middle Africa', languages: { por: Portuguese }, translations: { cym: { official: 'Republic of Angola', common: Angola }, deu: { official: 'Republik Angola', common: Angola }, fin: { official: 'Angolan tasavalta', common: Angola }, fra: { official: 'République d''Angola', common: Angola }, hrv: { official: 'Republika Angola', common: Angola }, ita: { official: 'Repubblica dell''Angola', common: Angola }, jpn: { official: アンゴラ共和国, common: アンゴラ }, nld: { official: 'Republiek Angola', common: Angola }, por: { official: 'República de Angola', common: Angola }, rus: { official: 'Республика Ангола', common: Ангола }, spa: { official: 'República de Angola', common: Angola } }, latlng: [-12.5, 18.5], demonym: Angolan, landlocked: false, borders: [COG, COD, ZMB, NAM], area: 1246700 }

Customising the output

The data files provided in the dist directory include all available fields, but is also possible to build a custom version of the data with certain fields excluded.

To do this, you will first need a working PHP installation, composer and a local copy of this repository. Once you have these, open a terminal in your local version of this project's root directory and run this command to install the necessary dependencies:

composer install

After this finishes, run the following command (here we will exclude the tld field from the output, but you can exclude any field you want):

php countries.php convert --exclude-field=tld

You can also exclude multiple fields:

php countries.php convert --exclude-field=tld --exclude-field=cca2

# Or using the shorter `-x` syntax:
php countries.php convert -x tld -x cca2

If you prefer to include only some fields (this can not be combined with --exclude-field):

php countries.php convert --include-field=name --include-field=area

# or using the shorter `-i` syntax:
php countries.php convert -i=name -i=area

The generated files are put into the dist directory, but you can change this to another existing directory:

mkdir foobar
php countries.php convert --output-dir=foobar

You can also choose to only generate some of the output formats:

mkdir foobar
php countries.php convert --format=json_unescaped --format=csv

# or using the shorter `-f` syntax:
php countries.php convert -f json_unescaped -f csv

Showcase

Projects using this dataset:

How to contribute?

Please refer to CONTRIBUTING.

To do

  • add the type of the country (country, sovereign state, public body, territory, etc.)
  • add missing translations
  • pull in data automatically from CLDR at build time (idea from @Munter, see #108)

Sources

https://www.currency-iso.org/ for currency codes.

Region and subregion are taken from https://github.com/hexorx/countries.

GeoJSON outlines come from http://thematicmapping.org/downloads/world_borders.php.

The rest comes from Wikipedia.

Credits

Thanks to:

Download Details:

Author: Mledoze
Source Code: https://github.com/mledoze/countries 
License: ODbL-1.0 license

#php #yaml #json #csv 

Countries: World Countries in JSON, CSV, XML and Yaml

4 Best YAML & ZIP Libraries with Swift

In this Swift article, we learn about YAML & ZIP: 4 Best YAML & ZIP Libraries with Swift

Table of contents:

YAML: 

  • YamlSwift - Load YAML and JSON documents.
  • Yams 🐧 - Sweet YAML parser.

ZIP:

  • Zip - Framework for zipping and unzipping files.
  • Zip Foundation - A library to create, read and modify ZIP archive files.

4 Best YAML & ZIP Libraries with Swift

  1. YamlSwift

Load YAML and JSON documents using Swift.

YamlSwift parses a string of YAML document(s) (or a JSON document) and returns a Yaml enum value representing that string.

Install

Use Carthage to build and install.

Or use CocoaPods : Add pod 'Yaml' to your Podfile and run pod install.

It supports Swift Package Manager.

        .package(
            url: "https://github.com/behrang/YamlSwift.git",
            from: "

And:

        .target(
            name: "YourProject",
            dependencies: ["Yaml"]),

API

import

To use it, you should import it using the following statement:

import Yaml

Yaml

A Yaml value can be any of these cases:

enum Yaml {
  case null
  case bool(Bool)
  case int(Int)
  case double(Double)
  case string(String)
  case array([Yaml])
  case dictionary([Yaml: Yaml])
}

View on GitHub


2.  Yams

A sweet and swifty YAML parser built on LibYAML.

Installation

Building Yams requires Xcode 12.5+ or a Swift 5.4+ toolchain with the Swift Package Manager or CMake and Ninja.

CMake

CMake 3.17.2 or newer is required, along with Ninja 1.9.0 or newer.

When building for non-Apple platforms:

cmake -B /path/to/build -G Ninja -S /path/to/yams -DCMAKE_BUILD_TYPE=Release -DFoundation_DIR=/path/to/foundation/build/cmake/modules
cmake --build /path/to/build

To build for Apple platforms (macOS, iOS, tvOS, watchOS), there is no need to spearately build Foundation because it is included as part of the SDK:

cmake -B /path/to/build -G Ninja -S /path/to/yams -DCMAKE_BUILD_TYPE=Release
cmake --build /path/to/build

Swift Package Manager

Add .package(url: "https://github.com/jpsim/Yams.git", from: "5.0.1") to your Package.swift file's dependencies.

CocoaPods

Add pod 'Yams' to your Podfile.

Carthage

Add github "jpsim/Yams" to your Cartfile.

Bazel

In your WORKSPACE file

YAMS_GIT_SHA = "SOME_SHA"
http_archive(
    name = "com_github_jpsim_yams",
    urls = [
        "https://github.com/jpsim/Yams/archive/%s.zip" % YAMS_GIT_SHA,
    ],
    strip_prefix = "Yams-%s" % YAMS_GIT_SHA,
)

View on GitHub


3.  Zip

A Swift framework for zipping and unzipping files. Simple and quick to use. Built on top of minizip.

Usage

Import Zip at the top of the Swift file.

import Zip

Quick functions

The easiest way to use Zip is through quick functions. Both take local file paths as NSURLs, throw if an error is encountered and return an NSURL to the destination if successful.

do {
    let filePath = Bundle.main.url(forResource: "file", withExtension: "zip")!
    let unzipDirectory = try Zip.quickUnzipFile(filePath) // Unzip
    let zipFilePath = try Zip.quickZipFiles([filePath], fileName: "archive") // Zip
}
catch {
  print("Something went wrong")
}

Advanced Zip

For more advanced usage, Zip has functions that let you set custom destination paths, work with password protected zips and use a progress handling closure. These functions throw if there is an error but don't return.

do {
    let filePath = Bundle.main.url(forResource: "file", withExtension: "zip")!
    let documentsDirectory = FileManager.default.urls(for:.documentDirectory, in: .userDomainMask)[0]
    try Zip.unzipFile(filePath, destination: documentsDirectory, overwrite: true, password: "password", progress: { (progress) -> () in
        print(progress)
    }) // Unzip

    let zipFilePath = documentsFolder.appendingPathComponent("archive.zip")
    try Zip.zipFiles([filePath], zipFilePath: zipFilePath, password: "password", progress: { (progress) -> () in
        print(progress)
    }) //Zip

}
catch {
  print("Something went wrong")
}

View on GitHub


4.  ZIPFoundation

ZIP Foundation is a library to create, read and modify ZIP archive files.

Installation

Swift Package Manager

The Swift Package Manager is a dependency manager integrated with the Swift build system. To learn how to use the Swift Package Manager for your project, please read the official documentation.
To add ZIP Foundation as a dependency, you have to add it to the dependencies of your Package.swift file and refer to that dependency in your target.

// swift-tools-version:5.0
import PackageDescription
let package = Package(
    name: "<Your Product Name>",
    dependencies: [
		.package(url: "https://github.com/weichsel/ZIPFoundation.git", .upToNextMajor(from: "0.9.0"))
    ],
    targets: [
        .target(
		name: "<Your Target Name>",
		dependencies: ["ZIPFoundation"]),
    ]
)

After adding the dependency, you can fetch the library with:

$ swift package resolve

Carthage

Carthage is a decentralized dependency manager.
Installation instructions can be found in the project's README file.

To integrate ZIPFoundation into your Xcode project using Carthage, you have to add it to your Cartfile:

github "weichsel/ZIPFoundation" ~> 0.9

After adding ZIPFoundation to the Cartfile, you have to fetch the sources by running:

carthage update --no-build

The fetched project has to be integrated into your workspace by dragging ZIPFoundation.xcodeproj to Xcode's Project Navigator. (See official Carhage docs.)

CocoaPods

CocoaPods is a dependency manager for Objective-C and Swift.
To learn more about setting up your project for CocoaPods, please refer to the official documentation.
To integrate ZIP Foundation into your Xcode project using CocoaPods, you have to add it to your project's Podfile:

source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '10.0'
use_frameworks!
target '<Your Target Name>' do
    pod 'ZIPFoundation', '~> 0.9'
end

Afterwards, run the following command:

$ pod install

View on GitHub


YAML & ZIP Swift FAQ

  • What is YAML used for?

YAML is a data serialization language that is often used for writing configuration files. Depending on whom you ask, YAML stands for yet another markup language or YAML ain't markup language (a recursive acronym), which emphasizes that YAML is for data, not documents

  • How does a YAML work?

YAML files store information, so they do not include actions and decisions. Unlike XML or JSON, YAML presents data in a way that makes it easy for a human to read. The simple syntax does not impact the language's capabilities. Any data or structure added to an XML or JSON file can also be stored in YAML.


Related videos:

Install, Setup & Use SwiftLint in Xcode for Swift


Related posts:

#swift #yaml 

4 Best YAML & ZIP Libraries with Swift
Jarvis  Maggio

Jarvis Maggio

1661992800

Yams: A Sweet and Swifty YAML Parser

Yams

A sweet and swifty YAML parser built on LibYAML.

Installation

Building Yams requires Xcode 12.5+ or a Swift 5.4+ toolchain with the Swift Package Manager or CMake and Ninja.

CMake

CMake 3.17.2 or newer is required, along with Ninja 1.9.0 or newer.

When building for non-Apple platforms:

cmake -B /path/to/build -G Ninja -S /path/to/yams -DCMAKE_BUILD_TYPE=Release -DFoundation_DIR=/path/to/foundation/build/cmake/modules
cmake --build /path/to/build

To build for Apple platforms (macOS, iOS, tvOS, watchOS), there is no need to spearately build Foundation because it is included as part of the SDK:

cmake -B /path/to/build -G Ninja -S /path/to/yams -DCMAKE_BUILD_TYPE=Release
cmake --build /path/to/build

Swift Package Manager

Add .package(url: "https://github.com/jpsim/Yams.git", from: "5.0.1") to your Package.swift file's dependencies.

CocoaPods

Add pod 'Yams' to your Podfile.

Carthage

Add github "jpsim/Yams" to your Cartfile.

Bazel

In your WORKSPACE file

YAMS_GIT_SHA = "SOME_SHA"
http_archive(
    name = "com_github_jpsim_yams",
    urls = [
        "https://github.com/jpsim/Yams/archive/%s.zip" % YAMS_GIT_SHA,
    ],
    strip_prefix = "Yams-%s" % YAMS_GIT_SHA,
)

Usage

Yams has three groups of conversion APIs: one for use with Codable types, another for Swift Standard Library types, and a third one for a Yams-native representation.

Codable types

  • Codable is an encoding & decoding strategy introduced in Swift 4 enabling easy conversion between YAML and other Encoders like JSONEncoder and PropertyListEncoder.
  • Lowest computational overhead, equivalent to Yams.Node.
  • Encoding: YAMLEncoder.encode(_:) Produces a YAML String from an instance of type conforming to Encodable.
  • Decoding: YAMLDecoder.decode(_:from:) Decodes an instance of type conforming to Decodable from YAML String or Data.
import Foundation
import Yams

struct S: Codable {
    var p: String
}

let s = S(p: "test")
let encoder = YAMLEncoder()
let encodedYAML = try encoder.encode(s)
encodedYAML == """
p: test

"""
let decoder = YAMLDecoder()
let decoded = try decoder.decode(S.self, from: encodedYAML)
s.p == decoded.p

Swift Standard Library types

  • The type of Swift Standard Library is inferred from the contents of the internal Yams.Node representation by matching regular expressions.
  • This method has the largest computational overhead When decoding YAML, because the type inference of all objects is done up-front.
  • It may be easier to use in such a way as to handle objects created from JSONSerialization or if the input is already standard library types (Any, Dictionary, Array, etc.).
  • Encoding: Yams.dump(object:) Produces a YAML String from an instance of Swift Standard Library types.
  • Decoding: Yams.load(yaml:) Produces an instance of Swift Standard Library types as Any from YAML String.
// [String: Any]
let dictionary: [String: Any] = ["key": "value"]
let mapYAML: String = try Yams.dump(object: dictionary)
mapYAML == """
key: value

"""
let loadedDictionary = try Yams.load(yaml: mapYAML) as? [String: Any]

// [Any]
let array: [Int] = [1, 2, 3]
let sequenceYAML: String = try Yams.dump(object: array)
sequenceYAML == """
- 1
- 2
- 3

"""
let loadedArray: [Int]? = try Yams.load(yaml: sequenceYAML) as? [Int]

// Any
let string = "string"
let scalarYAML: String = try Yams.dump(object: string)
scalarYAML == """
string

"""
let loadedString: String? = try Yams.load(yaml: scalarYAML) as? String

Yams.Node

  • Yams' native model representing Nodes of YAML which provides all functions such as detection and customization of the YAML format.
  • Depending on how it is used, computational overhead can be minimized.
  • Encoding: Yams.serialize(node:) Produces a YAML String from an instance of Node.
  • Decoding Yams.compose(yaml:) Produces an instance of Node from YAML String.
var map: Yams.Node = [
    "array": [
        1, 2, 3
    ]
]
map.mapping?.style = .flow
map["array"]?.sequence?.style = .flow
let yaml = try Yams.serialize(node: map)
yaml == """
{array: [1, 2, 3]}

"""
let node = try Yams.compose(yaml: yaml)
map == node

Integrating with Combine

When Apple's Combine framework is available, YAMLDecoder conforms to the TopLevelDecoder protocol, which allows it to be used with the decode(type:decoder:) operator:

import Combine
import Foundation
import Yams

func fetchBook(from url: URL) -> AnyPublisher<Book, Error> {
    URLSession.shared.dataTaskPublisher(for: url)
        .map(\.data)
        .decode(type: Book.self, decoder: YAMLDecoder())
        .eraseToAnyPublisher()
}

Download Details:

Author: jpsim
Source code: https://github.com/jpsim/Yams

License: MIT license
#swift #yaml 

Yams: A Sweet and Swifty YAML Parser
Jarvis  Maggio

Jarvis Maggio

1661985480

YamlSwift: Load YAML and JSON Documents using Swift

YamlSwift

Load YAML and JSON documents using Swift.

YamlSwift parses a string of YAML document(s) (or a JSON document) and returns a Yaml enum value representing that string.

Install

Use Carthage to build and install.

Or use CocoaPods : Add pod 'Yaml' to your Podfile and run pod install.

It supports Swift Package Manager.

        .package(
            url: "https://github.com/behrang/YamlSwift.git",
            from: "

And:

        .target(
            name: "YourProject",
            dependencies: ["Yaml"]),

API

import

To use it, you should import it using the following statement:

import Yaml

Yaml

A Yaml value can be any of these cases:

enum Yaml {
  case null
  case bool(Bool)
  case int(Int)
  case double(Double)
  case string(String)
  case array([Yaml])
  case dictionary([Yaml: Yaml])
}

Yaml.load

Yaml.load (String) throws -> Yaml

Takes a string of a YAML document and returns a Yaml enum.

let value = try! Yaml.load("a: 1\nb: 2")
print(value["a"])  // Int(1)
print(value["b"])  // Int(2)
print(value["c"])  // Null

If the input document is invalid or contains more than one YAML document, an error is thrown.

do {
  let value = try Yaml.load("a\nb: 2")
}
catch {
  print(error)  // expected end, near "b: 2"
}

Yaml.loadMultiple

Yaml.loadMultiple (String) throws -> [Yaml]

Takes a string of one or more YAML documents and returns [Yaml].

let value = try! Yaml.loadMultiple("---\na: 1\nb: 2\n---\na: 3\nb: 4")
print(value[0]["a"])  // Int(1)
print(value[1]["a"])  // Int(3)

It will throw an error if an error is encountered in any of the documents.

Yaml#[Int] -> Yaml

value[Int] -> Yaml
value[Int] = Yaml

If used on a Yaml.array value, it will return the value at the specified index. If the index is invalid or the value is not a Yaml.array, Yaml.null is returned. You can also set a value at a specific index. Enough elements will be added to the wrapped array to set the specified index. If the value is not a Yaml.array, it will change to it after set.

var value = try! Yaml.load("- Behrang\n- Maryam")
print(value[0])  // String(Behrang)
print(value[1])  // String(Maryam)
print(value[2])  // Null
value[2] = "Radin"
print(value[2])  // String(Radin)

Yaml#[Yaml] -> Yaml

value[Yaml] -> Yaml
value[Yaml] = Yaml

If used on a Yaml.dictionary value, it will return the value for the specified key. If a value for the specified key does not exist, or value is not a Yaml.dictionary, Yaml.null is returned. You can also set a value for a specific key. If the value is not a Yaml.dictionary, it will change to it after set.

Since Yaml is a literal convertible type, you can pass simple values to this method.

var value = try! Yaml.load("first name: Behrang\nlast name: Noruzi Niya")
print(value["first name"])  // String(Behrang)
print(value["last name"])  // String(Noruzi Niya)
print(value["age"])  // Null
value["first name"] = "Radin"
value["age"] = 1
print(value["first name"])  // String(Radin)
print(value["last name"])  // String(Noruzi Niya)
print(value["age"])  // Int(1)

Yaml#bool

value.bool -> Bool?

Returns an Optional<Bool> value. If the value is a Yaml.bool value, the wrapped value is returned. Otherwise nil is returned.

let value = try! Yaml.load("animate: true\nshow tip: false\nusage: 25")
print(value["animate"].bool)  // Optional(true)
print(value["show tip"].bool)  // Optional(false)
print(value["usage"].bool)  // nil

Yaml#int

value.int -> Int?

Returns an Optional<Int> value. If the value is a Yaml.int value, the wrapped value is returned. Otherwise nil is returned.

let value = try! Yaml.load("a: 1\nb: 2.0\nc: 2.5")
print(value["a"].int)  // Optional(1)
print(value["b"].int)  // Optional(2)
print(value["c"].int)  // nil

Yaml#double

value.double -> Double?

Returns an Optional<Double> value. If the value is a Yaml.double value, the wrapped value is returned. Otherwise nil is returned.

let value = try! Yaml.load("a: 1\nb: 2.0\nc: 2.5\nd: true")
print(value["a"].double)  // Optional(1.0)
print(value["b"].double)  // Optional(2.0)
print(value["c"].double)  // Optional(2.5)
print(value["d"].double)  // nil

Yaml#string

value.string -> String?

Returns an Optional<String> value. If the value is a Yaml.string value, the wrapped value is returned. Otherwise nil is returned.

let value = try! Yaml.load("first name: Behrang\nlast name: Noruzi Niya\nage: 33")
print(value["first name"].string)  // Optional("Behrang")
print(value["last name"].string)  // Optional("Noruzi Niya")
print(value["age"].string)  // nil

Yaml#array

value.array -> [Yaml]?

Returns an Optional<Array<Yaml>> value. If the value is a Yaml.array value, the wrapped value is returned. Otherwise nil is returned.

let value = try! Yaml.load("languages:\n - Swift: true\n - Objective C: false")
print(value.array)  // nil
print(value["languages"].array)  // Optional([Dictionary([String(Swift): Bool(true)]), Dictionary([String(Objective C): Bool(false)])])

Yaml#dictionary

value.dictionary -> [Yaml: Yaml]?

Returns an Optional<Dictionary<Yaml, Yaml>> value. If the value is a Yaml.dictionary value, the wrapped value is returned. Otherwise nil is returned.

let value = try! Yaml.load("- Swift: true\n- Objective C: false")
print(value.dictionary)  // nil
print(value[0].dictionary)  // Optional([String(Swift): Bool(true)])

Yaml#count

value.count -> Int?

Returns an Optional<Int> value. If the value is either a Yaml.array or a Yaml.dictionary value, the count of elements is returned. Otherwise nil is returned.

let value = try! Yaml.load("- Swift: true\n- Objective C: false")
print(value.count)  // Optional(2)
print(value[0].count)  // Optional(1)
print(value[0]["Swift"].count)  // nil

Download Details:

Author: behrang
Source code: https://github.com/behrang/YamlSwift

License: MIT license 
#swift #yaml 

YamlSwift: Load YAML and JSON Documents using Swift
Python  Library

Python Library

1661844540

Conda Replicate: Manage Local Mirrored anaconda Channels

What is this?

conda-replicate is a command line tool for creating and updating local mirrored anaconda channels.

Notable features

  • Uses the standard match specification syntax to identify packages
  • Resolves all necessary dependencies of specified packages
  • Allows for channel evolution via direct updates or transferable patches
  • Synchronizes upstream package hotfixes with local channels

Does this violate Anaconda's terms-of-service?

Disclaimer: I am an analyst, not a lawyer. The Anaconda terms-of-service expressly forbids mirroring of the default anaconda repository on repo.anaconda.com. However, as explained in a post on the conda-forge blog, this does not apply to conda-forge or any other channel hosted on anaconda.org. Therefore, conda-replicate uses conda-forge as it's default upstream channel. You are of course welcome to specify another channel, but please be respectful of Anaconda's terms-of-service and do not mirror the default anaconda repository.

Installation

Due to dependencies on modern versions of conda (for searching) and conda-build (for indexing), conda-replicate is currently only available on conda-forge:

conda install conda-replicate --channel conda-forge --override-channels

Usage

  1. Creating a local channel
  2. Updating an existing local channel

1. Creating a local channel

Suppose that you want to mirror all of the conda-forge python 3.9 micro releases (3.9.1, 3.9.2, ...) in a local channel called my-custom-channel, this can be accomplished by simply running the update sub-command:

> conda-replicate update "python >=3.9,<3.10" --target ./my-custom-channel

Because the number of command line arguments can quickly get out of hand, it is recommended that you use a configuration file:

# config.yml
channel: conda-forge
requirements:
  - python >=3.9,<3.10

With this configuration file saved as config.yml one can re-run the above command:

> conda-replicate update --config ./config.yaml --target ./my-custom-channel

Once either of these commands has finished you will have a fully accessible local channel that conda can use to install packages. For example you can do the following:

> conda create -n conda-replicate-test --channel ./my-custom-channel --override-channels -y

> conda activate conda-replicate-test

> python --version
Python 3.9.13

> conda list
# packages in environment at /path/to/Anaconda3/envs/conda-local-test-env:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h8ffe710_4    file:///path/to/my-custom-channel
ca-certificates           2022.6.15            h5b45459_0    file:///path/to/my-custom-channel
libffi                    3.4.2                h8ffe710_5    file:///path/to/my-custom-channel
libsqlite                 3.39.2               h8ffe710_1    file:///path/to/my-custom-channel
libzlib                   1.2.12               h8ffe710_2    file:///path/to/my-custom-channel
openssl                   3.0.5                h8ffe710_1    file:///path/to/my-custom-channel
pip                       22.2.2             pyhd8ed1ab_0    file:///path/to/my-custom-channel
python                    3.9.13       hcf16a7b_0_cpython    file:///path/to/my-custom-channel
python_abi                3.9                      2_cp39    file:///path/to/my-custom-channel
setuptools                65..0           py39hcbf5309_0     file:///path/to/my-custom-channel
sqlite                    3.39.2               h8ffe710_1    file:///path/to/my-custom-channel
tk                        8.6.12               h8ffe710_0    file:///path/to/my-custom-channel
tzdata                    2022c                h191b570_0    file:///path/to/my-custom-channel
ucrt                      10.0.20348.0         h57928b3_0    file:///path/to/my-custom-channel
vc                        14.2                 hb210afc_6    file:///path/to/my-custom-channel
vs2015_runtime            14.29.30037          h902a5da_6    file:///path/to/my-custom-channel
wheel                     0.37.1             pyhd8ed1ab_0    file:///path/to/my-custom-channel
xz                        5.2.6                h8d14728_0    file:///path/to/my-custom-channel

Notice that it appears our local channel has all of the direct and transitive dependencies for python 3.9.13. In fact, it has the direct and transitive dependencies for all of the micro versions of python 3.9. We can see a summary of these dependencies by using the query sub-command, which will query conda-forge and determine what packages are needed to satisfy the python 3.9 specification.

> conda-replicate query --config ./config.yaml

  Packages to add    (26)   Number   Size [MB]
 ──────────────────────────────────────────────
  python                    35          659.94
  pypy3.9                   16          469.51
  openssl                   19          147.41
  setuptools                111         141.61
  pip                       38           48.08
  vs2015_runtime            15           32.38
  sqlite                    20           24.89
  tk                        3            11.67
  ca-certificates           30            5.73
  zlib                      17            2.64
  certifi                   15            2.28
  tzdata                    15            1.91
  pyparsing                 24            1.58
  xz                        3             1.35
  ucrt                      1             1.23
  bzip2                     5             0.76
  expat                     2             0.74
  libsqlite                 1             0.65
  libzlib                   6             0.40
  packaging                 8             0.28
  wheel                     9             0.27
  libffi                    6             0.25
  vc                        14            0.17
  wincertstore              6             0.09
  six                       3             0.04
  python_abi                3             0.01
  Total                     425        1555.88

Note that the query sub-command is most commonly used when a target is included in the configuration file (or on the command line via --target). When a target is specified, the query sub-command will calculate results relative the given target channel. This also applies to other conda-replicate sub-commands such as update and patch. We will make use of this when we update our local channel below, but for now, we want the examine the complete, non-relative, results of query.

As you can see the original update installed quite a few packages, and they take up quite a bit of space! This result may prompt a few questions.

How are dependencies determined?

conda-replicate uses the conda.api to recursively examine the dependencies of user-supplied "root" specifications (like python>=3.9,<3.10 given above) and constructs a directed dependency graph. After this graph is completed, unsatisfied nodes (specifications that have no connected packages) are pruned. Additionally, nodes that have no possible connecting path to at least one of the root specifications are pruned as well. What is left are packages that satisfy either a root specification, a direct dependency of a root specification, or a transitive dependency further down the graph. Note that if a root specification is unsatisfied an UnsatisfiedRequirementsError exception is raised.

As a quick aside, you can use the conda query --info command to look at the dependencies of individual conda packages (where ⋮ indicates hidden output):

> conda search python==3.9.13 --info --channel conda-forge  --override-channels

⋮
python 3.9.13 hcf16a7b_0_cpython
--------------------------------
⋮
dependencies:
  - bzip2 >=1.0.8,<2.0a0
  - libffi >=3.4.2,<3.5.0a0
  - libzlib >=1.2.11,<1.3.0a0
  - openssl >=3.0.3,<4.0a0
  - sqlite >=3.38.5,<4.0a0
  - tk >=8.6.12,<8.7.0a0
  - tzdata
  - vc >=14.1,<15
  - vs2015_runtime >=14.16.27033
  - xz >=5.2.5,<5.3.0a0
  - pip

> conda search pip==22.2.2 -c conda-forge  --override-channels --info

⋮
pip 22.2.2 pyhd8ed1ab_0
-----------------------
⋮
dependencies:
  - python >=3.7
  - setuptools
  - wheel

Why are there so many "extra" packages?

Predominantly, board specifications are the usual the culprit for "extra" packages. Specifically, lets look at the following:

  • 35 different packages of python. This can be traced back to our root specification of python>=3.9,<3.10. This specification includes not only all of the micro versions, but all of the conda-forge builds for those packages as well.
  • 111 packages of setuptools and 38 of pip. Python has a dependency on pip which in turn has a dependency on setuptools (both seen in the aside above). These specifications do not include version numbers and therefore match all packages of setuptools and pip.
  • 16 packages of pypy3.9. In this case some packages depend on the python_abi 3.9-2. There is special build of this package that depends on the pypy3.8 interpreter. Therefore, the relevant pypy packages (and their dependencies) are included in our local channel.

Can we exclude these "extra" packages?

Yes, by using exclusions in the configuration file (or --exclude on the command line option) . Let's assume that you are repeating the process of creating my-custom-channel from above. However instead of jumping right to the update sub-command you do the following:

  1. Run the query sub-command in summary mode (the default mode used above) to see the overall package distribution
> conda-replicate query --config ./config.yaml

2.   If we find some unexpected packages we can re-run query in list mode to zero in on the individual version of those packages. As you can see below there is a wide range of package versions for python, pip, setuptools, pyp3.9.

> python -m conda_local query --config ./config.yaml --output list

Packages to add:
⋮
pip-20.0.2-py_2.tar.bz2
pip-20.1-pyh9f0ad1d_0.tar.bz2
pip-20.1.1-py_1.tar.bz2
⋮
pip-22.2-pyhd8ed1ab_0.tar.bz2
pip-22.2.1-pyhd8ed1ab_0.tar.bz2
pip-22.2.2-pyhd8ed1ab_0.tar.bz2
⋮
pypy3.9-7.3.8-h1738a25_0.tar.bz2
pypy3.9-7.3.8-h1738a25_1.tar.bz2
pypy3.9-7.3.8-h1738a25_2.tar.bz2
pypy3.9-7.3.8-hc3b0203_0.tar.bz2
pypy3.9-7.3.8-hc3b0203_1.tar.bz2
pypy3.9-7.3.8-hc3b0203_2.tar.bz2
pypy3.9-7.3.9-h1738a25_0.tar.bz2
pypy3.9-7.3.9-h1738a25_1.tar.bz2
pypy3.9-7.3.9-h1738a25_2.tar.bz2
pypy3.9-7.3.9-h1738a25_3.tar.bz2
pypy3.9-7.3.9-h1738a25_4.tar.bz2
pypy3.9-7.3.9-hc3b0203_0.tar.bz2
pypy3.9-7.3.9-hc3b0203_1.tar.bz2
pypy3.9-7.3.9-hc3b0203_2.tar.bz2
pypy3.9-7.3.9-hc3b0203_3.tar.bz2
pypy3.9-7.3.9-hc3b0203_4.tar.bz2
⋮
python-3.9.0-h408a966_4_cpython.tar.bz2
python-3.9.1-h7840368_0_cpython.tar.bz2
⋮
python-3.9.12-hcf16a7b_1_cpython.tar.bz2
python-3.9.13-h9a09f29_0_cpython.tar.bz2
⋮
setuptools-49.6.0-py39h467e6f4_2.tar.bz2
setuptools-49.6.0-py39hcbf5309_3.tar.bz2
setuptools-57.4.0-py39h0d475fb_1.tar.bz2
⋮
setuptools-65.1.1-py39hcbf5309_0.tar.bz2
setuptools-65.2.0-py39h0d475fb_0.tar.bz2
setuptools-65.3.0-py39hcbf5309_0.tar.bz2
⋮

3.   Having identified the version ranges of theses packages we can refine our call to the update sub-command by tightening our root specification and making use of exclusions in the configuration file. The entire process is updatable, so we don't need to loss sleep over our ranges right now:

# config.yml
channel: conda-forge
requirements:
  - python >=3.9.8,<3.10  # updated line
exclusions:
  - setuptools <=60.0     # new line
  - pip <=21.0            # new line
  - pypy3.9               # new line
 > conda-replicate query --config ./config.yml

 Packages to add    (21)   Number   Size [MB]
 ──────────────────────────────────────────────
 python                    14          258.60
 openssl                   14          117.20
 setuptools                53           69.28
 vs2015_runtime            15           32.38
 pip                       22           30.10
 sqlite                    10           12.51
 tk                        3            11.67
 ca-certificates           30            5.73
 tzdata                    15            1.91
 pyparsing                 24            1.58
 xz                        3             1.35
 ucrt                      1             1.23
 bzip2                     5             0.76
 libsqlite                 1             0.65
 libzlib                   6             0.40
 packaging                 8             0.28
 wheel                     9             0.27
 libffi                    6             0.25
 vc                        14            0.17
 six                       3             0.04
 python_abi                2             0.01
 Total                     230         546.38

This brings the number of packages and overall size down to a more reasonable level.

4.   Finally we can re-run the update sub-command (as we did above):

conda-replicate update --config ./config.yaml --target ./my-custom-channel

It should be mentioned that sometimes the reasons for why a package was included require a more detailed dependency investigation. In those cases calls to conda search --info, conda-replicate query --output json, or as a last resort conda-replicate query --debug, are very useful.

2. Updating an existing local channel

Once a local channel has been created it can be updated at any time. Updates preform the following actions:

Add, delete, or revoke packages in response to changes in our specifications or the upstream channel

Synchronize local package hotfixes with those in the upstream channel

Refresh the package index in response to package and/or hotfix changes (via conda_build.api)

There are two ways that local channels can be updated, either directly or through patches. Lets examine both options starting from my-custom-channel in the previous section. We ended up with a configuration file that looked like the following:

# config.yml
channel: conda-forge
requirements:
  - python >=3.9.8,<3.10
exclusions:
  - setuptools <=60.0
  - pip <=21.0
  - pypy3.9

Now, let's assume that after creating my-custom-channel we want to further tighten our python and setuptools requirements, and add a new requirement for pydantic (note that this example will only simulate changes to our specifications, not changes to upstream channel). We would also like to include the target field in the configuration file:

#config.yml
channel: conda-forge
target: ./my-custom-channel   # new line
requirements:
  - python >=3.9.10,<3.10     # updated line
  - pydantic                  # new line
exclusions:
  - setuptools <=62.0         # updated line
  - pip <=21.0
  - pypy3.9

Remembering the lessons from the last section, we first run the query sub-command. Because our new configuration file defines a target, we we will see the results of the query relative my-custom-channel. This effectively describes what packages will be added and removed when we run the update:

> conda-replicate query --config ./config.yml

  Packages to add    (4)   Number   Size [MB]
 ─────────────────────────────────────────────
  pydantic                 17           11.87
  typing_extensions        10            0.28
  typing-extensions        10            0.08
  dataclasses              3             0.02
  Total                    40           12.25


  Packages to remove (2)   Number   Size [MB]
 ─────────────────────────────────────────────
  python                   2            36.44
  setuptools               28           33.54
  Total                    30           69.98

How do we perform a direct update?

At this point performing the direct update is simple:

> conda-replicate update --config ./config.yml

How do we perform a patched update?

Patched updates are accomplished by using two different sub-commands: patch and merge. The first of these, patch, works similar to update, in that it will calculate the packages to add or remove relative our target. It will then download the packages and hotfixes into a separate patch directory (controlled by the --parent and --name command line options). It is important to note that the package index of the patch directory is not updated and therefore cannot be used by conda to install packages!

> conda-replicate patch --config ./config.yml --parent ./patch-archive --name patch_20220824

This patch directory can then be merged into an existing channel using the merge sub-command. The merging process not only copies the packages and modified hotfixes form the patch directory, but it also updates the package index. Note that those packages patch determined should be removed are passed to merge via the hotfixes.

> conda-replicate merge ./patch-archive/patch_20220824 my-air-gapped-channel

The patch and merge commands are particularly well suited for updating air-gapped systems, however there are some things to consider:

  1. You must be able to transfer files to the air-gapped system from a network facing system (via a bridge or manual drive).
  2. You need to maintain a parallel channel on your network facing system that is used generate the patches.
  3. The very first transfer to the air-gapped system must be an indexed conda channel. This means that you need to use the update sub-command to create a channel on your network facing system and then transfer the entire channel to the air gapped system. All subsequent transfers can be updated via the patch and merge sub-commands.
  4. Your configuration needs to include conda-replicate as a requirement. If not, you will not be able install conda-replicate on the air-gapped system, which means you cannot run the merge sub-command.

Commands

The following commands are available in conda-replicate:

query

 Usage: conda-replicate query [OPTIONS] [REQUIREMENTS]...

 Search an upstream channel for the specified package REQUIREMENTS and report results.

  • Resulting packages are reported to the user in the specified output form (see --output)
  • Include both direct and transitive dependencies of required packages

 Package requirement notes:

  • Requirements are constructed using the anaconda package query syntax
  • Unsatisfied requirements will raise an error by default (see --no-validate)
  • Requirements specified on the command line augment those specified in a configuration file

┌─ Options ────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                  │
│  --channel   -c   TEXT     Upstream anaconda channel. Can be specified using the canonical       │
│                            channel name on anaconda.org (conda-forge), a fully qualified URL     │
│                            (https://conda.anaconda.org/conda-forge/), or a local directory       │
│                            path.                                                                 │
│                            [default: conda-forge]                                                │
│                                                                                                  │
│  --target    -t   TEXT     Target anaconda channel. When specified, this channel will act as a   │
│                            baseline for the package search process - only package differences    │
│                            (additions or deletions) will be reported to the user.                │
│                                                                                                  │
│  --exclude        TEXT     Packages excluded from the search process. Specified using the        │
│                            anaconda package query syntax. Multiple options may be passed at one  │
│                            time.                                                                 │
│                                                                                                  │
│  --dispose        TEXT     Packages that are used in the search process but not included in the  │
│                            final results. Specified using the anaconda package query syntax.     │
│                            Multiple options may be passed at one time.                           │
│                                                                                                  │
│  --subdir         SUBDIR   Selected platform sub-directories. Multiple options may be passed at  │
│                            one time. Allowed values: {linux-32, linux-64, linux-aarch64,         │
│                            linux-armv6l, linux-armv7l, linux-ppc64, linux-ppc64le, linux-s390x,  │
│                            noarch, osx-64, osx-arm64, win-32, win-64, zos-z}.                    │
│                            [default: win-64, noarch]                                             │
│                                                                                                  │
│  --output         OUTPUT   Specifies the format of the search results. Allowed values: {table,   │
│                            list, json}.                                                          │
│                                                                                                  │
│  --config         FILE     Path to the yaml configuration file.                                  │
│                                                                                                  │
│  --quiet                   Quite mode. suppress all superfluous output.                          │
│                                                                                                  │
│  --debug     -d            Enable debugging logs. Can be repeated to increase log level          │
│                                                                                                  │
│  --help                    Show this message and exit.                                           │
│                                                                                                  │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

update

 Usage: conda-replicate update [OPTIONS] [REQUIREMENTS]...

 Update a local channel based on specified upstream package REQUIREMENTS.

  • Packages are downloaded or removed from the local channel prior to re-indexing
  • Includes both direct and transitive dependencies of required packages
  • Includes update to the platform specific patch instructions (hotfixes)

 Package requirement notes:

  • Requirements are constructed using the anaconda package query syntax
  • Unsatisfied requirements will raise an error by default (see --no-validate)
  • Requirements specified on the command line augment those specified in a configuration file

┌─ Options ────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                  │
│  *   --target    -t   TEXT     Local anaconda channel where the update will occur. If this       │
│                                local channel already exists it will act as a baseline for the    │
│                                package search process - only package differences (additions or   │
│                                deletions) will be updated.                                       │
│                                [required]                                                        │
│                                                                                                  │
│      --channel   -c   TEXT     Upstream anaconda channel. Can be specified using the canonical   │
│                                channel name on anaconda.org (conda-forge), a fully qualified     │
│                                URL (https://conda.anaconda.org/conda-forge/), or a local         │
│                                directory path.                                                   │
│                                [default: conda-forge]                                            │
│                                                                                                  │
│      --exclude        TEXT     Packages excluded from the search process. Specified using the    │
│                                anaconda package query syntax. Multiple options may be passed at  │
│                                one time.                                                         │
│                                                                                                  │
│      --dispose        TEXT     Packages that are used in the search process but not included in  │
│                                the final results. Specified using the anaconda package query     │
│                                syntax. Multiple options may be passed at one time.               │
│                                                                                                  │
│      --subdir         SUBDIR   Selected platform sub-directories. Multiple options may be        │
│                                passed at one time. Allowed values: {linux-32, linux-64,          │
│                                linux-aarch64, linux-armv6l, linux-armv7l, linux-ppc64,           │
│                                linux-ppc64le, linux-s390x, noarch, osx-64, osx-arm64, win-32,    │
│                                win-64, zos-z}.                                                   │
│                                [default: win-64, noarch]                                         │
│                                                                                                  │
│      --config         FILE     Path to the yaml configuration file.                              │
│                                                                                                  │
│      --quiet                   Quite mode. suppress all superfluous output.                      │
│                                                                                                  │
│      --debug     -d            Enable debugging logs. Can be repeated to increase log level      │
│                                                                                                  │
│      --help                    Show this message and exit.                                       │
│                                                                                                  │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

patch

 Usage: conda-replicate patch [OPTIONS] [REQUIREMENTS]...

 Create a patch from an upstream channel based on specified package REQUIREMENTS.

  • Packages are downloaded to a local patch directory (see --name and --parent)
  • Patches can be merged into existing local channels (see merge sub-command)
  • Includes both direct and transitive dependencies of required packages
  • Includes update to the platform specific patch instructions (hotfixes)

 Package requirement notes:

  • Requirements are constructed using the anaconda package query syntax
  • Unsatisfied requirements will raise an error by default (see --no-validate)
  • Requirements specified on the command line augment those specified in a configuration file

┌─ Options ────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                  │
│  --target    -t   TEXT     Target anaconda channel. When specified, this channel will act as a   │
│                            baseline for the package search process - only package differences    │
│                            (additions or deletions) will be included in the patch.               │
│                                                                                                  │
│  --name           TEXT     Name of the patch directory. [patch_%Y%m%d_%H%M%S]                    │
│                                                                                                  │
│  --parent         PATH     Parent directory of the patch. [current directory]                    │
│                                                                                                  │
│  --channel   -c   TEXT     Upstream anaconda channel. Can be specified using the canonical       │
│                            channel name on anaconda.org (conda-forge), a fully qualified URL     │
│                            (https://conda.anaconda.org/conda-forge/), or a local directory       │
│                            path.                                                                 │
│                            [default: conda-forge]                                                │
│                                                                                                  │
│  --exclude        TEXT     Packages excluded from the search process. Specified using the        │
│                            anaconda package query syntax. Multiple options may be passed at one  │
│                            time.                                                                 │
│                                                                                                  │
│  --dispose        TEXT     Packages that are used in the search process but not included in the  │
│                            final results. Specified using the anaconda package query syntax.     │
│                            Multiple options may be passed at one time.                           │
│                                                                                                  │
│  --subdir         SUBDIR   Selected platform sub-directories. Multiple options may be passed at  │
│                            one time. Allowed values: {linux-32, linux-64, linux-aarch64,         │
│                            linux-armv6l, linux-armv7l, linux-ppc64, linux-ppc64le, linux-s390x,  │
│                            noarch, osx-64, osx-arm64, win-32, win-64, zos-z}.                    │
│                            [default: win-64, noarch]                                             │
│                                                                                                  │
│  --config         FILE     Path to the yaml configuration file.                                  │
│                                                                                                  │
│  --quiet                   Quite mode. suppress all superfluous output.                          │
│                                                                                                  │
│  --debug     -d            Enable debugging logs. Can be repeated to increase log level          │
│                                                                                                  │
│  --help                    Show this message and exit.                                           │
│                                                                                                  │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

merge

 Usage: conda-replicate merge [OPTIONS] PATCH CHANNEL

 Merge a PATCH into a local CHANNEL and update the local package index.

┌─ Options ────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                  │
│  --quiet           Quite mode. suppress all superfluous output.                                  │
│                                                                                                  │
│  --debug   -d      Enable debugging logs. Can be repeated to increase log level                  │
│                                                                                                  │
│  --help            Show this message and exit.                                                   │
│                                                                                                  │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

index

 Usage: conda-replicate index [OPTIONS] CHANNEL

 Update the package index of a local CHANNEL.

┌─ Options ────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                  │
│  --quiet           Quite mode. suppress all superfluous output.                                  │
│                                                                                                  │
│  --debug   -d      Enable debugging logs. Can be repeated to increase log level                  │
│                                                                                                  │
│  --help            Show this message and exit.                                                   │
│                                                                                                  │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

Configuration File

The YAML configuration file can be used to specify any of the following:

  • channel (string)
  • target (string)
  • requirements (list)
  • exclusions (list)
  • disposables (list)
  • subdirs (list)

Download details:

Author: cswartzvi
Source code: https://github.com/cswartzvi/conda-replicate 
License: View license

#python #yaml #anaconda

Conda Replicate: Manage Local Mirrored anaconda Channels

JuliaData/YAML.jl: Parse Yer YAMLs

YAML

YAML is a flexible data serialization format that is designed to be easily read and written by human beings.

This library parses YAML documents into native Julia types and dumps them back into YAML documents.

Synopsis

For most purposes there is one important function: YAML.load, which takes a string and parses it the first YAML document it finds.

To parse a file use YAML.load_file, and to parse every document in a file use YAML.load_all or YAML.load_all_file.

Given a YAML document like the following

receipt:     Oz-Ware Purchase Invoice
date:        2012-08-06
customer:
    given:   Dorothy
    family:  Gale

items:
    - part_no:   A4786
      descrip:   Water Bucket (Filled)
      price:     1.47
      quantity:  4

    - part_no:   E1628
      descrip:   High Heeled "Ruby" Slippers
      size:      8
      price:     100.27
      quantity:  1

bill-to:  &id001
    street: |
            123 Tornado Alley
            Suite 16
    city:   East Centerville
    state:  KS

ship-to:  *id001

specialDelivery:  >
    Follow the Yellow Brick
    Road to the Emerald City.
    Pay no attention to the
    man behind the curtain.

It can be loaded with

import YAML
data = YAML.load_file("test.yml")
println(data)

Which will show you something like this.

{"date"=>Aug 6, 2012 12:00:00 AM PDT,"ship-to"=>{"street"=>"123 Tornado Alley\nSuite 16\n","state"=>"KS","city"=>"East Centerville"},"customer"=>{"given"=>"Dorothy","family"=>"Gale"},"specialDelivery"=>"Follow the Yellow Brick\nRoad to the Emerald City.\nPay no attention to the\nman behind the curtain.\n","items"=>{{"price"=>1.47,"descrip"=>"Water Bucket (Filled)","part_no"=>"A4786","quantity"=>4}  …  {"price"=>100.27,"size"=>8,"descrip"=>"High Heeled \"Ruby\" Slippers","part_no"=>"E1628","quantity"=>1}},"bill-to"=>{"street"=>"123 Tornado Alley\nSuite 16\n","state"=>"KS","city"=>"East Centerville"},"receipt"=>"Oz-Ware Purchase Invoice"}

Note that ints and floats are recognized, as well as timestamps which are parsed into CalendarTime objects. Also, anchors and references work as expected, without making a copy.

Dictionaries are parsed into instances of Dict{Any,Any} by default. You can, however, specify a custom type in which to parse all dictionaries.

# using Symbol keys
data = YAML.load_file("test.yml"; dicttype=Dict{Symbol,Any})

# maintaining the order from the YAML file
using OrderedCollections
data = YAML.load_file("test.yml"; dicttype=OrderedDict{String,Any})

# specifying a default value
using DataStructures
data = YAML.load_file("test.yml"; dicttype=()->DefaultDict{String,Any}(Missing))

Writing to YAML

Similar to reading files, you can emit Julia objects to YAML files by calling write_file, or to a string object by calling write.

For example, you can reproduce the above file from the variable data

import YAML
YAML.write_file("test-output.yml", data)

which gives you (omitting the precise format but maintaining the content)

receipt: "Oz-Ware Purchase Invoice"
items:
  - part_no: "A4786"
    price: 1.47
    descrip: "Water Bucket (Filled)"
    quantity: 4
  - part_no: "E1628"
    price: 100.27
    size: 8
    descrip: "High Heeled "Ruby" Slippers"
    quantity: 1
customer:
  given: "Dorothy"
  family: "Gale"
ship-to:
  city: "East Centerville"
  street: |
      123 Tornado Alley
      Suite 16

  state: "KS"
bill-to:
  city: "East Centerville"
  street: |
      123 Tornado Alley
      Suite 16

  state: "KS"
specialDelivery: |
    Follow the Yellow Brick Road to the Emerald City. Pay no attention to the man behind the curtain.

date: 2012-08-06

Not yet implemented

  • When writing YAML files, you cannot use additional constructors like you can when reading.
  • Parsing sexigesimal numbers.
  • Fractions of seconds in timestamps.
  • Specific time-zone offsets in timestamps.
  • Application specific tags.

Download Details:

Author: JuliaData
Source Code: https://github.com/JuliaData/YAML.jl 
License: View license

#julia #yaml 

JuliaData/YAML.jl: Parse Yer YAMLs
Nat  Grady

Nat Grady

1659727560

R-yaml: R Package for Converting Objects to and From YAML

R YAML package

The R YAML package implements the libyaml YAML parser and emitter for R.

What is YAML?

YAML is a human-readable markup language. With it, you can create easily readable documents that can be consumed by a variety of programming languages.

Examples

Hash of baseball teams per league:

american:
- Boston Red Sox
- Detroit Tigers
- New York Yankees
national:
- New York Mets
- Chicago Cubs
- Atlanta Braves

Data dictionary specification:

- field: ID
  description: primary identifier
  type: integer
  primary key: yes
- field: DOB
  description: date of birth
  type: date
  format: yyyy-mm-dd
- field: State
  description: state of residence
  type: string

Installation

CRAN

You can install this package directly from CRAN by running (from within R): install.packages('yaml')

Zip/Tarball

  1. Download the appropriate zip file or tar.gz file from the Github releases page.
  2. Run R CMD INSTALL <filename>

Git (via devtools)

  1. Install the devtools package from CRAN.
  2. In R, run the following:
library(devtools)
install_github('viking/r-yaml')

Usage

The yaml package provides three functions: yaml.load, yaml.load_file and as.yaml.

yaml.load

yaml.load is the YAML parsing function. It accepts a YAML document as a string. Here's a simple example that parses a YAML sequence:

x <- "
- 1
- 2
- 3
"
yaml.load(x)  #=> [1] 1 2 3

Scalars

A YAML scalar is the basic building block of YAML documents. Example of a YAML document with one element:

1.2345

In this case, the scalar "1.2345" is typed as a float (or numeric) by the parser. yaml.load would return a numeric vector of length 1 for this document.

yaml.load("1.2345")  #=> [1] 1.2345

Sequences

A YAML sequence is a list of elements. Here's an example of a simple YAML sequence:

- this
- is
- a
- simple
- sequence
- of
- scalars

If you pass a YAML sequence to yaml.load, a couple of things can happen. If all of the elements in the sequence are uniform, yaml.load will return a vector of that type (i.e. character, integer, real, or logical). If the elements are not uniform, yaml.load will return a list of the elements.

Maps

A YAML map is a list of paired keys and values, or hash, of elements. Here's an example of a simple YAML map:

one: 1
two: 2
three: 3
four: 4

Passing a map to yaml.load will produce a named list by default. That is, keys are coerced to strings. Since it is possible for the keys of a YAML map to be almost anything (not just strings), you might not want yaml.load to return a named list. If you want to preserve the data type of keys, you can pass as.named.list = FALSE to yaml.load. If as.named.list is FALSE, yaml.load will create a keys attribute for the list it returns instead of coercing the keys into strings.

Handlers

yaml.load has the capability to accept custom handler functions. With handlers, you can customize yaml.load to do almost anything you want. Example of handler usage:

integer.handler <- function(x) { as.integer(x) + 123 }
yaml.load("123", handlers = list(int = integer.handler))  #=> [1] 246

Handlers are passed to yaml.load through the handlers argument. The handlers argument must be a named list of functions, where each name is the YAML type that you want to be handled by your function. The functions you provide must accept one argument and must return an R object.

Handler functions will be passed a string or list, depending on the original type of the object. In the example above, integer.handler was passed the string "123".

Sequence handlers

Custom sequence handlers will be passed a list of objects. You can then convert the list into whatever you want and return it. Example:

sequence.handler <- function(x) {
  tmp <- as.numeric(x)
  tmp / 5
}
string <- "
- foo
- bar
- 123
- 4.567
"
yaml.load(string, handlers = list(seq = sequence.handler))  #=> [1]      NA      NA 24.6000  0.9134

Map handlers

Custom map handlers work much in the same way as custom list handlers. A map handler function is passed a named list, or a list with a keys attribute (depending on the value of as.named.list). Example:

string <- "
a:
- 1
- 2
b:
- 3
- 4
"
yaml.load(string, handlers = list(map = function(x) { as.data.frame(x) }))

Returns:

  a b
1 1 3
2 2 4

yaml.load_file

yaml.load_file does the same thing as yaml.load, except it reads a file from a connection. For example:

x <- yaml.load_file("Data/document.yml")

This function takes the same arguments as yaml.load, with the exception that the first argument is a filename or a connection.

read_yaml

The read_yaml function is a convenience function that works similarly to functions in the readr package. You can use it instead of yaml.load_file if you prefer.

as.yaml

as.yaml is used to convert R objects into YAML strings. Example as.yaml usage:

x <- as.yaml(1:5)
cat(x, "\n")

Output from above example:

- 1
- 2
- 3
- 4
- 5

Notable arguments

indent

You can control the number of spaces used to indent by setting the indent option. By default, indent is 2.

For example:

cat(as.yaml(list(foo = list(bar = 'baz')), indent = 3))

Outputs:

foo:
   bar: baz

indent.mapping.sequence

By default, sequences that are within a mapping context are not indented.

For example:

cat(as.yaml(list(foo = 1:10)))

Outputs:

foo:
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10

If you want sequences to be indented in this context, set the indent.mapping.sequence option to TRUE.

For example:

cat(as.yaml(list(foo = 1:10), indent.mapping.sequence=TRUE))

Outputs:

foo:
  - 1
  - 2
  - 3
  - 4
  - 5
  - 6
  - 7
  - 8
  - 9
  - 10

column.major

The column.major option determines how a data frame is converted into YAML. By default, column.major is TRUE.

Example of as.yaml when column.major is TRUE:

x <- data.frame(a=1:5, b=6:10)
y <- as.yaml(x, column.major = TRUE)
cat(y, "\n")

Outputs:

a:
- 1
- 2
- 3
- 4
- 5
b:
- 6
- 7
- 8
- 9
- 10

Whereas:

x <- data.frame(a=1:5, b=6:10)
y <- as.yaml(x, column.major = FALSE)
cat(y, "\n")

Outputs:

- a: 1
  b: 6
- a: 2
  b: 7
- a: 3
  b: 8
- a: 4
  b: 9
- a: 5
  b: 10

handlers

You can specify custom handler functions via the handlers argument. This argument must be a named list of functions, where the names are R object class names (i.e., 'numeric', 'data.frame', 'list', etc). The function(s) you provide will be passed one argument (the R object) and can return any R object. The returned object will be emitted normally.

Special features

Verbatim(-ish) text

Character vectors that have a class of 'verbatim' will not be quoted in the output YAML document except when the YAML specification requires it. This means that you cannot do anything that would result in an invalid YAML document, but you can emit strings that would otherwise be quoted. This is useful for changing how logical vectors are emitted. For example:

as.yaml(c(TRUE, FALSE), handlers = list(
  logical = function(x) {
    result <- ifelse(x, "true", "false")
    class(result) <- "verbatim"
    return(result)
  }
))

Quoted Strings

There are times you might need to ensure a string scalar is quoted. Apply a non-null attribute of "quoted" to the string you need quoted and it will come out with double quotes around it.

port_def <- "80:80"
attr(port_def, "quoted") <- TRUE
x <- list(ports = list(port_def))
as.yaml(x)

Custom tags

You can specify YAML tags for R objects by setting the 'tag' attribute to a character vector of length 1. If you set a tag for a vector, the tag will be applied to the YAML sequence as a whole, unless the vector has only 1 element. If you wish to tag individual elements, you must use a list of 1-length vectors, each with a tag attribute. Likewise, if you set a tag for an object that would be emitted as a YAML mapping (like a data frame or a named list), it will be applied to the mapping as a whole. Tags can be used in conjunction with YAML deserialization functions like yaml.load via custom handlers, however, if you set an internal tag on an incompatible data type (like !seq 1.0), errors will occur when you try to deserialize the document.

write_yaml

The write_yaml function is a convenience function that works similarly to functions in the readr package. It calls as.yaml and writes the result to a file or a connection.

Additional documentation

For more information, run help(package='yaml') or example('yaml-package') for some examples.

Development

There is a Makefile for use with GNU Make to help with development. There are several make targets for building, debugging, and testing. You can run these by executing make <target-name> if you have the make program installed.

Target nameDescription
compileCompile the source files
checkRun CRAN checks
gct-checkRun CRAN checks with gctorture
testRun unit tests
gdb-testRun unit tests with gdb
valgrind-testRun unit tests with valgrind
tarballCreate tarball suitable for CRAN submission
allDefault target, runs compile and test

Implicit tag discovery

The algorithm used whenever there is no YAML tag explicitly provided is located in the implicit.re file. This file is used to create the implicit.c file via the re2c program. If you want to change this algorithm, make your changes in implicit.re, not implicit.c. The make targets will automatically update the C file as needed, but you'll need to have the re2c program installed for it to work.

VERSION file

The VERSION file is used to track the current version of the package. Warnings are displayed if the DESCRIPTION and CHANGELOG files are not properly updated when creating a tarball. This is to help prevent problems during the CRAN submission process.

Author: Vubiostat
Source Code: https://github.com/vubiostat/r-yaml 
License: Unknown, Unknown licenses found

#r #yaml 

R-yaml: R Package for Converting Objects to and From YAML
Hunter  Krajcik

Hunter Krajcik

1658591220

Settings_yaml: A Very Simple Method to Use Yaml Files

Provide a very simple method to use yaml files for reading/writing an app's configuration.

Saving a config data

void save() {
  /// create a new .settings.yaml
  var settings = SettingsYaml.load(pathToSettings: '.settings.yaml');

  settings['dbname'] = 'billing';
  settings['dbusername'] = 'username';
  settings['dbpassword'] = 'apassword';
  settings['timeout'] = 200;
  settings['ats'] = <String>['cat', 'bat', 'rat'];
  settings['hosts'] = <String, String>{
        'host1': 'one',
        'host2': 'two',
        'host3': 'three'
      };

  settings.save();
}

Loading config data

void load() {

  /// load an existing .settings.yaml, if it doesn't exist then create it.
  var settings = SettingsYaml.load(pathToSettings: '.settings.yaml', create: true);

  var dbname = settings['dbname'] as String;
  var username = settings['dbusername']as String;
  var password = settings['dbpassword']as String;
  var timeout = settings['timeout']as String;
  var ats = settings['ats']; // List<dynamic>

  List<String> atsAsString = yaml.asStringList['ats'];
  var hosts = settings['hosts'] as Map<String, String>;

  settings['a_String'] = 'hello world';
  settings['an_int'] = 10;
  settings['a_double'] = 10.0;
  settings['a_bool'] = true;

  var a_String = settings.asString('a_String');
  var an_int = settings.asInt('an_int');
  var a_double = settings.asDoule('a_double');
  var a_bool = settings.asBool('a_bool');


  print('dbname $dbname, username: $username, password: $password, timeout: $timeout');

  /// change something
  var newPassword = ask('password');
  settings['dbpassword'] = newPassword;
  settings['hosts'] = <String, String>{
        'host1': 'one',
        'host2': 'two',
        'host3': 'three';
  settings['ats'] = <String>['cat', 'bat', 'rat'];

  settings.save();
}

Access nested content

SettingsYaml also provides access to nested attributes using path selectors:

 var content = '''name: brett
hostname: slayer
port: 10
active: true
volume: 10.0
imageid: "65385002e970"
people:
  - person:
    name: brett
  - person:
    name: john
''';

    var settings = SettingsYaml.fromString(content: content, filePath: path);
    // use a path selector to read an attribute.
    expect(settings.selectAsString('people.person[0].name'), equals('brett'));
    expect(settings.selectAsString('people.person[1].name'), equals('john'));

Each selector is made up of a word.

To access an element in a list use 'word[n]'. where 'n' is the nth instance of the word in a yaml array.

e.g.

one:
  - two
  - two
    three:
      name: brett
      password: abc123

To return the value of four

settings.selectAsString('one.two[1].three.name') == 'brett'
settings.selectAsString('one.two[1].three.password') == 'abc123'

You can also get a Dart map or list:

settings.selectAsString('one.two[1].three') is Map;
settings.selectAsString('one')  is List;

Installing

Use this package as a library

Depend on it

Run this command:

With Dart:

 $ dart pub add settings_yaml

With Flutter:

 $ flutter pub add settings_yaml

This will add a line like this to your package's pubspec.yaml (and run an implicit dart pub get):

dependencies:
  settings_yaml: ^3.4.2

Alternatively, your editor might support dart pub get or flutter pub get. Check the docs for your editor to learn more.

Import it

Now in your Dart code, you can use:

import 'package:settings_yaml/settings_yaml.dart';

example/example.dart

/* Copyright (C) S. Brett Sutton - All Rights Reserved
 * Unauthorized copying of this file, via any medium is strictly prohibited
 * Proprietary and confidential
 * Written by Brett Sutton <bsutton@onepub.dev>, Jan 2022
 */

import 'package:settings_yaml/settings_yaml.dart';

void main() {
  save();

  load();
}

void save() {
  final settings = SettingsYaml.load(pathToSettings: '.settings.yaml');

  settings['dbname'] = 'billing';
  settings['dbusername'] = 'username';
  settings['dbpassword'] = 'apassword';
  settings['timeout'] = 300;
  settings['coefficient'] = 10.85;
  settings['active'] = true;

  settings.save();
}

void load() {
  final settings = SettingsYaml.load(pathToSettings: '.settings.yaml');

  assert(settings.validString('dbname'), 'Should be a string');
  assert(settings.validInt('timeout'), 'Should be an int');
  assert(settings.validDouble('coefficient'), 'Should be a double');
  assert(settings.validBool('active'), 'Should be a bool');

  final dbname = settings['dbname'] as String;
  // we haven't validated the dbusername and dbpassword so
  // they could be null.
  final username = settings['dbusername'] as String?;
  final password = settings['dbpassword'] as String?;

  final timeout = settings['timeout'] as int;
  final coefficient = settings['coefficient'] as double;
  final active = settings['active'] as bool;

  print('dbname $dbname, username: $username, password: $password, '
      'timeout: $timeout, coefficient: $coefficient, active: $active');

  settings.save();
}

Author: Onepub-dev
Source Code: https://github.com/onepub-dev/settings_yaml 
License: MIT license

#flutter #dart #setting #yaml 

Settings_yaml: A Very Simple Method to Use Yaml Files

Yaml-config: Yaml Configuration Support for CakePHP 3

Yaml for CakePHP 3

implements most of the YAML 1.2 specification using Symfony Yaml Component to CakePHP 3 for parsing config files

Requirements

The 3.0 branch has the following requirements:

  • CakePHP 3.0.0 or greater.

Installation

  • Install the plugin with composer from your CakePHP Project's ROOT directory (where composer.json file is located)
php composer.phar require chobo1210/yaml "dev-master"

OR add this lines to your composer.json

"require": {
  "chobo1210/Yaml": "dev-master"
}

And run php composer.phar update

then add this lines to your config/bootstrap.php

use Yaml\Configure\Engine\YamlConfig;

try {
  Configure::config('yaml', new YamlConfig());
  Configure::load('your_file', 'yaml');
} catch (\Exception $e) {
  die('Unable to load config/your_file.yml.');
}

your file must be in the config/ directory replace your_file with the name of your YAML file without the extension

Usage

if you want to use the plugin Shell to convert your current config add the lines to config/bootstrap.php

Plugin::load('Yaml');

then in your Console or Terminal type :

bin/cake convert

Optionally you can set the name of your YAML file ( without the extension ) default is : app.yml

bin/cake convert your_file_name

the file will be create in your config/ directory.

Example

debug: true

Asset:
  timestamp: true

# Default Database Connection informations
default_configuration: &default_configuration
  className: Cake\Database\Connection
  driver: Cake\Database\Driver\Mysql
  persistent: false
  host: localhost
  login: root
  password: secret
  prefix: false
  encoding: utf8
  timezone: +01:00
  cacheMetadata: true
  quoteIdentifiers: false  



# Datasources
Datasources:
  # Default datasource
  default: 
    <<: *default_configuration
    database: my_database
  # PHPUnit tests datasource
  test:
    <<: *default_configuration
    database: my_database_test



# Email Configuration
EmailTransport:
  default:
      className: Mail
      host: localhost
      port: 1025
      timeout: 30
      # username: user
      # password: secret
      client: null
      tls: null
Email:
  default:
      transport: default
      from: contact@localhost.dz
      cherset: utf-8
      headerCharset: utf-8   

Author: Guemidiborhane
Source Code: https://github.com/guemidiborhane/yaml-config 
License: 

#php #yaml #config #cakephp 

Yaml-config: Yaml Configuration Support for CakePHP 3

Yaml-route Provides Possibility to Configure CakePHP 3 Routes

Yaml-route     

Yaml-route provides possibility to configure CakePHP 3 routes with simple YAML files. This is basically just wrapper for CakePHP core routing that parses YAML files and makes proper calls to Cake\Core\Router.

Main goal is not to implement all fancy features that CakePHP 3 routing provides, only those ones that are really needed. Of course this is relative to developer, so feel free to fork and commit your own code.

Installation

composer require makallio85/yaml-route

Usage

  1. Replace all contents in config/routes.php file with single method call makallio85\YamlRoute\Generator::getInstance()->run()
  2. Load all plugins by calling makallio85\YamlRoute\Plugin::getInstance()->load($plugin, $options) Method is basically just wrapper for Cake\Core\Plugin::load() method. Note that Cake\Core\Plugin::loadAll() method is not supported and all plugins should be loaded one at time.
  3. Add your own route to routes.yml files to your project and desired plugins.

About route configuration

Every route is automatically named with its key. Root route should be named as root by convention. Route can contain path and config keys. Path is always string but config can be string that references to another YAML file that contains routes configuration. Syntax for external path is "PluginName.RouteFileNameWithoutExtension". All route configurations should be placed in config folder of project or plugin.

Route can also contain subroutes and they are defined inside config.routes key

config key can contain keys listed below

KeyTypeDescription
controllerstringRoute controller
actionstringRoute action
pluginstringRoute plugin
_methodarray or stringRoute method
extensionsarrayAllowed extensions
routesarraySubroutes
validatearrayList of variables to validate

Note that subroutes can't contain routes so config.routes for subroutes is not available.

Examples

Basic routing

config/routes.yml like this

root:
  path: /

Turns into this

\Cake\Routing\Router::scope('/', [], function ($routes) {
    $routes->fallbacks('DashedRoute');
});

\Cake\Core\Plugin::routes();

Plugin Routing

plugins/PluginCars/config/routes.yml like this

cars:
  path: /cars
  config:
    plugin: PluginCars
    controller: Cars
    action: index
    extensions:
      - json
      - xml
    routes:
      bmws_list:
        path: /bmws
        config:
          controller: Bmws
      bmws_view:
        path: /bmws/{id}
        config:
          _method: GET
          controller: Bmws
          action: view
          validate:
            id: '[0-9]+'
      bmws_add:
        path: /bmws/add
        config:
          _method: POST
          controller: Bmws
          action: add
      ladas:
        path: /ladas
        config:
          controller: Ladas

Turns into this

\Cake\Routing\Router::plugin('PluginCars', ['path' => '/cars'], function ($routes) {
  $routes->extensions(['0' => 'json', '1' => 'xml']);
  $routes->connect('/', ['plugin' => 'PluginCars', 'controller' => 'Cars', 'action' => 'index'], ['_name' => 'cars']);
  $routes->connect('/bmws', ['controller' => 'Bmws'], ['_name' => 'bmws_list']);
  $routes->connect('/bmws/:id', ['_method' => 'GET', 'controller' => 'Bmws', 'action' => 'view'], ['_name' => 'bmws_view', 'pass' => ['0' => 'id'], 'id' => '[0-9]+']);
  $routes->connect('/bmws/add', ['_method' => 'POST', 'controller' => 'Bmws', 'action' => 'add'], ['_name' => 'bmws_add']);
  $routes->connect('/ladas', ['controller' => 'Ladas'], ['_name' => 'ladas']);
  $routes->fallbacks('DashedRoute');
});

\Cake\Core\Plugin::routes();

Debugging

If you want to debug generated routes, you can set debug parameter to true when calling makallio85\YamlRoute\Generator::getInstance()->run(true). After that, you are able to get executed calls by calling makallio85\YamlRoute\Generator::getInstance()->getDump().

toDo

  • Add support for true inheritance by allowing subroute to contain subroute
  • Add tests Add more tests
  • Refactor classes
  • Add support for extensions
  • Improve exception handling

Author: Makallio85
Source Code: https://github.com/makallio85/yaml-route 
License: MIT license

#php #cakephp #yaml 

Yaml-route Provides Possibility to Configure CakePHP 3 Routes
Awesome  Rust

Awesome Rust

1654979700

STFU-8: Sorta Text format in UTF-8 Written In Rust

STFU-8 is a hacky text encoding/decoding protocol for data that might be not quite UTF-8 but is still mostly UTF-8. It is based on the syntax of the repr created when you write (or print) binary text in rust, python, C or other common programming languages.

Its primary purpose is to be able to allow a human to visualize and edit "data" that is mostly (or fully) visible UTF-8 text. It encodes all non visible or non UTF-8 compliant bytes as longform text (i.e. ESC becomes the full string r"\x1B"). It can also encode/decode ill-formed UTF-16.

Comparision to other formats:

  • UTF-8 (i.e. std::str): UTF-8 is a standardized format for encoding human understandable text in any language on the planet. It is the reason the internet can be understood by almost anyone and should be the primary way that text is encoded. However, not everything that is "UTF-8 like" follows the standard exactly. For instance:
    • The linux command line defines ANSI escape codes to provide styles like color, bold, italic, etc. Even though almost everything printed to a terminal is UTF-8 text these "escape codes" might not be, and even if they are UTF-8, they are not visible characters.
    • Windows paths are not necessarily UTF-8 compliant as they can have [ill formed text][utf-16-ill-formed-text].
    • There might be other cases you can think of or want to create. In general, try not to create more use cases if you don't have to.
  • rust's OsStr: OsStr is the "cross platform" type for handling system specific strings, mainly in file paths. Unlike STFU-8 it not (always) coercible into UTF-8 and therefore cannot be serialized into JSON or other formats.
  • WTF-8 (rust-wtf8): is great for interoperating with different UTF standards but cannot be used to transmit data over the internet. The spec states: "WTF-8 must not be used to represent text in a file format or for transmission over the Internet."
  • base64 (base64): also encodes binary data as UTF-8. If your data is actually binary (i.e. not text) then use base64. However, if your data was formerly text (or mostly text) then encoding to base64 will make it completely un(human)readable.
  • Array[u8]: obviously great if your data is actually binary (i.e. NOT TEXT) and you don't need to put it into a UTF-8 encoding. However, an array of bytes (i.e. [0x72, 0x65, 0x61, 0x64, 0x20, 0x69, 0x74] is not human readable. Even if it were in pure ASCII the only ones who can read it efficiently are low-level programming Gods who have never figured out how to debug-print their ASCII.
  • STFU-8 (this crate): is "good" when you want to have only printable/hand-editable text (and your data is mostly UTF-8) but the data might have a couple of binary/non-printable/ill-formed pieces. It is very poor if your data is actually binary, requiring (on average) a mapping of 4/1 for binary data.

Specification

In simple terms, encoded STFU-8 is itself always valid unicode which decodes to binary (the binary is not necessarily UTF-8). It differs from unicode in that single \ items are illegal. The following patterns are legal:

  • \\: decodes to the backward-slash (\) byte (\x5c)
  • \t: decodes to the tab byte (\x09)
  • \n: decodes to the newline byte (\x0A)
  • \r: decodes to the linefeed byte (\x0D)
  • \xXX where XX are exactly two case-insensitive hexidecimal digits: decodes to the \xXX byte, where XX is a hexidecimal number (example: \x9F, \xaB or \x05). This never gets resolved into a code point, the value is pushed directly into the decoder stream.
  • \uXXXXXX where XXXXXX are exacty six case-insensitive hexidecimal digits, decodes to a 24bit number that typically represenents a unicode code point. If the value is a unicode code point it will always be decoded as such. Otherwise stfu8 will attempt to store the value into the decoder (if the value is too large for the decoding type it will be an error).

stfu8 provides 2 different categories of functions for encoding/decoding data that are not necessarily interoperable (don't decode output created from encode_u8 with decode_u16).

  • encode_u8(&[u8]) -> String and decode_u8(&str) -> Vec<u8>: encodes or decodes an array of u8 values to/from STFU-8, primarily used for interfacing with binary/nonvisible data that is almost UTF-8.
  • encode_u16(&[u16]) -> String and decode_u16(&str) -> Vec<u16>: encodes or decodes an array of u16 values to/from STFU-8, primarily used for interfacing with legacy UTF-16 formats that may contain [ill formed text][utf-16-ill-formed-text] but also converts unprintable characters.

There are some general rules for encoding and decoding:

  • If \u... cannot be resolved into a valid UTF code point it must fit into the decoder. For instance, trying to decode "\u00DEED" (which is an UTF-16 Trail surrogage) using decode_u8 will fail, but will succeed with decode_u16.
  • No escaped values are ever chained. For example, "\x01\x02" will be [0x01, 0x02] not [0x0102] -- even if you use decode_u16.
  • Values escaped with \x... are always copied verbatum into the decoder. I.e. \xFF is a valid UTF-32 code point, but if decoded with decode_u8 it will be 0xFE in the buffer, not two bytes of data as the UTF-8 character 'þ'. Note that with decode_u16 0xFE is a valid UTF-16 code point, so when re-encoded would be the 'þ' character. Moral of the story: don't mix inputs/outputs of the the u8 and u16 functions.

tab, newline, and line-feed characters are "visible", so encoding with them in "pretty form" is optional.

UTF-16 Ill Formed Text

The problem is succinctly stated here:

http://unicode.org/faq/utf_bom.html

Q: How do I convert an unpaired UTF-16 surrogate to UTF-8?

A different issue arises if an unpairedsurrogate is encountered when converting ill-formed UTF-16 data. By represented such an unpaired surrogate on its own as a 3-byte sequence, the resulting UTF-8 data stream would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in valid data stream. Therefore a convertermust treat this as an error. [AF]

Also, from the WTF-8 spec

As a result, [unpaired] surrogates do occur in practice and need to be preserved. For example:

In ECMAScript (a.k.a. JavaScript), a String value is defined as a sequence of 16-bit integers that usually represents UTF-16 text but may or may not be well-formed. Windows applications normally use UTF-16, but the file system treats path and file names as an opaque sequence of WCHARs (16-bit code units).

We say that strings in these systems are encoded in potentially ill-formed UTF-16 or WTF-16.

Basically: you can't (always) convert from UTF-16 to UTF-8 and it's a real bummer. WTF-8, while kindof an answer to this problem, doesn't allow me to serialize UTF-16 into a UTF-8 format, send it to my webapp, edit it (as a human), and send it back. That is what STFU-8 is for.

Download Details:
Author: vitiral
Source Code: https://github.com/vitiral/stfu8
License: Unknown, MIT licenses found

#rust  #rustlang  #encode  #yaml 

STFU-8: Sorta Text format in UTF-8 Written In Rust
Awesome  Rust

Awesome Rust

1654972380

YAML Rust: A Pure Rust YAML Implementation

yaml-rust

The missing YAML 1.2 implementation for Rust.

yaml-rust is a pure Rust YAML 1.2 implementation, which enjoys the memory safety property and other benefits from the Rust language. The parser is heavily influenced by libyaml and yaml-cpp.

Quick Start

Add the following to the Cargo.toml of your project:

[dependencies]
yaml-rust = "0.4"

and import:

extern crate yaml_rust;

Use yaml::YamlLoader to load the YAML documents and access it as Vec/HashMap:

extern crate yaml_rust;
use yaml_rust::{YamlLoader, YamlEmitter};

fn main() {
    let s =
"
foo:
    - list1
    - list2
bar:
    - 1
    - 2.0
";
    let docs = YamlLoader::load_from_str(s).unwrap();

    // Multi document support, doc is a yaml::Yaml
    let doc = &docs[0];

    // Debug support
    println!("{:?}", doc);

    // Index access for map & array
    assert_eq!(doc["foo"][0].as_str().unwrap(), "list1");
    assert_eq!(doc["bar"][1].as_f64().unwrap(), 2.0);

    // Chained key/array access is checked and won't panic,
    // return BadValue if they are not exist.
    assert!(doc["INVALID_KEY"][100].is_badvalue());

    // Dump the YAML object
    let mut out_str = String::new();
    {
        let mut emitter = YamlEmitter::new(&mut out_str);
        emitter.dump(doc).unwrap(); // dump the YAML object to a String
    }
    println!("{}", out_str);
}

Note that yaml_rust::Yaml implements Index<&'a str> & Index<usize>:

  • Index<usize> assumes the container is an Array
  • Index<&'a str> assumes the container is a string to value Map
  • otherwise, Yaml::BadValue is returned

If your document does not conform to this convention (e.g. map with complex type key), you can use the Yaml::as_XXX family API to access your documents.

Features

  • Pure Rust
  • Ruby-like Array/Hash access API
  • Low-level YAML events emission

Specification Compliance

This implementation aims to provide YAML parser fully compatible with the YAML 1.2 specification. The parser can correctly parse almost all examples in the specification, except for the following known bugs:

  • Empty plain scalar in certain contexts

However, the widely used library libyaml also fails to parse these examples, so it may not be a huge problem for most users.

Goals

  • Encoder
  • Tag directive
  • Alias while deserialization

Minimum Rust version policy

This crate's minimum supported rustc version is 1.31 (released with Rust 2018, after v0.4.3), as this is the currently known minimum version for regex as well.

Contribution

Fork & PR on Github.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Download Details:
Author: chyh1990
Source Code: https://github.com/chyh1990/yaml-rust
License: Apache-2.0, MIT licenses found

#rust #yaml  #rustlang  #encode 

YAML Rust: A Pure Rust YAML Implementation
Hermann  Frami

Hermann Frami

1654488420

Serverless Plugin IfElse

Serverless Plugin IfElse

While you can use serverless variables to define different values for your atrributes based on either stage or other properties, it sometimes is not as straightforward.

For example, If you have a serverless project with 3 functions and you want to deploy all 3 functions in one region but only 2 of them in other region, then there is no easier way to exclude the third function based on region you are deploying.

Another use case that inspired me to write this plugin was, I wanted to use iamRoleStatements for all my Lambda functions in staging but use a pre-define role in production. You cannot have both attributes in serverless.yml file as serverless ignores iamRoleStatements if there is role attribute.

This plugin allows you to write if else conditions in serverless.yml file to add, remove or change the values of attributes in the yml file. It works with both package and deploy commands. It also works with serverless-offline plugin.

Installation

npm i serverless-plugin-ifelse --save-dev

Example

serverless.yml

service: serverlessIfElseExample
custom:
  currentStage: ${opt:stage, self:provider.stage}
  serverlessIfElse:
      - If: '"${self:custom.currentStage}" == "dev"'
        Exclude:
          - provider.role
          - provider.environment.ENV1
          - functions.func3
          - functions.func1.events.0.http.authorizer
        Set:
          provider.timeout: 90
          provider.profile: dev
        ElseExclude:
          - provider.environment.ENV2
        ElseSet:
           provider.timeout: 120

      - ExcludeIf:
           functions.func1: '"${self:provider.region}" == "ap-southeast-2"'
           functions.func2: '"${opt:region}" == "us-east-1"'

      - If: '"${self:provider.region}" == "us-east-1"'
        Exclude:
          - functions.func1
        Set:
          provider.timeout: 300


plugins:
  - serverless-plugin-ifelse

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region : ap-southeast-2
  timeout: 60
  environment:
    ENV1: env-val-1
    ENV2: env-val-2
  profile: default
  iamRoleStatements:
    - Effect: Allow
      Action:
      - s3:*
      Resource: "*"
  role: arn:aws:iam::xxxxxxxxxxxx:role/testRole

functions:
  func1:
    name: Function 1
    handler: lambda.func1
    events:
      - http:
          path: "path1"
          method: "post"
          authorizer:
            arn: arn:aws:cognito-idp:us-east-1:123456789012:userpool/us-east-1_0acCDefgh

  func2:
    name: Function 2
    handler: lambda.func2
    events:
      - http:
           path: "path2"
           method: "post"

  func3:
    name: Function 3
    handler: lambda.func2
    events:
      - http:
           path: "path3"
           method: "post"

Put your conditions in custom variable serverlessIfElse.

- If

Write your if condition inside single quote ''. Inside your condition, all your variables that resolve in string or the string themselves should be inside double quote. The plugin will otherwise encounter undefined variable error and the condition will not work. The condition can be anything, like == , !=, <, > or even javascript regular expressions.

- If: '"${self:provider.stage}" == "dev"'

Exclude

If condition in If is true, all attibutes in Exclude will be ignored before serverless package or deploy your stack and hence serverless will not see those attributes.

Set

If condition in If is true, the value of the attribute will be updated with new value.

ElseExclude

If condition in If is false,the attibutes will be ignored.

ElseSet

If condition in If is false, the value of the attribute will be updated with new value.

- ExcludeIf

Use ExcludeIf, if you want to write conditions per attribute. If condition is true, only that attribute will be ignored.

You can write as many conditions as you like and exclude or set attributes any level deep in the yml file.

Note

This plugin will ignore or update value of attributes based on your conditions and does not evaluate if it causes any side effect. You are responsbile to make sure ignoring or setting new values will work as you expected and will not cause serverless to throw error.

The plugin will not remove or update any first level attributes in serverless.yml file like service or provider or functions.

Warning

The conditions are executed using eval, which executes the conditions as Javascript. Basically any javascript code in If or Else conditions are executed. This may cause security issues, especially if you have multiple (or community) contributors. Please make sure the code is properly reviewed and it does not cause any security issues.

Author: Anantab
Source Code: https://github.com/anantab/serverless-plugin-ifelse 
License: 

#serverless #plugin #yaml 

Serverless Plugin IfElse