1654349520
Zek: Generate A Go Struct From XML
zek
Zek is a prototype for creating a Go struct from an XML document. The resulting struct works best for reading XML (see also #14), to create XML, you might want to use something else.
It was developed at Leipzig University Library to shorten the time to go from raw XML to a struct that allows to access XML data in Go programs.
Skip the fluff, just the code.
Given some XML, run:
$ curl -s https://raw.githubusercontent.com/miku/zek/master/fixtures/e.xml | zek -e
// Rss was generated 2018-08-30 20:24:14 by tir on sol.
type Rss struct {
XMLName xml.Name `xml:"rss"`
Text string `xml:",chardata"`
Rdf string `xml:"rdf,attr"`
Dc string `xml:"dc,attr"`
Geoscan string `xml:"geoscan,attr"`
Media string `xml:"media,attr"`
Gml string `xml:"gml,attr"`
Taxo string `xml:"taxo,attr"`
Georss string `xml:"georss,attr"`
Content string `xml:"content,attr"`
Geo string `xml:"geo,attr"`
Version string `xml:"version,attr"`
Channel struct {
Text string `xml:",chardata"`
Title string `xml:"title"` // ESS New Releases (Display...
Link string `xml:"link"` // http://tinyurl.com/ESSNew...
Description string `xml:"description"` // New releases from the Ear...
LastBuildDate string `xml:"lastBuildDate"` // Mon, 27 Nov 2017 00:06:35...
Item []struct {
Text string `xml:",chardata"`
Title string `xml:"title"` // Surficial geology, Aberde...
Link string `xml:"link"` // https://geoscan.nrcan.gc....
Description string `xml:"description"` // Geological Survey of Cana...
Guid struct {
Text string `xml:",chardata"` // 304279, 306212, 306175, 3...
IsPermaLink string `xml:"isPermaLink,attr"`
} `xml:"guid"`
PubDate string `xml:"pubDate"` // Fri, 24 Nov 2017 00:00:00...
Polygon []string `xml:"polygon"` // 64.0000 -98.0000 64.0000 ...
Download string `xml:"download"` // https://geoscan.nrcan.gc....
License string `xml:"license"` // http://data.gc.ca/eng/ope...
Author string `xml:"author"` // Geological Survey of Cana...
Source string `xml:"source"` // Geological Survey of Cana...
SndSeries string `xml:"SndSeries"` // Bedford Institute of Ocea...
Publisher string `xml:"publisher"` // Natural Resources Canada,...
Edition string `xml:"edition"` // prelim., surficial data m...
Meeting string `xml:"meeting"` // Geological Association of...
Documenttype string `xml:"documenttype"` // serial, open file, serial...
Language string `xml:"language"` // English, English, English...
Maps string `xml:"maps"` // 1 map, 5 maps, Publicatio...
Mapinfo string `xml:"mapinfo"` // surficial geology, surfic...
Medium string `xml:"medium"` // on-line; digital, digital...
Province string `xml:"province"` // Nunavut, Northwest Territ...
Nts string `xml:"nts"` // 066B, 095J; 095N; 095O; 0...
Area string `xml:"area"` // Aberdeen Lake, Mackenzie ...
Subjects string `xml:"subjects"`
Program string `xml:"program"` // GEM2: Geo-mapping for Ene...
Project string `xml:"project"` // Rae Province Project Mana...
Projectnumber string `xml:"projectnumber"` // 340521, 343202, 340557, 3...
Abstract string `xml:"abstract"` // This new surficial geolog...
Links string `xml:"links"` // Online - En ligne (PDF, 9...
Readme string `xml:"readme"` // readme | https://geoscan....
PPIid string `xml:"PPIid"` // 34532, 35096, 35438, 2563...
} `xml:"item"`
} `xml:"channel"`
}
Online
Try it online at https://www.onlinetool.io/xmltogo/ -- thanks, kjk!
About
Upsides:
- it works fine for non-recursive structures,
- does not need XSD or DTD,
- it is relatively convenient to access attributes, children and text,
- will generate a single struct, which make for a quite compact representation,
- simple user interface,
- comments with examples,
- schema inference across multiple files.
Downsides:
- experimental, early, buggy, unstable prototype,
- no support for recursive types (similar to Russian Doll strategy, [1])
- no type inference, everything is accessible as string.
Bugs:
Mapping between XML elements and data structures is inherently flawed: an XML element is an order-dependent collection of anonymous values, while a data structure is an order-independent collection of named values.
https://golang.org/pkg/encoding/xml/#pkg-note-BUG
Related projects:
- https://github.com/bemasher/JSONGen
- https://github.com/dutchcoders/XMLGen
- https://github.com/gnewton/chidley
And other awesome XML utilities.
Presentations:
Install
$ go install github.com/miku/zek/cmd/zek@latest
Debian and RPM packages:
It's in AUR, too.
Usage
$ zek -h
Usage of zek:
-B use a fixed banner string (e.g. for CI)
-C emit less compact struct
-F skip formatting
-P string
if set, write out struct within a package with the given name
-c emit more compact struct (noop, as this is the default since 0.1.7)
-d debug output
-e add comments with example
-j add JSON tags
-m omit empty Text fields
-max-examples int
limit number of examples (default 10)
-n string
use a different name for the top-level struct
-o string
if set, write to output file, not stdout
-p write out an example program
-s strict parsing and writing
-t string
emit struct for tag matching this name
-u filter out duplicated examples
-version
show version
-x int
max chars for example (default 25)
Examples:
$ cat fixtures/a.xml
<a></a>
$ zek -C < fixtures/a.xml
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
}
Debug output dumps the internal tree as JSON to stdout.
$ zek -d < fixtures/a.xml
{"name":{"Space":"","Local":"a"}}
Example program:
package main
import (
"encoding/json"
"encoding/xml"
"fmt"
"log"
"os"
)
// A was generated 2017-12-05 17:35:21 by tir on apollo.
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
}
func main() {
dec := xml.NewDecoder(os.Stdin)
var doc A
if err := dec.Decode(&doc); err != nil {
log.Fatal(err)
}
b, err := json.Marshal(doc)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
$ zek -C -p < fixtures/a.xml > sample.go && go run sample.go < fixtures/a.xml | jq . && rm sample.go
{
"XMLName": {
"Space": "",
"Local": "a"
},
"Text": ""
}
More complex example:
$ zek < fixtures/d.xml
// Root was generated 2019-06-11 16:27:04 by tir on hayiti.
type Root struct {
XMLName xml.Name `xml:"root"`
Text string `xml:",chardata"`
A []struct {
Text string `xml:",chardata"`
B []struct {
Text string `xml:",chardata"`
C string `xml:"c"`
D string `xml:"d"`
} `xml:"b"`
} `xml:"a"`
}
$ zek -p < fixtures/d.xml > sample.go && go run sample.go < fixtures/d.xml | jq . && rm sample.go
{
"XMLName": {
"Space": "",
"Local": "root"
},
"Text": "\n\n\n\n",
"A": [
{
"Text": "\n \n \n",
"B": [
{
"Text": "\n \n ",
"C": "Hi",
"D": ""
},
{
"Text": "\n \n \n ",
"C": "World",
"D": ""
}
]
},
{
"Text": "\n \n",
"B": [
{
"Text": "\n \n ",
"C": "Hello",
"D": ""
}
]
},
{
"Text": "\n \n",
"B": [
{
"Text": "\n \n ",
"C": "",
"D": "World"
}
]
}
]
}
Annotate with comments:
$ zek -e < fixtures/l.xml
// Records was generated 2019-06-11 16:29:35 by tir on hayiti.
type Records struct {
XMLName xml.Name `xml:"Records"`
Text string `xml:",chardata"` // \n
Xsi string `xml:"xsi,attr"`
Record []struct {
Text string `xml:",chardata"`
Header struct {
Text string `xml:",chardata"`
Status string `xml:"status,attr"`
Identifier string `xml:"identifier"` // oai:ojs.localhost:article...
Datestamp string `xml:"datestamp"` // 2009-06-24T14:48:23Z, 200...
SetSpec string `xml:"setSpec"` // eppp:ART, eppp:ART, eppp:...
} `xml:"header"`
Metadata struct {
Text string `xml:",chardata"`
Rfc1807 struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
BibVersion string `xml:"bib-version"` // v2, v2, v2...
ID string `xml:"id"` // http://jou...
Entry string `xml:"entry"` // 2009-06-24...
Organization []string `xml:"organization"` // Proceeding...
Title string `xml:"title"` // Introducti...
Type string `xml:"type"`
Author []string `xml:"author"` // KRAMPEN, G..
Copyright string `xml:"copyright"` // Das Urhebe...
OtherAccess string `xml:"other_access"` // url:http:/...
Keyword string `xml:"keyword"`
Period []string `xml:"period"`
Monitoring string `xml:"monitoring"`
Language string `xml:"language"` // en, en, en, e...
Abstract string `xml:"abstract"` // After a short...
Date string `xml:"date"` // 2009-06-22 12...
} `xml:"rfc1807"`
} `xml:"metadata"`
About string `xml:"about"`
} `xml:"Record"`
}
Only consider a nested element
$ zek -t metadata fixtures/z.xml
// Metadata was generated 2019-06-11 16:33:26 by tir on hayiti.
type Metadata struct {
XMLName xml.Name `xml:"metadata"`
Text string `xml:",chardata"`
Dc struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Title struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
} `xml:"title"`
Identifier struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
} `xml:"identifier"`
Rights struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Lang string `xml:"lang,attr"`
} `xml:"rights"`
AccessRights struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
} `xml:"accessRights"`
} `xml:"dc"`
}
Inference across files
$ zek fixtures/a.xml fixtures/b.xml fixtures/c.xml
// A was generated 2017-12-05 17:40:14 by tir on apollo.
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
B []struct {
Text string `xml:",chardata"`
} `xml:"b"`
}
This is also useful, if you deal with archives containing XML files:
$ unzip -p 4082359.zip '*.xml' | zek -e
Given a directory full of zip files, you can combined find, unzip and zek:
$ for i in $(find ftp/b571 -type f -name "*zip"); do unzip -p $i '*xml'; done | zek -e
Another example (tarball with thousands of XML files, seemingly MARC):
$ tar -xOzf /tmp/20180725.125255.tar.gz | zek -e
// OAIPMH was generated 2018-09-26 15:03:29 by tir on sol.
type OAIPMH struct {
XMLName xml.Name `xml:"OAI-PMH"`
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
ListRecords struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Header struct {
Text string `xml:",chardata"`
Identifier struct {
Text string `xml:",chardata"` // aleph-pub:000000001, ...
} `xml:"identifier"`
} `xml:"header"`
Metadata struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
Leader struct
Text string `xml:",chardata"` // 00001nM2.01200024
} `xml:"leader"`
Controlfield []struct {
Text string `xml:",chardata"` // 00001nM2.01200024
Tag string `xml:"tag,attr"`
} `xml:"controlfield"`
Datafield []struct {
Text string `xml:",chardata"`
Tag string `xml:"tag,attr"`
Ind1 string `xml:"ind1,attr"`
Ind2 string `xml:"ind2,attr"`
Subfield []struct {
Text string `xml:",chardata"` // KM0000002
Code string `xml:"code,attr"`
} `xml:"subfield"`
} `xml:"datafield"`
} `xml:"record"`
} `xml:"metadata"`
} `xml:"record"`
} `xml:"ListRecords"`
}
Generate a package
If you want in include generated file in the build process, e.g. with go generate, you may find -P and -o helpful.
$ cat fixtures/b.xml
<a><b></b></a>
Run on the command line or via go generate:
$ zek -P mypkg -o data.go < fixtures/b.xml
This would write out the following in data.go file:
// Code generated by zek; DO NOT EDIT.
package mypkg
import "encoding/xml"
// A was generated 2021-09-16 11:23:06 by tir on trieste.
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
B string `xml:"b"`
}
Note that any existing file will be overwritten, without any warning.
Misc
As a side effect, zek seems to be a useful for debugging. Example:
This record is emitted from a typical OAI server (OJS, not even uncommon), yet one can quickly spot the flaw in the structure.
Over 30 different struct generated manually in the course of a few hours (around five minutes per source): https://git.io/vbTDo.
-- Current extent leader: 1532 lines struct
Author: Miku
Source Code: https://github.com/miku/zek
License: GPL-3.0 license
What is GEEK
Buddha Community
1678831740
Go-xmlstruct: Generate Go Structs From Multiple XML Documents
Go-xmlstruct
Generate Go structs from multiple XML documents.
What does go-xmlstruct do and why should I use it?
go-xmlstruct generates Go structs from XML documents. Alternatively put, go-xmlstruct infers XML schemas from one or more example XML documents. For example, given this XML document, go-xmlstruct generates this Go source code.
Compared to existing Go struct generators like zek, XMLGen, and chidley, go-xmlstruct offers:
- Takes multiple XML documents as input.
- Generates field types of
bool,int,string, ortime.Timeas appropriate. - Creates named types for all elements.
- Handles optional attributes and elements.
- Handles repeated attributes and elements.
- Ignores empty chardata.
- Provides a CLI for simple use.
- Usable as a Go package for advanced use, including configurable field naming.
go-xmlstruct is useful for quick-and-dirty unmarshalling of arbitrary XML documents, especially when you have no schema or the schema is extremely complex and you want something that "just works" with the documents you have.
Install
Install the goxmlstruct CLI with:
$ go install github.com/twpayne/go-xmlstruct/cmd/goxmlstruct@latest
Example
Feed goxmlstruct the simple XML document:
<parent>
<child flag="true">
chardata
</child>
</parent>
by running:
$ echo '<parent><child flag="true">text</child></parent>' | goxmlstruct
This produces the output:
// This file is automatically generated. DO NOT EDIT.
package main
type Parent struct {
Child struct {
Flag bool `xml:"flag,attr"`
CharData string `xml:",chardata"`
} `xml:"child"`
}
This demonstrates:
- A Go struct is generated from the structure of the input XML document.
- Attributes, child elements, and chardata are all considered.
- Field names are generated automatically.
- Field types are detected automatically.
For a full list of options to the goxmlstruct CLI run:
$ goxmlstruct -help
You can run a more advanced example with:
$ git clone https://github.com/twpayne/go-xmlstruct.git
$ cd go-xmlstruct
$ goxmlstruct internal/tests/gpx/testdata/*.gpx
This demonstrates generating a Go struct from multiple XML complex documents.
For an example of configurable field naming and named types by using go-xmlstruct as a package, see internal/tests/play/play_test.go.
For an example of a complex schema, see internal/tests/aixm/aixm_test.go.
How does go-xmlstruct work?
Similar to go-jsonstruct, go-xmlstruct consists of two phases:
- Firstly, go-xmlstruct explores all input XML documents to determine their structure. It gathers statistics on the types used for each attribute, chardata, and child element.
- Secondly, go-xmlstruct generates a Go struct based on the observed structure using the gathered statistics to determine the type of each field.
Download Details:
Author: twpayne
Source Code: https://github.com/twpayne/go-xmlstruct
License: MIT license
1599854400
What's new in the go 1.15
Go announced Go 1.15 version on 11 Aug 2020. Highlighted updates and features include Substantial improvements to the Go linker, Improved allocation for small objects at high core counts, X.509 CommonName deprecation, GOPROXY supports skipping proxies that return errors, New embedded tzdata package, Several Core Library improvements and more.
As Go promise for maintaining backward compatibility. After upgrading to the latest Go 1.15 version, almost all existing Golang applications or programs continue to compile and run as older Golang version.
#go #golang #go 1.15 #go features #go improvement #go package #go new features
1654349520
Zek: Generate A Go Struct From XML
zek
Zek is a prototype for creating a Go struct from an XML document. The resulting struct works best for reading XML (see also #14), to create XML, you might want to use something else.
It was developed at Leipzig University Library to shorten the time to go from raw XML to a struct that allows to access XML data in Go programs.
Skip the fluff, just the code.
Given some XML, run:
$ curl -s https://raw.githubusercontent.com/miku/zek/master/fixtures/e.xml | zek -e
// Rss was generated 2018-08-30 20:24:14 by tir on sol.
type Rss struct {
XMLName xml.Name `xml:"rss"`
Text string `xml:",chardata"`
Rdf string `xml:"rdf,attr"`
Dc string `xml:"dc,attr"`
Geoscan string `xml:"geoscan,attr"`
Media string `xml:"media,attr"`
Gml string `xml:"gml,attr"`
Taxo string `xml:"taxo,attr"`
Georss string `xml:"georss,attr"`
Content string `xml:"content,attr"`
Geo string `xml:"geo,attr"`
Version string `xml:"version,attr"`
Channel struct {
Text string `xml:",chardata"`
Title string `xml:"title"` // ESS New Releases (Display...
Link string `xml:"link"` // http://tinyurl.com/ESSNew...
Description string `xml:"description"` // New releases from the Ear...
LastBuildDate string `xml:"lastBuildDate"` // Mon, 27 Nov 2017 00:06:35...
Item []struct {
Text string `xml:",chardata"`
Title string `xml:"title"` // Surficial geology, Aberde...
Link string `xml:"link"` // https://geoscan.nrcan.gc....
Description string `xml:"description"` // Geological Survey of Cana...
Guid struct {
Text string `xml:",chardata"` // 304279, 306212, 306175, 3...
IsPermaLink string `xml:"isPermaLink,attr"`
} `xml:"guid"`
PubDate string `xml:"pubDate"` // Fri, 24 Nov 2017 00:00:00...
Polygon []string `xml:"polygon"` // 64.0000 -98.0000 64.0000 ...
Download string `xml:"download"` // https://geoscan.nrcan.gc....
License string `xml:"license"` // http://data.gc.ca/eng/ope...
Author string `xml:"author"` // Geological Survey of Cana...
Source string `xml:"source"` // Geological Survey of Cana...
SndSeries string `xml:"SndSeries"` // Bedford Institute of Ocea...
Publisher string `xml:"publisher"` // Natural Resources Canada,...
Edition string `xml:"edition"` // prelim., surficial data m...
Meeting string `xml:"meeting"` // Geological Association of...
Documenttype string `xml:"documenttype"` // serial, open file, serial...
Language string `xml:"language"` // English, English, English...
Maps string `xml:"maps"` // 1 map, 5 maps, Publicatio...
Mapinfo string `xml:"mapinfo"` // surficial geology, surfic...
Medium string `xml:"medium"` // on-line; digital, digital...
Province string `xml:"province"` // Nunavut, Northwest Territ...
Nts string `xml:"nts"` // 066B, 095J; 095N; 095O; 0...
Area string `xml:"area"` // Aberdeen Lake, Mackenzie ...
Subjects string `xml:"subjects"`
Program string `xml:"program"` // GEM2: Geo-mapping for Ene...
Project string `xml:"project"` // Rae Province Project Mana...
Projectnumber string `xml:"projectnumber"` // 340521, 343202, 340557, 3...
Abstract string `xml:"abstract"` // This new surficial geolog...
Links string `xml:"links"` // Online - En ligne (PDF, 9...
Readme string `xml:"readme"` // readme | https://geoscan....
PPIid string `xml:"PPIid"` // 34532, 35096, 35438, 2563...
} `xml:"item"`
} `xml:"channel"`
}
Online
Try it online at https://www.onlinetool.io/xmltogo/ -- thanks, kjk!
About
Upsides:
- it works fine for non-recursive structures,
- does not need XSD or DTD,
- it is relatively convenient to access attributes, children and text,
- will generate a single struct, which make for a quite compact representation,
- simple user interface,
- comments with examples,
- schema inference across multiple files.
Downsides:
- experimental, early, buggy, unstable prototype,
- no support for recursive types (similar to Russian Doll strategy, [1])
- no type inference, everything is accessible as string.
Bugs:
Mapping between XML elements and data structures is inherently flawed: an XML element is an order-dependent collection of anonymous values, while a data structure is an order-independent collection of named values.
https://golang.org/pkg/encoding/xml/#pkg-note-BUG
Related projects:
- https://github.com/bemasher/JSONGen
- https://github.com/dutchcoders/XMLGen
- https://github.com/gnewton/chidley
And other awesome XML utilities.
Presentations:
Install
$ go install github.com/miku/zek/cmd/zek@latest
Debian and RPM packages:
It's in AUR, too.
Usage
$ zek -h
Usage of zek:
-B use a fixed banner string (e.g. for CI)
-C emit less compact struct
-F skip formatting
-P string
if set, write out struct within a package with the given name
-c emit more compact struct (noop, as this is the default since 0.1.7)
-d debug output
-e add comments with example
-j add JSON tags
-m omit empty Text fields
-max-examples int
limit number of examples (default 10)
-n string
use a different name for the top-level struct
-o string
if set, write to output file, not stdout
-p write out an example program
-s strict parsing and writing
-t string
emit struct for tag matching this name
-u filter out duplicated examples
-version
show version
-x int
max chars for example (default 25)
Examples:
$ cat fixtures/a.xml
<a></a>
$ zek -C < fixtures/a.xml
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
}
Debug output dumps the internal tree as JSON to stdout.
$ zek -d < fixtures/a.xml
{"name":{"Space":"","Local":"a"}}
Example program:
package main
import (
"encoding/json"
"encoding/xml"
"fmt"
"log"
"os"
)
// A was generated 2017-12-05 17:35:21 by tir on apollo.
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
}
func main() {
dec := xml.NewDecoder(os.Stdin)
var doc A
if err := dec.Decode(&doc); err != nil {
log.Fatal(err)
}
b, err := json.Marshal(doc)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
$ zek -C -p < fixtures/a.xml > sample.go && go run sample.go < fixtures/a.xml | jq . && rm sample.go
{
"XMLName": {
"Space": "",
"Local": "a"
},
"Text": ""
}
More complex example:
$ zek < fixtures/d.xml
// Root was generated 2019-06-11 16:27:04 by tir on hayiti.
type Root struct {
XMLName xml.Name `xml:"root"`
Text string `xml:",chardata"`
A []struct {
Text string `xml:",chardata"`
B []struct {
Text string `xml:",chardata"`
C string `xml:"c"`
D string `xml:"d"`
} `xml:"b"`
} `xml:"a"`
}
$ zek -p < fixtures/d.xml > sample.go && go run sample.go < fixtures/d.xml | jq . && rm sample.go
{
"XMLName": {
"Space": "",
"Local": "root"
},
"Text": "\n\n\n\n",
"A": [
{
"Text": "\n \n \n",
"B": [
{
"Text": "\n \n ",
"C": "Hi",
"D": ""
},
{
"Text": "\n \n \n ",
"C": "World",
"D": ""
}
]
},
{
"Text": "\n \n",
"B": [
{
"Text": "\n \n ",
"C": "Hello",
"D": ""
}
]
},
{
"Text": "\n \n",
"B": [
{
"Text": "\n \n ",
"C": "",
"D": "World"
}
]
}
]
}
Annotate with comments:
$ zek -e < fixtures/l.xml
// Records was generated 2019-06-11 16:29:35 by tir on hayiti.
type Records struct {
XMLName xml.Name `xml:"Records"`
Text string `xml:",chardata"` // \n
Xsi string `xml:"xsi,attr"`
Record []struct {
Text string `xml:",chardata"`
Header struct {
Text string `xml:",chardata"`
Status string `xml:"status,attr"`
Identifier string `xml:"identifier"` // oai:ojs.localhost:article...
Datestamp string `xml:"datestamp"` // 2009-06-24T14:48:23Z, 200...
SetSpec string `xml:"setSpec"` // eppp:ART, eppp:ART, eppp:...
} `xml:"header"`
Metadata struct {
Text string `xml:",chardata"`
Rfc1807 struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
BibVersion string `xml:"bib-version"` // v2, v2, v2...
ID string `xml:"id"` // http://jou...
Entry string `xml:"entry"` // 2009-06-24...
Organization []string `xml:"organization"` // Proceeding...
Title string `xml:"title"` // Introducti...
Type string `xml:"type"`
Author []string `xml:"author"` // KRAMPEN, G..
Copyright string `xml:"copyright"` // Das Urhebe...
OtherAccess string `xml:"other_access"` // url:http:/...
Keyword string `xml:"keyword"`
Period []string `xml:"period"`
Monitoring string `xml:"monitoring"`
Language string `xml:"language"` // en, en, en, e...
Abstract string `xml:"abstract"` // After a short...
Date string `xml:"date"` // 2009-06-22 12...
} `xml:"rfc1807"`
} `xml:"metadata"`
About string `xml:"about"`
} `xml:"Record"`
}
Only consider a nested element
$ zek -t metadata fixtures/z.xml
// Metadata was generated 2019-06-11 16:33:26 by tir on hayiti.
type Metadata struct {
XMLName xml.Name `xml:"metadata"`
Text string `xml:",chardata"`
Dc struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Title struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
} `xml:"title"`
Identifier struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
} `xml:"identifier"`
Rights struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Lang string `xml:"lang,attr"`
} `xml:"rights"`
AccessRights struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
} `xml:"accessRights"`
} `xml:"dc"`
}
Inference across files
$ zek fixtures/a.xml fixtures/b.xml fixtures/c.xml
// A was generated 2017-12-05 17:40:14 by tir on apollo.
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
B []struct {
Text string `xml:",chardata"`
} `xml:"b"`
}
This is also useful, if you deal with archives containing XML files:
$ unzip -p 4082359.zip '*.xml' | zek -e
Given a directory full of zip files, you can combined find, unzip and zek:
$ for i in $(find ftp/b571 -type f -name "*zip"); do unzip -p $i '*xml'; done | zek -e
Another example (tarball with thousands of XML files, seemingly MARC):
$ tar -xOzf /tmp/20180725.125255.tar.gz | zek -e
// OAIPMH was generated 2018-09-26 15:03:29 by tir on sol.
type OAIPMH struct {
XMLName xml.Name `xml:"OAI-PMH"`
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
ListRecords struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Header struct {
Text string `xml:",chardata"`
Identifier struct {
Text string `xml:",chardata"` // aleph-pub:000000001, ...
} `xml:"identifier"`
} `xml:"header"`
Metadata struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
Leader struct
Text string `xml:",chardata"` // 00001nM2.01200024
} `xml:"leader"`
Controlfield []struct {
Text string `xml:",chardata"` // 00001nM2.01200024
Tag string `xml:"tag,attr"`
} `xml:"controlfield"`
Datafield []struct {
Text string `xml:",chardata"`
Tag string `xml:"tag,attr"`
Ind1 string `xml:"ind1,attr"`
Ind2 string `xml:"ind2,attr"`
Subfield []struct {
Text string `xml:",chardata"` // KM0000002
Code string `xml:"code,attr"`
} `xml:"subfield"`
} `xml:"datafield"`
} `xml:"record"`
} `xml:"metadata"`
} `xml:"record"`
} `xml:"ListRecords"`
}
Generate a package
If you want in include generated file in the build process, e.g. with go generate, you may find -P and -o helpful.
$ cat fixtures/b.xml
<a><b></b></a>
Run on the command line or via go generate:
$ zek -P mypkg -o data.go < fixtures/b.xml
This would write out the following in data.go file:
// Code generated by zek; DO NOT EDIT.
package mypkg
import "encoding/xml"
// A was generated 2021-09-16 11:23:06 by tir on trieste.
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
B string `xml:"b"`
}
Note that any existing file will be overwritten, without any warning.
Misc
As a side effect, zek seems to be a useful for debugging. Example:
This record is emitted from a typical OAI server (OJS, not even uncommon), yet one can quickly spot the flaw in the structure.
Over 30 different struct generated manually in the course of a few hours (around five minutes per source): https://git.io/vbTDo.
-- Current extent leader: 1532 lines struct
Author: Miku
Source Code: https://github.com/miku/zek
License: GPL-3.0 license
1591340335
How To Take Help Of Referencing Generator
APA Referencing Generator
Many students use APA style as the key citation style in their assignment in university or college. Although, many people find it quite difficult to write the reference of the source. You ought to miss the names and dates of authors. Hence, APA referencing generator is important for reducing the burden of students. They can now feel quite easy to do the assignments on time.
The functioning of APA referencing generator
If you are struggling hard to write the APA referencing then you can take the help of APA referencing generator. It will create an excellent list. You are required to enter the information about the source. Just ensure that the text is credible and original. If you will copy references then it is a copyright violation.
You can use a referencing generator in just a click. It will generate the right references for all the sources. You are required to organize in alphabetical order. The generator will make sure that you will get good grades.
How to use APA referencing generator?
Select what is required to be cited such as journal, book, film, and others. You can choose the type of required citations list and enter all the required fields. The fields are dates, author name, title, editor name, and editions, name of publishers, chapter number, page numbers, and title of journals. You can click for reference to be generated and you will get the desired result.
Chicago Referencing Generator
Do you require the citation style? You can rely on Chicago Referencing Generator and will ensure that you will get the right citation in just a click. The generator is created to provide solutions to students to cite their research paper in Chicago style. It has proved to be the quickest and best citation generator on the market. The generator helps to sort the homework issues in few seconds. It also saves a lot of time and energy.
This tool helps researchers, professional writers, and students to manage and generate text citation essays. It will help to write Chicago style in a fast and easy way. It also provides details and directions for formatting and cites resources.
So, you must stop wasting the time and can go for Chicago Referencing Generator or APA referencing generator. These citation generators will help to solve the problem of citation issues. You can easily create citations by using endnotes and footnotes.
So, you can generate bibliographies, references, in-text citations, and title pages. These are fully automatic referencing style. You are just required to enter certain details about the citation and you will get the citation in the proper and required format.
So, if you are feeling any problem in doing assignment then you can take the help of assignment help.
If you require help for Assignment then livewebtutors is the right place for you. If you see our prices, you will observe that they are actually very affordable. Also, you can always expect a discount. Our team is capable and versatile enough to offer you exactly what you need, the best services for the prices you can afford.
read more:- Are you struggling to write a bibliography? Use Harvard referencing generator
#apa referencing generator #harvard referencing generator #chicago referencing generator #mla referencing generator #deakin referencing generator #oxford referencing generator
1658977500
A Ruby Library for Generating Text with Recursive Template Grammars
Calyx
Calyx provides a simple API for generating text with declarative recursive grammars.
Install
Command Line
gem install calyx
Gemfile
gem 'calyx'
Examples
The best way to get started quickly is to install the gem and run the examples locally.
Any Gradient
Requires Roda and Rack to be available.
gem install roda
Demonstrates how to use Calyx to construct SVG graphics. Any Gradient generates a rectangle with a linear gradient of random colours.
Run as a web server and preview the output in a browser (http://localhost:9292):
ruby examples/any_gradient.rb
Or generate SVG files via a command line pipe:
ruby examples/any_gradient > gradient1.xml
Tiny Woodland Bot
Requires the Twitter client gem and API access configured for a specific Twitter handle.
gem install twitter
Demonstrates how to use Calyx to make a minimal Twitter bot that periodically posts unique tweets. See @tiny_woodland on Twitter and the writeup here.
TWITTER_CONSUMER_KEY=XXX-XXX
TWITTER_CONSUMER_SECRET=XXX-XXX
TWITTER_ACCESS_TOKEN=XXX-XXX
TWITTER_CONSUMER_SECRET=XXX-XXX
ruby examples/tiny_woodland_bot.rb
Faker
Faker is a popular library for generating fake names and associated sample data like internet addresses, company names and locations.
This example demonstrates how to use Calyx to reproduce the same functionality using custom lists defined in a YAML configuration file.
ruby examples/faker.rb
Usage
Require the library and inherit from Calyx::Grammar to construct a set of rules to generate a text.
require 'calyx'
class HelloWorld < Calyx::Grammar
start 'Hello world.'
end
To generate the text itself, initialize the object and call the generate method.
hello = HelloWorld.new
hello.generate
# > "Hello world."
Obviously, this hardcoded sentence isn’t very interesting by itself. Possible variations can be added to the text by adding additional rules which provide a named set of text strings. The rule delimiter syntax ({}) can be used to substitute the generated content of other rules.
class HelloWorld < Calyx::Grammar
start '{greeting} world.'
greeting 'Hello', 'Hi', 'Hey', 'Yo'
end
Each time #generate runs, it evaluates the tree and randomly selects variations of rules to construct a resulting string.
hello = HelloWorld.new
hello.generate
# > "Hi world."
hello.generate
# > "Hello world."
hello.generate
# > "Yo world."
By convention, the start rule specifies the default starting point for generating the final text. You can start from any other named rule by passing it explicitly to the generate method.
class HelloWorld < Calyx::Grammar
hello 'Hello world.'
end
hello = HelloWorld.new
hello.generate(:hello)
Block Constructors
As an alternative to subclassing, you can also construct rules unique to an instance by passing a block when initializing the class:
hello = Calyx::Grammar.new do
start '{greeting} world.'
greeting 'Hello', 'Hi', 'Hey', 'Yo'
end
hello.generate
Template Expressions
Basic rule substitution uses single curly brackets as delimiters for template expressions:
fruit = Calyx::Grammar.new do
start '{colour} {fruit}'
colour 'red', 'green', 'yellow'
fruit 'apple', 'pear', 'tomato'
end
6.times { fruit.generate }
# => "yellow pear"
# => "red apple"
# => "green tomato"
# => "red pear"
# => "yellow tomato"
# => "green apple"
Nesting and Substitution
Rules are recursive. They can be arbitrarily nested and connected to generate larger and more complex texts.
class HelloWorld < Calyx::Grammar
start '{greeting} {world_phrase}.'
greeting 'Hello', 'Hi', 'Hey', 'Yo'
world_phrase '{happy_adj} world', '{sad_adj} world', 'world'
happy_adj 'wonderful', 'amazing', 'bright', 'beautiful'
sad_adj 'cruel', 'miserable'
end
Nesting and hierarchy can be manipulated to balance consistency with novelty. The exact same word atoms can be combined in a variety of ways to produce strikingly different resulting texts.
module HelloWorld
class Sentiment < Calyx::Grammar
start '{happy_phrase}', '{sad_phrase}'
happy_phrase '{happy_greeting} {happy_adj} world.'
happy_greeting 'Hello', 'Hi', 'Hey', 'Yo'
happy_adj 'wonderful', 'amazing', 'bright', 'beautiful'
sad_phrase '{sad_greeting} {sad_adj} world.'
sad_greeting 'Goodbye', 'So long', 'Farewell'
sad_adj 'cruel', 'miserable'
end
class Mixed < Calyx::Grammar
start '{greeting} {adj} world.'
greeting 'Hello', 'Hi', 'Hey', 'Yo', 'Goodbye', 'So long', 'Farewell'
adj 'wonderful', 'amazing', 'bright', 'beautiful', 'cruel', 'miserable'
end
end
Random Sampling
By default, the outcomes of generated rules are selected with Ruby’s built-in pseudorandom number generator (as seen in methods like Kernel.rand and Array.sample). To seed the random number generator, pass in an integer seed value as the first argument to the constructor:
grammar = Calyx::Grammar.new(seed: 12345) do
# rules...
end
Alternatively, you can pass a preconfigured instance of Ruby’s stdlib Random class:
random = Random.new(12345)
grammar = Calyx::Grammar.new(rng: random) do
# rules...
end
When a random seed isn’t supplied, Time.new.to_i is used as the default seed, which makes each run of the generator relatively unique.
Weighted Choices
Choices can be weighted so that some rules have a greater probability of expanding than others.
Weights are defined by passing a hash instead of a list of rules where the keys are strings or symbols representing the grammar rules and the values are weights.
Weights can be represented as floats, integers or ranges.
- Floats must be in the interval 0..1 and the given weights for a production must sum to 1.
- Ranges must be contiguous and cover the entire interval from 1 to the maximum value of the largest range.
- Integers (Fixnums) will produce a distribution based on the sum of all given numbers, with each number being a fraction of that sum.
The following definitions produce an equivalent weighting of choices:
Calyx::Grammar.new do
start 'heads' => 1, 'tails' => 1
end
Calyx::Grammar.new do
start 'heads' => 0.5, 'tails' => 0.5
end
Calyx::Grammar.new do
start 'heads' => 1..5, 'tails' => 6..10
end
Calyx::Grammar.new do
start 'heads' => 50, 'tails' => 50
end
There’s a lot of interesting things you can do with this. For example, you can model the triangular distribution produced by rolling 2d6:
Calyx::Grammar.new do
start(
'2' => 1,
'3' => 2,
'4' => 3,
'5' => 4,
'6' => 5,
'7' => 6,
'8' => 5,
'9' => 4,
'10' => 3,
'11' => 2,
'12' => 1
)
end
Or reproduce Gary Gygax’s famous generation table from the original Dungeon Master’s Guide (page 171):
Calyx::Grammar.new do
start(
:empty => 0.6,
:monster => 0.1,
:monster_treasure => 0.15,
:special => 0.05,
:trick_trap => 0.05,
:treasure => 0.05
)
empty 'Empty'
monster 'Monster Only'
monster_treasure 'Monster and Treasure'
special 'Special'
trick_trap 'Trick/Trap.'
treasure 'Treasure'
end
String Modifiers
Dot-notation is supported in template expressions, allowing you to call any available method on the String object returned from a rule. Formatting methods can be chained arbitrarily and will execute in the same way as they would in native Ruby code.
greeting = Calyx::Grammar.new do
start '{hello.capitalize} there.', 'Why, {hello} there.'
hello 'hello', 'hi'
end
4.times { greeting.generate }
# => "Hello there."
# => "Hi there."
# => "Why, hello there."
# => "Why, hi there."
You can also extend the grammar with custom modifiers that provide useful formatting functions.
Filters
Filters accept an input string and return the transformed output:
greeting = Calyx::Grammar.new do
filter :shoutycaps do |input|
input.upcase
end
start '{hello.shoutycaps} there.', 'Why, {hello.shoutycaps} there.'
hello 'hello', 'hi'
end
4.times { greeting.generate }
# => "HELLO there."
# => "HI there."
# => "Why, HELLO there."
# => "Why, HI there."
Mappings
The mapping shortcut allows you to specify a map of regex patterns pointing to their resulting substitution strings:
green_bottle = Calyx::Grammar.new do
mapping :pluralize, /(.+)/ => '\\1s'
start 'One green {bottle}.', 'Two green {bottle.pluralize}.'
bottle 'bottle'
end
2.times { green_bottle.generate }
# => "One green bottle."
# => "Two green bottles."
Modifier Mixins
In order to use more intricate rewriting and formatting methods in a modifier chain, you can add methods to a module and embed it in a grammar using the modifier classmethod.
Modifier methods accept a single argument representing the input string from the previous step in the expression chain and must return a string, representing the modified output.
module FullStop
def full_stop(input)
input << '.'
end
end
hello = Calyx::Grammar.new do
modifier FullStop
start '{hello.capitalize.full_stop}'
hello 'hello'
end
hello.generate
# => "Hello."
To share custom modifiers across multiple grammars, you can include the module in Calyx::Modifiers. This will make the methods available to all subsequent instances:
module FullStop
def full_stop(input)
input << '.'
end
end
class Calyx::Modifiers
include FullStop
end
Monkeypatching String
Alternatively, you can combine methods from existing Gems that monkeypatch String:
require 'indefinite_article'
module FullStop
def full_stop
self << '.'
end
end
class String
include FullStop
end
noun_articles = Calyx::Grammar.new do
start '{fruit.with_indefinite_article.capitalize.full_stop}'
fruit 'apple', 'orange', 'banana', 'pear'
end
4.times { noun_articles.generate }
# => "An apple."
# => "An orange."
# => "A banana."
# => "A pear."
Memoized Rules
Rule expansions can be ‘memoized’ so that multiple references to the same rule return the same value. This is useful for picking a noun from a list and reusing it in multiple places within a text.
The @ sigil is used to mark memoized rules. This evaluates the rule and stores it in memory the first time it’s referenced. All subsequent references to the memoized rule use the same stored value.
# Without memoization
grammar = Calyx::Grammar.new do
start '{name} <{name.downcase}>'
name 'Daenerys', 'Tyrion', 'Jon'
end
3.times { grammar.generate }
# => Daenerys <jon>
# => Tyrion <daenerys>
# => Jon <tyrion>
# With memoization
grammar = Calyx::Grammar.new do
start '{@name} <{@name.downcase}>'
name 'Daenerys', 'Tyrion', 'Jon'
end
3.times { grammar.generate }
# => Tyrion <tyrion>
# => Daenerys <daenerys>
# => Jon <jon>
Note that the memoization symbol can only be used on the right hand side of a production rule.
Unique Rules
Rule expansions can be marked as ‘unique’, meaning that multiple references to the same rule always return a different value. This is useful for situations where the same result appearing twice would appear awkward and messy.
Unique rules are marked by the $ sigil.
grammar = Calyx::Grammar.new do
start "{$medal}, {$medal}, {$medal}"
medal 'Gold', 'Silver', 'Bronze'
end
grammar.generate
# => Silver, Bronze, Gold
Dynamically Constructing Rules
Template expansions can be dynamically constructed at runtime by passing a context map of rules to the #generate method:
class AppGreeting < Calyx::Grammar
start 'Hi {username}!', 'Welcome back {username}...', 'Hola {username}'
end
context = {
username: UserModel.username
}
greeting = AppGreeting.new
greeting.generate(context)
External File Formats
In addition to defining grammars in pure Ruby, you can load them from external JSON and YAML files:
hello = Calyx::Grammar.load('hello.yml')
hello.generate
The format requires a flat map with keys representing the left-hand side named symbols and the values representing the right hand side substitution rules.
In JSON:
{
"start": "{greeting} world.",
"greeting": ["Hello", "Hi", "Hey", "Yo"]
}
In YAML:
---
start: "{greeting} world."
greeting:
- Hello
- Hi
- Hey
- Yo
Accessing the Raw Generated Tree
Calling #evaluate on the grammar instance will give you access to the raw generated tree structure before it gets flattened into a string.
The tree is encoded as an array of nested arrays, with the leading symbols labeling the choices and rules selected, and the trailing terminal leaves encoding string values.
This may not make a lot of sense unless you’re familiar with the concept of s-expressions. It’s a fairly speculative feature at this stage, but it leads to some interesting possibilities.
grammar = Calyx::Grammar.new do
start 'Riddle me ree.'
end
grammar.evaluate
# => [:start, [:choice, [:concat, [[:atom, "Riddle me ree."]]]]]
Roadmap
Rough plan for stabilising the API and features for a 1.0 release.
| Version | Features planned |
|---|---|
0.6 | |
0.7 | |
0.8 | |
0.9 | |
0.10 | |
0.11 | |
0.12 | |
0.13 | |
0.14 | |
0.15 | |
0.16 | |
0.17 | |
Credits
Author & Maintainer
Contributors
Author: Maetl
Source Code: https://github.com/maetl/calyx
License: MIT license