JuliaParser.jl: A Rewrite Of Julia's Parser in Julia

JuliaParser

Note: This package is unmaintained and heavily bitrotted. It will not parse up to date Julia code correctly!

A pure Julia port of Julia's parser. It strives to be fully compatible with Julia's built-in parser.

Differences with Julia's built-in parser

BigInt and Int128 numbers are treated as literal values instead of expressions.
Literal negation is done as negated literals rather than using Expr(:-)
QuoteNodes are replaced with Expr(:quote).

Using JuliaParser as your primary parser

JuliaParser provides a script that will replace the built-in parser by itself. You may load it as follows:

julia -L ~/.julia/v0.5/JuliaParser/bin/repl.jl

TODO items

performance improvements
refactor number tokenization
refactor to make it more useful to use as a library (right now it is pretty monolithic)

Trying it out

julia> Pkg.clone("JuliaParser")

julia> import JuliaParser.Parser
julia> import JuliaParser.Lexer

julia> src = """
              function test(x::Int)
                  return x ^ 2
              end
              """
julia> ts = Lexer.TokenStream(src);

julia> Lexer.next_token(ts)
:function

julia> Lexer.next_token(ts)
:test

julia> Lexer.next_token(ts)
'('

julia> Lexer.next_token(ts)
:x

julia> Lexer.next_token(ts)
:(::)

julia> Lexer.next_token(ts)
:Int

julia> ast = Parser.parse(src);

julia> Meta.show_sexpr(ast)
(:function, (:call, :test, (:(::), :x, :Int)), (:block,
    (:line, 2, :none),
    (:return, (:call, :^, :x, 2))
  ))

julia> dump(ast)
Expr 
  head: Symbol function
  args: Array(Any,(2,))
    1: Expr 
      head: Symbol call
      args: Array(Any,(2,))
        1: Symbol test
        2: Expr 
          head: Symbol ::
          args: Array(Any,(2,))
          typ: Any
      typ: Any
    2: Expr 
      head: Symbol block
      args: Array(Any,(2,))
        1: Expr 
          head: Symbol line
          args: Array(Any,(2,))
          typ: Any
        2: Expr 
          head: Symbol return
          args: Array(Any,(1,))
          typ: Any
      typ: Any
  typ: Any

Author: JuliaLang
Source Code: https://github.com/JuliaLang/JuliaParser.jl
License: View license

#julia #nlp

JuliaParser.jl: A Rewrite Of Julia's Parser in Julia

JuliaParser

Note: This package is unmaintained and heavily bitrotted. It will not parse up to date Julia code correctly!

A pure Julia port of Julia's parser. It strives to be fully compatible with Julia's built-in parser.

Differences with Julia's built-in parser

BigInt and Int128 numbers are treated as literal values instead of expressions.
Literal negation is done as negated literals rather than using Expr(:-)
QuoteNodes are replaced with Expr(:quote).

Using JuliaParser as your primary parser

JuliaParser provides a script that will replace the built-in parser by itself. You may load it as follows:

julia -L ~/.julia/v0.5/JuliaParser/bin/repl.jl

TODO items

performance improvements
refactor number tokenization
refactor to make it more useful to use as a library (right now it is pretty monolithic)

Trying it out

julia> Pkg.clone("JuliaParser")

julia> import JuliaParser.Parser
julia> import JuliaParser.Lexer

julia> src = """
              function test(x::Int)
                  return x ^ 2
              end
              """
julia> ts = Lexer.TokenStream(src);

julia> Lexer.next_token(ts)
:function

julia> Lexer.next_token(ts)
:test

julia> Lexer.next_token(ts)
'('

julia> Lexer.next_token(ts)
:x

julia> Lexer.next_token(ts)
:(::)

julia> Lexer.next_token(ts)
:Int

julia> ast = Parser.parse(src);

julia> Meta.show_sexpr(ast)
(:function, (:call, :test, (:(::), :x, :Int)), (:block,
    (:line, 2, :none),
    (:return, (:call, :^, :x, 2))
  ))

julia> dump(ast)
Expr 
  head: Symbol function
  args: Array(Any,(2,))
    1: Expr 
      head: Symbol call
      args: Array(Any,(2,))
        1: Symbol test
        2: Expr 
          head: Symbol ::
          args: Array(Any,(2,))
          typ: Any
      typ: Any
    2: Expr 
      head: Symbol block
      args: Array(Any,(2,))
        1: Expr 
          head: Symbol line
          args: Array(Any,(2,))
          typ: Any
        2: Expr 
          head: Symbol return
          args: Array(Any,(1,))
          typ: Any
      typ: Any
  typ: Any

Author: JuliaLang
Source Code: https://github.com/JuliaLang/JuliaParser.jl
License: View license

#julia #nlp

Julia Programming

1661372820

OpenSMILES.jl: OpenSMILES parser in Julia

OpenSMILES.jl

This is a SMILES parser in Julia that aims to follow the OpenSMILES format (to the best of my ability). Theres probably bugs, this isn't inventive its just a parser that turns SMILES into a weighted Graphs.jl graph. Contributions welcome!

Notice

Although this package does mostly what it intends too (still missing chiral support, etc), an excellent package MolecularGraph has been released. It is highly reccommended users contribute to that package instead of this one, unless they specifically want LightGraphs.jl(or Graphs.jl) integration.

Examples

Tryptophan

using OpenSMILES, GraphPlot

# Tryptophan
Graph, Data = OpenSMILES.ParseSMILES("C1=CC=C2C(=C1)C(=CN2)CC(C(=O)O)N")
GraphPlot.gplot( Graph, nodelabel = OpenSMILES.abbreviation.( Data ) )
OpenSMILES.EmpiricalFormula( Data ) # C11H12N2O2

Tryptophan

# Bowtie ( not a real molecule :P )
Graph, Data = OpenSMILES.ParseSMILES("C1CC12CC2")
GraphPlot.gplot( Graph, nodelabel = OpenSMILES.abbreviation.( Data ) )
OpenSMILES.EmpiricalFormula( Data ) #C5H8

Bowtie

Cool? Enjoy!

Download Details:

Author: Caseykneale
Source Code: https://github.com/caseykneale/OpenSMILES.jl
License: MIT license

#julia #open

Monty Boehm

1658999040

TOML.jl: A TOML Parser for Julia

TOML.jl

A TOML v0.2.0 parser for Julia.

Usage:

julia> require("TOML")

julia> TOML.parse(readall("etc/example.toml"))
[
  "clients"=>[
    "data"=>[["gamma", "delta"], [1, 2]],
    "hosts"=>["alpha", "omega"]
  ],
  "database"=>[
    "enabled"=>true, "ports"=>[8001, 8001, 8002],
    "connection_max"=>5000, "server"=>"192.168.1.1"
  ],
  "title"=>"TOML Example",
  "servers"=>[
    "beta"=>["dc"=>"eqdc10","ip"=>"10.0.0.2"],
    "alpha"=>["dc"=>"eqdc10","ip"=>"10.0.0.1"]
  ],
  "owner"=>[
    "dob"=>TOML.DateTime(1979, 5, 27, 7, 32, 0),
    "organization"=>"GitHub",
    "name"=>"Tom Preston-Werner",
    "bio"=>"GitHub Cofounder & CEO\nLikes tater tots and beer."
  ]
]

The input must be convertible to UTF-8. Byte sequences that represent an invalid UTF-8 string will be rejected, per spec.

The TOML types are converted to their natural Julia counterparts (except datetimes, see below). Arrays are typed.

The parser is strict, and will throw a TOMLError on malformed input.

DateTime objects:

To keep the dependencies low (the Calendar package is very slow to load), and waiting for the implemetation of Timestamps in the Base Julia library, TOML DateTimes are currently converted to TOML.DateTime objects.

immutable DateTime
    year::Int
    month::Int
    date::Int
    hour::Int
    minute::Int
    second::Int
end

Author: pygy
Source Code: https://github.com/pygy/TOML.jl
License: View license

#julia #toml

Julia Programming

1661887800

OpenStreetMapParser.jl: Julia OpenStreetMap Parser

OpenStreetMapParser [Not recommended for use]

See OpenStreetMap.jl for now. But if you must:

This package provides basic functionality for parsing OpenStreetMap data in Julia, in the following file formats:

.osm (XML) # See http://wiki.openstreetmap.org/wiki/OSM_XML
.pbf (binary) # See http://wiki.openstreetmap.org/wiki/PBF_Format

For a complete introduction into the OSM project, the OSM API, and the OSM XML file format we refer to the project’s wiki available at http://wiki.openstreetmap.org/.

Installation

OSM Elements

The OpenStreetMap project provides data in the OSM XML format, which consists of three basic elements:

Node: The basic element. (defining points in space)
Way: An ordered interconnection of nodes (defining linear features and area boundaries)
Relation: A grouping of elements (nodes, ways, and relations), which are sometimes used to explain how other elements work together

The following functions are supported:

parseNodes() # document/examples
parseWays() # document/examples
parseRelations() # document/examples
parseMap() # document/examples

Each element has further attributes like the element ID (unique within the corresponding element group) and timestamp. Furthermore, each element may have an arbitrary number of tags (key-value pairs) which describe the element. Ways and relations, in addition, have references to their members’ IDs.

Remark: A distinction should be made between data elements (a data primitive used to represent semantic objects), and semantic elements (which represent the geometry of physical world objects). Data elements are an implementation detail, while semantic elements carry the desired meaning. This will be made clearer in the next section on OSM Features.

OSM Features

OpenStreetMap represents physical features on the ground (e.g., roads or buildings) using tags attached to its basic data structures (its nodes, ways, and relations). Each tag describes a geographic attribute of the feature being shown by that specific node, way or relation. The community agrees on certain key and value combinations for the most commonly used tags, which act as informal standards. For a comprehensive list of OSM features, we suggest visiting their wiki page here http://wiki.openstreetmap.org/wiki/Map_Features.

Scope of this Package

This package is meant for parsing of small/medium-sized (typically city-sized, <500MB) OSM files. If you're dealing with bigger files, you might want to scope it down into something smaller, or handle it through a database instead.

It will be possible with LibExpat, but not particularly profitable for us to selection/filtering of the OSM data within the parser itself. Given the size of the files we expect (c.f. #1), you can either filter/select them after the parsing, or roll out your own parser to perform the selection/filtering.

All coordinates are unprojected WGS84 (EPSG:4326). You can perform the necessary transformations through Geodesy.jl ~~or LibOGR.jl~~.

The availability of high-resolution aerial imagery has led to many features being recorded as areas (building or site outlines), not points, in OpenStreetMap. You will, for example, often find a restaurant or hotel drawn as an area. This might make processing difficult because you have to cater for both types of features even if you are not interested in areas. As the conversion from areas to points is not well-defined, we do not perform it automatically.

We will not be providing the following conveniences, but suggest packages that might help (in parentheses):

plotting/viewing of the map elements (Compose/Winston) # OpenStreetMapPlotter.jl
routing on the road network (LightGraphs/Graphs) # OpenStreetMapRouter.jl
map projections/transformations between different coordinate systems (Geodesy/OGR)
filtering/selection of data (DataFrames)
geometric operations (JuliaGeometry/LibGEOS)

We will, on the other hand, support Pull-Requests that updates the package to be in line with official/well-supported frameworks of OSM data.

References

Download Details:

Author: Yeesian
Source Code: https://github.com/yeesian/OpenStreetMapParser.jl
License: View license

#julia #map

Monty Boehm

1658950560

PEGParser.jl: PEG Parser for Julia

PEGParser

PEGParser is a PEG Parser for Julia with Packrat capabilties. PEGParser was inspired by pyparsing, parsimonious, boost::spirit, as well as several others.

Defining a grammar

To define a grammar you can write:

@grammar <name> begin
  rule1 = ...
  rule2 = ...
  ...
end

Allowed rules

The following rules can be used:

Terminals: Strings and characters
Or: a | b | c
And: a + b + c
Grouping: (a + b) | (c + d)
Optional: ?(a + b)
One or more: +((a + b) | (c + d))
Zero or more: *((a + b) | (c + d))
Look ahead: a > (b + c)
Regular expressions: r"[a-zA-Z]+"
Lists: list(rule, delim) or list(rule, delim, min=1)
Suppression: -rule
Semantic action: rule { expr }

For semantic actions, the expr may use the variables: node, value, first, last, and children. The value variable has a corresponding alias _0 and each element of children _i, where i is the index into children. See below for examples using this.

TODO

Multiple: (a+b)^(3, 5)

Example 1

Let's start by creating a simple calculator that can take two numbers and an operator to give a result.

We first define the grammar:

@grammar calc1 begin
  start = number + op + number
  op = plus | minus
  number = -space + r"[0-9]+"
  plus = -space + "+"
  minus = -space + "-"
  space = r"[ \t\n\r]*"
end

All grammars by default use start as the starting rule. You can specify a different starting rule in the parse function if you desire.

The starting rule is composed of two other rules: number and op. For this calculator, we only allow + and -. Note, that this could in fact be written more concisely with:

op = -space + r"[+-]"

The number rule just matches any digit between 0 to 9. You'll note that spaces appear in front of all terminals. This is because PEGs don't handle spaces automatically.

Now we can run this grammar with some input:

(ast, pos, error) = parse(calc1, "4+5")
println(ast)

will result in the following output:

node(start) {AndRule}
1: node(number) {AndRule}
  1: node(number.2) {'4',RegexRule}
2: node(plus) {AndRule}
  1: node(plus.2) {'+',Terminal}
3: node(number) {AndRule}
  1: node(number.2) {'5',RegexRule}

Our input is correctly parsed by our input, but we either have to traverse the tree to get out the result, or use change the output of the parse.

We can change the output of the parse with semantic actions. Every rule already has a semantic action attached to it. Normally it is set to either return a node in the tree or (for the or-rule) give the first child node.

For example, we can change the number rule to emit an actual number:

number = (-space + r"[0-9]+") { parseint(_1.value) }

The curly-braces after a rule allows either an expression or function to be used as the new action. In this case, the first child (the number, as the space is suppressed), as specified by _1, is parsed as an integer.

If we rewrite the grammar fully with actions defined for the rules, we end up with:

@grammar calc1 begin
  start = (number + op + number) {
    apply(eval(_2), _1, _3)
  }

  op = plus | minus
  number = (-space + r"[0-9]+") {parseint(_1.value)}
  plus = (-space + "+") {symbol(_1.value)}
  minus = (-space + "-") {symbol(_1.value)}
  space = r"[ \t\n\r]*"
end

data = "4+5"
(ast, pos, error) = parse(calc1, data)
println(ast)

We now get 9 as an answer. Thus, the parse is also doing the calculation. The code for this can be found in calc1.jl, with calc2.jl providing a more realistic (and useful) calculator.

Example 2

In calc3.jl, you can find a different approach to this problem. Instead of trying to calculate the answer immediately, the full syntax tree is created. This allows it to be transformed into different forms. In this example, we transform the tree into Julia code:

@grammar calc3 begin
  start = expr

  expr_op = term + op1 + expr
  expr = expr_op | term
  term_op = factor + op2 + term

  term = term_op | factor
  factor = number | pfactor
  pfactor = (lparen + expr + rparen) { _2 }
  op1 = add | sub
  op2 = mult | div

  number = (-space + float) { parsefloat(_1.value) } | (-space + integer) { parseint(_1.value) }
  add = (-space + "+") { symbol(_1.value) }
  sub = (-space + "-") { symbol(_1.value) }
  mult = (-space + "*") { symbol(_1.value) }
  div = (-space + "/") { symbol(_1.value) }

  lparen = (-space + "(") { _1 }
  rparen = (-space + ")") { _1 }
  space = r"[ \n\r\t]*"
end

You will also notice that instead of trying to define integer and float manually, we are now using pre-defined parsers. Custom parsers can be defined to both make defining new grammars easier as well as add new types of functionality (e.g. maintaining symbol tables).

The grammar is now ready to be used to parse strings:

(ast, pos, error) = parse(calc3, "3.145+5*(6-4.0)")

which results in the following AST:

node(start) {ReferencedRule}
  node(expr_op) {AndRule}
  1: 3.145 (Float64)
  2: + (Symbol)
  3: node(term_op) {AndRule}
    1: 5 (Int64)
    2: * (Symbol)
    3: node(expr_op) {AndRule}
      1: 6 (Int64)
      2: - (Symbol)
      3: 400.0 (Float64)

Now that we have an AST, we can create transforms to convert the AST into Julia code:

toexpr(node, cnodes, ::MatchRule{:default}) = cnodes
toexpr(node, cnodes, ::MatchRule{:term_op}) = Expr(:call, cnodes[2], cnodes[1], cnodes[3])
toexpr(node, cnodes, ::MatchRule{:expr_op}) = Expr(:call, cnodes[2], cnodes[1], cnodes[3])

and to use the transforms:

code = transform(toexpr, ast)

to generate the Expr:

Expr
  head: Symbol call
  args: Array(Any,(3,))
    1: Symbol +
    2: Float64 3.145
    3: Expr
      head: Symbol call
      args: Array(Any,(3,))
        1: Symbol *
        2: Int64 5
        3: Expr
        head: Symbol call
        args: Array(Any,(3,))
        typ: Any
      typ: Any
  typ: Any

Caveats

This is still very much a work in progress and doesn't yet have as much test coverage as I would like.

The error handling still needs a lot of work. Currently only a single error will be emitted, but the hope is to allow multiple errors to be returned.

Author: Abeschneider
Source Code: https://github.com/abeschneider/PEGParser.jl
License: View license

#julia #peg

JuliaParser.jl: A Rewrite Of Julia's Parser in Julia

JuliaParser

Differences with Julia's built-in parser

Using JuliaParser as your primary parser

TODO items

Trying it out

What is GEEK

Buddha Community

JuliaParser.jl: A Rewrite Of Julia's Parser in Julia

JuliaParser

Differences with Julia's built-in parser

Using JuliaParser as your primary parser

TODO items

Trying it out

OpenSMILES.jl: OpenSMILES parser in Julia

OpenSMILES.jl

Tryptophan

Download Details:

TOML.jl: A TOML Parser for Julia

TOML.jl

Usage:

DateTime objects:

OpenStreetMapParser.jl: Julia OpenStreetMap Parser

Installation

OSM Elements

OSM Features

Scope of this Package

References

Download Details:

PEGParser.jl: PEG Parser for Julia

PEGParser

Defining a grammar

Allowed rules

TODO

Example 1

Example 2

Caveats