1666818000
PtFEM.jl: The "Programming The Finite Element Method" toolkit
PtFEM
The "Programming the Finite Element Method" toolkit
This Julia package currently contains the programs in chapters 4, 5 and early sections of 6 as described in "Programming the Finite Element Method" by I M Smith, D V Griffiths and L. Margetts (PtFEM).
I use PtFEM when referring to the book and PtFEM.jl when referring to the Julia package. The authors and publisher have given permission to publish the Julia version of the PtFEM toolkit. Please refer to LICENSE for more details.
Documentation
PtFEM, the book, will always remain the primary documentation for this package. Additional programming documentation will be available through Julia's documenter package, e.g. in-line after installing the package:
use PtFEM
?StructuralElement
and full documentation can be found here.
Timeline
Please note that no hard timeline is set when this work in progress will be finished. TODO contains a list of next steps. VERSIONS holds the tagged version history.
Related work
Fundamental and great development work related to solving (partial) differential equations is done in several other Julia packages, e.g. ApproxFun.jl, DifferentialEquations.jl, JuliaFEM.jl and JuaFEM.jl to name a few.
Outside of Julia at least 2 other toolkits should be mentioned, i.e. deal.II and FEniCS.
Rerences
References will be kept here.
Participation and feedback
As always, feedback is welcome, please send me an email, file an issue on Github or generate a pull request (PR).
Rob J Goedman July 2018
Download Details:
Author: PtFEM
Source Code: https://github.com/PtFEM/PtFEM.jl
License: View license
1666526160
Graft.jl: Graph Toolkit for Julia
Graft.jl
A graph toolkit for Julia.
Graft stores vertex and edge metadata in separate dataframes. Adjacencies are stored in a sparsematrix, which also indexes into the edge dataframe. Vertex labels are supported for all external queries, using a bidirectional map. Vertex labels may be of any Julia type.
Data manipulation and analysis in Graft is accomplished with a pipelined query macro system adopted from Jplyr. User queries are parsed recursively, to build a DAG. The DAG is then executed from the bottom up. Results from the execution of intermediate nodes or table data-retrievals are cached to avoid redundant computations.
Installation
julia> Pkg.update()
julia> Pkg.add("Graft")
Examples
Acknowledgements
This project is supported by Google Summer of Code and mentored by Viral Shah and Shashi Gowda.
Download Details:
Author: Pranavtbhat
Source Code: https://github.com/pranavtbhat/Graft.jl
License: View license
1665369120
CellularAutomata.jl: Cellular Automata Simulation toolkit for Julia
Cellular Automata
A cellular automaton is a collection of "colored" cells on a grid of specified shape that evolves through a number of discrete time steps according to a set of rules based on the states of neighboring cells. The rules are then applied iteratively for as many time steps as desired.
mathworld.wolfram.com/CellularAutomaton
Elementary CA
To generate an elementary cellular automaton, use
ca = CellularAutomaton(rule, init, gen)
where rule is the Wolfram code (integer), init is a vector containing the initial starting condition and gen is the number of generations to be computed. For a single starting cell in the middle just omit the init vector.
To generate 15 generations of elementary cellular automaton of rule 90 use
using CellularAutomata
ca90 = CellularAutomaton(90, 16)
#
# #
# #
# # # #
# #
# # # #
# # # #
# # # # # # # #
# #
# # # #
# # # #
# # # # # # # #
# # # #
# # # # # # # #
# # # # # # # #
# # # # # # # # # # # # # # # #
Totalistic CA
For a more complex cellular automaton you can change the number of states k the cell can be and the radius r of neighbors that can influence the states. If k is changed to be larger than 2, a totalistic CA is computed where only the average value of all neighbors count. This can be done like this
ca = CellularAutomaton(993, 15, k=3)
X
XXX
X# #X
X X
XXX XXX
X# #X X# #X
X # X
XXX ### XXX
X# #X # X # X# #X
X # X # X
XXX ## X ## XXX
X# #X # X # X# #X
X X### XXX ###X X
XXX X XX # # XX X XXX
X# #X XX###X## ##X###XX X# #X
2 dimensional CAs
Two dimensional cellular automaton (like Conway's Game of Life) can be created by
ca = CA2d(B, S, init, gen)
where B and S are vectors that have the numbers of neighboring cells that define when cell is born or survives, init (matrix) is the initial starting condition and gen is the number of generations the CA is to be computed.
Game of life is then run for 9 generations for e.g. a turbine pattern by typing
ca = CA2d([3], [2, 3], init, 9)
1st step
###### ##
###### ##
##
## ##
## ##
## ##
##
## ######
## ######
2nd
####
# # ##
# # #
## #
## # #
# # # #
# # ##
# ##
# # #
## # #
####
3rd
##
####
# ## ## #
## #
## ## ###
#### # ###
# # # #
### # ####
### ## ##
# ##
# ## ## #
####
##
4th
# #
#
##
# ## #
# # #
# # ###
# #
### # #
# # #
# ## #
##
#
# #
5th
##
#
### ##
### # #
# # ##
# #
## # #
# # ###
## ###
#
##
6th
##
#
# # ##
# # ### #
# ######
## ##
###### #
# ### # #
## # #
#
##
7th
# # #
## # ###
# #
## #
# ##
# #
### # ##
# # #
8th
## ## #
## ## ##
#
##
## ##
##
#
## ## ##
# ## ##
9th
###### ##
###### ##
##
## ##
## ##
## ##
##
## ######
## ######
Running Tests
To run tests, execute the following command from the root folder of the repository:
julia tests/run_tests.jlDownload Details:
Author: Natj
Source Code: https://github.com/natj/CellularAutomata.jl
License: MIT license
1665025260
Testify: Thou Shalt Write Tests
Testify - Thou Shalt Write Tests
ℹ️ We are working on testify v2 and would love to hear what you'd like to see in it, have your say here: https://cutt.ly/testify
Go code (golang) set of packages that provide many tools for testifying that your code will behave as you intend.
Features include:
Get started:
- Install testify with one line of code, or update it with another
- For an introduction to writing test code in Go, see http://golang.org/doc/code.html#Testing
- Check out the API Documentation http://godoc.org/github.com/stretchr/testify
- To make your testing life easier, check out our other project, gorc
- A little about Test-Driven Development (TDD)
assert package
The assert package provides some helpful methods that allow you to write better test code in Go.
- Prints friendly, easy to read failure descriptions
- Allows for very readable code
- Optionally annotate each assertion with a message
See it in action:
package yours
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestSomething(t *testing.T) {
// assert equality
assert.Equal(t, 123, 123, "they should be equal")
// assert inequality
assert.NotEqual(t, 123, 456, "they should not be equal")
// assert for nil (good for errors)
assert.Nil(t, object)
// assert for not nil (good when you expect something)
if assert.NotNil(t, object) {
// now we know that object isn't nil, we are safe to make
// further assertions without causing any errors
assert.Equal(t, "Something", object.Value)
}
}
- Every assert func takes the
testing.Tobject as the first argument. This is how it writes the errors out through the normalgo testcapabilities. - Every assert func returns a bool indicating whether the assertion was successful or not, this is useful for if you want to go on making further assertions under certain conditions.
if you assert many times, use the below:
package yours
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestSomething(t *testing.T) {
assert := assert.New(t)
// assert equality
assert.Equal(123, 123, "they should be equal")
// assert inequality
assert.NotEqual(123, 456, "they should not be equal")
// assert for nil (good for errors)
assert.Nil(object)
// assert for not nil (good when you expect something)
if assert.NotNil(object) {
// now we know that object isn't nil, we are safe to make
// further assertions without causing any errors
assert.Equal("Something", object.Value)
}
}
require package
The require package provides same global functions as the assert package, but instead of returning a boolean result they terminate current test.
See t.FailNow for details.
mock package
The mock package provides a mechanism for easily writing mock objects that can be used in place of real objects when writing test code.
An example test function that tests a piece of code that relies on an external object testObj, can setup expectations (testify) and assert that they indeed happened:
package yours
import (
"testing"
"github.com/stretchr/testify/mock"
)
/*
Test objects
*/
// MyMockedObject is a mocked object that implements an interface
// that describes an object that the code I am testing relies on.
type MyMockedObject struct{
mock.Mock
}
// DoSomething is a method on MyMockedObject that implements some interface
// and just records the activity, and returns what the Mock object tells it to.
//
// In the real object, this method would do something useful, but since this
// is a mocked object - we're just going to stub it out.
//
// NOTE: This method is not being tested here, code that uses this object is.
func (m *MyMockedObject) DoSomething(number int) (bool, error) {
args := m.Called(number)
return args.Bool(0), args.Error(1)
}
/*
Actual test functions
*/
// TestSomething is an example of how to use our test object to
// make assertions about some target code we are testing.
func TestSomething(t *testing.T) {
// create an instance of our test object
testObj := new(MyMockedObject)
// setup expectations
testObj.On("DoSomething", 123).Return(true, nil)
// call the code we are testing
targetFuncThatDoesSomethingWithObj(testObj)
// assert that the expectations were met
testObj.AssertExpectations(t)
}
// TestSomethingWithPlaceholder is a second example of how to use our test object to
// make assertions about some target code we are testing.
// This time using a placeholder. Placeholders might be used when the
// data being passed in is normally dynamically generated and cannot be
// predicted beforehand (eg. containing hashes that are time sensitive)
func TestSomethingWithPlaceholder(t *testing.T) {
// create an instance of our test object
testObj := new(MyMockedObject)
// setup expectations with a placeholder in the argument list
testObj.On("DoSomething", mock.Anything).Return(true, nil)
// call the code we are testing
targetFuncThatDoesSomethingWithObj(testObj)
// assert that the expectations were met
testObj.AssertExpectations(t)
}
// TestSomethingElse2 is a third example that shows how you can use
// the Unset method to cleanup handlers and then add new ones.
func TestSomethingElse2(t *testing.T) {
// create an instance of our test object
testObj := new(MyMockedObject)
// setup expectations with a placeholder in the argument list
mockCall := testObj.On("DoSomething", mock.Anything).Return(true, nil)
// call the code we are testing
targetFuncThatDoesSomethingWithObj(testObj)
// assert that the expectations were met
testObj.AssertExpectations(t)
// remove the handler now so we can add another one that takes precedence
mockCall.Unset()
// return false now instead of true
testObj.On("DoSomething", mock.Anything).Return(false, nil)
testObj.AssertExpectations(t)
}
For more information on how to write mock code, check out the API documentation for the mock package.
You can use the mockery tool to autogenerate the mock code against an interface as well, making using mocks much quicker.
suite package
The suite package provides functionality that you might be used to from more common object oriented languages. With it, you can build a testing suite as a struct, build setup/teardown methods and testing methods on your struct, and run them with 'go test' as per normal.
An example suite is shown below:
// Basic imports
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
)
// Define the suite, and absorb the built-in basic suite
// functionality from testify - including a T() method which
// returns the current testing context
type ExampleTestSuite struct {
suite.Suite
VariableThatShouldStartAtFive int
}
// Make sure that VariableThatShouldStartAtFive is set to five
// before each test
func (suite *ExampleTestSuite) SetupTest() {
suite.VariableThatShouldStartAtFive = 5
}
// All methods that begin with "Test" are run as tests within a
// suite.
func (suite *ExampleTestSuite) TestExample() {
assert.Equal(suite.T(), 5, suite.VariableThatShouldStartAtFive)
}
// In order for 'go test' to run this suite, we need to create
// a normal test function and pass our suite to suite.Run
func TestExampleTestSuite(t *testing.T) {
suite.Run(t, new(ExampleTestSuite))
}
For a more complete example, using all of the functionality provided by the suite package, look at our example testing suite
For more information on writing suites, check out the API documentation for the suite package.
Suite object has assertion methods:
// Basic imports
import (
"testing"
"github.com/stretchr/testify/suite"
)
// Define the suite, and absorb the built-in basic suite
// functionality from testify - including assertion methods.
type ExampleTestSuite struct {
suite.Suite
VariableThatShouldStartAtFive int
}
// Make sure that VariableThatShouldStartAtFive is set to five
// before each test
func (suite *ExampleTestSuite) SetupTest() {
suite.VariableThatShouldStartAtFive = 5
}
// All methods that begin with "Test" are run as tests within a
// suite.
func (suite *ExampleTestSuite) TestExample() {
suite.Equal(suite.VariableThatShouldStartAtFive, 5)
}
// In order for 'go test' to run this suite, we need to create
// a normal test function and pass our suite to suite.Run
func TestExampleTestSuite(t *testing.T) {
suite.Run(t, new(ExampleTestSuite))
}
Installation
To install Testify, use go get:
go get github.com/stretchr/testify
This will then make the following packages available to you:
github.com/stretchr/testify/assert
github.com/stretchr/testify/require
github.com/stretchr/testify/mock
github.com/stretchr/testify/suite
github.com/stretchr/testify/http (deprecated)
Import the testify/assert package into your code using this template:
package yours
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestSomething(t *testing.T) {
assert.True(t, true, "True is true!")
}
Staying up to date
To update Testify to the latest version, use go get -u github.com/stretchr/testify.
Supported go versions
We currently support the most recent major Go versions from 1.13 onward.
Contributing
Please feel free to submit issues, fork the repository and send pull requests!
When submitting an issue, we ask that you please include a complete test function that demonstrates the issue. Extra credit for those using Testify to write the test code that demonstrates it.
Code generation is used. Look for CODE GENERATED AUTOMATICALLY at the top of some files. Run go generate ./... to update generated files.
We also chat on the Gophers Slack group in the #testify and #testify-dev channels.
Download Details:
Author: Stretchr
Source Code: https://github.com/stretchr/testify
License: MIT license
1665019980
NTK: Node.js Desktop UI toolkit
ntk
node.js desktop UI toolkit
A set of wrappers around low level node-x11 module to simplify X Window UI progamming - windows creation, event handling, 2d/3d graphics etc.
Installation
npm install ntk
Basic usage
require('ntk').createClient( (err, app) => {
var mainWnd = app.createWindow({ width: 500, height: 300, title: 'Hello' });
mainWnd.on('mousedown', (ev) => { mainWnd.setTitle('click: ' + [ev.x, ev.y].join(',')); });
mainWnd.map();
});
2d graphics
Each window ( or pixmap ) can be used to create 2d and 3d canvas. 2d canvas implements HTML context2d api ( some features not yet supported ) via XRender extension, and most operations are performed on the X Server side ( image composition, scaling, blur, text composition, gradients etc ). Initial font rendering is performed by ntk using freetype2 module and fonts names mapped to file names using font-manager module.
var ntk = require('ntk');
ntk.createClient( (err, app) => {
var wnd = app.createWindow({ width: 800, height: 600});
var ctx = wnd.getContext('2d');
wnd.on('mousemove', (ev) => {
var gradient = ctx.createRadialGradient(0, 0, 10, ev.x, ev.y, 500);
gradient.addColorStop(0, "red");
gradient.addColorStop(0.5, "green");
gradient.addColorStop(1, "rgba(255, 255, 255, 0)");
ctx.fillStyle = gradient;
ctx.fillRect(0, 0, ctx.width, ctx.height);
});
wnd.map();
});
ctx.drawImage() also accepts node-canvas as a source, for images with lot's of drawing calls it might be more efficient to perform all drawing locally and transfer pixels to server when ready:
var ntk = require('ntk');
var Canvas = require('canvas');
var canvas = new Canvas(800, 800);
var canvasCtx = canvas.getContext('2d');
function drawSomething(ctx) {
// ...
}
ntk.createClient( (err, app) => {
var wnd = app.createWindow({ width: 800, height: 800});
var ctx = wnd.getContext('2d');
setInterval(function() {
drawSomething(canvasCtx);
ctx.drawImage(canvas);
}, 20);
wnd.map();
});
3d graphics
At the moment only indirect GLX is supported with most of OpenGL 1.4 api implemented. Note that on some systems indirect GLX might be disabled by default, you'll need to enable it for gl to work.
var ntk = require('ntk');
var width = 300;
var height = 300;
var angle = 0;
function resize(gl) {
gl.Viewport(0, 0, width, height);
gl.MatrixMode(gl.PROJECTION);
gl.LoadIdentity();
if (width < height)
gl.Frustum(-1.0, 1.0, -height/width, height/width, -35.0, 20);
else
gl.Frustum(-width/height, width/height, -1.0, 1.0, -35.0, 20);
}
function draw(gl) {
gl.MatrixMode(gl.MODELVIEW);
gl.ClearColor(0.3,0.3,0.3,0.0);
gl.Clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
gl.LoadIdentity();
gl.Rotatef(-90 + 360*angle/width, 0, 0, 1);
gl.Begin(gl.TRIANGLE_STRIP);
gl.TexCoord2f(0.0,0.0);
gl.Vertex3f(-1, -1, 0);
gl.TexCoord2f(1.0, 0.0);
gl.Vertex3f(1, -1, 0);
gl.TexCoord2f(0.0, 1.0);
gl.Vertex3f(-1, 1, 0);
gl.TexCoord2f(1.0,1.0);
gl.Vertex3f(1, 1, 0);
gl.End();
gl.SwapBuffers();
}
var texture = require('baboon-image');
ntk.createClient(function(err, app) {
var wnd = app.createWindow({width: width, height: height});
wnd.map();
var gl = wnd.getContext('opengl');
gl.Enable(gl.TEXTURE_2D);
gl.GenTextures(1, function(err, textures) {
gl.BindTexture(gl.TEXTURE_2D, textures[0]);
gl.TexParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.TexParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.TexImage2D(gl.TEXTURE_2D, 0, gl.RGB, 512, 512, 0, gl.RGB, gl.UNSIGNED_BYTE, texture.data);
draw(gl);
wnd.on('resize', function(ev) {
width = ev.width;
height = ev.height;
resize(gl);
draw(gl);
}).on('mousemove', function(ev) {
angle = ev.x;
draw(gl);
});
});
});
High level widgets / layout management etc
Likely to be implemented outside as part of react-x11
Download Details:
Author: Sidorares
Source Code: https://github.com/sidorares/ntk
1662005580
jscodeshift: A JavaScript Codemod toolkit
jscodeshift
jscodeshift is a toolkit for running codemods over multiple JavaScript or TypeScript files. It provides:
- A runner, which executes the provided transform for each file passed to it. It also outputs a summary of how many files have (not) been transformed.
- A wrapper around recast, providing a different API. Recast is an AST-to-AST transform tool and also tries to preserve the style of original code as much as possible.
Install
Get jscodeshift from npm:
$ npm install -g jscodeshift
This will install the runner as jscodeshift.
VSCode Debugger
Configure VSCode to debug codemods
Usage (CLI)
The CLI provides the following options:
$ jscodeshift --help
Usage: jscodeshift [OPTION]... PATH...
or: jscodeshift [OPTION]... -t TRANSFORM_PATH PATH...
or: jscodeshift [OPTION]... -t URL PATH...
or: jscodeshift [OPTION]... --stdin < file_list.txt
Apply transform logic in TRANSFORM_PATH (recursively) to every PATH.
If --stdin is set, each line of the standard input is used as a path.
Options:
"..." behind an option means that it can be supplied multiple times.
All options are also passed to the transformer, which means you can supply custom options that are not listed here.
--(no-)babel apply babeljs to the transform file
(default: true)
-c, --cpus=N start at most N child processes to process source files
(default: max(all - 1, 1))
-d, --(no-)dry dry run (no changes are made to files)
(default: false)
--extensions=EXT transform files with these file extensions (comma separated list)
(default: js)
-h, --help print this help and exit
--ignore-config=FILE ... ignore files if they match patterns sourced from a configuration file (e.g. a .gitignore)
--ignore-pattern=GLOB ... ignore files that match a provided glob expression
--parser=babel|babylon|flow|ts|tsx the parser to use for parsing the source files
(default: babel)
--parser-config=FILE path to a JSON file containing a custom parser configuration for flow or babylon
-p, --(no-)print print transformed files to stdout, useful for development
(default: false)
--(no-)run-in-band run serially in the current process
(default: false)
-s, --(no-)silent do not write to stdout or stderr
(default: false)
--(no-)stdin read file/directory list from stdin
(default: false)
-t, --transform=FILE path to the transform file. Can be either a local path or url
(default: ./transform.js)
-v, --verbose=0|1|2 show more information about the transform process
(default: 0)
--version print version and exit
--fail-on-error return a 1 exit code when errors were found during execution of codemods
This passes the source of all passed through the transform module specified with -t or --transform (defaults to transform.js in the current directory). The next section explains the structure of the transform module.
Usage (JS)
const {run: jscodeshift} = require('jscodeshift/src/Runner')
const transformPath = 'transform.js'
const paths = ['foo.js', 'bar']
const options = {
dry: true,
print: true,
verbose: 1,
// ...
}
const res = await jscodeshift(transformPath, paths, options)
console.log(res)
/*
{
stats: {},
timeElapsed: '0.001',
error: 0,
ok: 0,
nochange: 0,
skip: 0
}
*/
Transform module
The transform is simply a module that exports a function of the form:
module.exports = function(fileInfo, api, options) {
// transform `fileInfo.source` here
// ...
// return changed source
return source;
};
As of v0.6.1, this module can also be written in TypeScript.
Arguments
fileInfo
Holds information about the currently processed file.
| Property | Description |
|---|---|
| path | File path |
| source | File content |
api
This object exposes the jscodeshift library and helper functions from the runner.
| Property | Description |
|---|---|
| jscodeshift | A reference to the jscodeshift library |
| stats | A function to collect statistics during --dry runs |
| report | Prints the passed string to stdout |
jscodeshift is a reference to the wrapper around recast and provides a jQuery-like API to navigate and transform the AST. Here is a quick example, a more detailed description can be found below.
/**
* This replaces every occurrence of variable "foo".
*/
module.exports = function(fileInfo, api, options) {
return api.jscodeshift(fileInfo.source)
.findVariableDeclarators('foo')
.renameTo('bar')
.toSource();
}
Note: This API is exposed for convenience, but you don't have to use it. You can use any tool to modify the source.
stats is a function that only works when the --dry options is set. It accepts a string, and will simply count how often it was called with that value.
At the end, the CLI will report those values. This can be useful while developing the transform, e.g. to find out how often a certain construct appears in the source(s).
report allows you do print arbitrary strings to stdout. This can be useful when other tools consume the output of jscodeshift. The reason to not directly use process.stdout in transform code is to avoid mangled output when many files are processed.
options
Contains all options that have been passed to runner. This allows you to pass additional options to the transform. For example, if the CLI is called with
$ jscodeshift -t myTransforms fileA fileB --foo=bar
options would contain {foo: 'bar'}.
Return value
The return value of the function determines the status of the transformation:
- If a string is returned and it is different from passed source, the transform is considered to be successful.
- If a string is returned but it's the same as the source, the transform is considered to be unsuccessful.
- If nothing is returned, the file is not supposed to be transformed (which is ok).
The CLI provides a summary of the transformation at the end. You can get more detailed information by setting the -v option to 1 or 2.
You can collect even more stats via the stats function as explained above.
Parser
The transform file can let jscodeshift know with which parser to parse the source files (and features like templates).
To do that, the transform module can export parser, which can either be one of the strings "babel", "babylon", "flow", "ts", or "tsx", or it can be a parser object that is compatible with recast and follows the estree spec.
Example: specifying parser type string in the transform file
module.exports = function transformer(file, api, options) {
const j = api.jscodeshift;
const rootSource = j(file.source);
// whatever other code...
return rootSource.toSource();
}
// use the flow parser
module.exports.parser = 'flow';
Example: specifying a custom parser object in the transform file
module.exports = function transformer(file, api, options) {
const j = api.jscodeshift;
const rootSource = j(file.source);
// whatever other code...
return rootSource.toSource();
}
module.exports.parser = {
parse: function(source) {
// return estree compatible AST
},
};
Example output
$ jscodeshift -t myTransform.js src
Processing 10 files...
Spawning 2 workers with 5 files each...
All workers done.
Results: 0 errors 2 unmodified 3 skipped 5 ok
The jscodeshift API
As already mentioned, jscodeshift also provides a wrapper around recast. In order to properly use the jscodeshift API, one has to understand the basic building blocks of recast (and ASTs) as well.
Core Concepts
AST nodes
An AST node is a plain JavaScript object with a specific set of fields, in accordance with the Mozilla Parser API. The primary way to identify nodes is via their type.
For example, string literals are represented via Literal nodes, which have the structure
// "foo"
{
type: 'Literal',
value: 'foo',
raw: '"foo"'
}
It's OK to not know the structure of every AST node type. The (esprima) AST explorer is an online tool to inspect the AST for a given piece of JS code.
Path objects
Recast itself relies heavily on ast-types which defines methods to traverse the AST, access node fields and build new nodes. ast-types wraps every AST node into a path object. Paths contain meta-information and helper methods to process AST nodes.
For example, the child-parent relationship between two nodes is not explicitly defined. Given a plain AST node, it is not possible to traverse the tree up. Given a path object however, the parent can be traversed to via path.parent.
For more information about the path object API, please have a look at ast-types.
Builders
To make creating AST nodes a bit simpler and "safer", ast-types defines a couple of builder methods, which are also exposed on jscodeshift.
For example, the following creates an AST equivalent to foo(bar):
// inside a module transform
var j = jscodeshift;
// foo(bar);
var ast = j.callExpression(
j.identifier('foo'),
[j.identifier('bar')]
);
The signature of each builder function is best learned by having a look at the definition files.
Collections and Traversal
In order to transform the AST, you have to traverse it and find the nodes that need to be changed. jscodeshift is built around the idea of collections of paths and thus provides a different way of processing an AST than recast or ast-types.
A collection has methods to process the nodes inside a collection, often resulting in a new collection. This results in a fluent interface, which can make the transform more readable.
Collections are "typed" which means that the type of a collection is the "lowest" type all AST nodes in the collection have in common. That means you cannot call a method for a FunctionExpression collection on an Identifier collection.
Here is an example of how one would find/traverse all Identifier nodes with jscodeshift and with recast:
// recast
var ast = recast.parse(src);
recast.visit(ast, {
visitIdentifier: function(path) {
// do something with path
return false;
}
});
// jscodeshift
jscodeshift(src)
.find(jscodeshift.Identifier)
.forEach(function(path) {
// do something with path
});
To learn about the provided methods, have a look at the Collection.js and its extensions.
Extensibility
jscodeshift provides an API to extend collections. By moving common operators into helper functions (which can be stored separately in other modules), a transform can be made more readable.
There are two types of extensions: generic extensions and type-specific extensions. Generic extensions are applicable to all collections. As such, they typically don't access specific node data, but rather traverse the AST from the nodes in the collection. Type-specific extensions work only on specific node types and are not callable on differently typed collections.
Examples
// Adding a method to all Identifiers
jscodeshift.registerMethods({
logNames: function() {
return this.forEach(function(path) {
console.log(path.node.name);
});
}
}, jscodeshift.Identifier);
// Adding a method to all collections
jscodeshift.registerMethods({
findIdentifiers: function() {
return this.find(jscodeshift.Identifier);
}
});
jscodeshift(ast).findIdentifiers().logNames();
jscodeshift(ast).logNames(); // error, unless `ast` only consists of Identifier nodes
Passing options to recast
You may want to change some of the output settings (like setting ' instead of "). This can be done by passing config options to recast.
.toSource({quote: 'single'}); // sets strings to use single quotes in transformed code.
You can also pass options to recast's parse method by passing an object to jscodeshift as second argument:
jscodeshift(source, {...})
More on config options here
Unit Testing
jscodeshift comes with a simple utility to allow easy unit testing with Jest, without having to write a lot of boilerplate code. This utility makes some assumptions in order to reduce the amount of configuration required:
- The test is located in a subdirectory under the directory the transform itself is located in (eg.
__tests__) - Test fixtures are located in a
__testfixtures__directory
This results in a directory structure like this:
/MyTransform.js
/__tests__/MyTransform-test.js
/__testfixtures__/MyTransform.input.js
/__testfixtures__/MyTransform.output.js
A simple example of unit tests is bundled in the sample directory.
The testUtils module exposes a number of useful helpers for unit testing.
defineTest
Defines a Jest/Jasmine test for a jscodeshift transform which depends on fixtures
jest.autoMockOff();
const defineTest = require('jscodeshift/dist/testUtils').defineTest;
defineTest(__dirname, 'MyTransform');
An alternate fixture filename can be provided as the fourth argument to defineTest. This also means that multiple test fixtures can be provided:
defineTest(__dirname, 'MyTransform', null, 'FirstFixture');
defineTest(__dirname, 'MyTransform', null, 'SecondFixture');
This will run two tests:
__testfixtures__/FirstFixture.input.js__testfixtures__/SecondFixture.input.js
defineInlineTest
Defines a Jest/Jasmine test suite for a jscodeshift transform which accepts inline values
This is a more flexible alternative to defineTest, as this allows to also provide options to your transform
const defineInlineTest = require('jscodeshift/dist/testUtils').defineInlineTest;
const transform = require('../myTransform');
const transformOptions = {};
defineInlineTest(transform, transformOptions, 'input', 'expected output', 'test name (optional)');
defineSnapshotTest
Similar to defineInlineTest but instead of requiring an output value, it uses Jest's toMatchSnapshot()
const defineSnapshotTest = require('jscodeshift/dist/testUtils').defineSnapshotTest;
const transform = require('../myTransform');
const transformOptions = {};
defineSnapshotTest(transform, transformOptions, 'input', 'test name (optional)');
For more information on snapshots, check out Jest's docs
defineSnapshotTestFromFixture
Similar to defineSnapshotTest but will load the file using same file-directory defaults as defineTest
const defineSnapshotTestDefault = require('jscodeshift/dist/testUtils').defineSnapshotTestDefault;
const transform = require('../myTransform');
const transformOptions = {};
defineSnapshotTestFromFixture(__dirname, transform, transformOptions, 'FirstFixture', 'test name (optional)');
applyTransform
Executes your transform using the options and the input given and returns the result. This function is used internally by the other helpers, but it can prove useful in other cases.
const applyTransform = require('jscodeshift/dist/testUtils').applyTransform;
const transform = require('../myTransform');
const transformOptions = {};
const output = applyTransform(transform, transformOptions, 'input');
ES modules
If you're authoring your transforms and tests using ES modules, make sure to import the transform's parser (if specified) in your tests:
// MyTransform.js
export const parser = 'flow'
export default function MyTransform(fileInfo, api, options) {
// ...
}
// __tests__/MyTransform-test.js
import { defineInlineTest } from 'jscodeshift/dist/testUtils
import * as transform from '../MyTransform
console.log(transform.parser) // 'flow'
defineInlineTest(transform, /* ... */)
Example Codemods
- react-codemod - React codemod scripts to update React APIs.
- js-codemod - Codemod scripts to transform code to next generation JS.
- js-transforms - Some documented codemod experiments to help you learn.
- fix-js - Codemods to fix some ESLint issues
Local Documentation Server
To update docs in /docs, use npm run docs.
To view these docs locally, use npx http-server ./docs
VSCode Debugging
It's recommended that you set up your codemod project to all debugging via the VSCode IDE. When you open your project in VSCode, add the following configuration to your launch.json debugging configuration.
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "pwa-node",
"request": "launch",
"name": "Debug Transform",
"skipFiles": [
"<node_internals>/**"
],
"program": "${workspaceRoot}/node_modules/.bin/jscodeshift",
"stopOnEntry": false,
"args": ["--dry", "--print", "-t", "${input:transformFile}", "--parser", "${input:parser}", "--run-in-band", "${file}"],
"preLaunchTask": null,
"runtimeExecutable": null,
"runtimeArgs": [
"--nolazy"
],
"console": "internalConsole",
"sourceMaps": true,
"outFiles": []
},
{
"name": "Debug All JSCodeshift Jest Tests",
"type": "node",
"request": "launch",
"runtimeArgs": [
"--inspect-brk",
"${workspaceRoot}/node_modules/jest/bin/jest.js",
"--runInBand",
"--testPathPattern=${fileBasenameNoExtension}"
],
"console": "integratedTerminal",
"internalConsoleOptions": "neverOpen",
"port": 9229
}
],
"inputs": [
{
"type": "pickString",
"id": "parser",
"description": "jscodeshift parser",
"options": [
"babel",
"babylon",
"flow",
"ts",
"tsx",
],
"default": "babel"
},
{
"type": "promptString",
"id": "transformFile",
"description": "jscodeshift transform file",
"default": "transform.js"
}
]
}
Once this has been added to the configuration
- Install jscodeshift as a package if you haven't done so already by running the command npm install --save jscodeshift. The debug configuration will not work otherwise.
- Once the jscodeshift local package has been installed, go to the VSCode file tree and select the file on which you want to run the transform. For example, if you wanted to run codemod transforms of foo.js file, you would click on the entry for foo.js file in your project tree.
- Select "Debug Transform" from the debugging menu's options menu.
- Click the "Start Debugging" button on the VSCode debugger.
- You will be then prompted for the name of jscodeshift transform file. Enter in the name of the transform file to use. If no name is given it will default to transform.js
- Select the parser to use from the presented selection list of parsers. The transform will otherwise default to using the babel parser.
- The transform will then be run, stopping at any breakpoints that have been set.
- If there are no errors and the transform is complete, then the results of the transform will be printed in the VSCode debugging console. The file with the contents that have been transformed will not be changed, as the debug configuration makes use the jscodeshift --dry option.
Recipes
Download Details:
Author: facebook
Source Code: https://github.com/facebook/jscodeshift
License: MIT license
1661389500
Memcache Session Support for Gorilla Web Toolkit
gorilla-sessions-memcache
Memcache session support for Gorilla Web Toolkit.
Dependencies
The usual gorilla stuff:
go get github.com/gorilla/sessions
For an ASCII memcache client:
go get github.com/bradfitz/gomemcache/memcache
For a binary memcache client with SASL authentication:
go get github.com/memcachier/mc
Usage
import (
"github.com/bradfitz/gomemcache/memcache"
// or
"github.com/memcachier/mc"
gsm "github.com/bradleypeabody/gorilla-sessions-memcache"
)
...
// set up your memcache client
memcacheClient := gsm.NewGoMemcacher(memcache.New("localhost:11211"))
// or
memcacheClient := mc.NewMC("localhost:11211", "username", "password")
// set up your session store
store := gsm.NewMemcacherStore(memcacheClient, "session_prefix_", []byte("secret-key-goes-here"))
// and the rest of it is the same as any other gorilla session handling:
func MyHandler(w http.ResponseWriter, r *http.Request) {
session, _ := store.Get(r, "session-name")
session.Values["foo"] = "bar"
session.Values[42] = 43
session.Save(r, w)
}
...
// you can also setup a MemCacheStore, which does not rely on the browser accepting cookies.
// this means, your client has to extract and send a configurable http Headerfield manually.
// e.g.
// set up your memcache client
memcacheClient := gsm.NewGoMemcacher(memcache.New("localhost:11211"))
// or
memcacheClient := mc.NewMC("localhost:11211", "username", "password")
// set up your session store relying on a http Headerfield: `X-CUSTOM-HEADER`
store := gsm.NewMemcacherStoreWithValueStorer(memcacheClient, &gsm.HeaderStorer{HeaderFieldName:"X-CUSTOM-HEADER"}, "session_prefix_", []byte("secret-key-goes-here"))
// and the rest of it is the same as any other gorilla session handling:
// The client has to send the session information in the header-field: `X-CUSTOM-HEADER`
func MyHandler(w http.ResponseWriter, r *http.Request) {
session, _ := store.Get(r, "session-name")
session.Values["foo"] = "bar"
session.Values[42] = 43
session.Save(r, w)
}
Storage Methods
I've added a few different methods of storage of the session data in memcache. You use them by setting the StoreMethod field.
- SecureCookie - uses the default securecookie encoding. Values are more secure as they are not readable from memcache without the secret key.
- Gob - uses the Gob encoder directly without any post processing. Faster. Result is Gob's usual binary gibber (not human readable)
- Json - uses the Json Marshaller. Result is human readable, slower but still pretty fast. Be careful - it will munch your data into stuff that works with JSON, and the keys must be strings. Example: you put in an int64 value and you'll get back a float64.
Example:
store := gsm.NewMemcacherStore(memcacheClient, "session_prefix_", []byte("..."))
// do one of these:
store.StoreMethod = gsm.StoreMethodSecureCookie // default, more secure
store.StoreMethod = gsm.StoreMethodGob // faster
store.StoreMethod = gsm.StoreMethodJson // human readable
// (but watch out, it munches your types
// to JSON compatible stuff)
Logging
Logging is available by setting the Logging field to > 0 after making your MemcacheStore.
store := gsm.NewMemcacherStore(memcacheClient, "session_prefix_", []byte("..."))
store.Logging = 1
That will output (using log.Printf) data about each session read/written from/to memcache. Useful for debugging
Things to Know
No official release has been done of this package but it should be stable for production use.
You can also call NewDumbMemorySessionStore() for local development without a memcache server (it's a stub that just stuffs your session data in a map - definitely do not use this for anything but local dev and testing).
Download Details:
Author: Bradleypeabody
Source Code: https://github.com/bradleypeabody/gorilla-sessions-memcache
License: Apache-2.0 license
1661368980
Graph-based Molecule Modeling toolkit for Cheminformatics
MolecularGraph.jl
MolecularGraph.jl is a graph-based molecule modeling and chemoinformatics analysis toolkit fully implemented in Julia.
Usage
Features
Chemical structure file I/O
- 2D structure image drawing and export to SVG
- 3D structure drawing
- SDFile import/export (.sdf, .mol)
- SMILES/SMARTS parser
Database
- InChI (InChI)
- Serialization (molecule object <-> JSON)
Basic descriptors
- H-bond donor/acceptor
- rotatable bonds
- Aromaticity
- Wildman-Crippen logP
Atomic mass
- standard atomic/molecular weight
- relative atomic/molecular mass
- isotopic composition
Molecular graph topology
- Ring, scaffold, connected components
- Minimum cycle basis (de Pina algorithm)
- Smallest set of smallest rings (SSSR)
- Planarity (left-right planarity test)
- Maximum matching
- Kekulization
- Graph traversal
2D geometry
- Stereochemistry drawing
- Coordinates generation (coordgenlibs)
Sub(super)structure
- Library search by using SMARTS query
- Subgraph isomorphism detection with VF2 algorithm
- Monomorphism, node-induced and edge-induced
- Constraints (mandatory/forbidden mapping)
SMARTS query-based substructure analysis
- functional group mining
- structural alerts (by using ChEMBL dataset)
Maximum common substructure (MCS)
- By clique detection algorithm
- Node-induced (MCIS) and edge-induced (MCES)
- Connected and disconnected
- Topological constraint (known as tdMCS)
- Diameter restriction (MCS-DR) and graph-based local similarity (GLS)
Download Details:
Author: Mojaie
Source Code: https://github.com/mojaie/MolecularGraph.jl
License: MIT license
1661368620
Gitflow-toolkit: A Simple toolkit for GitFlow
GitFlow ToolKit
GitFlow Toolkit is a gitflow commit tool written by go, used to standardize the format of git commit message and quickly create gitflow branches, It should be noted that GitFlow Toolkit currently only supports the generation of the commit message style of the Angular community specification.
Starting from the v2.1.1 version, the white theme terminal will be supported, and the white theme color scheme is being adjusted.
 Install |  Uninstall |
 Commit Success |  Commit Failed |
 Push Success |  Push Failed |
 Create Branch |
Installation
Just download the latest version from the Release page and execute the install command:
export VERSION='v2.1.5'
# download bin file
wget https://github.com/mritd/gitflow-toolkit/releases/download/${VERSION}/gitflow-toolkit-darwin-arm64
# add permissions
chmod +x gitflow-toolkit-darwin-arm64
# install
sudo ./gitflow-toolkit-darwin-arm64 installAfter the installation is complete, you can delete the bin file.
If the go language development environment is installed locally, you can install it through the go get command:
go install github.com/mritd/gitflow-toolkit/v2@latestComands
| cmd | desc |
|---|---|
git ci | Enter commit message interactively |
git ps | Push the current branch to the remote |
git feat NAME | Switch a new branch from the current branch (feat/NAME) |
git fix NAME | git switch -c fix/NAME |
git hotfix NAME | git switch -c hotfix/NAME |
git docs NAME | git switch -c docs/NAME |
git style NAME | git switch -c style/NAME |
git refactor NAME | git switch -c refactor/NAME |
git chore NAME | git switch -c chore/NAME |
git perf NAME | git switch -c perf/NAME |
git style NAME | git switch -c style/NAME |
Download Details:
Author: mritd
Source Code: https://github.com/mritd/gitflow-toolkit
License: MIT license
1659556200
Purrr: A Functional Programming toolkit for R
purrr
Overview
purrr enhances R’s functional programming (FP) toolkit by providing a complete and consistent set of tools for working with functions and vectors. If you’ve never heard of FP before, the best place to start is the family of map() functions which allow you to replace many for loops with code that is both more succinct and easier to read. The best place to learn about the map() functions is the iteration chapter in R for data science.
Installation
# The easiest way to get purrr is to install the whole tidyverse:
install.packages("tidyverse")
# Alternatively, install just purrr:
install.packages("purrr")
# Or the the development version from GitHub:
# install.packages("devtools")
devtools::install_github("tidyverse/purrr")
Cheatsheet
Usage
The following example uses purrr to solve a fairly realistic problem: split a data frame into pieces, fit a model to each piece, compute the summary, then extract the R2.
library(purrr)
mtcars %>%
split(.$cyl) %>% # from base R
map(~ lm(mpg ~ wt, data = .)) %>%
map(summary) %>%
map_dbl("r.squared")
#> 4 6 8
#> 0.5086326 0.4645102 0.4229655
This example illustrates some of the advantages of purrr functions over the equivalents in base R:
The first argument is always the data, so purrr works naturally with the pipe.
All purrr functions are type-stable. They always return the advertised output type (map() returns lists; map_dbl() returns double vectors), or they throw an error.
All map() functions either accept function, formulas (used for succinctly generating anonymous functions), a character vector (used to extract components by name), or a numeric vector (used to extract by position).
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
Author: Tidyverse
Source Code: https://github.com/tidyverse/purrr
License: Unknown, MIT licenses found
1659432660
RxJS-Dojo: Reactive Extensions bindings for the Dojo Toolkit
RxJS-Dojo 1.1 - Dojo Bindings for the Reactive Extensions for JavaScript
OVERVIEW
This project provides Reactive Extensions for JavaScript (RxJS) bindings for the Dojo Toolkit to abstract over the event binding, Ajax and other Dojo features. The RxJS libraries are not included with this release and must be installed separately.
GETTING STARTED
There are a number of ways to get started with the Dojo Bindings for RxJS. The files are available on cdnjs and jsDelivr.
Download the Source
To download the source of the Dojo Bindings for the Reactive Extensions for JavaScript, type in the following:
git clone https://github.com/Reactive-Extensions/RxJS-Dojo.git
cd ./RxJS-Dojo
Installing with NPM
npm install rx-dojo
Installing with Bower
bower install rx-dojo
Installing with Jam
jam install rx-dojo
Installing with NuGet
PM> Install-Package RxJS-Bridges-Dojo
API Documentation
You can find the documentation here as well as examples here.
Compatibility
RxJS has been thoroughly tested against all major browsers and supports IE6+, Chrome 4+, FireFox 1+, and Node.js v0.4+ with Dojo 1.7+.
Contributing
There are lots of ways to contribute to the project, and we appreciate our contributors.
You can contribute by reviewing and sending feedback on code checkins, suggesting and trying out new features as they are implemented, submit bugs and help us verify fixes as they are checked in, as well as submit code fixes or code contributions of your own. Note that all code submissions will be rigorously reviewed and tested by the Rx Team, and only those that meet an extremely high bar for both quality and design/roadmap appropriateness will be merged into the source.
Download Details:
Author: Reactive-Extensions
Source Code: https://github.com/Reactive-Extensions/RxJS-Dojo
License: View license
1659060180
RLTK: The Ruby Language toolkit
Welcome to the Ruby Language Toolkit
RLTK is a collection of classes and methods designed to help programmers work with languages in an easy to use and straightforward manner. This toolkit provides the following features:
- Lexer generator
- Parser generator
- AST node baseclass
- Class for representing context free grammars
- Low Level Virtual Machine (LLVM) bindings for code generation
In addition, RLTK includes several ready-made lexers and parsers and a Turing-complete language called Kazoo for use in your code and as examples for how to use the toolkit.
Why Use RLTK
Here are some reasons to use RLTK to build your lexers, parsers, and abstract syntax trees, as well as generating LLVM IR and native object files:
Lexer and Parser Definitions in Ruby - Many tools require you to write your lexer/parser definitions in their own format, which is then processed and used to generate Ruby code. RLTK lexers/parsers are written entirely in Ruby and use syntax you are already familiar with.
Re-entrant Code - The lexers and parsers generated by RLTK are fully re-entrant.
Multiple Lexers and Parsers - You can define as many lexers and parses as you want, and instantiate as many of them as you need.
Token Positions - Detailed information about a token's position is available in the parser.
Feature Rich Lexing and Parsing - Often, lexer and parser generators will try and force you to do everything their way. RLTK gives you more flexibility with features such as states and flags for lexers, and argument arrays for parsers. What's more, these features actually work (I'm looking at you REX).
LALR(1)/GLR Parsing - RLTK parsers use the LALR(1)/GLR parsing algorithms, which means you get both speed and the ability to handle any context-free grammar.
Parser Serialization - RLTK parsers can be serialized and saved after they are generated for faster loading the next time they are required.
Error Productions - RLTK parsers can use error productions to recover from, and report on, errors.
Fast Prototyping - If you need to change your lexer/parser you don't have to re-run the lexer and parser generation tools, simply make the changes and be on your way.
Parse Tree Graphs - RLTK parsers can print parse trees (in the DOT language) of accepted strings.
LLVM Bindings - RLTK provides wrappers for most of the C LLVM bindings.
The Contractor - LLVM's method of building instructions is a bit cumbersome, and is very imperative in style. RLTK provides the Contractor class to make things easier.
Documentation - We have it!
I Eat My Own Dog Food - I'm using RLTK for my own projects so if there is a bug I'll most likely be the first one to know.
Lexers
To create your own lexer using RLTK you simply need to subclass the {RLTK::Lexer} class and define the rules that will be used for matching text and generating tokens. Here we see a simple lexer for a calculator:
class Calculator < RLTK::Lexer
rule(/\+/) { :PLS }
rule(/-/) { :SUB }
rule(/\*/) { :MUL }
rule(/\//) { :DIV }
rule(/\(/) { :LPAREN }
rule(/\)/) { :RPAREN }
rule(/[0-9]+/) { |t| [:NUM, t.to_i] }
rule(/\s/)
end
The {RLTK::Lexer.rule} method's first argument is the regular expression used for matching text. The block passed to the function is the action that executes when a substring is matched by the rule. These blocks must return the type of the token (which must be in ALL CAPS; see the Parsers section), and optionally a value. In the latter case you must return an array containing the type and value, which you can see an example of in the Calculator lexer shown above. The values returned by the proc object are used to build a {RLTK::Token} object that includes the type and value information, as well as information about the line number the token was found on, the offset from the beginning of the line to the start of the token, and the length of the token's text. If the type value returned by the proc is nil the input is discarded and no token is produced.
The {RLTK::Lexer} class provides both {RLTK::Lexer.lex} and {RLTK::Lexer.lex_file}. The {RLTK::Lexer.lex} method takes a string as its argument and returns an array of tokens, with an end of stream token automatically added to the result. The {RLTK::Lexer.lex_file} method takes the name of a file as input, and lexes the contents of the specified file.
The Lexing Environment
The proc objects passed to the {RLTK::Lexer.rule} methods are evaluated inside an instance of the {RLTK::Lexer::Environment} class. This gives you access to methods for manipulating the lexer's state and flags (see bellow). You can also subclass the environment inside your lexer to provide additional functionality to your rule blocks. When doing so you need to ensure that you name your new class Environment like in the following example:
class MyLexer < RLTK::Lexer
...
class Environment < Environment
def helper_function
...
end
...
end
end
Using States
The lexing environment may be used to keep track of state inside your lexer. When rules are defined they are defined inside a given state, which is specified by the second parameter to {RLTK::Lexer.rule}. The default state is cleverly named :default. When the lexer is scanning the input string for matching rules, it only considers the rules for the given state.
The methods used to manipulate state are:
- RLTK::Lexer::Environment.push_state - Pushes a new state onto the stack.
- RLTK::Lexer::Environment.pop_state - Pops a state off of the stack.
- RLTK::Lexer::Environment.set_state - Sets the state at the top of the stack.
- RLTK::Lexer::Environment.state - Returns the current state.
States may be used to easily support nested comments.
class StateLexer < RLTK::Lexer
rule(/a/) { :A }
rule(/\s/)
rule(/\(\*/) { push_state(:comment) }
rule(/\(\*/, :comment) { push_state(:comment) }
rule(/\*\)/, :comment) { pop_state }
rule(/./, :comment)
end
By default the lexer will start in the :default state. To change this, you may use the {RLTK::Lexer.start} method.
Using Flags
The lexing environment also maintains a set of flags. This set is manipulated using the following methods:
- RLTK::Lexer::Environment.set_flag - Adds the specified flag to the set of flags.
- RLTK::Lexer::Environment.unset_flag - Removes the specified flag from the set of flags.
- RLTK::Lexer::Environment.clear_flags - Unsets all flags.
When rules are defined they may use a third parameter to specify a list of flags that must be set before the rule is considered when matching substrings. An example of this usage follows:
class FlagLexer < RLTK::Lexer
rule(/a/) { set_flag(:a); :A }
rule(/\s/)
rule(/b/, :default, [:a]) { set_flag(:b); :B }
rule(/c/, :default, [:a, :b]) { :C }
end
Instantiating Lexers
In addition to using the {RLTK::Lexer.lex} class method you may also instantiate lexer objects. The only difference then is that the lexing environment used between subsequent calls to {RLTK::Lexer#lex} is the same object, and therefor allows you to keep persistent state.
First and Longest Match
A RLTK::Lexer may be told to select either the first substring that is found to match a rule or the longest substring to match any rule. The default behavior is to match the longest substring possible, but you can change this by using the {RLTK::Lexer.match_first} method inside your class definition as follows:
class MyLexer < RLTK::Lexer
match_first
...
end
Match Data
Because it isn't RLTK's job to tell you how to write lexers and parsers, the MatchData object from a pattern match is available inside the Lexer::Environment object via the match accessor.
Context-Free Grammars
The {RLTK::CFG} class provides an abstraction for context-free grammars. For the purpose of this class terminal symbols appear in ALL CAPS, and non-terminal symbols appear in all lowercase. Once a grammar is defined the {RLTK::CFG#first_set} and {RLTK::CFG#follow_set} methods can be used to find first and follow sets.
Defining Grammars
A grammar is defined by first instantiating the {RLTK::CFG} class. The {RLTK::CFG#production} and {RLTK::CFG#clause} methods may then be used to define the productions of the grammar. The production method can take a Symbol denoting the left-hand side of the production and a string describing the right-hand side of the production, or the left-hand side symbol and a block. In the first usage a single production is created. In the second usage the block may contain repeated calls to the clause method, each call producing a new production with the same left-hand side but different right-hand sides. {RLTK::CFG#clause} may not be called outside of {RLTK::CFG#production}. Bellow we see a grammar definition that uses both methods:
grammar = RLTK::CFG.new
grammar.production(:s) do
clause('A G D')
clause('A a C')
clause('B a D')
clause('B G C')
end
grammar.production(:a, 'b')
grammar.production(:b, 'G')
Extended Backus–Naur Form
The RLTK::CFG class understands grammars written in the extended Backus–Naur form. This allows you to use the *, +, and ? operators in your grammar definitions. When each of these operators are encountered additional productions are generated. For example, if the right-hand side of a production contained NUM* a production of the form num_star -> | NUM num_star is added to the grammar. As such, your grammar should not contain productions with similar left-hand sides (e.g. foo_star, bar_question, or baz_plus).
As these additional productions are added internally to the grammar a callback functionality is provided to let you know when such an event occurs. The callback proc object can either be specified when the CFG object is created, or by using the {RLTK::CFG#callback} method. The callback will receive three arguments: the production generated, the operator that triggered the generation, and a symbol (:first or :second) specifying which clause of the production this callback is for.
Helper Functions
Once a grammar has been defined you can use the following functions to obtain information about it:
- {RLTK::CFG#first_set} - Returns the first set for the provided symbol or sentence.
- {RLTK::CFG#follow_set} - Returns the follow set for the provided symbol.
- {RLTK::CFG#nonterms} - Returns a list of the non-terminal symbols used in the grammar's definition.
- {RLTK::CFG#productions} - Provides either a hash or array of the grammar's productions.
- {RLTK::CFG#symbols} - Returns a list of all symbols used in the grammar's definition.
- {RLTK::CFG#terms} - Returns a list of the terminal symbols used in the grammar's definition.
Parsers
To create a parser using RLTK simply subclass RLTK::Parser, define the productions of the grammar you wish to parse, and call finalize. During finalization RLTK will build an LALR(1) parsing table, which may contain conflicts that can't be resolved with LALR(1) lookahead sets or precedence/associativity information. Traditionally, when parser generators such as YACC encounter conflicts during parsing table generation they will resolve shift/reduce conflicts in favor of shifts and reduce/reduce conflicts in favor of the production that was defined first. This means that the generated parsers can't handle ambiguous grammars.
RLTK parsers, on the other hand, can handle all context-free grammars by forking the parse stack when shift/reduce or reduce/reduce conflicts are encountered. This method is called the GLR parsing algorithm and allows the parser to explore multiple different possible derivations, discarding the ones that don't produce valid parse trees. GLR parsing is more expensive, in both time and space requirements, but these penalties are only payed when a parser for an ambiguous grammar is given an input with multiple parse trees, and as such most parsing should proceed using the faster LALR(1) base algorithm.
Defining a Grammar
Let us look at the simple prefix calculator included with RLTK:
class PrefixCalc < RLTK::Parser
production(:e) do
clause('NUM') {|n| n}
clause('PLS e e') { |_, e0, e1| e0 + e1 }
clause('SUB e e') { |_, e0, e1| e0 - e1 }
clause('MUL e e') { |_, e0, e1| e0 * e1 }
clause('DIV e e') { |_, e0, e1| e0 / e1 }
end
finalize
end
The parser uses the same method for defining productions as the {RLTK::CFG} class. In fact, the parser forwards the {RLTK::Parser.production} and {RLTK::Parser.clause} method invocations to an internal {RLTK::CFG} object after removing the parser specific information. To see a detailed description of grammar definitions please read the Context-Free Grammars section bellow.
It is important to note that the proc objects associated with productions should evaluate to the value you wish the left-hand side of the production to take.
The default starting symbol of the grammar is the left-hand side of the first production defined (in this case, e). This can be changed using the {RLTK::Parser.start} function when defining your parser.
Make sure you call finalize at the end of your parser definition, and only call it once.
Shortcuts
RLTK provides several shortcuts for common grammar constructs. Right now these shortcuts include the {RLTK::Parser.list} and {RLTK::Parser.nonempty_list} methods. A list may contain 0, 1, or more elements, with an optional token or tokens separating each element. A non-empty list contains at least 1 element. An empty list with only a single list element and an empty separator is equivalent to the Kleene star. Similarly, a list with only a single list element and an empty separator is equivalent to the Kleene plus.
This example shows how these shortcuts may be used to define a list of integers separated by a :COMMA token:
class ListParser < RLTK::Parser
nonempty_list(:int_list, :INT, :COMMA)
finalize
end
If you wanted to define a list of floats or integers you could define your parser like this:
class ListParser < RLTK::Parser
nonempty_list(:mixed_list, [:INT, :FLOAT], :COMMA)
finalize
end
If you don't want to require a separator you can do this:
class ListParser < RLTK::Parser
nonempty_list(:mixed_nonsep_list, [:INT, :FLOAT])
finalize
end
You can also use separators that are made up of multiple tokens:
class ListParser < RLTK::Parser
nonempty_list(:mixed_nonsep_list, [:INT, :FLOAT], 'COMMA NEWLINE?')
finalize
end
A list may also contain multiple tokens between the separator:
class ListParser < RLTK::Parser
nonempty_list(:foo_bar_list, 'FOO BAR', :COMMA)
finalize
end
Lastly, you can mix all of these features together:
class ListParser < RLTK::Parser
nonempty_list(:foo_list, ['FOO BAR', 'FOO BAZ+'], :COMMA)
finalize
end
The productions generated by these shortcuts will always evaluate to an array. In the first two examples above the productions will produce a 1-D array containing the values of the INT or FLOAT tokens. In the last two examples the productions foo_bar_list and foo_list will produce 2-D arrays where the top level array is composed of tuples corresponding to the values of FOO, and BAR or one or more BAZs.
Precedence and Associativity
To help you remove ambiguity from your grammars RLTK lets you assign precedence and associativity information to terminal symbols. Productions then get assigned precedence and associativity based on either the last terminal symbol on the right-hand side of the production, or an optional parameter to the {RLTK::Parser.production} or {RLTK::Parser.clause} methods. When an {RLTK::Parser} encounters a shift/reduce error it will attempt to resolve it using the following rules:
If there is precedence and associativity information present for all reduce actions involved and for the input token we attempt to resolve the conflict using the following rule. If not, no resolution is possible and the parser generator moves on. This conflict will later be reported to the programmer.
The precedence of the actions involved in the conflict are compared (a shift action's precedence is based on the input token), and the action with the highest precedence is selected. If two actions have the same precedence the associativity of the input symbol is used: left associativity means we select the reduce action, right associativity means we select the shift action, and non-associativity means that we have encountered an error.
To assign precedence to terminal symbols you can use the {RLTK::Parser.left}, {RLTK::Parser.right}, and {RLTK::Parser.nonassoc} methods inside your parser class definition. Later declarations of associativity have higher levels of precedence than earlier declarations of the same associativity.
Let's look at the infix calculator example now:
class InfixCalc < RLTK::Parser
left :PLS, :SUB
right :MUL, :DIV
production(:e) do
clause('NUM') { |n| n }
clause('LPAREN e RPAREN') { |_, e, _| e }
clause('e PLS e') { |e0, _, e1| e0 + e1 }
clause('e SUB e') { |e0, _, e1| e0 - e1 }
clause('e MUL e') { |e0, _, e1| e0 * e1 }
clause('e DIV e') { |e0, _, e1| e0 / e1 }
end
finalize
end
The standard order of mathematical operations tells us that the correct way to group the operations in the expression 2 + 3 * 4 is 2 + (3 * 4). However, our grammar tells us that (2 + 3) * 5 is also a valid way to parse the expression, leading to a shift/reduce error in the parser. To get rid of the shift/reduce error we need some way to tell the parser how to distinguish between these two parse trees. This is where associativity comes in. If the parser has already read NUM PLS NUM and the current symbol is a MUL symbol we want to tell the parser to shift the new MUL symbol onto the stack and continue on. We do this by making the MUL symbol right associative. When the parser generator encounters a shift/reduce error it looks at the token currently being read. If it has no associativity information, the error can't be resolved; if the token is left associative, it will remove the shift action from the parser (leaving only the reduce action); if the token is right associative, it will remove the reduce action from the parser (leaving only the shift action).
Now, let us consider the expression 3 - 2 - 1. Here, the correct way to parse the expression is (3 - 2) - 1. To ensure that this case is selected over 3 - (2 - 1) we can make the SUB token left associative. This will cause the symbols NUM SUB NUM to be reduced before the second SUB symbol is shifted onto the parse stack.
Not that, to resolve a shift/reduce or reduce/reduce conflict, precedence and associativity information must be present for all actions involved in the conflict. As such, it isn't enough to simply make the MUL and DIV tokens right associative; we must also make the PLS and SUB tokens left associative.
Token Selectors
In many cases productions contain tokens who's value is unimportant. In such situations passing nil to the production's action is not useful. To prevent this happening you may use token selectors. By placing a period (.) in front of a token you can indicate to the parser that the following token is important and you wish for its value to be passed to the action. In the following example selectors are used to only pass the sub-expressions' values to the action:
class InfixCalc < RLTK::Parser
left :PLS, :SUB
right :MUL, :DIV
production(:e) do
clause('NUM') { |n| n }
clause('LPAREN .e RPAREN') { |e| e }
clause('.e PLS .e') { |e0, e1| e0 + e1 }
clause('.e SUB .e') { |e0, e1| e0 - e1 }
clause('.e MUL .e') { |e0, e1| e0 * e1 }
clause('.e DIV .e') { |e0, e1| e0 / e1 }
end
finalize
end
Argument Passing for Actions
By default the proc objects associated with productions are passed one argument for each symbol on the right-hand side of the production. This can lead to long, unwieldy argument lists. To change this behaviour you can use the {RLTK::Parser.default_arg_type} method, which accepts the :splat (default) and :array arguments. Any production actions that are defined after a call to {RLTK::Parser.default_arg_type} will use the argument passing method currently set as the default. You can switch between the different argument passing methods by calling {RLTK::Parser.default_arg_type} repeatedly.
Individual productions may specify the argument type used by their action via the arg_type parameter. If the {RLTK::Parser.production} method is passed an argument type and a block, any clauses defined inside the block will use the argument type specified by the arg_type parameter.
The Parsing Environment
The parsing environment is the context in which the proc objects associated with productions are evaluated, and can be used to provide helper functions and to keep state while parsing. To define a custom environment simply subclass {RLTK::Parser::Environment} inside your parser definition as follows:
class MyParser < RLTK::Parser
...
class Environment < Environment
def helper_function
...
end
...
end
finalize
end
(The definition of the Environment class may occur anywhere inside the MyParser class definition.)
Instantiating Parsers
In addition to using the {RLTK::Parser.parse} class method you may also instantiate parser objects. The only difference then is that the parsing environment used between subsequent calls to {RLTK::Parser#parse} is the same object, and therefor allows you to keep persistent state.
Finalization Options
The {RLTK::Parser.finalize} method has several options that you should be aware of:
explain - Value should be true, false, an IO object, or a file name. Default value is false. If a non false (or nil) value is specified finalize will print an explanation of the parser to $stdout, the provided IO object, or the specified file. This explanation will include all of the productions defined, all of the terminal symbols used in the grammar definition, and the states present in the parsing table along with their items, actions, and conflicts.
lookahead - Either true or false. Default value is true. Specifies whether the parser generator should build an LALR(1) or LR(0) parsing table. The LALR(1) table may have the same actions as the LR(0) table or fewer reduce actions if it is possible to resolve conflicts using lookahead sets.
precedence - Either true or false. Default value is true. Specifies whether the parser generator should use precedence and associativity information to solve conflicts.
use - Value should be false, the name of a file, or a file object. If the file exists and hasn't been modified since the parser definition was RLTK will load the parser definition from the file, saving a bunch of time. If the file doesn't exist or the parser has been modified since it was last used RLTK will save the parser's data structures to this file.
Parsing Options
The {RLTK::Parser.parse} and {RLTK::Parser#parse} methods also have several options that you should be aware of:
accept - Either :first or :all. Default value is :first. This option tells the parser to accept the first successful parse-tree found, or all parse-trees that enter the accept state. It only affects the behavior of the parser if the defined grammar is ambiguous.
env - This option specifies the environment in which the productions' proc objects are evaluated. The RLTK::Parser::parse class function will create a new RLTK::Parser::Environment on each call unless one is specified. RLTK::Parser objects have an internal, per-instance, RLTK::Parser::Environment that is the default value for this option when calling RLTK::Parser.parse
parse_tree - Value should be true, false, an IO object, or a file name. Default value is false. If a non false (or nil) value is specified a DOT language description of all accepted parse trees will be printed out to $stdout, the provided IO object, or the specified file.
verbose - Value should be true, false, an IO object, or a file name. Default value is false. If a non false (or nil) value is specified a detailed description of the actions of the parser are printed to $stdout, the provided IO object, or the specified file as it parses the input.
Parse Trees
The above section briefly mentions the parse_tree option. So that this neat feature doesn't get lost in the rest of the documentation here is the tree generated by the Kazoo parser from Chapter 7 of the tutorial when it parses the line def fib(a) if a < 2 then 1 else fib(a-1) + fib(a-2);:
Parsing Exceptions
Calls to {RLTK::Parser.parse} may raise one of four exceptions:
- RLTK::BadToken - This exception is raised when a token is observed in the input stream that wasn't used in the language's definition.
- RLTK::HandledError - This exception is raised whenever an error production is encountered. The input stream is not actually in the language, but we were able to handle the encountered errors in a way that makes it appear that it is.
- RLTK::InternalParserError - This exception tells you that something REALLY went wrong. Users should never receive this exception.
- RLTK::NotInLanguage - This exception indicates that the input token stream is not in the parser's language.
Error Productions
Warning: this is the least tested feature of RLTK. If you encounter any problems while using it, please let me know so I can fix any bugs as soon as possible.
When an RLTK parser encounters a token for which there are no more valid actions (and it is on the last parse stack / possible parse-tree path) it will enter error handling mode. In this mode the parser pops states and input off of the parse stack (the parser is a pushdown automaton after all) until it finds a state that has a shift action for the ERROR terminal. A dummy ERROR terminal is then placed onto the parse stack and the shift action is taken. This error token will have the position information of the token that caused the parser to enter error handling mode. Additional tokens may have been discarded after this token.
If the input (including the ERROR token) can be reduced immediately the associated error handling proc is evaluated and we continue parsing. If no shift or reduce action is available the parser will being shifting tokens off of the input stack until a token appears with a valid action in the current state, in which case parsing resumes as normal.
The value of an ERROR non-terminal will be an array containing all of the tokens that were discarded while the parser was searching for a valid action.
The example below, based on one of the unit tests, shows a very basic usage of error productions:
class ErrorCalc < RLTK::Parser
left :ERROR
right :PLS, :SUB, :MUL, :DIV, :NUM
production(:e) do
clause('NUM') {|n| n}
clause('e PLS e') { |e0, _, e1| e0 + e1 }
clause('e SUB e') { |e0, _, e1| e0 - e1 }
clause('e MUL e') { |e0, _, e1| e0 * e1 }
clause('e DIV e') { |e0, _, e1| e0 / e1 }
clause('e PLS ERROR e') { |e0, _, err, e1| error("#{err.len} tokens skipped."); e0 + e1 }
end
finalize
end
A Note on Token Naming
In the world of RLTK both terminal and non-terminal symbols may contain only alphanumeric characters and underscores. The differences between terminal and non-terminal symbols is that terminals are ALL_UPPER_CASE and non-terminals are all_lower_case.
ASTNode
The {RLTK::ASTNode} base class is meant to be a good starting point for implementing your own abstract syntax tree nodes. By subclassing {RLTK::ASTNode} you automagically get features such as tree comparison, notes, value accessors with type checking, child node accessors and each and map methods (with type checking), and the ability to retrieve the root of a tree from any member node.
To create your own AST node classes you subclass the {RLTK::ASTNode} class and then use the {RLTK::ASTNode.child} and {RLTK::ASTNode.value} methods. By declaring the children and values of a node the class will define the appropriate accessors with type checking, know how to pack and unpack a node's children, and know how to handle constructor arguments.
Here we can see the definition of several AST node classes that might be used to implement binary operations for a language:
class Expression < RLTK::ASTNode; end
class Number < Expression
value :value, Fixnum
end
class BinOp < Expression
value :op, String
child :left, Expression
child :right, Expression
end
The assignment functions that are generated for the children and values perform type checking to make sure that the AST is well-formed. The type of a child must be a subclass of the {RLTK::ASTNode} class, whereas the type of a value can be any Ruby class. While child and value objects are stored as instance variables it is unsafe to assign to these variables directly, and it is strongly recommended to always use the accessor functions.
When instantiating a subclass of {RLTK::ASTNode} the arguments to the constructor should be the node's values (in order of definition) followed by the node's children (in order of definition). If a constructor is given fewer arguments then the number of values and children the remaining arguments are assumed to be nil. Example:
class Foo < RLTK::ASTNode
value :a, Fixnum
child :b, Bar
value :c, String
child :d, Bar
end
class Bar < RLTK::ASTNode
value :a, String
end
Foo.new(1, 'baz', Bar.new)
Lastly, the type of a child or value can be defined as an array of objects of a specific type as follows:
class Foo < RLTK::ASTNode
value :strings, [String]
end
Tree Iteration and Mapping
RLTK Abstract Syntax Trees may be traversed in three different ways:
- Pre-order
- Post-order
- Level-order
The order you wish to traverse the tree can be specified by passing the appropriate symbol to {RLTK::ASTNode#each}: :pre, :post, or :level.
You can also map one tree to another tree using the {RLTK::ASTNode#map} and {RLTK::ASTNode#map!} methods. In the former case a new tree is created and returned; in the latter case the current tree is transformed and the result of calling the provided block on the root node is returned. These methods will always visit nodes in post-order, so that all children of a node are visited before the node itself.
Code Generation
RLTK supports the generation of native code and LLVM IR, as well as JIT compilation and execution, through the {RLTK::CG} module. This module is built on top of bindings to LLVM and provides much, though not all, of the functionality of the LLVM libraries.
Acknowledgments and Discussion
Before we get started with the details, I would like to thank Jeremy Voorhis. The bindings present in RLTK are really a fork of the great work that he did on ruby-llvm.
Why did I fork ruby-llvm, and why might you want to use the RLTK bindings over ruby-llvm? There are a couple of reasons:
- Cleaner Codebase - The RLTK bindings present a cleaner interface to the LLVM library by conforming to more standard Ruby programming practices, providing better abstractions and cleaner inheritance hierarchies, overloading constructors and other methods properly, and performing type checking on objects to better aid in debugging.
- Documentation - RLTK's bindings provide better documentation.
- Completeness - The RLTK bindings provide several features that are missing from the ruby-llvm project. These include the ability to initialize LLVM for architectures besides x86 (RLTK supports all architectures supported by LLVM), the presence of all of LLVM's optimization passes, the ability to print the LLVM IR representation of modules and values to files and load modules from files, easy initialization of native architectures, initialization for ASM printers and parsers, and compiling modules to object files.
- Ease of Use - Several features have been added to make generating code easier such as automatic management of memory resources used by LLVM.
- Speed - The RLTK bindings are ever so slightly faster due to avoiding unnecessary FFI calls.
Before you dive into generating code, here are some resources you might want to look over to build up some background knowledge on how LLVM works:
LLVM
Since RLTK's code generation functionality is built on top of LLVM the first step in generating code is to inform LLVM of the target architecture. This is accomplished via the {RLTK::CG::LLVM.init} method, which is used like this: RLTK::CG::LLVM.init(:PPC). The {RLTK::CG::Bindings::ARCHS} constant provides a list of supported architectures. This call must appear before any other calls to the RLTK::CG module.
If you would like to see what version of LLVM is targeted by your version of RLTK you can either call the {RLTK::CG::LLVM.version} method or looking at the {RLTK::LLVM_TARGET_VERSION} constant.
Modules
Modules are one of the core building blocks of the code generation module. Functions, constants, and global variables all exist inside a particular module and, if you use the JIT compiler, a module provides the context for your executing code. New modules can be created using the {RLTK::CG::Module#initialize RLTK::CG::Module.new} method. While this method is overloaded you, as a library user, will always pass it a string as its first argument. This allows you to name your modules for easier debugging later.
Once you have created you can serialize the code inside of it into bitcode via the {RLTK::CG::Module#write_bitcode} method. This allows you to save partially generated code and then use it later. To load a module from bitcode you use the {RLTK::CG::Module.read_bitcode} method.
Types
Types are an important part of generating code using LLVM. Functions, operations, and other constructs use types to make sure that the generated code is sane. All types in RLTK are subclasses of the {RLTK::CG::Type} class, and have class names that end in "Type". Types can be grouped into to categories: fundamental and composite.
Fundamental types are those like {RLTK::CG::Int32Type} and {RLTK::CG::FloatType} that don't take any arguments when they are created. Indeed, these types are represented using a Singleton class, and so the new method is disabled. Instead you can use the instance method to get an instantiated type, or simply pass in the class itself whenever you need to reference the type. In this last case, the method you pass the class to will instantiate the type for you.
Composite types are constructed from other types. These include the {RLTK::CG::ArrayType}, {RLTK::CG::FunctionType}, and other classes. These types you must instantiate directly before they can be used, and you may not simply pass the type class as the type argument to functions inside the RLTK::CG module.
For convenience, the native integer type of the host platform is made available via {RLTK::CG::NativeIntType}.
Values
The {RLTK::CG::Value} class is the common ancestor of many classes inside the RLTK::CG module. The main way in which you, the library user, will interact with them is when creating constant values. Here is a list of some of value classes you might use:
- {RLTK::CG::Int1}
- {RLTK::CG::Int8}
- {RLTK::CG::Int16}
- {RLTK::CG::Int32}
- {RLTK::CG::Int64}
- {RLTK::CG::Float}
- {RLTK::CG::Double}
- {RLTK::CG::ConstantArray}
- {RLTK::CG::ConstantStruct}
Again, for convenience, the native integer class of the host platform is made available via {RLTK::CG::NativeInt}.
Functions
Functions in LLVM are much like C functions; they have a return type, argument types, and a body. Functions may be created in several ways, though they all require a module in which to place the function.
The first way to create functions is via a module's function collection:
mod.functions.add('my function', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
Here we have defined a function named 'my function' in the mod module. It takes two native integers as arguments and returns a native integer. It is also possible to define the type of a function ahead of time and pass it to this method:
type = RLTK::CG::FunctionType.new(RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
mod.functions.add('my function', type)
Functions may also be created directly via the {RLTK::CG::Function#initialize RLTK::CG::Function.new} method, though a reference to a module is still necessary:
mod = Module.new('my module')
fun = Function.new(mod, 'my function', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
or
mod = Module.new('my module')
type = RLTK::CG::FunctionType.new(RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
fun = Function.new(mod, 'my function', type)
Lastly, whenever you use one of these methods to create a function you may give it a block to be executed inside the context of the function object. This allows for easier building of functions:
mod.functions.add('my function', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType]) do
bb = blocks.append('entry)'
...
end
Basic Blocks
Once a function has been added to a module you will need to add {RLTK::CG::BasicBlock BasicBlocks} to the function. This can be done easily:
bb = fun.blocks.append('entry')
We now have a basic block that we can use to add instructions to our function and get it to actually do something. You can also instantiate basic blocks directly:
bb = RLTK::CG::BasicBlock.new(fun, 'entry')
The Builder
Now that you have a basic block you need to add instructions to it. This is accomplished using a {RLTK::CG::Builder builder}, either directly or indirectly.
To add instructions using a builder directly (this is most similar to how it is done using C/C++) you create the builder, position it where you want to add instructions, and then build them:
fun = mod.functions.add('add', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
bb = fun.blocks.append('entry')
builder = RLTK::CG::Builder.new
builder.position_at_end(bb)
# Generate an add instruction.
inst0 = builder.add(fun.params[0], fun.params[1])
# Generate a return instruction.
builder.ret(inst0)
You can get rid of some of those references to the builder by using the {RLTK::CG::Builder#build} method:
fun = mod.functions.add('add', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
bb = fun.blocks.append('entry')
builder = RLTK::CG::Builder.new
builder.build(bb) do
ret add(fun.params[0], fun.params[1])
end
To get rid of more code:
fun = mod.functions.add('add', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
bb = fun.blocks.append('entry')
RLTK::CG::Builder.new(bb) do
ret add(fun.params[0], fun.params[1])
end
or
fun = mod.functions.add('add', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType])
fun.blocks.append('entry') do
ret add(fun.params[0], fun.params[1])
end
or even
mod.functions.add('add', RLTK::CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType]) do
blocks.append('entry') do |fun|
ret add(fun.params[0], fun.params[1])
end
end
In the last two examples a new builder object is created for the block. It is possible to specify the builder to be used:
builder = RLTK::CG::Builder.new
mod.functions.add('add', RLTK:CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType]) do
blocks.append('entry', builder) do |fun|
ret add(fun.params[0], fun.params[1])
end
end
For an example of where this is useful, see the Kazoo tutorial.
The Contractor
An alternative to using the {RLTK::CG::Builder} class is to use the {RLTK::CG::Contractor} class, which is a subclass of the Builder and includes the Filigree::Visitor module. (Get it? It's a visiting builder!) By subclassing the Contractor you can define blocks of code for handling various types of AST nodes and leave the selection of the correct code up to the {RLTK::CG::Contractor#visit} method. In addition, the :at and :rcb options to the visit method make it much easier to manage the positioning of the Builder.
Here we can see how easy it is to define a block that builds the instructions for binary operations:
on Binary do |node|
left = visit node.left
right = visit node.right
case node
when Add then fadd(left, right, 'addtmp')
when Sub then fsub(left, right, 'subtmp')
when Mul then fmul(left, right, 'multmp')
when Div then fdiv(left, right, 'divtmp')
when LT then ui2fp(fcmp(:ult, left, right, 'cmptmp'), RLTK::CG::DoubleType, 'booltmp')
end
end
AST nodes whose translation requires the generation of control flow will require the creation of new BasicBlocks and the repositioning of the builder. This can be easily managed:
on If do |node|
cond_val = visit node.cond
fcmp :one, cond_val, ZERO, 'ifcond'
start_bb = current_block
fun = start_bb.parent
then_bb = fun.blocks.append('then')
then_val, new_then_bb = visit node.then, at: then_bb, rcb: true
else_bb = fun.blocks.append('else')
else_val, new_else_bb = visit node.else, at: else_bb, rcb: true
merge_bb = fun.blocks.append('merge', self)
phi_inst = build(merge_bb) { phi RLTK::CG::DoubleType, {new_then_bb => then_val, new_else_bb => else_val}, 'iftmp' }
build(start_bb) { cond cond_val, then_bb, else_bb }
build(new_then_bb) { br merge_bb }
build(new_else_bb) { br merge_bb }
returning(phi_inst) { target merge_bb }
end
More extensive examples of how to use the Contractor class can be found in the Kazoo tutorial chapters.
Execution Engines
Once you have generated your code you may want to run it. RLTK provides bindings to both the LLVM interpreter and JIT compiler to help you do just that. Creating a JIT compiler is pretty simple.
mod = RLTK::CG::Module.new('my module')
jit = RLTK::CG::JITCompiler(mod)
mod.functions.add('add', RLTK:CG::NativeIntType, [RLTK::CG::NativeIntType, RLTK::CG::NativeIntType]) do
blocks.append('entry', nil, nil, self) do |fun|
ret add(fun.params[0], fun.params[1])
end
end
Now you can run your 'add' function like this:
jit.run(fun, 1, 2)
The result will be a {RLTK::CG::GenericValue} object, and you will want to use its {RLTK::CG::GenericValue#to_i #to_i} and {RLTK::CG::GenericValue#to_f #to_f} methods to get the Ruby value result.
Tutorial
What follows is an in-depth example of how to use the Ruby Language Toolkit. This tutorial will show you how to use RLTK to build a lexer, parser, AST nodes, and compiler to create a toy language called Kazoo. The tutorial is based on the LLVM Kaleidoscope tutorial, but has been modified to:
- a) be done in Ruby
-
- use a lexer and parser generator and
- III) use a language that I call Kazoo, which is really just a cleaned up and simplified version of the Kaleidoscope language used in the LLVM tutorial (as opposed to the Kaleidoscope language from the 90′s).
The Kazoo toy language is a procedural language that allows you to define functions, use conditionals, and perform basic mathematical operations. Over the course of the tutorial we’ll extend Kazoo to support the if/then/else construct, for loops, JIT compilation, and a simple command line interface to the JIT.
Because we want to keep things simple the only datatype in Kazoo is a 64-bit floating point type (a C double or a Ruby float). As such, all values are implicitly double precision and the language doesn’t require type declarations. This gives the language a very nice and simple syntax. For example, the following example computes Fibonacci numbers:
def fib(x)
if x < 3 then
1
else
fib(x-1) + fib(x-2)
The tutorial is organized as follows:
- Chapter 1: The Lexer
- Chapter 2: The AST Nodes
- Chapter 3: The Parser
- Chapter 4: AST Translation
- Chapter 5: JIT Compilation
- Chapter 6: Adding Control Flow
- Chapter 7: Playtime
- Chapter 8: Mutable Variables
Before starting this tutorial you should know about regular expressions, the basic ideas behind lexing and parsing, and be able to read context-free grammar (CFG) definitions. By the end of this tutorial we will have written 372 lines of source code and have a JIT compiler for a Turing complete language.
Provided Lexers and Parsers
The following lexer and parser classes are included as part of RLTK:
- {RLTK::Lexers::Calculator}
- {RLTK::Lexers::EBNF}
- {RLTK::Parsers::PrefixCalc}
- {RLTK::Parsers::InfixCalc}
- {RLTK::Parsers::PostfixCalc}
Contributing
If you are interested in contributing to RLTK there are many aspects of the library that you can work on. A detailed TODO list can be found here. If you are looking for smaller units of work feel free to:
- Help provide unit tests. Not all of RLTK is tested as well as it could be. Specifically, more tests for the RLTK::CFG and RLTK::Parser classes would be appreciated.
- Write lexers or parsers that you think others might want to use. Possibilities include HTML, JSON/YAML, Javascript, and Ruby.
- Extend the RLTK::CFG class with additional functionality.
Lastly, I love hearing back from users. If you find any part of the documentation unclear or incomplete let me know. It is also helpful to me to know how people are using the library, so if you are using RLTK in your project send me an email. This lets me know what features are being used and where I should focus my development efforts.
News
Aaaaand we're back. Development of RLTK has been on hold for a while as I worked on other projects. If you want to see what I've been up to, you can check out Clang's new -Wconsumed flag and the Filigree gem.
The next version of RLTK is going to be updated to require Ruby 2.0 as well as LLVM 3.4. Previous versions of RLTK required my LLVM-ECB libarary to expose extra LLVM features through the C bindings; this is no longer necessary as this functionality has been moved into LLVM proper. If anyone has any requests for new or improved features for RLTK version 3.0, let me know.
Author: Chriswailes
Source Code: https://github.com/chriswailes/RLTK
License: NCSA license
1659044580
A Thin Julia Wrapper for Speech Signal Processing toolkit (SPTK) API
SPTK.jl
SPTK.jl is a Julia wrapper for the Speech Signal Processing Toolkit (SPTK), which provides a lot of functionalities for speech signal processing such as linear prediction analysis, mel-cepstrum analysis, generalized cepstrum analysis and mel-generalized cepstrum analysis to name a few. See the original project page for more details.
NOTE: SPTK.jl is based on a modified version of SPTK (r9y9/SPTK).
Documentation
A reference manual of the SPTK can be found at http://sp-tk.sourceforge.net/.
Demonstration notebook
- Introduction notebook: a brief intruduction to SPTK.jl, especially focused on mel-generalized cepstrum analysis
Getting started
Functions that SPTK.jl provides are basically same as the SPTK, so if you are new to SPTK, please take a look at the original documentation first and then use SPTK.jl for your need. Also notice that there is no function exported, so you should explicitly import functions if you need.
e.g.
import SPTK: mcep, mgcep, freqt, mc2b
Supported Platforms
- Linux
- Mac OS X
- Windows
Installation
Pkg.add("SPTK")
Related packages
There are a few packages on top of SPTK.jl, which provide more Julia-like interface:
- MelGeneralizedCepstrums.jl: Mel-Generalized Cepstrum analysis
- SynthesisFilters.jl: Speech waveform synthesis filters
Author: r9y9
Source Code: https://github.com/r9y9/SPTK.jl
License: View license
1656604740
Julia toolkit with Fairness Metrics & Bias Mitigation Algorithms
Fairness.jl
Fairness.jl is a comprehensive bias audit and mitigation toolkit in julia. Extensive support and functionality provided by MLJ has been used in this package.
Installation
using Pkg
Pkg.activate("my_environment", shared=true)
Pkg.add("Fairness")
Pkg.add("MLJ")
What Fairness.jl offers over its alternatives?
- As of writing, it is the only bias audit and mitigation toolkit to support data with multi-valued protected attribute. For eg. If the protected attribute, say race has more than 2 values: "Asian", "African", "American"..so on, then Fairness.jl can easily handle it with normal workflow.
- Multiple Fairness algorithms can be applied at the same time by wrapping the wrapped Model. Example is available in Documentation
- Due to the support for multi-valued protected attribute, intersectional fairness can also be dealt with this toolkit. For eg. If the data has 2 protected attributes, say race and gender, then Fairness.jl can be used to handle it by combining the attributes like "female_american", "male_asian"...so on.
- Extensive support and functionality provided by MLJ can be leveraged when using Fairness.jl
- Tuning of models using MLJTuning from MLJ. Numerious ML models from MLJModels can be used together with Fairness.jl
- It leverages the flexibility and speed of Julia to make it more efficient and easy-to-use at the same time
- Well structured and intutive design
- Extensive tests and Documentation
Getting Started
- Documentation is a good starting point for this package.
- To understand Fairness.jl, it is recommended that the user goes through the MLJ Documentation. It shall help the user in understanding the usage of machine, evaluate, etc.
- Incase of any difficulty or confusion feel free to open an issue.
Example
Following is an introductory example of using Fairness.jl. Observe how easy it has become to measure and mitigate bias in Machine Learning algorithms.
using Fairness, MLJ
X, y, ŷ = @load_toydata
julia> model = ConstantClassifier()
ConstantClassifier() @904
julia> wrappedModel = ReweighingSamplingWrapper(classifier=model, grp=:Sex)
ReweighingSamplingWrapper(
grp = :Sex,
classifier = ConstantClassifier(),
factor = 1) @312
julia> evaluate(
wrappedModel,
X, y,
measures=MetricWrappers(
[true_positive, true_positive_rate], grp=:Sex))
┌────────────────────┬─────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────────── ⋯
│ _.measure │ _.measurement │ _.per_fold ⋯
├────────────────────┼─────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────── ⋯
│ true_positive │ Dict{Any,Any}("M" => 2,"overall" => 4,"F" => 2) │ Dict{Any,Any}[Dict("M" => 0,"overall ⋯
│ true_positive_rate │ Dict{Any,Any}("M" => 0.8333333333333334,"overall" => 0.8333333333333334,"F" => 1.0) │ Dict{Any,Any}[Dict("M" => 4.99999999 ⋯
└────────────────────┴─────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────── ⋯
Components
Fairness.jl is divided into following components
FairTensor
It is a 3D matrix of values of TruePositives, False Negatives, etc for each group. It greatly helps in optimization and removing the redundant calculations.
Measures
CalcMetrics
| Name | Metric Instances |
|---|---|
| True Positive | truepositive, true_positive |
| True Negative | truenegative, true_negative |
| False Positive | falsepositive, false_positive |
| False Negative | falsenegative, false_negative |
| True Positive Rate | truepositive_rate, true_positive_rate, tpr, recall, sensitivity, hit_rate |
| True Negative Rate | truenegative_rate, true_negative_rate, tnr, specificity, selectivity |
| False Positive Rate | falsepositive_rate, false_positive_rate, fpr, fallout |
| False Negative Rate | falsenegative_rate, false_negative_rate, fnr, miss_rate |
| False Discovery Rate | falsediscovery_rate, false_discovery_rate, fdr |
| Precision | positivepredictive_value, positive_predictive_value, ppv |
| Negative Predictive Value | negativepredictive_value, negative_predictive_value, npv |
FairMetrics
| Name | Formula | Value for Custom function (func) |
|---|---|---|
| disparity | metric(Gᵢ)/metric(RefGrp) ∀ i | func(metric(Gᵢ), metric(RefGrp)) ∀ i |
| parity | [ (1-ϵ) <= dispariy_value[i] <= 1/(1-ϵ) ∀ i ] | [ func(disparity_value[i]) ∀ i ] |
BoolMetrics [WIP]
These metrics shall use either parity or shall have custom implementation to return boolean values
| Metric | Aliases |
|---|---|
| Demographic Parity | DemographicParity |
Fairness Algorithms
These algorithms are wrappers. These help in mitigating bias and improve fairness.
| Algorithm Name | Metric Optimised | Supports Multi-valued protected attribute | Type | Reference |
|---|---|---|---|---|
| Reweighing | General | :heavy_check_mark: | Preprocessing | Kamiran and Calders, 2012 |
| Reweighing-Sampling | General | :heavy_check_mark: | Preprocessing | Kamiran and Calders, 2012 |
| Equalized Odds Algorithm | Equalized Odds | :heavy_check_mark: | Postprocessing | Hardt et al., 2016 |
| Calibrated Equalized Odds Algorithm | Calibrated Equalized Odds | :x: | Postprocessing | Pleiss et al., 2017 |
| LinProg Algorithm | Any metric | :heavy_check_mark: | Postprocessing | Our own Algorithm |
| Penalty Algorithm | Any metric | :heavy_check_mark: | Inprocessing | Our own Algorithm |
Contributing
- Various Contribution opportunities are available. Some of the possible contributions have been listed at the pinned issue
- Feel free to open an issue or contact on slack. Let us know where your intersts and strengths lie and we can find possible contribution opportunities for you.
Citing Fairness.jl
@software{ashrya_agrawal_2020_3977197,
author = {Ashrya Agrawal and
Jiahao Chen and
Sebastian Vollmer and
Anthony Blaom},
title = {ashryaagr/Fairness.jl},
month = aug,
year = 2020,
publisher = {Zenodo},
version = {v0.1.2},
doi = {10.5281/zenodo.3977197},
url = {https://doi.org/10.5281/zenodo.3977197}
}For an introduction to Fairness.jl refer the notebook available at https://nextjournal.com/ashryaagr/fairness
⚠️ This is experimental software: We refer Aequitas for stable bias auditing.
Author: Ashryaagr
Source Code: https://github.com/ashryaagr/Fairness.jl
License: MIT license
1655818463
DFTK.jl: Density-functional toolkit
Density-functional toolkit
The density-functional toolkit, DFTK for short, is a collection of Julia routines for experimentation with plane-wave density-functional theory (DFT). The unique feature of this code is its emphasis on simplicity and flexibility with the goal of facilitating algorithmic and numerical developments as well as interdisciplinary collaboration in solid-state research.
Having started in 2019 we already support a sizeable set of features. Within the system size currently accessible to our code (ca. 1000 electrons) our performance is of the same order of magnitude as more established packages such as Abinit or Quantum Espresso.
For getting started with DFTK, see our documentation:
Note that at least Julia 1.6 is required.
DFTK summer school 2022
We will organise a summer school centred around the DFTK code and modern numerical approaches to density-functional theory from 29 to 31 August 2022 at Sorbonne Université, Paris. For more details see the school's website.
Support and citation
DFTK is mostly developed as part of academic research. Parts of DFTK have also been discussed in published papers. If you use our code as part of your research, teaching or other activities, we would be grateful if you cite them as appropriate. See the CITATION.bib in the root of this repo for relevant references. The current DFTK reference paper to cite is 
Funding
This project has received funding from the Institute of computing and data sciences (ISCD, Sorbonne Université), École des Ponts ParisTech, Inria Research Centre Paris, RWTH Aachen University, and from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No 810367).
Contributing
If you stumble across issues in using DFTK or have suggestions for future developments we are more than happy to hear about it. In this case please open an issue or contact us (@mfherbst and @antoine-levitt) directly.
Contributions to the code in any form is very welcome, just submit a pull request on github. If you want to contribute but are unsure where to start, take a look at the list of issues tagged good first issue (relatively easy tasks suitable for newcomers) or help wanted (more sizeable but well-defined and isolated). Don't hesitate to ask for help, through github, email or the JuliaMolSim slack.
| Documentation | Build Status | License |
|---|---|---|
 |
Author: JuliaMolSim
Source Code: https://github.com/JuliaMolSim/DFTK.jl
License: MIT license
