Chim: Cross-platform Binary Shims with Optional Remote Fetching

Chim

Cross-platform binary shims with optional remote fetching.

Quickstart (make an automatic fetching node.js wrapper)

Install chim: (see docs for alternate install methods)

$ cargo install chim
$ chim --version
chim 1.1.1

Create a chim in ./bin/node:

#!/usr/bin/env chim
[macos-arm64]
url = 'https://nodejs.org/dist/v18.7.0/node-v18.7.0-darwin-arm64.tar.xz'
path = 'node-v18.7.0-darwin-arm64/bin/node'
checksum = 'ef593cbb3a3f0aae9879b74a7850d794abab26178aa5e0f67ff182894811e6f0'

[linux-x64]
url = 'https://nodejs.org/dist/v18.7.0/node-v18.7.0-linux-x64.tar.xz'
path = 'node-v18.7.0-linux-x64/bin/node'
checksum = '8bc6a1b9deaed2586d726fc62d4bee9c1bfc5a30b96c1c4cff7edd15225a11a2'

[windows-x64]
url = 'https://nodejs.org/dist/v18.7.0/node-v18.7.0-win-x64.zip'
path = 'node-v18.7.0-win-x64\node.exe'
checksum = '9c0abfe32291dd5bed717463cb3590004289f03ab66011e383daa0fcec674683'

Now make it executable and run it:

$ chmod +x ./bin/node
$ ./bin/node -v
v18.7.0

The tarball is fetched once from nodejs.org, extracted into a cache directory, then reused on future ./bin/node calls.

What am I supposed to do with this?

Commit a set of chims into a /bin directory in your project's repo. Other people can just add this directory to their $PATH and won't have to manually install each utility. Also run the chims in CI/CD so you have dev/prod parity with your tools!

See also

• chim.sh - the documentation website
• GitHub (jdxcode/chim) - the repository
• GitHub (jdxcode/chim.sh) - the repository for the documentation
• GitHub Discussions - message board for chim
• GitHub Releases (Changelog) - see the latest releases
• Homebrew Formula - the Homebrew formula
• Crates.io (chim) - the crate on crates.io
• Discord - the chim discord server
• Introductory blog post - a blog post introducing chim

Download Details:

Author: jdxcode
Source Code: https://github.com/jdxcode/chim 
License: MIT license

#rust #cli #binary 

Dark-mode: Control The MacOS Dark Mode From The Command-line

Dark-mode

Control the macOS dark mode from the command-line

Requires macOS 10.10 or later. macOS 10.13 or earlier needs to download the Swift runtime support libraries.

Install

Homebrew

$ brew install dark-mode

npm

$ npm install --global dark-mode-cli

Manually

Download the binary and put it in /usr/local/bin.

Usage

$ dark-mode --help

  Usage
    $ dark-mode [command]

  Commands
    <none>  Toggle dark mode
    on      Enable dark mode
    off     Disable dark mode
    status  Dark mode status

Build

Run ./build. The binary can be found at ./bin/dark-mode.

Building in Xcode works, but it does not export a binary.

Related

• node-dark-mode - Node.js port
• alfred-dark-mode - Alfred workflow

Download Details:

Author: Sindresorhus
Source Code: https://github.com/sindresorhus/dark-mode 
License: MIT license

#swift #macos #cli #homebrew #binary 

BinaryTraits.jl: Can Do Or Not? It's Easy

BinaryTraits.jl

BinaryTraits focuses on usability - traits should be simple to understand and easy to use. For that reason, every trait is binary. An object either has the trait (positive) or does not have the trait (negative).

The design is heavily influenced by the Holy Traits pattern as explained in my Holy Traits book excerpt as well as in Julia manual's trait-based dispatch section. If you think about Holy Traits as the powerful manual transmission, then BinaryTraits gives you automatic transmission. The machinery is the same but it is a lot more pleasant to use.

NOTE: This package is under active development and we may introduce breaking changes. Please follow the issues list if you would like to contribute to the project or have a stake in the design.

Motivation

Just a quick example below. More details can be found in our documentation.

# Use package and import desired positive/negative trait type aliases
using BinaryTraits
using BinaryTraits.Prefix: Can

# Define a trait and its interface contracts
@trait Fly
@implement Can{Fly} by fly(_, destination::Location, speed::Float64)

# Define your data type and implementation
struct Bird end
fly(::Bird, destination::Location, speed::Float64) = "Wohoo! Arrived! 🐦"

# Assign your data type to a trait
@assign Bird with Can{Fly}

# Verify that your implementation is correct
@check(Bird)

# Dispatch for all flying things
@traitfn flap(::Can{Fly}, freq::Float64) = "Flapping wings at $freq Hz"

Main Features

Current features are listed below. Additional features are planned and logged in the issues list.

• Define traits and assigning them to your own data types
• Define composite traits that exhibits all of the underlying traits
• Define interface contracts required for a trait
• Verify that your data type fully implements all interface contracts
• Define traits/interfaces in one module and use them from another module
• Define trait dispatch functions easily

Credits

• Klaus Crusius for his ideas, articulation, and significant contributions to this project
• Jānis Erdmanis for his proposal of a new design based upon parametric types

Related Projects

There are quite a few traits libraries around. If this package isn't for you, take a look at these others:

• Traits.jl
• SimpleTraits.jl
• CanonicalTraits.jl
• TraitWrappers.jl

Download Details:

Author: tk3369
Source Code: https://github.com/tk3369/BinaryTraits.jl
License: MIT license

#julia #binary #interface #language #designpattern 

10 Best Golang Libraries and Tools for Binary Serialization

In today's post we will learn about 10 Best Golang Libraries and tools for Binary Serialization. 

What is binary serialization?

Binary serialization allows modifying private members inside an object and therefore changing the state of it. Because of this, other serialization frameworks, like System. Text. Json, that operate on the public API surface are recommended.

Table of contents:

• Asn1 - Asn.1 BER and DER encoding library for golang.
• Bambam - generator for Cap'n Proto schemas from go.
• Bel - Generate TypeScript interfaces from Go structs/interfaces. Useful for JSON RPC.
• Binstruct - Golang binary decoder for mapping data into the structure.
• Cbor - Small, safe, and easy CBOR encoding and decoding library.
• Colfer - Code generation for the Colfer binary format.
• CSvutil - High Performance, idiomatic CSV record encoding and decoding to native Go structures.
• Elastic - Convert slices, maps or any other unknown value across different types at run-time, no matter what.
• Fixedwidth - Fixed-width text formatting (UTF-8 supported).
• Fwencoder - Fixed width file parser (encoding and decoding library) for Go.
• Go-capnproto - Cap'n Proto library and parser for go.

1 - Asn1:

Asn.1 BER and DER encoding library for golang.

-- import "github.com/PromonLogicalis/asn1"

Package asn1 implements encoding and decoding of ASN.1 data structures using both Basic Encoding Rules (BER) or its subset, the Distinguished Encoding Rules (BER).

This package is highly inspired by the Go standard package "encoding/asn1" while supporting additional features such as BER encoding and decoding and ASN.1 CHOICE types.

By default and for convenience the package uses DER for encoding and BER for decoding. However it's possible to use a Context object to set the desired encoding and decoding rules as well other options.

Restrictions:

• BER allows STRING types, such as OCTET STRING and BIT STRING, to be encoded as constructed types containing inner elements that should be concatenated to form the complete string. The package does not support that, but in the future decoding of constructed strings should be included.

Usage

func Decode

func Decode(data []byte, obj interface{}) (rest []byte, err error)

Decode parses the given BER data into obj. The argument obj should be a reference to the value that will hold the parsed data. Decode uses a default Context and is equivalent to:

rest, err := asn1.NewContext().Decode(data, &obj)

func DecodeWithOptions

func DecodeWithOptions(data []byte, obj interface{}, options string) (rest []byte, err error)

DecodeWithOptions parses the given BER data into obj using the additional options. The argument obj should be a reference to the value that will hold the parsed data. Decode uses a default Context and is equivalent to:

rest, err := asn1.NewContext().DecodeWithOptions(data, &obj, options)

func Encode

func Encode(obj interface{}) (data []byte, err error)

Encode returns the DER encoding of obj. Encode uses a default Context and it's equivalent to:

data, err = asn1.NewContext().Encode(obj)

func EncodeWithOptions

func EncodeWithOptions(obj interface{}, options string) (data []byte, err error)

EncodeWithOptions returns the DER encoding of obj using additional options. EncodeWithOptions uses a default Context and it's equivalent to:

data, err = asn1.NewContext().EncodeWithOptions(obj, options)

type Choice

type Choice struct {
	Type    reflect.Type
	Options string
}

Choice represents one option available for a CHOICE element.

type Context

type Context struct {
}

Context keeps options that affect the ASN.1 encoding and decoding

Use the NewContext() function to create a new Context instance:

ctx := ber.NewContext()
// Set options, ex:
ctx.SetDer(true, true)
// And call decode or encode functions
bytes, err := ctx.EncodeWithOptions(value, "explicit,application,tag:5")
...

func NewContext

func NewContext() *Context

NewContext creates and initializes a new context. The returned Context does not contains any registered choice and it's set to DER encoding and BER decoding.

func (*Context) AddChoice

func (ctx *Context) AddChoice(choice string, entries []Choice) error

AddChoice registers a list of types as options to a given choice.

The string choice refers to a choice name defined into an element via additional options for DecodeWithOptions and EncodeWithOptions of via struct tags.

For example, considering that a field "Value" can be an INTEGER or an OCTET STRING indicating two types of errors, each error with a different tag number, the following can be used:

// Error types
type SimpleError string
type ComplextError string
// The main object
type SomeSequence struct {
	// ...
	Value	interface{}	`asn1:"choice:value"`
	// ...
}
// A Context with the registered choices
ctx := asn1.NewContext()
ctx.AddChoice("value", []asn1.Choice {
	{
		Type: reflect.TypeOf(int(0)),
	},
	{
		Type: reflect.TypeOf(SimpleError("")),
		Options: "tag:1",
	},
	{
		Type: reflect.TypeOf(ComplextError("")),
		Options: "tag:2",
	},
})

View on Github

2 - Bambam:

Generator for Cap'n Proto schemas from go.

bambam: auto-generate capnproto schema from your golang source files.

Adding capnproto serialization to an existing Go project used to mean writing a lot of boilerplate.

Not anymore.

Given a set of golang (Go) source files, bambam will generate a capnproto schema. Even better: bambam will also generate translation functions to readily convert between your golang structs and the new capnproto structs.

prereqs

You'll need a recent (up-to-date) version of go-capnproto. If you installed go-capnproto before, you'll want to update it [>= f9f239fc7f5ad9611cf4e88b10080a4b47c3951d / 16 Nov 2014].

Capnproto and go-capnproto should both be installed and on your PATH.

to install: run make. This lets us record the git commit in LASTGITCOMMITHASH to provide accurate version info. Otherwise you'll get an 'undefined: LASTGITCOMMITHASH' failure.

# be sure go-capnproto and capnpc are installed first.

$ go get -t github.com/glycerine/bambam  # the -t pulls in the test dependencies.

# ignore the initial compile error about 'undefined: LASTGITCOMMITHASH'. `make` will fix that.
$ cd $GOPATH/src/github.com/glycerine/bambam
$ make  # runs tests, build if all successful
$ go install

use

use: bambam -o outdir -p package myGoSourceFile.go myGoSourceFile2.go ...
     # Bambam makes it easy to use Capnproto serialization[1] from Go.
     # Bambam reads .go files and writes a .capnp schema and Go bindings.
     # options:
     #   -o="odir" specifies the directory to write to (created if need be).
     #   -p="main" specifies the package header to write (e.g. main, mypkg).
     #   -X exports private fields of Go structs. Default only maps public fields.
     #   -version   shows build version with git commit hash
     #   -OVERWRITE modify .go files in-place, adding capid tags (write to -o dir by default).
     # required: at least one .go source file for struct definitions. Must be last, after options.
     #
     # [1] https://github.com/glycerine/go-capnproto 

demo

See rw.go.txt. To see all the files compiled together in one project: (a) comment out the defer in the rw_test.go file; (b) run go test; (c) then cd testdir_* and look at the sample project files there. (d). run go build in the testdir_ to rebuild the binary. Notice that you will need all three .go files to successfully build. The two .capnp files should be kept so you can read your data from any capnp-supported language. Here's what is what in that example directory:

rw.go             # your original go source file (in this test)
translateCapn.go  # generated by bambam after reading rw.go
schema.capnp      # generated by bambam after reading rw.go
schema.capnp.go   # generated by `capnpc -ogo schema.capnp` <- you have to do this yourself or in your Makefile.
go.capnp          # always necessary boilerplate to let capnpc work, just copy it from bambam/go.capnp to your build dir.

View on Github

3 - Bel:

Generate TypeScript interfaces from Go structs/interfaces. Useful for JSON RPC.

Getting started

bel is easy to use. There are two steps involved: extract the Typescript information, and generate the Typescript code.

package main

import (
    "github.com/32leaves/bel"
)

type Demo struct {
    Foo string `json:"foo,omitempty"`
    Bar uint32
    Baz struct {
        FirstField  bool
        SecondField *string
    }
}

func main() {
    ts, err := bel.Extract(Demo{})
    if err != nil {
        panic(err)
    }

    err = bel.Render(ts)
    if err != nil {
        panic(err)
    }
}

produces something akin to (sans formatting):

export interface Demo {
    foo?: string
    Bar: number
    Baz: {
        FirstField: boolean
        SecondField: string
    }
}

Converting interfaces

You can also convert Golang interfaces to TypeScript interfaces. This is particularly handy for JSON RPC:

package main

import (
    "os"
    "github.com/32leaves/bel"
)

type DemoService interface {
    SayHello(name, msg string) (string, error)
}

func main() {
    ts, err := bel.Extract((*DemoService)(nil))
    if err != nil {
        panic(err)
    }

    err = bel.Render(ts)
    if err != nil {
        panic(err)
    }
}

produces something akin to (sans formatting):

export interface DemoService {
    SayHello(arg0: string, arg1: string): string
}

View on Github

4 - Binstruct:

Golang binary decoder for mapping data into the structure.

Install

go get -u github.com/ghostiam/binstruct

Examples

ZIP decoder
PNG decoder

Use

For struct

From file or other io.ReadSeeker:

package main

import (
	"encoding/binary"
	"fmt"
	"log"
	"os"

	"github.com/ghostiam/binstruct"
)

func main() {
	file, err := os.Open("testdata/file.bin")
	if err != nil {
		log.Fatal(err)
	}

	type dataStruct struct {
		Arr []int16 `bin:"len:4"`
	}

	var actual dataStruct
	decoder := binstruct.NewDecoder(file, binary.BigEndian)
	// decoder.SetDebug(true) // you can enable the output of bytes read for debugging
	err = decoder.Decode(&actual)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%+v", actual)

	// Output:
	// {Arr:[1 2 3 4]}
}

From bytes

package main

import (
	"fmt"
	"log"
	
	"github.com/ghostiam/binstruct"
)

func main() {
	data := []byte{
		0x00, 0x01,
		0x00, 0x02,
		0x00, 0x03,
		0x00, 0x04,
	}

	type dataStruct struct {
		Arr []int16 `bin:"len:4"`
	}

	var actual dataStruct
	err := binstruct.UnmarshalBE(data, &actual) // UnmarshalLE() or Unmarshal()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%+v", actual)

	// Output: {Arr:[1 2 3 4]}
}

View on Github

5 - Cbor:

Small, safe, and easy CBOR encoding and decoding library.

fxamacker/cbor is a modern CBOR codec in Go. It's like encoding/json for CBOR with time-saving features. It balances security, usability, speed, data size, program size, and other competing factors.

Features include CBOR tags, duplicate map key detection, float64→32→16, and Go struct tags (toarray, keyasint, omitempty). API is close to encoding/json plus predefined CBOR options like Core Deterministic Encoding, Preferred Serialization, CTAP2, etc.

Using CBOR Preferred Serialization with Go struct tags (toarray, keyasint, omitempty) reduces programming effort and creates smaller encoded data size.

Microsoft Corporation had NCC Group produce a security assessment (PDF) which includes portions of this library in its scope.

fxamacker/cbor has 98% coverage and is fuzz tested. It won't exhaust RAM decoding 9 bytes of bad CBOR data. It's used by Arm Ltd., Berlin Institute of Health at Charité, Chainlink, ConsenSys, Dapper Labs, Duo Labs (cisco), EdgeX Foundry, Mozilla, Netherlands (govt), Oasis Labs, Taurus SA, Teleport, and others.

Install with go get github.com/fxamacker/cbor/v2 and import "github.com/fxamacker/cbor/v2".
See Quick Start to save time.

What is CBOR?

CBOR is a concise binary data format inspired by JSON and MessagePack. CBOR is defined in RFC 8949 (December 2020) which obsoletes RFC 7049 (October 2013).

CBOR is an Internet Standard by IETF. It's used in other standards like WebAuthn by W3C, COSE (RFC 8152), CWT (RFC 8392), CDDL (RFC 8610) and more.

Reasons for choosing CBOR vary by project. Some projects replaced protobuf, encoding/json, encoding/gob, etc. with CBOR. For example, by replacing protobuf with CBOR in gRPC.

Why fxamacker/cbor?

fxamacker/cbor balances competing factors such as speed, size, safety, usability, maintainability, and etc.

Killer features include Go struct tags like toarray, keyasint, etc. They reduce encoded data size, improve speed, and reduce programming effort. For example, toarray automatically translates a Go struct to/from a CBOR array.

Modern CBOR features include Core Deterministic Encoding and Preferred Encoding. Other features include CBOR tags, big.Int, float64→32→16, an API like encoding/json, and more.

Security features include the option to detect duplicate map keys and options to set various max limits. And it's designed to make concurrent use of CBOR options easy and free from side-effects.

To prevent crashes, it has been fuzz-tested since before release 1.0 and code coverage is kept above 98%.

For portability and safety, it avoids using unsafe, which makes it portable and protected by Go1's compatibility guidelines.

For performance, it uses safe optimizations. When used properly, fxamacker/cbor can be faster than CBOR codecs that rely on unsafe. However, speed is only one factor and should be considered together with other competing factors.

CBOR Security

fxamacker/cbor is secure. It rejects malformed CBOR data and has an option to detect duplicate map keys. It doesn't crash when decoding bad CBOR data. It has extensive tests, coverage-guided fuzzing, data validation, and avoids Go's unsafe package.

Decoding 9 or 10 bytes of malformed CBOR data shouldn't exhaust memory. For example,
[]byte{0x9B, 0x00, 0x00, 0x42, 0xFA, 0x42, 0xFA, 0x42, 0xFA, 0x42}

*Speed and memory are for latest codec version listed in the row (compiled with Go 1.17.5).

fxamacker/cbor CBOR safety settings include: MaxNestedLevels, MaxArrayElements, MaxMapPairs, and IndefLength.

For more info, see:

• RFC 8949 Section 10 (Security Considerations) or RFC 7049 Section 8.
• Go warning, "Packages that import unsafe may be non-portable and are not protected by the Go 1 compatibility guidelines."

View on Github

6 - Colfer:

Code generation for the Colfer binary format.

Colfer is a binary serialization format optimized for speed and size.

The project's compiler colf(1) generates source code from schema definitions to marshal and unmarshall data structures.

This is free and unencumbered software released into the public domain. The format is inspired by Protocol Buffers.

Language Support

• C, ISO/IEC 9899:2011 compliant a.k.a. C11, C++ compatible
• Go, a.k.a. golang
• Java, Android compatible
• JavaScript, a.k.a. ECMAScript, NodeJS compatible

Features

• Simple and straightforward in use
• No dependencies other than the core library
• Both faster and smaller than the competition
• Robust against malicious input
• Maximum of 127 fields per data structure
• No support for enumerations
• Framed; suitable for concatenation/streaming

TODO's

• Rust and Python support
• Protocol revision

Use

Download a prebuilt compiler or run go get -u github.com/pascaldekloe/colfer/cmd/colf to make one yourself. Homebrew users can also brew install colfer.

The command prints its own manual when invoked without arguments.

NAME
	colf — compile Colfer schemas

SYNOPSIS
	colf [-h]
	colf [-vf] [-b directory] [-p package] \
		[-s expression] [-l expression] C [file ...]
	colf [-vf] [-b directory] [-p package] [-t files] \
		[-s expression] [-l expression] Go [file ...]
	colf [-vf] [-b directory] [-p package] [-t files] \
		[-x class] [-i interfaces] [-c file] \
		[-s expression] [-l expression] Java [file ...]
	colf [-vf] [-b directory] [-p package] \
		[-s expression] [-l expression] JavaScript [file ...]

DESCRIPTION
	The output is source code for either C, Go, Java or JavaScript.

	For each operand that names a file of a type other than
	directory, colf reads the content as schema input. For each
	named directory, colf reads all files with a .colf extension
	within that directory. If no operands are given, the contents of
	the current directory are used.

	A package definition may be spread over several schema files.
	The directory hierarchy of the input is not relevant to the
	generated code.

OPTIONS
  -b directory
    	Use a base directory for the generated code. (default ".")
  -c file
    	Insert a code snippet from a file.
  -f	Normalize the format of all schema input on the fly.
  -h	Prints the manual to standard error.
  -i interfaces
    	Make all generated classes implement one or more interfaces.
    	Use commas as a list separator.
  -l expression
    	Set the default upper limit for the number of elements in a
    	list. The expression is applied to the target language under
    	the name ColferListMax. (default "64 * 1024")
  -p package
    	Compile to a package prefix.
  -s expression
    	Set the default upper limit for serial byte sizes. The
    	expression is applied to the target language under the name
    	ColferSizeMax. (default "16 * 1024 * 1024")
  -t files
    	Supply custom tags with one or more files. Use commas as a list
    	separator. See the TAGS section for details.
  -v	Enable verbose reporting to standard error.
  -x class
    	Make all generated classes extend a super class.

TAGS
	Tags, a.k.a. annotations, are source code additions for structs
	and/or fields. Input for the compiler can be specified with the
	-f option. The data format is line-oriented.

		<line> :≡ <qual> <space> <code> ;
		<qual> :≡ <package> '.' <dest> ;
		<dest> :≡ <struct> | <struct> '.' <field> ;

	Lines starting with a '#' are ignored (as comments). Java output
	can take multiple tag lines for the same struct or field. Each
	code line is applied in order of appearance.

EXIT STATUS
	The command exits 0 on success, 1 on error and 2 when invoked
	without arguments.

EXAMPLES
	Compile ./io.colf with compact limits as C:

		colf -b src -s 2048 -l 96 C io.colf

	Compile ./*.colf with a common parent as Java:

		colf -p com.example.model -x com.example.io.IOBean Java

BUGS
	Report bugs at <https://github.com/pascaldekloe/colfer/issues>.

	Text validation is not part of the marshalling and unmarshalling
	process. C and Go just pass any malformed UTF-8 characters. Java
	and JavaScript replace unmappable content with the '?' character
	(ASCII 63).

SEE ALSO
	protoc(1), flatc(1)

It is recommended to commit the generated source code into the respective version control to preserve build consistency and minimise the need for compiler installations. Alternatively, you may use the Maven plugin.

<plugin>
	<groupId>net.quies.colfer</groupId>
	<artifactId>colfer-maven-plugin</artifactId>
	<version>1.11.2</version>
	<configuration>
		<packagePrefix>com/example</packagePrefix>
	</configuration>
</plugin>

View on Github

7 - CSvutil:

High Performance, idiomatic CSV record encoding and decoding to native Go structures.

Package csvutil provides fast, idiomatic, and dependency free mapping between CSV and Go (golang) values.

This package is not a CSV parser, it is based on the Reader and Writer interfaces which are implemented by eg. std Go (golang) csv package. This gives a possibility of choosing any other CSV writer or reader which may be more performant.

Installation

go get github.com/jszwec/csvutil

Requirements

• Go1.8+

Example

Unmarshal

Nice and easy Unmarshal is using the Go std csv.Reader with its default options. Use Decoder for streaming and more advanced use cases.

var csvInput = []byte(`
name,age,CreatedAt
jacek,26,2012-04-01T15:00:00Z
john,,0001-01-01T00:00:00Z`,
	)

	type User struct {
		Name      string `csv:"name"`
		Age       int    `csv:"age,omitempty"`
		CreatedAt time.Time
	}

	var users []User
	if err := csvutil.Unmarshal(csvInput, &users); err != nil {
		fmt.Println("error:", err)
	}

	for _, u := range users {
		fmt.Printf("%+v\n", u)
	}

	// Output:
	// {Name:jacek Age:26 CreatedAt:2012-04-01 15:00:00 +0000 UTC}
	// {Name:john Age:0 CreatedAt:0001-01-01 00:00:00 +0000 UTC}

Marshal

Marshal is using the Go std csv.Writer with its default options. Use Encoder for streaming or to use a different Writer.

type Address struct {
		City    string
		Country string
	}

	type User struct {
		Name string
		Address
		Age       int `csv:"age,omitempty"`
		CreatedAt time.Time
	}

	users := []User{
		{
			Name:      "John",
			Address:   Address{"Boston", "USA"},
			Age:       26,
			CreatedAt: time.Date(2010, 6, 2, 12, 0, 0, 0, time.UTC),
		},
		{
			Name:    "Alice",
			Address: Address{"SF", "USA"},
		},
	}

	b, err := csvutil.Marshal(users)
	if err != nil {
		fmt.Println("error:", err)
	}
	fmt.Println(string(b))

	// Output:
	// Name,City,Country,age,CreatedAt
	// John,Boston,USA,26,2010-06-02T12:00:00Z
	// Alice,SF,USA,,0001-01-01T00:00:00Z

Unmarshal and metadata

It may happen that your CSV input will not always have the same header. In addition to your base fields you may get extra metadata that you would still like to store. Decoder provides Unused method, which after each call to Decode can report which header indexes were not used during decoding. Based on that, it is possible to handle and store all these extra values.

type User struct {
		Name      string            `csv:"name"`
		City      string            `csv:"city"`
		Age       int               `csv:"age"`
		OtherData map[string]string `csv:"-"`
	}

	csvReader := csv.NewReader(strings.NewReader(`
name,age,city,zip
alice,25,la,90005
bob,30,ny,10005`))

	dec, err := csvutil.NewDecoder(csvReader)
	if err != nil {
		log.Fatal(err)
	}

	header := dec.Header()
	var users []User
	for {
		u := User{OtherData: make(map[string]string)}

		if err := dec.Decode(&u); err == io.EOF {
			break
		} else if err != nil {
			log.Fatal(err)
		}

		for _, i := range dec.Unused() {
			u.OtherData[header[i]] = dec.Record()[i]
		}
		users = append(users, u)
	}

	fmt.Println(users)

	// Output:
	// [{alice la 25 map[zip:90005]} {bob ny 30 map[zip:10005]}]

View on Github

8 - Elastic:

Convert slices, maps or any other unknown value across different types at run-time, no matter what.

Converts go types no matter what

elastic is a simple library that converts any type to another the best way possible. This is useful when the type is only known at run-time, which usually happens when serializing data. elastic allows your code to be flexible regarding type conversion if that is what you're looking for.

It is also capable of seeing through alias types and converting slices and maps to and from other types of slices and maps, providing there is some logical way to convert them.

Default conversion can be overridden by providing custom conversion functions for specific types. Struct types can also implement the ConverterTo interface to help with conversion to and from specific types.

Quick examples:

convert value types:

// note that using elastic wouldn't make sense if you are certain
    // f is a float64 at compile time.
    var f interface{} = float64(5.5)
	var i int
    
    err := elastic.Set(&i, f)
	if err != nil {
		log.Fatal(f)
	}

	fmt.Println(i) // prints 5

convert slices:

var ints []int
	err = elastic.Set(&ints, []interface{}{1, 2, 3, "4", float64(5), 6})
	if err != nil {
		log.Fatal(f)
	}

	fmt.Println(ints) // prints [1 2 3 4 5 6]

convert maps:

someMap := map[string]interface{}{
		"1": "uno",
		"2": "dos",
		"3": "tres",
	}

	intmap := make(map[int]string)
	err = elastic.Set(&intmap, someMap)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(intmap) // prints map[1:uno 2:dos 3:tres]

Simple API:

elastic.Convert()

Converts the passed value to the target type

Syntax:

elastic.Convert(source interface{}, targetType reflect.Type) (interface{}, error)

• source: value to convert
• targetType the type you want to convert source to

Returns

The converted value or an error if it fails.

View on Github

9 - Fixedwidth:

Fixed-width text formatting (UTF-8 supported).

Fixedwidth is a Go package that provides a simple way to define fixed-width data, fast encoding and decoding also is the project's target.

Character encoding supported

UTF-8

Getting Started

Installation

To start using Fixedwidth, run go get:

$ go get github.com/huydang284/fixedwidth

How we limit a struct field

To limit a struct field, we use fixed tag.

Example:

type people struct {
    Name string `fixed:"10"`
    Age  int    `fixed:"3"`
}

If the value of struct field is longer than the limit that we defined, redundant characters will be truncated.

Otherwise, if the value of struct field is less than the limit, additional spaces will be appended.

Encoding

We can use Marshal function directly to encode fixed-width data.

package main

import (
    "fmt"
    "github.com/huydang284/fixedwidth"
)

type people struct {
    Name string `fixed:"10"`
    Age  int    `fixed:"3"`
}

func main() {
    me := people {
        Name: "Huy",
        Age: 25,
    }
    data, _ := fixedwidth.Marshal(me)
    fmt.Println(string(data))
}

The result will be:

Huy       25 

View on Github

10 - Fwencoder:

Fixed width file parser (encoding and decoding library) for Go.

This library is using to parse fixed-width table data like:

Name            Address               Postcode Phone          Credit Limit Birthday
Evan Whitehouse V4560 Camel Back Road 3122     (918) 605-5383    1000000.5 19870101
Chuck Norris    P.O. Box 872          77868    (713) 868-6003     10909300 19651203

Install

To install the library use the following command:

$ go get -u github.com/o1egl/fwencoder

Decoding example

Parsing data from io.Reader:

type Person struct {
	Name        string
	Address     string
	Postcode    int
	Phone       string
	CreditLimit float64   `json:"Credit Limit"`
	Bday        time.Time `column:"Birthday" format:"20060102"`
}

f, _ := os.Open("/path/to/file")
defer f.Close

var people []Person
err := fwencoder.UnmarshalReader(f, &people)

You can also parse data from byte array:

b, _ := ioutil.ReadFile("/path/to/file")
var people []Person
err := fwencoder.Unmarshal(b, &people)

View on Github

Thank you for following this article.

Related videos:

Serialize protobuf message - Golang

#go #golang #binary #serialization 

BSON.jl: Julia Package for Working with The Binary JSON Serialisation

BSON

BSON.jl is a Julia package for working with the Binary JSON serialisation format. It can be used as a general store for Julia data structures, with the following features:

• Lightweight and ubiquitous, with a simple JSON-like data model and clients in many languages.
• Efficient for binary data (eg. arrays of floats).
• Flexible enough to handle anything you throw at it – closures, custom types, circular data structures, etc.
• Backwards compatible, so that if data layout changes old files will still load.

julia> using BSON

julia> bson("test.bson", Dict(:a => [1+2im, 3+4im], :b => "Hello, World!"))

julia> BSON.load("test.bson")
Dict{Symbol,Any} with 2 entries:
  :a => Complex{Int64}[1+2im, 3+4im]
  :b => "Hello, World!"

(Note that the top-level object in BSON is always a Dict{Symbol,Any}).

⚠️ Warning: Loading BSON files is not safe from malicious or erroneously constructed data. Loading BSON files can cause arbitrary code to execute on your machine. Do not load files from unknown or untrusted sources.

There a few utility methods for working with BSON files.

julia> using BSON

julia> bson("test.bson", a = 1, b = 2)

julia> BSON.load("test.bson")
Dict{Symbol,Any} with 2 entries:
  :a => 1
  :b => 2

julia> using BSON: @save, @load

julia> a, b = 1, 2
(1, 2)

julia> @save "test.bson" a b # Same as above

julia> @load "test.bson" a b # Loads `a` and `b` back into the workspace

For external files you can use BSON.parse to load raw BSON data structures without any Julia-specific interpretation. In basic cases, this will look that same, but Julia-specific types will be stored in a more complex format.

julia> BSON.parse("test.bson")
Dict{Symbol,Any} with 2 entries:
  :a => 1
  :b => 2

julia> BSON.parse("test.bson")[:data]
Dict{Symbol,Any} with 4 entries:
  :tag  => "array"
  :type => Dict(:tag=>"datatype",:params=>Any[],:name=>["Core","Int64"])
  :size => [3]
  :data => UInt8[0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00  …  ]

This is also how the data will appear to readers in other languages, should you wish to move data outside of Julia.

Notes

Below is some semi-official documentation on more advanced usage.

Loading custom data types within modules

For packages that use BSON.jl to load data, just writing BSON.load("mydata.bson") will not work with custom data types. Here's a simple example of that for DataFrames.jl:

module A
  using DataFrames, BSON
  d = DataFrame(a = 1:10, b = 5:14)
  bson("data.bson", Dict(:d=>d))
  d2 = BSON.load("data.bson") # this will throw an error
end

In these cases, you can specify the namespace under which to resolve types like so:

d2 = BSON.load("data.bson", @__MODULE__)

This will use the current module's namespace when loading the data. You could also pass any module name as the second argument (though almost all cases will use @__MODULE__). By default, the namespace is Main (i.e. the REPL).

Download Details:

Author: JuliaIO
Source Code: https://github.com/JuliaIO/BSON.jl 
License: View license

#julia #binary #json 

StrPack.jl: Swiss Army Knife for Encoding and Decoding Binary Streams

StrPack: Structured Binary Stream Tools

One of the things I find annoying about MATLAB is dealing with binary data. There's a lot of boilerplate, a lot of fread(), and some weird thing involving a =>.

Enter StrPack. StrPack decodes binary streams to Julia composite types, handling stream endianness and padding bytes for the source ABI along the way. StrPack also encodes instances of Julia composite types to binary streams, setting endianness and adding padding if required to meet an ABI. Along with Julia's IOBuffer type, StrPack can also be used to convert between Julia composite types and buffers suitable for some C function arguments.

StrPack is a serializer/deserializer only in a very limited sense. StrPack only handles Julia's bits types or user types for which read(io, ::UserType) and write(io, data::UserType) have been defined. However, you could use StrPack to build those things.

Example

Let’s create a C library as follows:

struct teststruct {
  int      int1;
  float    float1;
};

void getvalues(struct teststruct* ts)
{
  ts->int1 = 7;
  ts->float1 = 3.7;
}

The getvalues function just fills the two fields with specified values. Compile this as a shared library, which on Linux is achieved with gcc -fPIC teststruct.c -shared -o libteststruct.so.

Let’s also create the Julia analog of this structure:

using StrPack

@struct type TestStruct
    int1::Int32
    float1::Float32
end
#define a convenience constructor for 0.2 versions of Julia (in 0.3 this is automatic)
if VERSION < v"0.3.0-pre"
  TestStruct(i, f) = TestStruct(convert(Int32, i), convert(Float32, f))
end

Note that C’s int corresponds to Int32. Let’s initialize an object of this type:

s = TestStruct(-1, 1.2)

We can pack s into a form suitable to pass as the input to our C function getvalues, which we do in the following way:

iostr = IOBuffer()
pack(iostr, s)

It’s worth seeing what has happened here:

julia> iostr
IOBuffer([0xff, 0xff, 0xff, 0xff, 0x9a, 0x99, 0x99, 0x3f],9)

The first 4 bytes correspond to the Int32 representation of -1, and the last 4 to the Float32 representation of 1.2. In other words, this is just a packed memory buffer encoding s. (There are subtleties such as data alignment, endian status, etc. strpack knows about this stuff, and users who need to control its behavior manually can do so.)

Now we load our library and make a ccall:

const libtest = dlopen("libteststruct")
ccall(dlsym(libtest, :getvalues), Void, (Ptr{Void},), iostr.data)

The C function getvalues stores its output in the buffer we provided as input. We unpack this buffer back into a Julia type:

seek(iostr, 0)   # "rewind" to the beginning of the buffer
s2 = unpack(iostr, TestStruct)

Documentation

More complete documentation can be found on Read The Docs.

WARNING

This package is only semi-maintained. While it has been updated to work without warnings on Julia 1.0, there are no guarantees of correctness beyond the existing (and very limited) package tests. Use at your own risk.

Download Details:

Author: Pao
Source Code: https://github.com/pao/StrPack.jl 
License: MIT license

#julia #binary #stream 

Homebrew.jl: OSX Binary dependency provider for Julia

Unmaintained

Note: This package is unmaintained, all users are strongly encouraged to use JLL packages for their binary needs.

Homebrew.jl (OSX only)

Homebrew.jl sets up a homebrew installation inside your Julia package directory. It uses Homebrew to provide specialized binary packages to satisfy dependencies for other Julia packages, without the need for a compiler or other development tools; it is completely self-sufficient.

Package authors with dependencies that want binaries distributed in this manner should open an issue here for inclusion into the package database.

NOTE: If you have MacPorts installed, and are seeing issues with git or curl complaining about certificates, try to update the the curl and curl-ca-bundle packages before using Homebrew.jl. From the terminal, run:

port selfupdate
port upgrade curl curl-ca-bundle

Usage (Users)

As a user, you ideally shouldn't ever have to use Homebrew directly, short of installing it via Pkg.add("Homebrew"). However, there is a simple to use interface for interacting with the Homebrew package manager:

• Homebrew.add("pkg") will install pkg, note that if you want to install a package from a non-default tap, you can do so via Homebrew.add("user/tap/formula"). An example of this is installing the metis4 formula from the Homebrew/science tap via Homebrew.add("homebrew/science/metis4").
• Homebrew.rm("pkg") will uninstall pkg
• Homebrew.update() will update the available formulae for installation and upgrade installed packages if a newer version is available
• Homebrew.list() will list all installed packages and versions
• Homebrew.installed("pkg") will return a Bool denoting whether or not pkg is installed
• Homebrew.prefix() will return the prefix that all packages are installed to

Usage (Package Authors)

As a package author, the first thing to do is to write/find a Homebrew formula for whatever package you wish to create. The easiest way to tell if the binary will work out-of-the-box is Homebrew.add() it. Formulae from the default homebrew/core tap need no prefix, but if you are installing something from another tap, you need to prefix it with the appropriate tap name. For example, to install metis4 from the homebrew/science tap, you would run Homebrew.add("homebrew/science/metis4"). Programs installed to <prefix>/bin and libraries installed to <prefix>/lib will automatically be availble for run()'ing and dlopen()'ing.

If that doesn't "just work", there may be some special considerations necessary for your piece of software. Open an issue here with a link to your formula and we will discuss what the best approach for your software is. To see examples of formulae we have already included for special usage, peruse the homebrew-juliadeps repository.

To have your Julia package automatically install these precompiled binaries, Homebrew.jl offers a BinDeps provider which can be accessed as Homebrew.HB. Simply declare your dependency on Homebrew.jl via a @osx Homebrew in your REQUIRE files, create a BinDeps library_dependency and state that Homebrew provides that dependency:

using BinDeps
@BinDeps.setup
nettle = library_dependency("nettle", aliases = ["libnettle","libnettle-4-6"])

...
# Wrap in @osx_only to avoid non-OSX users from erroring out
@osx_only begin
    using Homebrew
    provides( Homebrew.HB, "nettle", nettle, os = :Darwin )
end

@BinDeps.install Dict(:nettle => :nettle)

Then, the Homebrew package will automatically download the requisite bottles for any dependencies you state it can provide. This example garnered from the build.jl file from Nettle.jl package.

Why Package Authors should use Homebrew.jl

A common question is why bother with Homebrew formulae and such when a package author could simply compile the .dylib's needed by their package, upload them somewhere and download them to a user's installation somewhere. There are multiple reasons, and although they are individually surmountable Homebrew offers a simpler (and standardized) method of solving many of these problems automatically:

On OSX shared libraries link via full paths. This means that unless you manually alter the path inside of a .dylib or binary to have an @rpath or @executable_path in it, the path will be attempting to point to the exact location on your harddrive that the shared library was found at compile-time. This is not an issue if all libraries linked to are standard system libraries, however as soon as you wish to link to a library in a non-standard location you must alter the paths. Homebrew does this for you automatically, rewriting the paths during installation via install_name_tool. To see the paths embedded in your libraries and executable files, run otool -L <file>.

Dependencies on other libraries are handled gracefully by Homebrew. If your package requires some heavy-weight library such as cairo, glib, etc... Homebrew already has those libraries ready to be installed for you.

Releasing new versions of binaries can be difficult. Homebrew.jl has builtin mechanisms for upgrading all old packages, and even detecting when a binary of the same version number has a new revision (e.g. if an old binary had an error embedded inside it).

Why doesn't this package use my system-wide Homebrew installation?

Some of the formulae in the staticfloat/juliadeps tap are specifically patched to work with Julia. Some of these patches have not (or will not) be merged back into Homebrew mainline, so we don't want to conflict with any packages the user may or may not have installed.

Users can modify Homebrew's internal workings, so it's better to have a known good Homebrew installation than to risk bug reports from users that have unknowingly merged patches into Homebrew that break functionality we require.

If you already have something installed, and it is usable, (e.g. BinDeps can load it and it passes any quick internal tests the Package authors have defined) then Homebrew.jl won't try to install it. BinDeps always checks to see if there is a library in the current load path that satisfies the requirements setup by package authors, and if there is, it doesn't build anything.

Advanced usage

Homebrew.jl provides a convenient wrapper around most of the functionality of Homebrew, however there are rare cases where access to the full suite of brew commands is necessary. To facilitate this, users that are familiar with the brew command set can use Homebrew.brew() to directly feed commands to the brew binary within Homebrew.jl. Example usage:

julia> using Homebrew

julia> Homebrew.brew(`info staticfloat/juliadeps/libgfortran`)
staticfloat/juliadeps/libgfortran: stable 6.2 (bottled)
http://gcc.gnu.org/wiki/GFortran
/Users/sabae/.julia/v0.5/Homebrew/deps/usr/Cellar/libgfortran/6.2 (9 files, 2M) *
  Poured from bottle on 2016-11-21 at 13:14:33
From: https://github.com/staticfloat/homebrew-juliadeps/blob/master/libgfortran.rb
==> Dependencies
Build: gcc ✘

Download Details: 

Author: JuliaPackaging
Source Code: https://github.com/JuliaPackaging/Homebrew.jl 
License: View license

#julia #binary 

CondaBinDeps.jl: Conda BinDeps provider for Julia

CondaBinDeps.jl 

This package, which builds on the Conda.jl package allows one to use conda as a BinDeps binary provider for Julia. While other binary providers like Homebrew.jl, AptGet or WinRPM.jl are platform-specific, CondaBinDeps.jl is a cross-platform alternative. It can also be used without administrator rights, in contrast to the current Linux-based providers.

As such, Conda.jl primary audience is Julia packages developers who have a dependency on some native library.

conda is a package manager which started as the binary package manager for the Anaconda Python distribution, but it also provides arbitrary packages. Instead of the full Anaconda distribution, Conda.jl uses the miniconda Python environment, which only includes conda and its dependencies.

CondaBinDeps.jl is NOT an alternative Julia package manager, nor a way to manage Python installations. It will not use any pre-existing Anaconda or Python installation on your machine.

Basic functionality

You can install this package by running Pkg.add("CondaBinDeps") at the Julia prompt. See the Conda.jl package for information on setting up conda environments, etcetera.

BinDeps integration: using Conda.jl as a package author

CondaBinDeps.jl can be used as a Provider for BinDeps with the Conda.Manager type. You first need to write a conda recipe, and upload the corresponding build to binstar. Then, add CondaBinDeps in your REQUIRE file, and add the following to your deps/build.jl file:

using BinDeps
@BinDeps.setup
netcdf = library_dependency("netcdf", aliases = ["libnetcdf" "libnetcdf4"])

...

using CondaBinDeps
provides(CondaBinDeps.Manager, "libnetcdf", netcdf)

If your dependency is available in another channel than the default one, you should register that channel.

CondaBinDeps.Conda.add_channel("my_channel")
provides(CondaBinDeps.Manager, "libnetcdf", netcdf)

If the binary dependency is only available for some OS, give this information to BinDeps:

provides(CondaBinDeps.Manager, "libnetcdf", netcdf, os=:Linux)

To tell BinDeps to install the package to an environment different from the root environment, use EnvManager.

provides(CondaBinDeps.EnvManager{:my_env}, "libnetcdf", netcdf)

Bugs and suggestions

CondaBinDeps has been tested on Linux, OS X, and Windows. It should work on all these platforms.

Please report any bug or suggestion as a github issue

Download Details: 

Author: JuliaPackaging
Source Code: https://github.com/JuliaPackaging/CondaBinDeps.jl 
License: View license

#julia #binary 

McSema: Framework for Lifting X86, Amd64, Aarch64, Sparc32 and Sparc64

McSema

McSema is an executable lifter. It translates ("lifts") executable binaries from native machine code to LLVM bitcode. LLVM bitcode is an intermediate representation form of a program that was originally created for the retargetable LLVM compiler, but which is also very useful for performing program analysis methods that would not be possible to perform on an executable binary directly.

McSema enables analysts to find and retroactively harden binary programs against security bugs, independently validate vendor source code, and generate application tests with high code coverage. McSema isn’t just for static analysis. The lifted LLVM bitcode can also be fuzzed with libFuzzer, an LLVM-based instrumented fuzzer that would otherwise require the target source code. The lifted bitcode can even be compiled back into a runnable program! This is a procedure known as static binary rewriting, binary translation, or binary recompilation.

McSema supports lifting both Linux (ELF) and Windows (PE) executables, and understands most x86 and amd64 instructions, including integer, X87, MMX, SSE and AVX operations. AARCH64 (ARMv8) instruction support is in active development.

Using McSema is a two-step process: control flow recovery, and instruction translation. Control flow recovery is performed using the mcsema-disass tool, which relies on IDA Pro to disassemble a binary file and produce a control flow graph. Instruction translation is then performed using the mcsema-lift tool, which converts the control flow graph into LLVM bitcode. Under the hood, the instruction translation capability of mcsema-lift is implemented in the remill library. The development of remill was a result of refactoring and improvements to McSema, and was first introduced with McSema version 2.0.0. Read more about remill here.

McSema and remill were developed and are maintained by Trail of Bits, funded by and used in research for DARPA and the US Department of Defense.

Features

• Lifts 32- and 64-bit Linux ELF and Windows PE binaries to bitcode, including executables and shared libraries for each platform.
• Supports a large subset of x86 and x86-64 instructions, including most integer, X87, MMX, SSE, and AVX operations.
• Supports a large subset of AArch64, SPARCv8+ (SPARC32), and SPARCv9 (SPARC64) instuctions.
• McSema runs on Windows and Linux and has been tested on Windows 7, 10, Ubuntu (14.04, 16.04, 18.04), and openSUSE.
• McSema can cross-lift: it can translate Linux binaries on Windows, or Windows binaries on Linux.
• Output bitcode is compatible with the LLVM toolchain (versions 3.5 and up).
• Translated bitcode can be analyzed or recompiled as a new, working executable with functionality identical to the original.

Use-cases

Why would anyone translate binaries back to bitcode?

Binary Patching And Modification. Lifting to LLVM IR lets you cleanly modify the target program. You can run obfuscation or hardening passes, add features, remove features, rewrite features, or even fix that pesky typo, grammatical error, or insane logic. When done, your new creation can be recompiled to a new binary sporting all those changes. In the Cyber Grand Challenge, we were able to use McSema to translate challenge binaries to bitcode, insert memory safety checks, and then re-emit working binaries.

Symbolic Execution with KLEE. KLEE operates on LLVM bitcode, usually generated by providing source to the LLVM toolchain. McSema can lift a binary to LLVM bitcode, permitting KLEE to operate on previously unavailable targets. See our walkthrough showing how to run KLEE on a symbolic maze.

Re-use existing LLVM-based tools. KLEE is not the only tool that becomes available for use on bitcode. It is possible to run LLVM optimization passes and other LLVM-based tools like libFuzzer on lifted bitcode.

Analyze the binary rather than the source. Source level analysis is great but not always possible (e.g. you don't have the source) and, even when it is available, it lacks compiler transformations, re-ordering, and optimizations. Analyzing the actual binary guarantees that you're analyzing the true executed behavior.

Write one set of analysis tools. Lifting to LLVM IR means that one set of analysis tools can work on both the source and the binary. Maintaining a single set of tools saves development time and effort, and allows for a single set of better tools.

Comparison with other machine code to LLVM bitcode lifters

Note: We label some architectures as "S2E" to mean any architecture supported by the S2E system. A system using "McSema" for control-flow recovery (e.g. RevGen) uses McSema's CFG.proto format for recovering control-flow. In the case of RevGen, only bitcode produced from DARPA Cyber Grand Challenge (CGC) binaries is executable.

Dependencies

• DynInst support is optional if you use the experimental DynInst disassembler. Note: We do not provide support for the DynInst disassembler.

Getting and building the code

Docker

Step 1: Clone the repository

git clone https://github.com/lifting-bits/mcsema
cd mcsema

Step 2: Add your disassembler to the Dockerfile

Currently IDA is the only supported frontend for control-flow recovery, it's left as an exercise to the reader to install your disassembler of choice. Experimental support for DynInst is available but may be buggy and sometimes get out of date, as we do not officially support it. DynInst support is provided as an exemplar of how to make a third-party disassembler.

Step 3: Build & Run Dockerfile

This will build the container for you and run it with your local directory mounted into the container (at /mcsema/local) such that your work in the container is saved locally:

# Build McSema container
ARCH=amd64; UBUNTU=18.04; LLVM=9; docker build . \
  -t mcsema:llvm${LLVM}-ubuntu${UBUNTU}-${ARCH} \
  -f Dockerfile \
  --build-arg UBUNTU_VERSION=${UBUNTU} \
  --build-arg LLVM_VERSION=${LLVM} \
  --build-arg ARCH=${ARCH}

# Run McSema container lifter
docker run --rm -it --ipc=host -v "$(pwd)":/mcsema/local mcsema:llvm${LLVM}-ubuntu${UBUNTU}-${ARCH}

# Run McSema container disassembler
docker run --rm -it --entrypoint=mcsema-disass --ipc=host -v "$(pwd)":/mcsema/local mcsema:llvm${LLVM}-ubuntu${UBUNTU}-${ARCH}

Native Build

Linux pre-requisites

Native builds on Linux are supported for Ubuntu 18.04 and 20.04. We only support LTS Ubuntu releases.

sudo apt-get update
sudo apt-get upgrade

sudo apt-get install \
     git \
     curl \
     cmake \
     python3 python3-pip python3-virtualenv \
     wget \
     xz-utils pixz \
     clang \
     rpm \
     build-essential \
     gcc-multilib g++-multilib \
     libtinfo-dev \
     lsb-release \
     zip \
     zlib1g-dev \
     ccache

macOS pre-requisites

Download and install the Homebrew package manager.

Make sure to download XCode from the App Store if you don't have it. After downloading XCode, make sure to open it at least once, as it might install more stuff.

brew update
xcode-select --install 2>&1 > /dev/null
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
brew install coreutils ccache
pip3 install requests

Double check that you have the correct Clang installed. You should see something like this:

% clang -v
Apple clang version 12.0.0 (clang-1200.0.32.21)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Step 1 (Optional): Create a virtualenv for your McSema installation

Using a virtualenv ensures that your McSema installation does not interfere with other software packages. This setup is especially helpful if you are hacking on McSema and want to avoid clobbering a global, working version with development code.

mkdir mcsema-ve
virtualenv mcsema-ve
cd mcsema-ve
source bin/activate

Step 2: Clone the repository and its dependencies

git clone https://github.com/lifting-bits/remill.git
pushd .
cd remill

# For latest LLVM versions (>=12)
git checkout -b release_93aba7c 93aba7c

# OR

# For LLVM versions (<=11)
git checkout -b all_llvm 9006baf7db


popd

After which;

git clone --depth 1 --single-branch --branch master https://github.com/lifting-bits/mcsema.git

# Get a compatible anvill version
git clone --branch master https://github.com/lifting-bits/anvill.git
( cd anvill && git checkout -b release_bc3183b bc3183b )

export CC="$(which clang)"
export CXX="$(which clang++)"


# Download cxx-common, build Remill. 
./remill/scripts/build.sh --llvm-version 11 --download-dir ./
pushd remill-build
sudo cmake --build . --target install
popd

# Build and install Anvill
mkdir anvill-build
pushd anvill-build
# Set VCPKG_ROOT to whatever directory the remill script downloaded
cmake -DVCPKG_ROOT=$(pwd)/../vcpkg_ubuntu-20.04_llvm-11_amd64 ../anvill
sudo cmake --build . --target install
popd

# Build and install McSema
mkdir mcsema-build
pushd mcsema-build
# Set VCPKG_ROOT to whatever directory the remill script downloaded
cmake -DVCPKG_ROOT=$(pwd)/../vcpkg_ubuntu-20.04_llvm-11_amd64 ../mcsema
sudo cmake --build . --target install

Once installed, you may use mcsema-disass for disassembling binaries, and mcsema-lift-9.0 for lifting the disassembled binaries. If you specified --llvm-version 9 to the build.sh script, then you would use mcsema-lift-9.0.

Step 3: Verifying Your McSema Installation

Step 2 specified --llvm-version 9 to Remill's build.sh script. This means that Remill, Anvill, and McSema have all been built against a copy of LLVM 9. To enable you to use multiple LLVM versions simultaneously, we suffix our binaries with the LLVM version. Thus, you may use mcsema-lift-9.0 to lift to LLVM 9 bitcode.

Try running mcsema-lift-9.0 --version to see if McSema has been installed.

Run the integration tests

In order to verify that McSema works correctly as built, head on over to the documentation on integration tests. Check that you can run the tests and that they pass.

On Windows (Experimental, may not work)

Step 1: Installing the toolchain

Visual Studio

1. Click on "Tools for Visual Studio 2019" and download the "Build Tools for Visual Studio 2019" installer from the Visual Studio downloads page
2. Select "MSVC v142 - VS 2019 C++ x64/x86 build tools" and confirm the installation

LLVM

1. Get the LLVM 9 (x64) installer from the LLVM download page: http://releases.llvm.org
2. Do NOT enable "Add to PATH"

Python

1. Get the latest Python 3 (X64) installer from the official download page: https://www.python.org/downloads/windows/
2. Enable "Add to PATH"

CMake

1. Download the CMake (x64) installer from https://cmake.org/download
2. Enable "Add to PATH"

Step 2: Obtaining the source code

git clone https://github.com/lifting-bits/remill.git --depth=1
git clone https://github.com/lifting-bits/mcsema.git --depth=1 remill/tools/mcsema

Note that for production usage you should always use a specific remill commit (remill/tools/mcsema/.remill_commit_id) when building McSema. At the time of writing, it is however best to use HEAD (or at least make sure that commit e7795be is present in the remill branch).

cd remill
git fetch --unshallow
git checkout -b production <commit>

Step 3: Enabling the LLVM toolchain for Visual Studio

Download the official extension from the market place: https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.llvm-toolchain

Automatic installation

Only works for the full Visual Studio IDE. Double clicking the extension should automatically install it.

Manual installation

The extension is in fact a ZIP archive; extract it and copy the VCTargets folder to the right location.

• Full Visual Studio: C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\VC\VCTargets
• Visual Studio Build Tools: C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160

Step 4: Dependencies

Its time to fetch library dependencies. You can either build them yourself using our cxx-common dependency manager or download a pre-built package.

There are two versions of LLVM used by Remill and McSema. One version (currently 7.0.1) builds remill and McSema. Another version (currently 5.0.1) is used to build the translation semantics.

On Windows, only the LLVM 5.0.1 package is supported for building semantics. If you build it yourself, use the Visual Studio 2017 Win64 generator with the LLVM 5.0.1 toolchain. The cxx-common script will automatically take care of this requirement.

Binaries (extract to C:\Projects\tob_libraries)

• LLVM 5

Step 5: Building

Make sure to always execute the vcvars64.bat script from the "x64 Native Tools Command Prompt": C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Auxiliary\Build\vcvars64.bat.

mkdir remill_build
cd remill_build

cmake -G "Visual Studio 16 2019" -T llvm -A x64 -DCMAKE_BUILD_TYPE=Release -DCXX_COMMON_REPOSITORY_ROOT=C:\Projects\tob_libraries -DCMAKE_INSTALL_PREFIX=C:\ ..\remill
cmake --build . --config Release -- /maxcpucount:%NUMBER_OF_PROCESSORS%

If you are using a recent CMake version (> 3.13) you can also use the newly introduced cross-platform -j parameter:

cmake --build . --config Release -j %NUMBER_OF_PROCESSORS%

Step 6: Installing

cmake --build . --config Release --target install

You should now have the following directories: C:\mcsema, C:\remill.

Step 7: Running McSema

Add the McSema python package to Python

Make extra sure it only contains ASCII characters with no newlines! The following command should work fine under cmd:

echo|set /p="C:\mcsema\Lib\site-packages" > "C:\Python3<version>\Lib\site-packages\mcsema.pth"

Install the libmagic DLL

pip install python-magic-bin

Update the PATH (cmd)

set PATH=%PATH%;C:\remill\bin;C:\mcsema\bin;C:\mcsema\Scripts

Update the PATH (PowerShell)

$env:PATH+="C:\remill\bin;C:\mcsema\bin;C:\mcsema\Scripts"

Getting help

If you are experiencing problems with McSema or just want to learn more and contribute, join the #binary-lifting channel of the Empire Hacking Slack. Alternatively, you can join our mailing list at mcsema-dev@googlegroups.com or email us privately at mcsema@trailofbits.com.

FAQ

How do you pronounce McSema and where did the name come from

This is a hotly contested issue. We must explore the etymology of the name to find an answer. The "Mc" in McSema was originally a contraction of the words "Machine Code," and the "sema" is short for "semantics." At that time, McSema used LLVM's instruction decoder to take machine code bytes, and turn them into llvm::MCInst data structures. It is possible that "MC" in that case is pronounced em-see. Alas, even those who understand the origin of the name pronounce it as if it were related to America's favorite fast food joint.

Why do I need IDA Pro to use McSema

IDA Pro is an excellent disassembler, and in our experience, it has been the most reliable disassembler to use in McSema. The way in which IDA Pro exposes information about cross-references closely matches how McSema stores that information in its CFG file, which is convenient. We also feature an experimental, not officially supported DynInst disassembler frontend. This frontent exists mostly to support open-source uses cases, but is not actively maintained, and may be out of date. The Dyninst frontend is a good example of how to make a new frontend.

What is Remill, and why does McSema need it

Remill is a library that McSema uses to lift individual machine code instructions to LLVM IR. You can think of McSema being to Remill as Clang is to LLVM. Remill's scope is small: it focuses on instruction semantics only, and it provides semantics for x86, x86-64, and AArch64 instruction semantics. McSema's scope is much bigger: it focuses on lifting entire programs. To do so, McSema must lift the individual instructions, but there's a lot more to lifting programs than just the instructions; there are code and data cross-references, segments, etc.

I'm a student and I'd like to contribute to McSema: how can I help

We would love to take you on as an intern to help improve McSema. We have several project ideas labelled intern project, as well as having smaller scale to-dos labelled under good first issue and help wanted on our issue tracker. You are not limited to those items: if you think of a great feature you want in McSema, let us know and we will sponsor it. Simply contact us on our Slack channel or via mcsema@trailofbits.com and let us know what you'd want to work on and why.

Author: lifting-bits
Source Code: https://github.com/lifting-bits/mcsema
License: AGPL-3.0 License

#cpluplus 

The Jakstab Static Analysis Platform for Binaries

Jakstab

Overview

Jakstab is an Abstract Interpretation-based, integrated disassembly and static analysis framework for designing analyses on executables and recovering reliable control flow graphs. It is designed to be adaptable to multiple hardware platforms using customized instruction decoding and processor specifications. It is written in Java, and in its current state supports x86 processors and 32-bit Windows PE or Linux ELF executables.

Jakstab translates machine code to a low level intermediate language on the fly as it performs data flow analysis on the growing control flow graph. Data flow information is used to resolve branch targets and discover new code locations. Other analyses can either be implemented in Jakstab to run together with the main control flow reconstruction to improve precision of the disassembly, or they can work on the resulting preprocessed control flow graph.

The most detailed description of the entire system so far is contained in Johannes Kinder`s dissertation:

• Johannes Kinder: Static Analysis of x86 Executables. Technische Universität Darmstadt, 2010. PDF

Running Jakstab

Jakstab is invoked via the command line, it comes with both a Windows and a Unix shell script for setting the correct classpath. The package contains a set of examples for unit testing, you can try it out on those by running

jakstab -m input/helloworld.exe

for a default Bounded Address Tracking run on the helloworld executable, or by running

jakstab -m input/jumptable.exe --cpa xfi

for analyzing a jumptable example with Bounded Address Tracking, forward expression substitution, and interval analysis. It is still a research prototype, so all interfaces are likely to change with new versions without further notice. Documentation is still sparse, but will hopefully improve over time.

Outputs

After finishing analysis, Jakstab creates the following files:

• filename_jak.asm - A disassembly with all reachable instructions
• filename_cfa.dot - A CFG in the intermediate language, instruction by instruction
• filename_asmcfg.dot - A CFG of assembly instructions, in basic blocks

Supported Analyses

The analyses (CPAs) that should be working correctly are:

• Bounded Address Tracking (x) (see FMCAD'10)
• VPC-lifted Bounded Address Tracking (v) (see WCRE'12)
• Constant Propagation (c)
• Forward Expression Substitution (f)
• Interval Analysis (i)
• K-Set Analysis (k)

Publications

The following publications, sorted chronologically, describe specific aspects of Jakstab, or applications and extensions of it.

The CAV 2008 tool paper describes an early implementation of Jakstab, which was based on iterative constant propagation and branch resolution:

• Johannes Kinder, Helmut Veith. Jakstab: A Static Analysis Platform for Binaries. In Proceedings of the 20th International Conference on Computer Aided Verification (CAV 2008), vol. 5123, Lecture Notes in Computer Science, Springer, July 2008, pp. 423–427.

Our VMCAI 2009 paper introduces a generic framework for disassembly and control flow reconstruction guided by data flow analysis and defines the theoretical background for Jakstab. The framework is not fixed in its choice of domain, but allows to combine control flow reconstruction with any data flow analysis that provides abstract evaluation of expressions:

• Johannes Kinder, Helmut Veith, Florian Zuleger. An Abstract Interpretation-Based Framework for Control Flow Reconstruction from Binaries. In Proceedings of the 10th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI

2009), vol. 5403, Lecture Notes in Computer Science, Springer, January 2009, pp. 214–228.

In FMCAD 2010, we give an overview on the Jakstab architecture and describe Bounded Address Tracking, a practical abstract domain used for control flow reconstruction and verification of API usage specifications on device driver binaries:

• Johannes Kinder, Helmut Veith. Precise Static Analysis of Untrusted Driver Binaries. In Proceedings of the 10th International Conference on Formal Methods in Computer-Aided Design (FMCAD 2010), October 2010, pp. 43–50.

In our paper at VMCAI 2012, we give a reformulation of control flow reconstruction using parameterized semantics, and show how it can be extended to accomodate under-approximations derived from concrete execution traces. A prototype implementation shows that under-approximations allow to reconstruct useful CFGs when the over-approximation would have to conservatively over-approximate indirect jump targets.

• Johannes Kinder, Dmitry Kravchenko. Alternating Control Flow Reconstruction. In Proceedings of the 13th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI

2012), vol. 7148, Lecture Notes in Computer Science, Springer, January 2012, pp. 267-282.

The WCRE 2012 paper proposes a method for using Jakstab to analyze binaries that have been protected using virtualization-obfuscation.

• Johannes Kinder. Towards Static Analysis of Virtualization-Obfuscated Binaries. In Proceedings of the 19th Working Conference on Reverse Engineering (WCRE 2012), IEEE, October 2012.

Author: jkinder
Source Code: https://github.com/jkinder/jakstab
License: GPL-2.0 License

#binary #java 

CWE Checker: Finds Vulnerable Patterns in Binary Executables

cwe_checker

What is cwe_checker?

cwe_checker is a suite of checks to detect common bug classes such as use of dangerous functions and simple integer overflows. These bug classes are formally known as Common Weakness Enumerations (CWEs). Its main goal is to aid analysts to quickly find vulnerable code paths.

Its main focus are ELF binaries that are commonly found on Linux and Unix operating systems. The cwe_checker uses Ghidra to disassemble binaries into one common intermediate representation and implements its own analyses on this IR. Hence, the analyses can be run on all CPU architectures that Ghidra can disassemble, which makes the cwe_checker a valuable tool for firmware analysis.

The following arguments should convince you to give cwe_checker a try:

• it is very easy to set up, just build the Docker container!
• it analyzes ELF binaries of several CPU architectures including x86, ARM, MIPS, and PPC
• it is extensible due to its plugin-based architecture
• it is configureable, e.g. apply analyses to new APIs
• view results annotated in Ghidra
• cwe_checker can be integrated as a plugin into FACT

Installation

Using the docker image

The simplest way is to pull the latest Docker image from dockerhub:

• docker pull fkiecad/cwe_checker:latest yields an image based on the current master branch.
• docker pull fkiecad/cwe_checker:stable yields an image based on the latest stable release version.

If you want to build the docker image yourself, just run docker build -t cwe_checker .

Local installation

The following dependencies must be installed in order to build and install the cwe_checker locally:

• Rust >= 1.57
• Ghidra >= 9.2

Run make all GHIDRA_PATH=/path/to/ghidra_folder (with the correct path to the local Ghidra installation inserted) to compile and install the cwe_checker.

Usage

The cwe_checker takes a binary as input, runs several checks based on static analysis on the binary and then outputs a list of CWE warnings that have been found during the analysis.

If you use the official docker image, just run

docker run --rm -v /PATH/TO/BINARY:/input fkiecad/cwe_checker /input

If you installed the cwe_checker locally, run

cwe_checker BINARY

You can adjust the behavior of most checks via a configuration file located at src/config.json. If you modify it, add the command line flag --config=src/config.json to tell the cwe_checker to use the modified file. For information about other available command line flags you can pass the --help flag to the cwe_checker.

If you use the stable version, you can also look at the online documentation for more information.

For Bare-Metal Binaries

The cwe_checker offers experimental support for analyzing bare-metal binaries. For that one needs to provide a bare metal configuration file via the --bare-metal-config command line option. An example for such a configuration file can be found at bare_metal/stm32f407vg.json (which was created and tested for an STM32F407VG MCU).

For more information build and read the documentation locally via make documentation. Note that this analysis mode is not yet included in the stable version of the cwe_checker.

Documentation and Tests

The test binaries for our test suite can be built with make compile_test_files (needs Docker to be installed!). The test suite can then be run with make test.

Source code documentation can be built with make documentation. For the stable version, the documentation can be found here.

Implemented Checks

So far the following analyses are implemented:

• CWE-78: OS Command Injection (currently disabled on standard runs)
• CWE-119 and its variants CWE-125 and CWE-787: Buffer Overflow
• CWE-134: Use of Externally-Controlled Format String
• CWE-190: Integer Overflow or Wraparound
• CWE-215: Information Exposure Through Debug Information
• CWE-243: Creation of chroot Jail Without Changing Working Directory
• CWE-332: Insufficient Entropy in PRNG
• CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition
• CWE-415: Double Free
• CWE-416: Use After Free
• CWE-426: Untrusted Search Path
• CWE-467: Use of sizeof() on a Pointer Type
• CWE-476: NULL Pointer Dereference
• CWE-560: Use of umask() with chmod-style Argument
• CWE-676: Use of Potentially Dangerous Function
• CWE-782: Exposed IOCTL with Insufficient Access Control

Please note that some of the above analyses are only partially implemented at the moment. Furthermore, both false positives and false negatives are to be expected due to shortcuts and the nature of static analysis as well as over-approximation.

Integration into other tools

cwe_checker comes with a script for Ghidra, which parses the output of the cwe_checker and annotates the found CWEs in the disassembler for easier manual analysis. The script is located at ghidra_plugin/cwe_checker_ghidra_plugin.py, usage instructions are contained in the file.

How does cwe_checker work internally?

Building the documentation using cargo doc --open --document-private-items will give you more information about the internal structure of the cwe_checker. However, the best documentation is still the source code itself. If you have questions, be sure to ask them on our discussions page! We are constantly striving to improve extensibility and documentation and your questions will help us to achieve that!

To get a quick/initial overview of its internals you can also look at the slides of conference presentations on the cwe_checker in the doc folder. We presented cwe_checker at the following conferences so far:

• Pass The SALT 2019 (slides)
• Black Hat USA 2019 (slides)

Contribute

Contributions are always welcome. Just fork it and open a pull request!

Acknowledgements

This project is partly financed by German Federal Office for Information Security (BSI).

A special thanks goes out to the BAP community (especially the official gitter) for answering questions and discussing solutions.

License

    Copyright (C) 2018 -       Fraunhofer FKIE  (firmware-security@fkie.fraunhofer.de)

    This library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Library General Public
    License as published by the Free Software Foundation; either
    version 3 of the License, or (at your option) any later version.

    This library is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    Library General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

Author: fkie-cad
Source Code: https://github.com/fkie-cad/cwe_checker
License: LGPL-3.0 License

#binary 

Bloaty: A Size Profiler For Binaries

Bloaty: a size profiler for binaries

Ever wondered what's making your binary big? Bloaty will show you a size profile of the binary so you can understand what's taking up space inside.

$ ./bloaty bloaty -d compileunits
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  34.8%  10.2Mi  43.4%  2.91Mi    [163 Others]
  17.2%  5.08Mi   4.3%   295Ki    third_party/protobuf/src/google/protobuf/descriptor.cc
   7.3%  2.14Mi   2.6%   179Ki    third_party/protobuf/src/google/protobuf/descriptor.pb.cc
   4.6%  1.36Mi   1.1%  78.4Ki    third_party/protobuf/src/google/protobuf/text_format.cc
   3.7%  1.10Mi   4.5%   311Ki    third_party/capstone/arch/ARM/ARMDisassembler.c
   1.3%   399Ki  15.9%  1.07Mi    third_party/capstone/arch/M68K/M68KDisassembler.c
   3.2%   980Ki   1.1%  75.3Ki    third_party/protobuf/src/google/protobuf/generated_message_reflection.cc
   3.2%   965Ki   0.6%  40.7Ki    third_party/protobuf/src/google/protobuf/descriptor_database.cc
   2.8%   854Ki  12.0%   819Ki    third_party/capstone/arch/X86/X86Mapping.c
   2.8%   846Ki   1.0%  66.4Ki    third_party/protobuf/src/google/protobuf/extension_set.cc
   2.7%   800Ki   0.6%  41.2Ki    third_party/protobuf/src/google/protobuf/generated_message_util.cc
   2.3%   709Ki   0.7%  50.7Ki    third_party/protobuf/src/google/protobuf/wire_format.cc
   2.1%   637Ki   1.7%   117Ki    third_party/demumble/third_party/libcxxabi/cxa_demangle.cpp
   1.8%   549Ki   1.7%   114Ki    src/bloaty.cc
   1.7%   503Ki   0.7%  48.1Ki    third_party/protobuf/src/google/protobuf/repeated_field.cc
   1.6%   469Ki   6.2%   427Ki    third_party/capstone/arch/X86/X86DisassemblerDecoder.c
   1.4%   434Ki   0.2%  15.9Ki    third_party/protobuf/src/google/protobuf/message.cc
   1.4%   422Ki   0.3%  23.4Ki    third_party/re2/re2/dfa.cc
   1.3%   407Ki   0.4%  24.9Ki    third_party/re2/re2/regexp.cc
   1.3%   407Ki   0.4%  29.9Ki    third_party/protobuf/src/google/protobuf/map_field.cc
   1.3%   397Ki   0.4%  24.8Ki    third_party/re2/re2/re2.cc
 100.0%  29.5Mi 100.0%  6.69Mi    TOTAL

Bloaty performs a deep analysis of the binary. Using custom ELF, DWARF, and Mach-O parsers, Bloaty aims to accurately attribute every byte of the binary to the symbol or compileunit that produced it. It will even disassemble the binary looking for references to anonymous data.

Bloaty supports many features:

• file formats: ELF, Mach-O, PE/COFF (experimental), WebAssembly (experimental)
• data sources: compileunit (shown above), symbol, section, segment, etc.
• hierarchical profiles: combine multiple data sources into a single report
• size diffs: see where the binary grew, perfect for CI tests
• separate debug files: strip the binary under test, while making debug data available for analysis
• flexible demangling: demangle C++ symbols, optionally discarding function/template parameters
• custom data sources: regex rewrites of built-in data sources, for custom munging/bucketing
• regex filtering: filter out parts of the binary that do or don't match a given regex
• easy to deploy: statically-linked C++ binary, easy to copy around

For detailed info on all of Bloaty's features, see the User Documentation.

For more information about the analysis performed by Bloaty, please see How Bloaty Works.

Install

To build, use cmake. For example:

$ cmake -B build -G Ninja -S .
$ cmake --build build
$ cmake --build build --target install

Bloaty bundles libprotobuf, re2, capstone, and pkg-config as Git submodules, and uses protoc build from libprotobuf, but it will prefer the system's versions of those dependencies if available. All other dependencies are included as Git submodules.

If the Git repository hasn't been cloned with the --recursive, the submodules can be checked out with:

$ git submodule update --init --recursive

To run the tests, see the info in tests/README.md.

Support

GitHub issues and PRs welcome. Please include tests when possible, see: tests/README.md.

This is not an official Google product.

Author: google
Source Code: https://github.com/google/bloaty
License: Apache-2.0 License

#google #binary 

BinSkim: A Binary Static Analysis Tool

BinSkim Binary Analyzer

This repository contains the source code for BinSkim, a Portable Executable (PE) light-weight scanner that validates compiler/linker settings and other security-relevant binary characteristics.

For Developers

1. Fork the repository -- Need Help?
2. Load and compile src\BinSkim.sln to develop changes for contribution.
3. Execute BuildAndTest.cmd at the root of the enlistment to validate before submitting a PR.

Submit Pull Requests

1. Run BuildAndTest.cmd at the root of the enlistment to ensure that all tests pass, release build succeeds, and NuGet packages are created
2. Submit a Pull Request to the 'develop' branch -- Need Help?

For Users

1. Download BinSkim from NuGet
2. Read the User Guide
3. Find out more about the Static Analysis Results Interchange Format (SARIF) used to output Binskim results

How to extract the exe file from the nuget package

If you only want to run the Binskim tool without installing anything, then you can

1. Download BinSkim from NuGet
2. Rename the file extension from .nupkg to .zip
3. Unzip
4. Executable files are now available in the folder tools\netcoreapp3.1

Command-Line Quick Guide

Example: binskim.exe analyze c:\bld\*.dll --recurse --output MyRun.sarif

Author: Microsoft
Source Code: https://github.com/Microsoft/binskim
License: View license

#binary #analyzer 

Binstruct: Golang Binary Decoder for Mapping Data into The Structure

binstruct

Golang binary decoder to structure

Install

go get -u github.com/ghostiam/binstruct

Use

For struct

From file or other io.ReadSeeker:

package main

import (
    "encoding/binary"
    "fmt"
    "log"
    "os"

    "github.com/ghostiam/binstruct"
)

func main() {
    file, err := os.Open("testdata/file.bin")
    if err != nil {
        log.Fatal(err)
    }

    type dataStruct struct {
        Arr []int16 `bin:"len:4"`
    }

    var actual dataStruct
    decoder := binstruct.NewDecoder(file, binary.BigEndian)
    // decoder.SetDebug(true) // you can enable the output of bytes read for debugging
    err = decoder.Decode(&actual)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("%+v", actual)

    // Output:
    // {Arr:[1 2 3 4]}
}

From bytes

package main

import (
    "fmt"
    "log"
    
    "github.com/ghostiam/binstruct"
)

func main() {
    data := []byte{
        0x00, 0x01,
        0x00, 0x02,
        0x00, 0x03,
        0x00, 0x04,
    }

    type dataStruct struct {
        Arr []int16 `bin:"len:4"`
    }

    var actual dataStruct
    err := binstruct.UnmarshalBE(data, &actual) // UnmarshalLE() or Unmarshal()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("%+v", actual)

    // Output: {Arr:[1 2 3 4]}
}

or just use reader without mapping data into the structure

You can not use the functionality for mapping data into the structure, you can use the interface to get data from the stream (io.ReadSeeker)

reader.go

type Reader interface {
    io.ReadSeeker

    // Peek returns the next n bytes without advancing the reader.
    Peek(n int) ([]byte, error)

    // ReadBytes reads up to n bytes. It returns the number of bytes
    // read, bytes and any error encountered.
    ReadBytes(n int) (an int, b []byte, err error)
    // ReadAll reads until an error or EOF and returns the data it read.
    ReadAll() ([]byte, error)

    // ReadByte read and return one byte
    ReadByte() (byte, error)
    // ReadBool read one byte and return boolean value
    ReadBool() (bool, error)

    // ReadUint8 read one byte and return uint8 value
    ReadUint8() (uint8, error)
    // ReadUint16 read two bytes and return uint16 value
    ReadUint16() (uint16, error)
    // ReadUint32 read four bytes and return uint32 value
    ReadUint32() (uint32, error)
    // ReadUint64 read eight bytes and return uint64 value
    ReadUint64() (uint64, error)
    // ReadUintX read X bytes and return uint64 value
    ReadUintX(x int) (uint64, error)

    // ReadInt8 read one byte and return int8 value
    ReadInt8() (int8, error)
    // ReadInt16 read two bytes and return int16 value
    ReadInt16() (int16, error)
    // ReadInt32 read four bytes and return int32 value
    ReadInt32() (int32, error)
    // ReadInt64 read eight bytes and return int64 value
    ReadInt64() (int64, error)
    // ReadIntX read X bytes and return int64 value
    ReadIntX(x int) (int64, error)

    // ReadFloat32 read four bytes and return float32 value
    ReadFloat32() (float32, error)
    // ReadFloat64 read eight bytes and return float64 value
    ReadFloat64() (float64, error)

    // Unmarshal parses the binary data and stores the result
    // in the value pointed to by v.
    Unmarshal(v interface{}) error

    // WithOrder changes the byte order for the new Reader
    WithOrder(order binary.ByteOrder) Reader
}

Example:

package main

import (
    "encoding/binary"
    "fmt"
    "log"
    
    "github.com/ghostiam/binstruct"
)

func main() {
    data := []byte{0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F}

    reader := binstruct.NewReaderFromBytes(data, binary.BigEndian, false)

    i16, err := reader.ReadInt16()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(i16)

    i32, err := reader.ReadInt32()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(i32)

    b, err := reader.Peek(4)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Peek bytes: %#v\n", b)

    an, b, err := reader.ReadBytes(4)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Read %d bytes: %#v\n", an, b)

    other, err := reader.ReadAll()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Read all: %#v\n", other)

    // Output:
    // 258
    // 50595078
    // Peek bytes: []byte{0x7, 0x8, 0x9, 0xa}
    // Read 4 bytes: []byte{0x7, 0x8, 0x9, 0xa}
    // Read all: []byte{0xb, 0xc, 0xd, 0xe, 0xf}
}

Decode to fields

type test struct {
    // Read 1 byte
    Field bool
    Field byte
    Field [1]byte
    Field int8
    Field uint8

    // Read 2 bytes
    Field int16
    Field uint16
    Field [2]byte

    // Read 4 bytes
    Field int32
    Field uint32
    Field [4]byte

    // Read 8 bytes
    Field int64
    Field uint64
    Field [8]byte

    // You can override length
    Field int64 `bin:"len:2"`
    // Or even use very weird byte lengths for int
    Field int64 `bin:"len:3"`
    Field int64 `bin:"len:5"`
    Field int64 `bin:"len:7"`

    // Fields of type int, uint and string are not read automatically 
    // because the size is not known, you need to set it manually
    Field int    `bin:"len:2"`
    Field uint   `bin:"len:4"`
    Field string `bin:"len:42"`
    
    // Can read arrays and slices
    Array [2]int32              // read 8 bytes (4+4byte for 2 int32)
    Slice []int32 `bin:"len:2"` // read 8 bytes (4+4byte for 2 int32)
    
    // Also two-dimensional slices work (binstruct_test.go:307 Test_SliceOfSlice)
    Slice2D [][]int32 `bin:"len:2,[len:2]"`
    // and even three-dimensional slices (binstruct_test.go:329 Test_SliceOfSliceOfSlice)
    Slice3D [][][]int32 `bin:"len:2,[len:2,[len:2]]"`
    
    // Structures and embedding are also supported.
    Struct struct {
        ...
    }
    OtherStruct Other
    Other // embedding
}

type Other struct {
    ...
}

Tags

type test struct {
    IgnoredField []byte `bin:"-"`          // ignore field
    CallMethod   []byte `bin:"MethodName"` // Call method "MethodName"
    ReadLength   []byte `bin:"len:42"`     // read 42 bytes

    // Offsets test binstruct_test.go:9
    Offset      byte `bin:"offset:42"`      // move to 42 bytes from current position and read byte
    OffsetStart byte `bin:"offsetStart:42"` // move to 42 bytes from start position and read byte
    OffsetEnd   byte `bin:"offsetEnd:-42"`  // move to -42 bytes from end position and read byte
    OffsetStart byte `bin:"offsetStart:42, offset:10"` // also worked and equally `offsetStart:52`

    // Calculations supported +,-,/,* and are performed from left to right that is 2+2*2=8 not 6!!!
    CalcTagValue []byte `bin:"len:10+5+2+3"` // equally len:20

    // You can refer to another field to get the value.
    DataLength              int    // actual length
    ValueFromOtherField     string `bin:"len:DataLength"`
    CalcValueFromOtherField string `bin:"len:DataLength+10"` // also work calculations

    // You can change the byte order directly from the tag
    UInt16LE uint16 `bin:"le"`
    UInt16BE uint16 `bin:"be"`
    // Or when you call the method, it will contain the Reader with the byte order you need
    CallMethodWithLEReader uint16 `bin:"MethodNameWithLEReader,le"`
    CallMethodWithBEReader uint16 `bin:"be,MethodNameWithBEReader"`
} 

// Method can be:
func (*test) MethodName(r binstruct.Reader) (error) {}
// or
func (*test) MethodName(r binstruct.Reader) (FieldType, error) {}

See the tests and examples for more information.

Examples

ZIP decoder 
PNG decoder

Author: Ghostiam
Source Code: https://github.com/ghostiam/binstruct 
License: MIT License

#go #golang #binary #decode 

Is FidoMeta a secure cryptocurrency?

FidoMeta is a new cryptocurrency that has recently entered the market. The BEP-20 is used to create the FidoMeta coin. Cryptocurrency is becoming more popular every day, and investing now can help you reap future benefits. FidoMeta is the most reliable and transparent digital currency. The coin was designed and developed by some of the brightest brains in the world, and it incorporates the most cutting-edge technology. 

Visit: www.fidometa.io
Mail: info@fidometa.io

#cryptotrading #cryptonews
#binance #forexlife #binary #euro #CryptoOnIndia #Cryptocurrency #Cryptoindiaupdates #CryptoCrown
#CryptoUpdates #doge
#cryptocurrency