General: The Official Registry Of General Julia Packages

General

General is the default Julia package registry. Package registries are used by Julia's package manager Pkg.jl and includes information about packages such as versions, dependencies and compatibility constraints.

The General registry is open for everyone to use and provides access to a large ecosystem of packages.

If you are registering a new package, please make sure that you have read the package naming guidelines.

Follow along new package registrations with the #new-packages-feed channels in the community Slack or Zulip!

See our Contributing Guidelines for ways to get involved!

Registering a package in General

New packages and new versions of packages are added to the General registry by pull requests against this GitHub repository. It is highly recommended that you use Registrator.jl to automate this process. Registrator can either be used as a GitHub App or through a web interface, as decribed in the Registrator README.

When Registrator is triggered a pull request is opened against this repository. Pull requests that meet certain guidelines is merged automatically, see Automatic merging of pull requests. Other pull requests need to be manually reviewed and merged by a human.

It is highly recommended to also use TagBot, which automatically tags a release in your repository after the new release of your package is merged into the registry.

Registered packages must have an Open Source Initiative approved license, clearly marked via a LICENSE.md, LICENSE, COPYING or similarly named file in the package repository. Packages that wrap proprietary libraries are acceptable if the licenses of those libraries permit open source distribution of the Julia wrapper code.

Automatic merging of pull requests

Pull requests that meet certain criteria are automatically merged periodically. Only pull requests that are opened by Registrator are candidates for automatic merging.

The full list of AutoMerge guidelines is available in the RegistryCI documentation.

Please report issues with automatic merging to the RegistryCI repo.

Currently the waiting period is as follows:

  • New Julia packages: 3 days (this allows time for community feedback)
  • New versions of existing packages: 15 minutes
  • JLL package (binary dependencies): 15 minutes, for either a new package or a new version

FAQ

Do I need to register a package to install it?

No, you can simply do using Pkg; Pkg.add(url="https://github.com/JuliaLang/Example.jl") or ] add https://github.com/JuliaLang/Example.jl in the Pkg REPL mode to e.g. install the package Example.jl, even if it was not registered. When a package is installed this way, the URL is saved in the Manifest.toml, so that file is needed to resolve Pkg environments that have unregistered packages installed.

Registering allows the package to be added by Pkg.add("Example") or ] add Example in the Pkg REPL mode. This is true if the package is installed in any registry you have installed, not just General; you can even create your own registry!

Should I register my package?

If your package might be useful to others, or provide functionality other packages in General might want to rely on, go for it! We only ask that you consider the following best practices.

  • It is easier for others to use your package if it has documentation that explains what the package is for and how to use it. This could be in the form of a README or hosted documentation such as that generated by Documenter.jl.
  • And in order to provide reliable functionality for your users, it is also important to setup tests (see the Pkg.jl docs and the Test stdlib docs), which can be automatically run by free continuous integration services such as GitHub Actions.
  • Also, note that the General registry is not a place for "personal packages" that consist of collections of "utility functions" nor for packages that are only useful for a closed group (like a research group or a company). For that, it is easy to set up your own registry using for example LocalRegistry.jl. The Pkg documentation about registries might be useful if you decide to go this route.

Packages like PkgTemplates.jl or PkgSkeleton.jl provide easy ways to setup documentation, tests, and continuous integration.

My pull request was not approved for automatic merging, what do I do?

It is recommended that you fix the release to conform to the guidelines and then retrigger Registrator on the branch/commit that includes the fix.

If you for some reason can't (or won't) adhere to the guidelines you will have to wait for a human to review/merge the pull request. You can contact a human in the #pkg-registration channel in the official Julia Slack to expedite this process.

My package fails to load because it needs proprietary software/additional setup to work, what can I do?

Before merging a pull request, AutoMerge will check that your package can be installed and loaded. It is OK for your package to not be fully functional, but making it at least load successfully would streamline registration, as it does not require manual intervention from the registry maintainers. This would also let other packages depend on it, and use its functionalities only when the proprietary software is available in the system, as done for example by the CUDA.jl package. If you are not able or willing to make your package always loadable without the proprietary dependency (which is the preferred solution), you can check if the environment variable JULIA_REGISTRYCI_AUTOMERGE is equal to true and make your package loadable during AutoMerge at least, so that it can be registered without manual intervention. Examples of packages with proprietary software that use the environment variable check include Gurobi.jl and CPLEX.jl.

My pull request has a merge conflict, what do I do?

Retrigger Registrator.

How do I retrigger Registrator in order to update my pull request?

Do what you did when you triggered Registrator the first time.

AutoMerge is blocked by one of my comments, how do I unblock it?

Simply edit [noblock] into all your comments. AutoMerge periodically checks each PR, and if there are no blocking comments when it checks (i.e. all comments have [noblock] present), it will continue to merge (assuming of course that all of its other checks have passed).

Are there any requirements for package names in the General registry?

There are no hard requirements, but it is highly recommended to follow the package naming guidelines.

What to do when asked to reconsider/update the package name?

If someone comments on the name of your package when you first release it it is often because it does not follow the naming guidelines. If you think that your package should not follow those conventions for some reason or another, just explain why. Otherwise, it is often a good idea to just rename the package -- it is more disruptive to do so after it is already registered, and sticking to the conventions makes it easier for users to navigate Julia's many varied packages.

As long as the package is not yet registered, renaming the package from OldName.jl to NewName.jl is reasonably straightforward:

  • Rename the GitHub repository to NewName.jl
  • Rename the file src/OldName.jl to src/NewName.jl
  • Rename the top-level module to NewName
  • Update tests, documentation, etc, to reference the new name
  • Once you are done renaming the package, retrigger registration. This will make a new pull request to General. It is helpful to comment in the old pull request that it can be closed, linking to the new one.

How do I rename an existing registered package?

Technically, you can't rename a package once registered, as this would break existing users. But you can re-register the package again under a new name with a new UUID, which basically has the same effect.

  • Follow the instructions above for renaming a package: rename on GitHub, rename files etc.
    • if you rename the repository so it has a new URL, make a PR to edit the URL stored in the registry for the old package name to point to the new URL (example). This allows the old versions of the package under the previous name to continue to work.
  • Generate a new UUID for the Project.toml
  • Increment the version in the Project.toml as a breaking change.
  • Register it as if it were a new package
  • Comment on the PR, that this is a rename.
  • It will have to go though the normal criteria for registring a new package.
    • In particular, even if you get it merged manually, it will need to wait 3 days from the PR being opened.
    • This gives others and yourself the chance to point out any naming issues.

You also should let your users know about the rename, e.g. by placing a note in the README, or opening PRs/issues on downstream packages to change over.

How do I transfer a package to an organization or another user?

Technically if you skip the second step things will keep working, because GitHub will redirect; but it is best practice. For this reason, when you try to register a new release, the Julia Registrator will complain if the second step is skipped.

Where do I report a problem with a package in the General registry?

Report it to the package repository.

How do I remove a package or version from the registry?

You can't. Package registrations are permanent. A version can not be overwritten in the registry, and code cannot be deleted.

Can my package be registered without an OSI approved license?

No, sorry. The registry is maintained by volunteers, and we don't have a legal team who can thoroughly review licenses. It is very easy to accidentally wander into legally murky territory when combining common OSI licenses[^1] like GPL with non-OSI licenses and we don't want to subject Julia users to that risk when installing packages registered in General. See these [comments] (https://github.com/JuliaRegistries/General/pull/31549#issuecomment-804196208) for more discussion. We are not lawyers and this is not legal advice.

[^1]: Note that even within the world of OSI licenses, there are combinations of OSI licenses which are not legal to use together, such as GPL2 with Apache2.

Registry maintenance

The General registry is a shared resource that belongs to the entire Julia community. Therefore, we welcome comments and suggestions from everyone in the Julia community. However, all decisions regarding the General registry are ultimately up to the discretion of the registry maintainers.

See our Contributing Guidelines for ways to get involved!

Disclaimer

The General registry is open for everyone to register packages in. The General registry is not a curated list of Julia packages. In particular this means that:

  • packages included in the General registry are not reviewed/scrutinized;
  • packages included in the General registry are not "official" packages and not endorsed/approved by the JuliaLang organization;
  • the General registry and its maintainers are not responsible for the package code you install through the General registry -- you are responsible for reviewing your code dependencies.
WorkflowStatus
AutoMergeAutoMerge status
Continuous Integration (CI)Continuous Integration (CI) status
TagBot TriggersTagBot Triggers status
Update ManifestsUpdate Manifests status

Download Details:

Author: JuliaRegistries
Source Code: https://github.com/JuliaRegistries/General 
License: MIT license

#julia #general 

What is GEEK

Buddha Community

General: The Official Registry Of General Julia Packages

General: The Official Registry Of General Julia Packages

General

General is the default Julia package registry. Package registries are used by Julia's package manager Pkg.jl and includes information about packages such as versions, dependencies and compatibility constraints.

The General registry is open for everyone to use and provides access to a large ecosystem of packages.

If you are registering a new package, please make sure that you have read the package naming guidelines.

Follow along new package registrations with the #new-packages-feed channels in the community Slack or Zulip!

See our Contributing Guidelines for ways to get involved!

Registering a package in General

New packages and new versions of packages are added to the General registry by pull requests against this GitHub repository. It is highly recommended that you use Registrator.jl to automate this process. Registrator can either be used as a GitHub App or through a web interface, as decribed in the Registrator README.

When Registrator is triggered a pull request is opened against this repository. Pull requests that meet certain guidelines is merged automatically, see Automatic merging of pull requests. Other pull requests need to be manually reviewed and merged by a human.

It is highly recommended to also use TagBot, which automatically tags a release in your repository after the new release of your package is merged into the registry.

Registered packages must have an Open Source Initiative approved license, clearly marked via a LICENSE.md, LICENSE, COPYING or similarly named file in the package repository. Packages that wrap proprietary libraries are acceptable if the licenses of those libraries permit open source distribution of the Julia wrapper code.

Automatic merging of pull requests

Pull requests that meet certain criteria are automatically merged periodically. Only pull requests that are opened by Registrator are candidates for automatic merging.

The full list of AutoMerge guidelines is available in the RegistryCI documentation.

Please report issues with automatic merging to the RegistryCI repo.

Currently the waiting period is as follows:

  • New Julia packages: 3 days (this allows time for community feedback)
  • New versions of existing packages: 15 minutes
  • JLL package (binary dependencies): 15 minutes, for either a new package or a new version

FAQ

Do I need to register a package to install it?

No, you can simply do using Pkg; Pkg.add(url="https://github.com/JuliaLang/Example.jl") or ] add https://github.com/JuliaLang/Example.jl in the Pkg REPL mode to e.g. install the package Example.jl, even if it was not registered. When a package is installed this way, the URL is saved in the Manifest.toml, so that file is needed to resolve Pkg environments that have unregistered packages installed.

Registering allows the package to be added by Pkg.add("Example") or ] add Example in the Pkg REPL mode. This is true if the package is installed in any registry you have installed, not just General; you can even create your own registry!

Should I register my package?

If your package might be useful to others, or provide functionality other packages in General might want to rely on, go for it! We only ask that you consider the following best practices.

  • It is easier for others to use your package if it has documentation that explains what the package is for and how to use it. This could be in the form of a README or hosted documentation such as that generated by Documenter.jl.
  • And in order to provide reliable functionality for your users, it is also important to setup tests (see the Pkg.jl docs and the Test stdlib docs), which can be automatically run by free continuous integration services such as GitHub Actions.
  • Also, note that the General registry is not a place for "personal packages" that consist of collections of "utility functions" nor for packages that are only useful for a closed group (like a research group or a company). For that, it is easy to set up your own registry using for example LocalRegistry.jl. The Pkg documentation about registries might be useful if you decide to go this route.

Packages like PkgTemplates.jl or PkgSkeleton.jl provide easy ways to setup documentation, tests, and continuous integration.

My pull request was not approved for automatic merging, what do I do?

It is recommended that you fix the release to conform to the guidelines and then retrigger Registrator on the branch/commit that includes the fix.

If you for some reason can't (or won't) adhere to the guidelines you will have to wait for a human to review/merge the pull request. You can contact a human in the #pkg-registration channel in the official Julia Slack to expedite this process.

My package fails to load because it needs proprietary software/additional setup to work, what can I do?

Before merging a pull request, AutoMerge will check that your package can be installed and loaded. It is OK for your package to not be fully functional, but making it at least load successfully would streamline registration, as it does not require manual intervention from the registry maintainers. This would also let other packages depend on it, and use its functionalities only when the proprietary software is available in the system, as done for example by the CUDA.jl package. If you are not able or willing to make your package always loadable without the proprietary dependency (which is the preferred solution), you can check if the environment variable JULIA_REGISTRYCI_AUTOMERGE is equal to true and make your package loadable during AutoMerge at least, so that it can be registered without manual intervention. Examples of packages with proprietary software that use the environment variable check include Gurobi.jl and CPLEX.jl.

My pull request has a merge conflict, what do I do?

Retrigger Registrator.

How do I retrigger Registrator in order to update my pull request?

Do what you did when you triggered Registrator the first time.

AutoMerge is blocked by one of my comments, how do I unblock it?

Simply edit [noblock] into all your comments. AutoMerge periodically checks each PR, and if there are no blocking comments when it checks (i.e. all comments have [noblock] present), it will continue to merge (assuming of course that all of its other checks have passed).

Are there any requirements for package names in the General registry?

There are no hard requirements, but it is highly recommended to follow the package naming guidelines.

What to do when asked to reconsider/update the package name?

If someone comments on the name of your package when you first release it it is often because it does not follow the naming guidelines. If you think that your package should not follow those conventions for some reason or another, just explain why. Otherwise, it is often a good idea to just rename the package -- it is more disruptive to do so after it is already registered, and sticking to the conventions makes it easier for users to navigate Julia's many varied packages.

As long as the package is not yet registered, renaming the package from OldName.jl to NewName.jl is reasonably straightforward:

  • Rename the GitHub repository to NewName.jl
  • Rename the file src/OldName.jl to src/NewName.jl
  • Rename the top-level module to NewName
  • Update tests, documentation, etc, to reference the new name
  • Once you are done renaming the package, retrigger registration. This will make a new pull request to General. It is helpful to comment in the old pull request that it can be closed, linking to the new one.

How do I rename an existing registered package?

Technically, you can't rename a package once registered, as this would break existing users. But you can re-register the package again under a new name with a new UUID, which basically has the same effect.

  • Follow the instructions above for renaming a package: rename on GitHub, rename files etc.
    • if you rename the repository so it has a new URL, make a PR to edit the URL stored in the registry for the old package name to point to the new URL (example). This allows the old versions of the package under the previous name to continue to work.
  • Generate a new UUID for the Project.toml
  • Increment the version in the Project.toml as a breaking change.
  • Register it as if it were a new package
  • Comment on the PR, that this is a rename.
  • It will have to go though the normal criteria for registring a new package.
    • In particular, even if you get it merged manually, it will need to wait 3 days from the PR being opened.
    • This gives others and yourself the chance to point out any naming issues.

You also should let your users know about the rename, e.g. by placing a note in the README, or opening PRs/issues on downstream packages to change over.

How do I transfer a package to an organization or another user?

Technically if you skip the second step things will keep working, because GitHub will redirect; but it is best practice. For this reason, when you try to register a new release, the Julia Registrator will complain if the second step is skipped.

Where do I report a problem with a package in the General registry?

Report it to the package repository.

How do I remove a package or version from the registry?

You can't. Package registrations are permanent. A version can not be overwritten in the registry, and code cannot be deleted.

Can my package be registered without an OSI approved license?

No, sorry. The registry is maintained by volunteers, and we don't have a legal team who can thoroughly review licenses. It is very easy to accidentally wander into legally murky territory when combining common OSI licenses[^1] like GPL with non-OSI licenses and we don't want to subject Julia users to that risk when installing packages registered in General. See these [comments] (https://github.com/JuliaRegistries/General/pull/31549#issuecomment-804196208) for more discussion. We are not lawyers and this is not legal advice.

[^1]: Note that even within the world of OSI licenses, there are combinations of OSI licenses which are not legal to use together, such as GPL2 with Apache2.

Registry maintenance

The General registry is a shared resource that belongs to the entire Julia community. Therefore, we welcome comments and suggestions from everyone in the Julia community. However, all decisions regarding the General registry are ultimately up to the discretion of the registry maintainers.

See our Contributing Guidelines for ways to get involved!

Disclaimer

The General registry is open for everyone to register packages in. The General registry is not a curated list of Julia packages. In particular this means that:

  • packages included in the General registry are not reviewed/scrutinized;
  • packages included in the General registry are not "official" packages and not endorsed/approved by the JuliaLang organization;
  • the General registry and its maintainers are not responsible for the package code you install through the General registry -- you are responsible for reviewing your code dependencies.
WorkflowStatus
AutoMergeAutoMerge status
Continuous Integration (CI)Continuous Integration (CI) status
TagBot TriggersTagBot Triggers status
Update ManifestsUpdate Manifests status

Download Details:

Author: JuliaRegistries
Source Code: https://github.com/JuliaRegistries/General 
License: MIT license

#julia #general 

Bibliography.jl: General Bibliography Manager in Pure Julia

Bibliography.jl

Bibliography.jl is a Julia package for handling both import/export from various bibliographic formats.

Short documentation

# Import a BibTeX file to the internal bib structure
imported_bib = import_bibtex(source_path::AbstractString)


# Select a part of a bibliography
selection = ["key1", "key2"]
selected_bib = select(imported_bib, selection) # select the intersection between the bibliography and `selection`
diff_bib = select(imported_bib, selection; complementary = true) # select the difference between the bibliography and `selection`

# Export from internal to BibTeX format
export_bibtex(target_path::AbstractString, bibliography)

# Check BibTeX rules, entry validity, clean and sort a bibtex file
export_bibtex(target_path::AbstractString, import_bibtex(path_to_file::AbstractString))

# Export from internal to the Web Format of StaticWebPages.jl
export_web(bibliography)

# Export from BibTeX to the Web Format of StaticWebPages.jl
bibtex_to_web(source_path::AbstractString)

Organization

This package comes as a set of 3 packages to convert bibliographies. This tool was split into three for the sake of the precompilation times.

  • Bibliography.jl: The interface to import/export bibliographic items.
  • BibInternal.jl: A julian internal format to translate from and into.
  • BibParser.jl: A container for different bibliographic format parsers (such as BibTeX).

Packages using Bibliographies

  • StaticWebPages.jl: a black-box generator for static websites oriented towards personal and/or academic pages. No knowledge of Julia nor any other programming language is required.

Contributions are welcome

  • Write new or integrate existing parsers to BibParser.jl (currently only a light BibTeX parser is available).
  • Add import/export from existing bibliographic formats to Bibliography.jl.
  • Add export for non-bibliographic formats (such as in StaticWebPages.jl).

Download Details:

Author: Humans-of-Julia
Source Code: https://github.com/Humans-of-Julia/Bibliography.jl 
License: MIT license

#julia #general 

GrammaticalEvolution: Grammatical Evolution Package for Julia

GrammaticalEvolution

GrammaticalEvolution provides the evolutionary technique of Grammatical Evolution optimization (cite) for Julia. The library focuses on providing the tools necessary to construct a grammar and evolve a population without forcing the user to use a large framework. No XML or configuration files are necessary.

One important aspect of the GrammaticalEvolution library is that uses the strength of Julia to insert arbritrary into the program. Instead of creating a string that then has to be parsed, GrammaticalEvolution instead directly generates Julia code that can then be run. The advantages of doing so are: speeed, simplification of the grammar, and the grammar does not have to be defined in an auxillary file.

Defining a grammar

The following code defines a grammar:

@grammar <name> begin
  rule1 = ...
  rule2 = ...
end

Allowed rules

The following rules are supported:

  • Terminals: Strings, numbers, and Exprs
  • Or: a | b | c
  • And: a + b + c
  • Grouping: (a + b) | (c + d)

Example

In the examples directory there is an example to learn an arbritrary mathemtical equation. Below is an annotated version of that code.

First, we can define the grammar:

  @grammar example_grammar begin
    start = ex
    ex = number | sum | product | (ex) | value
    sum = Expr(:call, :+, ex, ex)
    product = Expr(:call, :*, ex, ex)
    value = :x | :y | number
    number[convert_number] = digit + '.' + digit
    digit = 0:9
  end

Every grammar must define a start symbol. This grammar supports equations that add and multiply values together, but it's trivial to build more mathematical operations into the grammar.

There are several items of note:

  1. Some rules result in a Julia Expr. This results in the Julia code that makes a function call to the selected function.
  2. The number rule is followed by [convert_number]. This appends an action to the rule which is invoked when the rule gets applied.
  3. The rule digit = 0:9 is translated into digit = 0 | 1 | 2 | .. | 9

The action convert_number is defined as:

convert_number(lst) = float(join(lst))

Next individuals which are generated by the grammar can be defined as:

type ExampleIndividual <: Individual
  genome::Array{Int64, 1}
  fitness::Float64
  code

  function ExampleIndividual(size::Int64, max_value::Int64)
    genome = rand(1:max_value, size)
    return new(genome, -1.0, nothing)
  end

  ExampleIndividual(genome::Array{Int64, 1}) = new(genome, -1.0, nothing)
end

with a population of the individuals defined as:

type ExamplePopulation <: Population
  individuals::Array{ExampleIndividual, 1}

  function ExamplePopulation(population_size::Int64, genome_size::Int64)
    individuals = Array(ExampleIndividual, 0)
    for i=1:population_size
      push!(individuals, ExampleIndividual(genome_size, 1000))
    end

    return new(individuals)
  end
end

Finally, the evaluation function for the individuals can be defined as:

function evaluate!(grammar::Grammar, ind::ExampleIndividual)
  fitness::Array{Float64, 1} = {}

  # transform the individual's genome into Julia code
  try
    ind.code = transform(grammar, ind)
    @eval fn(x, y) = $(ind.code)
  catch e
    println("exception = $e")
    ind.fitness = Inf
    return
  end

  # evaluate the generated code at multiple points
  for x=0:10
    for y=0:10
      # this the value of the individual's code
      value = fn(x, y)

      # the difference between the ground truth
      diff = (value - gt(x, y)).^2

      if !isnan(diff) && diff > 0
        insert!(fitness, length(fitness)+1, sqrt(diff))
      elseif diff == 0
        insert!(fitness, length(fitness)+1, 0)
      end
    end
  end

  # total fitness is the average over all of the sample points
  ind.fitness = mean(fitness)
end

To use our defined grammar and population we can write:

# here is our ground truth
gt(x, y) = 2*x + 5*y

# create population
pop = ExamplePopulation(500, 100)

fitness = Inf
generation = 1
while fitness > 1.0
  # generate a new population (based off of fitness)
  pop = generate(example_grammar, pop, 0.1, 0.2, 0.2)

  # population is sorted, so first entry it the best
  fitness = pop[1].fitness
  println("generation: $generation, max fitness=$fitness, code=$(pop[1].code)")
  generation += 1
end

A couple items of note:

  • While generate is defined in the base package, it is easy to write your own version.
  • Both Population and Individual are user defined. The fields of these types can differ from what are used in ExamplePopulation and ExampleIndividual. However, this may require several methods to be defined so the library can index into the population and genome.

The following piece of code in evaluate! may look strange:

ind.code = transform(grammar, ind)
@eval fn(x, y) = $(ind.code)

So it's worth explaining it in a little more detail. The first part ind.code = transform(grammar, ind) takes the current genome of the genome and turns it into unevalauate Julia code. In the code we use the variables x and y which are currently not defined.

The variables x and y are defined in the loop further below. However, they must first be bound to the generated code. Simply running eval won't work because it is designed to only use variables defined in the global scope. The line @eval fn(x, y) = $(ind.code) creates a function that has its body bound to an evaluated version of the code. When fn(x, y) is called, the variables x and y will now be in scope of the code.

Build Status

Download Details:

Author: Abeschneider
Source Code: https://github.com/abeschneider/GrammaticalEvolution 
License: View license

#julia #package 

JuliaWebAPI.jl: Julia Package for Deploying APIs

JuliaWebAPI.jl

Facilitates wrapping Julia functions into a remote callable API via message queues (e.g. ZMQ, RabbitMQ) and HTTP.

It can plug in to a different messaging infrastructure through an implementation of transport (AbstractTransport) and message format (AbstractMsgFormat). Multiple instances of the front (HTTP API) and back (Julia methods) end can help scale an application. Bundled with the package are implementations for:

  • ZMQTransport: use ZMQ for transport
  • InProcTransport: use Julia Channel for transport within the same process
  • JSONMsgFormat: JSON as message format
  • SerializedMsgFormat: Julia serialization as message format
  • DictMsgFormat: Julia Dict as message format, for use within the same process

Combined with a HTTP/Messaging frontend (like JuliaBox), it helps deploy Julia packages and code snippets as hosted, auto-scaling HTTP APIs.

Some amount of basic request filtering and pre-processing is possible by registering a pre-processor with the HTTP frontend. The pre-processor is run at the HTTP server side, where it has access to the complete request. It can examine headers and data and take decision whether to allow calling the service or respond directly and immediately. It can also rewrite the request before passing it on to the service.

A pre-processor can be used to implement features like authentication, request rewriting and such. See example below.

Example Usage

Create a file srvr.jl with the following code

# Load required packages
using JuliaWebAPI

# Define functions testfn1 and testfn2 that we shall expose
function testfn1(arg1, arg2; narg1="1", narg2="2")
    return (parse(Int, arg1) * parse(Int, narg1)) + (parse(Int, arg2) * parse(Int, narg2))
end

testfn2(arg1, arg2; narg1="1", narg2="2") = testfn1(arg1, arg2; narg1=narg1, narg2=narg2)

# Expose testfn1 and testfn2 via a ZMQ listener
process(
    JuliaWebAPI.create_responder([
        (testfn1, true),
        (testfn2, false)
    ], "tcp://127.0.0.1:9999", true, "")
)

Start the server process in the background. This process will run the ZMQ listener.

julia srvr.jl &

Then, on a Julia REPL, run the following code

using JuliaWebAPI   #Load package

#Create the ZMQ client that talks to the ZMQ listener above
const apiclnt = APIInvoker("tcp://127.0.0.1:9999");

#Start the HTTP server in current process (Ctrl+C to interrupt)
run_http(apiclnt, 8888)

Then, on your browser, navigate to http://localhost:8888/testfn1/4/5?narg1=6&narg2=4

This will return the following JSON response to your browser, which is the result of running the testfn1 function defined above: {"data"=>44,"code"=>0}

Example of an authentication filter implemented using a pre-processor:

function auth_preproc(req::HTTP.Request)
    if !validate(req)
        return HTTP.Response(401)
    end
    return nothing
end
run_http(apiclnt, 8888, auth_preproc)

Download Details:

Author: JuliaWeb
Source Code: https://github.com/JuliaWeb/JuliaWebAPI.jl 
License: View license

#julia #web #package #api 

Monty  Boehm

Monty Boehm

1660036740

Diversity.jl: Julia Package for Diversity Measurement

Diversity

A package for measuring and partitioning diversity

Summary

Diversity is a Julia package that provides functionality for measuring alpha, beta and gamma diversity of metacommunities (e.g. ecosystems) and their constituent subcommunities. It uses the diversity measures described in the arXiv paper arXiv:1404.6520 (q-bio.QM), How to partition diversity. It also provides a series of other older diversity measures through sub-modules. Currently these are all ecological diversity measures, but this will be expanded through interfacing to EcoJulia and BioJulia.

This package is in beta now, but is cross-validated against our R package boydorr/rdiversity, which is developed independently, so please raise an issue if you find any problems. We now use a DataFrame as the common output format for all of the diversity calculations to provide consistency with our R package rdiversity. The code is not optimised for speed at the moment due to the substantial changes that have happened to it under the hood, and the Phylogenetics submodule is also recently revised, and may need further improvements.

Installation

The package is registered in the General registry on v1.x and so can be installed with add. For example on Julia v1.6:

(@v1.6) pkg> add Diversity
    Resolving package versions...
    Updating `~/.julia/environments/v1.6/Project.toml`
  [d3d5718d] + Diversity v0.5.5
    Updating `~/.julia/environments/v1.6/Manifest.toml`
  [d3d5718d] + Diversity v0.5.5
  
(@v1.6) pkg>

Project Status

The package is confirmed to work against the current LTS Julia v1.4 release and the latest release on Linux, macOS, and Windows. It is also tested against nightly.

Contributing and Questions

Contributions are very welcome, as are feature requests and suggestions. Please open an issue if you encounter any problems or would just like to ask a question.

Usage

Diversity Measures

The main package provides basic numbers-equivalent diversity measures (described in Hill, 1973), similarity-sensitive diversity measures (generalised from Hill, and described in Leinster and Cobbold, 2012), and related alpha, beta and gamma diversity measures at the level of the metacommunity and its component subcommunities (generalised in turn from Leinster and Cobbold, and described in arXiv:1404.6520 (q-bio.QM)). The diversity functions exist both with unicode names (e.g. ᾱ()), which are not automatically exported as we feel they are too short and with matching ascii names (e.g. NormalisedAlpha()), which are. We also provide a general function for extract any diversity measure for a series of subcommunity relative abundances.

Getting started

Before calculating diversity a Metacommunity object must be created. This object contains all the information needed to calculate diversity.

# Load the package into Julia
using Diversity

# Example population
pop = [1 1 0; 2 0 0; 3 1 4]
pop = pop / sum(pop)

# Create Metacommunity object
meta = Metacommunity(pop)

Calculating diversity

First we need to calculate the low-level diversity component seperately, by passing a metacommunity object to the appropriate function; RawAlpha(), NormalisedAlpha(), RawBeta(), NormalisedBeta(), RawRho(), NormalisedRho(), or Gamma().

# First, calculate the normalised alpha component
component = NormalisedAlpha(meta)

Afterwhich, subdiv() or metadiv() are used to calculate subcommunity or metacommunity diversity, respectively (since both subcommunity and metacommunity diversity measures are transformations of the same low-level components, this is computationally more efficient).

# Then, calculate species richness of the subcommunities
subdiv(component, 0)

# or the average (alpha) species richness across the whole population
metadiv(component, 0)

# We can also generate a diversity profile by calculating multiple q-values simultaneously
df = subdiv(component, 0:30)

In some instances, it may be useful to calculate all subcommunity (or metacommunity) measures. In which case, a Metacommunity object may be passed directly to subdiv() or metadiv():

# To calculate all subcommunity diversity measures
subdiv(meta, 0:2)

# To calculate all metacommunity diversity measures
metadiv(meta, 0:2)

Alternatively, if computational efficiency is not an issue, a single measure of diversity may be calculated directly by calling a wrapper function:

norm_sub_alpha(meta, 0:2)

A complete list of these functions is shown below:

  • raw_sub_alpha() : per-subcommunity estimate of naive-community metacommunity diversity
  • norm_sub_alpha() : similarity-sensitive diversity of each subcommunity in isolation
  • raw_sub_rho() : redundancy of individual subcommunities
  • norm_sub_rho() : representativeness of individual subcommunities
  • raw_sub_beta() : distinctiveness of individual subcommunities
  • norm_sub_beta() : per-subcommunity estimate of effective number of distinct subcommunities
  • sub_gamma() : contribution per individual in a subcommunity toward metacommunity diversity
  • raw_meta_alpha() : naive-community metacommunity diversity
  • norm_meta_alpha() : average similarity-sensitive diversity of subcommunities
  • raw_meta_rho() : average redundancy of subcommunities
  • norm_meta_rho() : average representativeness of subcommunities
  • raw_meta_beta() : average distinctiveness of subcommunities
  • norm_meta_beta() : effective number of distinct subcommunities
  • meta_gamma() : metacommunity similarity-sensitive diversity

Phylogenetic diversity

Phylogenetic diversity (described here) is automatically included in the Diversity module when the Phylo package is loaded. Documentation for these diversity measures can be found here. The phylogenetics code relies on the Phylo package to generate trees to incorporate into the diversity code, and the ppropriate code will be created when both main packages are loaded:

julia> using Diversity, Phylo

julia> communities = [4 1; 3 2; 1 0; 0 1] / 12;

julia> nt = rand(Nonultrametric(4))
RootedTree with 4 tips, 7 nodes and 6 branches.
Leaf names are tip 1, tip 2, tip 4 and tip 3


julia> metaphylo = Metacommunity(communities, PhyloBranches(nt));

julia> raw_meta_rho(metaphylo, [1, 2])
2×8 DataFrame
│ Row │ div_type            │ measure │ q     │ type_level │ type_name │ partition_level │ partition_name │ diversity │
│     │ String              │ String  │ Int64 │ String     │ String    │ String          │ String         │ Float64   │
├─────┼─────────────────────┼─────────┼───────┼────────────┼───────────┼─────────────────┼────────────────┼───────────┤
│ 1   │ Phylogenetic Branch │ RawRho  │ 1     │ types      │           │ metacommunity   │                │ 1.7787    │
│ 2   │ Phylogenetic Branch │ RawRho  │ 2     │ types      │           │ metacommunity   │                │ 1.66749   │

The package also provides some other sub-modules for related measures:

Diversity.Ecology

Many existing ecological diversity measures can be derived from our diversity measures, and so we provide them in the Diversity.Ecology submodule along with generalised versions of them that relate to our general measures of alpha, beta and gamma diversity at subcommunity and metacommunity levels. The generalisations of species richness, Shannon entropy and Simpson's index are the only standard measures we are aware of whose subcommunity components sum directly to the corresponding metacommunity measure (although note that Simpson's index decreases for increased diversity, so small components are more diverse). Documentation for these diversity measures can be found here.

Diversity.Hill

Hill numbers are found in the Diversity.Hill sub-module. Documentation for these diversity measures can be found here.

Diversity.Jost

Lou Jost's diversity measures are found in the Diversity.Jost sub-module. Documentation for these diversity measures is here.

Documentation

Documentation is generated by the Base documentation in Julia and online via the Documenter package.

Using the docs

Accessing the documentation in Julia is easy:

using Diversity

# Returns any documentation for the subdiv() function
?subdiv

The documentation is also available online.

Latest release

The online documentation for the current stable release is here.

Development branch

The online documentation for the latest dev (unreleased) branch is here.

Download Details:

Author: EcoJulia
Source Code: https://github.com/EcoJulia/Diversity.jl 
License: BSD-2-Clause license

#julia #packaging