Ben Taylor

Ben Taylor

1643269598

What is Elastic Supply Tokens | Why Do We Need Elastic Supply Tokens

Elastic supply tokens have a changing circulating supply. The idea is that instead of price volatility, what changes is the token supply through events called rebases. 

Imagine if the Bitcoin protocol could adjust how much bitcoin is in user wallets to achieve a target price. You have 1 BTC today. You wake up tomorrow, and now you have 2 BTC, but they’re each worth half of what they were yesterday. That’s how a rebase mechanism works.

Introduction

Decentralized Finance (DeFi) has seen an explosion of new types of financial products on the blockchain. We’ve already discussed yield farming, tokenized Bitcoin on Ethereum, Uniswap, and flash loans. One other segment of the crypto space that has been interesting to watch is elastic supply tokens, or rebase tokens.

The unique mechanism behind them allows for a lot of experimentation. Let’s see how these tokens work.

What is an elastic supply token?

An elastic supply (or rebase) token works in a way that the circulating supply expands or contracts due to changes in token price. This increase or decrease in supply works with a mechanism called rebasing. When a rebase occurs, the supply of the token is increased or decreased algorithmically, based on the current price of each token.

In some ways, elastic supply tokens can be paralleled with stablecoins. They aim to achieve a target price, and these rebase mechanics facilitate that. However, the key difference is that rebasing tokens aim to achieve it with a changing (elastic) supply. 

Wait, aren’t many cryptocurrencies operating with a changing supply? Yes, somewhat. Currently, 6.25 new BTC is minted with every block. After the 2024 halving, this is going to be reduced to 3.125 per block. It is a predictable rate, so we can estimate how much BTC will exist next year or after the next halving. 

Supply-elastic tokens work differently. As mentioned, the rebasing mechanism adjusts the token circulating supply periodically. Let’s say we have an elastic supply token that aims to achieve a value of 1 USD. If the price is above 1 USD, the rebase increases the current supply, reducing the value of each token. Conversely, if the price is below 1 USD, the rebase will decrease the supply, making each token worth more.

What does this mean from a practical standpoint? The amount of tokens in user wallets changes if a rebase occurs. Let’s say we have Rebase USD (rUSD), a hypothetical token that targets a price of 1 USD. You have 100 rUSD safely sitting in your hardware wallet. Let’s say the price goes below 1 USD. After the rebase occurs, you’ll have only 96 rUSD in your wallet, but at the same time, each will be worth proportionally more than before the rebase.

The idea is that your holdings proportional to the total supply haven’t changed with the rebase. If you had 1% of the supply before the rebase, you should still have 1% after it, even if the number of coins in your wallet has changed. In essence, you retain your share of the network no matter what the price is.

How Do Elastic Tokens Work?

Tokens with elastic supply operate on a special technique termed rebase. This is an algorithm-based token supply regulation. It’s worth mentioning that during these arrangements customers’ proportional assets eventually aren’t diluted and remain the same.

The goal of rebases is to connect a token with a definite price considering supply and demand. As an instance, let’s view a case when there is a rebase token with the objective to deliver a value of 1 USD. If its price surpasses 1 USD, the algorithm will heighten the existing supply and the cost of a single token will go down. If the cost is under 1 USD, the opposite action takes place. Accordingly, rebases can be positive or negative.

Following a rebase, the quantity of tokens in user wallets modifies accordingly.

At some points, elastic supply tokens have similarities with stablecoins – both of them intend to correspond to the particular price, but the used techniques differ.

  • Stablecoins are described as semi-fixed supply currencies that are governed (as they have the option to mint more coins to corresponding demand when being collateralized). In contrast, rebase tokens are actively adapting supply to reach a non-collateralized peg rate.
  • In the case of fiat collateralized stablecoins (like USDT), there is a trust issue as users need to trust that the other side really holds the stocks it states. So, there is counterparty risk. As for elastic tokens, their work is based on algorithms.
  • Stable coins don’t intend to bring income or trade like stocks. Likewise, elastic tokens are a store of value. Nevertheless, they can bring income given that when the market cap grows, users acquire rebases. To accomplish this, the rebase technique issues recently minted currencies to users, without the dilution of their possessions.

At any rate, it’s essential to be alert of the risks that investing in tokens with not constant supply has. Rebases expend holders’ capital when the rate is growing, but also cause more losses on the way down.

Symmetric and Asymmetric Rebases

Symmetric rebase also called standard rebase suggests changing the number of tokens in the customers’ wallets equally when an adjustment is managed. As for asymmetric (non-standard) rebase it doesn’t impact all wallets in the same way. This means users can volunteer to reduce their token supply. By doing so, they will acquire higher returns in case of a positive rebase.

Originally, elastic supply tokens mainly used asymmetric rebase standard. Afterwards, aiming to reduce the supply contraction, the asymmetric rebase model appeared.

Anyway, no matter what structure of rebasing is adopted, elastic supply tokens should keep relative stability, or they will fail to perform their basic function.

Why Do We Need Elastic Supply Tokens?

Well, it’s a little tricky to understand the use-cases for elastic supply tokens. One might say, elastic supply tokens are quite similar in nature to that of stablecoins. However, there’s one major difference. Stablecoins, on one hand, aims to keep the price stable or pegged to another asset to facilitate the ease of trading.

Whereas, elastic supply tokens, on the other hand, aims for a given target price to fight the volatile nature of cryptocurrencies.

Are there any risks with elastic supply tokens?

Investing in tokens with an elastic price can be considered risky. With elastic supply tokens, the chances of losing funds could be higher. Sure, this can amplify your gains to the upside, but it can also boost your losses. If rebases occur while the token price is going down, you not only lose money from the token price going down, you’ll also own fewer and fewer tokens after each rebase! 

Another reason why investing in elastic supply tokens may be risky is that they are an experimental asset that increases the chances for projects to have bugs in their smart contract code.

Elastic supply tokens are highly risky and very dangerous investments. You should only invest in them if you completely understand what you’re doing. Remember, looking at price charts isn’t going to be all that helpful, since the amount of tokens you hold will change after rebases occur. 

Sure, this can amplify your gains to the upside, but it can also amplify your losses. If rebases occur while the token price is going down, you not only lose money from the token price going down, you’ll also own less and less tokens after each rebase! 

Since they’re quite tricky to understand, investing in rebasing tokens will likely result in a loss for most traders. Only invest in elastic supply tokens if you can fully grasp the mechanisms behind them. Otherwise, you’re not in control of your investment and won’t be able to make informed decisions.

Rebasing token examples

Ampleforth

Ampleforth is one of the first coins to work with an elastic supply. Ampleforth aims to be an uncollateralized synthetic commodity, where 1 AMPL targets a price of 1 USD. Rebases occur once every 24 hours.

The project had relatively little traction until the introduction of a liquidity mining campaign called Geyser. What’s particularly interesting about this scheme is its duration. It distributes tokens for participants over a 10-year period. Geyser is a prime example of how liquidity incentives can create significant traction for a DeFi project.

While technically a stablecoin, the AMPL price chart shows you how volatile elastic supply tokens get.

The AMPL price targets $1, but it can be quite volatile nevertheless.

Bear in mind that this price chart only shows the price of individual AMPL tokens, and doesn’t take into account the changes in supply. Even so, Ampleforth is highly volatile, making it a risky coin to play around with.

It might make more sense to chart elastic supply tokens in terms of market capitalization. Since the price of individual units doesn’t matter as much, the market cap can be a more accurate barometer of the network’s growth and traction.

AMPL market cap on a logarithmic scale.

Yam Finance

Yam Finance is one of the other elastic supply token projects that has gained some traction. The Yam protocol’s overall design is sort of a mashup between Ampleforth’s elastic supply, Synthetix’s staking system, and yearn.finance’s fair launch. YAMs also aims to achieve a price target of 1 USD.

YAM is a completely community-owned experiment, as all tokens were distributed through liquidity mining. There was no premine, no founder allocation – the playing field to acquire these tokens was even for everyone through a yield farming scheme.

As a completely new and unknown project, Yam had achieved 600 million dollars of value locked in its staking pools in less than two days. What may have attracted a lot of liquidity is how YAM farming was specifically targeting the holders of some of the most popular DeFi coins. These were COMP, LEND, LINK, MKR, SNX, ETH, YFI, and ETH-AMPL Uniswap LP tokens.

However, due to a bug in the rebasing mechanism, much more supply was minted as planned. The project was ultimately relaunched and migrated to a new token contract thanks to a community-funded audit and joint effort. The future of Yam is completely in the hands of YAM holders now.

#blockchain #cryptocurrency #tokenization 

What is GEEK

Buddha Community

What is Elastic Supply Tokens | Why Do We Need Elastic Supply Tokens
Ben Taylor

Ben Taylor

1643269598

What is Elastic Supply Tokens | Why Do We Need Elastic Supply Tokens

Elastic supply tokens have a changing circulating supply. The idea is that instead of price volatility, what changes is the token supply through events called rebases. 

Imagine if the Bitcoin protocol could adjust how much bitcoin is in user wallets to achieve a target price. You have 1 BTC today. You wake up tomorrow, and now you have 2 BTC, but they’re each worth half of what they were yesterday. That’s how a rebase mechanism works.

Introduction

Decentralized Finance (DeFi) has seen an explosion of new types of financial products on the blockchain. We’ve already discussed yield farming, tokenized Bitcoin on Ethereum, Uniswap, and flash loans. One other segment of the crypto space that has been interesting to watch is elastic supply tokens, or rebase tokens.

The unique mechanism behind them allows for a lot of experimentation. Let’s see how these tokens work.

What is an elastic supply token?

An elastic supply (or rebase) token works in a way that the circulating supply expands or contracts due to changes in token price. This increase or decrease in supply works with a mechanism called rebasing. When a rebase occurs, the supply of the token is increased or decreased algorithmically, based on the current price of each token.

In some ways, elastic supply tokens can be paralleled with stablecoins. They aim to achieve a target price, and these rebase mechanics facilitate that. However, the key difference is that rebasing tokens aim to achieve it with a changing (elastic) supply. 

Wait, aren’t many cryptocurrencies operating with a changing supply? Yes, somewhat. Currently, 6.25 new BTC is minted with every block. After the 2024 halving, this is going to be reduced to 3.125 per block. It is a predictable rate, so we can estimate how much BTC will exist next year or after the next halving. 

Supply-elastic tokens work differently. As mentioned, the rebasing mechanism adjusts the token circulating supply periodically. Let’s say we have an elastic supply token that aims to achieve a value of 1 USD. If the price is above 1 USD, the rebase increases the current supply, reducing the value of each token. Conversely, if the price is below 1 USD, the rebase will decrease the supply, making each token worth more.

What does this mean from a practical standpoint? The amount of tokens in user wallets changes if a rebase occurs. Let’s say we have Rebase USD (rUSD), a hypothetical token that targets a price of 1 USD. You have 100 rUSD safely sitting in your hardware wallet. Let’s say the price goes below 1 USD. After the rebase occurs, you’ll have only 96 rUSD in your wallet, but at the same time, each will be worth proportionally more than before the rebase.

The idea is that your holdings proportional to the total supply haven’t changed with the rebase. If you had 1% of the supply before the rebase, you should still have 1% after it, even if the number of coins in your wallet has changed. In essence, you retain your share of the network no matter what the price is.

How Do Elastic Tokens Work?

Tokens with elastic supply operate on a special technique termed rebase. This is an algorithm-based token supply regulation. It’s worth mentioning that during these arrangements customers’ proportional assets eventually aren’t diluted and remain the same.

The goal of rebases is to connect a token with a definite price considering supply and demand. As an instance, let’s view a case when there is a rebase token with the objective to deliver a value of 1 USD. If its price surpasses 1 USD, the algorithm will heighten the existing supply and the cost of a single token will go down. If the cost is under 1 USD, the opposite action takes place. Accordingly, rebases can be positive or negative.

Following a rebase, the quantity of tokens in user wallets modifies accordingly.

At some points, elastic supply tokens have similarities with stablecoins – both of them intend to correspond to the particular price, but the used techniques differ.

  • Stablecoins are described as semi-fixed supply currencies that are governed (as they have the option to mint more coins to corresponding demand when being collateralized). In contrast, rebase tokens are actively adapting supply to reach a non-collateralized peg rate.
  • In the case of fiat collateralized stablecoins (like USDT), there is a trust issue as users need to trust that the other side really holds the stocks it states. So, there is counterparty risk. As for elastic tokens, their work is based on algorithms.
  • Stable coins don’t intend to bring income or trade like stocks. Likewise, elastic tokens are a store of value. Nevertheless, they can bring income given that when the market cap grows, users acquire rebases. To accomplish this, the rebase technique issues recently minted currencies to users, without the dilution of their possessions.

At any rate, it’s essential to be alert of the risks that investing in tokens with not constant supply has. Rebases expend holders’ capital when the rate is growing, but also cause more losses on the way down.

Symmetric and Asymmetric Rebases

Symmetric rebase also called standard rebase suggests changing the number of tokens in the customers’ wallets equally when an adjustment is managed. As for asymmetric (non-standard) rebase it doesn’t impact all wallets in the same way. This means users can volunteer to reduce their token supply. By doing so, they will acquire higher returns in case of a positive rebase.

Originally, elastic supply tokens mainly used asymmetric rebase standard. Afterwards, aiming to reduce the supply contraction, the asymmetric rebase model appeared.

Anyway, no matter what structure of rebasing is adopted, elastic supply tokens should keep relative stability, or they will fail to perform their basic function.

Why Do We Need Elastic Supply Tokens?

Well, it’s a little tricky to understand the use-cases for elastic supply tokens. One might say, elastic supply tokens are quite similar in nature to that of stablecoins. However, there’s one major difference. Stablecoins, on one hand, aims to keep the price stable or pegged to another asset to facilitate the ease of trading.

Whereas, elastic supply tokens, on the other hand, aims for a given target price to fight the volatile nature of cryptocurrencies.

Are there any risks with elastic supply tokens?

Investing in tokens with an elastic price can be considered risky. With elastic supply tokens, the chances of losing funds could be higher. Sure, this can amplify your gains to the upside, but it can also boost your losses. If rebases occur while the token price is going down, you not only lose money from the token price going down, you’ll also own fewer and fewer tokens after each rebase! 

Another reason why investing in elastic supply tokens may be risky is that they are an experimental asset that increases the chances for projects to have bugs in their smart contract code.

Elastic supply tokens are highly risky and very dangerous investments. You should only invest in them if you completely understand what you’re doing. Remember, looking at price charts isn’t going to be all that helpful, since the amount of tokens you hold will change after rebases occur. 

Sure, this can amplify your gains to the upside, but it can also amplify your losses. If rebases occur while the token price is going down, you not only lose money from the token price going down, you’ll also own less and less tokens after each rebase! 

Since they’re quite tricky to understand, investing in rebasing tokens will likely result in a loss for most traders. Only invest in elastic supply tokens if you can fully grasp the mechanisms behind them. Otherwise, you’re not in control of your investment and won’t be able to make informed decisions.

Rebasing token examples

Ampleforth

Ampleforth is one of the first coins to work with an elastic supply. Ampleforth aims to be an uncollateralized synthetic commodity, where 1 AMPL targets a price of 1 USD. Rebases occur once every 24 hours.

The project had relatively little traction until the introduction of a liquidity mining campaign called Geyser. What’s particularly interesting about this scheme is its duration. It distributes tokens for participants over a 10-year period. Geyser is a prime example of how liquidity incentives can create significant traction for a DeFi project.

While technically a stablecoin, the AMPL price chart shows you how volatile elastic supply tokens get.

The AMPL price targets $1, but it can be quite volatile nevertheless.

Bear in mind that this price chart only shows the price of individual AMPL tokens, and doesn’t take into account the changes in supply. Even so, Ampleforth is highly volatile, making it a risky coin to play around with.

It might make more sense to chart elastic supply tokens in terms of market capitalization. Since the price of individual units doesn’t matter as much, the market cap can be a more accurate barometer of the network’s growth and traction.

AMPL market cap on a logarithmic scale.

Yam Finance

Yam Finance is one of the other elastic supply token projects that has gained some traction. The Yam protocol’s overall design is sort of a mashup between Ampleforth’s elastic supply, Synthetix’s staking system, and yearn.finance’s fair launch. YAMs also aims to achieve a price target of 1 USD.

YAM is a completely community-owned experiment, as all tokens were distributed through liquidity mining. There was no premine, no founder allocation – the playing field to acquire these tokens was even for everyone through a yield farming scheme.

As a completely new and unknown project, Yam had achieved 600 million dollars of value locked in its staking pools in less than two days. What may have attracted a lot of liquidity is how YAM farming was specifically targeting the holders of some of the most popular DeFi coins. These were COMP, LEND, LINK, MKR, SNX, ETH, YFI, and ETH-AMPL Uniswap LP tokens.

However, due to a bug in the rebasing mechanism, much more supply was minted as planned. The project was ultimately relaunched and migrated to a new token contract thanks to a community-funded audit and joint effort. The future of Yam is completely in the hands of YAM holders now.

#blockchain #cryptocurrency #tokenization 

Words Counted: A Ruby Natural Language Processor.

WordsCounted

We are all in the gutter, but some of us are looking at the stars.

-- Oscar Wilde

WordsCounted is a Ruby NLP (natural language processor). WordsCounted lets you implement powerful tokensation strategies with a very flexible tokeniser class.

Are you using WordsCounted to do something interesting? Please tell me about it.

 

Demo

Visit this website for one example of what you can do with WordsCounted.

Features

  • Out of the box, get the following data from any string or readable file, or URL:
    • Token count and unique token count
    • Token densities, frequencies, and lengths
    • Char count and average chars per token
    • The longest tokens and their lengths
    • The most frequent tokens and their frequencies.
  • A flexible way to exclude tokens from the tokeniser. You can pass a string, regexp, symbol, lambda, or an array of any combination of those types for powerful tokenisation strategies.
  • Pass your own regexp rules to the tokeniser if you prefer. The default regexp filters special characters but keeps hyphens and apostrophes. It also plays nicely with diacritics (UTF and unicode characters): Bayrūt is treated as ["Bayrūt"] and not ["Bayr", "ū", "t"], for example.
  • Opens and reads files. Pass in a file path or a url instead of a string.

Installation

Add this line to your application's Gemfile:

gem 'words_counted'

And then execute:

$ bundle

Or install it yourself as:

$ gem install words_counted

Usage

Pass in a string or a file path, and an optional filter and/or regexp.

counter = WordsCounted.count(
  "We are all in the gutter, but some of us are looking at the stars."
)

# Using a file
counter = WordsCounted.from_file("path/or/url/to/my/file.txt")

.count and .from_file are convenience methods that take an input, tokenise it, and return an instance of WordsCounted::Counter initialized with the tokens. The WordsCounted::Tokeniser and WordsCounted::Counter classes can be used alone, however.

API

WordsCounted

WordsCounted.count(input, options = {})

Tokenises input and initializes a WordsCounted::Counter object with the resulting tokens.

counter = WordsCounted.count("Hello Beirut!")

Accepts two options: exclude and regexp. See Excluding tokens from the analyser and Passing in a custom regexp respectively.

WordsCounted.from_file(path, options = {})

Reads and tokenises a file, and initializes a WordsCounted::Counter object with the resulting tokens.

counter = WordsCounted.from_file("hello_beirut.txt")

Accepts the same options as .count.

Tokeniser

The tokeniser allows you to tokenise text in a variety of ways. You can pass in your own rules for tokenisation, and apply a powerful filter with any combination of rules as long as they can boil down into a lambda.

Out of the box the tokeniser includes only alpha chars. Hyphenated tokens and tokens with apostrophes are considered a single token.

#tokenise([pattern: TOKEN_REGEXP, exclude: nil])

tokeniser = WordsCounted::Tokeniser.new("Hello Beirut!").tokenise

# With `exclude`
tokeniser = WordsCounted::Tokeniser.new("Hello Beirut!").tokenise(exclude: "hello")

# With `pattern`
tokeniser = WordsCounted::Tokeniser.new("I <3 Beirut!").tokenise(pattern: /[a-z]/i)

See Excluding tokens from the analyser and Passing in a custom regexp for more information.

Counter

The WordsCounted::Counter class allows you to collect various statistics from an array of tokens.

#token_count

Returns the token count of a given string.

counter.token_count #=> 15

#token_frequency

Returns a sorted (unstable) two-dimensional array where each element is a token and its frequency. The array is sorted by frequency in descending order.

counter.token_frequency

[
  ["the", 2],
  ["are", 2],
  ["we",  1],
  # ...
  ["all", 1]
]

#most_frequent_tokens

Returns a hash where each key-value pair is a token and its frequency.

counter.most_frequent_tokens

{ "are" => 2, "the" => 2 }

#token_lengths

Returns a sorted (unstable) two-dimentional array where each element contains a token and its length. The array is sorted by length in descending order.

counter.token_lengths

[
  ["looking", 7],
  ["gutter",  6],
  ["stars",   5],
  # ...
  ["in",      2]
]

#longest_tokens

Returns a hash where each key-value pair is a token and its length.

counter.longest_tokens

{ "looking" => 7 }

#token_density([ precision: 2 ])

Returns a sorted (unstable) two-dimentional array where each element contains a token and its density as a float, rounded to a precision of two. The array is sorted by density in descending order. It accepts a precision argument, which must be a float.

counter.token_density

[
  ["are",     0.13],
  ["the",     0.13],
  ["but",     0.07 ],
  # ...
  ["we",      0.07 ]
]

#char_count

Returns the char count of tokens.

counter.char_count #=> 76

#average_chars_per_token([ precision: 2 ])

Returns the average char count per token rounded to two decimal places. Accepts a precision argument which defaults to two. Precision must be a float.

counter.average_chars_per_token #=> 4

#uniq_token_count

Returns the number of unique tokens.

counter.uniq_token_count #=> 13

Excluding tokens from the tokeniser

You can exclude anything you want from the input by passing the exclude option. The exclude option accepts a variety of filters and is extremely flexible.

  1. A space-delimited string. The filter will normalise the string.
  2. A regular expression.
  3. A lambda.
  4. A symbol that names a predicate method. For example :odd?.
  5. An array of any combination of the above.
tokeniser =
  WordsCounted::Tokeniser.new(
    "Magnificent! That was magnificent, Trevor."
  )

# Using a string
tokeniser.tokenise(exclude: "was magnificent")
# => ["that", "trevor"]

# Using a regular expression
tokeniser.tokenise(exclude: /trevor/)
# => ["magnificent", "that", "was", "magnificent"]

# Using a lambda
tokeniser.tokenise(exclude: ->(t) { t.length < 4 })
# => ["magnificent", "that", "magnificent", "trevor"]

# Using symbol
tokeniser = WordsCounted::Tokeniser.new("Hello! محمد")
tokeniser.tokenise(exclude: :ascii_only?)
# => ["محمد"]

# Using an array
tokeniser = WordsCounted::Tokeniser.new(
  "Hello! اسماءنا هي محمد، كارولينا، سامي، وداني"
)
tokeniser.tokenise(
  exclude: [:ascii_only?, /محمد/, ->(t) { t.length > 6}, "و"]
)
# => ["هي", "سامي", "وداني"]

Passing in a custom regexp

The default regexp accounts for letters, hyphenated tokens, and apostrophes. This means twenty-one is treated as one token. So is Mohamad's.

/[\p{Alpha}\-']+/

You can pass your own criteria as a Ruby regular expression to split your string as desired.

For example, if you wanted to include numbers, you can override the regular expression:

counter = WordsCounted.count("Numbers 1, 2, and 3", pattern: /[\p{Alnum}\-']+/)
counter.tokens
#=> ["numbers", "1", "2", "and", "3"]

Opening and reading files

Use the from_file method to open files. from_file accepts the same options as .count. The file path can be a URL.

counter = WordsCounted.from_file("url/or/path/to/file.text")

Gotchas

A hyphen used in leu of an em or en dash will form part of the token. This affects the tokeniser algorithm.

counter = WordsCounted.count("How do you do?-you are well, I see.")
counter.token_frequency

[
  ["do",   2],
  ["how",  1],
  ["you",  1],
  ["-you", 1], # WTF, mate!
  ["are",  1],
  # ...
]

In this example -you and you are separate tokens. Also, the tokeniser does not include numbers by default. Remember that you can pass your own regular expression if the default behaviour does not fit your needs.

A note on case sensitivity

The program will normalise (downcase) all incoming strings for consistency and filters.

Roadmap

Ability to open URLs

def self.from_url
  # open url and send string here after removing html
end

Contributors

See contributors.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Author: abitdodgy
Source code: https://github.com/abitdodgy/words_counted
License: MIT license

#ruby  #ruby-on-rails 

Royce  Reinger

Royce Reinger

1658068560

WordsCounted: A Ruby Natural Language Processor

WordsCounted

We are all in the gutter, but some of us are looking at the stars.

-- Oscar Wilde

WordsCounted is a Ruby NLP (natural language processor). WordsCounted lets you implement powerful tokensation strategies with a very flexible tokeniser class.

Features

  • Out of the box, get the following data from any string or readable file, or URL:
    • Token count and unique token count
    • Token densities, frequencies, and lengths
    • Char count and average chars per token
    • The longest tokens and their lengths
    • The most frequent tokens and their frequencies.
  • A flexible way to exclude tokens from the tokeniser. You can pass a string, regexp, symbol, lambda, or an array of any combination of those types for powerful tokenisation strategies.
  • Pass your own regexp rules to the tokeniser if you prefer. The default regexp filters special characters but keeps hyphens and apostrophes. It also plays nicely with diacritics (UTF and unicode characters): Bayrūt is treated as ["Bayrūt"] and not ["Bayr", "ū", "t"], for example.
  • Opens and reads files. Pass in a file path or a url instead of a string.

Installation

Add this line to your application's Gemfile:

gem 'words_counted'

And then execute:

$ bundle

Or install it yourself as:

$ gem install words_counted

Usage

Pass in a string or a file path, and an optional filter and/or regexp.

counter = WordsCounted.count(
  "We are all in the gutter, but some of us are looking at the stars."
)

# Using a file
counter = WordsCounted.from_file("path/or/url/to/my/file.txt")

.count and .from_file are convenience methods that take an input, tokenise it, and return an instance of WordsCounted::Counter initialized with the tokens. The WordsCounted::Tokeniser and WordsCounted::Counter classes can be used alone, however.

API

WordsCounted

WordsCounted.count(input, options = {})

Tokenises input and initializes a WordsCounted::Counter object with the resulting tokens.

counter = WordsCounted.count("Hello Beirut!")

Accepts two options: exclude and regexp. See Excluding tokens from the analyser and Passing in a custom regexp respectively.

WordsCounted.from_file(path, options = {})

Reads and tokenises a file, and initializes a WordsCounted::Counter object with the resulting tokens.

counter = WordsCounted.from_file("hello_beirut.txt")

Accepts the same options as .count.

Tokeniser

The tokeniser allows you to tokenise text in a variety of ways. You can pass in your own rules for tokenisation, and apply a powerful filter with any combination of rules as long as they can boil down into a lambda.

Out of the box the tokeniser includes only alpha chars. Hyphenated tokens and tokens with apostrophes are considered a single token.

#tokenise([pattern: TOKEN_REGEXP, exclude: nil])

tokeniser = WordsCounted::Tokeniser.new("Hello Beirut!").tokenise

# With `exclude`
tokeniser = WordsCounted::Tokeniser.new("Hello Beirut!").tokenise(exclude: "hello")

# With `pattern`
tokeniser = WordsCounted::Tokeniser.new("I <3 Beirut!").tokenise(pattern: /[a-z]/i)

See Excluding tokens from the analyser and Passing in a custom regexp for more information.

Counter

The WordsCounted::Counter class allows you to collect various statistics from an array of tokens.

#token_count

Returns the token count of a given string.

counter.token_count #=> 15

#token_frequency

Returns a sorted (unstable) two-dimensional array where each element is a token and its frequency. The array is sorted by frequency in descending order.

counter.token_frequency

[
  ["the", 2],
  ["are", 2],
  ["we",  1],
  # ...
  ["all", 1]
]

#most_frequent_tokens

Returns a hash where each key-value pair is a token and its frequency.

counter.most_frequent_tokens

{ "are" => 2, "the" => 2 }

#token_lengths

Returns a sorted (unstable) two-dimentional array where each element contains a token and its length. The array is sorted by length in descending order.

counter.token_lengths

[
  ["looking", 7],
  ["gutter",  6],
  ["stars",   5],
  # ...
  ["in",      2]
]

#longest_tokens

Returns a hash where each key-value pair is a token and its length.

counter.longest_tokens

{ "looking" => 7 }

#token_density([ precision: 2 ])

Returns a sorted (unstable) two-dimentional array where each element contains a token and its density as a float, rounded to a precision of two. The array is sorted by density in descending order. It accepts a precision argument, which must be a float.

counter.token_density

[
  ["are",     0.13],
  ["the",     0.13],
  ["but",     0.07 ],
  # ...
  ["we",      0.07 ]
]

#char_count

Returns the char count of tokens.

counter.char_count #=> 76

#average_chars_per_token([ precision: 2 ])

Returns the average char count per token rounded to two decimal places. Accepts a precision argument which defaults to two. Precision must be a float.

counter.average_chars_per_token #=> 4

#uniq_token_count

Returns the number of unique tokens.

counter.uniq_token_count #=> 13

Excluding tokens from the tokeniser

You can exclude anything you want from the input by passing the exclude option. The exclude option accepts a variety of filters and is extremely flexible.

  1. A space-delimited string. The filter will normalise the string.
  2. A regular expression.
  3. A lambda.
  4. A symbol that names a predicate method. For example :odd?.
  5. An array of any combination of the above.
tokeniser =
  WordsCounted::Tokeniser.new(
    "Magnificent! That was magnificent, Trevor."
  )

# Using a string
tokeniser.tokenise(exclude: "was magnificent")
# => ["that", "trevor"]

# Using a regular expression
tokeniser.tokenise(exclude: /trevor/)
# => ["magnificent", "that", "was", "magnificent"]

# Using a lambda
tokeniser.tokenise(exclude: ->(t) { t.length < 4 })
# => ["magnificent", "that", "magnificent", "trevor"]

# Using symbol
tokeniser = WordsCounted::Tokeniser.new("Hello! محمد")
tokeniser.tokenise(exclude: :ascii_only?)
# => ["محمد"]

# Using an array
tokeniser = WordsCounted::Tokeniser.new(
  "Hello! اسماءنا هي محمد، كارولينا، سامي، وداني"
)
tokeniser.tokenise(
  exclude: [:ascii_only?, /محمد/, ->(t) { t.length > 6}, "و"]
)
# => ["هي", "سامي", "وداني"]

Passing in a custom regexp

The default regexp accounts for letters, hyphenated tokens, and apostrophes. This means twenty-one is treated as one token. So is Mohamad's.

/[\p{Alpha}\-']+/

You can pass your own criteria as a Ruby regular expression to split your string as desired.

For example, if you wanted to include numbers, you can override the regular expression:

counter = WordsCounted.count("Numbers 1, 2, and 3", pattern: /[\p{Alnum}\-']+/)
counter.tokens
#=> ["numbers", "1", "2", "and", "3"]

Opening and reading files

Use the from_file method to open files. from_file accepts the same options as .count. The file path can be a URL.

counter = WordsCounted.from_file("url/or/path/to/file.text")

Gotchas

A hyphen used in leu of an em or en dash will form part of the token. This affects the tokeniser algorithm.

counter = WordsCounted.count("How do you do?-you are well, I see.")
counter.token_frequency

[
  ["do",   2],
  ["how",  1],
  ["you",  1],
  ["-you", 1], # WTF, mate!
  ["are",  1],
  # ...
]

In this example -you and you are separate tokens. Also, the tokeniser does not include numbers by default. Remember that you can pass your own regular expression if the default behaviour does not fit your needs.

A note on case sensitivity

The program will normalise (downcase) all incoming strings for consistency and filters.

Roadmap

Ability to open URLs

def self.from_url
  # open url and send string here after removing html
end

Are you using WordsCounted to do something interesting? Please tell me about it.

Gem Version 

RubyDoc documentation.

Demo

Visit this website for one example of what you can do with WordsCounted.


Contributors

See contributors.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Author: Abitdodgy
Source Code: https://github.com/abitdodgy/words_counted 
License: MIT license

#ruby #nlp 

aaron silva

aaron silva

1622197808

SafeMoon Clone | Create A DeFi Token Like SafeMoon | DeFi token like SafeMoon

SafeMoon is a decentralized finance (DeFi) token. This token consists of RFI tokenomics and auto-liquidity generating protocol. A DeFi token like SafeMoon has reached the mainstream standards under the Binance Smart Chain. Its success and popularity have been immense, thus, making the majority of the business firms adopt this style of cryptocurrency as an alternative.

A DeFi token like SafeMoon is almost similar to the other crypto-token, but the only difference being that it charges a 10% transaction fee from the users who sell their tokens, in which 5% of the fee is distributed to the remaining SafeMoon owners. This feature rewards the owners for holding onto their tokens.

Read More @ https://bit.ly/3oFbJoJ

#create a defi token like safemoon #defi token like safemoon #safemoon token #safemoon token clone #defi token

aaron silva

aaron silva

1621844791

SafeMoon Clone | SafeMoon Token Clone | SafeMoon Token Clone Development

The SafeMoon Token Clone Development is the new trendsetter in the digital world that brought significant changes to benefit the growth of investors’ business in a short period. The SafeMoon token clone is the most widely discussed topic among global users for its value soaring high in the marketplace. The SafeMoon token development is a combination of RFI tokenomics and the auto-liquidity generating process. The SafeMoon token is a replica of decentralized finance (DeFi) tokens that are highly scalable and implemented with tamper-proof security.

The SafeMoon tokens execute efficient functionalities like RFI Static Rewards, Automated Liquidity Provisions, and Automatic Token Burns. The SafeMoon token is considered the most advanced stable coin in the crypto market. It gained global audience attention for managing the stability of asset value without any fluctuations in the marketplace. The SafeMoon token clone is completely decentralized that eliminates the need for intermediaries and benefits the users with less transaction fee and wait time to overtake the traditional banking process.

Reasons to invest in SafeMoon Token Clone :

  • The SafeMoon token clone benefits the investors with Automated Liquidity Pool as a unique feature since it adds more revenue for their business growth in less time. The traders can experience instant trade round the clock for reaping profits with less investment towards the SafeMoon token.
  • It is integrated with high-end security protocols like two-factor authentication and signature process to prevent various hacks and vulnerable activities. The Smart Contract system in SafeMoon token development manages the overall operation of transactions without any delay,
  • The users can obtain a reward amount based on the volume of SafeMoon tokens traded in the marketplace. The efficient trading mechanism allows the users to trade the SafeMoon tokens at the best price for farming. The user can earn higher rewards based on the staking volume of tokens by users in the trade market.
  • It allows the token holders to gain complete ownership over their SafeMoon tokens after purchasing from DeFi exchanges. The SafeMoon community governs the token distribution, price fluctuations, staking, and every other token activity. The community boosts the value of SafeMoon tokens.
  • The Automated Burning tokens result in the community no longer having control over the SafeMoon tokens. Instead, the community can control the burn of the tokens efficiently for promoting its value in the marketplace. The transaction of SafeMoon tokens on the blockchain platform is fast, safe, and secure.

The SafeMoon Token Clone Development is a promising future for upcoming investors and startups to increase their business revenue in less time. The SafeMoon token clone has great demand in the real world among millions of users for its value in the market. Investors can contact leading Infinite Block Tech to gain proper assistance in developing a world-class SafeMoon token clone that increases the business growth in less time.

#safemoon token #safemoon token clone #safemoon token clone development #defi token