What is Theta Fuel (TFUEL) | What is Theta Fuel token | What is TFUEL token

What Is Theta Fuel (TFUEL)?

Theta Fuel (TFUEL) is one of the two native tokens on the Theta blockchain. TFUEL should not be mistaken for  Theta Token (THETA), which is the governance token of the Theta blockchain supported by thousands of community-run Guardian nodes and its Enterprise validators including Google, Binance, Blockchain ventures, Gumi and Samsung.

TFUEL is the second token on the Theta blockchain that serves as the utility token in decentralized video and data delivery, it also acts as a gas token. This means that it is used to power all operations on the Theta blockchain, like payments to relayers for sharing a video stream, for deploying and interacting with smart contracts, and as fees associated with transacting NTFs and DeFi applications.

In addition, it is also used to incentivize users in the ecosystem who share their redundant computing power as Edge Compute nodes as well as bandwidth resources as Edge Caching nodes for video streams and other data.

What’s the difference between the Theta Token and Theta Fuel?

Image for post

Theta Token (THETA): The governance token of the Theta protocol. THETA is used to stake as a Validator or Guardian node, contributing to block production and the protocol governance of the Theta Network. By staking and running a node, users will earn a proportional amount of the new TFUEL generated. The supply of THETA is fixed at 1 billion and will never increase.

Image for post

Theta Fuel (TFUEL): The operational token of the Theta protocol. TFUEL powers on-chain operations like payments to relayers for sharing a video stream, or for deploying or interacting with smart contracts. Relayers earn TFUEL for every video stream they relay to other users on the network. You can think of Theta Fuel as the “gas” of the protocol. At Mainnet launch on March 15th, TFUEL will be created as a native token on the Theta blockchain.

Read more ☞What is THETA Network (THETA) | What is THETA Network token | What is THETA token

How will Theta Fuel be generated? At what rate?

The genesis distribution of Theta Fuel will happen when then Theta mainnet launches on March 15th. For each THETA that you hold when the Theta Mainnet launches, you will also receive 5 TFUEL to seed the ecosystem. To ensure you receive this initial distribution, make sure to follow our  mainnet token swap procedures.

After the initial distribution of 5 billion TFUEL (5 for each of the 1 billion THETA), the supply will increase at an initial annual target rate of 5%. The new supply rate will be determined at the protocol level, and can be adjusted as needed by protocol consensus to provide the appropriate amount of new supply as demanded by platforms on the Theta Network. Each network participant running a Validator or Guardian node will earn part of that new supply of TFUEL, proportional to the amount of THETA they are staking.

Will each viewer need to pay TFUEL to pull video streams on the Theta Network?

Technically that’s true at the protocol level, but the actual model implemented on  SLIVER.tv, and our initial partners like MBN and  Samsung VR, is that the cost of TFUEL falls on the video platform. Platforms subsidize users with the TFUEL necessary to pull video streams from relayers on the Theta Network. This makes sense, because video platforms are the ones most directly gaining from getting more viewers to pull their video stream from the Theta Network, in the form of lower CDN costs and higher user engagement. We think it’s critical that the end-user never has to go out and purchase any TFUEL tokens just to watch videos on the Theta Network — it’s just too much of a friction point for adoption.

Why introduce a second currency at all?

There are two key reasons to introduce a second token:

  1. First, this allows the utility and purpose of each token to be separated. THETA is used strictly for staking and securing the network, while TFUEL is used to power microtransactions and operations of the network. This is necessary because staking inherently decreases circulating supply, but video relay transactions and smart contracts will require a highly-liquid token that can facilitate millions of daily transactions.
  2. Second, two tokens are needed to solve governance issues that arise from using the same token for staking and operations. Because the token used for operations must be liquid, it would be easier for a malicious actor to accumulate a significant number of that frequently-traded token on the open market. If that same token is also used for staking, they could potentially threaten the security of the Theta Network by quietly acquiring a large amount of the staking token. By separating the two functions (staking and operations) into different tokens, that risk is greatly decreased.

Who Are the Founders of Theta Fuel?

Theta Fuel, and in extension, the entire Theta Ecosystem, was co-founded by Mitch Liu and Jieyi Long in 2017. Liu has a long history in the gaming, video and virtual reality industries, co-founding video advertising firm Tapjoy, mobile social gaming startup Gameview Studios, and THETA.tv, the live streaming platform whose DApp was the first to be built on the Theta protocol.

Jieyi Long is Theta’s second co-founder and CTO, following similar multi-year experience in design automation, gaming, VR, and large scale distributed systems. He authored multiple peer-reviewed academic papers and holds various patents in video streaming, blockchain and virtual reality.

Theta now has a modest team, and its official website lists strategic corporate investors as Samsung NEXT, Sony Innovation Fund, media investors BDMI Bertelsmann Digital Media Investments, CAA Creative Artists Agency, and traditional Silicon Valley VCs including DCM, Sierra Ventures and the VR Fund.

What Makes Theta Fuel Unique?

Theta’s main use case is decentralizing video streaming, data delivery and edge computing, making it more efficient, cost-effective and fair for industry participants. The network runs on a native blockchain, with two native tokens, known as Theta (THETA) and Theta Fuel (TFUEL), powering the internal economy.

Theta’s appeal is threefold: viewers get rewarded with better quality streaming service, content creators improve their earnings and middlemen — video platforms — save money on building infrastructure and increase advertising, subscription revenues. Users have an incentive to both watch network content and share network resources, as rewards come in the form of TFUEL tokens.

The platform is open source, and token holders receive governance powers as with many proof-of-stake (PoS)-based blockchain ecosystems. In addition to video, data and computing, Theta caters to developers looking to launch decentralized applications (DApps) such as DeFi and NFTs on its fully featured EVM-compatible smart contract platform.

How Many Theta Fuel (TFUEL) Coins Are There in Circulation?

As earlier mentioned, there are two coins in the Theta ecosystem; THETA and TFUEL. Both coins are used as the main tools for interaction on the Theta blockchain, yet serve different functions.

There are currently 5,232,675,200 TFUEL coins in circulation, which grows each year as new TFUEL is generated for staking rewards. Theta Tokens are fixed and at their max supply, which means there are 1,000,000,000 coins in circulation which will never increase.

Theta runs on the proof-of-stake algorithm, which means staking is possible. Since the amount of THETA in circulation is the max supply there will ever be, TFUEL is the reward for staking on the network. To qualify for staking, you have to stake at least 1,000 THETA to a Guardian Node or run your own Guardian Node.

How Is the Theta Network Secured?

The Theta Blockchain relies on a proof-of-stake consensus algorithm. It, however, adds a multi-level Byzantine Fault Tolerance ( BFT) consensus mechanism on its blockchain to achieve better security along with higher transaction throughput.

At the time of Theta Mainnet launch in March 2019, Theta introduced Guardian Nodes. With the Guardian Node, no single entity controls the majority of THETA tokens being staked at any one time. This further helps the network achieve a high transaction throughput of 1000+ TPS.

Theta uses a financial incentive scheme to ensure user participation in governance activities, and hence its network is secured by its own users.

Would you like to earn TFUEL right now! ☞ CLICK HERE

How and Where to Buy Theta Fuel (TFUEL) ?

TFUEL has been listed on a number of crypto exchanges, unlike other main cryptocurrencies, it cannot be directly purchased with fiats money. However, You can still easily buy this coin by first buying Bitcoin, ETH, USDT from any large exchanges and then transfer to the exchange that offers to trade this coin, in this guide article we will walk you through in detail the steps to buy TFUEL

You will have to first buy one of the major cryptocurrencies, usually either Bitcoin (BTC), Ethereum (ETH), Tether (USDT)…

We will use Binance Exchange here as it is one of the largest crypto exchanges that accept fiat deposits.

Binance is a popular cryptocurrency exchange which was started in China but then moved their headquarters to the crypto-friendly Island of Malta in the EU. Binance is popular for its crypto to crypto exchange services. Binance exploded onto the scene in the mania of 2017 and has since gone on to become the top crypto exchange in the world.

Once you finished the KYC process. You will be asked to add a payment method. Here you can either choose to provide a credit/debit card or use a bank transfer, and buy one of the major cryptocurrencies, usually either Bitcoin (BTC), Ethereum (ETH), Tether (USDT)

SIGN UP ON BINANCE

Step by Step Guide : What is Binance | How to Create an account on Binance (Updated 2021)

After the deposit is confirmed you may then purchase TFUEL from the Binance exchange.

Exchange: Binance, Upbit, Gate.io, BitMart, and Hotbit

Apart from the exchange(s) above, there are a few popular crypto exchanges where they have decent daily trading volumes and a huge user base. This will ensure you will be able to sell your coins at any time and the fees will usually be lower. It is suggested that you also register on these exchanges since once TFUEL gets listed there it will attract a large amount of trading volumes from the users there, that means you will be having some great trading opportunities!

Top exchanges for token-coin trading. Follow instructions and make unlimited money

https://www.binance.com
https://www.bittrex.com
https://www.poloniex.com
https://www.bitfinex.com
https://www.huobi.com
https://www.mxc.ai
https://www.probit.com
https://www.gate.io
https://www.coinbase.com

Find more information TFUEL

WebsiteExplorerWhitepaperSource CodeSocial ChannelSocial Channel 2Social Channel 3Coinmarketcap

Learn about Cryptocurrency in this article ☞ What You Should Know Before Investing in Cryptocurrency - For Beginner

I hope this post will help you. If you liked this, please sharing it with others. Thank you!

#blockchain #bitcoin #theta fuel #tfuel #theta

What is GEEK

Buddha Community

What is Theta Fuel (TFUEL) | What is Theta Fuel token | What is TFUEL token

What is Theta Fuel (TFUEL) | What is Theta Fuel token | What is TFUEL token

What Is Theta Fuel (TFUEL)?

Theta Fuel (TFUEL) is one of the two native tokens on the Theta blockchain. TFUEL should not be mistaken for  Theta Token (THETA), which is the governance token of the Theta blockchain supported by thousands of community-run Guardian nodes and its Enterprise validators including Google, Binance, Blockchain ventures, Gumi and Samsung.

TFUEL is the second token on the Theta blockchain that serves as the utility token in decentralized video and data delivery, it also acts as a gas token. This means that it is used to power all operations on the Theta blockchain, like payments to relayers for sharing a video stream, for deploying and interacting with smart contracts, and as fees associated with transacting NTFs and DeFi applications.

In addition, it is also used to incentivize users in the ecosystem who share their redundant computing power as Edge Compute nodes as well as bandwidth resources as Edge Caching nodes for video streams and other data.

What’s the difference between the Theta Token and Theta Fuel?

Image for post

Theta Token (THETA): The governance token of the Theta protocol. THETA is used to stake as a Validator or Guardian node, contributing to block production and the protocol governance of the Theta Network. By staking and running a node, users will earn a proportional amount of the new TFUEL generated. The supply of THETA is fixed at 1 billion and will never increase.

Image for post

Theta Fuel (TFUEL): The operational token of the Theta protocol. TFUEL powers on-chain operations like payments to relayers for sharing a video stream, or for deploying or interacting with smart contracts. Relayers earn TFUEL for every video stream they relay to other users on the network. You can think of Theta Fuel as the “gas” of the protocol. At Mainnet launch on March 15th, TFUEL will be created as a native token on the Theta blockchain.

Read more ☞What is THETA Network (THETA) | What is THETA Network token | What is THETA token

How will Theta Fuel be generated? At what rate?

The genesis distribution of Theta Fuel will happen when then Theta mainnet launches on March 15th. For each THETA that you hold when the Theta Mainnet launches, you will also receive 5 TFUEL to seed the ecosystem. To ensure you receive this initial distribution, make sure to follow our  mainnet token swap procedures.

After the initial distribution of 5 billion TFUEL (5 for each of the 1 billion THETA), the supply will increase at an initial annual target rate of 5%. The new supply rate will be determined at the protocol level, and can be adjusted as needed by protocol consensus to provide the appropriate amount of new supply as demanded by platforms on the Theta Network. Each network participant running a Validator or Guardian node will earn part of that new supply of TFUEL, proportional to the amount of THETA they are staking.

Will each viewer need to pay TFUEL to pull video streams on the Theta Network?

Technically that’s true at the protocol level, but the actual model implemented on  SLIVER.tv, and our initial partners like MBN and  Samsung VR, is that the cost of TFUEL falls on the video platform. Platforms subsidize users with the TFUEL necessary to pull video streams from relayers on the Theta Network. This makes sense, because video platforms are the ones most directly gaining from getting more viewers to pull their video stream from the Theta Network, in the form of lower CDN costs and higher user engagement. We think it’s critical that the end-user never has to go out and purchase any TFUEL tokens just to watch videos on the Theta Network — it’s just too much of a friction point for adoption.

Why introduce a second currency at all?

There are two key reasons to introduce a second token:

  1. First, this allows the utility and purpose of each token to be separated. THETA is used strictly for staking and securing the network, while TFUEL is used to power microtransactions and operations of the network. This is necessary because staking inherently decreases circulating supply, but video relay transactions and smart contracts will require a highly-liquid token that can facilitate millions of daily transactions.
  2. Second, two tokens are needed to solve governance issues that arise from using the same token for staking and operations. Because the token used for operations must be liquid, it would be easier for a malicious actor to accumulate a significant number of that frequently-traded token on the open market. If that same token is also used for staking, they could potentially threaten the security of the Theta Network by quietly acquiring a large amount of the staking token. By separating the two functions (staking and operations) into different tokens, that risk is greatly decreased.

Who Are the Founders of Theta Fuel?

Theta Fuel, and in extension, the entire Theta Ecosystem, was co-founded by Mitch Liu and Jieyi Long in 2017. Liu has a long history in the gaming, video and virtual reality industries, co-founding video advertising firm Tapjoy, mobile social gaming startup Gameview Studios, and THETA.tv, the live streaming platform whose DApp was the first to be built on the Theta protocol.

Jieyi Long is Theta’s second co-founder and CTO, following similar multi-year experience in design automation, gaming, VR, and large scale distributed systems. He authored multiple peer-reviewed academic papers and holds various patents in video streaming, blockchain and virtual reality.

Theta now has a modest team, and its official website lists strategic corporate investors as Samsung NEXT, Sony Innovation Fund, media investors BDMI Bertelsmann Digital Media Investments, CAA Creative Artists Agency, and traditional Silicon Valley VCs including DCM, Sierra Ventures and the VR Fund.

What Makes Theta Fuel Unique?

Theta’s main use case is decentralizing video streaming, data delivery and edge computing, making it more efficient, cost-effective and fair for industry participants. The network runs on a native blockchain, with two native tokens, known as Theta (THETA) and Theta Fuel (TFUEL), powering the internal economy.

Theta’s appeal is threefold: viewers get rewarded with better quality streaming service, content creators improve their earnings and middlemen — video platforms — save money on building infrastructure and increase advertising, subscription revenues. Users have an incentive to both watch network content and share network resources, as rewards come in the form of TFUEL tokens.

The platform is open source, and token holders receive governance powers as with many proof-of-stake (PoS)-based blockchain ecosystems. In addition to video, data and computing, Theta caters to developers looking to launch decentralized applications (DApps) such as DeFi and NFTs on its fully featured EVM-compatible smart contract platform.

How Many Theta Fuel (TFUEL) Coins Are There in Circulation?

As earlier mentioned, there are two coins in the Theta ecosystem; THETA and TFUEL. Both coins are used as the main tools for interaction on the Theta blockchain, yet serve different functions.

There are currently 5,232,675,200 TFUEL coins in circulation, which grows each year as new TFUEL is generated for staking rewards. Theta Tokens are fixed and at their max supply, which means there are 1,000,000,000 coins in circulation which will never increase.

Theta runs on the proof-of-stake algorithm, which means staking is possible. Since the amount of THETA in circulation is the max supply there will ever be, TFUEL is the reward for staking on the network. To qualify for staking, you have to stake at least 1,000 THETA to a Guardian Node or run your own Guardian Node.

How Is the Theta Network Secured?

The Theta Blockchain relies on a proof-of-stake consensus algorithm. It, however, adds a multi-level Byzantine Fault Tolerance ( BFT) consensus mechanism on its blockchain to achieve better security along with higher transaction throughput.

At the time of Theta Mainnet launch in March 2019, Theta introduced Guardian Nodes. With the Guardian Node, no single entity controls the majority of THETA tokens being staked at any one time. This further helps the network achieve a high transaction throughput of 1000+ TPS.

Theta uses a financial incentive scheme to ensure user participation in governance activities, and hence its network is secured by its own users.

Would you like to earn TFUEL right now! ☞ CLICK HERE

How and Where to Buy Theta Fuel (TFUEL) ?

TFUEL has been listed on a number of crypto exchanges, unlike other main cryptocurrencies, it cannot be directly purchased with fiats money. However, You can still easily buy this coin by first buying Bitcoin, ETH, USDT from any large exchanges and then transfer to the exchange that offers to trade this coin, in this guide article we will walk you through in detail the steps to buy TFUEL

You will have to first buy one of the major cryptocurrencies, usually either Bitcoin (BTC), Ethereum (ETH), Tether (USDT)…

We will use Binance Exchange here as it is one of the largest crypto exchanges that accept fiat deposits.

Binance is a popular cryptocurrency exchange which was started in China but then moved their headquarters to the crypto-friendly Island of Malta in the EU. Binance is popular for its crypto to crypto exchange services. Binance exploded onto the scene in the mania of 2017 and has since gone on to become the top crypto exchange in the world.

Once you finished the KYC process. You will be asked to add a payment method. Here you can either choose to provide a credit/debit card or use a bank transfer, and buy one of the major cryptocurrencies, usually either Bitcoin (BTC), Ethereum (ETH), Tether (USDT)

SIGN UP ON BINANCE

Step by Step Guide : What is Binance | How to Create an account on Binance (Updated 2021)

After the deposit is confirmed you may then purchase TFUEL from the Binance exchange.

Exchange: Binance, Upbit, Gate.io, BitMart, and Hotbit

Apart from the exchange(s) above, there are a few popular crypto exchanges where they have decent daily trading volumes and a huge user base. This will ensure you will be able to sell your coins at any time and the fees will usually be lower. It is suggested that you also register on these exchanges since once TFUEL gets listed there it will attract a large amount of trading volumes from the users there, that means you will be having some great trading opportunities!

Top exchanges for token-coin trading. Follow instructions and make unlimited money

https://www.binance.com
https://www.bittrex.com
https://www.poloniex.com
https://www.bitfinex.com
https://www.huobi.com
https://www.mxc.ai
https://www.probit.com
https://www.gate.io
https://www.coinbase.com

Find more information TFUEL

WebsiteExplorerWhitepaperSource CodeSocial ChannelSocial Channel 2Social Channel 3Coinmarketcap

Learn about Cryptocurrency in this article ☞ What You Should Know Before Investing in Cryptocurrency - For Beginner

I hope this post will help you. If you liked this, please sharing it with others. Thank you!

#blockchain #bitcoin #theta fuel #tfuel #theta

Linda nano

Linda nano

1624392000

What is #TFUEL and why #THETA is the future of video streaming!

In this video I discuss the dual token model of $theta token and $tfuel. These tokens will disrupt the entire video streaming industry, and will revolutionize it. What do you think of theta and tfuel?!

📺 The video in this post was made by Crypto 2103
The origin of the article: https://www.youtube.com/watch?v=dst1tR5FI7M
🔺 DISCLAIMER: The article is for information sharing. The content of this video is solely the opinions of the speaker who is not a licensed financial advisor or registered investment advisor. Not investment advice or legal advice.
Cryptocurrency trading is VERY risky. Make sure you understand these risks and that you are responsible for what you do with your money
🔥 If you’re a beginner. I believe the article below will be useful to you ☞ What You Should Know Before Investing in Cryptocurrency - For Beginner
⭐ ⭐ ⭐The project is of interest to the community. Join to Get free ‘GEEK coin’ (GEEKCASH coin)!
☞ **-----CLICK HERE-----**⭐ ⭐ ⭐
Thanks for visiting and watching! Please don’t forget to leave a like, comment and share!

#bitcoin #blockchain #theta #tfuel #what is #tfuel #what is #tfuel and why #theta is the future of video streaming!

ACACIA  ADAM

ACACIA ADAM

1623457800

How To Buy Theta Fuel Crypto Coin On Binance (TFUEL Token)

In this video I will show you how to buy theta fuel crypto coin on binance. It’s really easy and it will take you less than a minute to do so!

📺 The video in this post was made by How To Explained
️ The origin of the article: https://www.youtube.com/watch?v=_iw-obZ_OHc

🔺 DISCLAIMER: The article is for information sharing. The content of this video is solely the opinions of the speaker who is not a licensed financial advisor or registered investment advisor. Not investment advice or legal advice.
Cryptocurrency trading is VERY risky. Make sure you understand these risks and that you are responsible for what you do with your money
🔥 If you’re a beginner. I believe the article below will be useful to you ☞ What You Should Know Before Investing in Cryptocurrency - For Beginner
⭐ ⭐ ⭐The project is of interest to the community. Join to Get free ‘GEEK coin’ (GEEKCASH coin)!
⭐ ⭐ ⭐ Join to Get free ‘GEEK coin’ (GEEKCASH coin)! ☞ https://geekcash.org⭐ ⭐ ⭐
(There is no limit to the amount of credit you can earn through referrals)
Thanks for visiting and watching! Please don’t forget to leave a like, comment and share!

#bitcoin #blockchain #theta fuel #binance (tfuel token) #crypto #coin

Linda nano

Linda nano

1624205280

$THETA Tfuel BIG Update - THE NEXT BIG ONE

Theta and tfuel will decenentralize data deliver, what a wonderful project. What do you think?
What is #TFUEL and why #THETA is the future of video streaming!
📺 The video in this post was made by Crypto 2103
The origin of the article: https://www.youtube.com/watch?v=iYQuOlG_K7A
🔺 DISCLAIMER: The article is for information sharing. The content of this video is solely the opinions of the speaker who is not a licensed financial advisor or registered investment advisor. Not investment advice or legal advice.
Cryptocurrency trading is VERY risky. Make sure you understand these risks and that you are responsible for what you do with your money
🔥 If you’re a beginner. I believe the article below will be useful to you ☞ What You Should Know Before Investing in Cryptocurrency - For Beginner
⭐ ⭐ ⭐The project is of interest to the community. Join to Get free ‘GEEK coin’ (GEEKCASH coin)!
☞ **-----CLICK HERE-----**⭐ ⭐ ⭐
Thanks for visiting and watching! Please don’t forget to leave a like, comment and share!

#bitcoin #blockchain #$theta #$theta tfuel big update #$theta tfuel big update - the next big one

Words Counted: A Ruby Natural Language Processor.

WordsCounted

We are all in the gutter, but some of us are looking at the stars.

-- Oscar Wilde

WordsCounted is a Ruby NLP (natural language processor). WordsCounted lets you implement powerful tokensation strategies with a very flexible tokeniser class.

Are you using WordsCounted to do something interesting? Please tell me about it.

 

Demo

Visit this website for one example of what you can do with WordsCounted.

Features

  • Out of the box, get the following data from any string or readable file, or URL:
    • Token count and unique token count
    • Token densities, frequencies, and lengths
    • Char count and average chars per token
    • The longest tokens and their lengths
    • The most frequent tokens and their frequencies.
  • A flexible way to exclude tokens from the tokeniser. You can pass a string, regexp, symbol, lambda, or an array of any combination of those types for powerful tokenisation strategies.
  • Pass your own regexp rules to the tokeniser if you prefer. The default regexp filters special characters but keeps hyphens and apostrophes. It also plays nicely with diacritics (UTF and unicode characters): Bayrūt is treated as ["Bayrūt"] and not ["Bayr", "ū", "t"], for example.
  • Opens and reads files. Pass in a file path or a url instead of a string.

Installation

Add this line to your application's Gemfile:

gem 'words_counted'

And then execute:

$ bundle

Or install it yourself as:

$ gem install words_counted

Usage

Pass in a string or a file path, and an optional filter and/or regexp.

counter = WordsCounted.count(
  "We are all in the gutter, but some of us are looking at the stars."
)

# Using a file
counter = WordsCounted.from_file("path/or/url/to/my/file.txt")

.count and .from_file are convenience methods that take an input, tokenise it, and return an instance of WordsCounted::Counter initialized with the tokens. The WordsCounted::Tokeniser and WordsCounted::Counter classes can be used alone, however.

API

WordsCounted

WordsCounted.count(input, options = {})

Tokenises input and initializes a WordsCounted::Counter object with the resulting tokens.

counter = WordsCounted.count("Hello Beirut!")

Accepts two options: exclude and regexp. See Excluding tokens from the analyser and Passing in a custom regexp respectively.

WordsCounted.from_file(path, options = {})

Reads and tokenises a file, and initializes a WordsCounted::Counter object with the resulting tokens.

counter = WordsCounted.from_file("hello_beirut.txt")

Accepts the same options as .count.

Tokeniser

The tokeniser allows you to tokenise text in a variety of ways. You can pass in your own rules for tokenisation, and apply a powerful filter with any combination of rules as long as they can boil down into a lambda.

Out of the box the tokeniser includes only alpha chars. Hyphenated tokens and tokens with apostrophes are considered a single token.

#tokenise([pattern: TOKEN_REGEXP, exclude: nil])

tokeniser = WordsCounted::Tokeniser.new("Hello Beirut!").tokenise

# With `exclude`
tokeniser = WordsCounted::Tokeniser.new("Hello Beirut!").tokenise(exclude: "hello")

# With `pattern`
tokeniser = WordsCounted::Tokeniser.new("I <3 Beirut!").tokenise(pattern: /[a-z]/i)

See Excluding tokens from the analyser and Passing in a custom regexp for more information.

Counter

The WordsCounted::Counter class allows you to collect various statistics from an array of tokens.

#token_count

Returns the token count of a given string.

counter.token_count #=> 15

#token_frequency

Returns a sorted (unstable) two-dimensional array where each element is a token and its frequency. The array is sorted by frequency in descending order.

counter.token_frequency

[
  ["the", 2],
  ["are", 2],
  ["we",  1],
  # ...
  ["all", 1]
]

#most_frequent_tokens

Returns a hash where each key-value pair is a token and its frequency.

counter.most_frequent_tokens

{ "are" => 2, "the" => 2 }

#token_lengths

Returns a sorted (unstable) two-dimentional array where each element contains a token and its length. The array is sorted by length in descending order.

counter.token_lengths

[
  ["looking", 7],
  ["gutter",  6],
  ["stars",   5],
  # ...
  ["in",      2]
]

#longest_tokens

Returns a hash where each key-value pair is a token and its length.

counter.longest_tokens

{ "looking" => 7 }

#token_density([ precision: 2 ])

Returns a sorted (unstable) two-dimentional array where each element contains a token and its density as a float, rounded to a precision of two. The array is sorted by density in descending order. It accepts a precision argument, which must be a float.

counter.token_density

[
  ["are",     0.13],
  ["the",     0.13],
  ["but",     0.07 ],
  # ...
  ["we",      0.07 ]
]

#char_count

Returns the char count of tokens.

counter.char_count #=> 76

#average_chars_per_token([ precision: 2 ])

Returns the average char count per token rounded to two decimal places. Accepts a precision argument which defaults to two. Precision must be a float.

counter.average_chars_per_token #=> 4

#uniq_token_count

Returns the number of unique tokens.

counter.uniq_token_count #=> 13

Excluding tokens from the tokeniser

You can exclude anything you want from the input by passing the exclude option. The exclude option accepts a variety of filters and is extremely flexible.

  1. A space-delimited string. The filter will normalise the string.
  2. A regular expression.
  3. A lambda.
  4. A symbol that names a predicate method. For example :odd?.
  5. An array of any combination of the above.
tokeniser =
  WordsCounted::Tokeniser.new(
    "Magnificent! That was magnificent, Trevor."
  )

# Using a string
tokeniser.tokenise(exclude: "was magnificent")
# => ["that", "trevor"]

# Using a regular expression
tokeniser.tokenise(exclude: /trevor/)
# => ["magnificent", "that", "was", "magnificent"]

# Using a lambda
tokeniser.tokenise(exclude: ->(t) { t.length < 4 })
# => ["magnificent", "that", "magnificent", "trevor"]

# Using symbol
tokeniser = WordsCounted::Tokeniser.new("Hello! محمد")
tokeniser.tokenise(exclude: :ascii_only?)
# => ["محمد"]

# Using an array
tokeniser = WordsCounted::Tokeniser.new(
  "Hello! اسماءنا هي محمد، كارولينا، سامي، وداني"
)
tokeniser.tokenise(
  exclude: [:ascii_only?, /محمد/, ->(t) { t.length > 6}, "و"]
)
# => ["هي", "سامي", "وداني"]

Passing in a custom regexp

The default regexp accounts for letters, hyphenated tokens, and apostrophes. This means twenty-one is treated as one token. So is Mohamad's.

/[\p{Alpha}\-']+/

You can pass your own criteria as a Ruby regular expression to split your string as desired.

For example, if you wanted to include numbers, you can override the regular expression:

counter = WordsCounted.count("Numbers 1, 2, and 3", pattern: /[\p{Alnum}\-']+/)
counter.tokens
#=> ["numbers", "1", "2", "and", "3"]

Opening and reading files

Use the from_file method to open files. from_file accepts the same options as .count. The file path can be a URL.

counter = WordsCounted.from_file("url/or/path/to/file.text")

Gotchas

A hyphen used in leu of an em or en dash will form part of the token. This affects the tokeniser algorithm.

counter = WordsCounted.count("How do you do?-you are well, I see.")
counter.token_frequency

[
  ["do",   2],
  ["how",  1],
  ["you",  1],
  ["-you", 1], # WTF, mate!
  ["are",  1],
  # ...
]

In this example -you and you are separate tokens. Also, the tokeniser does not include numbers by default. Remember that you can pass your own regular expression if the default behaviour does not fit your needs.

A note on case sensitivity

The program will normalise (downcase) all incoming strings for consistency and filters.

Roadmap

Ability to open URLs

def self.from_url
  # open url and send string here after removing html
end

Contributors

See contributors.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Author: abitdodgy
Source code: https://github.com/abitdodgy/words_counted
License: MIT license

#ruby  #ruby-on-rails