RWKV Cpp Node: Elevating Projects with AI Mastery

RWKV.cpp NodeJS bindings

Arguably the easiest way to get RWKV.cpp running on node.js

This project primary use case, is to be used as a nodejs library for running RWKV.cpp

The CLI tooling, is simply a helper tooling, to do quick demo's or benchmark via node.js locally

World model is not yet supported

This is not a pure JS solution, and depends on the precompiled RWKV.cpp binaries found here

Additionally V2 breaks compatiblity with V1, due to changes in quantization weights.

Running it as a JS lib

const RWKV = require("RWKV-cpp-node");

// Load the module with the pre-qunatized cpp weights
const raven = new RWKV("<path-to-your-model-bin-files>")

// You must call the setup before completion
await raven.setup();

// Call the completion API
let res = await raven.completion("RWKV is a")

// And log, or do something with the result
console.log( res.completion )

Running it as a CLI

# Install globally
npm install -g rwkv-cpp-node

# This will start the interactive CLI, 
# which will guide you in downloading, and running the chat model
rwkv-cpp-node --setup

# You can run the chat model, with thread count
# / gpu offload % (experimental, not optimized)
rwkv-cpp-node --threads 4 --gpu 100%

# For benchmarking
rwkv-cpp-node --threads 4 --gpu 0 --dragon --size 100

What is RWKV?

RWKV, is a LLM which which can switch between "transformer" and "RNN" mode.

This gives it the best of both worlds

  • High scalable training in transformer
  • Low overheads when infering each token in RNN mode

Along with the following benefits

  • Theoretically Infinite context size
  • Embedding support via hidden states

For more details on the math involved, and how this model works on a more technical basis. Refer to the official project

JS CLI demo

If you just want to give it a spin, the fastest way is to use npm. First perform the setup (it will download the RWKV files into your home directory)

# Install globally
npm install -g rwkv-cpp-node

# First run the setup
rwkv-cpp-node --setup

You can then choose a model to download ...

--setup call detected, starting setup process...
RWKV model will be downloaded into ~/.rwkv/
? Select a RWKV raven model to download:  (Use arrow keys)
❯ RWKV raven 1B5 v11 (Small, Fast) - 2.82 GB 
  RWKV raven 7B v11 (Q8_0) - 8.09 GB 
  RWKV raven 7B v11 (Q8_0, multilingual, performs slightly worse for english) - 8.09 GB 
  RWKV raven 14B v11 (Q8_0) - 15.25 GB 
  RWKV Pile 169M (Q8_0, lacks instruct tuning, use only for testing) - 0.24 GB 

PS: The file size equals to the approximate amount of storage and ram your system needs

Subsequently, you can run the interactive chat mode

# Load the interactive chat

Which would start an interactive shell session, with something like the following

Starting RWKV chat mode
Loading model from /root/.rwkv/raven_1b5_v11.bin ...
The following is a conversation between the user and bot
? User: Hi
Bot:    How can I help you?
? User: Tell me something interesting about ravens
Bot:    RAVEN. I am most fascinated by the raven because of its incredible rate of survival. Ravens have been observed to live longer than any other bird, rumored to reach over 200 years old. They have the ability to live for over 1,000 years, a remarkable feat. This makes them the odd man out among birds!

PS: RWKV like all chat models, can and do lie about stuff.

Finally if you want to run a custom model, or just run the benchmark

# If you want to run with a pre downloaded model
rwkv-cpp-node --modelPath "<path to the model bin file>"

# If you want to run the "--dragon" prompt benchmark
rwkv-cpp-node --dragon
rwkv-cpp-node --modelPath "<path to the model bin file>" --dragon

JS Lib Setup

Install the node module

npm i rwkv-cpp-node

Download one of the prequantized rwkv.cpp weights, from hugging face (raven, is RWKV pretrained weights with fine-tuned instruction sets)

Alternatively you can download one of the raven pretrained weights from the hugging face repo. And perform your own quantization conversion using the original rwkv.cpp project

JS Usage Details

The JS interface for the RWKV model is async/promises based

const RWKV = require("RWKV-cpp-node");

// Load the module with the pre-qunatized cpp weights
const raven = new RWKV({
    path: "<path-to-your-model-bin-files>"
    ... other params ...

// You must call the setup before completion
await raven.setup();

// Call the completion API
let res = await raven.completion("RWKV is a")

// And log, or do something with the result
console.log( res.completion )

Advance setup options

// You can setup with the following parameters with a config object (instead of a string path)
const raven = new RWKV({
    // Path to your cpp weights
    path: "<path-to-your-model-bin-files>",

    // Threads count to use, this is auto detected based on your number of vCPU
    // if its not configured, uses 4 with gpu offloading, else uses half of vCPU detected
    threads: 4,

    // Number of layers (eg. 12), or % of the model (eg: 50%) to offload to the gpu
    // defaults: 0
    gpuOffload: 0,

    // Number of concurrent inferences, the model is cloned while sharing the weights
    // for each concurrent instances configured. This is only useful in server prod env
    // deafults: 1
    concurrent: 1,

    // Batch size of the input to process, this is only useful with gpuOffload
    // Defaults to 64 with gpuOffload, else 1
    // ---
    // batchSize: 64,

    // Cache size for the RKWV state, This help optimize the repeated RWKV calls
    // in use cases such as "conversation", allow it to skip the previous chat computation
    // it is worth noting that the 7B model takes up about 2.64 MB for the state buffer, 
    // meaning you will need atleast 264 MB of RAM for a cachesize of 100
    // This defaults to 50
    // Set to false or 0 to disable
    stateCacheSize: 50
await raven.setup();

Completion API options

// Lets perform a completion, with more options
let res = await raven.completion({

    // The prompt to use
    prompt: "<prompt str>",

    // Completion default settings
    // See openai docs for more details on what these do for your output if you do not understand them
    max_tokens: 64,
    temperature: 1.0,
    top_p: 1.0,
    stop: [ "\n" ],

    // Streaming of output, either token by token, or the full complete output stream
    streamCallback: function(tokenStr, fullCompletionStr) {
        // ....

    // Existing RWKV hidden state, represented as a Flaot32Array
    // do not use this unless you REALLY KNOW WHAT YOUR DOING
    // This will skip the state caching logic 
    initState: (Special Float32Array)

// Additionally if you have a commonly reused instruction set prefix, you can preload this
// using either of the following (requires the stateCacheSize to not be disabled)
await raven.preloadPrompt( "<prompt prefix string>" )
await raven.completion({ prompt:"<prompt prefix string>", max_tokens:0 })

Completion output format

// The following is a sample of the result object format
let resFormat = {
    // Completion generated
    completion: '<completion string used>',

    // Prompt used
    prompt: '<prompt string used>',

    // Token usage numbers
    usage: {
        promptTokens: 41,
        completionTokens: 64,
        totalTokens: 105,
        // number of tokens in the prompt that was previously cached
        promptTokensCached: 39 

    // Performance statistics of the completion operation
    // the following perf numbers is from a single 
    // `Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz`
    // an old 2014 processor, with 28 vCPU 
    // with the 14B model Q8_0 quantized
    perf: {
        // Time taken in ms for each segment
        promptTime: 954,
        completionTime: 35907,
        totalTime: 36861,

        // Time taken in ms to process each token at the respective phase
        timePerPrompt: 477, // This excludes cached tokens
        timePerCompletion: 561.046875,
        timePerFullPrompt: 23.26829268292683, // This includes cached tokens (if any)

        // The average tokens per second
        promptPerSecond: 2.0964360587002098, // This excludes cached tokens
        completionPerSecond: 1.7823822652964603,
        fullPromptPerSecond: 42.9769392033543 // This includes cached tokens (if any)

Want lower level CPP based binding access?

You can call our cpp_bind interface code via

const cpp_bind = require("rwkv-cpp-node").cpp_bind;

// You can find the code here :

What can be improved?

Known issues

  • You need macOS 12 and above

How to run the unit test?

# Download the test model
mkdir -p ./raven/
wget -O raven_1b5_v12_Q8_0.bin 

# Run the test
npm run test

Designated maintainer

@picocreator - is the current maintainer of the project, ping him on the RWKV discord if you have any questions on this project

Special thanks & refrences

@saharNooby - original rwkv.cpp implementation

@BlinkDL - for the main rwkv project

[ THIS IS OUTDATED ] Time taken per token completion for RWKV.cpp v1

Model SizeDownload SizeRAM usageAWS c6g.4xlarge (arm64, 8 Core, 16 vCPU)AWS c6gd.16xlarge (arm64, 32 Core, 64 vCPU)M2 Pro, Mac Mini (6 P core + 4 E core)Oracle A1 (4 Cores)AMD Ryzen 7 3700X (x64, 8 Core, 16 vCPU)
1.5B2.82 GB~ 3.0 GB94.699 ms81.497 ms57.448 ms177.025 ms283.681 ms
3B5.56 GB~ 5.7 GB139.038 ms109.676 ms103.013 ms317.793 ms564.116 ms
7B (Q5_1)5.65 GB~ 7.1 GB  180.137 ms482.916 ms 
7B (Q8_0)8.09 GB~ 8.3 GB167.148 ms126.856 ms140.261 ms382.687 ms406.984 ms
7B13.77 GB~ 14.9 GB259.888 ms175.069 ms210.280 ms733.818 ms729.948 ms
14B (Q8_0)15.25 GB~ 16.4 GB269.201 ms199.114 ms243.889 ms688.014 ms738.947 ms
14B26.36 GB~ 27.9 GB460.963 ms273.277 ms 883.386 ms 

** Note: There are know performance bottleneck issue in the tokenizer, and sampler written in nodejs, as its a single threaded operation, between each "token" in nodejs (which takes ~10ms). And would penalize smaller model more then larger models.

Thanks to @Tomeno & @Cahya for contributing benchmark numbers ofr their A1 and M2 Pro respectively

The above is done by downloading the respective model via rwkv-cpp-node --setup, and performing the rwkv-cpp-node --dragon benchmark. Which would give the following JSON at the end

... output of the benchmark ...

timePerCompletion : is then extracted and used in the above table.

Minor notes: 7B (Q5_1) uses ~ 7.1 GB ram, 7B (Q4_3) uses ~ 6.3 GB ram, making them ideal targets for 8GB ram systems

Download Details:

Author: RWKV
Source Code: 
License: MIT license

#ai #node #modules 

RWKV Cpp Node: Elevating Projects with AI Mastery
1.55 GEEK