Idris Brhane

Idris Brhane

1605984060

Streaming Json Parser in Javascript for Node.js and The Browser

@streamparser/json

Fast dependency-free library to parse a JSON stream using utf-8 encoding in Node.js, Deno or any modern browser. Fully compliant with the JSON spec and JSON.parse(...).

tldr;

import { JSONparser } from '@streamparser/json';

const parser = new JSONparser();
parser.onValue = (value) => { /* process data */}

// Or passing the stream in several chunks 
try {
  parser.write('{ "test": ["a"] }');
  // onValue will be called 3 times:
  // "a"
  // ["a"]
  // { test: ["a"] }
} catch (err) {
  console.log(err); // handler errors 
}

Dependencies / Polyfilling

@streamparser/json requires a few ES6 classes:

If you are targeting browsers or systems in which these might be missing, you need to polyfil them.

Components

Tokenizer

A JSON compliant tokenizer that parses a utf-8 stream into JSON tokens

import { Tokenizer } from '@streamparser/json';

const tokenizer = new Tokenizer(opts);

The available options are:

{
  stringBufferSize: <bufferSize>, // set to 0 to don't buffer. Min valid value is 4.
  numberBufferSize: <bufferSize>, // set to 0 to don't buffer
  separator: <string>, // separator between object. For example `\n` for nd-js.
}

If buffer sizes are set to anything else than zero, instead of using a string to apppend the data as it comes in, the data is buffered using a TypedArray. A reasonable size could be 64 * 1024 (64 KB).

Buffering

When parsing strings or numbers, the parser needs to gather the data in-memory until the whole value is ready.

Strings are inmutable in Javascript so every string operation creates a new string. The V8 engine, behind Node, Deno and most modern browsers, performs a many different types of optimization. One of this optimizations is to over-allocate memory when it detects many string concatenations. This increases significatly the memory consumption and can easily exhaust your memory when parsing JSON containing very large strings or numbers. For those cases, the parser can buffer the characters using a TypedArray. This requires encoding/decoding from/to the buffer into an actual string once the value is ready. This is done using the TextEncoder and TextDecoder APIs. Unfortunately, these APIs creates a significant overhead when the strings are small so should be used only when strictly necessary.

Properties & Methods
  • write(data: string|typedArray|buffer) push data into the tokenizer.
  • end() closes the tokenizer so it can not be used anymore. Throws an error if the tokenizer was in the middle of parsing.
  • isEnded readonly boolean property indicating whether the Tokenizer is ended or is still accepting data.
  • parseNumber(numberStr) method used internally to parse numbers. By default, it is equivalent to Number(numberStr) but the user can override it if he wants some other behaviour.
  • onToken(token: TokenType, value: any, offset: number) no-op method that the user should override to follow the tokenization process.
  • onError(err: Error) no-op method that the user can override to act on errors. If not set, the write method simply throws synchronously.
  • onEnd() no-op method that the user can override to act when the tokenizer is ended.
// You can override the overridable methods by creating your own class extending Tokenizer
class MyTokenizer extends Tokenizer {
  parseNumber(numberStr) {
    const number = super.parseNumber(numberStr);
    // if number is too large. Just keep the string.
    return Number.isFinite(numberStr)) ? number : numberStr;
  }
  onToken(token: TokenType, value: any) {
    if (token = TokenTypes.NUMBER && typeof value === 'string') {
      super(TokenTypes.STRING, value);
    } else {
      super(token, value);
    }
  }
}

const myTokenizer = new MyTokenizer();

// or just overriding it
const tokenizer = new Tokenizer();
tokenizer.parseNumber = (numberStr) => { ... };
tokenizer.onToken = (token, value, offset) => { ... };

TokenParser

A token parser that processes JSON tokens as emitted by the Tokenizer and emits JSON values/objects.

import { TokenParser} from '@streamparser/json';

const tokenParser = new TokenParser(opts);

The available options are:

{
  paths: <string[]>,
  keepStack: <boolean>, // whether to keep all the properties in the stack
  separator: <string>, // separator between object. For example `\n` for nd-js. If left empty or set to undefined, the token parser will end after parsing the first object. To parse multiple object without any delimiter just set it to the empty string `''`.
}
  • paths: Array of paths to emit. Defaults to undefined which emits everything. The paths are intended to suppot jsonpath although at the time being it only supports the root object selector ($) and subproperties selectors including wildcards ($.a, $.*, $.a.b, , $.*.b, etc).
  • keepStack: Whether to keep full objects on the stack even if they won’t be emitted. Defaults to true. When set to false the it does preserve properties in the parent object some ancestor will be emitted. This means that the parent object passed to the onValue function will be empty, which doesn’t reflect the truth, but it’s more memory-efficient.
Properties & Methods
  • write(token: TokenType, value: any) push data into the token parser.
  • end() closes the token parser so it can not be used anymore. Throws an error if the tokenizer was in the middle of parsing.
  • isEnded readonly boolean property indicating whether the token parser is ended or is still accepting data.
  • onValue(value: any) no-op method that the user should override to get the parsed value.
  • onError(err: Error) no-op method that the user should override to act on errors. If not set, the write method simply throws synchronously.
  • onEnd() no-op method that the user should override to act when the token parser is ended.
// You can override the overridable methods by creating your own class extending Tokenizer
class MyTokenParser extends TokenParser {
  onValue(value: any) {
    // ...
  }
}

const myTokenParser = new MyTokenParser();

// or just overriding it
const tokenParser = new TokenParser();
tokenParser.onValue = (value) => { ... };

JSONparser

A drop-in replacement of JSONparse (with few breaking changes improvements. See below.).

import { JSONParser } from '@streamparser/json';

const parser = new JSONParser();

It takes the same options as the tokenizer.

This class is just for convenience. In reality, it simply connects the tokenizer and the parser:

const tokenizer = new Tokenizer(opts);
const tokenParser = new TokenParser();
tokenizer.onToken = this.tokenParser.write.bind(this.parser);
tokenParser.onValue = (value) => { /* Process values */ }
Properties & Methods
  • write(token: TokenType, value: any) alias to the Tokenizer write method.
  • end() alias to the Tokenizer end method.
  • isEnded readonly boolean property indicating whether the JSONparser is ended or is still accepting data.
  • onToken(token: TokenType, value: any, offset: number) alias to the Tokenizer onToken method. (write only).
  • onValue(value: any) alias to the Token Parser onValue method (write only).
  • onError(err: Error) alias to the Tokenizer/Token Parser onError method (write only).
  • onEnd() alias to the Tokenizer onEnd method (which will call the Token Parser onEnd methods) (write only).
// You can override the overridable methods by creating your own class extending Tokenizer
class MyJsonParser extends JSONParser {
  onToken(value: any) {
    // ...
  }
  onValue(value: any) {
    // ...
  }
}

const myJsonParser = new MyJsonParser();

// or just overriding it
const jsonParser = new JSONParser();
jsonParser.onToken = (token, value, offset) => { ... };
jsonParser.onValue = (value) => { ... };

Usage

You can use both components independently as

const tokenizer = new Tokenizer(opts);
const tokenParser = new TokenParser();
this.tokenizer.onToken = this.tokenParser.write.bind(this.tokenParser);

You push data using the write method which takes a string or an array-like object.

You can subscribe to the resulting data using the

import { JSONParser } from '@streamparser/json';

const parser = new JSONParser({ stringBufferSize: undefined, paths: ['$'] });
parser.onValue = console.log;

parser.write('"Hello world!"'); // logs "Hello world!"

// Or passing the stream in several chunks 
parser.write('"');
parser.write('Hello');
parser.write(' ');
parser.write('world!');
parser.write('"');// logs "Hello world!"

Write is always a synchronous operation so any error during the parsing of the stream will be thrown during the write operation. After an error, the parser can’t continue parsing.

import { JSONParser } from '@streamparser/json';

const parser = new JSONParser({ stringBufferSize: undefined });
parser.onValue = console.log;

try {
  parser.write('"""');
} catch (err) {
  console.log(err); // logs 
}

You can also handle errors using callbacks:

import { JSONParser } from '@streamparser/json';

const parser = new JSONParser({ stringBufferSize: undefined });
parser.onValue = console.log;
parser.onError = console.error;

parser.write('"""');

Examples

Stream-parsing a fetch request returning a JSONstream

Imagine an endpoint that send a large amount of JSON objects one after the other ({"id":1}{"id":2}{"id":3}...).

  import { JSONParser} from '@streamparser/json';

  const jsonparser = new JSONParser();
  jsonparser.onValue = (value, key, parent, stack) => {
	if (stack > 0) return; // ignore inner values
    // TODO process element
  }

  const response = await fetch('http://example.com/');
  const reader = response.body.getReader();
  while(true) {
    const { done, value } = await reader.read();
    if (done) break;
    jsonparser.write(value);
  }

Stream-parsing a fetch request returning a JSON array

Imagine an endpoint that send a large amount of JSON objects one after the other ([{"id":1},{"id":2},{"id":3},...]).

  import { JSONParser } from '@streamparser/json';

  const jsonparser = new JSONParser({ stringBufferSize: undefined, paths: ['$.*'] });
  jsonparser.onValue = (value, key, parent, stack) => {
    if (stack.length === 0) /* We are done. Exit. */; 
    // By default, the parser keeps all the child elements in memory until the root parent is emitted.
    // Let's delete the objects after processing them in order to optimize memory.
    delete parent[key];
    // TODO process `value` which will be each of the values in the array.
  }

  const response = await fetch('http://example.com/');
  const reader = response.body.getReader();
  while(true) {
    const { done, value } = await reader.read();
    if (done) break;
    jsonparser.write(value);
  }

Why building this if we have JSONparse

JSONParser was awesome… in 2011.

@streamparser/json strengths include:

  • As performant as the original an even faster in some cases.
  • Works on the browser.
  • Allows selector of what to emit.
  • Well documented.
  • Better designed and more plugable/configurable by clearly separating the tokenizer and token parser processes.
  • Simpler and cleaner code. Uses ES6 and doesn’t rely on deprecated Node.js methods.
  • 100% unit test coverage.
  • Fully compliant with the JSON spec. You will always get the same result as using JSON.parse().

Breaking changes Improvements compared to JSONparse

  • JSONparse errors keep big number as a string which is not compliant with the spec. With @streamparser/json you can achieve such behaviour by simply overriding the parseNumber method.
  • JSONparse errors on characters above 244 which is not compliant with the spec. @streamparser/json parsed them correctly.
  • JSONparse incorrectly allows trailing comas in objects or arrays which is not compliant with the spec. @streamparser/json do not.
  • JSONparse’s uses the onError callback to handle errors. Since the write method is synchronous, @streamparser/json defaults to throwing on error, so wrapping the write operation in a try-catch block captures all possible errors. If the onError callback is set, nothing is thrown.
  • JSONparse uses buffers to parse strings to avoid memory exhaustion if your JSON include very long strings (due to V8 optimizations). This has a performance impact and it is not necessary for most use cases. @streamparser/json uses a string as internal buffer by default to improve performance and allows the user to get the exact same behaviour as in JSONparse by setting the stringBufferSize option to 64 * 1024.
  • JSONparse parses all valid JSON objects that come through the stream and doesn’t support ending the processing. @streamparser/json ends the processing after a single object unless the user explicitly configure a separator. When using a separator, the user can end the processing by calling the end method which will end the processing and throw and error if the stream is in the middle of parsing something i.e. the JSON passed so far was incomplete/incorrect. Users can use the onEnd callback to act when the processing ends.
  • JSONparse will fail to emit a number until is followed by a non-numeric character, i.e. it will not parse a single number which is valid JSON. @streamparser/json uses the end method to emit any possible number that was being parsed before completely ending the processing.

Download Details:

Author: juanjoDiaz

Source Code: https://github.com/juanjoDiaz/streamparser-json

#nodejs #node #javascript

What is GEEK

Buddha Community

Streaming Json Parser in Javascript for Node.js and The Browser
Marlon  Boyle

Marlon Boyle

1589634120

Hands on with Node.Js Streams | Examples & Approach

Never heard of Node.js? Node.js is an accessible asynchronous environment based on Javascript which contains several core modules helpful for performing various tasks. Node.js is famous worldwide due to its efficiency and being open-source, it brings a lot to the table. Node.js allows the developers to handle multiple requests on a single thread and thereby allowing them more breathing space.

Node.js handles data using two approaches – Buffered and Streamed. In the buffered approach, you have to write the entire data before the receiver may read it. Such an approach doesn’t support its asynchronous paradigm. When it comes to the Streamed approach, the information starts the interpreting process as soon as you enter it.

Before you read further, we would like to inform you that this article is about streams. Streams are an essential part of the Node.js environment. What it stream, and what do they do? What are the different types of streams? We have tried to cover several important questions that may help you in understanding Node.js Streams. Let’s get started.

#nodejs #streams in node.js #using streams in node js #node.js streams #node.js tutorial #data streams

Hire Dedicated Node.js Developers - Hire Node.js Developers

If you look at the backend technology used by today’s most popular apps there is one thing you would find common among them and that is the use of NodeJS Framework. Yes, the NodeJS framework is that effective and successful.

If you wish to have a strong backend for efficient app performance then have NodeJS at the backend.

WebClues Infotech offers different levels of experienced and expert professionals for your app development needs. So hire a dedicated NodeJS developer from WebClues Infotech with your experience requirement and expertise.

So what are you waiting for? Get your app developed with strong performance parameters from WebClues Infotech

For inquiry click here: https://www.webcluesinfotech.com/hire-nodejs-developer/

Book Free Interview: https://bit.ly/3dDShFg

#hire dedicated node.js developers #hire node.js developers #hire top dedicated node.js developers #hire node.js developers in usa & india #hire node js development company #hire the best node.js developers & programmers

Aria Barnes

Aria Barnes

1622719015

Why use Node.js for Web Development? Benefits and Examples of Apps

Front-end web development has been overwhelmed by JavaScript highlights for quite a long time. Google, Facebook, Wikipedia, and most of all online pages use JS for customer side activities. As of late, it additionally made a shift to cross-platform mobile development as a main technology in React Native, Nativescript, Apache Cordova, and other crossover devices. 

Throughout the most recent couple of years, Node.js moved to backend development as well. Designers need to utilize a similar tech stack for the whole web project without learning another language for server-side development. Node.js is a device that adjusts JS usefulness and syntax to the backend. 

What is Node.js? 

Node.js isn’t a language, or library, or system. It’s a runtime situation: commonly JavaScript needs a program to work, however Node.js makes appropriate settings for JS to run outside of the program. It’s based on a JavaScript V8 motor that can run in Chrome, different programs, or independently. 

The extent of V8 is to change JS program situated code into machine code — so JS turns into a broadly useful language and can be perceived by servers. This is one of the advantages of utilizing Node.js in web application development: it expands the usefulness of JavaScript, permitting designers to coordinate the language with APIs, different languages, and outside libraries.

What Are the Advantages of Node.js Web Application Development? 

Of late, organizations have been effectively changing from their backend tech stacks to Node.js. LinkedIn picked Node.js over Ruby on Rails since it took care of expanding responsibility better and decreased the quantity of servers by multiple times. PayPal and Netflix did something comparative, just they had a goal to change their design to microservices. We should investigate the motivations to pick Node.JS for web application development and when we are planning to hire node js developers. 

Amazing Tech Stack for Web Development 

The principal thing that makes Node.js a go-to environment for web development is its JavaScript legacy. It’s the most well known language right now with a great many free devices and a functioning local area. Node.js, because of its association with JS, immediately rose in ubiquity — presently it has in excess of 368 million downloads and a great many free tools in the bundle module. 

Alongside prevalence, Node.js additionally acquired the fundamental JS benefits: 

  • quick execution and information preparing; 
  • exceptionally reusable code; 
  • the code is not difficult to learn, compose, read, and keep up; 
  • tremendous asset library, a huge number of free aides, and a functioning local area. 

In addition, it’s a piece of a well known MEAN tech stack (the blend of MongoDB, Express.js, Angular, and Node.js — four tools that handle all vital parts of web application development). 

Designers Can Utilize JavaScript for the Whole Undertaking 

This is perhaps the most clear advantage of Node.js web application development. JavaScript is an unquestionable requirement for web development. Regardless of whether you construct a multi-page or single-page application, you need to know JS well. On the off chance that you are now OK with JavaScript, learning Node.js won’t be an issue. Grammar, fundamental usefulness, primary standards — every one of these things are comparable. 

In the event that you have JS designers in your group, it will be simpler for them to learn JS-based Node than a totally new dialect. What’s more, the front-end and back-end codebase will be basically the same, simple to peruse, and keep up — in light of the fact that they are both JS-based. 

A Quick Environment for Microservice Development 

There’s another motivation behind why Node.js got famous so rapidly. The environment suits well the idea of microservice development (spilling stone monument usefulness into handfuls or many more modest administrations). 

Microservices need to speak with one another rapidly — and Node.js is probably the quickest device in information handling. Among the fundamental Node.js benefits for programming development are its non-obstructing algorithms.

Node.js measures a few demands all at once without trusting that the first will be concluded. Many microservices can send messages to one another, and they will be gotten and addressed all the while. 

Versatile Web Application Development 

Node.js was worked in view of adaptability — its name really says it. The environment permits numerous hubs to run all the while and speak with one another. Here’s the reason Node.js adaptability is better than other web backend development arrangements. 

Node.js has a module that is liable for load adjusting for each running CPU center. This is one of numerous Node.js module benefits: you can run various hubs all at once, and the environment will naturally adjust the responsibility. 

Node.js permits even apportioning: you can part your application into various situations. You show various forms of the application to different clients, in light of their age, interests, area, language, and so on. This builds personalization and diminishes responsibility. Hub accomplishes this with kid measures — tasks that rapidly speak with one another and share a similar root. 

What’s more, Node’s non-hindering solicitation handling framework adds to fast, letting applications measure a great many solicitations. 

Control Stream Highlights

Numerous designers consider nonconcurrent to be one of the two impediments and benefits of Node.js web application development. In Node, at whatever point the capacity is executed, the code consequently sends a callback. As the quantity of capacities develops, so does the number of callbacks — and you end up in a circumstance known as the callback damnation. 

In any case, Node.js offers an exit plan. You can utilize systems that will plan capacities and sort through callbacks. Systems will associate comparable capacities consequently — so you can track down an essential component via search or in an envelope. At that point, there’s no compelling reason to look through callbacks.

 

Final Words

So, these are some of the top benefits of Nodejs in web application development. This is how Nodejs is contributing a lot to the field of web application development. 

I hope now you are totally aware of the whole process of how Nodejs is really important for your web project. If you are looking to hire a node js development company in India then I would suggest that you take a little consultancy too whenever you call. 

Good Luck!

Original Source

#node.js development company in india #node js development company #hire node js developers #hire node.js developers in india #node.js development services #node.js development

Node JS Development Company| Node JS Web Developers-SISGAIN

Top organizations and start-ups hire Node.js developers from SISGAIN for their strategic software development projects in Illinois, USA. On the off chance that you are searching for a first rate innovation to assemble a constant Node.js web application development or a module, Node.js applications are the most appropriate alternative to pick. As Leading Node.js development company, we leverage our profound information on its segments and convey solutions that bring noteworthy business results. For more information email us at hello@sisgain.com

#node.js development services #hire node.js developers #node.js web application development #node.js development company #node js application

Node JS Development Advantages - How Your App Will Benefit From This JavaScript Framework

Web development has been controlling the JavaScript system features for many years. Many big online sites use Java Script for their everyday operations. And recently there has been a change and a shift towards cross-platform mobile application development. The main software frameworks in work these days are React native, apache Cordova, native script and hybrid tools. In the last ten years, Node.JS has been used as a backend development framework. Developers nowadays want to learn and use the same technologies for one entire website. They do not want to learn an entire language for server development. And Node.JS is able to adapt all the functions and syntaxes to the backend services from JavaScript. If you do not know the languages or syntaxes for Node JS development, you can look for an online guide. These guides have a detailed overview of the additional functions and basic systems. You will also find simple tasks in these guides. To read more click on the link.

#node js development services #node js development #node js development company #hire node js developers #node js mobile app developmen #website developer