ipc-stream: Duplex stream that runs over Electron's IPC

electron-ipc-stream

Duplex stream that run over Electron's IPC mechanism.

Why?

This allows you to use any Node.js stream readable/writable and easily communicate between your main/renderer process.

Since your renderer process is also responsible for UI/DOM, etc, you may not want to do any heavy processing on the renderer process. You could leverage this module to have the renderer stream data to the main process for processing and then the main module could stream results back to the renderer process for consumption.

Install

npm i --save electron-ipc-stream

Usage

Example 1: Pipe file from main process to renderer.

main.js:

var app = require('app')
var fs = require('fs')
var path = require('path')
var window = require('electron-window')
var IPCStream = require('electron-ipc-stream')

app.on('ready', function () {
  var win = window.createWindow({ height: 600, with: 1000 })

  var ipcs = new IPCStream('any-arbitrary-channel-name', win)
  win.showUrl(path.resolve(__dirname, './index.html'), function () {
    // window is visible, dom is ready in window
    fs.createReadStream('/tmp/mainfile').pipe(ipcs)
  })
})

rend.js:

var fs = require('fs')
var ipc = require('ipc')
var IPCStream = require('electron-ipc-stream')
var ipcs = new IPCStream('any-arbitrary-channel-name')

document.addEventListener('DOMContentLoaded', function () {
  ipcs.pipe(fs.createWriteStream('/tmp/rendfile')).on('finish', function () {
    console.log('done')
  })
})

Example 2: Pipe file from renderer process to main.

main.js:

var app = require('app')
var fs = require('fs')
var path = require('path')
var window = require('electron-window')
var IPCStream = require('electron-ipc-stream')

var tmpfile = '/tmp/mainfile'
app.on('ready', function () {
  var win = window.createWindow({ height: 600, with: 1000 })
  var ipcs = new IPCStream('any-arbitrary-channel-name', win)
  ipcs.pipe(fs.createWriteStream(tmpfile)).on('finish', function () {
    console.log('done')
  })
  win.showUrl(path.resolve(__dirname, './index.html'), function () { })
})

rend.js:

var crypt = require('crypto') // notice this is 'crypt' and not 'crypto'
var fs = require('fs')
var ipc = require('ipc')
var IPCStream = require('electron-ipc-stream')
var ipcs = new IPCStream('any-arbitrary-channel-name')

fs.writeFileSync('/tmp/rendfile', crypt.randomBytes(10000))
document.addEventListener('DOMContentLoaded', function () {
  fs.createReadStream(tmpfile).pipe(ipcs)
})

API

Main Process

IPCStream(channel, [browserWindow], [streamOptions])

Create a new IPCStream in the main process.

Renderer Process

IPCStream(channel, [streamOptions])

Create a new IPCStream in the renderer process.

Stream Options

You shouldn't have to mess with objectMode. Under the hood, objectMode is true. Buffers are serialized to JSON. This is because of the way that Electron handles buffers in renderer. See: https://github.com/atom/electron/blob/master/docs/api/remote.md for more detail. You also may need to adjust highWaterMark.

JSON Objects

It is completely safe to call write on either end of the stream with objects.

source:

myStream.write({name: 'JP'})

dest:

// streams 1 (flowing):
myStream.on('data', function (data) {
  console.dir(data) // => {name: 'JP'}
})

// streams 2/3 (pull, if you prefer):
myStream.on('readable', function () {
  var data
  while (null !=== (data = myStream.read())) {
    console.dir(data) // => {name: 'JP'}
  }
})

Examples

In the ./test folder, you'll see two examples. You can run these by installing electron-prebuilt:

npm i -g electron-prebuilt
electron ./test/main-to-rend
electron ./test/rend-to-main

Author: jprichardson
Source Code: https://github.com/jprichardson/electron-ipc-stream 
License: MIT

#electron #stream 

What is GEEK

Buddha Community

ipc-stream: Duplex stream that runs over Electron's IPC

electron-ipc-stream

Duplex stream that run over Electron's IPC mechanism.

Why?

This allows you to use any Node.js stream readable/writable and easily communicate between your main/renderer process.

Since your renderer process is also responsible for UI/DOM, etc, you may not want to do any heavy processing on the renderer process. You could leverage this module to have the renderer stream data to the main process for processing and then the main module could stream results back to the renderer process for consumption.

Install

npm i --save electron-ipc-stream

Usage

Example 1: Pipe file from main process to renderer.

main.js:

var app = require('app')
var fs = require('fs')
var path = require('path')
var window = require('electron-window')
var IPCStream = require('electron-ipc-stream')

app.on('ready', function () {
  var win = window.createWindow({ height: 600, with: 1000 })

  var ipcs = new IPCStream('any-arbitrary-channel-name', win)
  win.showUrl(path.resolve(__dirname, './index.html'), function () {
    // window is visible, dom is ready in window
    fs.createReadStream('/tmp/mainfile').pipe(ipcs)
  })
})

rend.js:

var fs = require('fs')
var ipc = require('ipc')
var IPCStream = require('electron-ipc-stream')
var ipcs = new IPCStream('any-arbitrary-channel-name')

document.addEventListener('DOMContentLoaded', function () {
  ipcs.pipe(fs.createWriteStream('/tmp/rendfile')).on('finish', function () {
    console.log('done')
  })
})

Example 2: Pipe file from renderer process to main.

main.js:

var app = require('app')
var fs = require('fs')
var path = require('path')
var window = require('electron-window')
var IPCStream = require('electron-ipc-stream')

var tmpfile = '/tmp/mainfile'
app.on('ready', function () {
  var win = window.createWindow({ height: 600, with: 1000 })
  var ipcs = new IPCStream('any-arbitrary-channel-name', win)
  ipcs.pipe(fs.createWriteStream(tmpfile)).on('finish', function () {
    console.log('done')
  })
  win.showUrl(path.resolve(__dirname, './index.html'), function () { })
})

rend.js:

var crypt = require('crypto') // notice this is 'crypt' and not 'crypto'
var fs = require('fs')
var ipc = require('ipc')
var IPCStream = require('electron-ipc-stream')
var ipcs = new IPCStream('any-arbitrary-channel-name')

fs.writeFileSync('/tmp/rendfile', crypt.randomBytes(10000))
document.addEventListener('DOMContentLoaded', function () {
  fs.createReadStream(tmpfile).pipe(ipcs)
})

API

Main Process

IPCStream(channel, [browserWindow], [streamOptions])

Create a new IPCStream in the main process.

Renderer Process

IPCStream(channel, [streamOptions])

Create a new IPCStream in the renderer process.

Stream Options

You shouldn't have to mess with objectMode. Under the hood, objectMode is true. Buffers are serialized to JSON. This is because of the way that Electron handles buffers in renderer. See: https://github.com/atom/electron/blob/master/docs/api/remote.md for more detail. You also may need to adjust highWaterMark.

JSON Objects

It is completely safe to call write on either end of the stream with objects.

source:

myStream.write({name: 'JP'})

dest:

// streams 1 (flowing):
myStream.on('data', function (data) {
  console.dir(data) // => {name: 'JP'}
})

// streams 2/3 (pull, if you prefer):
myStream.on('readable', function () {
  var data
  while (null !=== (data = myStream.read())) {
    console.dir(data) // => {name: 'JP'}
  }
})

Examples

In the ./test folder, you'll see two examples. You can run these by installing electron-prebuilt:

npm i -g electron-prebuilt
electron ./test/main-to-rend
electron ./test/rend-to-main

Author: jprichardson
Source Code: https://github.com/jprichardson/electron-ipc-stream 
License: MIT

#electron #stream 

Stateful stream processing with Apache Flink(part 1): An introduction

Apache Flink, a 4th generation Big Data processing framework provides robust **stateful stream processing capabilitie**s. So, in a few parts of the blogs, we will learn what is Stateful stream processing. And how we can use Flink to write a stateful streaming application.

What is stateful stream processing?

In general, stateful stream processing is an application design pattern for processing an unbounded stream of events. Stateful stream processing means a** “State”** is shared between events(stream entities). And therefore past events can influence the way the current events are processed.

Let’s try to understand it with a real-world scenario. Suppose we have a system that is responsible for generating a report. It comprising the total number of vehicles passed from a toll Plaza per hour/day. To achieve it, we will save the count of the vehicles passed from the toll plaza within one hour. That count will be used to accumulate it with the further next hour’s count to find the total number of vehicles passed from toll Plaza within 24 hours. Here we are saving or storing a count and it is nothing but the “State” of the application.

Might be it seems very simple, but in a distributed system it is very hard to achieve stateful stream processing. Stateful stream processing is much more difficult to scale up because we need different workers to share the state. Flink does provide ease of use, high efficiency, and high reliability for the**_ state management_** in a distributed environment.

#apache flink #big data and fast data #flink #streaming #streaming solutions ##apache flink #big data analytics #fast data analytics #flink streaming #stateful streaming #streaming analytics

Spark Structured Streaming – Stateful Streaming

Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Internals of Structured Streaming“. And this blog pertains to Stateful Streaming in Spark Structured Streaming. So let’s get started.

Let’s start from the very basic understanding of what is Stateful Stream Processing. But to understand that, let’s first understand what Stateless Stream Processing is.

In my previous blogs of this series, I’ve discussed Stateless Stream Processing.

You can check them before moving ahead – Introduction to Structured Streaming and Internals of Structured Streaming

#analytics #apache spark #big data and fast data #ml #ai and data engineering #scala #spark #streaming #streaming solutions #tech blogs #stateful streaming #structured streaming

Electron-stream: Streaming Wrapper Around Electron

electron-stream

Write JavaScript to electron, get console output back!

Example

Boot a hidden electron instance, log to stdout and clean up:

var electron = require('electron-stream');

var browser = electron();

browser.pipe(process.stdout);

browser.write('console.log(window.location.href);');
browser.write('window.close();');
browser.end();

Alternatively, use an existing http server. Note you cannot write to electron-stream when outside http server is in use.

var electron = require('electron-stream');
var http = require('http');

var server = http.createServer((req, res) => {
  if (/^\/bundle\.js/.test(req.url)) {
    res.setHeader('content-type', 'application/javascript');
    res.setHeader('cache-control', 'no-cache');
    res.end('console.log("hello");window.close();');
    return;
  }

  if (req.url == '/') {
    res.setHeader('Content-Type', 'text/html');
    res.end(`<!DOCTYPE html><meta charset="utf8"><body><script src="/bundle.js"></script></body>`);
    return;
  }
});

server.listen(8000);
var browser = electron({ loc: 'http://localhost:8000' });
browser.pipe(process.stdout);
browser.end();

Output streams

electron-stream lets you read all of the console output together, or split up into stdout and stderr:

// console.log and console.error
browser.pipe(...);
browser.stdall.pipe(...);

// console.log only
browser.stdout.pipe(...);

// console.error only
browser.stderr.pipe(...);

Installation

To install as a library:

$ npm install electron-stream

To install as a binary:

$ npm install -g electron-stream
$ echo "console.log('foo');window.close()" | electron-stream

Travis

To use electron on travis, add this to your travis.yml:

addons:
  apt:
    packages:
      - xvfb
install:
  - export DISPLAY=':99.0'
  - Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
  - npm install

Source.

API

electron([opts])

Create a writable stream around a newly spawned electron which forwards written data to electron. This module bundles electron-prebuilt.

Options:

• show: Show the electron window. Defaults to false.
• node: Enable node integration. Defaults to false.
• basedir: Set this if you need to require node modules in node mode
• static: Serve static files from this directory at /
• loc: a full url like http://localhost:8080/ for using an existing http server. When loc is supplied, options node, basedir, and static are all ignored.
• sandbox: Run electron with sandbox. Disable to emit debug information when using Docker. Defaults to true.

electron#stdout

electron#stderr

electron#stdall

Readable streams containing the console output. console.log will be forwarded to .stdout, console.error to .stderr. .stdall has them both.

electron#kill()

Kill the child process.

electron#on('exit', fn)

Emitted when the underlying electron exits. There can be multiple reasons for this:

• electron#kill() was called
• window.close() was sent as a script
• there was a fatal error

Author: juliangruber
Source Code: https://github.com/juliangruber/electron-stream 
License: MIT

#electron #stream 

On-Demand Music Streaming App I Live Streaming App Development Company

https://www.mobiwebtech.com/music-streaming-app-development/

On-Demand Live Streaming App Development Company- Create on-demand music streaming apps with hi-tech Music streaming portal and app development experts.

#music streaming app development #music streaming app development company #music streaming software development #create music streaming app #live streaming app development