Collect-stream, from2, to2

In notebook:
FrontEndMasters Networking and Streams
Created at:
2017-09-30
Updated:
2017-10-01
Tags:
backend Node JS JavaScript

collect-stream

Very useful for doing unit tests. It gives you an array of the chunks.

collect a stream's output into a single buffer

for object streams, collect output into an array of objects

var collect = require('collect-stream')
var split = require('split2')

var sp = process.stdin.pipe(split(JSON.parse))
collect(sp, function (err, rows) {
  if (err) console.error(err)
  else console.log(rows)
})

This module is very useful for unit tests.


from2

The core readable API is very hard to use.

The function argument it takes is called each time a downstream consumer (that you pipe into) needs data (calls its next() function). This is much easier to set up a readable stream, than from the NodeJS core.

create a readable stream with a pull function

var from = require('from2')
var messages = [ 'hello', ' world\n', null ]

from(function (size, next) {
  next(null, messages.shift())
}).pipe(process.stdout)

to2

When you want to use through2 but you are not piping out anywhere, then you can use this module instead. It just implements the writable stream (you cannot pipe out of it)

create a writable stream with a write and flush function

var to = require('to2')
var split = require('split2')

process.stdin.pipe(split()).pipe(to(function (buf, next) {
  console.log(buf.length)
  next()
}))

We can refactor the line count example with this:

  //	****		line-count		****

var split = require('split2')
var to = require('to2')
var through = require('through2') // will not be necessary
var lineCount = 0

process.stdin
  .pipe(split())
  // .pipe(through(write,end))
  // **** 1. use to instead  ↴
  .pipe(to(write,end))
  
function write (buf, enc, next) {
  lineCount++
  next()
}
function end (next) {
  console.log(lineCount)
  next()
}

It's good practice to think ahead and evaluate if you can use to2 instead of through2 because if you change your code later, it can add bugs (data not piped out anymore).