#streams

Node.jsadvanced

Use stream.finished for reliable cleanup

import { createReadStream, createWriteStream } from "node:fs";
import { finished } from "node:stream/promises";

export const copyWithCleanup = async (src: string, dest: string, signal?: AbortSignal) => {
  const read = createReadStream(src);

Wait for a stream to finish and clean up resources without ad-hoc listeners.

Node.jsadvanced

Bridge Node streams to web streams

import { createReadStream } from "node:fs";
import { Readable } from "node:stream";

export const compressedFileResponse = (path: string) => {
  const nodeStream = createReadStream(path, { highWaterMark: 64 * 1024 });

Turn a Node Readable into a web-compatible stream with Readable.toWeb().

Node.jsintermediate

Readable.from() to turn any async iterable into a stream

import { Readable } from "node:stream";

async function* paginatedResults(pageCount: number) {
  for (let page = 1; page <= pageCount; page++) {
    await new Promise((resolve) => setTimeout(resolve, 10)); // simulate a network call

Readable.from() wraps any (async) generator or iterable in a real Readable stream, so it can be piped anywhere a stream is expected.

Bunadvanced

Piped-stdio subprocesses with Bun.spawn

const proc = Bun.spawn(["bun", "-e", "console.log((await Bun.stdin.text()).toUpperCase())"], {
  stdin: "pipe",
});

proc.stdin.write("hello from the parent\n");

Bun.spawn's stdin/stdout are a FileSink and a ReadableStream, so you can write to a child process incrementally and read its output as it streams back.

Node.jsadvanced

Streaming a large JSON array without loading it into memory

import type { Writable } from "node:stream";

export const streamJsonArray = async (rows: AsyncIterable<unknown>, out: Writable) => {
  out.write("[");
  let first = true;

Write a large JSON array to a stream item by item as it's produced, instead of building the whole array in memory before calling JSON.stringify.

Node.jsadvanced

Piping a Readable through a Transform stream with backpressure

import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";
import { Transform } from "node:stream";

const upperCaseChunks = new Transform({

Use stream/promises pipeline() to move data through a Transform without ever buffering more than the stream can hold.