Skip to content

Node.js files and streams: fs, paths and pipelines

Back-office services spend a surprising amount of time on files: order exports, invoice PDFs, import batches and logs. This page covers the three layers Node.js gives you for that work: the promise-based fs API for whole files and directories, the path module for building locations that work on every operating system, and streams for data too large to hold in memory. The running example is an OrdersPortal export job that saves order files, then turns a large CSV export into compressed JSON lines.

All samples were run with Node.js 22.23 as ES modules: save as .mjs and run with node file.mjs.

Topic Use it when
File system promises API You read or write whole files, list directories or check metadata.
Paths You build file locations relative to the module or the working directory.
Streams Data is large, arrives over time, or flows between files, sockets and compressors.
Pipelines and backpressure You connect streams safely and keep a fast producer from flooding a slow consumer.

The node:fs module comes in three flavors: promise-based (node:fs/promises), callback-based and synchronous (readFileSync and friends). In servers use the promise API with await; the synchronous API blocks the event loop and belongs only in startup code and short CLIs. readFile and writeFile take an encoding ('utf8') to work with strings, otherwise they return and accept Buffers.

Useful calls: mkdir(path, { recursive: true }) creates missing parents and does not fail if the folder exists; readdir(path, { withFileTypes: true }) returns Dirent objects so you can tell files from folders without an extra stat; rm(path, { recursive: true, force: true }) deletes a tree; rename moves a file within one file system; copyFile copies. Errors carry a code such as ENOENT (not found), EACCES (permission denied), EEXIST (already exists), EISDIR and ENOTDIR.

A common production need is an atomic write: if the process crashes halfway through writeFile, readers can see a truncated file. Writing to a temporary file in the same folder and then renaming it over the target avoids that, because a rename within one file system replaces the file in one step on POSIX systems.

fs-orders.mjs
import { mkdir, writeFile, readFile, readdir, stat, rename, rm } from 'node:fs/promises';
import { join } from 'node:path';
const dataDir = join(import.meta.dirname, 'orders-data');
await rm(dataDir, { recursive: true, force: true });
await mkdir(join(dataDir, 'archive'), { recursive: true });
// Write atomically: write a temp file, then rename it over the target.
async function saveOrder(order) {
const target = join(dataDir, `order-${order.id}.json`);
const temp = `${target}.tmp`;
await writeFile(temp, JSON.stringify(order, null, 2), 'utf8');
await rename(temp, target);
return target;
}
await saveOrder({ id: 1001, customer: 'ada@example.test', total: 61.98 });
await saveOrder({ id: 1002, customer: 'ben@example.test', total: 12.5 });
const entries = (await readdir(dataDir, { withFileTypes: true }))
.sort((a, b) => a.name.localeCompare(b.name));
for (const entry of entries) {
if (!entry.isFile()) { console.log(`dir ${entry.name}/`); continue; }
const info = await stat(join(dataDir, entry.name));
const order = JSON.parse(await readFile(join(dataDir, entry.name), 'utf8'));
console.log(`file ${entry.name} ${info.size} bytes, total ${order.total}`);
}
try {
await readFile(join(dataDir, 'order-9999.json'), 'utf8');
} catch (err) {
if (err.code !== 'ENOENT') throw err;
console.log(`order 9999: not found (${err.code})`);
}

Output:

dir archive/
file order-1001.json 68 bytes, total 61.98
file order-1002.json 67 bytes, total 12.5
order 9999: not found (ENOENT)

Pitfalls:

  • Check-then-act races. access() followed by readFile() can fail if the file disappears in between. Just try the operation and handle ENOENT, as above.
  • readFile on huge files loads everything into memory; use a stream once files can reach hundreds of megabytes.
  • readdir order is not guaranteed. It depends on the file system; sort when order matters.

The node:path module builds and takes apart file paths using the separator of the current platform. path.join concatenates segments and normalizes .. and duplicate slashes; path.resolve returns an absolute path, resolving relative segments against the current working directory; basename, dirname and extname split a path into parts. Never build paths with string concatenation and '/': it breaks on Windows and on inputs with trailing slashes.

The working directory (process.cwd()) is wherever the process was started, which is rarely what you want for files that ship with your code. Resolve those relative to the module instead: import.meta.dirname in ES modules (Node.js 20.11 and later; on older versions use path.dirname(fileURLToPath(import.meta.url))), or __dirname in CommonJS.

paths.mjs
import path from 'node:path';
const templates = path.join(import.meta.dirname, 'templates', 'invoice.html');
console.log(path.basename(templates)); // invoice.html
console.log(path.extname('order-1001.json')); // .json
console.log(path.join('/srv/exports/', '../imports', 'batch-7.csv'));
console.log(path.resolve('exports', 'today.csv') === path.join(process.cwd(), 'exports', 'today.csv'));

Output (on Linux or macOS):

invoice.html
.json
/srv/imports/batch-7.csv
true

Pitfalls:

  • Path traversal. Joining user input (path.join(uploadDir, req.params.name)) can escape the folder with ../. Resolve the result and check it still starts with the allowed folder, or accept only IDs, never names.
  • path.join with a URL. import.meta.url is a file: URL, not a path; convert it with fileURLToPath.

A stream processes data in chunks as it arrives instead of all at once, so memory use stays roughly constant whatever the input size. Node.js has four kinds: Readable (a source, such as fs.createReadStream or an incoming HTTP request), Writable (a destination, such as fs.createWriteStream or an HTTP response), Duplex (both, independently, such as a TCP socket) and Transform (a duplex that changes data as it passes, such as zlib.createGzip()).

Modern code rarely subclasses these. Readable.from() turns any iterable or async generator into a readable stream, for await...of consumes a readable chunk by chunk, and readline.createInterface splits a byte stream into lines. Streams are also how HTTP works in Node.js: req is readable and res is writable, so you can send a large file with pipeline(createReadStream(file), res) without buffering it.

Pitfalls:

  • Mixing data events and for await. Pick one consumption style per stream.
  • Assuming chunks are lines. A chunk boundary can fall in the middle of a line or a multi-byte character; use readline or set an encoding.
  • Collecting a whole stream into an array brings back the memory problem streams were meant to solve.

pipeline() from node:stream/promises connects a source, any number of transforms and a destination, and returns a promise that settles when everything has finished. Unlike chained .pipe() calls, it forwards errors from every stage and destroys all the streams on failure, so a crashed transform does not leave a file handle open. A stage can also be an async generator function that receives the previous stage as an async iterable, which is the easiest way to write a transform.

Backpressure is how a stream says “slow down”. writable.write() returns false once the internal buffer reaches highWaterMark; the producer should stop and wait for the 'drain' event. pipeline() does this for you. The first sample streams a 200,000-row export through a filter and gzip; the second writes manually and respects drain.

streams-orders.mjs
import { createReadStream, createWriteStream } from 'node:fs';
import { stat } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
import { Readable } from 'node:stream';
import { createGzip } from 'node:zlib';
import { createInterface } from 'node:readline';
const csvPath = 'orders-export.csv';
const outPath = 'large-orders.ndjson.gz';
// 1. Produce a 200,000-row CSV export without building it in memory.
async function* exportRows() {
yield 'id,customer,total\n';
for (let id = 1; id <= 200_000; id++) {
yield `${id},customer-${id % 500}@example.test,${((id * 37) % 1000) / 10}\n`;
}
}
await pipeline(Readable.from(exportRows()), createWriteStream(csvPath));
// 2. Read it line by line, keep orders over 90, write gzip-compressed NDJSON.
let kept = 0;
async function* toLargeOrderJson(source) {
let header = true;
for await (const line of source) {
if (header) { header = false; continue; }
const [id, customer, total] = line.split(',');
if (Number(total) > 90) {
kept++;
yield JSON.stringify({ id: Number(id), customer, total: Number(total) }) + '\n';
}
}
}
const lines = createInterface({ input: createReadStream(csvPath), crlfDelay: Infinity });
await pipeline(lines, toLargeOrderJson, createGzip(), createWriteStream(outPath));
const [csv, gz] = await Promise.all([stat(csvPath), stat(outPath)]);
console.log(`CSV export: ${(csv.size / 1e6).toFixed(1)} MB`);
console.log(`Large orders kept: ${kept}`);
console.log(`Compressed NDJSON: ${(gz.size / 1e3).toFixed(0)} kB`);
console.log(`Peak resident memory under 100 MB: ${process.resourceUsage().maxRSS < 100 * 1024}`);

Output:

CSV export: 7.4 MB
Large orders kept: 19800
Compressed NDJSON: 53 kB
Peak resident memory under 100 MB: true

When you write to a stream yourself, check the return value of write():

backpressure.mjs
import { Writable } from 'node:stream';
import { once } from 'node:events';
import { setTimeout as delay } from 'node:timers/promises';
// A slow destination, such as a rate-limited audit API: 1 ms per record.
const auditSink = new Writable({
objectMode: true,
highWaterMark: 16, // buffer at most 16 records
async write(record, _encoding, callback) {
await delay(1);
callback();
},
});
let pauses = 0;
let maxBuffered = 0;
for (let id = 1; id <= 200; id++) {
const ok = auditSink.write({ id, event: 'order.shipped' });
maxBuffered = Math.max(maxBuffered, auditSink.writableLength);
if (!ok) {
pauses++;
await once(auditSink, 'drain'); // wait until the buffer empties
}
}
auditSink.end();
await once(auditSink, 'finish');
console.log(`wrote 200 records, paused ${pauses} times, max buffered ${maxBuffered}`);

Output:

wrote 200 records, paused 12 times, max buffered 16

Without the drain wait, all 200 records would sit in memory at once; with a million records from a fast database cursor, that is how a process runs out of heap.

Pitfalls:

  • Chained .pipe() without error handling. An error in one stage does not close the others; use pipeline().
  • Ignoring write()’s return value turns a stream into an unbounded buffer.
  • Forgetting end(). A writable that is never ended never emits 'finish', and the file may not be flushed.
Situation Use
Config file, small JSON, template readFile / writeFile from node:fs/promises
Startup-only or CLI code The sync API is acceptable
Large or unbounded data, uploads, downloads, exports Streams with pipeline()
Line-by-line processing readline.createInterface with for await
Custom transform An async generator stage in pipeline()

When would you use readFile instead of a stream? When the file is small and bounded and you need it whole anyway, such as configuration or a template. readFile is simpler and fast. Once size is unbounded or large, streams keep memory flat and let you start processing before the read finishes.

What is backpressure and how does Node.js signal it? It is the mechanism that slows a producer down to the consumer’s pace. write() returns false when the buffer passes highWaterMark, and the stream emits 'drain' when it is safe to continue. pipeline() and .pipe() respect it automatically.

Why prefer pipeline() over .pipe()? pipeline() propagates an error from any stage, destroys every stream on failure, and returns a promise you can await. With .pipe() you must attach error handlers to every stream and clean up yourself.

How do you safely serve a file chosen by the user? Never join raw user input into a path. Map an ID to a stored file name, or resolve the joined path and confirm it still sits inside the allowed folder, then stream it with pipeline(createReadStream(file), res) and handle ENOENT as a 404.

Senior scenario: a nightly import of a 5 GB CSV crashes with “JavaScript heap out of memory”. How do you redesign it? The job almost certainly reads the whole file or accumulates rows. Rewrite it as a pipeline: createReadStream into readline, parse and validate each row in an async generator, batch rows (for example 1,000 at a time) and write each batch to the database before reading more, so backpressure flows from the database to the file. Record progress (byte offset or last ID) so a failed run can resume, and send invalid rows to a reject file instead of failing the whole import.