Node.js Streams In Depth

A comprehensive repository demonstrating the core concepts of Streams in Node.js, including Readable Streams, Writable Streams, Backpressure, Pipe, and Pipeline.

---

Overview

Streams are one of the most powerful features in Node.js. They allow processing large amounts of data efficiently without loading everything into memory.

This repository covers:

• Readable Streams
• Writable Streams
• Backpressure
• Pause and Resume
• Drain Event
• Pipe
• Pipeline
• Error Handling
• Stream Lifecycle

---

Project Structure

.
├── generator.js  - Generates a source.txt file with 10,000 lines of sample data
├── source.txt    - Source file containing generated data
├── copy.txt      - Destination file created after stream operations
│
├── app.js        - Demonstrates Readable & Writable Streams with manual backpressure handling
├── pipe.js       - Demonstrates file copying using the pipe() method
├── pipeline.js   - Demonstrates the pipeline() API with automatic error handling
│
├── package.json
└── README.md

---

📁 File Descriptions

File	Description
generator.js	Generates a source.txt file containing 10,000 lines of sample text for stream testing.
source.txt	Input file used as the source for all stream operations.
copy.txt	Output file where streamed data is written.
app.js	Demonstrates Readable and Writable Streams with manual backpressure handling using pause(), resume(), and the drain event.
pipe.js	Demonstrates automatic data transfer between streams using the pipe() method.
pipeline.js	Demonstrates the pipeline() API for safer stream handling with automatic cleanup and error management.
README.md	Project documentation, setup instructions, and explanation of stream concepts.

Prerequisites

• Node.js >= 18
• npm

Verify Installation

node -v
npm -v

---

Installation

Clone the repository

git clone https://github.com/MilanEkanna/Node-Streams.git

Move into the project

cd nodejs-streams-in-depth

Install dependencies

npm install

---

Streams Basics

A Stream is an abstract interface for working with streaming data in Node.js.

Instead of reading an entire file into memory, data is processed chunk by chunk.

Benefits:

• Lower Memory Usage
• Better Performance
• Handles Large Files
• Real-Time Processing

---

stream.js

Demonstrates:

• Readable Stream
• Writable Stream
• Manual Backpressure Handling
• Pause and Resume
• Drain Event
• Finish Event

Run:

node stream.js

Workflow:

Readable Stream
        │
        ▼
Receive Chunk
        │
        ▼
Writable Stream
        │
        ├── Buffer Full?
        │       │
        │      Yes
        │       │
        ▼       ▼
     Pause    Wait
               │
               ▼
            Drain
               │
               ▼
            Resume

---

pipe.js

Demonstrates automatic stream piping.

Run:

node pipe.js

Example:

const fs = require('fs');

const readStream = fs.createReadStream('source.txt');
const writeStream = fs.createWriteStream('copy.txt');

readStream.pipe(writeStream);

Advantages:

• Less Code
• Automatic Backpressure Handling
• Cleaner Implementation

Flow:

Readable Stream
       │
       ▼
      pipe()
       │
       ▼
Writable Stream

---

pipeline.js

Demonstrates the use of Node.js Pipeline API.

Run:

node pipeline.js

Example:

const { pipeline } = require('stream');

Benefits:

• Automatic Cleanup
• Better Error Handling
• Production Ready
• Prevents Resource Leaks

Flow:

Readable Stream
       │
       ▼
   Pipeline
       │
       ▼
Writable Stream

---

Backpressure

Backpressure occurs when:

Producer Speed
      >
Consumer Speed

Meaning:

The readable stream generates data faster than the writable stream can process it.

Solution:

const canWrite = writeStream.write(chunk);

if (!canWrite) {
  readStream.pause();
}

Resume after:

writeStream.on('drain', () => {
  readStream.resume();
});

---

Stream Events

Readable Stream Events

Event	Description
data	Chunk received
end	No data left
error	Error occurred
close	Stream closed

---

Writable Stream Events

Event	Description
drain	Buffer emptied
finish	Writing completed
error	Error occurred
close	Stream closed

---

Memory Efficiency

Without Streams:

Entire File
      ↓
Memory
      ↓
Processing

With Streams:

Chunk
 ↓
Process
 ↓
Chunk
 ↓
Process

Streams avoid loading large files entirely into RAM.

---

Learning Outcomes

After completing this repository, you will understand:

• How streams work internally
• Readable streams
• Writable streams
• Buffering
• Backpressure
• Pipe
• Pipeline
• Stream Events
• Memory Optimization

---

Commands Summary

Run Manual Stream

node stream.js

Run Pipe Example

node pipe.js

Run Pipeline Example

node pipeline.js

---

Author

Milan Ekanna

MERN Stack Developer