Node.

js Design Patterns
Third Edition

Design and implement production-grade Node.js

applications using proven patterns and techniques

Mario Casciaro
Luciano Mammino

BIRMINGHAM - MUMBAI
Node.js Design Patterns
Third Edition
Copyright © 2020 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the authors, nor Packt
Publishing or its dealers and distributors, will be held liable for any damages
caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
Producer: Tushar Gupta
Acquisition Editor – Peer Reviews: Suresh Jain
Project Editor: Tom Jacob
Content Development Editors: Joanne Lovell, Bhavesh Amin
Copy Editor: Safis Editing
Technical Editor: Saby D'silva
Proofreader: Safis Editing
Indexer: Manju Arasan
Presentation Designer: Sandip Tadge

First published: December 2014

Second Edition: July 2016
Third Edition: July 2020
Production reference: 1240720
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.

ISBN 978-1-83921-411-0

www.packt.com
 Table of Contents
Prefacexi
Chapter 1: The Node.js Platform 1
The Node.js philosophy 2
Small core 2
Small modules 2
Small surface area 3
Simplicity and pragmatism 4
How Node.js works 5
I/O is slow 5
Blocking I/O 5
Non-blocking I/O 6
Event demultiplexing 7
The reactor pattern 9
Libuv, the I/O engine of Node.js 11
The recipe for Node.js 12
JavaScript in Node.js 13
Run the latest JavaScript with confidence 13
The module system 14
Full access to operating system services 14
Running native code 15
Summary16
Chapter 2: The Module System 17
The need for modules 18
Module systems in JavaScript and Node.js 19
The module system and its patterns 20
The revealing module pattern 20
[i]
Table of Contents

CommonJS modules 22
A homemade module loader 22
Defining a module 24
module.exports versus exports 25
The require function is synchronous 26
The resolving algorithm 26
The module cache 28
Circular dependencies 29
Module definition patterns 33
Named exports 33
Exporting a function 34
Exporting a class 35
Exporting an instance 36
Modifying other modules or the global scope 37
ESM: ECMAScript modules 38
Using ESM in Node.js 39
Named exports and imports 39
Default exports and imports 42
Mixed exports 43
Module identifiers 45
Async imports 45
Module loading in depth 48
Loading phases 48
Read-only live bindings 49
Circular dependency resolution 50
Modifying other modules 56
ESM and CommonJS differences and interoperability 60
ESM runs in strict mode 60
Missing references in ESM 60
Interoperability61
Summary62
Chapter 3: Callbacks and Events 63
The Callback pattern 64
The continuation-passing style 64
Synchronous CPS 65
Asynchronous CPS 65
Non-CPS callbacks 67
Synchronous or asynchronous? 67
An unpredictable function 68
Unleashing Zalgo 68
Using synchronous APIs 70
Guaranteeing asynchronicity with deferred execution 72

[ ii ]
 Table of Contents

Node.js callback conventions 73

The callback comes last 73
Any error always comes first 74
Propagating errors 74
Uncaught exceptions 75
The Observer pattern 77
The EventEmitter 78
Creating and using the EventEmitter 79
Propagating errors 80
Making any object observable 80
EventEmitter and memory leaks 82
Synchronous and asynchronous events 83
EventEmitter versus callbacks 85
Combining callbacks and events 86
Summary88
Exercises88
Chapter 4: Asynchronous Control Flow Patterns with Callbacks 89
The difficulties of asynchronous programming 90
Creating a simple web spider 90
Callback hell 93
Callback best practices and control flow patterns 94
Callback discipline 95
Applying the callback discipline 95
Sequential execution 98
Executing a known set of tasks in sequence 99
Sequential iteration 100
Parallel execution 104
Web spider version 3 106
The pattern 108
Fixing race conditions with concurrent tasks 108
Limited parallel execution 110
Limiting concurrency 112
Globally limiting concurrency 113
The async library 119
Summary120
Exercises121
Chapter 5: Asynchronous Control Flow Patterns
with Promises and Async/Await 123
Promises124
What is a promise? 125
Promises/A+ and thenables 127
The promise API 128

[ iii ]
Table of Contents

Creating a promise 130

Promisification 131
Sequential execution and iteration 133
Parallel execution 136
Limited parallel execution 137
Implementing the TaskQueue class with promises 138
Updating the web spider 139
Async/await141
Async functions and the await expression 141
Error handling with async/await 143
A unified try...catch experience 143
The "return" versus "return await" trap 144
Sequential execution and iteration 145
Antipattern – using async/await with Array.forEach for serial execution 147
Parallel execution 147
Limited parallel execution 149
The problem with infinite recursive promise resolution chains 152
Summary156
Exercises157
Chapter 6: Coding with Streams 159
Discovering the importance of streams 160
Buffering versus streaming 160
Spatial efficiency 161
Gzipping using a buffered API 162
Gzipping using streams 163
Time efficiency 163
Composability167
Adding client-side encryption 167
Adding server-side decryption 169
Getting started with streams 170
Anatomy of streams 170
Readable streams 171
Reading from a stream 171
Implementing Readable streams 174
Writable streams 179
Writing to a stream 179
Backpressure181
Implementing Writable streams 182
Duplex streams 185
Transform streams 185
Implementing Transform streams 186
Filtering and aggregating data with Transform streams 189
PassThrough streams 193
Observability193

[ iv ]
 Table of Contents

Late piping 194

Lazy streams 197
Connecting streams using pipes 198
Pipes and error handling 200
Better error handling with pipeline() 201
Asynchronous control flow patterns with streams 203
Sequential execution 203
Unordered parallel execution 206
Implementing an unordered parallel stream 206
Implementing a URL status monitoring application 208
Unordered limited parallel execution 210
Ordered parallel execution 212
Piping patterns 214
Combining streams 214
Implementing a combined stream 217
Forking streams 219
Implementing a multiple checksum generator 220
Merging streams 221
Merging text files 221
Multiplexing and demultiplexing 223
Building a remote logger 224
Multiplexing and demultiplexing object streams 229
Summary230
Exercises230
Chapter 7: Creational Design Patterns 233
Factory234
Decoupling object creation and implementation 235
A mechanism to enforce encapsulation 236
Building a simple code profiler 238
In the wild 241
Builder241
Implementing a URL object builder 244
In the wild 248
Revealing Constructor 249
Building an immutable buffer 250
In the wild 253
Singleton 253
Wiring modules 257
Singleton dependencies 258
Dependency Injection 261
Summary266
Exercises267

[v]
Table of Contents

Chapter 8: Structural Design Patterns 269

Proxy269
Techniques for implementing proxies 271
Object composition 272
Object augmentation 275
The built-in Proxy object 277
A comparison of the different proxying techniques 280
Creating a logging Writable stream 281
Change observer with Proxy 282
In the wild 285
Decorator285
Techniques for implementing decorators 286
Composition286
Object augmentation 288
Decorating with the Proxy object 289
Decorating a LevelUP database 290
Introducing LevelUP and LevelDB 290
Implementing a LevelUP plugin 291
In the wild 293
The line between proxy and decorator 294
Adapter294
Using LevelUP through the filesystem API 295
In the wild 298
Summary299
Exercises300
Chapter 9: Behavioral Design Patterns 301
Strategy 302
Multi-format configuration objects 304
In the wild 308
State308
Implementing a basic failsafe socket 310
Template315
A configuration manager template 316
In the wild 318
Iterator319
The iterator protocol 319
The iterable protocol 322
Iterators and iterables as a native JavaScript interface 324
Generators326
Generators in theory 327
A simple generator function 327
Controlling a generator iterator 328
How to use generators in place of iterators 330

[ vi ]
 Table of Contents

Async iterators 331

Async generators 334
Async iterators and Node.js streams 335
In the wild 336
Middleware337
Middleware in Express 337
Middleware as a pattern 338
Creating a middleware framework for ZeroMQ 340
The Middleware Manager 340
Implementing the middleware to process messages 342
Using the ZeroMQ middleware framework 344
In the wild 347
Command347
The Task pattern 349
A more complex command 349
Summary353
Exercises354
Chapter 10: Universal JavaScript for Web Applications 357
Sharing code with the browser 358
JavaScript modules in a cross-platform context 359
Module bundlers 360
How a module bundler works 363
Using webpack 369
Fundamentals of cross-platform development 371
Runtime code branching 372
Challenges of runtime code branching 373
Build-time code branching 374
Module swapping 377
Design patterns for cross-platform development 378
A brief introduction to React 379
Hello React 381
Alternatives to react.createElement 383
Stateful components 385
Creating a Universal JavaScript app 391
Frontend-only app 392
Server-side rendering 399
Asynchronous data retrieval 405
Universal data retrieval 411
Two-pass rendering 412
Async pages 414
Implementing async pages 416
Summary425
Exercises426
[ vii ]
Table of Contents

Chapter 11: Advanced Recipes 427

Dealing with asynchronously initialized components 428
The issue with asynchronously initialized components 428
Local initialization check 429
Delayed startup 430
Pre-initialization queues 431
In the wild 435
Asynchronous request batching and caching 435
What's asynchronous request batching? 436
Optimal asynchronous request caching 437
An API server without caching or batching 439
Batching and caching with promises 441
Batching requests in the total sales web server 442
Caching requests in the total sales web server 443
Notes about implementing caching mechanisms 445
Canceling asynchronous operations 445
A basic recipe for creating cancelable functions 446
Wrapping asynchronous invocations 447
Cancelable async functions with generators 449
Running CPU-bound tasks 453
Solving the subset sum problem 453
Interleaving with setImmediate 457
Interleaving the steps of the subset sum algorithm 457
Considerations on the interleaving approach 459
Using external processes 460
Delegating the subset sum task to an external process 461
Considerations for the multi-process approach 467
Using worker threads 468
Running the subset sum task in a worker thread 469
Running CPU-bound tasks in production 472
Summary473
Exercises473
Chapter 12: Scalability and Architectural Patterns 475
An introduction to application scaling 476
Scaling Node.js applications 477
The three dimensions of scalability 477
Cloning and load balancing 479
The cluster module 480
Notes on the behavior of the cluster module 481
Building a simple HTTP server 482
Scaling with the cluster module 484
Resiliency and availability with the cluster module 486
Zero-downtime restart 488

[ viii ]
 Table of Contents

Dealing with stateful communications 490

Sharing the state across multiple instances 491
Sticky load balancing 492
Scaling with a reverse proxy 494
Load balancing with Nginx 496
Dynamic horizontal scaling 501
Using a service registry 501
Implementing a dynamic load balancer with http-proxy and Consul 503
Peer-to-peer load balancing 510
Implementing an HTTP client that can balance requests across multiple servers 511
Scaling applications using containers 513
What is a container? 513
Creating and running a container with Docker 514
What is Kubernetes? 517
Deploying and scaling an application on Kubernetes 519
Decomposing complex applications 523
Monolithic architecture 524
The microservice architecture 526
An example of a microservice architecture 526
Microservices – advantages and disadvantages 528
Integration patterns in a microservice architecture 530
The API proxy 531
API orchestration 532
Integration with a message broker 536
Summary538
Exercises539
Chapter 13: Messaging and Integration Patterns 541
Fundamentals of a messaging system 542
One way versus request/reply patterns 542
Message types 544
Command Messages 544
Event Messages 545
Document Messages 545
Asynchronous messaging, queues, and streams 545
Peer-to-peer or broker-based messaging 547
Publish/Subscribe pattern 549
Building a minimalist real-time chat application 550
Implementing the server side 550
Implementing the client side 551
Running and scaling the chat application 553
Using Redis as a simple message broker 554
Peer-to-peer Publish/Subscribe with ZeroMQ 557
Introducing ZeroMQ 557
Designing a peer-to-peer architecture for the chat server 558
Using the ZeroMQ PUB/SUB sockets 559

[ ix ]
Table of Contents

Reliable message delivery with queues 562

Introducing AMQP 564
Durable subscribers with AMQP and RabbitMQ 566
Reliable messaging with streams 571
Characteristics of a streaming platform 571
Streams versus message queues 573
Implementing the chat application using Redis Streams 573
Task distribution patterns 577
The ZeroMQ Fanout/Fanin pattern 579
PUSH/PULL sockets 580
Building a distributed hashsum cracker with ZeroMQ 580
Pipelines and competing consumers in AMQP 587
Point-to-point communications and competing consumers 588
Implementing the hashsum cracker using AMQP 588
Distributing tasks with Redis Streams 592
Redis consumer groups 593
Implementing the hashsum cracker using Redis Streams 594
Request/Reply patterns 598
Correlation Identifier 598
Implementing a request/reply abstraction using correlation identifiers 599
Return address 605
Implementing the Return Address pattern in AMQP 605
Summary611
Exercises612
Other Books You May Enjoy 615
Index619

[x]