Resourceful NodeJS Packages for ETL

In my previous blog post, we spoke about using NodeJS for doing ETL jobs. In this post, I’m going to highlight a few of the NodeJS packages that I’ve used in the past. I’ll explain their importance and how they are used. Let’s get started and hopefully, you’ll find this post to be of use to you.

Async

Async is one of the most useful Node.js packages I have used and can be applied to any number of Node.js projects.  With over 60 million downloads in the past month alone, I’m not the only one who would think so. This package provides a module with robust functions that work with asynchronous JavaScript. It is very well-documented, easy to learn, and has plenty of functions specifically for control flow.  If you’ve ever found yourself nesting loops that are returning live variable references in an event loop or are trying to run an ordered series of functions, async will make your life much easier.  Plus it will help keep your Node.js scripts from turning into an unending mess of callbacks-within-callbacks.

Express-stream

Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications. It gives us a lot of tools that make writing node applications easier and more fun. It also implements a lot of functionalities that will allow us to focus on the business logic, set up routes, and render things to the screen. Express also requires us to use template engines to create our HTML files and to dynamically inject values into those files. I’m a big fan of Express because you can insert almost any compatible middleware you like into the request handling chain, in almost any order you like. Express also allows you to structure applications in single or multiple files, with no directory restrictions. 

Express-stream is an add-on that exposes two middleware functions: Stream.pipe() and Stream.stream(). Stream.pipe() is a minimal opinionated BigPipe implementation that is ideal for client-side rendering. Stream.stream() is ideal for server-side rendering. Both methods get content as fast as possible by taking forms of latency and making them occur in parallel. 

Body-Parser

Another useful node package is Body-parser which parses your request, as an HTTP request body, in a middleware and converts it into a format from which you can easily extract relevant information that you may need. Using body parser allows you to access “req.body” from within your routes, and use that data. An example would be that you have a sign-up form at your frontend. You are filling it, and requesting the server to save the details somewhere. Extracting the username and password from each request would only take a few lines of code. This would save you from having to assemble the data from every request you send out. 

Lodash

Lodash is a very popular NPM Package Manager repository that it is used to simplify your work of managing/editing objects and arrays by providing lots of utility methods to do so. Some of the methods that I found quite handy and helped ease my development process are _pick, _map, _filter, _has. These methods are exceptionally helpful with iterating arrays, objects, & strings, manipulating & testing values and creating composite functions. It is also very functional on the server and browser side without any issues. I cherish lodash because my code stays neat, the functions are easy to remember and it has no learning curve. It currently has more than 20 million weekly downloads, so it’s in high demand these days. 

Babel

Babel is a transpiler which was originally created to convert next-generation JavaScript (ES6) into browser compatible JavaScript but has since been updated to also convert JSX and Flow to JavaScript. It gives you the ability to transform syntax, invoke polyfill features that are missing and perform source code transformations. As developers, we have our own preferences when it comes to the languages and environments we want to work with. With this package, I am able to choose my preferred development language. This is what makes Babel so versatile and encouraging. Plugins can also be created if there isn’t a suitable tool.

The packages explored in this post are ones that I personally find to be the most resourceful. We all operate at different levels and there are a ton of there useful packages out there. However, I hope some of these tools will make work and life a little easier and more fun.

We are a technology company that specializes in building business platforms. If you have any questions about the topic discussed in this post or about any of our services, send us an email!

Photo by Brooke Cagle on Unsplash


Join Anant's Newsletter

Subscribe to our monthly newsletter below and never miss the latest Cassandra and data engineering news!