How to Read and Filter a Directory Recursively in Node.js

For this tutorial, we're going to use CheatCode's full-stack JavaScript framework, Joystick. Joystick brings together a front-end UI framework with a Node.js back-end for building apps.

To begin, we'll want to install Joystick via NPM. Make sure you're using Node.js 16+ before installing to ensure compatibility (give this tutorial a read first if you need to learn how to install Node.js or run multiple versions on your computer):

Terminal

npm i -g @joystick.js/cli

This will install Joystick globally on your computer. Once installed, next, let's create a fresh project:

Terminal

joystick create app

After a few seconds, you will see a message logged out to cd into your new project and run joystick start:

Terminal

cd app && joystick start

After this, your app should be running and we're ready to get started.

Writing a function for getting files in a directory

To start, we're going to wire up the function that we'll be using to read our directory. In the /lib folder that was created as part of the project you created with joystick create app above, create another directory node and within that, a file called readDirectory.js:

Heads up: the node directory here is required. In a Joystick app, the /lib/node folder is used to isolate any code that will need to be built for a Node.js environment. The root /lib folder is built as universal JavaScript, meaning it can run in the browser or a Node.js environment. Because the code we'll write below must have access to Node.js, we use the /lib/node folder to isolate the functionality.

/lib/node/readDirectory.js

import { resolve } from 'path';
import { readdir } from 'fs/promises';

const readDirectory = async (path = '', pathsToFilter = []) => {
  // We'll implement reading the directory here...
};

export default readDirectory;

First, we want to mock out the function that we'll be calling to list out our directory. Here, we see a function readDirectory being defined, taking in two arguments:

1. path which will be the path of the directory we want to list the files from.
2. pathsToFilter an optional array of path strings (or folder names) to filter out of the resulting list of files.

Up at the top of our file, we're importing two dependencies, both from built-in Node.js packages/libraries (i.e., we don't need to install these separately):

1. resolve from the path package which is a function which will help us determine (or, "resolve") the path of the current directory we're listing.
2. readdir from the fs/promises package which is the function that will actually read and list the files in the directory.

/lib/node/readDirectory.js

import { resolve } from 'path';
import { readdir } from 'fs/promises';

const readDirectory = async (path = '', pathsToFilter = []) => {
  const filesInPath = await readdir(path, { withFileTypes: true });
  // We'll implement the rest of our function here...
};

export default readDirectory;

Adding some detail, next, just inside the body of our function we want to call to the readdir() function we imported up above. Notice that we've imported this function not from the more common fs package, but one if its folders: fs/promises. The /promises here gives us access to functions from the fs package which return a JavaScript Promise (as opposed to using the standard asynchronous callback or synchronous function approach).

This is why we have the async keyword in front of our function definition. We want to call the readdir() function, receiving a Promise, and use the await keyword to "wait on" the Promise to resolve. This keeps our code clean but also plays nicely with the recursion we'll implement next.

Here, in order to use the await keyword, the async prefixing our function is required as JavaScript will throw a runtime error telling us await is a reserved keyword and can only be used in an async context (this is why we often hear this concept referred to as async/await—they're two peas in a pod).

Focusing on the readdir() function call, as the first argument we pass the path we want to read, and second, an options object. For our options here, we've set the withFileTypes option to true. This is a bit confusing.

Though it doesn't sound like it, this option tells the readdir() function to return what's known as a Dirent object (directory entry). This object not only contains the name of each file in the path, but also adds some helpful methods for working with the file. Without this, we'd only get back a string which is only part of what we need to complete our work here.

/lib/node/readDirectory.js

import { resolve } from 'path';
import { readdir } from 'fs/promises';

const readDirectory = async (path = '', pathsToFilter = []) => {
  const filesInPath = await readdir(path, { withFileTypes: true });

  const files = await Promise.all(filesInPath.map((fileInPath) => {
    const resolvedPath = resolve(path, fileInPath.name);
    return fileInPath.isDirectory() ? readDirectory(resolvedPath) : resolvedPath;
  }));

  // We'll handle filtering and returning the file list here...
};

export default readDirectory;

Now we get into the tricky stuff. Here, we've added a call to Promise.all() with an await in front of it. To it, we've passed in a call to filesInPath.map() where filesInPath is the list of Dirent objects we just got back from readdir().

For each Dirent object—here, we're mapping over and referring to as fileInPath—we need to do two things:

1. Use the resolve() method we imported up top to "resolve" or give us the full path for the current file we're mapping over as resolvedPath.
2. Determine if the fileInPath is itself a directory that needs to be listed, or, if it's just a plain file.

This is where the Dirent object (versus the plain string) comes into play. Notice that from our .map() function, we're returning a JavaScript Ternary statement (think of this like a condensed if/else statement). This allows us to conditionally return some value inline.

Here, we call the .isDirectory() function on the Dirent object (again, cast as fileInPath for context) we received back from readdir() to figure out if the file we're currently mapping over is a directory. If it is, we want to call readDirectory() (the function we're currently writing) recursively (meaning, to tell the function to call itself again with different arguments).

In the event that fileInPath is not directory, we just want to return the plain resolvedPath that we constructed with resolve().

So wait, what just happened? Consider what our readDirectory() function is doing. It takes in a path and then reads the contents of that path. For each path found in that directory, we check to see if that path is a directory, and if it is, we recursively call readDirectory() to list its contents. If it's not, we return the file.

Because we're making calls recursively inside of a Promise.all(), what we're saying in effect is that if a "file" returned from our initial path is itself a directory, we want to wait on it to return its list of files (in essence, we're actually waiting on the await fs.readdir() and any recursive calls to await readDirectory()). Because a directory might have directories within it, we wait on those too. So, for example, if we have a folder at the root called A and it has a folder called B who has a folder called C, the A folder will have to wait for C to return to B before B is returned to A.

If that's confusing, apply the logic of "turtles all the way down" and it should make a bit more sense.

So what's the result here? Assuming that we've .map()ed over every file until we've run out of subdirectories, we'll get back the full list of files in the const files variable. There's a tiny gotcha, though: due to us ultimately returning an array from readDirectory(), what we'll actually get back (if we did have subdirectories) is an array of nested arrays, like this:

['/somefile.txt', ['/someDirectory/somefile.txt'] ]

In other words, for each subdirectory, we get back an array of its files. To avoid having to write a bunch of weird loops, we can get around this with a call to the JavaScript .flat() method on our const files array:

/lib/node/readDirectory.js

import { resolve } from 'path';
import { readdir } from 'fs/promises';

const readDirectory = async (path = '', pathsToFilter = []) => {
  const filesInPath = await readdir(path, { withFileTypes: true });

  const files = await Promise.all(filesInPath.map((fileInPath) => {
    const resolvedPath = resolve(path, fileInPath.name);
    return fileInPath.isDirectory() ? readDirectory(resolvedPath) : resolvedPath;
  }));

  return files.flat()?.filter((file = '') => {
    return !pathsToFilter.some((pathToFilter) => {
      return file.includes(pathToFilter);
    });
  });
};

export default readDirectory;

Here, from our readDirectory() function, we're returning a call to files.flat() to give us back a flattened copy of all the nested arrays in our const files variable. On the end of that, we call to the .filter() method on the resulting array to make use of the pathsToFilter array we talked about earlier.

To filter the array, we're trying to answer the question, "does the current file include any of the paths we want to filter?" To do it, from our .filter() we return a call to .some() which will return true or false if some of the items in the array we call the method on—here, pathsToFilter—return true.

In this case, for each pathToFilter we want to pass it to a call to file.includes(). If file does include the pathToFilter we'll return true, which in turn means we return true for .some(), meaning, the current file should be filtered because it matched some of the pathsToFilter.

That's it! Again, consider that we're calling this recursively until we run out of subdirectories to list. So, for each recursive call, we're returning a flattened, filtered array, with the final resulting array being returned back to our original call to readDirectory().

To make sense of that, let's put this to use and see what we mean.

Putting the function to use

Now, we want to actually use our readDirectory() function to list out some files. To test it, we're just going to list the directory for our project, or, ./:

/index.server.js

import node from "@joystick.js/node";
import api from "./api";
import readDirectory from './lib/node/readDirectory';

node.app({
  api,
  routes: { ... },
}).then(async () => {
  const files = await readDirectory(`./`, ['node_modules', '.joystick']);
  console.log(files);
});

Above, we've opened up the /index.server.js file at the root of the app that was created for us at the start of this tutorial. Inside, we have the node.app() function that gets called to start up our Joystick app. We expect this function to return a JavaScript Promise immediately after our HTTP server is started.

Here, on the end of that function, we've added a call to .then() which is passed a function itself to call once node.app() resolves, signifying a server startup. Inside, we set the function passed to .then() to be async so we can use the await keyword with our readDirectory() function.

To readDirectory(), we pass ./ meaning "the current directory" along with an array containing two paths as strings: node_modules and .joystick. By doing this, we expect the list of files we get back to contain all of the files in our project, except for those in the node_modules folder or the .joystick folder.

If our server is already up and running we should see the list of files printed to the server console (in your terminal/command line), like this:

[
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/api/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/i18n/en-US.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.client.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.css',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.html',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.server.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/lib/node/readDirectory.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/package-lock.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/package.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/apple-touch-icon-152x152.png',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/favicon.ico',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/manifest.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/service-worker.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/splash-screen-1024x1024.png',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/settings.development.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/components/quote/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/layouts/app/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/pages/error/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/pages/index/index.js'
]

If we remove the .joystick path from the pathsToFilter array, we'll get something like this:

[
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/PROCESS_ID',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/api/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/fileMap.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/i18n/en-US.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/index.client.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/index.css',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/index.html',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/index.server.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/lib/node/readDirectory.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/package-lock.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/package.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/public/apple-touch-icon-152x152.png',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/public/favicon.ico',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/public/manifest.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/public/service-worker.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/public/splash-screen-1024x1024.png',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/settings.development.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/ui/components/quote/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/ui/layouts/app/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/ui/pages/error/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/build/ui/pages/index/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/WiredTiger',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/WiredTiger.lock',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/WiredTiger.turtle',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/WiredTiger.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/WiredTigerHS.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/_mdb_catalog.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/collection-0-8421897594299168219.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/collection-2-8421897594299168219.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/collection-4-8421897594299168219.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/diagnostic.data/metrics.2022-12-02T02-51-54Z-00000',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/diagnostic.data/metrics.2022-12-02T02-52-28Z-00000',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/diagnostic.data/metrics.2022-12-02T03-25-24Z-00000',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/diagnostic.data/metrics.interim',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/index-1-8421897594299168219.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/index-3-8421897594299168219.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/index-5-8421897594299168219.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/index-6-8421897594299168219.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/journal/WiredTigerLog.0000000003',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/journal/WiredTigerPreplog.0000000001',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/journal/WiredTigerPreplog.0000000002',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/mongod.lock',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/sizeStorer.wt',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/.joystick/data/mongodb/storage.bson',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/api/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/i18n/en-US.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.client.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.css',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.html',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/index.server.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/lib/node/readDirectory.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/package-lock.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/package.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/apple-touch-icon-152x152.png',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/favicon.ico',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/manifest.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/service-worker.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/public/splash-screen-1024x1024.png',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/settings.development.json',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/components/quote/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/layouts/app/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/pages/error/index.js',
  '/Users/rglover/projects/cheatcode/tutorials/geH5Mxans1wE8SnD/app/ui/pages/index/index.js'
]

That should do it! Keep in mind the pathsToFilter is optional so you can either pass an empty array or nothing for the second argument to readDirectory().

Wrapping up

In this tutorial, we learned how to read and list the files in a directory in Node.js. We learned how to write a function that acted as a wrapper around fs.readdir(), implementing a recursive pattern to help us also get back files in subdirectories. To make our code even more useful, we learned how to pass an array of paths to filter by, so we could eliminate any paths from our list that we didn't want or don't need.