How to Stream a File in Response to an HTTP Request in Node.js

For this tutorial, we're going to use CheatCode's full-stack JavaScript framework, Joystick. Joystick brings together a front-end UI framework with a Node.js back-end for building apps.

To begin, we'll want to install Joystick via NPM. Make sure you're using Node.js 16+ before installing to ensure compatibility (give this tutorial a read first if you need to learn how to install Node.js or run multiple versions on your computer):

Terminal

npm i -g @joystick.js/cli

This will install Joystick globally on your computer. Once installed, next, let's create a fresh project:

Terminal

joystick create app

After a few seconds, you will see a message logged out to cd into your new project and run joystick start. Before you do that, we need to install one dependency mime:

Terminal

cd app && npm i mime

After that's installed, you can start up your server:

Terminal

joystick start

After this, your app should be running and we're ready to get started.

Why?

If you're building an app that's handling HTTP requests for large files (e.g., images, videos, or big documents like PDFs), knowing how to use streams is important. When reading a file from the file system in Node.js, typically, you might be used to using something like fs.readFile() or fs.readFileSync(). The "gotcha" with these methods is that they read the entire file into memory. This means that if your server uses either of these to read a file before responding to a request, it's eating up the memory of the machine your app is running on.

In contrast, streams don't load anything into memory. Instead, they send (or "pipe") the data directly to the request, meaning it never gets loaded into memory, it's just directly transferred. The disadvantage with this approach is that, depending on the size of the file you're streaming to the request, there might be a delay on the receiving end (e.g., when you see a video "buffer" in the browser, it's likely receiving data as a stream). If this is of little (or no) concern for your app, stream's are a great way to maximize efficiency.

Adding a route that returns a file stream

To show this off, we're going to set up a simple route inside of the app we just created at /files/:fileName where :fileName is a route parameter that can be replaced with the name of any file (e.g., video.mp4 or potato.png). For testing, we're going to use some randomly generated images from This Person Does Not Exist and an edited chunk of a VFX graphics reel. All files used for this tutorial can be downloaded from CheatCode's S3 bucket here.

/index.server.js

import node from "@joystick.js/node";
import api from "./api";

node.app({
  api,
  routes: {
    "/": (req, res) => {
      res.render("ui/pages/index/index.js", {
        layout: "ui/layouts/app/index.js",
      });
    },
    "/files/:fileName": (req, res) => {   
      // TODO: We'll implement our file stream response here...
    },
    "*": (req, res) => {
      res.render("ui/pages/error/index.js", {
        layout: "ui/layouts/app/index.js",
        props: {
          statusCode: 404,
        },
      });
    },
  },
});

To start, we want to open up the /index.server.js file at the root of the app we just created when we ran joystick create app above. Inside this file is the code—here, the node.app() function—used to start the HTTP server (behind the scenes, this runs an Express.js server) for your app and wire up your routes, API, and other configuration.

On the routes object here, we've defined a property /files/:fileName assigned to the route handler function used by Express.js to "handle" requests to that URL. Like we suggested earlier, the idea will be that we can send an HTTP GET request to this route, passing the name of some file we expect to exist in the position of :fileName, for example: http://localhost:2600/files/cat.jpg.

/index.server.js

import node from "@joystick.js/node";
import fs from 'fs';
import api from "./api";

node.app({
  api,
  routes: {
    "/": (req, res) => {
      res.render("ui/pages/index/index.js", {
        layout: "ui/layouts/app/index.js",
      });
    },
    "/files/:fileName": (req, res) => {   
      const filePath = `public/files/${req?.params?.fileName}`;
       
      if (fs.existsSync(filePath)) {
        // TODO: If the file exists, we'll stream it to the response here...
      }

      return res.status(404).send(`404 – File ${filePath} not found.`);
    },
    "*": (req, res) => {
      res.render("ui/pages/error/index.js", {
        layout: "ui/layouts/app/index.js",
        props: {
          statusCode: 404,
        },
      });
    },
  },
});

Next, inside of that route handler function, we create a variable const filePath which is assigned to an interpolated (meaning it takes some plain text and injects or embeds a dynamic value into it) string combining the path public/files/ with the file name passed as :fileName in our route (accessed in our code here as req.params.fileName).

The idea here is that in the public folder at the root of our app, we want to create another folder files where we'll store the files to test out our streaming. This is arbitrary and purely for example. The reason we chose this location is that the /public folder contains data we intend to be publicly available and the nested /files folder is just a way to visually separate our test data from other public files. Technically, the file you stream can come from anywhere on your server. Just be careful not to expose files you don't intend to.

What we care about most here is the if statement and the fs.existsSync() passed to it. This function (from the imported fs dependency we've added up top—a built-in Node.js library) returns a Boolean true or false telling us whether or not the given path actually exists. In our code here, we only want to stream the file if it actually exists. If it doesn't, at the bottom of our function we want to send back an HTTP 404 status code and a message letting the requester know the file does not exist.

Terminal

import node from "@joystick.js/node";
import fs from 'fs';
import mime from 'mime';
import api from "./api";

node.app({
  api,
  routes: {
    "/": (req, res) => {
      res.render("ui/pages/index/index.js", {
        layout: "ui/layouts/app/index.js",
      });
    },
    "/files/:fileName": (req, res) => {   
      const filePath = `public/files/${req?.params?.fileName}`;
       
      if (fs.existsSync(filePath)) {
        res.setHeader('Content-Type', mime.getType(filePath));
        res.setHeader('Content-Disposition', `attachment; filename="${req?.params?.fileName}"`);
        const stream = fs.createReadStream(filePath);
        return stream.pipe(res);
      }

      return res.status(404).send(`404 – File ${filePath} not found.`);
    },
    "*": (req, res) => {
      res.render("ui/pages/error/index.js", {
        layout: "ui/layouts/app/index.js",
        props: {
          statusCode: 404,
        },
      });
    },
  },
});

Now for the important stuff. First, up top, we've added an import for the mime package which will help us to dynamically detect the MIME-type ("Multipurpose Internet Mail Extensions," a well-supported standard format for describing multimedia files) for the file. This is important as we need to communicate back to the requester what the stream contains so they know how to properly handle it.

To do this, if our file exists, we begin by calling the res.setHeader() function provided by Express.js, passing the name of the header we want to set, followed by the value for that header. Here, Content-Type (the standard HTTP header for a response type format on the web) is set to the value of what mime.getType() returns for our filePath.

Next, we set Content-Disposition which is another standard HTTP header which contains instructions for how the requester should handle the file. There are two possible values for this: either 'inline' which suggests the browser/requester should just load the file directly, or, attachment; filename="<name>" which suggests the file should be downloaded (learn more here). Technically, this behavior is up to the browser or requester receiving the file to respect, so it's not worth stressing over.

Next, the important part for this tutorial: in order to create our stream, we call to fs.createReadStream() passing in the filePath and storing the result (a stream object) in a variable const stream. Now for the "magic" part. What's neat about a stream is that it can be "piped" elsewhere. This term "pipe" is taken from the same convention in Linux/Unix systems where you can do things like cat settings.development.json | grep mongodb (here the | pipe character tells the operating system to "hand off" or "pipe" the result of cat settings.development.json to grep mongodb).

In our code here, we want to pipe our stream to the Express.js response object for our route with stream.pipe(res) (best read as "pipe the stream to res"). In other words, we want to respond to a request for this route with the stream of our file.

That's it! Now, if we open up a browser and hit a URL like http://localhost:2600/files/art.mp4 (assuming you're using the example files linked from the S3 bucket above), you should see the video start loading in the browser. Pay attention to how the video's "loaded" amount continues to buffer/grow over time. This is the streaming data making its way to the browser (our requester).

Wrapping up

In this tutorial, we learned how to use streams to respond to HTTP requests. We learned how to set up a simple route, first checking to see if a file exists (returning a 404 if it doesn't) and then, how to dynamically retrieve the MIME-type for a file and then create and pipe a stream of that file's contents back to our HTTP request's response.