How to Upload Files to Amazon S3 Using the File Reader API

Getting Started

For this tutorial, we're going to need a back-end and a front-end. Our back-end will be used to communicate with Amazon S3 while the front-end will give us a user interface where we can upload our file.

To speed us up, we're going to use CheatCode's Node.js Boilerplate for the back-end and CheatCode's Next.js Boilerplate for the front-end. To get these setup, we need to clone them from Github.

We'll start with the back-end:

Terminal

git clone https://github.com/cheatcode/nodejs-server-boilerplate.git server

Once cloned, cd into the project and install its dependencies:

Terminal

cd server && npm install

Next, we need to install one additional dependency, aws-sdk:

Terminal

npm i aws-sdk

Once all of the dependencies are installed, start the server with:

Terminal

npm run dev

With your server running, in another terminal window or tab, we need to clone the front-end:

Terminal

git clone https://github.com/cheatcode/nextjs-boilerplate.git client

Once cloned, cd into the project and install its dependencies:

Terminal

cd client && npm install

Once all of the dependencies are installed, start the front-end with:

Terminal

npm run dev

With that, we're ready to start.

Increasing the body-parser limit

Looking at our server code, the first thing we need to do is modify the upload limit for the body-parser middleware in the boilerplate. This middleware is responsible for, as the name implies, parsing the raw body data of an HTTP request sent to the server (an Express.js server).

/server/middleware/bodyParser.js

import bodyParser from "body-parser";

export default (req, res, next) => {
  const contentType = req.headers["content-type"];

  if (contentType && contentType === "application/x-www-form-urlencoded") {
    return bodyParser.urlencoded({ extended: true })(req, res, next);
  }

  return bodyParser.json({ limit: "50mb" })(req, res, next);
};

In Express.js, middleware is the term used to refer to code that runs between an HTTP request initially hitting the server and being passed off to a matching path/route (if one exists).

Above, the function we're exporting is an Express.js middleware function that's a part of the CheatCode Node.js Boilerplate. This function takes in an HTTP request from Express.js—we can identify that we intend this to be a request passed to us by Express by the req, res, and next arguments that Express passes to its route callbacks—and then hands off that request to the appropriate method from the body-parser dependency included in the boilerplate.

The idea here is that we want to use the appropriate "converter" from bodyParser to ensure that the raw body data we get from the HTTP request is usable in our app.

For this tutorial, we're going to be sending JSON-formatted data from the browser. So, we can expect any requests we send (file uploads) to be handed off to the bodyParser.json() method. Above, we can see that we're passing in an object with one property limit set to 50mb. This gets around the default limit of 100kb on the HTTP request body imposed by the library.

Because we're uploading files of varied size, we need to increase this so that we don't receive any errors on upload. Here, we're using a "best guess" of 50 megabytes as the max body size we'll receive.

Adding an Express.js route

Next, we need to add a route where we'll send our uploads. Like we hinted at above, we're using Express.js in the boilerplate. To keep our code organized, we've split off different groups of routes that are accessed via functions called to from the main index.js file where the Express server is started in /server/index.js.

There, we call to a function api() which loads the API-related routes for the boilerplate.

/server/api/index.js

import graphql from "./graphql/server";
import s3 from "./s3";

export default (app) => {
  graphql(app);
  s3(app);
};

In that file, below the call to graphql(), we want to add another call to a function s3() which we'll create next. Here, app represents the Express.js app instance that we'll add our routes to. Let's create that s3() function now.

/server/api/s3/index.js

import uploadToS3 from "./uploadToS3";

export default (app) => {
  app.use("/uploads/s3", async (req, res) => {
    await uploadToS3({
      bucket: "cheatcode-tutorials",
      acl: "public-read",
      key: req.body?.key,
      data: req.body?.data,
      contentType: req.body?.contentType,
    });

    res.send("Uploaded to S3!");
  });
};

Here, we take in the Express app instance we passed in and call to the .use() method, passing the path where we'd like our route to be available, /uploads/s3. Inside of the callback for the route, we call to a function uploadToS3 which we'll define in the next section.

It's important to note: we intend uploadToS3 to return a JavaScript Promise. This is why we have the await keyword in front of the method. When we perform the upload, we want to "wait on" the Promise to be resolved before responding to the original HTTP request we sent from the client. To make sure this works, too, we've prefixed the keyword async on our route's callback function. Without this, JavaScript will throw an error about await being a reserved keyword when this code runs.

Let's jump into that uploadToS3 function now and see how to get our files handed off to AWS.

Wiring up the upload to Amazon S3 on the server

Now for the important part. To get our upload over to Amazon S3, we need to set up a connection to AWS and an instance of the .S3() method in the aws-sdk library that we installed earlier.

/server/api/s3/uploadToS3.js

import AWS from "aws-sdk";
import settings from "../../lib/settings";

AWS.config = new AWS.Config({
  accessKeyId: settings?.aws?.akid,
  secretAccessKey: settings?.aws?.sak,
  region: "us-east-1",
});

const s3 = new AWS.S3();

export default async (options = {}) => { ... };

Before we jump into the body of our function, first, we need to wire up an instance of AWS. More specifically, we need to pass in an AWS Access Key ID and Secret Access Key. This pair does two things:

1. Authenticates our request with AWS.
2. Validates that this pair has the correct permissions for the action we're trying to perform (in this case s3.putObject()).

Obtaining these keys is outside of the scope of this tutorial, but give this documentation from Amazon Web Services a read to learn how to set them up.

Assuming you've obtained your keys—or have an existing pair you can use—next, we're going to leverage the settings implementation in the CheatCode Node.js Boilerplate to securely store our keys.

/server/settings-development.json

{
  "authentication": {
    "token": "abcdefghijklmnopqrstuvwxyz1234567890"
  },
  "aws": {
    "akid": "Type your Access Key ID here...",
    "sak":" "Type your Secret Access Key here..."
  },
  [...]
}

Inside of /server/settings-development.json, above, we add a new object aws, setting it equal to another object with two properties:

• akid - This will be set to the Access Key ID that you obtain from AWS.
• sak - This will be set to the Secret Access Key that you obtain from AWS.

Inside of /server/lib/settings.js, this file is automatically loaded into memory when the server starts up. You'll notice that this file is called settings-development.json. The -development part tells us that this file will only be loaded when process.env.NODE_ENV (the current Node.js environment) is equal to development. Similarly, in production, we'd create a separate file settings-production.json.

Fair Warning: The settings-development.json file is committed to your Git repository. DO NOT upload this file with your keys in it to a public Github repo. There are scammer bots that scan Github for insecure keys and use them to rack up bills on your AWS account.

The point of this is security and avoiding using your production keys in a development environment. Separate files avoid unnecessary leakage and mixing of keys.

/server/api/s3/uploadToS3.js

import AWS from "aws-sdk";
import settings from "../../lib/settings";

AWS.config = new AWS.Config({
  accessKeyId: settings?.aws?.akid,
  secretAccessKey: settings?.aws?.sak,
  region: "us-east-1",
});

const s3 = new AWS.S3();

export default async (options = {}) => { ... };

Back in our uploadToS3.js file, next, we import the settings file we mentioned above from /server/lib/settings.js and from that, we grab the aws.akid and aws.sak values we just set.

Finally, before we dig into the function definition, we create a new instance of the S3 class, storing it in the s3 variable with new AWS.S3(). With this, let's jump into the core of our function:

/server/api/s3/uploadToS3.js

import AWS from "aws-sdk";

[...]

const s3 = new AWS.S3();

export default async (options = {}) => {
  await s3
    .putObject({
      Bucket: options.bucket,
      ACL: options.acl || "public-read",
      Key: options.key,
      Body: Buffer.from(options.data, "base64"),
      ContentType: options.contentType,
    })
    .promise();

  return {
    url: `https://${options.bucket}.s3.amazonaws.com/${options.key}`,
    name: options.key,
    type: options.contentType || "application/",
  };
};

There's not much to it so we've logged everything out here. The core function that we're going to call on the s3 instance is .putObject(). To .putObject(), we pass an options object with a few settings:

• Bucket - The Amazon S3 bucket where you'd like to store the object (an S3 term for file) that you upload.
• ACL - The "Access Control List" that you'd like to use for the file permissions. This tells AWS who is allowed to access the file. You can pass in any of the Canned ACLs Amazon offers here (we're using public-read to grant open access).
• Key - The name of the file as it will exist in the Amazon S3 bucket.
• Body - The contents of the file you're uploading.
• ContentType - The MIME type for the file that you're uploading.

Focusing on Body, we can see something unique happening. Here, we're calling to the Buffer.from() method that's built in to Node.js. As we'll see in a bit, when we get our file back from the FileReader in the browser, it will be formatted as a base64 string.

To ensure AWS can interpret the data we send it, we need to convert the string we've passed up from the client into a Buffer. Here, we pass our options.data—the base64 string—as the first argument and then base64 as the second argument to let Buffer.from() know the encoding it needs to convert the string from.

With this, we have what we need wired up to send over to Amazon. To make our code more readable, here, we chain the .promise() method onto the end of our call to s3.putObject(). This tells the aws-sdk that we want it to return a JavaScript Promise.

Just like we saw back in our route callback, we need to add the async keyword to our function so that we can utilize the await keyword to "wait on" the response from Amazon S3. Technically speaking, we don't need to wait on S3 to respond (we could omit the async/await here) but doing so in this tutorial will help us to verify that the upload is complete (more on this when we head to the client).

Once our upload is complete, from our function, we return an object describing the url, name, and type of the file we just uploaded. Here, notice that url is formatted to be the URL of the file as it exists in your Amazon S3 bucket.

With that, we're all done with the server. Let's jump down to the client to wire up our upload interface and get this working.

Wiring up the FileReader API on the client

Because we're using Next.js on the client, we're going to create a new upload page in our /pages directory that will host an example component with our upload code:

/client/pages/upload/index.js

import React, { useState } from "react";
import pong from "../../lib/pong";

const Upload = () => {
  const [uploading, setUploading] = useState(false);

  const handleUpload = (uploadEvent) => { ... };

  return (
    <div>
      <header className="page-header">
        <h4>Upload a File</h4>
      </header>
      <form className="mb-3">
        <label className="form-label">File to Upload</label>
        <input
          disabled={uploading}
          type="file"
          className="form-control"
          onChange={handleUpload}
        />
      </form>
      {uploading && <p>Uploading your file to S3...</p>}
    </div>
  );
};

Upload.propTypes = {};

export default Upload;

First, we set up a React component with just enough markup to get us a basic user interface. For the styling, we're relying on Bootstrap which is automatically set up for us in the boilerplate.

The important part here is the <input type="file" /> which is the file input we'll attach a FileReader instance to. When we select a file using this, the onChange function will be called, passing the DOM event containing our selected files. Here, we're defining a new function handleUpload that we'll use for this event.

/client/pages/upload/index.js

import React, { useState } from "react";
import pong from "../../lib/pong";

const Upload = () => {
  const [uploading, setUploading] = useState(false);

  const handleUpload = (uploadEvent) => {
    uploadEvent.persist();
    setUploading(true);

    const [file] = uploadEvent.target.files;
    const reader = new FileReader();

    reader.onloadend = (onLoadEndEvent) => {
      fetch("http://localhost:5001/uploads/s3", {
        method: "POST",
        mode: "cors",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          key: file.name,
          data: onLoadEndEvent.target.result.split(",")[1],
          contentType: file.type,
        }),
      })
        .then(() => {
          setUploading(false);
          pong.success("File uploaded!");
          uploadEvent.target.value = "";
        })
        .catch((error) => {
          setUploading(false);
          pong.danger(error.message || error.reason || error);
          uploadEvent.target.value = "";
        });
    };

    reader.readAsDataURL(file);
  };

  return (
    <div>
      <header className="page-header">
        <h4>Upload a File</h4>
      </header>
      <form className="mb-3">
        <label className="form-label">File to Upload</label>
        <input
          disabled={uploading}
          type="file"
          className="form-control"
          onChange={handleUpload}
        />
      </form>
      {uploading && <p>Uploading your file to S3...</p>}
    </div>
  );
};

Upload.propTypes = {};

export default Upload;

Filling in the handleUpload function, we have a few things to do. First, just inside of the function body, we add a call to React's .persist() method on the uploadEvent (this is the DOM event passed in via the onChange method on our <input />). We need to do this because React creates something known as a synthetic event which is not available inside of functions outside the main execution thread (more on this in a bit).

Following this, we use the useState() hook from React to create a state variable uploading and toggle it to true. If you look down in our markup, you can see we use this to disable the file input while we're mid-upload and display a feedback message to confirm the process is underway.

After this, we dig into the core functionality. First, we need to get the file that we picked from the browser. To do it, we call to uploadEvent.target.files and use JavaScript Array Destructuring to "pluck off" the first file in the files array and assign it to the variable file.

Next, we create our instance of the FileReader() in the browser. This is built-in to modern browsers so there's nothing to import.

In response, we get back a reader instance. Skipping past reader.onloadend for a sec, at the bottom of our handleUpload function, we have a call to reader.readAsDataURL(), passing in the file we just destructured from the uploadEvent.target.files array. This line is responsible for telling the file reader what format we want our file read into memory as. Here, a data URL gets us back something like this:

Example Base64 String

data:text/plain;base64,4oCcVGhlcmXigJlzIG5vIHJvb20gZm9yIHN1YnRsZXR5IG9uIHRoZSBpbnRlcm5ldC7igJ0g4oCUIEdlb3JnZSBIb3R6

Though it may not look like it, this string is capable of representing the entire contents of a file. When our reader has fully loaded our file into memory, the reader.onloadend function event is called, passing in the onloadevent object as an argument. From this event object, we can get access to the data URL representing our file's contents.

Before we do, we set up a call to fetch(), passing in the presumed URL of our upload route on the server (when you run npm run dev in the boilerplate, it runs the server on port 5001). In the options object for fetch() we make sure to set the HTTP method to POST so that we can send a body along with our request.

We also make sure to set the mode cors to true so that our request makes it pass the CORS middleware on the server (this limits what URLs can access a server—this is pre-configured to work between the Next.js boilerplate and Node.js boilerplates for you). After this, we also set the Content-Type header which is a standard HTTP header that tells our server in what format our POST body is in. Keep in mind, this is not the same as our file type.

In the body field, we call to JSON.stringify()—fetch() requires that we pass body as a string, not an object—and to that, pass an object with the data we'll need on the server to upload our file to S3.

Here, key is set to file.name to ensure the file we put in the S3 bucket is identical to the name of the file selected from our computer. contentType is set to the MIME type automatically provided to us in the browser's file object (e.g., if we opened a .png file this would be set to image/png).

The important part here is data. Notice that we're making use of the onLoadEndEvent like we hinted at above. This contains the contents of our file as a base64 string in its target.result field. Here, the call to .split(',') on the end is saying "split this into two chunks, the first being the metadata about the base64 string and the second being the actual base64 string."

We need to do this because only the part after the comma in our data URL (see the example above) is an actual base64 string. If we do not take this out, Amazon S3 will store our file but when we open it, it will be unreadable. To finish this line out, we use array bracket notation to say "give us the second item in the array (position 1 in a zero-based JavaScript array)."

With this, our request is sent up to the server. To finish up, we add a .then() callback—fetch returns us a JavaScript Promise—which confirms the uploads success and "resets" our UI. We setUploading() to false, clear out the <input />, and then use the pong alerts library built-in to the Next.js boilerplate to display a message on screen.

In the event that there's a failure, we do the same thing, however, providing an error message (if available) instead of a success message.

If all is working according to plan, we should see something like this:

Wrapping Up

In this tutorial, we learned how to upload files to Amazon S3 using the FileReader API in the browser. We learned how to set up a connection to Amazon S3 via the aws-sdk, as well as how to create an HTTP route that we could call to from the client.

In the browser, we learned how to use the FileReader API to convert our file into a Base64 string and then use fetch() to pass our file up to the HTTP route we created.