How to Upload Files to Multiple Locations Simultaneously with Joystick

Getting Started

For this tutorial, we're going to use CheatCode's full-stack JavaScript framework, Joystick. Joystick brings together a front-end UI framework with a Node.js back-end for building apps.

To begin, we'll want to install Joystick via NPM. Make sure you're using Node.js 16+ before installing to ensure compatibility (give this tutorial a read first if you need to learn how to install Node.js or run multiple versions on your computer):

Terminal

npm i -g @joystick.js/cli

This will install Joystick globally on your computer. Once installed, next, let's create a fresh project:

Terminal

joystick create app

After a few seconds, you will see a message logged out to cd into your new project and run joystick start. Before you do, we need to install one dependency, uuid:

Terminal

cd app && npm i uuid

We'll use this to generate an arbitrary UUID that we can pass along with our upload to demonstrate passing data with your upload. After that's installed, you can start up your server:

Terminal

joystick start

After this, your app should be running and we're ready to get started.

Setting up an Amazon S3 bucket

For this tutorial, one of the two locations we upload our files to will be Amazon S3 (the other will be to a folder locally within the app). For S3, we'll need to make sure we have a few things:

1. An Amazon Web Services account.
2. An Amazon IAM user to provider credentials for accessing the bucket.
3. An Amazon S3 bucket.

If you already have access to these, you can skip to the "Wiring up an uploader on the server" section below.

If you don't have these, first, head over to Amazon Web Services and create a new account here.

Once you're signed up, make sure you've completed any steps to add your billing information and then head over to the IAM Security Credentials page. From the left-hand menu, click on the "Users" option under the "Access management" subheading.

In the top-right hand corner on this page, click the blue "Add users" button. On the next page, in the "User name" box, type in a username for your IAM (Identity Access Management) user and under "Select AWS access type" tick the box next to "Access key - Programmatic access." After these are set, click "Next: Permissions" at the bottom-right corner of the page.

On the next screen, click the third box labeled "Attach existing policies directly" and then in the search box next to "Filter policies" in the middle of the page, type in "s3full" to filter the list to the AmazonS3FullAccess option. Tick the box next to this item and then click the "Next: Tags" button at the bottom-right of the page.

The "tags" page can be skipped as well as the one after it (unless you're familiar with these and would like to complete them). After these, your IAM user's credentials will be revealed.

Note: IAM credentials are like GOLD for thieves. Do not under any circumstances put these into a public Github repository or give them to someone you do not know/trust. It is very easy to leak these keys and find a surprise bill from Amazon at the end of the month with charges you didn't accrue (I speak from experience).

It's best to store these credentials in a secure location like 1Password, LastPass, or another password management tool you trust.

Once you have your credentials set up, head back to the "Users" list we started from above and click on the user you just created to reveal the "Summary" page. From here, you will want to copy the long "User ARN" string just beneath the page heading. We'll use this next to set up your bucket.

Once you have this copied, in the search box at the very top of the page (to the right of the "AWS" logo) type in s3 and select the first option that appears underneath "Services" in the search results.

On the next page, click the orange "Create bucket" button in the top-right corner of the page. From this page, we need to fill out the following fields:

1. For "Bucket name," enter a unique name (bucket names must be unique to the region that you select for the second option) that describes what your bucket will hold.
2. For "AWS Region" select the region that's either closest to the majority of your users, or, closest to yourself.
3. Under "Object Ownership," select the "ACLs enabled" box. Even though this isn't recommended, we'll need this in order to customize permissions on a per-uploader basis in your app.
4. For "Block Public Access..." this option is up to you. If your bucket will NOT store sensitive files or files that you'd like to keep private, you can untick this box (and check the "I acknowledge" warning that appears when you do). For the bucket in use for the rest of the tutorial, we've unticked this box to allow for public objects.

After those are set, you can skip the other settings and click on "Create bucket" at the bottom of the page. Once your bucket is created, locate it in the list of buckets and click on it to reveal it in the dashboard. From here, locate the "Permissions" tab at the top of the page and on this tab, locate and click the "Edit" button in the "Bucket policy" block.

In the box that pops up, you will want to paste in the following statement, replacing the <bucket-name> placeholder with the name of the bucket you just created and <user arn you copied> with the "User ARN" we copied above.

Example Amazon S3 Bucket Policy

{
  "Id": "Policy1654277614273",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1654277612532",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::<bucket-name>/*",
      "Principal": {
        "AWS": [
          "<user arn you copied>"
        ]
      }
    }
  ]
}

After this is customized for your bucket and user, scroll down and click the orange "Save changes" button. Once this is set, what we just accomplished was allowing the IAM user credentials we just created to have full access to the bucket we just created. This will come into play when we configure our uploader next and set the "ACL" ("access control list" in AWS-speak) we hinted at above.

Wiring up an uploader on the server

In order to support uploading files in a Joystick app, we need to define an uploader on the server in our /index.server.js file. Let's take a look at the basic setup and walk through it:

/index.server.js

import node from "@joystick.js/node";
import api from "./api";

node.app({
  api,
  uploaders: {
    photos: {
      providers: ['local', 's3'],
      local: {
        path: 'uploads',
      },
      s3: {
        region: 'us-east-1',
        accessKeyId: joystick?.settings?.private?.aws?.accessKeyId,
        secretAccessKey: joystick?.settings?.private?.aws?.secretAccessKey,
        bucket: 'cheatcode-tutorials',
        acl: 'public-read',
      },
      mimeTypes: ['image/jpeg', 'image/png', 'image/svg+xml', 'image/webp'],
      maxSizeInMegabytes: 5,
      fileName: ({ input, fileName, mimeType }) => {
        // NOTE: Return the full path and file name that you want the file to be stored in
        // relative to the provider.
        return `photos/${input?.photoId}_${fileName}`;
      },
    },
  },
  routes: { ... },
});

This is everything we need to support multiple-location uploads. First, up top, we're calling to the node.app() function imported from the @joystick.js/node package that starts up our server for us (using Express.js behind the scenes). To that function, we can pass options on an object to customize the behavior of our app.

Here, the uploaders option takes an object where each property defines one of the uploaders that we want to support in our app (here, we're defining an uploader called photos). To that property, we pass the object or "definition" for our uploader.

At the top, we pass a providers array of strings to specify where we want our upload to go (Joystick automatically routes the upload of a file to these providers). Here, we can specify one or more providers that will receive an upload. In this case, we want to upload to two locations: our local machine and Amazon S3.

Based on the providers that we pass, next, we need to define configuration for those specific providers.

For local, we pass an object with a single object path which specifies the local path (relative to the root of our application) where our files will be stored.

For s3, things are a bit more involved. Here, we need to specify a few different properties:

• region which is the AWS region shortcode for the region where our bucket is located.
• accessKeyId which is the "Access Key ID" you generated alongside your IAM user earlier.
• secretAccessKey which is the "Secret Access Key" you generated alongside your IAM user earlier.
• bucket which is the name of the bucket where you want your files to be stored.
• acl which is the "access control list" or catch-all permission you want to apply to all files uploaded via this uploader. For our example, we're using public-read which means files are read-only for public users.

Note: for the accessKeyId and secretAccessKey values here, notice that we're pulling these values from joystick?.settings?.private?.aws. In a Joystick app, you can specify settings for each environment in your app in the settings.<env>.json file at the root of your app (where <env> is some environment supported by your app).

Here, because we're in the development environment, we expect these values to be defined in our settings.development.json file. Here's an updated version of this file (you will need to fill in your accessKeyId and secretAccessKey that you obtained from AWS earlier):

/settings.development.json

{
  "config": {
    "databases": [
      {
        "provider": "mongodb",
        "users": true,
        "options": {}
      }
    ],
    "i18n": {
      "defaultLanguage": "en-US"
    },
    "middleware": {},
    "email": {
      "from": "",
      "smtp": {
        "host": "",
        "port": 587,
        "username": "",
        "password": ""
      }
    }
  },
  "global": {},
  "public": {},
  "private": {
    "aws": {
      "accessKeyId": "",
      "secretAccessKey": ""
    }
  }
}

A settings file in Joystick supports four root properties: config, global, public, and private. Here, we utilize the private object which is only accessible on the server to store our AWS credentials (we DO NOT want to put these in global or public as they will be exposed to the browser if we do).

Back in our uploader definition, after s3, we have some generic settings specific to the uploader. These include:

• mimeTypes which is an array of strings specifying the MIME types supported by this uploader (e.g., we only pass image MIME types here to avoid things like videos, documents, or audio files being uploaded).
• maxSizeInMegabytes the maximum file size (in megabytes) allowed for this uploader. Files over this limit will be rejected by the uploader.
• fileName a function which gives us the opportunity to customize the path/file name for the file we're uploading. This function receives an object containing the fileName, fileSize, fileExtension, and mimeType for the uploaded file as well as the input we pass from the client (more on this later). Here, we return a path which nests uploads in a folder photos and prefixes the fileName of the uploaded file with the photoId passed via the input object.

That's it! With this, now we have an uploader ready-to-go on the server. Let's jump over to the client and see how we actually upload files.

Calling to an uploader on the client

Fortunately, calling an uploader from the client is quite simple: we just need to call a single function upload from the @joystick.js/ui package (the same one we use to define our components). To make our work a bit easier here, we're going to reuse the existing /ui/pages/index/index.js file that was already created for us when we ran joystick create app earlier.

Let's replace the existing contents of that with what's below and step through it:

/ui/pages/index/index.js

import ui, { upload } from "@joystick.js/ui";
import { v4 as uuid } from "uuid";

const Index = ui.component({
  state: {
    uploads: [],
    progress: 0,
  },
  events: {
    'change input[type="file"]': (event, component) => {
      component.setState({ urls: [], }, () => {
        upload('photos', {
          files: event.target.files,
          input: {
            // NOTE: Arbitrary, just to demonstrate passing data alongside your upload.
            // This is accessible within the `fileName` function on your uploader definition.
            photoId: uuid(),
          },
          onProgress: (progress = 0, provider = '') => {
            component.setState({ progress, provider });
          },
        }).then((uploads) => {
          component.setState({ progress: 0, uploads });
        }).catch((errors) => {
          console.warn(errors);
        });
      });
    },
  },
  css: `
    .progress-bar {
      width: 100%;
      height: 10px;
      border-radius: 30px;
      background: #eee;
      margin-top: 30px;
    }

    .progress-bar .progress {
      height: 10px;
      background: #ffcc00;
      border-radius: 30px;
    }
  `,
  render: ({ when, state, each }) => {
    return `
      <div>
        <input type="file" />
        ${when(state.progress > 0, `
          <div class="progress-bar">
            <div class="progress" style="width:${state.progress}%;"></div>
          </div>
        `)}
        ${when(state.uploads?.length > 0, `
          <ul>
            ${each(state.uploads, (upload) => {
              return `<li>${upload.provider}: ${upload.url ? `<a href="${upload.url}">${upload.url}</a>` : upload.error}</li>`;
            })}
          </ul>
        `)}
      </div>
    `;
  },
});

export default Index;

Starting down at the render function, here, we specify some HTML that we want to render for our component. The important part here is the <input type="file" /> tag which is how we'll select files to upload from our computer.

Beneath this, using the when render function (this is the name used for the special "contextual" functions passed to a component's render function in Joystick) to say "when the value of state.progress is greater than 0, render this HTML." "This HTML," here, is the markup for a progress bar which will fill as our upload completes.

To simulate the fill, we've added an inline style attribute which sets the CSS width property dynamically on the inner <div class="progress"></div> element to the value of state.progress concatenated with a % percentage symbol (Joystick automatically provides us the upload completion percentage as a float/decimal value).

Beneath this, again using the when() function, if we see that state.uploads has a length greater than 0 (meaning we've uploaded a file and have received a response from all of our providers), we want to render a <ul></ul> tag which lists out the providers and URLs returned by those providers for our files.

Here, we utilize the each() render function, which, like the name implies helps us to render some HTML for each item in an array. Here, for each expected object inside of state.uploads, we return an <li></li> tag which tells us the provider for the specific uploads (e.g., local or s3) along with the URL returned by the provider.

Just above this, utilizing the css option on our components, we pass some simple styling for our progress bar (feel free to copy this and tweak it for your own app).

The important part here is the events block just above css. Here, we define the JavaScript DOM event listeners we want to listen for within our component (i.e., Joystick automatically scopes the event listeners defined here to this component). To events, we pass an object with properties defined as a string combining two values with a space in the middle: the type of DOM event we want to listen for and the element we want to listen for the event on (<event> <element>).

In this case, we want to listen for a change event on our <input type="file" /> element. When this occurs, it means that our user has selected a file they want to upload; a perfect time to trigger the upload of that file. To this property, we pass the function that Joystick will call when this event is detected on our file input.

Inside, first, we call to component.setState() to empty out our state.urls value, assuming that we're running our uploader multiple times and don't want to mix up the response URLs.

Next, inside, we call to the upload() function we've imported from @joystick.js/ui up above. This function is almost identical to the get() and set() functions in Joystick that are used for calling to API endpoints defined as getters and setters in your Joystick app.

It takes two arguments:

1. The name of the uploader that we defined on the server that will handle this upload (e.g., here, we pass 'photos' as that's the name we used for our uploader on the server).
2. An options object which provides the files we want to upload, any miscellaneous input data we want to pass along, and an onProgress function which is called whenever the progress of our upload changes.

For files here, we're just passing event.target.files which contains the browser File array provided on the change event for a file input (this is required as it tells Joystick which files we're trying to upload). For input, just for the sake of demonstration, we pass an object with a single property photoId set to a call to uuid(). This is a function from the uuid package that we installed earlier (see the import at the top of this file) that generates a random UUID value. While this isn't necessary, it demonstrates how to get extra data passed alongside our uploader for use with the fileName() function in our uploader definition.

For onProgress, whenever Joystick receives a progress event from the server, it calls the function we pass to onProgress here with two arguments: first, the progress of the upload as a percentage and provider which is the name of the provider that progress belongs to. For example, here, because we're uploading to local and s3, we would expect this to get called with some progress percentage and either local or s3 for the provider value. This allows us to track progress on a per-provider basis if we wish.

Finally, because we expect upload() to return a JavaScript Promise, we've added a .then() callback and .catch() callback on the end. If our upload completes without any issues, the .then() callback will fire, receiving an array of objects describing the upload result for each provider (i.e., one object for local, one object for s3, etc).

Because we're rendering our list of uploads down in our render() function, here, we just take the raw array and set it on state.uploads (remember, this is what we reference in our render() function).

So it's clear, at the very top of our options object passed to ui.component() we've provided a state object which sets some defaults for our two state values: uploads as an empty array [] and progress as 0.

That should do it! Now, if we select an image file from our computer and upload it, we should see our progress bar fill and a list of URLs rendered to the screen after it completes.

Wrapping up

In this tutorial, we learned how to add uploads to a Joystick app. We learned how to define an uploader on the server, specifying multiple providers/destinations, passing config for each provider, and how to customize the allowed mimeTypes, fileSize, and fileName for the file we're uploading. On the client, we learned how to call our uploader, handling both the progress for the upload as well as the resulting URLs after our upload completes.