How to Upload Files to Multiple Locations Simultaneously with Joystick
Getting Started
For this tutorial, we're going to use CheatCode's full-stack JavaScript framework, Joystick. Joystick brings together a front-end UI framework with a Node.js back-end for building apps.
To begin, we'll want to install Joystick via NPM. Make sure you're using Node.js 16+ before installing to ensure compatibility (give this tutorial a read first if you need to learn how to install Node.js or run multiple versions on your computer):
Terminal
npm i -g @joystick.js/cli
This will install Joystick globally on your computer. Once installed, next, let's create a fresh project:
Terminal
joystick create app
After a few seconds, you will see a message logged out to cd
into your new project and run joystick start
. Before you do, we need to install one dependency, uuid
:
Terminal
cd app && npm i uuid
We'll use this to generate an arbitrary UUID that we can pass along with our upload to demonstrate passing data with your upload. After that's installed, you can start up your server:
Terminal
joystick start
After this, your app should be running and we're ready to get started.
Setting up an Amazon S3 bucket
For this tutorial, one of the two locations we upload our files to will be Amazon S3 (the other will be to a folder locally within the app). For S3, we'll need to make sure we have a few things:
- An Amazon Web Services account.
- An Amazon IAM user to provider credentials for accessing the bucket.
- An Amazon S3 bucket.
If you already have access to these, you can skip to the "Wiring up an uploader on the server" section below.
If you don't have these, first, head over to Amazon Web Services and create a new account here.
Once you're signed up, make sure you've completed any steps to add your billing information and then head over to the IAM Security Credentials page. From the left-hand menu, click on the "Users" option under the "Access management" subheading.
In the top-right hand corner on this page, click the blue "Add users" button. On the next page, in the "User name" box, type in a username for your IAM (Identity Access Management) user and under "Select AWS access type" tick the box next to "Access key - Programmatic access." After these are set, click "Next: Permissions" at the bottom-right corner of the page.
On the next screen, click the third box labeled "Attach existing policies directly" and then in the search box next to "Filter policies" in the middle of the page, type in "s3full" to filter the list to the AmazonS3FullAccess
option. Tick the box next to this item and then click the "Next: Tags" button at the bottom-right of the page.
The "tags" page can be skipped as well as the one after it (unless you're familiar with these and would like to complete them). After these, your IAM user's credentials will be revealed.
Note: IAM credentials are like GOLD for thieves. Do not under any circumstances put these into a public Github repository or give them to someone you do not know/trust. It is very easy to leak these keys and find a surprise bill from Amazon at the end of the month with charges you didn't accrue (I speak from experience).
It's best to store these credentials in a secure location like 1Password, LastPass, or another password management tool you trust.
Once you have your credentials set up, head back to the "Users" list we started from above and click on the user you just created to reveal the "Summary" page. From here, you will want to copy the long "User ARN" string just beneath the page heading. We'll use this next to set up your bucket.
Once you have this copied, in the search box at the very top of the page (to the right of the "AWS" logo) type in s3 and select the first option that appears underneath "Services" in the search results.
On the next page, click the orange "Create bucket" button in the top-right corner of the page. From this page, we need to fill out the following fields:
- For "Bucket name," enter a unique name (bucket names must be unique to the region that you select for the second option) that describes what your bucket will hold.
- For "AWS Region" select the region that's either closest to the majority of your users, or, closest to yourself.
- Under "Object Ownership," select the "ACLs enabled" box. Even though this isn't recommended, we'll need this in order to customize permissions on a per-uploader basis in your app.
- For "Block Public Access..." this option is up to you. If your bucket will NOT store sensitive files or files that you'd like to keep private, you can untick this box (and check the "I acknowledge" warning that appears when you do). For the bucket in use for the rest of the tutorial, we've unticked this box to allow for public objects.
After those are set, you can skip the other settings and click on "Create bucket" at the bottom of the page. Once your bucket is created, locate it in the list of buckets and click on it to reveal it in the dashboard. From here, locate the "Permissions" tab at the top of the page and on this tab, locate and click the "Edit" button in the "Bucket policy" block.
In the box that pops up, you will want to paste in the following statement, replacing the <bucket-name>
placeholder with the name of the bucket you just created and <user arn you copied>
with the "User ARN" we copied above.
Example Amazon S3 Bucket Policy
{
"Id": "Policy1654277614273",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1654277612532",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::<bucket-name>/*",
"Principal": {
"AWS": [
"<user arn you copied>"
]
}
}
]
}
After this is customized for your bucket and user, scroll down and click the orange "Save changes" button. Once this is set, what we just accomplished was allowing the IAM user credentials we just created to have full access to the bucket we just created. This will come into play when we configure our uploader next and set the "ACL" ("access control list" in AWS-speak) we hinted at above.
Wiring up an uploader on the server
In order to support uploading files in a Joystick app, we need to define an uploader on the server in our /index.server.js
file. Let's take a look at the basic setup and walk through it:
/index.server.js
import node from "@joystick.js/node";
import api from "./api";
node.app({
api,
uploaders: {
photos: {
providers: ['local', 's3'],
local: {
path: 'uploads',
},
s3: {
region: 'us-east-1',
accessKeyId: joystick?.settings?.private?.aws?.accessKeyId,
secretAccessKey: joystick?.settings?.private?.aws?.secretAccessKey,
bucket: 'cheatcode-tutorials',
acl: 'public-read',
},
mimeTypes: ['image/jpeg', 'image/png', 'image/svg+xml', 'image/webp'],
maxSizeInMegabytes: 5,
fileName: ({ input, fileName, mimeType }) => {
// NOTE: Return the full path and file name that you want the file to be stored in
// relative to the provider.
return `photos/${input?.photoId}_${fileName}`;
},
},
},
routes: { ... },
});
This is everything we need to support multiple-location uploads. First, up top, we're calling to the node.app()
function imported from the @joystick.js/node
package that starts up our server for us (using Express.js behind the scenes). To that function, we can pass options on an object to customize the behavior of our app.
Here, the uploaders
option takes an object where each property defines one of the uploaders that we want to support in our app (here, we're defining an uploader called photos
). To that property, we pass the object or "definition" for our uploader.
At the top, we pass a providers
array of strings to specify where we want our upload to go (Joystick automatically routes the upload of a file to these providers). Here, we can specify one or more providers that will receive an upload. In this case, we want to upload to two locations: our local machine and Amazon S3.
Based on the providers
that we pass, next, we need to define configuration for those specific providers.
For local
, we pass an object with a single object path
which specifies the local path (relative to the root of our application) where our files will be stored.
For s3
, things are a bit more involved. Here, we need to specify a few different properties:
region
which is the AWS region shortcode for the region where our bucket is located.accessKeyId
which is the "Access Key ID" you generated alongside your IAM user earlier.secretAccessKey
which is the "Secret Access Key" you generated alongside your IAM user earlier.bucket
which is the name of the bucket where you want your files to be stored.acl
which is the "access control list" or catch-all permission you want to apply to all files uploaded via this uploader. For our example, we're usingpublic-read
which means files are read-only for public users.
Note: for the accessKeyId
and secretAccessKey
values here, notice that we're pulling these values from joystick?.settings?.private?.aws
. In a Joystick app, you can specify settings for each environment in your app in the settings.<env>.json
file at the root of your app (where <env>
is some environment supported by your app).
Here, because we're in the development
environment, we expect these values to be defined in our settings.development.json
file. Here's an updated version of this file (you will need to fill in your accessKeyId
and secretAccessKey
that you obtained from AWS earlier):
/settings.development.json
{
"config": {
"databases": [
{
"provider": "mongodb",
"users": true,
"options": {}
}
],
"i18n": {
"defaultLanguage": "en-US"
},
"middleware": {},
"email": {
"from": "",
"smtp": {
"host": "",
"port": 587,
"username": "",
"password": ""
}
}
},
"global": {},
"public": {},
"private": {
"aws": {
"accessKeyId": "",
"secretAccessKey": ""
}
}
}
A settings file in Joystick supports four root properties: config
, global
, public
, and private
. Here, we utilize the private
object which is only accessible on the server to store our AWS credentials (we DO NOT want to put these in global
or public
as they will be exposed to the browser if we do).
Back in our uploader definition, after s3
, we have some generic settings specific to the uploader. These include:
mimeTypes
which is an array of strings specifying the MIME types supported by this uploader (e.g., we only pass image MIME types here to avoid things like videos, documents, or audio files being uploaded).maxSizeInMegabytes
the maximum file size (in megabytes) allowed for this uploader. Files over this limit will be rejected by the uploader.fileName
a function which gives us the opportunity to customize the path/file name for the file we're uploading. This function receives an object containing thefileName
,fileSize
,fileExtension
, andmimeType
for the uploaded file as well as theinput
we pass from the client (more on this later). Here, we return a path which nests uploads in a folderphotos
and prefixes thefileName
of the uploaded file with thephotoId
passed via theinput
object.
That's it! With this, now we have an uploader ready-to-go on the server. Let's jump over to the client and see how we actually upload files.
Calling to an uploader on the client
Fortunately, calling an uploader from the client is quite simple: we just need to call a single function upload
from the @joystick.js/ui
package (the same one we use to define our components). To make our work a bit easier here, we're going to reuse the existing /ui/pages/index/index.js
file that was already created for us when we ran joystick create app
earlier.
Let's replace the existing contents of that with what's below and step through it:
/ui/pages/index/index.js
import ui, { upload } from "@joystick.js/ui";
import { v4 as uuid } from "uuid";
const Index = ui.component({
state: {
uploads: [],
progress: 0,
},
events: {
'change input[type="file"]': (event, component) => {
component.setState({ urls: [], }, () => {
upload('photos', {
files: event.target.files,
input: {
// NOTE: Arbitrary, just to demonstrate passing data alongside your upload.
// This is accessible within the `fileName` function on your uploader definition.
photoId: uuid(),
},
onProgress: (progress = 0, provider = '') => {
component.setState({ progress, provider });
},
}).then((uploads) => {
component.setState({ progress: 0, uploads });
}).catch((errors) => {
console.warn(errors);
});
});
},
},
css: `
.progress-bar {
width: 100%;
height: 10px;
border-radius: 30px;
background: #eee;
margin-top: 30px;
}
.progress-bar .progress {
height: 10px;
background: #ffcc00;
border-radius: 30px;
}
`,
render: ({ when, state, each }) => {
return `
<div>
<input type="file" />
${when(state.progress > 0, `
<div class="progress-bar">
<div class="progress" style="width:${state.progress}%;"></div>
</div>
`)}
${when(state.uploads?.length > 0, `
<ul>
${each(state.uploads, (upload) => {
return `<li>${upload.provider}: ${upload.url ? `<a href="${upload.url}">${upload.url}</a>` : upload.error}</li>`;
})}
</ul>
`)}
</div>
`;
},
});
export default Index;
Starting down at the render
function, here, we specify some HTML that we want to render for our component. The important part here is the <input type="file" />
tag which is how we'll select files to upload from our computer.
Beneath this, using the when
render function (this is the name used for the special "contextual" functions passed to a component's render
function in Joystick) to say "when the value of state.progress
is greater than 0
, render this HTML." "This HTML," here, is the markup for a progress bar which will fill as our upload completes.
To simulate the fill, we've added an inline style
attribute which sets the CSS width
property dynamically on the inner <div class="progress"></div>
element to the value of state.progress
concatenated with a %
percentage symbol (Joystick automatically provides us the upload completion percentage as a float/decimal value).
Beneath this, again using the when()
function, if we see that state.uploads
has a length greater than 0
(meaning we've uploaded a file and have received a response from all of our providers), we want to render a <ul></ul>
tag which lists out the providers and URLs returned by those providers for our files.
Here, we utilize the each()
render function, which, like the name implies helps us to render some HTML for each item in an array. Here, for each expected object inside of state.uploads
, we return an <li></li>
tag which tells us the provider
for the specific uploads (e.g., local
or s3
) along with the URL returned by the provider.
Just above this, utilizing the css
option on our components, we pass some simple styling for our progress bar (feel free to copy this and tweak it for your own app).
The important part here is the events
block just above css
. Here, we define the JavaScript DOM event listeners we want to listen for within our component (i.e., Joystick automatically scopes the event listeners defined here to this component). To events
, we pass an object with properties defined as a string combining two values with a space in the middle: the type of DOM event
we want to listen for and the element
we want to listen for the event on (<event> <element>
).
In this case, we want to listen for a change
event on our <input type="file" />
element. When this occurs, it means that our user has selected a file they want to upload; a perfect time to trigger the upload of that file. To this property, we pass the function that Joystick will call when this event is detected on our file input.
Inside, first, we call to component.setState()
to empty out our state.urls
value, assuming that we're running our uploader multiple times and don't want to mix up the response URLs.
Next, inside, we call to the upload()
function we've imported from @joystick.js/ui
up above. This function is almost identical to the get()
and set()
functions in Joystick that are used for calling to API endpoints defined as getters and setters in your Joystick app.
It takes two arguments:
- The name of the uploader that we defined on the server that will handle this upload (e.g., here, we pass
'photos'
as that's the name we used for our uploader on the server). - An options object which provides the
files
we want to upload, any miscellaneousinput
data we want to pass along, and anonProgress
function which is called whenever the progress of our upload changes.
For files
here, we're just passing event.target.files
which contains the browser File array provided on the change
event for a file input (this is required as it tells Joystick which files we're trying to upload). For input
, just for the sake of demonstration, we pass an object with a single property photoId
set to a call to uuid()
. This is a function from the uuid
package that we installed earlier (see the import at the top of this file) that generates a random UUID value. While this isn't necessary, it demonstrates how to get extra data passed alongside our uploader for use with the fileName()
function in our uploader definition.
For onProgress
, whenever Joystick receives a progress event from the server, it calls the function we pass to onProgress
here with two arguments: first, the progress
of the upload as a percentage and provider
which is the name of the provider that progress belongs to. For example, here, because we're uploading to local
and s3
, we would expect this to get called with some progress
percentage and either local
or s3
for the provider
value. This allows us to track progress on a per-provider basis if we wish.
Finally, because we expect upload()
to return a JavaScript Promise, we've added a .then()
callback and .catch()
callback on the end. If our upload completes without any issues, the .then()
callback will fire, receiving an array of objects describing the upload result for each provider (i.e., one object for local
, one object for s3
, etc).
Because we're rendering our list of uploads down in our render()
function, here, we just take the raw array and set it on state.uploads
(remember, this is what we reference in our render()
function).
So it's clear, at the very top of our options object passed to ui.component()
we've provided a state
object which sets some defaults for our two state values: uploads
as an empty array []
and progress
as 0
.
That should do it! Now, if we select an image file from our computer and upload it, we should see our progress bar fill and a list of URLs rendered to the screen after it completes.
Wrapping up
In this tutorial, we learned how to add uploads to a Joystick app. We learned how to define an uploader on the server, specifying multiple providers/destinations, passing config for each provider, and how to customize the allowed mimeTypes
, fileSize
, and fileName
for the file we're uploading. On the client, we learned how to call our uploader, handling both the progress for the upload as well as the resulting URLs after our upload completes.