Postmortem: October 16th Downtime & Site Rebuild

On October 16th, 2022, cheatcode.co went completely offline and was inaccessible for ~24 hours until the evening of October 17th, 2022.

What Happened?

An automatic upgrade by CheatCode's host (DigitalOcean)—that couldn't be halted—on the Kubernetes cluster running the site migrated to a version that didn't support the site's existing Kubernetes configuration.

Specifically, the upgrade was incompatible with the site's .yaml config files (Kubernetes deprecated the file structure the cluster was dependent on between versions) and the version of Nginx Ingress Controller that was installed on the cluster.

How I Fixed It

I scrapped Kubernetes. I was already seeing intermittent outages leading up to the post-upgrade collapse which had me moving as fast as possible on getting Joystick and Joystick Deploy ready for the prime time.

The ultimate fix was to rebuild and redeploy a simple version of the site using Joystick and Joystick Deploy which only ended up taking an evening (I had spent the bulk of the day wrestling with Kubernetes only to say "screw it, let's give it a go" later that night). Since I made the move, the site has seen zero downtime and I've made significant progress on a redesign that was planned for next year thanks to that move.

Outcome

Mostly positive. Despite ~24 hours of downtime, I was able to bring the site back up with all data intact. The real win here is that it was an excellent, successful test of Joystick and the soon to be released Joystick Deploy. Redeploying the site on that stack (what I'm referring to as "The CheatCode Stack" moving forward) went off without a hitch.

The only real casualty was the site's SEO cratering due to URLs going offline, the sitemap becoming uncrawlable, and new content releases halting while I rebuilt everything (consistency is handsomely rewarded by search engines). This is slowly starting to heal but I expect it will take several months to regain previous ranking. It's not the end of the world, but still frustrating as the site had grown to ~32K per month in uniques and has slammed down to roughly 1/3 to 1/2 that (October was 13K, November is on track to do ~8-10K).

Despite that, the site will see a yearly unique total of ~275K, up 153K from the 2021 total.

The Future

The punchline: doubling and tripling down on my commitment to simple, battle-proofed tech.

What's great about this move is that now, the site runs on some seriously boring technology. Behind the scenes, Joystick Deploy utilizes HAProxy for its load balancer and then to run the app, a simple ~15 line systemd script. No Docker, no fancy scripting. Good ol' fashioned infra that just works.

While I won't divulge all of the secrets around how Joystick Deploy works, the punchline is that's the meat and potatoes of it. It's remarkably zen, and now that I've had the chance to really put it to the test: I'll never go back.

What's more, this has emboldened my "simple is best" vision for The CheatCode Stack and my drive to onboard more developers so they can benefit from it, too. While I've still got a good bit to do for a 1.0 release, with every passing day my confidence in the viability of the long-term vision for Joystick grows.

From here, my goal is to continue to steady the ship, build the remaining features I wanted to for the redesign "in transit," and keep sailing. I've been reluctant to share some of the big plans I've been working on (sneaky, sneaky) behind the scenes but I'm excited to see it all—finally—come together in 2023. It's a tad surreal to see it all working well in the real world.

As always, thank you for your continued support and patience. If you have any questions for me—or are a paying customer that would like a refund for the downtime—don't hesitate to get in touch: business@cheatcode.co.

Ryan