Moved to self-hosting with Coolify
The move to self-hosting has to be like pulling a band-aid, fast and the sooner the better. When you get locked in all of the juicy perks the vendors have to offer, it's very hard to live without them.
Not another cloud billing horror story
There are plenty of horror stories with obscene bills from cloud providers. Fortunately, I am not in that category, but I don’t want to become complacent either.
Old infrastructure
GetSite started with a hybrid infrastructure. Some services were deployed on Hetzner with Rancher, and others were split between AWS and MongoDB.
On the MongoDB cloud, I was using Atlas and ran a serverless instance for the database. The invoices started to ramp up, but not in direct proportion with the traffic we were getting. At first it was a couple of bucks per month and very quickly got up to $40 / mo. As you might have guessed, this was a strong motivator to move from Mongo.
Before making the move, I tried to figure out what inflated the invoices so much. The database instance was getting 20 req/s at all times, and had no idea why. The official support was quite unresponsive and not helpful in this regard.
Mind you, this spike in requests has started about a month ago, however the traffic to GetSite didn’t increase. Something’s fishy if you ask me, but I didn’t want to spend more time looking into it.
On AWS, I had a few Elastic Beanstalk instances that were quite expensive compared with Hetzner. The instances were part of an autoscaling group.
Did I mention I was using Rancher 1 for the self hosted services? It is a great piece of software, but it was discontinued a while back. Rancher 2 is very complex to use and geared towards enterprise users and big clusters.
New infrastructure
Enter Coolify, the successor to Rancher. I have been trying to find a replacement for Rancher for a few years. All container managers (Portainer included) seem to be tailored to the enterprise. I read about Coolify a few months ago but wasn’t convinced. Their website is not doing them justice.
A couple of weeks ago, after the inflated MongoDB invoice, I decided to jump in and install Coolify on a Hetzner server. The onboarding process is a bit unintuitive, but it grew on me as I started using it.
I moved all services from AWS, MongoDB, Rancher, and some old MySQL and Redis servers to Coolify in a day. Spent another day setting up workers, load balancers, backups, security, and testing everything out. All of this while keeping 100% uptime (ok… maybe like 99.99%).
Costs
I cut costs down to about $40 / month for the whole infrastructure. That’s how much the last MongoDB invoice was, for reference. While bringing the cost down, I also beefed up the server capacity. The current setup has plenty of spare room for future growth.