Good day everyone! As we all know we are getting closer to this years Gain Jam and I wanted to take some time to go over our plan for upgrading the forums.
As many of you may have seen during last years jam the forum was having a bit of trouble keeping up with the weight of the traffic last year (if you would pardon the expression). And as our little community continues to grow its been putting more and more strain on our back end servers. For right now load is holding at around 60% but we are seeing more and more spikes pushing us to ~80-90%.
Due to this fact, needing to consolidate services for the main site, and us wanting to make sure we are as prepared for this years gain jam as possible we are going to being slowly migrating the forums over to AWS. We will also be taking this chance to adjust our setup from a single server setup to a more complex but flexible multi-tiered setup. In order to do this with minimum disruption we came up with a simple 3 phase plan with the goal to try to have phase 2 finished before the gain jam this year.
Phase 1 - Migrate Uploads Storage to S3
In our first phase the goal is fairly simple, to move upload storage to S3. Currently all site uploads are hosted on the same server the site runs on putting strain on its storage, so moving all uploads to a S3 bucket will help reduce this storage overhead while giving us a more flexible way to store files. This is also the first step to decentralizing the server which will be required latter for phase 3.
Luckily, this task is not too hard thanks to discourse’s built in S3 integration but will require us to run a special migration task to pull over the existing uploads which could put the forums in read only mode. Also, unlike the other stages a S3 upload migration is not reversible so once we do it there is no turning back making us want to ensure we have everything planned out well beforehand.
Phase 2 - Migrate the DB to AWS’s RDS
The second phase is one of the most important, and that is migrating the discourse DB from the intergraded postgres server to a RDS hosted postgres server.
This step is much more complex then phase 1 and will require us to run several staging tests before we can switch over RDS. Once we have a good idea of how to move the data over safely and reliably and hook up the existing servers to it we will have to likely put the site into read only mode for a day while we take a recent backup and preform the migration. Once this is done though, this will give us the flexibility to scale our DB as required now and take a fair bit of strain of the existing hardware.
Phase 3 - Migration of the App Servers to AWS
Finally, we will be able to move onto the last step and that is moving the app server to AWS. This task alone is actually not to difficult due to how Discourse works, but requires a fair bit of additional infrastructure to be moved over and setup before we can move forward with it.
The plan is to host two app services on a kubernetes cluster hosted on AWS. This is the same system that will host the main site services as well, and by hosting 2 instances it will allow us to preform rolling updates on the forums minimizing site downtime for maintenance.
Unfortunately, before we can do this we need to setup a container registry, update our build system, create an EKS cluster, then migrate our DNSs over to Rout53 so we can reliably setup HTTPs connection in AWS. Luckily, we need to do most of this for the main site anyway but just make this phase take longer.
Unfortunately, this is not cheap and we expect to see our operating costs increase between $75-$200 a month. Luckily, the ad rev we are getting right now will help offset most of this cost but as usual, if you like what we do and wish to support us feel free to check out our Patreon!