Hi All,
As some of you may have realised, the planned upgrade sort of crashed everything, and we had our longest period of downtime since the site began.
This is partly because I had to go to sleep (thanks to a newborn and a job).
The good news is that the backup process worked! We’ve restored to seconds before the upgrade took the site offline.
The bad news is that federation is likely to be… wonky… for a little while. The site may also go up and down while I undo some of the fixes I tried.
Ultimately the issue came down to the upgrade failing (I am not sure why - will be digging into this now the priority is no longer getting the site up) and then the containers not talking to eachother, so the UI wouldn’t talk to lemmy, and lemmy wouldn’t talk to the database.
I rebuilt the containers, restored the backup, restarted everything, and it’s all come back up (admittedly not perfect right now).
Importantly, I want to issue an apology. This isn’t what I want for Lemmy.zip, and it should’ve been handled way better by myself. I’m always learning but this took way longer than it should’ve, and while I take some solace in the fact the backup process worked and has been proven to work in production, the delay in being able to get this back up is entirely my fault and frankly unacceptable.
I’ll be working to document this outage, the steps it took to get it back up, and some form of repeatable plan so a repair can be replicated in the future if I’m not available.
In terms of upgrading to 0.19.11 - I will have to try again soon as it’s got some security fixes we desperately need to implement.
Thanks
Demigodrick
That graph is really kind of neat, but it seems to only be synchronizing with a single instance at a time from what I can tell. I saw the world line has dropped significantly, but the other lines don’t look like they’ve fallen yet.
Yes, the lemmy.world admins kindly manually reset the timer for their instance so it started updating straight away!
If an instance goes down, other instances slowly back off sending retries of activities so not to waste sending them to dead instances.
You can use this tool to see this info. It links lemmy.world but you can search for any instance, and then look up lemmy.zip either under failed or lagging instances. You’ll see on the far right the “next send try” time and date. Looks like a lot will try again around 9pm (although I’m not entirely sure on the timezone there) - so over the next few hours instances will send another try, see that lemmy.zip is back up, and then start federation with us again :)
It’s cool to see that there’s logic built-in that keeps instances from sending Federation requests to dead instances. But when an instance comes back online, they will re-synchronize themselves. An instance may drop out of the Federation, but when it comes back, it will get everything it missed. Eventually.