June 1-3, 2026 Server Outage Report

[Isaac Eiland-Hall]

Well. This has been some stressful time.

I awoke June 1 to reports - thank you Roberon! - that the server was down. Investigation quickly determined that the server was down HARD, and googling quickly showed that the datacenter was down.

Although it was reported initially as a tornado, it seems a microburst directly impacted the datacenter, taking out power and destroying around half of the backup generators onsite.

An update on June 2 from datacenter staff indicated that initially 10,000 servers were down, but by that point only 319 were remaining down - alas, ours was one.

I received a reply to the ticket earlier today indicating that power was restored to the server, but that it was not booting due to file system issues. Being an unmanaged server, it was on me to reboot into the rescue mode and do what I could.

Took less than two hours to fix. DNS was set to re-propagate domains from the new temporary server back to this now-operational-again server, so because of caching at multiple levels, it will be anywhere from nowish to a day or two for everyone to see the sites back on the server here.

Postmortem: I got behind in checking for backups, which I will double-check. Also, while I was putting backups off-server, they were not off-site. I will find off-site backup storage to feed my backups to.

[prissi]

Thank you, I can imagine how stressful.