The International Simutrans Forum

Information and Announcements => Information & Announcements => Topic started by: Isaac Eiland-Hall on August 31, 2023, 01:01:54 AM

Title: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on August 31, 2023, 01:01:54 AM
Well, the server went down again last night. It last went down nearly two months ago, but this time when rebooted, did not come back up.

Long story short, I've obtained a new server. I was able to get backups which covered almost all accounts - some of my personal accounts are lost, but I have most of the files even there, just not the databases.

Meanwhile, restoring individual accounts takes time, so a lot of the smaller Simutrans accounts will be a day or two before I am able to restore them.

I believe there was a hardware issue with the previous server, but they would have had to take it offline for a day or more to check it which is why I put up with occasional failures.

Hopefully this new server will prove stable.

My apologies for all the inconvenience and annoyance.

NOTE: Backups were from 27th, so up to three days of posts/etc are lost. I'm sorry.
Title: Re: Server Outage 2023-08-30
Post by: isidoro on August 31, 2023, 01:23:38 AM
No problem.  Thanks, Isaac, for your dedication.
Title: Re: Server Outage 2023-08-30
Post by: jamespetts on August 31, 2023, 11:52:02 AM
Thank you for your continued work on this - it is much appreciated!
Title: Re: Server Outage 2023-08-30
Post by: Matthew on August 31, 2023, 03:06:56 PM
Thank you for dealing with this so quickly, Isaac. Hardware failures happen and a few hours' or days' downtime is no big deal for a hobby project.

The main thing is that your backup strategy worked and we still have the forum data from years gone by. Phew!

Title: Re: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on September 01, 2023, 02:11:16 AM
I'm hopeful this will lead to better server stability. This has been an occasional issue for nearly two years, but they would have had to take the server down for a day or so to check for problems, so I felt it was better to hang on and reboot whenever there was an issue. Not sure in hindsight if that was truly best, but... well... I'll remember that in future. lol

I'm irritated though that the backups of my personal sites and one friend (non-paying client from old days) didn't survive. For some reason, no backups generated. So we might find some missing sites that I haven't noticed yet.

Haven't had time to start restoring all the smaller Simutrans sites yet. Will be this weekend.

On the other hand, this also takes the server cost from around $150/mo to around $100/mo. :) (Server has very similar specs, yay for falling costs)
Title: Re: Server Outage 2023-08-30
Post by: makie on September 01, 2023, 12:26:26 PM
Link: https://translator.simutrans.com/ don´t work.

Maybe the IP Address is not set or wrong.

Thanks you for taking care of the forum. And yes, a good backup is, if it is present when it is need. Good work.
Title: Re: Server Outage 2023-08-30
Post by: prissi on September 01, 2023, 12:55:17 PM
Thank you Isaac for your efforts.

I think this concerns all dns forwarding, the servers.simutrans.org is also not working. It was shortly working between 1:33 and 5:35

So the server.simutrans.org (which is internally CT234.pm2-host.1awww.com) should point to
2001:41d0:700:5b4b::234
176.31.12.55

Would be nice if the DNS entry could be added again. Thank you.
Title: Re: Server Outage 2023-08-30
Post by: jamespetts on September 02, 2023, 06:36:33 PM
I note that the Extended list-server is currently not working.
Title: Re: Server Outage 2023-08-30
Post by: jamespetts on September 04, 2023, 05:40:27 PM
It seems to be back - excellent!

Or apparently not, sadly.
Title: Re: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on September 06, 2023, 06:11:20 PM
I apologize for the delays. I hope to make significant progress starting this evening. Work has been a beast and my health has been delicate. :/
Title: Re: Server Outage 2023-08-30
Post by: jamespetts on September 07, 2023, 06:00:13 PM
Quote from: Isaac Eiland-Hall on September 06, 2023, 06:11:20 PMI apologize for the delays. I hope to make significant progress starting this evening. Work has been a beast and my health has been delicate. :/
I hope that you feel better soon!
Title: Re: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on September 10, 2023, 01:59:52 AM
Quote from: prissi on September 01, 2023, 12:55:17 PMserver.simutrans.org

It took me far too long to figure out that I don't know where simutrans.org was attached on the old server, but it didn't' make it to the new one.

I've parked it on top of simutrans.com and added A/AAAA (ipv4/6) records for that, so propagation time - it should work soon.
Title: Re: Server Outage 2023-08-30
Post by: prissi on September 10, 2023, 08:38:12 AM
Thank you very much and I hope you are doing well.
Title: Re: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on September 12, 2023, 07:57:09 PM
The forum was down for unknown time today as a table became corrupted.

I've noticed some Jetpack warnings of short outages that I don't yet know if are legit or not. Load on the server does not seem excessive and this server should be plenty to handle our needs...

Keeping an eye on it for now.
Title: Re: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on September 14, 2023, 03:33:47 PM
:sigh:

Overnight, monitoring showed the server unavailable for short periods of time, and I found the forum with a crashed table.

I'm currently deciding what to try to do.
Title: Re: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on September 22, 2023, 06:42:03 PM
Update: Five days ago, I had them pull the server and test the hardware. A bad processor was found and replaced.

Four days ago, the server came back up and has been running since. So it appears we have — at least for now — achieved stability. Finally.

All server resources that I'm aware of have been restored by now, so if something is missing, it might have been lost. :/ (I had some personal backups that were corrupted)
Title: Re: Server Outage 2023-08-30
Post by: Roboron on September 22, 2023, 10:18:41 PM
Normality at last! Thanks!
Title: Re: Server Outage 2023-08-30
Post by: Isaac Eiland-Hall on September 22, 2023, 10:45:41 PM
Quote from: Roboron on September 22, 2023, 10:18:41 PMNormality

Nono, I'm still around. ;-)