Server Outage 2023-08-30

[Isaac Eiland-Hall]

Well, the server went down again last night. It last went down nearly two months ago, but this time when rebooted, did not come back up.

Long story short, I've obtained a new server. I was able to get backups which covered almost all accounts - some of my personal accounts are lost, but I have most of the files even there, just not the databases.

Meanwhile, restoring individual accounts takes time, so a lot of the smaller Simutrans accounts will be a day or two before I am able to restore them.

I believe there was a hardware issue with the previous server, but they would have had to take it offline for a day or more to check it which is why I put up with occasional failures.

Hopefully this new server will prove stable.

My apologies for all the inconvenience and annoyance.

NOTE: Backups were from 27th, so up to three days of posts/etc are lost. I'm sorry.

[isidoro]

No problem.  Thanks, Isaac, for your dedication.

[jamespetts]

Thank you for your continued work on this - it is much appreciated!

[Matthew]

Thank you for dealing with this so quickly, Isaac. Hardware failures happen and a few hours' or days' downtime is no big deal for a hobby project.

The main thing is that your backup strategy worked and we still have the forum data from years gone by. Phew!

[Isaac Eiland-Hall]

I'm hopeful this will lead to better server stability. This has been an occasional issue for nearly two years, but they would have had to take the server down for a day or so to check for problems, so I felt it was better to hang on and reboot whenever there was an issue. Not sure in hindsight if that was truly best, but... well... I'll remember that in future. lol

I'm irritated though that the backups of my personal sites and one friend (non-paying client from old days) didn't survive. For some reason, no backups generated. So we might find some missing sites that I haven't noticed yet.

Haven't had time to start restoring all the smaller Simutrans sites yet. Will be this weekend.

On the other hand, this also takes the server cost from around $150/mo to around $100/mo. (Server has very similar specs, yay for falling costs)

[makie]

Link: https://translator.simutrans.com/ don´t work.

Maybe the IP Address is not set or wrong.

Thanks you for taking care of the forum. And yes, a good backup is, if it is present when it is need. Good work.

[prissi]

Thank you Isaac for your efforts.

I think this concerns all dns forwarding, the servers.simutrans.org is also not working. It was shortly working between 1:33 and 5:35

So the server.simutrans.org (which is internally CT234.pm2-host.1awww.com) should point to
2001:41d0:700:5b4b::234
176.31.12.55

Would be nice if the DNS entry could be added again. Thank you.

[jamespetts]

I note that the Extended list-server is currently not working.

[jamespetts]

It seems to be back - excellent!

Or apparently not, sadly.

[Isaac Eiland-Hall]

I apologize for the delays. I hope to make significant progress starting this evening. Work has been a beast and my health has been delicate. :/

[jamespetts]

I apologize for the delays. I hope to make significant progress starting this evening. Work has been a beast and my health has been delicate. :/

I hope that you feel better soon!

[Isaac Eiland-Hall]

server.simutrans.org

It took me far too long to figure out that I don't know where simutrans.org was attached on the old server, but it didn't' make it to the new one.

I've parked it on top of simutrans.com and added A/AAAA (ipv4/6) records for that, so propagation time - it should work soon.

[prissi]

Thank you very much and I hope you are doing well.

[Isaac Eiland-Hall]

The forum was down for unknown time today as a table became corrupted.

I've noticed some Jetpack warnings of short outages that I don't yet know if are legit or not. Load on the server does not seem excessive and this server should be plenty to handle our needs...

Keeping an eye on it for now.

[Isaac Eiland-Hall]

:sigh:

Overnight, monitoring showed the server unavailable for short periods of time, and I found the forum with a crashed table.

I'm currently deciding what to try to do.

[Isaac Eiland-Hall]

Update: Five days ago, I had them pull the server and test the hardware. A bad processor was found and replaced.

Four days ago, the server came back up and has been running since. So it appears we have — at least for now — achieved stability. Finally.

All server resources that I'm aware of have been restored by now, so if something is missing, it might have been lost. :/ (I had some personal backups that were corrupted)

[Roboron]

Normality at last! Thanks!

[Isaac Eiland-Hall]

Normality

Nono, I'm still around.