So the logfiles and the memory test did not reveal anything. If the machine goes down again, I will ask support to take a look at the machine instead of just power-cycling it.
Maybe there is just an issue with the network card or similar that is causing issues.
The errors that I got over the last 24 hours during this and the last crash pulled me off-track to find out what is wrong. The first thing that happened yesterday and today was that on restart of the server the “session.lock” files of the worlds where reported to be write-protected. This can happen when the server crashes mid-session, so deleting those does the trick normally. But then, several worlds reported files to be inaccessible despite the fact that the user rights on these files were good. I guess there is some instance that when those files are written to while the server goes down, there are some issues leftover and the file has to be reloaded from backup. Luckily, and strangely, so far only the draftlands, hunger and city were affected. I have no idea why because those have been hardly accessed at all. This made me worry that I have another HD crash like some time ago. I still cannot rule out this issue.
The next thing I ran into was the issue that I did not auto-start the ntpd, which automatically synchronizes the time. The server, when crashing at 10:00am HKG time, restarts with that time, assuming it’s UTC. It then reads in a config file that the time should be HKG time and adds 8 hours to it. Now we have 18:00 in the evening. After the crash yesterday I started everything up, realized we have the wrong time and then fixed it by rewinding the clock 8 hours. Problem was that there is a file cache for PHP that does not report any file changes to the webserver and continues to use he old versions that were all seemingly 8 hours newer than what was on disk…. So for 3 hours I went mad because none of the file changes I did made an effect on the server, until I figured out I needed to restart the webserver…. Since I now have the time fixed on start, this wont be an issue anymore.
I am still, as written above, not clear why the server crashed at all. All I can see from the logfile in Minecraft is that all users lose network connection before that happens.
Another thing that was affected was enderchests, they were rolled back a while.
Ender chests have always been unstable. Best just hide your valuables rather than put them in enderchests.
My inv was rolled back around 24 hours, but not my chests, yesterday was a bad day to craft diamond armor :P