So I had a lengthy chat with a system admin at the hosting company to try and nail down the issue and see what we can do now that the server is up and running. From a look at the logfiles and the different other indicators, we concluded that the most likely thing is a network/network card hardware/network card software issue.

When the server goes down, the minecraft server actually continues to run but stops doing outgoing connections. The webserver gets incoming connections but does not serve replies to the outside. So it’s as if the server server can still hear but not talk.

To test this and what could be wrong, I would need to shutdown  the server and wait until support personnel has time to take a look at it. Since we were already 6 hours down today, I wanted to avoid that. So I will just wait if the issue happens again and then ask for support. The chances that the issue can be properly identified then should be quite good, and better than if we do a “cold check” without the issue occurring.

The good news is that we most likely will not have to cold-crash the server with a power cycle since we know that the server does not seem to 100% freeze up during this issue. Also since I spent already some hours yesterday and today to recover from the crash I now know how to get back online much faster and stop searching for issues that do not exist.