[Disclaimer: This is a fairly technical post about the management of this server. It does not affect gameplay, so no need to read this for your game experience]
I have been working the last 2 days on optimizing the server performance. I installed some months ago a server monitoring tool to better understand how the resources of the server work. This mostly is balanced between Memory, processor capacity and hard disk read/write speed. There are some settings I can change to shift load on the server between memory and hard disk writes for example.
Then, on Monday, I investigated the above tool and found that I can add specific reports for our database system (MySQL). On top of this, I am now observing a more detailed status report provided my the server and observe if I can weed out any poor performance. This showed me that the basic configuration that the server was installed with by default was quite bad for the current usage that we have and that some of the queries that we are doing on the database might not be optimized properly. So here is the monitor tool chart during the last 24 hours when I started to adjust the configuration of the database server:
What you can see on the left end of the chart is that the red and pink line (Buffer usage) were at 100% and the cyan line (cache usage) was at zero. So that means that I did not use at all or completely exhausted some of the resources that the server provides to reduce its work load. If there is no available cache for example, each query that is performed several times needs to be re-executed from scratch instead of simple remembering the result somewhere and re-using that.
So since the first time when the chart starts changing (around 10:00 am yesterday) I re-configured the database config several times (each with a restart of the database and minecraft server) and tried to apply generous cache and other resource settings. Once I am happy with the result based on the detailed status report, I can again reduce the memory and so on provided to give more to other processes (like the web server, the minecraft server etc).
For example, originally I had a maximum number of connections to the server of 300 (green line), a quota which was utilized most of the time around 70% and sometimes even fully exhausted (which basically stopped the minecraft server from working) to 600, which might be too many (now the usage is only around 10%). So I avoided the server rejecting connections because of the limit. The usage of connections might be reduced or increased by the change of other config settings however. So I rather run with excess resources until I removed all the alerts the server is giving me, and once I am happy with the result, I can reduce that limit to a reasonable amount.
The biggest work however will be to find out the most inefficient queries to the database that the server is doing and fix those.
So I hope you understand if I need to restart the server in the coming days a couple of times since every time I change the database config, I need to shutdown the minecraft server, too.