I wanted to outline here once how our backup system works right now so you know what is being done and what you can expect in case of a crash but also to get input in case someone with server maintenance experience wants to share possibilities for improvement.

1. We are running all limited-size world on a RAMdisk. Nether and Darklands are not. They are just too big for that. We are copying changes to the ramdisk to a save-folder with rsync to the normal disk every 5 minutes. If there is any issue (server crashes, system powers down), we should have 5 minutes loss maximum. On hardware restarts, the RAMDisk is built from these saves.
2. Every day once, I stop auto-saves on the MC server, copy all changed files with rsync to a mirror folder on a second hard drive and continue saving afterwards. This prevents us from issues where the main HD fails.
3. Every day we zip this mirror on the same 2ndary HD and store the archive in folders numbered 1-31 depending on the day of the month. Like that I keep always the last 31 days as a backup. Same happens with the databases.
4. Every week we zip the folders on top of that in folders numbered 1-52 depending on the number of the week. This way, in case we can track back an issue to originate further back than one month, we have weekly backups for one year. Same happens with the databases.
5. On top of that, I have a crashplan subscription which makes off-site backups to a cloud server of the daily databases and the weekly file mirrors of the drives. I had this subscription already to backup my photos to a secure location so no extra costs there. This is newly setup for minecraft since I did not know that I can install the crashplan client on a headless linux server.
6. Nether and darklands are neither mirrored nor backed up. The nether is now 2.6GB and the darklands 2.3GB which is together larger than all the rest of the files (5.4 GB). Given that we delete those from time to time, there was never a backup and there won’t be one.

So if anyone has some more suggestions on how to improve the system, I am happy for feedback. Finally I am not a learned sysadmin and still learn with every crash :)

Please see the latest upgrades from the voting system:

• lukecreative (from ArchitectDonator to DesignerDonator)

Congratz and thanks to all voters!

I also now resolved issues with teamspeak so the server is back online.

MineConomy is repeatedly disconnecting from the MySQL server. Since I have re-setup everything from scratch on the new server, there might be some default setting on the new version I am using now that is causing this, but Mineconomy as opposed to other plugins is not automatically reconnecting to the server. So after a while, it simply stops working.

Since Minecononmy was not updated since 1.5.2, I am not sure if the admin is still maintaining it. I sent a bug report, but it seems that the same bug was identified by other users for the last 2 versions already, marked as fixed but we still have the issue.

So what I did now: I wrote a simply plugin that can deal with money since MineConomy was basically only used to see the account status and to give money to users. The only real advantage of having a economy plugin is if other plugins have money functions (such as a shop) but we do not use any external plugins that do that. We wrote all money-related plugins ourselves. So in the end we got rid of another plugin by writing some lines of code.

The new commands are now /money status and /money give <user> <amount>. There is a appropriate help in the system and on the website.

Please see the latest upgrades from the voting system:

• kzr_nvd (from ArchitectDonatorPlus to DesignerDonatorPlus)

Congratz and thanks to all voters!

Please see the latest upgrades from the voting system:

• thegypsyminer (from Citizen to Architect)

Congratz and thanks to all voters!

The server is back online, except for some automated stuff like restarts and backups. Please check if all is as you have left it: Inventories, money, lots etc.

I had do delete a couple of nether files and one darklands file. Most of them were REALLY far out, so I do not think something got damaged. Since we do not backup the nether and the darklands, there is nothing I can do. One Kingdom file got reset by one day, here I could fallback on a backup.

Since at least one of the two HDs of the server has defects, the server is down while we fix this issue. Thanks for your patience.

In line with the latest blog posts, the server will shutdown in 7 hours from now (18:00 New York, Midnight Berlin, 7:00 Hong Kong), but not restart afterwards. This will happen at the same time when the server usually goes for daily reboot anyhow. I will then take the latest backups created, copy them to a secure location and ask the hoster to re-setup the system from scratch.

With this process the chances are the highest that the system will be back up for Thursday/Friday and that there are no unsaved changes that could be lost.

Please see the latest upgrades from the voting system:

• alerwrig (from CitizenDonatorPlus to ArchitectDonatorPlus)

Congratz and thanks to all voters!

I just fixed a voting issue where votes were closed when they got enough support but the form would not really reflect that. Users were still able to change their votes and thus we had some closed votes without the needed support. I hope I was able now to fix this. I re-opened the votes with not enough points.

Minecarts should also be fixed (again). Azkedar kindly looked into the code and made sure that accidental loops of spawning minecarts would not crash the server so easily.

In the meantime I am still doing the backups for the server. There were some technical glitches and I had to start over. I found some better ways to manage the server in the future in the process, so at least some good came out of it.

I have now (see last post) found out that the main HD of the server is failing. The hosting company will replace both HDs (main & backup). While I am pretty sure that the second HD is fine, I had to rent a FTP server on their farm to make an additional backup.

I am currently copying all needed data to the backup FTP, then I will ask the hoster to put new HDs into the system, then will have to setup everything again and copy over the data from the backup. Normally the re-setup of the server will take 5 hours, I will most likely need another couple of hours to setup everything again. If the old HD is not too broken, there is a chance that this goes quickly. We will see.

Since it’s getting late here, I will simply let the backup run overnight while I sleep and then try to do the rest tomorrow. Since I never did that before and since I do not know if I will be very busy tomorrow with other stuff and when the hoster actually will be able to replace the HDs, there is not ETA for this yet. The server will be down sometime (including the website), and then hopefully be back up again not too much later. Please check the facebook page for updates and IRC to chat.

Recently my backup process is throwing this error:

rsync: read errors mapping "/home/minecraft/server/bukkit/nether/DIM-1/region/r.2.-1.mca": Input/output error (5)

I cannot copy the file somewhere else. I know it’s not an access rights issue. Unfortunately, I am not 100% sure what to do best. There are 2 processes available that I know of:

1. restart the server (the whole machine) and schedule a disk check on startup. This means a longer downtime. Last time I suspected HD errors (not sure anymore what that was) and tried to do that, the machine would not start up by itself and I had to wait for tech support to go in and restart it.
2. Try to just delete the file and either restore the area from backup or regenerate the area from scratch. The issue with this is that in case there is an issue with the harddisk, we would not know.

Any one with more linux troubleshooting experience who can help how to best approach this?

Please be aware that the IRC channel currently has a misconfiguration and all chat channel s are displayed. Will fix it when I get back home.All fixed now.

I have now fixed IRC, and all the messages should look like in-game. The only thing odd now is that a /me event in-game shows two messages on IRC. Will have to figure this one out.

Also, I came across an issue with minecarts. If you put a pressure plate next to a black wool spawner, it can happen that the cart is spawned, pushes you out of the way, on the pressure plate, spawns a minecart again, pushes you back on the pressure plate etc. Somehow the new limiter is triggered, but it seems still to cause an endless loop that freezes the server. I have now disabled automated spawning until we resolved that issue too.

I have found that the IRC system is quite broken, a lot of in-game messages to not arrive on the IRC channel.

I therefore am researching on other systems that are better supported than the rather old plugin we were using so far. The best maintained plugin seems to have some challenges however, so until I figure out how to make it work properly (I am in contact with the plugin author), the system might be broken.

Please stand by for further information.