I have discovered with the help of the lag mapping tool that there is an area in the nether that caused extreme levels of lag:
Yellow here means that a chunk as loaded at least once when lag occurred, Red means the chunks were loaded when the worst lag occurred and blue means the chunk was live ALWAYS when lag was occurred. So the yellow areas one can ignore, but the red and blue ones means that this area has been the single biggest source of lag for the whole server, and it’s huge. Every square in this is 16 blocks. It spanned 2×2 regions or more precisely 400×400 blocks. When looking closer at the location, I found the following:
What you can see here is a MASSIVE area of lava flow. Someone (I know at least one person who is involved) has made a huge dig of a nether fortress while building a blaze spawner. While the blaze spawner was not an issue, the lava flow created a huge amount of lag. The server basically immediately went from 20 tps to 10 tps lag (minimum TPS, not 1-minute average) the moment I showed up there – despite the fact that I was alone on the server. Since the lava flow area was so huge and distributed over such a wide area, I could not even fix it. I had to reset the wider area of 4 region files, i.e. 1024×1024 blocks.
I won’t start naming names or banning people here, but this should be a reminder NOT to create unnecessary lava flows.