So I have been investigating on what is keeping the Villager chunks alive and got some replies there. It seems that there are some very old bugs in Spigot that they tried to fix to make it more work like the Vanilla server, which in turn caused the Villagers to become persistent despite no users being near. This has caused issues also for other servers.

The end result is that I tried to change a spigot setting that might fix the issue, but I need to observe this going forward. For this, I need to get more info from my lag-tracking plugin. There is an issue with it however. If you know basic Java, please stop by on GitHub and help me solve the issue. The issue has been resolved!