For a really long time I’ve seen many cPanel servers have improper space reporting and quota problems due to the VirtFS system, which is used for the “jailshell” login for users (if a user logs into SSH [or presumably Telnet] whom has their shell set to “/usr/local/cpanel/bin/jailshell” in “/etc/passwd”). Whenever a user logs into their jailshell, it creates a virtual directory structure to chroot them into, and “hard links” their home directory inside this virtfs directory, which doubles their apparent hard disk usage from their home directory (does not include their sql files, for example). Sometimes, the virtfs directory is not unmounted (presumably if the user does not log out correctly, which I have not confirmed), so the doubled hard disk usage is permanent, causing them to go over their quota if a full quota update is ran.
I had given this up as a lost cause a while back, because all the information I could find on the Internet said to just leave it alone and not worry about it, and that it couldn’t be fixed. I picked the problem back up today when one of our servers decided to do a quota check update and a bunch of accounts went over. It seems a script was added recently at “/scripts/clear_orphaned_virtfs_mounts” that fixes the problem :-) (not sure when it was added, but the creation, modification, and access times for the files on all 3 of my cPanel servers shows as today...). Surprisingly, I could not find this file documented or mentioned anywhere on the Internet, and still no mentions anywhere on how to fix this problem.
So the simplest way to fix the problem is to run
/scripts/clear_orphaned_virtfs_mounts --clearall
I did some digging and found that that script calls the function “clean_user_virtfs” in “/scripts/cPScript/Filesys/Virtfs”, so you can just clear one user directory “properly” by running
cd /scripts
perl -e 'use cPScript::Filesys::Virtfs();cPScript::Filesys::Virtfs::clean_user_virtfs("ACCOUNTNAME");'
All this script really does is unmount the home directory inside the jailed failed system (so the hard link is done through a “mount” that doesn’t show up in “df”... interesting). So the problem can also be fixed by
umount -f /home/virtfs/USERNAME/home/USERNAME
After this is done, a simple
/scripts/fixquotas
can be used to set the users’ quotas back to what they should be. Do note that this operation goes through every file on the file system to count users’ disk usages.
