While attempting to automate some filesytem creation that involved LVM I kept running into an issue occasionally with some holding open the logical volumes. I would attempt to disable the volume using the following command:

$ lvchange -an system/storage
Logical volume system/storage contains a filesystem in use.

All of the mounts for the filesystems that were on the volume were unmounted, so it must have been a process. The trick to finding this out is to query all the processes mount files to find out what is holding this open. In my case since I’m searching for the system volume I’ll need to filter out systemd. The following the command will find the offending processes:

$ grep system /proc/*/mounts | grep -vE '(cgroup|systemd)'
/proc/338/mounts:/dev/mapper/system-storage /mnt/storage_target xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/proc/421/mounts:/dev/mapper/system-storage /mnt/storage_target xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0

From here we need to identify the offending processes using the PID identifiers from the previous output:

$ ps -q 338,421 -o comm=

Disabling, shutting down, or kill the offending processes allow the LVM to be disabled and properly removed. Of course it was systemd…