Running out of vnodes while running Synth

Synth has frozen on more than one occasion while it was running commands like umount /var/synth/builders/synth-live/SL04/ccache and building more than one package at a time. Manually running commands such as df -ah appeared deadlocked since they never finished, and Ctrl+C never aborted the processes. A reboot was needed, well, I always ended up giving the reboot command from the kernel debugger. Incidently, I could instead have issued the panic command, giving me a crash dump to examine.

Googling for “freebsd vfs deadlock nullfs zfs synth” led me to https://lists.freebsd.org/pipermail/freebsd-current/2013-July/043152.html and https://forums.freebsd.org/threads/what-price-do-you-pay-when-the-system-runs-out-of-vnodes.70877/.

ZFS needs a new vnode from vnlru and is waiting on it, while vnlru has to wait on ZFS. — Andriy Gapon

Yes, that can very well be what I observed in my FreeBSD/amd64 14-STABLE system.

My system had initially kern.maxvnodes set to 4222708. This value is determined by the kernel at boottime.

I decided to double this value at runtime, stored the new value in /etc/sysctl.conf for posterity, and this seems to have done some good.

# sysctl -n kern.maxvnodes
4222708

# sysctl kern.maxvnodes=8445416
kern.maxvnodes: 4222708 -> 8445416

I also wrote the shell script shown below.

#!/bin/sh
#set -x
# Ref 1: Last parapgraph of https://lists.freebsd.org/pipermail/freebsd-current/2013-July/043152.html
# Ref 2: https://forums.freebsd.org/threads/what-price-do-you-pay-when-the-system-runs-out-of-vnodes.70877/

tsutc=`date -u +%FT%T%z`
maxvnodes=`sysctl -n kern.maxvnodes`
numvnodes=`sysctl -n vfs.numvnodes`
freevnodes=`sysctl -n vfs.freevnodes`
wantfreevnodes=`sysctl -n vfs.wantfreevnodes`

computation=$(( ${maxvnodes} - ${numvnodes} + ${freevnodes} ))
bottleneck=$(( ${computation} < ${wantfreevnodes} ))

echo "Timestamp      = ${tsutc}"
echo "maxvnodes      = ${maxvnodes}"
echo "numvnodes      = ${numvnodes}"
echo "freevnodes     = ${freevnodes}"
echo "wantfreevnodes = ${wantfreevnodes}"
echo "computation    = ${maxvnodes} - ${numvnodes} + ${freevnodes} = ${computation}"

echo
echo -n "Verdict: "

if [ "${bottleneck}" -eq 0 ]; then
  echo "No bottlenecks detected."
else
  echo "Bottleneck detected."
fi

# EOF

Here’s a sample:

Timestamp      = 2025-10-30T14:05:11+0000
maxvnodes      = 8445416
numvnodes      = 8375803
freevnodes     = 4885182
wantfreevnodes = 2111354
computation    = 8445416 - 8375803 + 4885182 = 4954795

Verdict: No bottlenecks detected.

Leave a Reply Cancel Reply