Wednesday, March 31, 2010

Is kern.openfiles more reliable than fstat?

Is kern.openfiles more reliable than fstat?

> I'm running Apache with MySQL and for the first time yesterday I hit
> the maximum file table length of my kernel, so I've been trying to
> monitor the number file descriptors in order to track down the rogue
> process that is causing the problem.
> However, I can't seem to figure out which of these two commands
> actually returns the number of active file descriptors:
> % fstat | wc -l
> % sysctl kern.openfiles
> Now I totally understand if fstat and kern.openfiles are not exact in
> their results (since the system is going to be in different states and
> both commands themselves contribute to the total number of open files),
> but they are usually in vast disagreement sometimes by as much as 250
> file descriptors. In some cases fstat indicates there are far more
> files open than kern.maxfiles allows, which doesn't make sense either.
> % fstat | wc -l ; sysctl kern.openfiles
> 1702
> kern.openfiles: 1458
> Here's some of the information from sysctl:
> kern.ostype: FreeBSD
> kern.osrelease: 4.7-RELEASE-p28
> kern.osrevision: 199506
> kern.version: FreeBSD 4.7-RELEASE-p28 #43: Wed Jul 20 09:35:30 MDT 2005
> root@fc2:/usr/src/sys/compile/VKERN
> kern.maxvnodes: 239578
> kern.maxproc: 150
> kern.maxfiles: 1600
> kern.argmax: 65536
> kern.maxfilesperproc: 11095
> kern.maxprocperuid: 5547
> kern.openfiles: 1465
> kern.maxusers: 384
> Any help is greatly appreciated,
> --Randall

kern.openfiles indicates the number of structures representing files
are in uses in the kernel. "fstat | wc -l" gives a sum over all
processes of where these files are attached to processes. These are
slightly different things.

For example, if a process forks, then the parent and child processes
will both have the file open (so that's two instances as far as
fstat is concerned) but these can reference the same structure in the
(only one reference for kern.openfiles).

Alternatively, if file discriptor that is passed from one process
to another, is closed by the sender but not yet received by the
receiver, then this file discriptor is currently attached to no
processes. That's one for kern.openfiles but zero for fstat.

Files which are only memory mapped do not usually show up in fstat's
output, but I think they will count towards kern.openfiles. Giving
fstat the "-m" option should encourage it to show you the mmapped
files. This might help you track down what's going on.

(Also, file discriptors that are leaked due to kernel bugs will be
counted by kern.openfiles but are unlikely to show up in fstat.
Thankfully these should be rare, but I do have some patches for
fstat to make it show the list of all files rather than the list
of files attached to processes.)


Install the lsof utility from ports. You may find it helpful.

No comments: