ufs_dirbad panic with mangled entries on FreeBSD

FreeBSD UFS makes usually a very good job in staying sane and repairing itself. However, it can happen that UFS is not capable of repairing and some “mangled entries” appear which result in kernel panics. Unfortunatlely these are not repairable by fsck. How do they look like?

/: bad dir ino 32578 AT OFFSET 33812: MANGLED ENTRY
panic: ufs_dirbad: bad dir

So what should we do if i encounter one of these lookalike messages? Well…

Reboot your machiine in single user mode, usually option 2 in the boot menu. After bootup start a filesystem check with repairing:

$ fsck -y /
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
112346 files, 42158044 used, 59151489 free (9993 frags, 7392687 blocks, 0.0% fragmentation)

Even if no error was found and the disk was marked clean, there can be problems with some inodes/files. You may bring your system back up without any further work, however if it panics again with the same message (check out the inode number -> ino), you are likely to have unfixable corruption.

Now how can you fix it? Use the filesystem debugger fsdb. Please note that in our example the concerned inode is 32578 (ino 32578 in our error message), this is likely to change in your error.

$ fsdb /dev/da0p2
** /dev/da0p2
Editing file system '/dev/da0p2'
Last mounted on /
...
fsdb (inum: 2)>

Now go to the inode mentioned in the panic and delete it: WARNING: you will lose data when you clear the inode! Keep it in mind.

fsdb (inum: 2)> inode 32578
...
fsdb (inum: 32578)> clri 32578
fsdb (inum: 32578)> quit

**** FILE SYSTEM STILL DIRTY *****
*** FILE SYSTEM MARKED DIRTY
*** BE SURE TO RUN FSDK TO CLEAN UP ANY DAMAGE
*** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload

Now start fsck again like in the beginning of this article. Run it until no more errors are shown and it is MARKED AS CLEAN.

That’s it. Reboot normally and hope that not more inodes are faulty. If so, repeat this for every inode throwing the initial error/panic.

How to Change Open Files Limit on OS X and macOS

To check the current limits on your Mac OS X system, run:

$ launchctl limit maxfiles

The last two columns are the soft and hard limits, respectively.

Adjusting Open File Limits

To adjust open files limits on a system-wide basis, you must create a configuration file. It is a property list (aka plist) file in /Library/LaunchDaemons/limit.maxfiles.plist that contains the following XML configuration:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
      <string>launchctl</string>
      <string>limit</string>
      <string>maxfiles</string>
      <string>200000</string>
      <string>200000</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>

This will set the open files limit to 200000. The file must be owned by root:wheel and have permissions -rw-r–r–. This permissions should be in place by default, but you can ensure that they are in place by running:

$ sudo chmod 644 /Library/LaunchDaemons/limit.maxfiles.plist