The FUSE Protocol

Filesystem in Userspace (FUSE) is a protocol for implementing UNIX-style filesystems outside of the OS kernel. It was initially developed for Linux, and has seen some limited adoption by other kernels.

I wanted to write a library for the userspace side of FUSE as an exercise in learning Rust, but got stuck on a lack of documentation regarding the protocol, its versions, and how it varies across kernels. This page contains my notes on the FUSE protocol.

Similar documents elsewhere on the web:

Other resources:

Versions

The FUSE protocol is versioned with a (major, minor) tuple. Backwards compatibility is freely broken in “minor” releases, so it’s not a SemVer-style version. I tend to think of it as separate “handshake version” and “protocol version”, each being equivalent to a SemVer major version.

In this version table, I use Linux as the reference version and note when other kernel-side implementations break Linux compatibility. In practice this seems to happen only for the Darwin (macOS) port, which is maintained as a third-party kernel module.

VersionDateLinuxFreeBSDDarwin (macOS)
v7.22005-10-272.6.14
v7.3 (diff)2006-01-032.6.15
v7.6 (diff)2006-03-202.6.16
v7.7 (diff)2006-09-202.6.18
v7.8 (diff)2007-02-052.6.20
2007-10-26MacFUSE 1.0
2014-01-2010.0
v7.8-darwin1 (diff)2008-04-25MacFUSE 1.5
v7.8-darwin2 (diff)2008-06-30MacFUSE 1.7
v7.8-darwin3 (diff)2008-12-08MacFUSE 2.0
v7.9 (diff)2008-01-242.6.24
v7.10 (diff)2008-12-252.6.28
v7.11 (diff)2009-03-232.6.29
v7.12 (diff)2009-09-092.6.31
v7.13 (diff)2009-12-032.6.32
v7.14 (diff)2010-08-012.6.35
v7.15 (diff)2010-10-202.6.36
v7.16 (diff)2011-03-142.6.38
v7.17 (diff)2011-10-243.1
v7.18 (diff)2012-03-183.3
v7.19 (diff)2012-07-213.5
v7.19-darwin1 (diff)2015-01-11FUSE for OS X 3.0.0
v7.19-darwin2 (diff)2015-03-09FUSE for OS X 3.0.2
v7.20 (diff)2012-09-303.6
v7.21 (diff)2013-04-283.9
v7.22 (diff)2013-06-303.10
v7.23 (diff)2014-06-083.15
v7.24 (diff)2016-03-134.5
v7.25 (diff)2016-07-244.7
v7.26 (diff)2016-12-114.9
v7.27 (diff)2018-08-124.18

FreeBSD

FUSE was maintained as a “fusefs-kmod” port until it merged into the kernel in FreeBSD v10.0 as sys/fs/fuse/fuse_kernel.h1.

The FreeBSD developers have kept their implementation pinned to v7.8 to maintain ABI compatibility, instead of using FUSE version negotiation. There are occasional ABI-compatible changes relative to the Linux version, such as removing fuse_mknod_in or splitting the xattr request/response types2.

Darwin (macOS)

The first FUSE implementation for Darwin was MacFUSE, written by Amit Singh3 in 2007 as a loadable kernel module:

The MacFUSE kernel ABI was based on FUSE v7.8 but had a few Darwin-specific struct fields and opcodes. The last stable release of MacFUSE I can find was v2.0, released in 2008.

In 2011, Benjamin Fleischer forked MacFUSE into OSXFUSE and resumed development. In 2015 it was rebased to the v7.19 ABI, keeping the Darwin-specific fields and opcodes. As of 2018 there have been no further changes to the kernel ABI.

Other Kernels

Wire Format

TODO

fuse_request_header
offset+0+1+2+3+4+5+6+7
0x00lengthopcode
0x08request_id
0x10node_id
0x08user_idgroup_id
0x20thread_id

fuse_response_header
offset+0+1+2+3+4+5+6+7
0x00lengtherror (signed)
0x08request_id

TODO

Mounting

Linux (/dev/fuse)

TODO

Linux (fusermount)

TODO

FreeBSD

TODO

Darwin (osxfuse)

TODO

Special Topics

Extended Attributes

Extended attributes or “xattrs” are key-value items that may be associated with filesystem nodes. Keys are C-style null-terminated strings; values are arbitrary byte blobs. See xattr(7) for more details on their use and semantics.

FUSE supports xattrs through four opcodes that directly map to libattr functions, documented by :

Because no UNIX API would be complete without some sharp corners to stub your toes on, the libattr authors invented ENOATTR. There no no such error code defined in the POSIX standard and it’s not guaranteed to be defined by system headers, so libattr defines ENOATTR equal to ENODATA if it’s not already set:

ENOATTR
The named attribute does not exist, or the process has no access to this attribute. (ENOATTR is defined to be a synonym for ENODATA in <attr/xattr.h>.)

Read that again!

The API of extended attributes depends on the content of third-party userland headers!

And if that’s not enough, ENODATA is itself optional – UNIX systems that don’t implement the XSI STREAMS Option Group might not have a definition of ENODATA. FreeBSD is in this category.

PlatformENOATTRENODATA
Linux (x86-64)61
Linux (sparc)111
Darwin (x86-64)9396
FreeBSD x86-64)87

In practice I’ve found it easiest to hardcode the error behavior to whatever that platform’s native filesystems do, even if the resulting behavior deviates from the libattr manpages.

If this isn’t handled well by the FUSE library, then filesystem authors will try to do it themselves and probably get it wrong. See [tech-kern@netbsd.org] ENOATTR vs ENODATA for the trouble caused by a filesystem assuming ENODATA == ENOATTR.

CUSE

Character Devices in Userspace (CUSE) lets a FUSE server export operations as a Linux character device instead of a filesystem. Most of the behavior is the same, and the CUSE “mount” acts like a filesystem containing a single file.

Differences from standard FUSE:

Block Devices (fuseblk)

TODO

This seems to exist so the kernel can use a FUSE server to interpret bytes on a block device. ntfs-3g is the main user?

Debugging

TODO

The user can mount a “control filesystem” to inspect FUSE state and forcefully abort an existing FUSE server mount.

1
2
3
4
5
$ mount -t fusectl none /sys/fs/fuse/connections
$ ls /sys/fs/fuse/connections
42/  44/  46/  47/  48/  50/  51/  52/  53/
$ ls /sys/fs/fuse/connections/42
abort  congestion_threshold  max_background  waiting

There’s some basic docs at https://www.kernel.org/doc/Documentation/filesystems/fuse.txt

Multi-Threading

Background: When a file is opened, the Linux kernel creates a “file description” for the I/O state, and returns a “file descriptor” to userland. That descriptor can be freely passed to the dup(2) functions to duplicate the descriptor, but the underlying description remains unary.

The FUSE kernel driver implicitly locks access to the /dev/fuse file descriptor so that each read() and write() syscall is atomic. This implies that multiple threads can safely share the descriptor, but also that they will face lock contention and reduced performance.

To get the best performance out of a multi-threaded filesystem server, open /dev/fuse once as a “session FD” and again in each thread as “worker FDs”. After initializing the session with a standard FUSE handshake, the workers can be associated with the session by calling ioctl(worker_fd, FUSE_DEV_IOC_CLONE, &session_fd).

This allows multiple threads to serve FUSE requests without contending for the descriptor lock.

POSIX ACLs

TODO

Notifications

TODO

Locks

TODO

Opcodes

namevalueversionnotes
FUSE_LOOKUP1
FUSE_FORGET2
FUSE_GETATTR3
FUSE_SETATTR4
FUSE_READLINK5
FUSE_SYMLINK6
7
FUSE_MKNOD8
FUSE_MKDIR9
FUSE_UNLINK10
FUSE_RMDIR11
FUSE_RENAME12
FUSE_LINK13
FUSE_OPEN14
FUSE_READ15
FUSE_WRITE16
FUSE_STATFS17
FUSE_RELEASE18
19
FUSE_FSYNC20
FUSE_SETXATTR21
FUSE_GETXATTR22
FUSE_LISTXATTR23
FUSE_REMOVEXATTR24
FUSE_FLUSH25
FUSE_INIT26
FUSE_OPENDIR27
FUSE_READDIR28
FUSE_RELEASEDIR29
FUSE_FSYNCDIR30
FUSE_GETLK31v7.7
FUSE_SETLK32v7.7
FUSE_SETLKW33v7.7
FUSE_ACCESS34v7.3
FUSE_CREATE35v7.3
FUSE_INTERRUPT36v7.7
FUSE_BMAP37v7.8
FUSE_DESTROY38v7.8
FUSE_IOCTL39v7.11
FUSE_POLL40v7.11
FUSE_NOTIFY_REPLY41v7.15
FUSE_BATCH_FORGET42v7.16
FUSE_FALLOCATE42v7.19
FUSE_READDIRPLUS44v7.21
FUSE_RENAME245v7.23
FUSE_LSEEK46v7.24
FUSE_SETVOLNAME61v7.8-darwin3macOS only
FUSE_GETXTIMES62v7.8-darwin3macOS only
FUSE_EXCHANGE63v7.8-darwin3macOS only
CUSE_INIT4096v7.12

FUSE_ACCESS

If the default_permissions mount option is unset, the kernel will delegate permission checks to the FUSE server.

fuse_access_request
offset+0+1+2+3+4+5+6+7
0x00fuse_request_header
0x28mode

The mode is a bitmask of requested operations, matching the semantics of the POSIX access() syscall.

PermissionMode bit
execute0x1
write0x2
read0x4

The response body is empty, but the return value is significant:

Other return codes are OS-dependent.

FUSE_BATCH_FORGET

TODO

fuse_batch_forget_in
byte→
↓word
01234567
0count
fuse_forget_one[count]

fuse_forget_one
byte→
↓word
01234567
0nodeid
1nlookup

FUSE_BMAP

TODO

FUSE_CREATE

TODO

FUSE_DESTROY

Sent just before the kernel unmounts the filesystem. Might be received by the server after the kernel has terminated the session.

No request or response.

FUSE_EXCHANGE

TODO

https://www.unix.com/man-page/osx/2/exchangedata/

FUSE_FALLOCATE

TODO

FUSE_FLUSH

TODO

FUSE_FORGET

Reduces the reference count of a lookup’d inode.

fuse_forget_in
byte→
↓word
01234567
0nlookup

FUSE_FSYNC

TODO

FUSE_FSYNCDIR

TODO

FUSE_GETATTR

TODO

FUSE_GETLK

TODO

FUSE_GETXATTR

FUSE_GETXATTR
byte→
↓word
01234567
0size

FUSE_GETXATTR (v7.8-darwin3)
byte→
↓word
01234567
0size
1position

FUSE_GETXATTR
byte→
↓word
01234567
0size

FUSE_GETXTIMES

TODO

FUSE_INIT

FUSE_INIT (v7.2)
byte→
↓word
01234567
fuse_request_header
5majorminor

FUSE_INIT (v7.6)
byte→
↓word
01234567
FUSE_INIT (v7.2)
6max_readaheadflags

Negotiated features (“flags”):

featurebitmaskversionnotes
FUSE_ASYNC_READ0x1v7.6
FUSE_POSIX_LOCKS0x2v7.7
FUSE_FILE_OPS0x4v7.9
FUSE_ATOMIC_O_TRUNC0x8v7.9
FUSE_EXPORT_SUPPORT0x10v7.10
FUSE_BIG_WRITES0x20v7.10
FUSE_DONT_MASK0x40v7.12
FUSE_SPLICE_WRITE0x80v7.20
FUSE_SPLICE_MOVE0x100v7.20
FUSE_SPLICE_READ0x200v7.20
FUSE_FLOCK_LOCKS0x400v7.17
FUSE_HAS_IOCTL_DIR0x800v7.20
FUSE_AUTO_INVAL_DATA0x1000v7.20
FUSE_DO_READDIRPLUS0x2000v7.21
FUSE_READDIRPLUS_AUTO0x4000v7.21
FUSE_ASYNC_DIO0x8000v7.22
FUSE_WRITEBACK_CACHE0x10000v7.23
FUSE_NO_OPEN_SUPPORT0x20000v7.24
FUSE_PARALLEL_DIROPS0x40000v7.25
FUSE_HANDLE_KILLPRIV0x80000v7.26
FUSE_POSIX_ACL0x100000v7.26
FUSE_ABORT_ERROR0x200000v7.27
FUSE_CASE_INSENSITIVE0x20000000v7.8-darwin3macOS only
FUSE_VOL_RENAME0x40000000v7.8-darwin3macOS only
FUSE_XTIMES0x80000000v7.8-darwin3macOS only

fuse_init_out (v7.2)
byte→
↓word
01234567
0majorminor

fuse_init_out (v7.6)
byte→
↓word
01234567
fuse_init_out (v7.2)
1max_readaheadflags
2max_write

fuse_init_out (v7.13)
byte→
↓word
01234567
fuse_init_out (v7.2)
1max_readaheadflags
2max_backgroundcongestion_thresholdmax_write

fuse_init_out (v7.23)
byte→
↓word
01234567
fuse_init_out (v7.13)
3time_gran
4
5
6
7

FUSE_INTERRUPT

TODO

FUSE_IOCTL

TODO

TODO

FUSE_LISTXATTR

TODO

FUSE_LOOKUP

The request is a NUL-terminated bytestring. Incoming name length is constrained to some maximum length by the kernel:

Response is a fuse_entry_out.

Notes:

FUSE_LSEEK

TODO

FUSE_MKDIR

TODO

FUSE_MKNOD

TODO

FUSE_NOTIFY_REPLY

TODO

FUSE_OPEN

TODO

FUSE_OPENDIR

TODO

FUSE_POLL

TODO

FUSE_READ

TODO

FUSE_READDIR

TODO

FUSE_READDIRPLUS

TODO

https://tools.ietf.org/html/rfc1813#section-3.3.17

3.3.17 Procedure 17: READDIRPLUS - Extended read from directory

TODO

FUSE_RELEASE

FUSE_RELEASE (v7.2)
byte→
↓word
01234567
fuse_request_header
5fh
6flags

FUSE_RELEASE (v7.8)
byte→
↓word
01234567
fuse_request_header
5fh
6flagsrelease_flags
7lock_owner

TODO

FUSE_RELEASEDIR

TODO

FUSE_REMOVEXATTR

TODO

FUSE_RENAME

fuse_rename_request
offset+0+1+2+3+4+5+6+7
0x00fuse_request_header
0x28newdir
old_name
new_name

FUSE_RENAME2

fuse_rename2_request
offset+0+1+2+3+4+5+6+7
0x00fuse_request_header
0x28newdir
0x30flagsnewdir
old_name
new_name

TODO

https://lwn.net/Articles/606237/

https://www.spinics.net/lists/linux-fsdevel/msg72068.html

https://www.systutorials.com/docs/linux/man/2-renameat2/

FUSE_RMDIR

TODO

FUSE_SETATTR

TODO

FUSE_SETLK

TODO

https://sourceforge.net/p/fuse/mailman/message/35018434/

FUSE supports mandatory locking and BSD flock if that’s your thing < http://0pointer.de/blog/projects/locking.html >.

FUSE_SETLKW

TODO

FUSE_SETVOLNAME

TODO

FUSE_SETXATTR

FUSE_SETXATTR
byte→
↓word
01234567
fuse_request_header
5sizeflags

FUSE_SETXATTR (v7.8-darwin3)
byte→
↓word
01234567
fuse_request_header
5sizeflags
6position

FUSE_STATFS

TODO

TODO

TODO

FUSE_WRITE

TODO

CUSE_INIT

TODO

Appendix A: Structs

fuse_attr

fuse_attr (v7.2)
byte→
↓word
01234567
0ino
1size
2blocks
3atime
4mtime
5ctime
6atimensecmtimensec
7ctimensecmode
8nlinkuid
9gidrdev

fuse_attr (v7.9)
byte→
↓word
01234567
fuse_attr (v7.2)
10blksize

fuse_attr (v7.8-darwin3)
byte→
↓word
01234567
0ino
1size
2blocks
3atime
4mtime
5ctime
6crtime
7atimensecmtimensec
8ctimenseccrtimensec
9modenlink
10uidgid
11rdevflags

fuse_attr (v7.19-darwin1)
byte→
↓word
01234567
fuse_attr (v7.8-darwin3)
12blksize

fuse_entry_out

fuse_entry_out
byte→
↓word
01234567
0nodeid
1generation
2entry_valid
3attr_valid
4entry_valid_nsecattr_valid_nsec
attr (fuse_attr)


  1. The FreeBSD release notes describe their implementation as “state of the art”, but the ABI was ~7 years behind Linux at the time it was released.
  2. https://reviews.freebsd.org/D13528
  3. Amit worked at Google from 2006 to 20114, left to found a startup, got acquired by Facebook in 2013, and vanished from the Internet. I assume he’s either racing yachts around a private island or trapped in Mark Zuckerberg’s human zoo.
  4. http://www.kernelthread.com/resume/