Deep Dive into Distributed File System Permission Management: Linux Security Integration

In multi-user environments with high-security requirements, robust permission controls are fundamental for resource isolation. Linux's file permission model provides a flexible access control mechanism, ensuring system security through user/group permission settings. For distributed file systems supporting Linux, compliance with this model is critical for consistent security.

This article explores key Linux permission mechanisms and their implementation in a FUSE-based distributed file system.

Access Control Models: DAC and MAC

Access control is a critical component of system security, managing resource access such as files and network services. Common Linux models include:

• Discretionary access control (DAC): Decentralized, user-managed permissions
• Mandatory access control (MAC): Centralized permissions enforced by administrators

The table below shows their differences:

Common Unix permissions and POSIX ACLs implement DAC, enabling resource owners to freely manage access rights for files and directories. In contrast, SELinux and AppArmor enforce MAC through system-defined security policies, thereby providing more stringent security restrictions.

In the following sections, we’ll elaborate on the technical details of these mechanisms and their implementation in our file system.

Unix Permissions

Unix permissions are an access control mechanism inherited from Unix systems, used to manage access rights for files and directories. This mechanism can be summarized as follows:

Subjects (actors) perform actions on objects (targets) based on defined rules.

• Subjects: Users or processes, categorized into three types (ugo)
  • u: Owner
  • g: Owner group
  • o: Other
• Objects: Files or directories
• Permission rules (rwx) with different meanings for files and directories:
  • r (read permission)
    • For files: Read file contents, extended attributes, and symbolic links
    • For directories: List directory contents
  • w (write permission)
    • For files: Modify file contents and extended attributes
    • For directories: Create, delete, or move files or directories
  • x (execute permission)
    • For files: Execute files
    • For directories: Access the directory and view file metadata

Special Permission Bits

Beyond the basic ugo + rwx permission model, Unix systems support additional special permission bits:

• Set User ID (SUID): If a file is assigned the SUID permission, when other users execute the file, their permissions are temporarily elevated to those of the file owner.
• Set Group ID (SGID): The SGID permission is similar to the SUID permission, but it affects the group permissions of the file instead of the owner permissions. If a file is assigned the SGID permission, when other users execute the file, their permissions are temporarily elevated to the permissions of the group to which the file belongs.
• Sticky Bit (SBIT): The SBIT permission is a special directory permission. When a directory is assigned this permission, users can only delete their own files, even if they have write and execute permissions on the directory. This permission is often used on temporary file directories (such as /tmp) to prevent users from deleting other users' files.

Our Permission Verification

As a FUSE-based userspace file system, JuiceFS enables the default_permissions option by default. This activates kernel-level permission checks. Before file operations reach userspace, the kernel validates requests against Unix permission modes in file attributes, and only approved operations are forwarded to the file system.

In addition, the file system implements userspace permission logic to handle special scenarios:

• SDK access: The file system provides multiple SDK access methods. These SDK requests do not go through the kernel, so the file system needs to implement permission verification by itself.
• The squash feature: When the squash feature is enabled, the file system maps certain users to specified users. In this case, it’s impossible to rely on kernel permission verification, so the system enables user-mode permission verification.

Our file system has two squash modes:

• Root squash
• All squash

Root Squash (Root Privilege Demotion)

This mode maps the local root user to other users, thereby enhancing system security. In this mode, even the root user in the operating system is subject to permission control when performing file operations and cannot modify other users' files at will.

# root in /tmp/jfs
$ ll /tmp/jfs/f1    
-rw------- 1 user2 user2 0  Feb  19 16:26 /tmp/jfs/f1
# Reading user2's file succeeds.
$ cat /tmp/jfs/f1
hello

After mapping:

# Mapping root user to regular user1 (uid=1001,gid=1001).
./juicefs mount sqlite3://test.db /tmp/jfs -o allow_other -root-squash=1001:1001
# Trying to read user2's file fails (permission denied).
$ cat /tmp/jfs/f1
cat: /tmp/jfs/f1: Permission denied

All Squash (All Users Mapping to a Specified Account)

In all squash mode, the system maps all users to a designated account (such as user1). This mode is typically used for unified permission management, where all operations are performed under the specified user's identity.

# Map all users to user1 (uid=1001,gid=1001).
$ ./juicefs mount sqlite3://test.db /tmp/jfs -o allow_other -all-squash=1001:1001
# User2 creates a file.
$ whoami
user2
$ touch f1
# User3 creates a file.
$ whoami
user3
$ touch f2
# User1 owns both files.
$ ls -l .
total 0
-rw-rw-r-- 1 user1 user1 0  Feb 19 14:10 f1
-rw-rw-r-- 1 user1 user1 0  Feb 19 14:11 f2

POSIX ACLs

While standard Unix permissions satisfy most access control needs, certain scenarios require finer-grained permission management. For example, when you need to open file permissions for a certain user, you can use ACLs for fine-grained control. POSIX ACLs provide this extended control mechanism. Although there is no unified standard, POSIX 1003.1e draft 17 is widely accepted. The implementation based on this draft was added to the 2.5.46 version of the Linux kernel in November 2002.

Traditional Unix permissions only classify subjects into three categories (owner, group, and other), while POSIX ACLs classify subjects more finely and can assign rwx permissions to any user or group. All user permission definitions can be abstracted into an entry definition in ACLs. Each entry contains the user ID and its permission definition. The structure is defined as follows:

type Permission struct {
    Owner Mode
    Group Mode
    Other Mode
}

type ACL struct {
    Owner Mode
    Group Mode
    Other Mode
    Mask Mode // Maximum allowed permissions
    NamedUsers  Entry // User-specific permissions
    NamedGroups Entry // Group-specific permissions
}

type Entry struct {
    Uid uint32
    Perm Mode
}

POSIX ACLs add permissions for NamedUsers and NamedGroup. From the definition point of view, we can classify the newly added entries (including “mask”) of POSIX ACLs into the group class, and extend the definition of the group class from the permissions of the original group to the upper limit of permissions of all entries in the group class.

When no additional named users and named groups are configured, it’s called minimal ACLs, which is equivalent to Unix permissions.

• Minimal ACLs: Equivalent to ordinary Unix permissions
• Extended ACLs: Any named user or named group entries are configured

The following operation shows the relationship between POSIX ACLs and Unix permissions.

# Check permissions.
$ ls -l
drwxr-xr-x 2 root root 4.0K  Apr 12 09:32 d1

# Set the group1 permissions of the d1 directory through setfacl.
$ setfacl -m g:group1:rwx d1

# Check permissions again.
$ ls -l
drwxrwxr-x+ 2 root root 4.0K  Apr 12 09:32 d1

After setting the ACL, the group class of the file changed, adding the w permission. This is because the ACL extends the group class (refer to the grouping in the first table).

In Unix permissions, the group class only represents the permission of the owner group, while in POSIX ACLs, it serves as the permission ceiling for all entries within the group class. Therefore, the permission bits of the group class will also change with the entry changes.

The New ACL Entry: Mask

As mentioned earlier, the POSIX ACL introduces a “mask” entry to manage dynamic permission changes within the group class. Once the ACL is configured, the group class of permission mode displays “mask.” When configuring the ACL using setfacl, the system automatically computes the “mask” by default.

ACL Types

ACLs are categorized into two types:

• Access ACLs: They’re used for permission definition and checks, applicable to files and directories, as defined in the section above.
• Default ACLs: Their structure is the same as access ACLs, but it’s only applied to directories. Access ACLs of a subfile or subdirectory inherit from the default ACLs of the parent directory. (The inherited permissions are combined with the mode parameter of the system call that creates the file object to generate the final access ACL, where owner/mask/other permissions are intersected with the mode bits.)

# Set the access ACL.
$ setfacl -m u:user1:rwx d1
# Set the default ACL. (with the -d flag)
setfacl -d -m u:user1:rwx d1
# View the ACL.
$ getfacl -c d1
# Access ACL entries (top 5 lines):
user::rwx
user:user1:rwx
group::rwx
mask::rwx
other::r-x
# Default ACL entries (bottom 5 lines):
default:user::rwx
default:user:user1:rwx
default:group::rwx
default:mask::rwx
default:other::r-x

Access ACL Permission Evaluation Rules

When a process requests access to a file object, permission validation follows these rules:

1. Matching entry selection: The system identifies the most applicable ACL entry by checking in this strict order:

1. Owner
2. Named users
3. Owning or named groups
4. Others

A process may belong to multiple groups. If there are multiple group class entries that match (the third item above), the entry that has the permission is selected. If there is a matching group class entry, but none of them meets the permission requirement, the request is denied.

2. Check whether the permission requirements of this entry meet the request.

Our ACL Implementation

Our file system optimizes ACL storage by decoupling ACLs from extended attributes (xattrs). Rather than storing a separate ACL copy for each file, the system stores ACL configurations in a dedicated metadata structure and references them via globally unique ACL IDs.

This approach reduces storage duplication and improves performance, especially when creating files in directories with default ACLs, reducing the frequency of storage ACL requests.

Capability: Granular Permission Control to Reduce System Security Risks

Traditional Unix permissions only distinguish between two types of processes: privileged (root) and non-privileged. Root processes possess all system privileges, allowing unrestricted access and modification to file objects. Checks like Unix permissions and ACLs are ineffective against root processes.

Starting with Linux kernel version 2.2, the capability mechanism was introduced. It divides privileges into distinct capabilities. This allows non-privileged processes to be granted specific privileges, thereby reducing system security risks.

The capability set of a binary file is stored in the extended attributes of the security namespace. You can use the getcap and setcap commands to view and set capabilities.

For our file system, enabling the capability support requires both extended attributes (the --enable-xattr mount option) and capability (the --enable-cap mount option). By default, capability is disabled because it’s rarely needed in most scenarios. Enabling it generates additional metadata requests for extended attributes, impacting performance.

# Mounting, requires enabling xattr and cap.
$ ./juicefs mount sqlite3://test.db /tmp/jfs --enable-cap --enable-xattr
# Setting capability: Allows a regular user executing the 'bin' binary to modify file ownership. 
$ sudo setcap 'cap_chown=+ep' bin
# Viewing capability.
$ getcap bin
bin cap_chown=ep

SELinux: A Kernel Module to Enhance Linux System Security

The previous discussion explained the principles and differences between DAC and MAC models. While DAC is flexible, it lacks strictness and poses certain security risks. In traditional Linux systems, the excessive privileges of the root account under DAC create potential security threats.

The MAC model addresses this issue by implementing a mandatory access control mechanism, where system administrators define policies that are strictly enforced. Security-enhanced Linux (SELinux) is a kernel module that enhances security by implementing MAC, thereby restricting the unlimited power of root.

Although both SELinux and AppArmor are Linux security modules implementing MAC, they’re different in usage and implementation. Different Linux distributions adopt different defaults: SELinux is primarily used in Red Hat Enterprise Linux (RHEL) and Fedora, while AppArmor is common in Ubuntu and SUSE. This section focuses on SELinux.

In a Linux system with SELinux enabled, permission checks first pass through DAC before being evaluated against SELinux policies. In SELinux, every object (such as a file) has a security context, stored in the file's extended attributes, defining its permissions (as specified by SELinux).

To use SELinux in our file system, both extended attributes (the --enable-xattr mount option) and SELinux support (the --enable-selinux mount option) must be enabled. By default, --enable-selinux is disabled.

Summary

This article explored Linux file permission models and examined how a distributed file system implements permission management based on them. As a FUSE-based userspace distributed file system, it not only supports traditional Unix permissions but also provides flexible access control through userspace permission validation, root squash, and all squash. These capabilities ensure stable operation in environments demanding high security and complex user management.