File Systems
Sometimes we'll need a newer copy of the userspace tools for our file system of choice, so let's find out how to build these tools. Also provide some usage examples.
btrfs
btrfs-progs
Build btrfs-progs:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git btrfs-progs-git cd btrfs-progs-git sudo apt-get install e2fslibs-dev libacl1-dev libattr1-dev libblkid-dev liblzo2-dev uuid-dev # Debian or Ubuntu sudo dnf install e2fsprogs-devel libacl-devel libattr-devel libblkid-devel lzo-devel uuid-devel # Fedora make && sudo make prefix=/opt/btrfs-progs/ install
Note: we might still need a patch for this to compile:
Usage
TBD
- BTRFS and Free Space - Emergency Response (2019-02-11)
- Fixing Btrfs Filesystem Full Problems (2014-05-04)
Ext4
e2fsprogs
The e2fsprogs can be obtained via Git:
git clone git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git e2fsprogs-git cd e2fsprogs-git ./configure --prefix=/opt/e2fsprogs --enable-htree make && sudo make install
Usage
We can use tune2fs
to modify some file system properties. Let's have the device checked every 20 mounts or 6 months, and turn it read-only in case of errors:
tune2fs -c 20 -i 6m -e remount-ro ${DEV}
e2image
We can use e2image to create an image of the file system:
e2image -rap ${DEV} disk.img # -r create a raw image # -a include all data # -p show progress while creating the image
Restore the image with:
e2image -rap disk.img ${DEV}
dump
To clone a file system on file system level (as opposed to file or block level), we can use dump.
Now let's dump
the currently mounted ext4
file system of to another disk, ${DEV}
:
$ mount -t ext4 ${DEV} /mnt/disk/
$ cd /mnt/disk/
$ dump -0 -f - / | restore -rf -
DUMP: Date of this level 0 dump: Thu Aug 24 23:30:19 2017
DUMP: Dumping /dev/sda1 (/) to standard output
DUMP: Label: none
DUMP: Writing 10 Kilobyte records
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 2708627 blocks.
DUMP: Volume 1 started with block 1 at: Thu Aug 24 23:30:19 2017
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
restore: ./lost+found: File exists
./tmp/rstdir1503642619: (inode 805) not found on tape
./tmp/rstmode1503642619: (inode 951) not found on tape
DUMP: Volume 1 completed at: Thu Aug 24 23:31:02 2017
DUMP: Volume 1 2701050 blocks (2637.74MB)
DUMP: Volume 1 took 0:00:43
DUMP: Volume 1 transfer rate: 62815 kB/s
DUMP: 2701050 blocks (2637.74MB)
DUMP: finished in 43 seconds, throughput 62815 kBytes/sec
DUMP: Date of this level 0 dump: Thu Aug 24 23:30:19 2017
DUMP: Date this dump completed: Thu Aug 24 23:31:02 2017
DUMP: Average transfer rate: 62815 kB/s
DUMP: DUMP IS DONE
Since this was our root
disk and we want to boot from the clone, we need to install the bootloader as well:
mount -t devtmpfs udev /mnt/disk/dev/ mount -t proc proc /mnt/disk/proc/ mount -t sysfs sysfs /mnt/disk/sys/ chroot /mnt/disk/ > grub-install --recheck /dev/sdc Installing for i386-pc platform. Installation finished. No error reported. > update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-4.9.0-0.bpo.2-amd64 Found initrd image: /boot/initrd.img-4.9.0-0.bpo.2-amd64 No volume groups found Found Debian GNU/Linux (8.9) on /dev/sdc done
As a final step, check the bootloader's configuration and /etc/fstab
if the device names reflect the new disk, $DEV
.
JFS
jfsutils
cvs -z3 -d:pserver:anonymous@jfs.cvs.sourceforge.net:/cvsroot/jfs co -P jfsutils mv jfsutils{,-cvs} cd jfsutils-cvs sudo apt-get install uuid-dev # Debian or Ubuntu sudo dnf install uuid-devel # Fedora ./autogen.sh --prefix=/opt/jfsutils make && sudo make install
Usage
TBD
OverlayFS
Usage
Basic OverlayFS usage - in this case, we want to make use of a read-only NFS share:
mkdir -p /mnt/nfs /usr/local/src /mnt/overlay_{upper,work} mount -t nfs -o ro server:/data /mnt/nfs
mount -t overlay -o lowerdir=/mnt/nfs,upperdir=/mnt/overlay_upper,workdir=/mnt/overlay_work \ overlayfs /usr/local/src
With that, we can still write to /usr/local/src
, although the read-only NFS share would not support this:
$ touch /usr/local/src/foo $ ls -l /mnt/nfs /usr/local/src /mnt/nfs: drwx--xr-x 1 dummy staff 50 Sep 2 2015 dir_name -rw-r--r-- 1 dummy staff 892 Feb 27 2015 file.exe /usr/local/src: drwx--xr-x 1 dummy staff 50 Sep 2 2015 dir_name -rw-r--r-- 1 dummy staff 892 Feb 27 2015 file.exe -rw------- 1 dummy staff 0 Aug 14 23:32 foo
ReiserFS
reiserfsprogs
Build reiserfsprogs:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeffm/reiserfsprogs.git reiserfsprogs-git cd reiserfsprogs-git sudo apt-get install libacl1-dev # Debian or Ubuntu sudo dnf install libacl-devel # Fedora libtoolize --copy --install --force && aclocal && autoheader && autoconf && automake --add-missing ./configure --prefix=/opt/reiserfsprogs make && make install
Usage
TBD
XFS
xfsprogs
Build xfsprogs:
git clone https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git xfsprogs-git cd xfsprogs-git sudo apt-get install automake gettext libattr1-dev libblkid-dev libicu-dev libtool m4 pkg-config uuid-dev # Debian or Ubuntu sudo dnf install automake gettext libattr-devel libblkid-devel libtool m4 pkgconf-pkg-config libuuid-devel # Fedora make configure && ./configure --prefix=/opt/xfsprogs make && sudo make install sudo chown -R root:root /opt/xfsprogs && sudo chmod -R a+rX /opt/xfsprogs
xfs_scrub
We might want to[1] try another xfsprogs
branch supporting xfs_scrub
[2], along with kernel support for the same feature.
cd /usr/local/src/linux-git/ git remote add djwong-xfs-linux https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git git fetch --tags djwong-xfs-linux git checkout -b local-xfs djwong-xfs-linux/master
Build and install the kernel. While this is going on, we can install the userspace part too:
cd /usr/local/src/xfsprogs-git git remote add djwong-xfsprogs-dev https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git git fetch --tags djwong-xfsprogs-dev git checkout -b local-djwong djwong-xfsprogs-dev/djwong-devel make configure && ./configure --prefix=/opt/xfsprogs-dev && make -j8 && sudo make install
xfstests
The xfstests is not exactly a filesystem checking
tool, but a full blown file system test suite.
Install prerequisites:
sudo apt-get install acl attr autoconf automake bc btrfs-progs dbench dump e2fsprogs fio gawk gcc git jfsutils libacl1-dev libaio-dev libattr1-dev libcap2-bin libgdbm-dev libssl-dev libtool-bin lvm2 quota reiser4progs reiserfsprogs uuid-dev xfsdump xfslibs-dev xfsprogs
sudo dnf install acl attr autoconf automake bc btrfs-progs dbench dump e2fsprogs fio gawk gcc gdbm-devel git jfsutils libacl-devel libaio-devel libattr-devel libcap libtool libuuid-devel lvm2 openssl-devel quota reiserfs-utils uuid-devel xfsdump xfsprogs xfsprogs-devel
sudo zypper install acl attr autoconf automake btrfsprogs dbench dump e2fsprogs fio gawk gcc gdbm-devel git-core jfsutils libacl-devel libaio-devel libattr-devel libcap-progs libopenssl-devel libtool libuuid-devel lvm2 make quota reiserfs uuid-devel xfsdump xfsprogs xfsprogs-devel
Checkout the source:
git clone https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git xfstests-git cd xfstests-git make configure ./configure --prefix=/opt/xfstests && make sudo make install # Optional id fsgqa || sudo useradd -m -U fsgqa # Add the fsgqa test user & group and create # its home directory
Set up some environment variables:
export TEST_DEV=/dev/mapper/vg0-lv0 TEST_DIR=/mnt/test SCRATCH_DEV=/dev/mapper/vg0-lv1 SCRATCH_MNT=/mnt/scratch mkdir -p ${TEST_DIR} ${SCRATCH_MNT} mkfs.xfs ${TEST_DEV}
Run (a single) test:
cd /opt/xfstests/xfstests/ ./check tests/xfs/002
Follow along with:
tail -f /opt/xfstests/xfstests/results/xfs/002.full
Usage
Create a filesystem with metadata checksums and a separate free inode btree index:
mkfs.xfs -m crc=1,finobt=1 /dev/sdX
Check for fragmentation:[3][4]
$ ulimit -d unlimited # Otherwise xfs_db
may fail[5]
$ xfs_db -c frag -r /dev/sde1
actual 922480, ideal 28640, fragmentation factor 96.90%
Online de-fragment with xfs_fsr
, the XFS filesystem reorganizer:
xfs_fsr -v /mnt/disk
Check (and repair) an XFS filesystem:
xfs_repair /dev/sde1
If errors cannot be recovered at once, one might want to save a xfs_metadump
copy before continuing, so that the issue can be analyzed later on.
xfs_quota
TBD