File Systems
Sometimes we'll need a newer copy of the userspace tools for our file system of choice, so let's find out how to build these tools. Also provide some usage examples.
btrfs
btrfs-progs
Build btrfs-progs:
git clone https://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git btrfs-progs-git cd btrfs-progs-git sudo apt-get install e2fslibs-dev libblkid-dev zlib1g-dev libzstd-dev libudev-dev liblzo2-dev # Debian or Ubuntu sudo dnf install e2fsprogs-devel libacl-devel libattr-devel libblkid-devel lzo-devel uuid-devel # Fedora ./autogen.sh && ./configure --prefix=/opt/btrfs-progs/ --disable-documentation --disable-python make && sudo install
Usage
Grow
Resize[1], let's grow the file system first:
$ parted /dev/vdb (parted) print Number Start End Size Type File system Flags 1 1049kB 5369MB 5368MB primary btrfs (parted) resizepart 1 80% (parted) print Number Start End Size Type File system Flags 1 1049kB 8590MB 8589MB primary btrfs ^D $ df -h /mnt/ Filesystem Size Used Avail Use% Mounted on /dev/vdb1 5.0G 3.8M 4.5G 1% /mnt $ btrfs filesystem resize max /mnt/ Resize device id 1 (/dev/vdb1) from 5.00GiB to max $ df -h /mnt/ Filesystem Size Used Avail Use% Mounted on /dev/vdb1 8.0G 3.8M 7.5G 1% /mnt
Shrink
To shrink[1] the file system, it's the other way around:
$ df -h /mnt/ Filesystem Size Used Avail Use% Mounted on /dev/vdb1 1.1G 219M 713M 24% /mnt $ btrfs filesystem resize 700M /mnt/ # 100 MB safety margin, see below Resize device id 1 (/dev/vdb1) from 1.02GiB to 700.00MiB $ umount /mnt $ parted /dev/vdb (parted) print Number Start End Size Type File system Flags 1 1049kB 1100MB 1099MB primary btrfs (parted) resizepart 1 800M # 100 MB safety margin (parted) print Number Start End Size Type File system Flags 1 1049kB 800MB 799MB primary btrfs ^D $ mount -t btrfs /dev/vdb1 /mnt/ $ df -h /mnt/ Filesystem Size Used Avail Use% Mounted on /dev/vdb1 700M 219M 365M 38% /mnt
Compression
Compress files on an existing file system:
$ sudo compsize -x / Processed 122774 files, 63682 regular extents (70231 refs), 62514 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 84% 3.9G 4.6G 5.4G none 100% 3.6G 3.6G 3.6G zstd 34% 378M 1.0G 1.7G $ sudo btrfs filesystem defragment -r -czstd / # Add -v to see which files are being compressed $ sudo compsize -x / Processed 122774 files, 75777 regular extents (82367 refs), 70167 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 45% 2.1G 4.6G 5.4G none 100% 769M 769M 769M zstd 35% 1.3G 3.9G 4.6G
Swap
Do as described:
SWAP=/var/tmp/swap.img
truncate -s 0 ${SWAP} # Create, but don't allocat just yet.
chattr +C ${SWAP} # Disable copy-on-write updates
fallocate -l 2G ${SWAP} # Allocate space
chmod 0600 ${SWAP}
mkswap ${SWAP}
swapon ${SWAP}
Or, with suffiently recent enough userspace:
btrfs filesystem mkswapfile --size 2G ${SWAP}
swapon swapfile
Ext4
e2fsprogs
The e2fsprogs can be obtained via Git:
git clone git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git e2fsprogs-git cd e2fsprogs-git ./configure --prefix=/opt/e2fsprogs --enable-htree make && sudo make install
Usage
We can use tune2fs to modify some file system properties. Let's have the device checked every 20 mounts or 6 months, and turn it read-only in case of errors:
tune2fs -c 20 -i 6m -e remount-ro ${DEV}
Resizing the file system to grow to its maxium size can be done online, but we have to resize the partition first:
parted /dev/vda "print" parted /dev/vda "resizepart 1 100%"
Reboot (because it's the root partition here), then:
resize2fs -p /dev/vda1
e2image
We can use e2image to create an image of the file system:
e2image -rap ${DEV} disk.img # -r create a raw image
# -a include all data
# -p show progress while creating the image
Restore the image with:
e2image -rap disk.img ${DEV}
Or, to create a disk-to-disk copy:[2]
e2image -rap /dev/${SOURCE} /dev/${DEST}
dump
To clone a file system on file system level (as opposed to file or block level), we can use dump.
Now let's dump the currently mounted ext4 file system of to another disk, ${DEV}:
$ mount -t ext4 ${DEV} /mnt/disk/
$ cd /mnt/disk/
$ dump -0 -f - / | restore -rf -
DUMP: Date of this level 0 dump: Thu Aug 24 23:30:19 2017
DUMP: Dumping /dev/sda1 (/) to standard output
DUMP: Label: none
DUMP: Writing 10 Kilobyte records
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 2708627 blocks.
DUMP: Volume 1 started with block 1 at: Thu Aug 24 23:30:19 2017
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
restore: ./lost+found: File exists
./tmp/rstdir1503642619: (inode 805) not found on tape
./tmp/rstmode1503642619: (inode 951) not found on tape
DUMP: Volume 1 completed at: Thu Aug 24 23:31:02 2017
DUMP: Volume 1 2701050 blocks (2637.74MB)
DUMP: Volume 1 took 0:00:43
DUMP: Volume 1 transfer rate: 62815 kB/s
DUMP: 2701050 blocks (2637.74MB)
DUMP: finished in 43 seconds, throughput 62815 kBytes/sec
DUMP: Date of this level 0 dump: Thu Aug 24 23:30:19 2017
DUMP: Date this dump completed: Thu Aug 24 23:31:02 2017
DUMP: Average transfer rate: 62815 kB/s
DUMP: DUMP IS DONE
Since this was our root disk and we want to boot from the clone, we need to install the bootloader as well:
mount -t devtmpfs udev /mnt/disk/dev/ mount -t proc proc /mnt/disk/proc/ mount -t sysfs sysfs /mnt/disk/sys/ chroot /mnt/disk/ > grub-install --recheck /dev/sdc Installing for i386-pc platform. Installation finished. No error reported. > update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-4.9.0-0.bpo.2-amd64 Found initrd image: /boot/initrd.img-4.9.0-0.bpo.2-amd64 No volume groups found Found Debian GNU/Linux (8.9) on /dev/sdc done
As a final step, check the bootloader's configuration and /etc/fstab if the device names reflect the new disk, $DEV.
JFS
jfsutils
cvs -z3 -d:pserver:anonymous@jfs.cvs.sourceforge.net:/cvsroot/jfs co -P jfsutils
mv jfsutils{,-cvs}
cd jfsutils-cvs
sudo apt-get install uuid-dev # Debian or Ubuntu
sudo dnf install uuid-devel # Fedora
./autogen.sh --prefix=/opt/jfsutils
make && sudo make install
Usage
TBD
OverlayFS
Usage
Basic OverlayFS usage - in this case, we want to make use of a read-only NFS share:
mkdir -p /mnt/nfs /usr/local/src /mnt/overlay_{upper,work} mount -t nfs -o ro server:/data /mnt/nfs
mount -t overlay -o lowerdir=/mnt/nfs,upperdir=/mnt/overlay_upper,workdir=/mnt/overlay_work \ overlayfs /usr/local/src
With that, we can still write to /usr/local/src, although the read-only NFS share would not support this:
$ touch /usr/local/src/foo $ ls -l /mnt/nfs /usr/local/src /mnt/nfs: drwx--xr-x 1 dummy staff 50 Sep 2 2015 dir_name -rw-r--r-- 1 dummy staff 892 Feb 27 2015 file.exe /usr/local/src: drwx--xr-x 1 dummy staff 50 Sep 2 2015 dir_name -rw-r--r-- 1 dummy staff 892 Feb 27 2015 file.exe -rw------- 1 dummy staff 0 Aug 14 23:32 foo
ReiserFS
reiserfsprogs
Build reiserfsprogs:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeffm/reiserfsprogs.git reiserfsprogs-git cd reiserfsprogs-git sudo apt-get install libacl1-dev # Debian or Ubuntu sudo dnf install libacl-devel # Fedora libtoolize --copy --install --force && aclocal && autoheader && autoconf && automake --add-missing ./configure --prefix=/opt/reiserfsprogs make && make install
Usage
TBD
XFS
xfsprogs
Build xfsprogs:
git clone https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git xfsprogs-git cd xfsprogs-git sudo apt-get install automake gettext libattr1-dev libblkid-dev libicu-dev libtool m4 pkg-config uuid-dev # Debian or Ubuntu sudo dnf install automake gettext libattr-devel libblkid-devel libtool m4 pkgconf-pkg-config libuuid-devel # Fedora make configure && ./configure --prefix=/opt/xfsprogs make && sudo make install sudo chown -R root:root /opt/xfsprogs && sudo chmod -R a+rX /opt/xfsprogs
xfs_scrub
We might want to[3] try another xfsprogs branch supporting xfs_scrub[4], along with kernel support for the same feature.
cd /usr/local/src/linux-git/ git remote add djwong-xfs-linux https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git git fetch --tags djwong-xfs-linux git checkout -b local-xfs djwong-xfs-linux/master
Build and install the kernel. While this is going on, we can install the userspace part too:
cd /usr/local/src/xfsprogs-git git remote add djwong-xfsprogs-dev https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git git fetch --tags djwong-xfsprogs-dev git checkout -b local-djwong djwong-xfsprogs-dev/djwong-devel make configure && ./configure --prefix=/opt/xfsprogs-dev && make -j8 && sudo make install
xfstests
The xfstests is not exactly a filesystem checking tool, but a full blown file system test suite.
Install prerequisites:
sudo apt-get install acl attr automake bc dbench dump e2fsprogs fio gawk \
gcc git indent libacl1-dev libaio-dev libcap-dev libgdbm-dev libtool \
libtool-bin liburing-dev libuuid1 lvm2 make psmisc python3 quota sed \
uuid-dev uuid-runtime xfsprogs linux-headers-$(uname -r) sqlite3
sudo dnf install acl attr automake bc dbench dump e2fsprogs fio gawk gcc \
gdbm-devel git indent kernel-devel libacl-devel libaio-devel \
libcap-devel libtool liburing-devel libuuid-devel lvm2 make psmisc \
python3 quota sed sqlite xfsprogs
sudo zypper install acl attr autoconf automake btrfsprogs dbench dump e2fsprogs \
fio gawk gcc gdbm-devel git-core jfsutils libacl-devel libaio-devel \
libattr-devel libcap-progs libopenssl-devel libtool libuuid-devel lvm2 \
make quota reiserfs uuid-devel xfsdump xfsprogs xfsprogs-devel
sudo pacman -Sy --needed acl attr autoconf automake bc btrfs-progs compsize \
e2fsprogs fio gawk gcc gdbm git indent inetutils jfsutils libaio libcap \
libtool liburing linux-aarch64-headers linux-api-headers lvm2 m4 make \
psmisc python3 quota-tools reiserfsprogs sed sqlite3 util-linux-libs xfsprogs
Checkout the source:
git clone https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git xfstests-git
cd xfstests-git
make configure
make # No need to make install really, we can just execute
# the most current tests from our source directory.
id fsgqa || sudo useradd -m -U fsgqa # Add the test users & group and create
123456-fsgqa || sudo useradd 123456-fsgqa # their home directories
id fsgqa2 || sudo useradd fsgqa2
Create test devices, for example via RAM disks:
modprobe brd rd_nr=3 rd_size=$((10*1024*1024))
Or via LVM:
for d in test scratch s{0..4} log{,writes}; do echo lvcreate --size 10G --name vg0/${d}; done
lvcreate --size 500M --name vg0/logscratch # SCRATCH_LOGDEV can be way smaller than the actual test devices.
Set up some environment variables:
export TEST_DEV=/dev/mapper/vg0-lv0 TEST_DIR=/mnt/test \
SCRATCH_DEV=/dev/mapper/vg0-lv1 SCRATCH_MNT=/mnt/scratch FSTYP=xfs
We can use a SCRATCH device pool too, and unset SCRATCH_DEV then:
unset SCRATCH_DEV export SCRATCH_DEV_POOL="/dev/mapper/vg0-s0 /dev/mapper/vg0-s1 /dev/mapper/vg0-s2 /dev/mapper/vg0-s3 /dev/mapper/vg0-s4"
Maybe we want to test external LOG devices as well:
export TEST_LOGDEV=/dev/mapper/vg0-log LOGWRITES_DEV=/dev/mapper/vg0-logwrites SCRATCH_LOGDEV=/dev/mapper/vg0-logscratch USE_EXTERNAL=yes
Create test directories, and a filesystem on our TEST_DEV:
mkdir -p ${TEST_DIR} ${SCRATCH_MNT}
mkfs -t ${FSTYP} ${TEST_DEV}
Run (a single) test:
cd /opt/xfstests/xfstests/ ./check tests/generic/001
Follow along with:
tail -f results/generic/001.full
Usage
Create a filesystem with metadata checksums and a separate free inode btree index:
mkfs.xfs -m bigtime=1,crc=1,rmapbt=1,reflink=1 /dev/sdX
Check for fragmentation:[5][6]
$ ulimit -d unlimited # Otherwise xfs_db may fail[7]
$ xfs_db -c frag -r /dev/sde1
actual 922480, ideal 28640, fragmentation factor 96.90%
Online de-fragment with xfs_fsr, the XFS filesystem reorganizer:
xfs_fsr -v /mnt/disk
Check (and repair) an XFS filesystem:
xfs_repair /dev/sde1
If errors cannot be recovered at once, one might want to save a xfs_metadump copy before continuing, so that the issue can be analyzed later on.
xfs_quota
TBD