Proof of Concept

Encrypted ZFS Replication Service

Cenk Yildiz // SB-SPC IT
cenk.yildiz@epfl.ch

ZFS: What and Why

ZFS: File system + Volume Manager https://openzfs.github.io/

• Open source
• Copy on write file system -> data integrity/snapshotting/replication
• Compression and encryption
• Expandable (RAID, mirror etc...), Portable, Highly tunable
• Great CLI

# Create pool from multiple disks
zpool create mypool raidz2 sda sdb sdc sdd sde sdf

# Create dataset with quota, mount on /data
zfs create -o mountpoint=/data -o quota=100G mypool/data

# Create home directory, assigning 10G quota to each user
zfs create \
    -o mountpount=/home \
    -o quota=1T \
    -o defaultuserquota=10G \
    mypool/home

# Create RAID Array
mdadm --create /dev/md5 --level=5 --raid-devices=6 \
    /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

# Make filesystem
mkfs.xfs /dev/md5

# Create folder
mkdir -p /data

# Add to fstab and mount
uuid=$(blkid -o value -s UUID /dev/md5)
echo "UUID=${uuid} /data xfs  defaults 0 0" >> /etc/fstab
mount -a

ZFS Concepts

man zpoolconcepts
man zfsconcepts
┌──────────────────────────────────────────────────────────┐
│                      POOL: mypool                      │
├──────────────────────────────────────────────────────────┤
│   ┌────────────────────────────────────────────────────┐ │
│   │   VDEV (mirror)                                  │ │
│   │  ┌──────────┐ ┌──────────┐                         │ │
│   │  │   sda  │ │   sdb  │                         │ │
│   │  └──────────┘ └──────────┘                         │ │
│   │  data mirrored across disks                        │ │
│   └────────────────────────────────────────────────────┘ │
│   ┌────────────────────────────────────────────────────┐ │
│   │   VDEV (mirror)                                  │ │
│   │  ┌──────────┐  ┌──────────┐                        │ │
│   │  │   sdc  │  │   sdd  │                        │ │
│   │  └──────────┘  └──────────┘                        │ │
│   │  data mirrored across disks                        │ │
│   └────────────────────────────────────────────────────┘ │
│   ┌────────────────────────────────────────────────────┐ │
│   │   datasets (zfs)                                 │ │
│   │   mypool/home     mypool/data     mypool/vms │ │
│   └────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘

ZFS Concepts relevant to this talk

• Pool: Logical collection of devices that provide space for datasets
• Dataset: Smallest unit that can be snapshotted

┌──────────────────────────────────────────────────────────┐
│                      POOL: mypool                      │
├──────────────────────────────────────────────────────────┤
│   ┌────────────────────────────────────────────────────┐ │
│   │   datasets (zfs)                                 │ │
│   │   mypool/home     mypool/data     mypool/vms │ │
│   └────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘

What runs on ZFS

• pfsense
• opnsense
• TrueNAS
• Proxmox
• Qnap
• LLNL HPC Cluster (https://computing.llnl.gov/projects/openzfs)

Encrypted ZFS Replication Service

Use cases

• Linux laptop with ZFS, encrypted dataset of private data

• DB (mysql, postgresql) with sensitive information

• VM disks on libvirt/proxmox

• Common point: Source that is ZFS, backup to a non-trusted target

ZFS Encryption

• Key: passphrase, raw file on disk(USB key), remote url
• Raw replication of encrypted dataset -> No key needed on target

ZFS Snapshot/Replication

• Good: Fast, incremental, block level, consistent state, most recovery is local
• Bad: per dataset

ZFS replication vs other common open source tools

            Consistency   Encryption  Compres/Dedup  Speed     Direction   Granularity
            -----------   ----------  -------------  -----     --------    -----------
rsync       No            No          No             Depends   push/pull   Single File
restic/borg No            Yes         Yes            Depends   push        Single File
zfs         Yes           Yes         Yes            Good      push/pull   ZFS Dataset

Big wins for:

• Small changes in large files
• Many small files

ZFS Permission Delegation

• Permissions per dataset can be delegated to non-root users
• Example: Allow user to snapshot/replicate

zfs allow yildiz create,destroy,hold,receive,rollback,send,snapshot mypool/mydataset

https://openzfs.github.io/openzfs-docs/Basic Concepts/Operations/Delegated Administration.html

ZFS Replication Service

• VM on EPFL XAAS, Ubuntu 26.04 -> zfsrepl-tests.epfl.ch
• EPFL LDAP group: zfs-replica-service
• Daily creation of dataset for group members with fixed quota
• Access delegation: User only have access to their dataset
• Datasets are not mounted
• Provisioned via ansible using epfl-si/ansible.suitcase
  https://gitlab.epfl.ch/yildiz/zfs-replica-service

Demo - 1

ZFSPERMS=compression,create,destroy,hold,receive,rollback,send,snapshot,mount,mountpoint
ZFSPOOL=mypool
ZFSDATASET=mypool/sensitive
USER=$(id -un)
WORKDIR=/"$ZFSDATASET"/$USER
ZFSREMOTEDATASET="replica/users/$USER/sensitive"
SYNCOID_OPTS="--no-sync-snap --sendoptions=raw  --no-privilege-elevation"

# Create a mock pool from a file
mkdir -p /tmp/zfstest && dd if=/dev/zero of=/tmp/zfstest/sda bs=1G count=4

sudo zpool create $ZFSPOOL /tmp/zfstest/sda
sudo zfs create -o encryption=on -o keyformat=passphrase "$ZFSDATASET"
sudo zfs allow "$USER" "$ZFSPERMS" "$ZFSPOOL"
sudo mkdir "$WORKDIR"
sudo chown "$USER" "$WORKDIR"

# Modify files...

# Show file contents and snapshots
watch -n 1 "printf '===== contents ===== \n\n' && tree -a /$ZFSDATASET && printf '\n===== snapshots =====\n' && zfs list -t snapshot $ZFSDATASET"

Demo - 2

• Modify/snapshot/replicate - easy way with Syncoid

# Create random file
dd if=/dev/urandom of="$WORKDIR"/rand1.dat bs=1M count=3 status=progress

zfs snapshot "$ZFSDATASET"@snap1

# During initial snapshot, dataset must NOT exist
syncoid $SYNCOID_OPTS "$ZFSDATASET" zfsrepl-tests.epfl.ch:"$ZFSREMOTEDATASET"

dd if=/dev/urandom of="$WORKDIR"/rand2.dat bs=128M count=2 status=progress

zfs snapshot "$ZFSDATASET"@snap2

syncoid $SYNCOID_OPTS "$ZFSDATASET" zfsrepl-tests.epfl.ch:"$ZFSREMOTEDATASET"

# Modify 256KB block in the middle
dd if=/dev/urandom of="$WORKDIR/rand2.dat" bs=1 seek=134217728 count=262144 conv=notrunc status=progress

zfs snapshot "$ZFSDATASET"@snap3
syncoid $SYNCOID_OPTS "$ZFSDATASET" zfsrepl-tests.epfl.ch:"$ZFSREMOTEDATASET"

zfs list -t snapshot "$ZFSDATASET"
ssh zfsrepl-tests.epfl.ch zfs list -t snapshot "$ZFSREMOTEDATASET"

• Hard way: Use zfs send/zfs receive

Demo - Recovery - Local

• Some normal late night sysadmining

  rm -f "$WORKDIR"/rand*
  

• 

• Partial rescue

# Make snapshot dirs visible, mounting them read-only
sudo zfs set snapdir=visible "$ZFSDATASET"

# Copy from last snapshot
cp /mypool/sensitive/.zfs/snapshot/snap3/yildiz/rand* "$WORKDIR"

# Make them hidden again
sudo zfs set snapdir=hidden "$ZFSDATASET"

• Full rollback

zfs rollback mypool/sensitive@snap3

Demo - Recovery - Remote

rm -f "$WORKDIR"/rand*
zfs destroy mypool/sensitive@snap2
zfs destroy mypool/sensitive@snap3

• Recovery

# OPTION 1: Just replicate from backup
syncoid $SYNCOID_OPTS yildiz@zfsrepl-tests.epfl.ch:"$ZFSREMOTEDATASET" "$ZFSDATASET"

# OPTION 2: Replicate from backup to a new dataset
syncoid $SYNCOID_OPTS yildiz@zfsrepl-tests.epfl.ch:"$ZFSREMOTEDATASET" "$ZFSPOOL"/sensitive-rescued
sudo zfs load-key "$ZFSPOOL"/sensitive-rescued
sudo zfs mount "$ZFSPOOL"/sensitive-rescued

Issues

• It's not a zero trust system for server side
• Limit access by resticted shells/ssh Force command? -> Too hacky
• zfs receive is not designed for safety - a malicious data with faulty headers can crash the remote system.
  ( But ZFS will refuse to replicate from tampered snapshots, thus safe for users)
• All commercial services that allow zfs replication have a VM and zfs pool per account

What is next ?

• ZFS users, feel free send me a mail, to be added to group (32GB quota per person)
• Non ZFS users, feel free to become ZFS user
• Do not put critical data!
• Do not sent non-encrypted datasets
• Support: Best effort

Resources:

• https://gitlab.epfl.ch/yildiz/zfs-replica-service -> This project, including these slides

• https://openzfs.github.io/

• https://2.5admins.com/ -> Podcast by Klara Systems Allan Jude, Jim Salter (sanoid/syncoid), Joe Ressington

• https://klarasystems.com/ -> Open source development/consulting company focused on ZFS/FreeBSD

• https://jrs-s.net/2016/05/16/zfs-practicing-failures/ A neat way to create block devices to emulate storage

BACKUP SLIDES

ZFS Features

• Open source
  • https://openzfs.github.io/openzfs-docs/License.html
• Copy on write file system
  • https://openzfs.github.io/openzfs-docs/Basic Concepts/Copy-on-write.html
• Data integrity/self healing
  • https://openzfs.github.io/openzfs-docs/Basic Concepts/Operations/Scrub and Resilver.html
• Compression (lz4, zstd)
  • https://openzfs.github.io/openzfs-docs/Basic Concepts/Data Storage/Compression.html
• Encryption
  • https://openzfs.github.io/openzfs-docs/Basic Concepts/Data Storage/Encryption.html
• Snapshotting -> Instant, cheap
  • https://openzfs.github.io/openzfs-docs/Basic Concepts/Datasets/Snapshots and Clones.html
• Incremental replication from snapshots
  • https://openzfs.github.io/openzfs-docs/Basic Concepts/Operations/Send and Receive.html

ZFS Snapshot/Replication - Many small files

• Real life Example:
  • Daily backup of ~1.6TB /home directory of ~300 users on linux
  • 1-10 GB data modified per day

            Backup duration   Excluded files       Recovery
            ---------------   ----------------     ----------
restic      2min30            .cache etc           "restic restore" over network

zfs         17sec             None                 local:                  cp /home/.zfs/snapshots/2026-9-15/yildiz/myfile ./
                                                   remote(single file):    scp $bkptarget:/bkppool/home/.zfs/snapshots/2026-9-15/yildiz/myfile ./
                                                   remote(full snapshot):  syncoid $bkptarget:bkppool/home@2026-9-15 mypool/home

• ZFS Compression -> 1.6TB data -> 1.06 TB space thanks to native compression

A note on space use

• Same dataset above (1.6 TB home directory) occupies 1.06T space
• 48 past snapshots (up to 3 months in time) occupy 0.11T

yildiz@spcnode01: ~/ zfs get available,used,referenced,logicalused,logicalreferenced,compressratio
 mypool/shares/home

NAME         PROPERTY           VALUE     SOURCE
mypool/home  available          9.60T     -       -> total space
mypool/home  used               1.17T     -       -> Space used after compression by data+snapshots
mypool/home  referenced         1.06T     -       -> Space used after compression by data (du -s)
mypool/home  logicalused        1.79T     -       -> Logical space used before compression data+snapshot
mypool/home  logicalreferenced  1.61T     -       -> Logical space used before compression data+snapshot (du -s --apparent-size)
mypool/home  compressratio      1.55x     -       -> Compression ratio

Sanoid/Syncoid

# Ubuntu/Debian
apt install -y sanoid

#RHEL/Fedora/other
url=https://github.com/jimsalterjrs/sanoid/archive/refs/tags/v2.3.0.tar.gz
curl -fsSL $url | tar -xz

# Automate with cron or systemd timers

/etc/sanoid/sanoid.conf

[rpool]
        use_template = production
        recursive = yes

[rpool/USERDATA/home-yildiz]                 
        use_template = daily

#############################
# templates below this line #
#############################

[template_production]
        hourly = 12
        daily = 30
        monthly = 3
        yearly = 2
        autosnap = yes
        autoprune = yes

[template_daily]
        daily = 30
        autosnap = yes
        autoprune = yes