{"id":859,"date":"2025-07-20T14:58:55","date_gmt":"2025-07-20T09:28:55","guid":{"rendered":"https:\/\/www.lguruprasad.in\/blog\/?p=859"},"modified":"2025-07-20T14:58:55","modified_gmt":"2025-07-20T09:28:55","slug":"my-zfs-snapshot-and-replication-setup-on-ubuntu-ft-sanoid-and-syncoid","status":"publish","type":"post","link":"https:\/\/www.lguruprasad.in\/blog\/2025\/07\/20\/my-zfs-snapshot-and-replication-setup-on-ubuntu-ft-sanoid-and-syncoid\/","title":{"rendered":"My ZFS snapshot and replication setup on Ubuntu ft. sanoid and syncoid"},"content":{"rendered":"\n<p>I have known about ZFS since 2009, when I was working for Sun Microsystems as a campus ambassador at my college. But it wasn&#8217;t until I started hearing <a href=\"https:\/\/mercenarysysadmin.com\" target=\"_blank\" rel=\"noreferrer noopener\">Jim Salter<\/a> (on the TechSNAP and 2.5 Admins podcasts) and <a href=\"http:\/\/www.allanjude.com\" data-type=\"link\" data-id=\"http:\/\/www.allanjude.com\" target=\"_blank\" rel=\"noreferrer noopener\">Allan Jude<\/a> (on the 2.5 Admins podcast) evangelize ZFS that I became interested in using it on my computers and servers. With Ubuntu shipping ZFS in the kernel for many years now, I had access to native ZFS!,<\/p>\n\n\n\n<p>Here is an overview of my setup running Ubuntu + ZFS before I explain and document some of the details.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>cube<\/code> &#8211; A headless server running Ubuntu 24.04 LTS (at the time of writing) with ZFS on root and a lot of ZFS storage powered by mirror <code>vdev<\/code>s. Has <code>sanoid<\/code> for automatic snapshots.<\/li>\n\n\n\n<li>Desktops and laptops in my home run (K)Ubuntu (24.04 or later; versions vary) with encrypted (ZFS native encryption) ZFS on root and <a href=\"https:\/\/docs.zfsbootmenu.org\/en\/v3.0.x\/guides\/ubuntu\/noble-uefi.html\">ZFSBootMenu<\/a>. These computers also use <code><a href=\"https:\/\/github.com\/jimsalterjrs\/sanoid\" data-type=\"link\" data-id=\"https:\/\/github.com\/jimsalterjrs\/sanoid\" target=\"_blank\" rel=\"noreferrer noopener\">sanoid<\/a><\/code> for automatic snapshots.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Sanoid configuration<\/h2>\n\n\n\n<p>On my personal computers, I use a minimal <code>sanoid<\/code> configuration that looks like<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>############\n# datasets #\n############\n\n&#91;zroot]\n        use_template = production\n        recursive = zfs\n\n\n#############\n# templates #\n#############\n\n&#91;template_production]\n        frequently = 0\n        hourly = 26\n        daily = 30\n        monthly = 3\n        yearly = 0\n        autosnap = yes\n        autoprune = yes\n\n&#91;template_ignore]\n        autoprune = no\n        autosnap = no\n        monitor = no<\/code><\/pre>\n\n\n\n<p>On servers, the <code>sanoid<\/code> configuration has some additional tweaks, like the following template to not snapshot replicated datasets.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;template_backup]\n        frequently = 0\n        hourly = 36\n        daily = 30\n        monthly = 3\n        yearly = 0\n        # don't take new snapshots - snapshots\n        # on backup datasets are replicated in\n        # from source, not generated locally\n        autosnap = no<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Pre-apt snapshots<\/h2>\n\n\n\n<p>While <code>sanoid<\/code> provides periodic ZFS snapshots, I also wanted to wrap <code>apt<\/code> transactions in ZFS snapshots for the ability to roll back any bad updates\/upgrades. For this, I used the following shell script, <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/usr\/bin\/env bash\n\nDATE=\"$(\/bin\/date +%F-%T)\"\nzfs snapshot -r zroot@snap_pre_apt_\"$DATE\"<\/code><\/pre>\n\n\n\n<p>with the following <code>apt<\/code> hook in <code>\/etc\/apt\/apt.conf.d\/90zfs-pre-apt-snapshot<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/\/ Takes a snapshot of the system before package changes.\nDPkg::Pre-Invoke {\"&#91; -x \/usr\/local\/sbin\/zfs-pre-apt-snapshot ] &amp;&amp; \/usr\/local\/sbin\/zfs-pre-apt-snapshot || true\";};<\/code><\/pre>\n\n\n\n<p>This handles taking snapshots before <code>apt<\/code> transactions but doesn&#8217;t prune the snapshots at all. For that, I used the <code>zfs-prune-snapshots<\/code> script (from <a href=\"https:\/\/github.com\/bahamas10\/zfs-prune-snapshots\">https:\/\/github.com\/bahamas10\/zfs-prune-snapshots<\/a>) in a wrapper cron shell (schedule varies per computer) script that looks like<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/bin\/sh\n\n\/usr\/local\/sbin\/zfs-prune-snapshots \\\n    -p 'snap_pre_apt_' \\\n    1w 2>&amp;1 | logger \\\n    -t cleanup-zfs-pre-apt-snapshots<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Snapshot replication<\/h2>\n\n\n\n<p>The <code>cube<\/code> server has sufficient disk space to provide a replication target for all my other personal computers using ZFS. It has a pool named <code>dpool<\/code>, which will be referenced in the details to follow.<\/p>\n\n\n\n<p>For automating snapshot replication, I chose to use <code>syncoid<\/code> from the same <code>sanoid<\/code> package. To avoid giving privileged access to the sending and the receiving user accounts, my setup closely follows the path in <a href=\"https:\/\/klarasystems.com\/articles\/improving-replication-security-with-openzfs-delegation\/\">https:\/\/klarasystems.com\/articles\/improving-replication-security-with-openzfs-delegation\/<\/a>.<\/p>\n\n\n\n<p>On my personal computer, I granted my unprivileged (but has <code>sudo<\/code> \ud83e\udd37\u200d\u2642\ufe0f) local user account the <code>hold<\/code> and <code>send<\/code> permissions on the root dataset, <code>zroot<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo zfs allow send-user hold,send zroot\n\nzfs allow zroot\n---- Permissions on zroot --------------------------------------------\nLocal+Descendent permissions:\n        user send-user hold,send<\/code><\/pre>\n\n\n\n<p>On the <code>cube<\/code> server, I created an unprivileged user (no <code>sudo<\/code> permissions here \ud83d\ude0c) and granted it the <code>create,mount,receive<\/code> permissions temporarily on the parent of the target dataset, <code>dpool<\/code>.<\/p>\n\n\n\n<p>Then I performed an initial full replication of a local snapshot by running the following commands as the unprivileged user.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>zfs send \\\n  zroot@snapshot-name | ssh \\\n  remote-user@cube \\\n  zfs receive -u \\\n  dpool\/local-hostname\n\nzfs send \\\n  zroot\/ROOT@snapshot-name | ssh \\\n  remote-user@cube \\\n  zfs receive -u \\\n  dpool\/local-hostname\/ROOT\n\nzfs send \\\n  zroot\/ROOT\/os-name@snapshot-name | ssh \\\n  remote-user@cube \\\n  zfs receive -u \\\n    dpool\/local-hostname\/ROOT\/os-name\n\nzfs send \\\n  zroot\/home@snapshot-name | ssh \\\n  remote-user@cube \\\n  zfs receive -u \\\n  dpool\/local-hostname\/home<\/code><\/pre>\n\n\n\n<p>The <code>-u<\/code> flag in the <code>zfs receive<\/code> commands above will prevent it from trying to mount the remote dataset. The target remote dataset <strong>must<\/strong> not exist when running this initial full replication.<\/p>\n\n\n\n<p>As it is not a good practice to allow unprivileged users to mount filesystems, I disabled automatic mounting by running<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>zfs set mountpoint=none dpool\/local-hostname<\/code><\/pre>\n\n\n\n<p>as the <code>sudo<\/code> user on the target server.<\/p>\n\n\n\n<p>Then I narrowed down the permissions of the receiving user to only its own dataset by running<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>zfs unallow remote-user \\\n  create,mount,receive dpool\n\nzfs allow remote-user \\\n  create,mount,receive dpool\/local-hostname<\/code><\/pre>\n\n\n\n<p>on the target server.<\/p>\n\n\n\n<p>Next, I tried to test the snapshot replication by running <code>syncoid<\/code> manually like<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>syncoid -r \\\n  --no-privilege-elevation \\\n  --no-sync-snap \\\n  zroot \\\n  remote-user@cube:dpool\/local-hostname<\/code><\/pre>\n\n\n\n<p>and it replicated all the other snapshots all on the local datasets (we had only replicated one snapshot previously).<\/p>\n\n\n\n<p>The <code>sanoid<\/code> package in Debian and Ubuntu does not ship with a systemd timer for <code>syncoid<\/code>. So I created a user service and a timer that look like the following examples.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># ~\/.config\/systemd\/user\/syncoid.service\n&#91;Unit]\nDescription=Replicate sanoid snapshots\n\n&#91;Service]\nType=oneshot\nExecStart=\/usr\/sbin\/syncoid -r --no-privilege-elevation --no-sync-snap zroot remote-user@cube:dpool\/local-hostname<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code># ~\/.config\/systemd\/user\/syncoid.timer\n&#91;Unit]\nDescription=Run Syncoid to replicate ZFS snapshots to cube\n\n&#91;Timer]\nOnCalendar=*:0\/15\nPersistent=true\n\n&#91;Install]\nWantedBy=timers.target<\/code><\/pre>\n\n\n\n<p>Then I reloaded systemd, enabled and started the above timer to have everything working smoothly.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have known about ZFS since 2009, when I was working for Sun Microsystems as a campus ambassador at my college. But it wasn&#8217;t until I started hearing Jim Salter (on the TechSNAP and 2.5 Admins podcasts) and Allan Jude (on the 2.5 Admins podcast) evangelize ZFS that I became interested in using it on [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_s2mail":"","activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":4,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"federated","footnotes":""},"categories":[321,286,34,393],"tags":[395,398,396,394,397],"class_list":["post-859","post","type-post","status-publish","format-standard","hentry","category-self-hosting","category-technology","category-ubuntu","category-zfs","tag-sanoid","tag-snapshot-replication","tag-syncoid","tag-zfs","tag-zfsbootmenu"],"_links":{"self":[{"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/posts\/859","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/comments?post=859"}],"version-history":[{"count":36,"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/posts\/859\/revisions"}],"predecessor-version":[{"id":897,"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/posts\/859\/revisions\/897"}],"wp:attachment":[{"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/media?parent=859"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/categories?post=859"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lguruprasad.in\/blog\/wp-json\/wp\/v2\/tags?post=859"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}