www.fabiankeil.de/gehacktes/zogftw/

zogftw: ZFS on geli for the win

1. Example output
   1. Import all external pools
   2. Synchronize all snapshots stored on the pools
   3. List available subcommands and internal functions that can be leveraged
   4. Count snapshots on an imported pool by leveraging an internal function
   5. Create a new snapshot (and trigger custom hooks that provide additional information)
   6. Dealing with remote storage
   7. Diffing snapshots or files on snapshots more conveniently
   8. Automatically resume a prematurely aborted transfer
2. Example configuration
3. Download
4. Tests
5. Change log
6. License
7. Donations welcome
8. Dedication
9. zogftw presentation at OpenRheinRuhr

For a long time OpenZFS had no integrated full disk encryption support and while encryption support is now available it doesn't cover all the meta data. On ElectroBSD and FreeBSD this is mitigated by using geli.

It works great out of the box for pools that are always attached, but using external pools, which are exported most of the time, is inconvenient. Automatically importing available pools doesn't work because the ZFS metadata isn't available until the geli provider has been attached. It's especially inconvenient when multiple pools are supposed to be attached at the same time, specifying lots of passphrases and optionally keyfiles is rather boring.

Manually using the external pools for backups is time consuming even without encryption being involved. One has to provide the proper snapshot names as the zfs process that sends the snapshot doesn't see the receiving dataset and thus can't easily send whatever is missing.

zogftw makes importing, exporting and sending snapshots to such pools more convenient and has a bunch of additional ZFS-related features. It's extendable in shell (and thus pretty much any language you might care about). Additionally it can be used as function library for other shell scripts or interactive shells.

For documentation have a look at the zogftw man page, for examples see below.

Example output

These examples rely on the configuration below. The commands work out of the box, but without custom hooks the output and effect will be different.

Import all external pools

GnuPG is (optionally) used to get the passphrases, sudo is used to run commands that require privileges. The provider labels are used to figure out where the encrypted passphrase and the optional geli keyfile are.

fk@r500 ~ $zogftw import
2012-12-28 16:20:20 zogftw: No pool name specified. Trying all unattached labels: gfu gfu2 wde4 
2012-12-28 16:20:23 zogftw: No geli keyfile found at /home/fk/.config/zogftw/geli/keyfiles/gfu.key. Not using any.

You need a passphrase to unlock the secret key for
user: "Fabian Keil <fk@fabiankeil.de>"
4096-bit ELG-E key, ID 351A59E5, created 2006-08-19 (main key ID BF2EA563)

2012-12-28 16:20:30 zogftw: gfu attached
2012-12-28 16:20:31 zogftw: gfu imported
2012-12-28 16:20:31 zogftw: No geli keyfile found at /home/fk/.config/zogftw/geli/keyfiles/gfu2.key. Not using any.

You need a passphrase to unlock the secret key for
user: "Fabian Keil <fk@fabiankeil.de>"
4096-bit ELG-E key, ID 351A59E5, created 2006-08-19 (main key ID BF2EA563)

2012-12-28 16:20:35 zogftw: gfu2 attached
2012-12-28 16:20:38 zogftw: gfu2 imported
2012-12-28 16:20:38 zogftw: Using geli keyfile /home/fk/.config/zogftw/geli/keyfiles/wde4.key

You need a passphrase to unlock the secret key for
user: "Fabian Keil <fk@fabiankeil.de>"
4096-bit ELG-E key, ID 351A59E5, created 2006-08-19 (main key ID BF2EA563)

2012-12-28 16:20:41 zogftw: wde4 attached
2012-12-28 16:20:51 zogftw: wde4 imported

Synchronize all snapshots stored on the pools (and trigger a custom hook to garbage collect some snapshots)

Two pools in this example are small and only contain backups from a few datasets, one is larger and contains more datasets. zogftw has been extended with a custom hook to call zsd to destroy the oldest snapshot after successfully sending a new one to a dataset located on a small pool. On the large pool no snapshots are automatically destroyed.

fk@r500 ~ $zogftw sync
2012-12-28 16:20:57 zogftw: No destination pool specified. Synchronizing all pools prepared for receiving: gfu gfu2 wde4
2012-12-28 16:20:57 zogftw: gfu/backup/r500/tank/etc@2012-12-22_16:33 is the most recent snapshot
2012-12-28 16:20:57 zogftw: gfu/backup/r500/tank/home/fk@2012-12-24_15:27 is behind tank/home/fk@2012-12-28_13:20
receiving incremental stream of tank/home/fk@2012-12-28_13:20 into gfu/backup/r500/tank/home/fk@2012-12-28_13:20
in @ 14.7 MiB/s, out @ 5653 KiB/s, 10.6 MiB total, buffer   3% full
summary: 17.2 MiByte in 20.1 sec - average of  877 KiB/s
received 17.2MB stream in 7 seconds (2.45MB/sec)
2012-12-28 16:21:21 zogftw: zogftw giveth, and zogftw taketh away
There are 2 snapshots on gfu/backup/r500/tank/home/fk that could be destroyed. Will destroy 1 and keep 1.
Calling zfs destroy 'gfu/backup/r500/tank/home/fk@2012-12-20_19:49' ignoring the exit code
2012-12-28 16:21:21 zogftw: gfu/backup/r500/tank/home/fk/.liferea_1.8@2012-12-04_22:33 is the most recent snapshot
2012-12-28 16:21:21 zogftw: Destination dataset gfu/backup/r500/tank/home/fk/.mozilla doesn't seem to exist yet
receiving full stream of tank/home/fk/.mozilla@2012-12-28_11:06 into gfu/backup/r500/tank/home/fk/.mozilla@2012-12-28_11:06
in @  0.0 KiB/s, out @  0.0 KiB/s, 59.3 MiB total, buffer   3% full
summary: 63.7 MiByte in  8.4 sec - average of 7749 KiB/s
received 63.7MB stream in 18 seconds (3.54MB/sec)
2012-12-28 16:21:40 zogftw: gfu/backup/r500/tank/home/fk/git/privoxy@2012-12-16_12:22 is the most recent snapshot
2012-12-28 16:21:40 zogftw: gfu/backup/r500/tank/home/fk/papers@2012-12-20_13:33 is the most recent snapshot
2012-12-28 16:21:40 zogftw: gfu/backup/r500/tank/home/fk/web@2012-10-05_12:33 is the most recent snapshot
2012-12-28 16:21:40 zogftw: gfu/backup/r500/tank/home/fk/web/www.fabiankeil.de@2012-12-22_13:18 is behind tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35
receiving incremental stream of tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35 into gfu/backup/r500/tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35
summary: 2117 KiByte in  0.6 sec - average of 3645 KiB/s
received 2.07MB stream in 6 seconds (353KB/sec)
2012-12-28 16:21:47 zogftw: zogftw giveth, and zogftw taketh away
There are 2 snapshots on gfu/backup/r500/tank/home/fk/web/www.fabiankeil.de that could be destroyed. Will destroy 1 and keep 1.
Calling zfs destroy 'gfu/backup/r500/tank/home/fk/web/www.fabiankeil.de@2012-12-15_16:39' ignoring the exit code
2012-12-28 16:21:48 zogftw: gfu2/backup/r500/tank/etc@2012-12-06_21:06 is behind tank/etc@2012-12-22_16:33
receiving incremental stream of tank/etc@2012-12-22_16:33 into gfu2/backup/r500/tank/etc@2012-12-22_16:33
in @  128 KiB/s, out @  0.0 KiB/s,  0.0 KiB total, buffer   0% full
summary:  274 KiByte in  2.2 sec - average of  126 KiB/s
received 274KB stream in 3 seconds (91.4KB/sec)
2012-12-28 16:21:53 zogftw: zogftw giveth, and zogftw taketh away
There are 2 snapshots on gfu2/backup/r500/tank/etc that could be destroyed. Will destroy 1 and keep 1.
Calling zfs destroy 'gfu2/backup/r500/tank/etc@2012-11-09_19:50' ignoring the exit code
2012-12-28 16:21:55 zogftw: gfu2/backup/r500/tank/home/fk@2012-12-20_12:03 is behind tank/home/fk@2012-12-28_13:20
receiving incremental stream of tank/home/fk@2012-12-28_13:20 into gfu2/backup/r500/tank/home/fk@2012-12-28_13:20
in @  0.0 KiB/s, out @ 59.9 KiB/s, 33.6 MiB total, buffer   0% full
summary: 34.0 MiByte in 32.8 sec - average of 1060 KiB/s
received 34.0MB stream in 18 seconds (1.89MB/sec)
2012-12-28 16:22:30 zogftw: zogftw giveth, and zogftw taketh away
There are 26 snapshots on gfu2/backup/r500/tank/home/fk that could be destroyed. Will destroy 1 and keep 25.
Calling zfs destroy 'gfu2/backup/r500/tank/home/fk@2012-06-14_16:49' ignoring the exit code
2012-12-28 16:22:31 zogftw: gfu2/backup/r500/tank/home/fk/.e@2012-08-14_12:40 is the most recent snapshot
2012-12-28 16:22:31 zogftw: gfu2/backup/r500/tank/home/fk/.liferea_1.8@2012-12-04_22:33 is the most recent snapshot
2012-12-28 16:22:31 zogftw: Destination dataset gfu2/backup/r500/tank/home/fk/.mozilla doesn't seem to exist yet
receiving full stream of tank/home/fk/.mozilla@2012-12-28_11:06 into gfu2/backup/r500/tank/home/fk/.mozilla@2012-12-28_11:06
in @  0.0 KiB/s, out @ 2612 KiB/s, 60.8 MiB total, buffer   2% full
summary: 63.7 MiByte in 14.9 sec - average of 4393 KiB/s
received 63.7MB stream in 18 seconds (3.54MB/sec)
2012-12-28 16:22:51 zogftw: gfu2/backup/r500/tank/home/fk/git/privoxy@2012-12-16_12:22 is the most recent snapshot
2012-12-28 16:22:51 zogftw: gfu2/backup/r500/tank/home/fk/web@2012-10-05_12:33 is the most recent snapshot
2012-12-28 16:22:51 zogftw: gfu2/backup/r500/tank/home/fk/web/www.fabiankeil.de@2012-12-15_16:39 is behind tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35
receiving incremental stream of tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35 into gfu2/backup/r500/tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35
in @ 6185 KiB/s, out @  0.0 KiB/s,  0.0 KiB total, buffer   4% full
summary: 5584 KiByte in  1.1 sec - average of 4922 KiB/s
received 5.45MB stream in 7 seconds (798KB/sec)
2012-12-28 16:22:59 zogftw: zogftw giveth, and zogftw taketh away
There are 2 snapshots on gfu2/backup/r500/tank/home/fk/web/www.fabiankeil.de that could be destroyed. Will destroy 1 and keep 1.
Calling zfs destroy 'gfu2/backup/r500/tank/home/fk/web/www.fabiankeil.de@2012-12-03_15:33' ignoring the exit code
2012-12-28 16:23:00 zogftw: gfu2/backup/r500/tank/usr/local/etc@2012-12-20_12:21 is the most recent snapshot
2012-12-28 16:23:01 zogftw: wde4/backup/r500/tank/etc@2012-12-22_16:33 is the most recent snapshot
2012-12-28 16:23:01 zogftw: wde4/backup/r500/tank/home/fk@2012-12-28_13:20 is the most recent snapshot
2012-12-28 16:23:01 zogftw: wde4/backup/r500/tank/home/fk/.dvdcss@2012-11-03_19:29 is the most recent snapshot
2012-12-28 16:23:01 zogftw: wde4/backup/r500/tank/home/fk/.e@2012-08-14_12:40 is the most recent snapshot
2012-12-28 16:23:01 zogftw: wde4/backup/r500/tank/home/fk/.liferea_1.8@2012-12-04_22:33 is the most recent snapshot
2012-12-28 16:23:01 zogftw: wde4/backup/r500/tank/home/fk/.mozilla@2012-12-28_11:06 is the most recent snapshot
2012-12-28 16:23:02 zogftw: wde4/backup/r500/tank/home/fk/bilder@2012-12-22_13:18 is the most recent snapshot
2012-12-28 16:23:02 zogftw: wde4/backup/r500/tank/home/fk/git/curl/.git@2012-11-17_15:55 is the most recent snapshot
2012-12-28 16:23:02 zogftw: wde4/backup/r500/tank/home/fk/git/dvdbackup@2012-10-31_19:18 is the most recent snapshot
2012-12-28 16:23:02 zogftw: wde4/backup/r500/tank/home/fk/git/privoxy@2012-12-16_12:22 is the most recent snapshot
[...]
2012-12-28 16:23:04 zogftw: wde4/backup/r500/tank/var/db@2012-12-26_19:55 is the most recent snapshot

Export all external pools again

fk@r500 ~ $zogftw export
2012-12-28 16:35:22 zogftw: No zpool specified. Exporting all external ones: gfu gfu2 wde4 
2012-12-28 16:35:22 zogftw: Exporting gfu
2012-12-28 16:35:23 zogftw: Exporting gfu2
2012-12-28 16:35:25 zogftw: Exporting wde4

List available subcommands and internal functions that can be leveraged

fk@r500 ~ $zogftw help -v
zogftw (ZFS on geli for the win) 2013-10-06-4760e2c
usage: zogftw cmd command-to-execute
       zogftw config
       zogftw create zpool-name device-to-label
       zogftw export [-f] [zpool-to-export]
       zogftw help [-v]
       zogftw init
       zogftw import [zpool-to-import]
       zogftw snap[shot] [dataset-to-snapshot-specified-by-name-or-path]
       zogftw source
       zogftw sync [receiving-zpool]
       zogftw zcmd function-without-prefix-to-execute
       zogftw zpool subcommand [prefix_args] [postfix_args]
       zogftw diff file-or-directory
       zogftw list [-v]
       zogftw ggatec create
       zogftw ggatec destroy
       zogftw lookup grep-pattern
Internal functions and their parameters:
zogftw_asciidoc()
zogftw_clear_device() device_name
zogftw_cmd() command_to_execute
zogftw_config() 
zogftw_create() pool_name device_name 
zogftw_create_ecrypted_geli_passphrase_file() passphrase_file 
zogftw_create_missing_parent_datasets() dataset 
zogftw_dataset_does_exist() dataset
zogftw_dataset_has_been_specified_by_path() potential_path
zogftw_echo_list_members_with_attribute() attribute list 
zogftw_export() zpools_to_export 
zogftw_export_external_zpool() external_zpool 
zogftw_exporting_external_zpool_failed_hook()
zogftw_exporting_external_zpool_hook()
zogftw_foreach_zpool() zpool_subcommand prefix_args postfix_args 
zogftw_fyi() information 
zogftw_geli_attach() zpool_name 
zogftw_geli_initialize() provider 
zogftw_get_all_snapshots() dataset 
zogftw_get_config_variables()
zogftw_get_dataset_from_path() path
zogftw_get_exportable_zpools() 
zogftw_get_function_prototypes() 
zogftw_get_fyi_prefix()
zogftw_get_last_snapshot() dataset 
zogftw_get_last_snapshot_name() dataset 
zogftw_get_new_snapshot_name()
zogftw_get_passphrase() passphrase_file 
zogftw_get_sorted_src_dataset_information()
zogftw_get_src_dataset_information()
zogftw_get_sync_worthy_zpools() 
zogftw_get_unattached_labels() 
zogftw_get_zfs_property_value() value dataset
zogftw_get_zpool_size() dest_zpool
zogftw_import() zpool_name 
zogftw_import_got_labels_to_attach_hook()
zogftw_import_got_no_labels_to_attach_hook()
zogftw_import_zpool() zpool_name
zogftw_init() 
zogftw_initialize_unset_optional_config_variables() 
zogftw_label_device() device_label device_name
zogftw_load_config_file() 
zogftw_main() mode param_1 param_2
zogftw_main_hook()
zogftw_normalize_dataset() dataset
zogftw_pipe_buffer()
zogftw_prepare_pool_for_receiving() dest_zpool
zogftw_prepared_pool_for_receiving_hook()
zogftw_request_passphrase() promt passphrase_variable
zogftw_send_dataset_incrementally() src_dataset dest_dataset
zogftw_show_hook_capabilities() hook_name variables 
zogftw_snapshot() dataset
zogftw_snapshot_dataset() dataset 
zogftw_snapshot_not_yet_created_hook()
zogftw_snapshot_successfully_created_hook()
zogftw_snapshot_successfully_sent_hook()
zogftw_sudo() cmd_to_run_with_sudo
zogftw_sync() dest_zpool 
zogftw_sync_zpool() dest_zpool 
zogftw_sync_zpool_hook()
zogftw_transfer_is_impossible_no_snapshot_found_on_src_dataset_hook()
zogftw_transfer_is_necessary_hook()
zogftw_transfer_is_not_necessary_last_snapshots_are_equal_hook()
zogftw_transfer_last_snapshot() incremental_mode src_dataset dest_dataset 
zogftw_unregistered_mode_hook()
zogftw_usage() flag 
zogftw_usage_hook()
zogftw_user_has_zfs_permission() permission dataset 
zogftw_wtf() complaints
zogftw_zpool_does_exist() zpool_name
zogftw_zpool_import_successful_hook()
zogftw_zpool_is_puny() zpool_name

Count snapshots on an imported pool by leveraging an internal function

fk@r500 ~ $zogftw cmd zogftw_get_all_snapshots wde4/backup/r500/tank/home/fk | wc -l
    1119

If you expect to call lots of internal functions you can source zogftw first, in which case the shell's autocompletion should work as well:

fk@r500 ~ $. zogftw source
fk@r500 ~ $zogftw_get_all_snapshots wde4/backup/r500/tank/home/fk | wc -l
     1119

Create a new snapshot (and trigger custom hooks that provide additional information)

fk@r500 ~ $zogftw snap ~/web/www.fabiankeil.de/
2012-12-28 17:53:56 zogftw: The last snapshot tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35 references 93.3M
2012-12-28 17:53:57 zogftw: The new  snapshot tank/home/fk/web/www.fabiankeil.de@2012-12-28_17:53 references 93.4M
2012-12-28 17:53:57 zogftw: Changes between tank/home/fk/web/www.fabiankeil.de@2012-12-28_12:35 and tank/home/fk/web/www.fabiankeil.de@2012-12-28_17:53:
M	/home/fk/web/www.fabiankeil.de/
M	/home/fk/web/www.fabiankeil.de/RCS
+	/home/fk/web/www.fabiankeil.de/RCS/index.html,v
+	/home/fk/web/www.fabiankeil.de/index.html
+	/home/fk/web/www.fabiankeil.de/zogftw
+	/home/fk/web/www.fabiankeil.de/RCS/nutzloseinfos.html,v
+	/home/fk/web/www.fabiankeil.de/nutzloseinfos.html
+	/home/fk/web/www.fabiankeil.de/zogftw/RCS
+	/home/fk/web/www.fabiankeil.de/zogftw/RCS/index.html,v
+	/home/fk/web/www.fabiankeil.de/zogftw/index.html
+	/home/fk/web/www.fabiankeil.de/zogftw/zogftw.1.html

Dealing with remote storage

zogftw currently has no native cloud support, but you can use a hook to implement a subcommand to automatically attach remote storage using ggatec:

fk@r500 ~ $zogftw ggatec create
2013-08-29 10:43:22 zogftw: Successfully attached kendra.sshtunnel:/dev/ad4s3
2013-08-29 10:43:22 zogftw: Successfully attached kendra.sshtunnel:/dev/ad0

At which point it can be used like local storage:

fk@r500 ~ $zogftw import
2013-08-29 10:43:39 zogftw: No pool name specified. Trying all unattached labels: hd400ld kendra-ad4s3 
2013-08-29 10:43:39 zogftw: Using geli keyfile /home/fk/.config/zogftw/geli/keyfiles/hd400ld.key

You need a passphrase to unlock the secret key for
user: "Fabian Keil <fk@fabiankeil.de>"
4096-bit ELG-E key, ID 351A59E5, created 2006-08-19 (main key ID BF2EA563)

2013-08-29 10:43:43 zogftw: hd400ld attached
2013-08-29 10:43:46 zogftw: hd400ld imported
2013-08-29 10:43:46 zogftw: No geli keyfile found at /home/fk/.config/zogftw/geli/keyfiles/kendra-ad4s3.key. Not using any.

You need a passphrase to unlock the secret key for
user: "Fabian Keil <fk@fabiankeil.de>"
4096-bit ELG-E key, ID 351A59E5, created 2006-08-19 (main key ID BF2EA563)

2013-08-29 10:43:49 zogftw: kendra-ad4s3 attached
2013-08-29 10:43:51 zogftw: kendra-ad4s3 imported

While the server in the cloud gets to see the glabel and geli metadata, the block encryption happens on the system running zogftw. The server's security level is thus less important and you don't have to completely trust the hoster or the server admin.

After syncronizing the pools you can export them like local pools and detach the storage again:

fk@r500 ~ $zogftw export
2013-08-29 10:48:13 zogftw: No zpool specified. Exporting all external ones: hd400ld kendra-ad4s3 
2013-08-29 10:48:13 zogftw: Exporting hd400ld
2013-08-29 10:48:14 zogftw: Exporting kendra-ad4s3
fk@r500 ~ $zogftw ggatec destroy
2013-08-29 10:48:18 zogftw: Detaching ggate0
2013-08-29 10:48:19 zogftw: Detaching ggate1

Diffing snapshots or files on snapshots more conveniently

Hooks can also be used to implement a diff subcommand that either calls zfs diff or diff -u.

fk@r500 ~ $zogftw diff web/www.fabiankeil.de/
2013-09-10 12:46:23 zogftw: web/www.fabiankeil.de/ is located on tank/home/fk/web/www.fabiankeil.de whose last snapshot is 2013-09-04_21:37
M	/home/fk/web/www.fabiankeil.de/sourcecode/freebsd
M	/home/fk/web/www.fabiankeil.de/gpg-keys/fk-8BA2371C.asc
+	/home/fk/web/www.fabiankeil.de/sourcecode/freebsd/geli-Add-missing-line-breaks-in-geli-s-kern.geom.confxml.diff
fk@r500 ~ $zogftw diff web/www.fabiankeil.de/gpg-keys/fk-8BA2371C.asc
2013-09-10 12:46:33 zogftw: Diffing /home/fk/web/www.fabiankeil.de/gpg-keys/fk-8BA2371C.asc with its version on tank/home/fk/web/www.fabiankeil.de@2013-09-04_21:37
--- /home/fk/web/www.fabiankeil.de/.zfs/snapshot/2013-09-04_21:37/gpg-keys/fk-8BA2371C.asc	2013-07-17 15:59:06.016517305 +0200
+++ /home/fk/web/www.fabiankeil.de/gpg-keys/fk-8BA2371C.asc	2013-09-10 12:46:16.870299438 +0200
@@ -27,7 +27,272 @@
 gFRhLnHVxcX2kSe33qXIq8H7l+OTrxChP68BznWx38GIRgQQEQgABgUCUHQlWAAK
 CRBIxVIfvy6lY+oEAJ9f7h97Blp62mIc7TVjEaIHJC/LegCggU6KLOtNOKbuJ54R
 aCVqp6jo1T+IRgQQEQgABgUCUHQliwAKCRAFiohV/3dUnV83AJ9OOLIX1HFUY3zP
-SEbpT3vIFBWCoACgloLSZXYD+B21VScTJUAaCz2Zjaa5Ag0EUHQkVgEQALhGTcFa
+SEbpT3vIFBWCoACgloLSZXYD+B21VScTJUAaCz2ZjaaIRgQQEQIABgUCUh5oEwAK
[...]
+WZWJgF519F+qsnTr28vpPGLV3/TFHbte9hFlySurZCi5Ag0EUHQkVgEQALhGTcFa
 bUA1uI2Uk7bG/cYIHQ4yxAg6VUNPidAK5pNBbcq9V6VCV4GyRN0lUYrSnvxM9UvT
 /MepG2m/3NOOagYwfK7qeE6vgtDj6lXLWb7dQn6sFDwyUv6fj6EYoBvyNMNeMYPl
 YZ8K4C+0isbKL3h502qVzxZgqTAqq0RdzGOE4urIrYhQU0vWG3WXeha7768FKqVk
@@ -51,5 +316,5 @@
 Y9u5/lurO1rG4I9W5snDsa4keKzfTg/VXDRr/G7KRms8942YlAlB5VOIeQI7DUNQ
 RxTdQEZEy48bjY+PJogSPOGgQWq0bKBPVc1efEx+MnEnSLizVUKFvoAKokqHSgu/
 HSH3eROHn8U=
-=3ivf
+=6uhk
 -----END PGP PUBLIC KEY BLOCK-----

Automatically resume a prematurely aborted transfer

When configured to do so, zogftw automatically resumes previously interupted transfers:

fk@r500 ~ $zogftw sync
2015-11-21 13:46:32 zogftw: No destination pool specified. Synchronizing all pools prepared for receiving: audio
2015-11-21 13:46:34 zogftw: audio/backup/r500/tank/home/fk/audio/mp3s@2015-02-13_00:52 is the most recent snapshot
2015-11-21 13:46:40 zogftw: Resuming transfer to audio/backup/r500/tank/home/fk/audio/vorbis using token 1-dbe3066e5-d8-789[...]e4
receiving full stream of tank/home/fk/audio/vorbis@2015-11-09_12:16 into audio/backup/r500/tank/home/fk/audio/vorbis@2015-11-09_12:16
in @  124 KiB/s, out @  124 KiB/s,  104 MiB total, buffer   0% full

This is useful when sending backups through slow or unstable links.

For details see the changes in 2015-11-21-0a21288 and the zogftw man page.

Example configuration

This is an excerpt of the configuration I used for the examples above. The main configuration file:

fk@r500 ~ $zogftw cmd eval cat \$ZOGFTW_CONFIG_FILE
# If the zpool size in MB is below this limit, assume the zpool is
# space-constrained and should thus only get the incremental to the
# most recent snapshot as opposed to all the missing ones.
ZOGFTW_MAX_SPACE_CONSTRAINED_ZPOOL_SIZE=${ZOGFTW_MAX_SPACE_CONSTRAINED_ZPOOL_SIZE-20000}

# These are always syncronized
ZOGFTW_REQUIRED_SRC_DATASETS=${ZOGFTW_REQUIRED_SRC_DATASETS-"tank/home/fk tank/etc tank/home/fk/Mail"}

# These are only syncronized if a matching
# dataset exists on the destination zpool.
# The source datasets have to be always available.
ZOGFTW_OPTIONAL_SRC_DATASETS=$(grep -v "^#" "${ZOGFTW_CONFIG_DIRECTORY}/optional-src-datasets")

# These are like ZOGFTW_OPTIONAL_SRC_DATASETS
# except that the source datasets do not have
# to be always available
ZOGFTW_EXTERNAL_SRC_DATASETS=$(grep -v "^#" "${ZOGFTW_CONFIG_DIRECTORY}/external-src-datasets")

ZOGFTW_GELI_INIT_FLAGS="-l 256 -s 4k"

ZOGFTW_ZFS_INCREMENTAL_RECEIVE_FLAGS="-v -u -F -s"
ZOGFTW_ZFS_NON_INCREMENTAL_RECEIVE_FLAGS="-v -u -s"

# Datasets to snapshot if no dataset is specified on the command line
ZOGFTW_DEFAULT_SNAPSHOT_DATASETS="${ZOGFTW_DEFAULT_SNAPSHOT_DATASETS-tank/home/fk tank/home/fk/Mail tank/home/fk/Mail/Tor}"

# Do not zero out the provider before or after geli init
ZOGFTW_INITIALIZE_WITHOUT_DEVICE_ZEROING=${ZOGFTW_INITIALIZE_WITHOUT_DEVICE_ZEROING-1}

# Expect the snapshot names to reflect the chronological
# order of their creation, in which case they can be listed
# a lot faster. Even if there are only a few hundred snapshots
# this already makes a huge difference.
#
# Only set if it isn't already set to allow overwriting it
# through the environment for testing.
ZOGFTW_SORTING_SNAPSHOTS_BY_NAME_KEEPS_CHRONOLOGICAL_ORDER=${ZOGFTW_SORTING_SNAPSHOTS_BY_NAME_KEEPS_CHRONOLOGICAL_ORDER-1}

# Prefix messages coming from zogftw with a timestamp and the origin.
#
# This function doesn't end with _hook as its stdout output affects
# zogftw whereas the stdout output for hooks doesn't matter.
#
# zogftw functions that don't end with _hook should only be overwritten
# after understanding what their side effects are.
zogftw_get_fyi_prefix() {
    local timestamp=$(date "+%Y-%m-%d %H:%M:%S")
    echo "${timestamp} zogftw: "
}

# Gimme my hooks. Using a separate file for them is optional.
. "${ZOGFTW_CONFIG_DIRECTORY}/zogftw.hooks"

# Variables below this point are only relevant for the hooks.
# The prefix is optional.

ZOGFTW_HOOK_LOCATION_HIDDEN_ZPOOLS="tor1 tor2 tor3 tor4"

# Used by ggatec_create().
# Requires a ggatec version with ElectroBSD patches.
ZOGFTW_HOOK_GGATEC_SOCKS_ADDRESS="tor-jail"
ZOGFTW_HOOK_GGATEC_SOCKS_PORT="9049"

# XXX: Potentionally using a single logfile for multiple ggatec processes.
ZOGFTW_HOOK_GGATEC_CREATE_FLAGS="-l /var/log/ggatec-zogftw.log"

# Firefox has a dedicated user now, point out obsolete backups
ZOGFTW_HOOK_DATASETS_TO_POINT_OUT="tank/home/fk/.mozilla"

And the sourced file zogftw.hooks.

Download

Are you using a FreeBSD-based operating system and still reading? Download zogftw 2022-06-25-03982c7 (signature) now!

Note that zogftw is part of the FreeBSD ports collection. All the cool kids install it with: cd /usr/ports/sysutils/zogftw && sudo make install clean

Tests

To execute the tests, use the execute-tests target:

fk@r500 /usr/ports/sysutils/zogftw $make execute-tests
/usr/bin/env HOME=/usr/obj-ports/usr/ports/sysutils/zogftw/work kyua test --kyuafile /usr/obj-ports/usr/ports/sysutils/zogftw/work/zogftw-2015-11-21-0a21288/tests/Kyuafile
zogftw_test.sh:detect_environment_pollution  ->  passed  [0.050s]
zogftw_test.sh:echo_supports_n  ->  passed  [0.065s]
zogftw_test.sh:no_untested_zogftw_functions  ->  passed  [0.270s]
zogftw_test.sh:zogftw_auto_resume  ->  passed  [0.111s]
zogftw_test.sh:zogftw_change_space_escaping  ->  passed  [0.130s]
zogftw_test.sh:zogftw_clear_device  ->  passed  [0.065s]
zogftw_test.sh:zogftw_cmd  ->  passed  [0.068s]
zogftw_test.sh:zogftw_config  ->  passed  [0.049s]
zogftw_test.sh:zogftw_create_encrypted_geli_passphrase_file_gpg_failure  ->  passed  [0.072s]
zogftw_test.sh:zogftw_create_encrypted_geli_passphrase_file_invalid_arguments  ->  passed  [0.058s]
zogftw_test.sh:zogftw_create_encrypted_geli_passphrase_file_passphrase_mismatch  ->  passed  [0.053s]
zogftw_test.sh:zogftw_create_encrypted_geli_passphrase_file_success  ->  passed  [0.057s]
zogftw_test.sh:zogftw_create_failures  ->  passed  [0.061s]
zogftw_test.sh:zogftw_create_missing_parent_datasets  ->  passed  [0.120s]
zogftw_test.sh:zogftw_create_success  ->  passed  [0.122s]
zogftw_test.sh:zogftw_dataset_does_exist  ->  passed  [0.105s]
zogftw_test.sh:zogftw_dataset_has_been_specified_by_path  ->  passed  [0.173s]
zogftw_test.sh:zogftw_dataset_has_snapshots  ->  passed  [0.113s]
zogftw_test.sh:zogftw_echo_list_members_with_attribute  ->  passed  [0.078s]
zogftw_test.sh:zogftw_escape_spaces  ->  passed  [0.148s]
zogftw_test.sh:zogftw_export  ->  passed  [0.099s]
zogftw_test.sh:zogftw_export_external_zpool  ->  passed  [0.125s]
zogftw_test.sh:zogftw_foreach_zpool_failures  ->  passed  [0.128s]
zogftw_test.sh:zogftw_foreach_zpool_success  ->  passed  [0.138s]
zogftw_test.sh:zogftw_fyi  ->  passed  [0.057s]
zogftw_test.sh:zogftw_geli_attach_failure  ->  passed  [0.080s]
zogftw_test.sh:zogftw_geli_attach_success  ->  passed  [0.080s]
zogftw_test.sh:zogftw_geli_geom_exists  ->  passed  [0.077s]
zogftw_test.sh:zogftw_geli_initialize_failure_due_to_existing_metadata  ->  passed  [0.050s]
zogftw_test.sh:zogftw_geli_initialize_success_not_saving_metadata  ->  passed  [0.059s]
zogftw_test.sh:zogftw_geli_initialize_success_with_keyfile  ->  passed  [0.060s]
zogftw_test.sh:zogftw_geli_initialize_success_without_keyfile  ->  passed  [0.059s]
zogftw_test.sh:zogftw_geli_initialize_using_existing_gpg_passphrase_file  ->  passed  [0.060s]
zogftw_test.sh:zogftw_get_all_snapshots  ->  passed  [0.193s]
zogftw_test.sh:zogftw_get_asciidoc  ->  passed  [0.048s]
zogftw_test.sh:zogftw_get_config_variables  ->  passed  [0.046s]
zogftw_test.sh:zogftw_get_dataset_from_path  ->  passed  [0.095s]
zogftw_test.sh:zogftw_get_exportable_zpools  ->  passed  [0.143s]
zogftw_test.sh:zogftw_get_first_snapshot  ->  passed  [0.243s]
zogftw_test.sh:zogftw_get_function_prototypes  ->  passed  [0.055s]
zogftw_test.sh:zogftw_get_fyi_prefix  ->  passed  [0.046s]
zogftw_test.sh:zogftw_get_last_snapshot  ->  passed  [0.177s]
zogftw_test.sh:zogftw_get_last_snapshot_name  ->  passed  [0.181s]
zogftw_test.sh:zogftw_get_new_snapshot_name  ->  passed  [0.069s]
zogftw_test.sh:zogftw_get_passphrase  ->  passed  [1.440s]
zogftw_test.sh:zogftw_get_receive_resume_token  ->  passed  [0.148s]
zogftw_test.sh:zogftw_get_sorted_src_dataset_information  ->  passed  [0.179s]
zogftw_test.sh:zogftw_get_sorted_src_dataset_information_with_spaces  ->  passed  [0.048s]
zogftw_test.sh:zogftw_get_src_dataset_information  ->  passed  [0.118s]
zogftw_test.sh:zogftw_get_sync_worthy_zpools  ->  passed  [0.109s]
zogftw_test.sh:zogftw_get_unattached_labels  ->  passed  [0.155s]
zogftw_test.sh:zogftw_get_zfs_property_value  ->  passed  [0.106s]
zogftw_test.sh:zogftw_get_zpool_size  ->  passed  [0.141s]
zogftw_test.sh:zogftw_help  ->  passed  [0.090s]
zogftw_test.sh:zogftw_import_success  ->  passed  [0.121s]
zogftw_test.sh:zogftw_import_without_geli_on_label  ->  passed  [0.063s]
zogftw_test.sh:zogftw_import_without_labels_to_attach  ->  passed  [0.086s]
zogftw_test.sh:zogftw_import_without_zpool_on_geli  ->  passed  [0.070s]
zogftw_test.sh:zogftw_import_zpool  ->  passed  [0.131s]
zogftw_test.sh:zogftw_init  ->  passed  [0.088s]
zogftw_test.sh:zogftw_initialize_unset_optional_config_variables  ->  passed  [0.044s]
zogftw_test.sh:zogftw_invalid_subcommand  ->  passed  [0.050s]
zogftw_test.sh:zogftw_label_device  ->  passed  [0.092s]
zogftw_test.sh:zogftw_load_config_file  ->  passed  [0.044s]
zogftw_test.sh:zogftw_main  ->  passed  [0.064s]
zogftw_test.sh:zogftw_nop  ->  passed  [0.046s]
zogftw_test.sh:zogftw_normalize_dataset  ->  passed  [0.097s]
zogftw_test.sh:zogftw_pipe_buffer  ->  passed  [0.068s]
zogftw_test.sh:zogftw_prepare_pool_for_receiving  ->  passed  [0.097s]
zogftw_test.sh:zogftw_prime_sudo  ->  passed  [0.051s]
zogftw_test.sh:zogftw_receive_resume_token_belongs_to_dataset  ->  passed  [0.202s]
zogftw_test.sh:zogftw_replace_pattern  ->  passed  [0.133s]
zogftw_test.sh:zogftw_request_passphrase  ->  passed  [0.181s]
zogftw_test.sh:zogftw_resume_transfer  ->  passed  [0.107s]
zogftw_test.sh:zogftw_send_dataset_incrementally  ->  passed  [0.123s]
zogftw_test.sh:zogftw_show_hook_capabilities  ->  passed  [0.058s]
zogftw_test.sh:zogftw_snapshot  ->  passed  [0.500s]
zogftw_test.sh:zogftw_snapshot_creation_failures  ->  passed  [0.058s]
zogftw_test.sh:zogftw_snapshot_creation_using_defaults  ->  passed  [0.388s]
zogftw_test.sh:zogftw_snapshot_dataset  ->  passed  [0.180s]
zogftw_test.sh:zogftw_successful_snapshot_creation  ->  passed  [0.115s]
zogftw_test.sh:zogftw_successful_snapshot_creation_without_sudo  ->  passed  [0.072s]
zogftw_test.sh:zogftw_sudo  ->  passed  [0.067s]
zogftw_test.sh:zogftw_sync_creating_destination_dataset  ->  passed  [0.320s]
zogftw_test.sh:zogftw_sync_failures  ->  passed  [0.102s]
zogftw_test.sh:zogftw_sync_skip_required_dataset_creation  ->  passed  [0.246s]
zogftw_test.sh:zogftw_sync_with_existing_destination_dataset  ->  passed  [0.101s]
zogftw_test.sh:zogftw_sync_with_unsyncable_destination_dataset  ->  passed  [0.102s]
zogftw_test.sh:zogftw_sync_without_src_datasets  ->  passed  [0.079s]
zogftw_test.sh:zogftw_sync_zpool  ->  passed  [0.130s]
zogftw_test.sh:zogftw_transfer_first_snapshot  ->  passed  [0.117s]
zogftw_test.sh:zogftw_transfer_first_snapshot_with_spaces  ->  passed  [0.081s]
zogftw_test.sh:zogftw_transfer_last_snapshot  ->  passed  [0.179s]
zogftw_test.sh:zogftw_unattached_label_exists  ->  passed  [0.214s]
zogftw_test.sh:zogftw_unescape_spaces  ->  passed  [0.156s]
zogftw_test.sh:zogftw_usage  ->  passed  [0.072s]
zogftw_test.sh:zogftw_user_has_zfs_permission  ->  passed  [0.424s]
zogftw_test.sh:zogftw_wtf  ->  passed  [0.215s]
zogftw_test.sh:zogftw_zpool_does_exist  ->  passed  [0.079s]
zogftw_test.sh:zogftw_zpool_gets_new_datasets  ->  passed  [0.117s]
zogftw_test.sh:zogftw_zpool_is_puny  ->  passed  [0.117s]

Results file id is usr_obj-ports_usr_ports_sysutils_zogftw_work_zogftw-2015-11-21-0a21288_tests.20151121-122240-561123
Results saved to /usr/obj-ports/usr/ports/sysutils/zogftw/work/.kyua/store/results.usr_obj-ports_usr_ports_sysutils_zogftw_work_zogftw-2015-11-21-0a21288_tests.20151121-122240-561123.db

101/101 passed (0 failed)

Change log

2013-02-08-3b9e2c2

• Add and use zogftw_sudo() and introduce ZOGFTW_SUDO_FLAGS which default to '-c -'. This is a workaround for bin/174831 (geli(8) segfaults with the new locked memory limit default).
• Make it easier to replace sudo with something else. This is a side effect of the previous change, it only makes sense if you aren't using sudo already and I didn't bother to test this.
• zogftw cmd zogftw_usage -v works as expected as long as zogftw can be reached through the PATH.
• Added the zcmd subcommand.
• Be more explicit about zogftw_init() not second-guessing locations and permissions.

2013-03-31-80ef6e9

• Added subcommand zpool to execute zpool commands on all managed pools.
• Added zogftw_usage_hook() which can be used to extend the help output.
• Cosmetical code changes.

2013-10-06-4760e2c

• Extended the export subcommand to optionally export pools forcefully.
• Added zogftw_exporting_external_zpool_hook() which can be used to keep a dataset listing.
• Cosmetical code changes.

2014-09-20-28bc862

• Added ATF tests. Every function that isn't a hook is being tested. As a result, various error conditions are now handled more consistently, error messages have been clarified and code could be refactored without modifying the tested behaviour.
• When sending a source dataset to a destination dataset that doesn't exist yet, the first snapshot is sent followed by the incremental snapshot(s) to the last one. Previously only the last one was being sent.
• Let zogftw_sync() abort syncing right away if zogftw_sync_zpool() fails. Makes it more likely that the user notices the problem.
• Support positional arguments and datasets names with spaces more consistently.
• Support escaped spaces ("\ ") in config variables containing datasets.
• Don't allow to create GEOM labels with spaces as FreeBSD doesn't properly support them. Previously this resulted in a unclear error message as glabel reported success while zpool failed to access the label.
• Do not sync to pools whose names contain spaces. GEOM labels with spaces currently aren't supported on FreeBSD anyway, but a user with poor taste could create such pools manually.
• Unbreak zogftw cmd zogftw_main. It may not be a particular useful command, but it shouldn't fail with: "[: !=: unexpected operator" either.
• Rename zogftw_asciidoc() to zogftw_get_asciidoc() to be more consistent.
• Spell function name zogftw_create_encrypted_geli_passphrase_file() properly. Previously an 'n' was missing.
• Let zogftw_user_has_zfs_permission() check more carefully. Previously the permissions from user foobar were also considered when checking permissions for user foo.
• Add config variable ZOGFTW_GPG_DECRYPT_FLAGS.
• Add config variable ZOGFTW_BACKUP_DATASET_COMPRESSION and change the default from 'on' to 'lz4'.
• The zogftw_snapshot_successfully_created_hook() now gets the snapshot that has been created through the variable $snapshot which was previously incorrectly named $snapshot_name. You may have to adjust your hooks accordingly.

2014-12-12-71f792b

• Use zpool get's "-p" option instead of reimplementing it (poorly). zogftw_get_zpool_size() is now deprecated.
• Incovenient sudo passphrase promts are now suppressed by using sudo's -v option before executing sudo pipes. Previously "sudo false" was executed. It works as well, but wasn't being used consistently and caused log messages the user might consider surprising.
• Missing config files are no longer fatal when using the subcommand "help" but result in a hopefully helpful suggestion instead.
• The test that checks for untested zogftw functions is now more reliably, previously two untested functions were missed due to partial matches. The missing tests have been added as well.
• The man page now has a "SEE ALSO" section.
• The test for zogftw_get_passphrase() now works with more recent gpg versions and requires gpg2.
• Minor code and grammar cosmetic.
• Use zpool get's "-p" option instead of reimplementing it (poorly). zogftw_get_zpool_size() is now deprecated.

2015-11-21-0a21288

• Automatically resume aborted zfs receive operations. For this to work, the system has to suport it (FreeBSD r289362 or later) and the variables ZOGFTW_ZFS_INCREMENTAL_RECEIVE_FLAGS and ZOGFTW_ZFS_NON_INCREMENTAL_RECEIVE_FLAGS have to include the -s flag so the already received data isn't discarded when a receive operation fails. Example output.
• Don't create required datasets on the receiving pool if the "de.fabiankeil:zogftw:new_datasets" property is set on the receiving pool's parent dataset and its value is not "yes".
• Polish documentation.
• Update and add new tests.

2022-01-11-e290c82

• Add new hooks: zogftw_wtf_message_emitted_hook(), zogftw_fyi_message_emitted_hook(), and zogftw_zpool_export_successful_hook().
• Don't explicitly log that no key file has been found before attaching a provider.
• zogftw_main(): Don't return success if zogftw_unregistered_mode_hook() fails.
• Fix function name in error messages from zogftw_zpool_gets_new_datasets().
• Allow to set additional "zfs send" flags though ZOGFTW_ZFS_SEND_FLAGS.
• zogftw_geli_attach(): Emit an error message if the keyfile exists but is not accessible as file.
• zogftw_sync_zpool(): After resuming, check for additional snapshots that may need syncing. Previously snapshots after the failed and resumed ones were not transfered until the next sync cycle.
• Add undocumented and nearly untested "restore" subcommand.

2022-02-17-47c83cb

• Allow the zogftw_snapshot_not_yet_created_hook() to abort the snapshotting (for example because no new data has been written since the last snapshot).

2022-06-25-03982c7

• After synchronising a dataset, store the last snapshot as property so it can be looked up faster. This can speed up the synchronisation process by several minutes if the disk is slow (for example USB2) and the receiving datasets have lots of snapshots.
• Add a "update-last-snapshot-properties" subcommand to put the last-snapshot property on all the receiving datasets without waiting that they receive updates.

License

zogftw is free software:

Copyright (c) 2010-2022 Fabian Keil <fk@fabiankeil.de>

Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Donations welcome

You can support zogftw development by funding ElectroBSD.

Dedication

zogftw is dedicated to the one I love a couple of individuals affiliated with the NRW Polizei Bergisch Gladbach (not actually shown on the photo).

They may or may not have provided me with a free zogftw penetration test from the point of view of an unskilled attacker, after pretending not to be able to recognize a Tor server that has been running for years.

While I have my hardware back, I do not (yet) know at which point of the investigation they lost interest. Either way, it's the thought that counts.

zogftw presentation at OpenRheinRuhr

There was a zogftw presentation at OpenRheinRuhr 2014. In case you missed it, you may be interested in the slides: zogftw: Verschlüsselte externe ZFS-Backup-Pools schmerzarm verwalten.