1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
|
.\"
.\" CDDL HEADER START
.\"
.\" The contents of this file are subject to the terms of the
.\" Common Development and Distribution License (the "License").
.\" You may not use this file except in compliance with the License.
.\"
.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
.\" or http://www.opensolaris.org/os/licensing.
.\" See the License for the specific language governing permissions
.\" and limitations under the License.
.\"
.\" When distributing Covered Code, include this CDDL HEADER in each
.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE.
.\" If applicable, add the following below this CDDL HEADER, with the
.\" fields enclosed by brackets "[]" replaced with your own identifying
.\" information: Portions Copyright [yyyy] [name of copyright owner]
.\"
.\" CDDL HEADER END
.\"
.\" Copyright (c) 2009 Sun Microsystems, Inc. All Rights Reserved.
.\" Copyright 2011 Joshua M. Clulow <josh@sysmgr.org>
.\" Copyright (c) 2011, 2019 by Delphix. All rights reserved.
.\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
.\" Copyright (c) 2014, Joyent, Inc. All rights reserved.
.\" Copyright (c) 2014 by Adam Stevko. All rights reserved.
.\" Copyright (c) 2014 Integros [integros.com]
.\" Copyright 2019 Richard Laager. All rights reserved.
.\" Copyright 2018 Nexenta Systems, Inc.
.\" Copyright 2019 Joyent, Inc.
.\"
.Dd April 15, 2021
.Dt ZFS-SEND 8
.Os
.
.Sh NAME
.Nm zfs-send
.Nd generate backup stream of ZFS dataset
.Sh SYNOPSIS
.Nm zfs
.Cm send
.Op Fl DLPRbcehnpsvw
.Op Oo Fl I Ns | Ns Fl i Oc Ar snapshot
.Ar snapshot
.Nm zfs
.Cm send
.Op Fl DLPcensvw
.Op Fl i Ar snapshot Ns | Ns Ar bookmark
.Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot
.Nm zfs
.Cm send
.Fl -redact Ar redaction_bookmark
.Op Fl DLPcenpv
.Op Fl i Ar snapshot Ns | Ns Ar bookmark
.Ar snapshot
.Nm zfs
.Cm send
.Op Fl Penv
.Fl t
.Ar receive_resume_token
.Nm zfs
.Cm send
.Op Fl Pnv
.Fl S Ar filesystem
.Nm zfs
.Cm redact
.Ar snapshot redaction_bookmark
.Ar redaction_snapshot Ns …
.
.Sh DESCRIPTION
.Bl -tag -width ""
.It Xo
.Nm zfs
.Cm send
.Op Fl DLPRbcehnpvw
.Op Oo Fl I Ns | Ns Fl i Oc Ar snapshot
.Ar snapshot
.Xc
Creates a stream representation of the second
.Ar snapshot ,
which is written to standard output.
The output can be redirected to a file or to a different system
.Po for example, using
.Xr ssh 1
.Pc .
By default, a full stream is generated.
.Bl -tag -width "-D"
.It Fl D , -dedup
Deduplicated send is no longer supported.
This flag is accepted for backwards compatibility, but a regular,
non-deduplicated stream will be generated.
.It Fl I Ar snapshot
Generate a stream package that sends all intermediary snapshots from the first
snapshot to the second snapshot.
For example,
.Fl I Em @a Em fs@d
is similar to
.Fl i Em @a Em fs@b Ns \&; Fl i Em @b Em fs@c Ns \&; Fl i Em @c Em fs@d .
The incremental source may be specified as with the
.Fl i
option.
.It Fl L , -large-block
Generate a stream which may contain blocks larger than 128KB.
This flag has no effect if the
.Sy large_blocks
pool feature is disabled, or if the
.Sy recordsize
property of this filesystem has never been set above 128KB.
The receiving system must have the
.Sy large_blocks
pool feature enabled as well.
See
.Xr zpool-features 7
for details on ZFS feature flags and the
.Sy large_blocks
feature.
.It Fl P , -parsable
Print machine-parsable verbose information about the stream package generated.
.It Fl R , -replicate
Generate a replication stream package, which will replicate the specified
file system, and all descendent file systems, up to the named snapshot.
When received, all properties, snapshots, descendent file systems, and clones
are preserved.
.Pp
If the
.Fl i
or
.Fl I
flags are used in conjunction with the
.Fl R
flag, an incremental replication stream is generated.
The current values of properties, and current snapshot and file system names are
set when the stream is received.
If the
.Fl F
flag is specified when this stream is received, snapshots and file systems that
do not exist on the sending side are destroyed.
If the
.Fl R
flag is used to send encrypted datasets, then
.Fl w
must also be specified.
.It Fl e , -embed
Generate a more compact stream by using
.Sy WRITE_EMBEDDED
records for blocks which are stored more compactly on disk by the
.Sy embedded_data
pool feature.
This flag has no effect if the
.Sy embedded_data
feature is disabled.
The receiving system must have the
.Sy embedded_data
feature enabled.
If the
.Sy lz4_compress
feature is active on the sending system, then the receiving system must have
that feature enabled as well.
Datasets that are sent with this flag may not be
received as an encrypted dataset, since encrypted datasets cannot use the
.Sy embedded_data
feature.
See
.Xr zpool-features 7
for details on ZFS feature flags and the
.Sy embedded_data
feature.
.It Fl b , -backup
Sends only received property values whether or not they are overridden by local
settings, but only if the dataset has ever been received.
Use this option when you want
.Nm zfs Cm receive
to restore received properties backed up on the sent dataset and to avoid
sending local settings that may have nothing to do with the source dataset,
but only with how the data is backed up.
.It Fl c , -compressed
Generate a more compact stream by using compressed WRITE records for blocks
which are compressed on disk and in memory
.Po see the
.Sy compression
property for details
.Pc .
If the
.Sy lz4_compress
feature is active on the sending system, then the receiving system must have
that feature enabled as well.
If the
.Sy large_blocks
feature is enabled on the sending system but the
.Fl L
option is not supplied in conjunction with
.Fl c ,
then the data will be decompressed before sending so it can be split into
smaller block sizes.
Streams sent with
.Fl c
will not have their data recompressed on the receiver side using
.Fl o Sy compress Ns = Ar value .
The data will stay compressed as it was from the sender.
The new compression property will be set for future data.
Note that uncompressed data from the sender will still attempt to
compress on the receiver, unless you specify
.Fl o Sy compress Ns = Em off .
.It Fl w , -raw
For encrypted datasets, send data exactly as it exists on disk.
This allows backups to be taken even if encryption keys are not currently loaded.
The backup may then be received on an untrusted machine since that machine will
not have the encryption keys to read the protected data or alter it without
being detected.
Upon being received, the dataset will have the same encryption
keys as it did on the send side, although the
.Sy keylocation
property will be defaulted to
.Sy prompt
if not otherwise provided.
For unencrypted datasets, this flag will be equivalent to
.Fl Lec .
Note that if you do not use this flag for sending encrypted datasets, data will
be sent unencrypted and may be re-encrypted with a different encryption key on
the receiving system, which will disable the ability to do a raw send to that
system for incrementals.
.It Fl h , -holds
Generate a stream package that includes any snapshot holds (created with the
.Nm zfs Cm hold
command), and indicating to
.Nm zfs Cm receive
that the holds be applied to the dataset on the receiving system.
.It Fl i Ar snapshot
Generate an incremental stream from the first
.Ar snapshot
.Pq the incremental source
to the second
.Ar snapshot
.Pq the incremental target .
The incremental source can be specified as the last component of the snapshot
name
.Po the
.Sy @
character and following
.Pc
and it is assumed to be from the same file system as the incremental target.
.Pp
If the destination is a clone, the source may be the origin snapshot, which must
be fully specified
.Po for example,
.Em pool/fs@origin ,
not just
.Em @origin
.Pc .
.It Fl n , -dryrun
Do a dry-run
.Pq Qq No-op
send.
Do not generate any actual send data.
This is useful in conjunction with the
.Fl v
or
.Fl P
flags to determine what data will be sent.
In this case, the verbose output will be written to standard output
.Po contrast with a non-dry-run, where the stream is written to standard output
and the verbose output goes to standard error
.Pc .
.It Fl p , -props
Include the dataset's properties in the stream.
This flag is implicit when
.Fl R
is specified.
The receiving system must also support this feature.
Sends of encrypted datasets must use
.Fl w
when using this flag.
.It Fl s , -skip-missing
Allows sending a replication stream even when there are snapshots missing in the
hierarchy.
When a snapshot is missing, instead of throwing an error and aborting the send,
a warning is printed to the standard error stream and the dataset to which it belongs
and its descendents are skipped.
This flag can only be used in conjunction with
.Fl R .
.It Fl v , -verbose
Print verbose information about the stream package generated.
This information includes a per-second report of how much data has been sent.
.Pp
The format of the stream is committed.
You will be able to receive your streams on future versions of ZFS.
.El
.It Xo
.Nm zfs
.Cm send
.Op Fl DLPcenvw
.Op Fl i Ar snapshot Ns | Ns Ar bookmark
.Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot
.Xc
Generate a send stream, which may be of a filesystem, and may be incremental
from a bookmark.
If the destination is a filesystem or volume, the pool must be read-only, or the
filesystem must not be mounted.
When the stream generated from a filesystem or volume is received, the default
snapshot name will be
.Qq --head-- .
.Bl -tag -width "-D"
.It Fl D , -dedup
Deduplicated send is no longer supported.
This flag is accepted for backwards compatibility, but a regular,
non-deduplicated stream will be generated.
.It Fl L , -large-block
Generate a stream which may contain blocks larger than 128KB.
This flag has no effect if the
.Sy large_blocks
pool feature is disabled, or if the
.Sy recordsize
property of this filesystem has never been set above 128KB.
The receiving system must have the
.Sy large_blocks
pool feature enabled as well.
See
.Xr zpool-features 7
for details on ZFS feature flags and the
.Sy large_blocks
feature.
.It Fl P , -parsable
Print machine-parsable verbose information about the stream package generated.
.It Fl c , -compressed
Generate a more compact stream by using compressed WRITE records for blocks
which are compressed on disk and in memory
.Po see the
.Sy compression
property for details
.Pc .
If the
.Sy lz4_compress
feature is active on the sending system, then the receiving system must have
that feature enabled as well.
If the
.Sy large_blocks
feature is enabled on the sending system but the
.Fl L
option is not supplied in conjunction with
.Fl c ,
then the data will be decompressed before sending so it can be split into
smaller block sizes.
.It Fl w , -raw
For encrypted datasets, send data exactly as it exists on disk.
This allows backups to be taken even if encryption keys are not currently loaded.
The backup may then be received on an untrusted machine since that machine will
not have the encryption keys to read the protected data or alter it without
being detected.
Upon being received, the dataset will have the same encryption
keys as it did on the send side, although the
.Sy keylocation
property will be defaulted to
.Sy prompt
if not otherwise provided.
For unencrypted datasets, this flag will be equivalent to
.Fl Lec .
Note that if you do not use this flag for sending encrypted datasets, data will
be sent unencrypted and may be re-encrypted with a different encryption key on
the receiving system, which will disable the ability to do a raw send to that
system for incrementals.
.It Fl e , -embed
Generate a more compact stream by using
.Sy WRITE_EMBEDDED
records for blocks which are stored more compactly on disk by the
.Sy embedded_data
pool feature.
This flag has no effect if the
.Sy embedded_data
feature is disabled.
The receiving system must have the
.Sy embedded_data
feature enabled.
If the
.Sy lz4_compress
feature is active on the sending system, then the receiving system must have
that feature enabled as well.
Datasets that are sent with this flag may not be received as an encrypted dataset,
since encrypted datasets cannot use the
.Sy embedded_data
feature.
See
.Xr zpool-features 7
for details on ZFS feature flags and the
.Sy embedded_data
feature.
.It Fl i Ar snapshot Ns | Ns Ar bookmark
Generate an incremental send stream.
The incremental source must be an earlier snapshot in the destination's history.
It will commonly be an earlier snapshot in the destination's file system, in
which case it can be specified as the last component of the name
.Po the
.Sy #
or
.Sy @
character and following
.Pc .
.Pp
If the incremental target is a clone, the incremental source can be the origin
snapshot, or an earlier snapshot in the origin's filesystem, or the origin's
origin, etc.
.It Fl n , -dryrun
Do a dry-run
.Pq Qq No-op
send.
Do not generate any actual send data.
This is useful in conjunction with the
.Fl v
or
.Fl P
flags to determine what data will be sent.
In this case, the verbose output will be written to standard output
.Po contrast with a non-dry-run, where the stream is written to standard output
and the verbose output goes to standard error
.Pc .
.It Fl v , -verbose
Print verbose information about the stream package generated.
This information includes a per-second report of how much data has been sent.
.El
.It Xo
.Nm zfs
.Cm send
.Fl -redact Ar redaction_bookmark
.Op Fl DLPcenpv
.Op Fl i Ar snapshot Ns | Ns Ar bookmark
.Ar snapshot
.Xc
Generate a redacted send stream.
This send stream contains all blocks from the snapshot being sent that aren't
included in the redaction list contained in the bookmark specified by the
.Fl -redact
(or
.Fl d )
flag.
The resulting send stream is said to be redacted with respect to the snapshots
the bookmark specified by the
.Fl -redact No flag was created with.
The bookmark must have been created by running
.Nm zfs Cm redact
on the snapshot being sent.
.Pp
This feature can be used to allow clones of a filesystem to be made available on
a remote system, in the case where their parent need not (or needs to not) be
usable.
For example, if a filesystem contains sensitive data, and it has clones where
that sensitive data has been secured or replaced with dummy data, redacted sends
can be used to replicate the secured data without replicating the original
sensitive data, while still sharing all possible blocks.
A snapshot that has been redacted with respect to a set of snapshots will
contain all blocks referenced by at least one snapshot in the set, but will
contain none of the blocks referenced by none of the snapshots in the set.
In other words, if all snapshots in the set have modified a given block in the
parent, that block will not be sent; but if one or more snapshots have not
modified a block in the parent, they will still reference the parent's block, so
that block will be sent.
Note that only user data will be redacted.
.Pp
When the redacted send stream is received, we will generate a redacted
snapshot.
Due to the nature of redaction, a redacted dataset can only be used in the
following ways:
.Bl -enum -width "a."
.It
To receive, as a clone, an incremental send from the original snapshot to one
of the snapshots it was redacted with respect to.
In this case, the stream will produce a valid dataset when received because all
blocks that were redacted in the parent are guaranteed to be present in the
child's send stream.
This use case will produce a normal snapshot, which can be used just like other
snapshots.
.
.It
To receive an incremental send from the original snapshot to something
redacted with respect to a subset of the set of snapshots the initial snapshot
was redacted with respect to.
In this case, each block that was redacted in the original is still redacted
(redacting with respect to additional snapshots causes less data to be redacted
(because the snapshots define what is permitted, and everything else is
redacted)).
This use case will produce a new redacted snapshot.
.It
To receive an incremental send from a redaction bookmark of the original
snapshot that was created when redacting with respect to a subset of the set of
snapshots the initial snapshot was created with respect to
anything else.
A send stream from such a redaction bookmark will contain all of the blocks
necessary to fill in any redacted data, should it be needed, because the sending
system is aware of what blocks were originally redacted.
This will either produce a normal snapshot or a redacted one, depending on
whether the new send stream is redacted.
.It
To receive an incremental send from a redacted version of the initial
snapshot that is redacted with respect to a subject of the set of snapshots the
initial snapshot was created with respect to.
A send stream from a compatible redacted dataset will contain all of the blocks
necessary to fill in any redacted data.
This will either produce a normal snapshot or a redacted one, depending on
whether the new send stream is redacted.
.It
To receive a full send as a clone of the redacted snapshot.
Since the stream is a full send, it definitionally contains all the data needed
to create a new dataset.
This use case will either produce a normal snapshot or a redacted one, depending
on whether the full send stream was redacted.
.El
.Pp
These restrictions are detected and enforced by
.Nm zfs Cm receive ;
a redacted send stream will contain the list of snapshots that the stream is
redacted with respect to.
These are stored with the redacted snapshot, and are used to detect and
correctly handle the cases above.
Note that for technical reasons,
raw sends and redacted sends cannot be combined at this time.
.It Xo
.Nm zfs
.Cm send
.Op Fl Penv
.Fl t
.Ar receive_resume_token
.Xc
Creates a send stream which resumes an interrupted receive.
The
.Ar receive_resume_token
is the value of this property on the filesystem or volume that was being
received into.
See the documentation for
.Nm zfs Cm receive Fl s
for more details.
.It Xo
.Nm zfs
.Cm send
.Op Fl Pnv
.Op Fl i Ar snapshot Ns | Ns Ar bookmark
.Fl S
.Ar filesystem
.Xc
Generate a send stream from a dataset that has been partially received.
.Bl -tag -width "-L"
.It Fl S , -saved
This flag requires that the specified filesystem previously received a resumable
send that did not finish and was interrupted.
In such scenarios this flag
enables the user to send this partially received state.
Using this flag will always use the last fully received snapshot
as the incremental source if it exists.
.El
.It Xo
.Nm zfs
.Cm redact
.Ar snapshot redaction_bookmark
.Ar redaction_snapshot Ns …
.Xc
Generate a new redaction bookmark.
In addition to the typical bookmark information, a redaction bookmark contains
the list of redacted blocks and the list of redaction snapshots specified.
The redacted blocks are blocks in the snapshot which are not referenced by any
of the redaction snapshots.
These blocks are found by iterating over the metadata in each redaction snapshot
to determine what has been changed since the target snapshot.
Redaction is designed to support redacted zfs sends; see the entry for
.Nm zfs Cm send
for more information on the purpose of this operation.
If a redact operation fails partway through (due to an error or a system
failure), the redaction can be resumed by rerunning the same command.
.El
.Ss Redaction
ZFS has support for a limited version of data subsetting, in the form of
redaction.
Using the
.Nm zfs Cm redact
command, a
.Sy redaction bookmark
can be created that stores a list of blocks containing sensitive information.
When provided to
.Nm zfs Cm send ,
this causes a
.Sy redacted send
to occur.
Redacted sends omit the blocks containing sensitive information,
replacing them with REDACT records.
When these send streams are received, a
.Sy redacted dataset
is created.
A redacted dataset cannot be mounted by default, since it is incomplete.
It can be used to receive other send streams.
In this way datasets can be used for data backup and replication,
with all the benefits that zfs send and receive have to offer,
while protecting sensitive information from being
stored on less-trusted machines or services.
.Pp
For the purposes of redaction, there are two steps to the process.
A redact step, and a send/receive step.
First, a redaction bookmark is created.
This is done by providing the
.Nm zfs Cm redact
command with a parent snapshot, a bookmark to be created, and a number of
redaction snapshots.
These redaction snapshots must be descendants of the parent snapshot,
and they should modify data that is considered sensitive in some way.
Any blocks of data modified by all of the redaction snapshots will
be listed in the redaction bookmark, because it represents the truly sensitive
information.
When it comes to the send step, the send process will not send
the blocks listed in the redaction bookmark, instead replacing them with
REDACT records.
When received on the target system, this will create a
redacted dataset, missing the data that corresponds to the blocks in the
redaction bookmark on the sending system.
The incremental send streams from
the original parent to the redaction snapshots can then also be received on
the target system, and this will produce a complete snapshot that can be used
normally.
Incrementals from one snapshot on the parent filesystem and another
can also be done by sending from the redaction bookmark, rather than the
snapshots themselves.
.Pp
In order to make the purpose of the feature more clear, an example is provided.
Consider a zfs filesystem containing four files.
These files represent information for an online shopping service.
One file contains a list of usernames and passwords, another contains purchase histories,
a third contains click tracking data, and a fourth contains user preferences.
The owner of this data wants to make it available for their development teams to
test against, and their market research teams to do analysis on.
The development teams need information about user preferences and the click
tracking data, while the market research teams need information about purchase
histories and user preferences.
Neither needs access to the usernames and passwords.
However, because all of this data is stored in one ZFS filesystem,
it must all be sent and received together.
In addition, the owner of the data
wants to take advantage of features like compression, checksumming, and
snapshots, so they do want to continue to use ZFS to store and transmit their data.
Redaction can help them do so.
First, they would make two clones of a snapshot of the data on the source.
In one clone, they create the setup they want their market research team to see;
they delete the usernames and passwords file,
and overwrite the click tracking data with dummy information.
In another, they create the setup they want the development teams
to see, by replacing the passwords with fake information and replacing the
purchase histories with randomly generated ones.
They would then create a redaction bookmark on the parent snapshot,
using snapshots on the two clones as redaction snapshots.
The parent can then be sent, redacted, to the target
server where the research and development teams have access.
Finally, incremental sends from the parent snapshot to each of the clones can be sent
to and received on the target server; these snapshots are identical to the
ones on the source, and are ready to be used, while the parent snapshot on the
target contains none of the username and password data present on the source,
because it was removed by the redacted send operation.
.
.Sh SEE ALSO
.Xr zfs-bookmark 8 ,
.Xr zfs-receive 8 ,
.Xr zfs-redact 8 ,
.Xr zfs-snapshot 8
|