Commit | Line | Data |
---|---|---|
69305044 MW |
1 | .ie t .ds o \(bu |
2 | .el .ds o o | |
3 | .de hP | |
4 | .IP | |
5 | \h'-\w'\fB\\$1\ \fP'u'\fB\\$1\ \fP\c | |
6 | .. | |
f6b4ffdc | 7 | .TH rsync-backup 8 "7 October 2012" rsync-backup |
977d0da9 MW |
8 | .SH NAME |
9 | rsync-backup \- back up files using rsync | |
f6b4ffdc MW |
10 | .SH SYNOPSIS |
11 | .B rsync-backup | |
3f496b2b | 12 | .RB [ \-nv ] |
f6b4ffdc MW |
13 | .RB [ \-c |
14 | .IR config-file ] | |
15 | .SH DESCRIPTION | |
16 | The | |
17 | .B rsync-backup | |
18 | script is a backup program of the currently popular | |
19 | .RB ` rsync (1) | |
20 | .BR \-\-link-dest ' | |
21 | variety. It uses | |
22 | .BR rsync 's | |
23 | ability to create hardlinks from (apparently) similar existing local | |
24 | trees to make incremental dumps efficient, even from remote sources. | |
25 | Restoring files is easy because the backups created are just directories | |
26 | full of files, exactly as they were on the source \(en and this is | |
27 | verified using the | |
28 | .BR fshash (1) | |
29 | program. | |
30 | .PP | |
31 | The script does more than just running | |
32 | .BR rsync . | |
33 | It is also responsible for creating and removing snapshots of volumes to | |
34 | be backed up, and expiring old dumps according to a user-specified | |
35 | retention policy. | |
36 | .SS Installation | |
37 | The idea is that the | |
38 | .B rsync-backup | |
39 | script should be installed and run on a central backup server with local | |
40 | access to the backup volumes. | |
41 | .PP | |
42 | The script should be run with full (root) privileges, so that it can | |
43 | correctly record file ownership information. The server should also be | |
44 | able to connect via | |
45 | .BR ssh (1) | |
46 | to the client machines, and run processes there as root. (This is not a | |
47 | security disaster. Remember that the backup server is, in the end, | |
48 | responsible for the integrity of the backup data. A dishonest backup | |
49 | server can easily compromise a client which is being restored from | |
50 | corrupt backup data.) | |
69305044 MW |
51 | .SS Command-line options |
52 | Most of the behaviour of | |
53 | .B rsync-backup | |
54 | is controlled by a configuration file, described starting with the | |
55 | section named | |
56 | .B Configuration commands | |
57 | below. | |
58 | But a few features are controlled by command-line options. | |
59 | .TP | |
60 | .B \-h | |
61 | Show a brief help message for the program, and exit successfully. | |
62 | .TP | |
63 | .B \-V | |
64 | Show | |
65 | .BR rsync-backup 's | |
66 | version number and some choice pieces of build-time configuration, and | |
67 | exit successfully. | |
68 | .TP | |
69 | .BI "\-c " conf | |
70 | Read | |
71 | .I conf | |
72 | instead of the default configuration file (shown as | |
73 | .B conf | |
74 | in the | |
75 | .B \-V | |
76 | output). | |
77 | .TP | |
3f496b2b MW |
78 | .B \-n |
79 | Don't actually take a backup, or write proper logs: instead, write a | |
80 | description of what would be done to standard error. | |
81 | .TP | |
69305044 MW |
82 | .B \-v |
83 | Produce verbose progress information on standard output while the backup | |
84 | is running. This keeps one amused while running a backup | |
85 | interactively. In any event, | |
86 | .B rsync-backup | |
87 | will report failures to standard error, and otherwise run silently, so | |
88 | it doesn't annoy unnecessarily if run by | |
89 | .BR cron (8). | |
90 | .SS Backup process | |
91 | Backing up a filesystem works as follows. | |
92 | .hP \*o | |
93 | Make a snapshot of the filesystem on the client, and ensure that the | |
94 | snapshot is mounted. There are some `trivial' snapshot types which use | |
95 | the existing mounted filesystem, and either prevent processes writing to | |
96 | it during the backup, or just hope for the best. Other snapshot types | |
97 | require the snapshot to be mounted somewhere distinct from the main | |
98 | filesystem, so that the latter can continue being used. | |
99 | .hP \*o | |
100 | Run | |
101 | .B rsync | |
102 | to copy the snapshot to the backup volume \(en specifically, to | |
103 | .IB host / fs / new \fR. | |
104 | If this directory already exists, then it's presumed to be debris from a | |
105 | previous attempt to dump this filesystem: | |
106 | .B rsync | |
107 | will update it appropriately, by adding, deleting or modifying the | |
108 | files. This means that retrying a failed dump \(en after fixing whatever | |
109 | caused it to go wrong, obviously! \(en is usually fairly quick. | |
110 | .hP \*o | |
111 | Run | |
112 | .B fshash | |
113 | on the client to generate a `digest' describing the contents of the | |
114 | filesystem, and send this to the server as | |
115 | .IB host / fs / new .fshash \fR. | |
116 | .hP \*o | |
117 | Release the snapshot: we don't need it any more. | |
118 | .hP \*o | |
119 | Run | |
120 | .B fshash | |
121 | over the new backup; specifically, to | |
122 | .BI tmp/fshash. host . fs . date \fR. | |
123 | This gives us a digest for what the backup volume actually stored. | |
124 | .hP \*o | |
125 | Compare the two | |
126 | .B fshash | |
127 | digests. If they differ then dump the differences to the log file and | |
128 | report a backup failure. (Backups aren't any good if they don't | |
129 | actually back up the right thing. And you stand a better chance of | |
130 | fixing them if you know that they're going wrong.) | |
131 | .hP \*o | |
132 | Commit the backup, by renaming the dump directory to | |
133 | .IB host / fs / date | |
134 | and the | |
135 | .B fshash | |
136 | digest file to | |
137 | .IB host / fs / date .fshash \fR. | |
f6b4ffdc | 138 | .PP |
69305044 | 139 | The backup is now complete. |
f6b4ffdc MW |
140 | .SS Configuration commands |
141 | The configuration file is simply a Bash shell fragment: configuration | |
142 | commands are shell functions. | |
143 | .TP | |
144 | .BI "backup " "fs\fR[:\fIfsarg\fR] ..." | |
145 | Back up the named filesystems. The corresponding | |
146 | .IR fsarg s | |
147 | may be required by the snapshot type. | |
148 | .TP | |
149 | .BI "host " host | |
150 | Future | |
151 | .B backup | |
152 | commands will back up filesystems on the named | |
153 | .IR host . | |
69305044 MW |
154 | To back up filesystems on the backup server itself, use its hostname: |
155 | .B rsync-backup | |
156 | will avoid inefficient and pointless messing about | |
157 | .BR ssh (1) | |
158 | in this case. | |
159 | This command clears the | |
f6b4ffdc | 160 | .B like |
f8d0b27d MW |
161 | list, and resets the retention policy to its default (i.e., the to |
162 | policy defined prior to the first | |
163 | .B host | |
164 | command). | |
f6b4ffdc MW |
165 | .TP |
166 | .BI "like " "host\fR ..." | |
167 | Declare that subsequent filesystems are `similar' to like-named | |
168 | filesystems on the named | |
169 | .IR host s, | |
170 | and that | |
171 | .B rsync | |
172 | should use those trees as potential sources of hardlinkable files. Be | |
173 | careful when using this option without | |
174 | .BR rsync 's | |
175 | .B \-\-checksum | |
176 | option: an erroneous hardlink will cause the backup to fail. (The | |
177 | backup won't be left silently incorrect.) | |
178 | .TP | |
179 | .BI "retain " frequency " " duration | |
180 | Define part a backup retention policy: backup trees of the | |
181 | .I frequency | |
182 | should be kept for the | |
183 | .IR duration . | |
184 | The | |
185 | .I frequency | |
186 | can be | |
187 | .BR daily , | |
188 | .BR weekly , | |
189 | .BR monthly , | |
190 | or | |
69305044 | 191 | .B annually |
f6b4ffdc MW |
192 | (or |
193 | .BR yearly , | |
194 | which means the same); the | |
195 | .I duration | |
196 | may be any of | |
197 | .BR week , | |
198 | .BR month , | |
199 | .BR year , | |
200 | or | |
201 | .BR forever . | |
202 | Expiry considers each existing dump against the policy lines in order: | |
203 | the last applicable line determines the dump's fate \(en so you should | |
204 | probably write the lines in decreasing order of duration. | |
f8d0b27d MW |
205 | .PP |
206 | Groups of | |
207 | .B retain | |
208 | commands between | |
209 | .B host | |
210 | and/or | |
211 | .B backup | |
212 | commands collectively define a retention policy. Once a policy is | |
213 | defined, subsequent | |
214 | .B backup | |
215 | operations use the policy. The first | |
216 | .B retain | |
217 | command after a | |
218 | .B host | |
219 | or | |
220 | .B backup | |
221 | command clears the policy and starts defining a new one. The policy | |
222 | defined before the first | |
223 | .B host | |
224 | is the | |
225 | .I default | |
226 | policy: at the start of each | |
227 | .B host | |
228 | stanza, the policy is reset to the default. | |
f6b4ffdc | 229 | .TP |
5675acda MW |
230 | .BI "retry " count |
231 | The | |
232 | .B live | |
233 | snapshot type (see below) doesn't prevent a filesystem from being | |
234 | modified while it's being backed up. If this happens, the | |
235 | .B fshash | |
236 | pass will detect the difference and fail. If the filesystem in question | |
237 | is relatively quiescent, then maybe retrying the backup will result in a | |
238 | successful consistent copy. Following this command, a backup which | |
239 | results in an | |
240 | .B fshash | |
241 | mismatch will be retried up to | |
242 | .I count | |
243 | times before being declared a failure. | |
244 | .TP | |
f6b4ffdc MW |
245 | .BI "snap " type " " \fR[\fIargs\fR...] |
246 | Use the snapshot | |
247 | .I type | |
248 | for subsequent backups. Some snapshot types require additional | |
5675acda MW |
249 | arguments, which may be supplied here. This command clears the |
250 | .B retry | |
251 | counter. | |
f6b4ffdc MW |
252 | .SS Configuration variables |
253 | The following shell variables may be overridden by the configuration | |
254 | file. | |
255 | .TP | |
8e40e6cf MW |
256 | .B HASH |
257 | The hash function to use for verifying archive integrity. This is | |
258 | passed to the | |
259 | .B \-H | |
260 | option of | |
261 | .BR fshash , | |
262 | so it must name one of the hash functions supported by your Python's | |
263 | .B hashlib | |
264 | module. | |
265 | The default is | |
266 | .BR sha256 . | |
267 | .TP | |
a8447303 MW |
268 | .B INDEXDB |
269 | The name of a SQLite database initialized by | |
270 | .BR update-bkp-index (8) | |
271 | in which an index is maintained of which dumps are on which backup | |
272 | volumes. If the file doesn't exist, then no index is maintained. The | |
273 | default is | |
274 | .IB localstatedir /lib/bkp/index.db | |
275 | where | |
276 | .I localstatedir | |
277 | is the state directory configured at build time. | |
278 | .TP | |
f6b4ffdc MW |
279 | .B MAXLOG |
280 | The number of log files to be kept for each filesystem. Old logfiles | |
281 | are deleted to keep the total number below this bound. The default | |
282 | value is 14. | |
283 | .TP | |
a8447303 MW |
284 | .B METADIR |
285 | The metadata directory for the currently mounted backup volume. | |
286 | The default is | |
287 | .IB mntbkpdir /meta | |
288 | where | |
289 | .I mntbkpdir | |
290 | is the backup mount directory configured at build time. | |
291 | .TP | |
f6b4ffdc MW |
292 | .B RSYNCOPTS |
293 | Command-line options to pass to | |
294 | .BR rsync (1) | |
295 | in addition to the basic set: | |
69305044 MW |
296 | .B \-\-archive |
297 | .B \-\-hard-links | |
298 | .B \-\-numeric-ids | |
299 | .B \-\-del | |
300 | .B \-\-sparse | |
301 | .B \-\-compress | |
302 | .B \-\-one-file-system | |
303 | .B \-\-partial | |
304 | .BR "\-\-filter=""dir-merge .rsync-backup""" . | |
f6b4ffdc MW |
305 | The default is |
306 | .BR \-\-verbose . | |
307 | .TP | |
308 | .B SNAPDIR | |
309 | LVM (and | |
310 | .BR rfreezefs ) | |
311 | snapshots are mounted on subdirectories below the | |
312 | .B SNAPDIR | |
313 | .IR "on backup clients" . | |
314 | The default is | |
315 | .IB mntbkpdir /snap | |
316 | where | |
317 | .I mntbkpdir | |
318 | is the backup mount directory configured at build time. | |
319 | .TP | |
320 | .B SNAPSIZE | |
321 | The volume size option to pass to | |
322 | .BR lvcreate (8) | |
323 | when creating a snapshot. The default is | |
324 | .B \-l10%ORIGIN | |
325 | which seems to work fairly well. | |
326 | .TP | |
327 | .B STOREDIR | |
328 | Where the actual backup trees should be stored. See the section on | |
329 | .B Archive structure | |
330 | below. | |
331 | The default is | |
332 | .IB mntbkpdir /store | |
333 | where | |
334 | .I mntbkpdir | |
335 | is the backup mount directory configured at build time. | |
336 | .TP | |
a8447303 MW |
337 | .B VOLUME |
338 | The name of the current volume. If this is left unset, the volume name | |
339 | is read from the file | |
340 | .IB METADIR /volume | |
341 | once at the start of the backup run. | |
f6b4ffdc MW |
342 | .SS Hook functions |
343 | The configuration file may define shell functions to perform custom | |
344 | actions at various points in the backup process. | |
345 | .TP | |
346 | .BI "backup_precommit_hook " host " " fs " " date | |
347 | Called after a backup has been verified complete and about to be | |
348 | committed. The backup tree is in | |
349 | .B new | |
350 | in the current directory, and the | |
351 | .B fshash | |
352 | manifest is in | |
353 | .BR new.fshash . | |
354 | A typical action would be to create a digital signature on the | |
355 | manifest. | |
356 | .TP | |
357 | .BI "backup_commit_hook " host " " fs " " date | |
358 | Called during the commit procedure. The backup tree and manifest have | |
359 | been renamed into their proper places. Typically one would use this | |
360 | hook to rename files created by the | |
361 | .B backup_precommit_hook | |
362 | function. | |
363 | .TP | |
364 | .BR "whine " [ \-n ] " " \fItext\fR... | |
365 | Called to report `interesting' events when the | |
366 | .B \-v | |
367 | option is in force. The default action is to echo the | |
368 | .I text | |
369 | to (what was initially) standard output, followed by a newline unless | |
370 | .B \-n | |
371 | is given. | |
372 | .SS Snapshot types | |
373 | The following snapshot types are available. | |
374 | .TP | |
375 | .B live | |
376 | A trivial snapshot type: attempts to back up a live filesystem. How | |
377 | well this works depends on how active the filesystem is. If files | |
378 | change while the dump is in progress then the | |
379 | .B fshash | |
380 | verification will likely fail. Backups using this snapshot type must | |
381 | specify the filesystem mount point as the | |
382 | .IR fsarg . | |
383 | .TP | |
384 | .B ro | |
385 | A slightly less trivial snapshot type: make the filesystem read-only | |
386 | while the dump is in progress. Backups using this snapshot type must | |
387 | specify the filesystem mount point as the | |
388 | .IR fsarg . | |
389 | .TP | |
390 | .BI "lvm " vg | |
391 | Create snapshots using LVM. The snapshot argument is interpreted as the | |
392 | relevant volume group. The filesystem name is interpreted as the origin | |
393 | volume name; the snapshot will be called | |
394 | .IB fs .bkp | |
395 | and mounted on | |
396 | .IB SNAPDIR / fs \fR; | |
397 | space will be allocated to it according to the | |
398 | .I SNAPSIZE | |
399 | variable. | |
400 | .TP | |
401 | .BI "rfreezefs " client " " vg | |
402 | This gets complicated. Suppose that a server has an LVM volume group, | |
403 | and exports (somehow) a logical volume to a client. Examples are a host | |
404 | providing a virtual disk to a guest, or a server providing | |
405 | network-attached storage to a client. The server can create a snapshot | |
406 | of the volume using LVM, but must synchronize with the client to ensure | |
407 | that the filesystem image captured in the snapshot is clean. The | |
408 | .BR rfreezefs (8) | |
409 | program should be installed on the client to perform this rather | |
410 | delicate synchronization. Declare the server using the | |
411 | .B host | |
412 | command as usual; pass the client's name as the | |
413 | .I client | |
414 | and the | |
415 | server's volume group name as the | |
416 | .I vg | |
417 | snapshot arguments. Finally, backups using this snapshot type must | |
418 | specify the filesystem mount point (or, actually, any file in the | |
419 | filesystem) on the client, as the | |
420 | .IR fsarg . | |
421 | .PP | |
422 | Additional snapshot types can be defined in the configuration file. A | |
423 | snapshot type requires two shell functions. | |
424 | .TP | |
425 | .BI snap_ type " " snapargs " " fs " " fsarg | |
426 | Create the snapshot, and write the mountpoint (on the client host) to | |
427 | standard output, in a form suitable as an argument to | |
428 | .BR rsync . | |
429 | .TP | |
430 | .BI unsnap_ type " " snapargs " " fs " " fsarg | |
431 | Remove the snapshot. | |
432 | .PP | |
433 | There are a number of utility functions which can be used by snapshot | |
434 | type handlers: please see the script for details. Please send the | |
435 | author interesting snapshot handlers for inclusion in the main | |
436 | distribution. | |
437 | .SS Archive structure | |
69305044 MW |
438 | Backup trees are stored in a fairly straightforward directory tree. |
439 | .PP | |
440 | At the top level is one directory for each client host. There are also | |
441 | some special entries: | |
442 | .TP | |
6037bdb3 MW |
443 | .B \&.rsync-backup-store |
444 | This file must be present in order to indicate that a backup volume is | |
445 | present (and not just an empty mount point). | |
446 | .TP | |
69305044 MW |
447 | .B fshash.cache |
448 | The cache database used for improving performance of local file | |
449 | hashing. There may be other | |
450 | .B fshash.cache-* | |
451 | files used by SQLite for its own purposes. | |
452 | .TP | |
453 | .B lost+found | |
454 | Part of the filesystem used on the backup volume. You don't want to | |
455 | mess with this. | |
456 | .TP | |
457 | .B tmp | |
458 | Used to store temporary files during the backup process. (Some of them | |
459 | want to be on the same filesystem as the rest of the backup.) When | |
460 | things go wrong, files are left behind in the hope that they might help | |
461 | someone debug the mess. It's always safe to delete the files in here | |
462 | when no backup is running. | |
463 | .PP | |
464 | So don't use those names for your hosts. | |
465 | .PP | |
466 | The next layer down contains a directory for each filesystem on the given host. | |
467 | .PP | |
468 | The bottom layer contains a directory for each dump of that filesystem, | |
469 | named with the date at which the dump was started (in ISO8601 | |
470 | .IB yyyy \(en mm \(en dd | |
471 | format), together with associated files named | |
472 | .IB date .* \fR. | |
473 | .SH SEE ALSO | |
474 | .BR fshash (1), | |
475 | .BR lvm (8), | |
476 | .BR rfreezefs (8), | |
477 | .BR rsync (1), | |
a8447303 MW |
478 | .BR ssh (1), |
479 | .BR update-bkp-index (8). | |
69305044 MW |
480 | .SH AUTHOR |
481 | Mark Wooding, <mdw@distorted.org.uk> |