BorgBackup

Stuff related to BorgBackup, a deduplicating archiver with compression and encryption.

Patches

USeful recipes

Running borg non-root using capabilities.

List repository showing archive sizes.

List an archive along with the chunk IDs associated to files.

Performance benchmarks

Compression

Results of a non-scientific test of some of the compression options offered by BorgBackup.

Tests performed on a Lenovo Thinkpad P52, Core i7-8850H, 64 GB RAM, 2 TB Samsung 970 EVO SSD (LUKS encrypted).

The entire system was backed up, consisting of the Debian buster system files, sereveral Debinan buster and stretch VM images, a few Debian installer and live ISO images, some photos and my Email, yielding a somewhat representative mix of data.

A new repository was created each time, so different chunker hash seeds are part of the equiation.

BorgBackup version 1.1.9 was used for these tests.

Method Time Original Size Compressed Deduplicated
none 34:08 332.36 GB 332.38 GB 145.74 GB
lz4 31:36 332.36 GB 133.98 GB 90.67 GB
auto,zstd 35:20 332.36 GB 115.12 GB 78.87 GB
zstd 34:44 332.36 GB 114.50 GB 78.57 GB
zstd,6 44:58 332.40 GB 113.18 GB 77.73 GB
auto,zstd,10 59:55 332.37 GB 112.35 GB 77.51 GB
zstd,7 48:17 332.41 GB 112.35 GB 77.18 GB
zstd,8 55:55 332.50 GB 112.03 GB 77.11 GB
zstd,9 57:06 332.50 GB 111.85 GB 77.11 GB
zstd,10 1:05:11 332.49 GB 111.61 GB 76.92 GB
zstd,16 3:34:29 332.40 GB 110.35 GB 76.10 GB
auto,zstd,22 6:27:56 332.48 GB 109.26 GB 75.59 GB
zstd,22 7:34:02 332.41 GB 108.35 GB 74.72 GB
zstd,19 6:22:30 332.45 GB 108.33 GB 74.61 GB
auto,lzma 5:34:14 332.45 GB 106.42 GB 73.50 GB
lzma 7:58:28 332.36 GB 105.38 GB 73.09 GB

Compression of RAW fises

Results of a non-scientific test of some types of RAW image formats using LZMA compression as well as ZSTD in the case where it did prove efficient, LZ4 did in none of the cases. Deduplication among the unique images did not happen in a noticeable way.

A new repository was created once and each test used a copy of that empty repository, so identical chunker hash seeds and thus identical chunking and deduplication are used.

BorgBackup version 1.1.11 with above patches was used for these tests, ie. including the copmression improvementns.

RAW Image Original Size Compressed RAW Image Size (avg.)
Camera Type Method Size Ratio Original Compressed
Canon EOS 6D CR2 8.01 GB lzma 7.75 GB 1.033 22.2 MB 21.5 MB
Sony A7R IV ARW (compressed) 4.65 GB lzma 4.23 GB 1.099 59 MB 53.7 MB
Sony A7R IV ARW (uncompressed) 8.62 GB zstd,6 6.27 GB 1.375 117.4 MB 85.3 MB
Sony A7R IV ARW (uncompressed) 8.62 GB lzma 5.14 GB 1.677 117.4 MB 70 MB

Check

Benchmarks of borg check operations, comparing different hardware, to get a feeling for the demands and run-time of these in practice.

Tests performed using identical copies of the same repository on a Samsung 970 EVO Plus 2 TB M.2 NVME and Western Digital WD Elements 10 TB USB 3 hard disk. Each installed into, respectively connected to a Supermicro X11SCA-F, Xeon E-2146G, 64 GB ECC RAM, with the NVME either in the on-board M.2 slot connected via DMI (yielding about 1.7 GB/s sequential read) or using a Supermicro AOC-SLG3-2M2-O PCIe to M.2 adapter connected installed in a PCIe slot directly connected to the CPU (yielding about 3.2 GB/s sequential read). Lenovo Thinkpad P52, Core i7-8850H, 64 GB RAM

BorgBackup version 1.1.9 was used for these benchmarks.

Sequential disk read speeds

Approximate sequential read speed estimated using dd with 4 GB block size.

Disk System OS Squential read speed
Samsung 970 EVO Plus 2 TB Thinkpad P52 Debian buster 3.6 GB/s
Samsung 970 EVO Plus 2 TB X11SCA-F (CPU PCIe) Debian stretch 3.2 GB/s
Samsung 970 EVO Plus 2 TB X11SCA-F (DMI PCIe) Debian buster 2.5 GB/s
Samsung 970 EVO Plus 2 TB X11SCA-F (DMI PCIe) Debian stretch 1.7 GB/s
WD Elements 10 TB 190 MB/s

Repository

This LZMA compressed repository was used in the benchmarks:

Archives Original Size Compressed Deduplicated Unique chunks
110 56.31 TB 44.69 TB 1.68 TB 2024761

borg check

Client Server Disk Time
Core i7-8850H - Samsung 970 EVO Plus 2 TB
(4 MB readahead)
21:48
Xeon E-2146G
(Debian buster)
- Samsung 970 EVO Plus 2 TB
(4 MB readahead)
23:05
Xeon E-2146G Core i7-8850H Samsung 970 EVO Plus 2 TB
(4 MB readahead)
25:10
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(CPU PCIe, 4 MB readahead)
25:59
Core i7-8850H - Samsung 970 EVO Plus 2 TB 25:53
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(4 MB readahead)
26:39
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(CPU PCIe)
27:49
Xeon E-2146G
(Debian buster)
- Samsung 970 EVO Plus 2 TB 29:47
Xeon E-2146G Core i7-8850H Samsung 970 EVO Plus 2 TB 29:48
Xeon E-2146G - Samsung 970 EVO Plus 2 TB 31:27
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(LUKS encrypted)
47:17
Xeon E-2146G - WD Elements 10 TB 2:37:20
Xeon E-2146G Core i7-8850H WD Elements 10 TB 2:37:31
Xeon E-2146G AMD GX-412TC SOC WD Elements 10 TB
(4 MB readahead)
2:40:01
Xeon E-2146G AMD GX-412TC SOC WD Elements 10 TB 3:05:30

The segment verification step is I/O bound and takes the majority of the time of the entire operation. The other verification steps taking place after this are mostly CPU bound.

borg check --verify-data

Client Server Disk Time
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(CPU PCIe, 4 MB readahead)
11:19:31
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(4 MB readahead)
11:24:27
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(CPU PCIe)
11:28:59
Xeon E-2146G - Samsung 970 EVO Plus 2 TB 11:41:20
Xeon E-2146G - Samsung 970 EVO Plus 2 TB
(LUKS encrypted)
12:19:14
Xeon E-2146G Core i7-8850H Samsung 970 EVO Plus 2 TB 13:55:40
Xeon E-2146G - WD Elements 10 TB 16:36:56
Xeon E-2146G Core i7-8850H WD Elements 10 TB 16:43:17
Xeon E-2146G AMD GX-412TC SOC WD Elements 10 TB
(4 MB readahead)
27:15:09
Xeon E-2146G AMD GX-412TC SOC WD Elements 10 TB 28:39:23

The data verification stepe is mostly client CPU bound, but I/O latency seems to be important, as well.

Links

BorgBackup homepage
BorgBackup Documentation