rdedup

rdedup is the data deduplication engine and backup software

rdedup is written in Rust and provides both command line tool and library API (rdedup-lib).

rdedup is generally similar to existing software like duplicacy, restic, attic, duplicity, zbackup, etc.

Features

• support for public-key encryption (the only tool like that I'm aware of, and primary reason rdedup was created)
• flat-file synchronization friendly (Dropbox, syncthing) backend
  • cloud backends are WIP
• garbage collection
• variety of supported algorithms:
  • chunking: bup, gear
  • hashing: blake2b, sha256
  • compression: deflate, xz2, bzip2, zstd, none
  • encryption: curve25519, none
  • very easy to add new ones
  • check rdedup init --help output for up-to-date list
• extreme performance and parallelism - see Rust fearless concurrency in rdedup
• attention to reliability (eg. rdedup is using fsync + rename to avoid data corruption even in case of hardware crash)

Strong parts

It's written in Rust. It's a modern language, that is actually really nice to use. Rust makes it easy to have a very robust and fast software.

The author is a nice person, welcomes contributions, and helps users. Or at least he's trying... :)

Shortcomings and missing features:

rdedup currently does not implement own backup/restore functionality (own directory traversal), and because of that it's typically paired with tar or rdup tools. Built-in directory traversal could improve deduplication ratio for workloads with many small files.

Garbage collection could be optimized and made more scalable.

Cloud storage integrations are missing. The architecture to support it is mostly implemented, but the actual backends are not.

Installation

If you have cargo installed:

cargo install rdedup

If not, I highly recommend installing rustup (think pip, npm but for Rust)

In case of troubles, check rdedup building issues or report a new one (sorry)!

Usage

See rdedup -h for help.

Supported commands:

• rdedup init - create a repo directory with keypair used for encryption.
• rdedup ls - list all stored names.
• rdedup store <name> - store data read from standard input under given name.
• rdedup load <name> - load data stored under given name and write it on standard output
• rdedup rm <name> - remove the given name. This by itself does not remove the data.
• rdedup gc - remove any no longer reachable data

Check rdedup init --help for repository configuration options.

In combination with rdup this can be used to store and restore your backup like this:

rdup -x /dev/null "$HOME" | rdedup store home
rdedup load home | rdup-up "$HOME.restored"

rdedup is data agnostic, so formats like tar, cpio and other will work, but to get benefits of deduplication, archive format should not be compressed or encrypted already.

RDEDUP_PASSPHRASE environment variable

While it's not advised, if RDEDUP_PASSPHRASE is defined, it will be used instead of interactively asking user for password.

License

rdedup is licensed under: MPL-2.0