DAR over tar for dedublication targets?

[stilsch]

Hi!
I have a deduplication appliance (Cohesity) to which I want to write thousands of similar backups of operating system and applications.
I came across this paper, which states that TAR might not be a good choice since the standard TAR structure is not easily deduplicated if changes are made to the source. ChatGPT pointed out that DAR might be a better choice since it addresses this issue more effectively.

However, while reading through DAR's extensive manuals, I could not find information on this, nor any specific options to set (apart from not compressing, which is logical).

So, I have some questions:

• Is DAR better suited to dedupe at the target than TAR?
• If so, which options should be set and which should be avoided?

Thank you very much!

[Edrusb]

Hi,

Dar backups have not been designed to be de-duplicated from the 'outside' more than tar ones.

The difference is when compression is used with tar the whole backup is compressed at once. This means that the next time a backup is done, if just one files is modified somewhere, the whole compressed stream will be completely different after that point. Thus de-duplication between two different tar backups is not likely to be good.

If also a same file is found multiple time in a backup, there are much chances that the compressed bytes results will not be the same, because the file bytes occurrence will be compressed with different values located before and after it, not mentioning the offset in the stream of bytes to compress that might lead to different compressed data... I guess the article you referred to (which I have not had the time to read) explains that, and very probably in much detailed and precised terms.

For dar the compression is performed per file, thus if in a given backup you have several occurrences of the same file content, they will be compressed the same and the result will be the same compressed stream of bytes located a different offsets in the dar backup.

This also means that de-duplication is possible between several dar backups where the same file is found in each backup (eventually multiple time), assuming the fact the compression algorithm and compression level are the same among backups.

Another difference between tar and dar is the fact that dar does not compress small files nor the files you decided not to be compressed in the backup (like already compressed files or of known poorly compressible data type). These bytes are stored as is. Well this is not completely true if sparse file mechanism is in use: each hole will be removed from the data stream and replaced by a small structure indicating the hole size. But with sparse file detection enabled, several occurrences of a given file's data will be stored the same, eventually the same as without sparse file detection if no more than 15 zeroed bytes in sequence can be found in the data stream.

The metadata around the data is usually very small (filename, permission, dates, ownership...) and not compressed (Well, I should double check the code) but protected by a CRC stored at then end of the metadata. Depending on the granularity of the de-duplication engine, this might provide some gain if hundred or tens of thousand files are stored with the same UID, GID, and/or permission...

Last without any compression, I'm pretty sure tar and dar backups would give the same result in term of de-duplication, which would be more related to the nature of the data backed up than the archive format itself.

For your second question:

If using compression or sparse-file detection, using the same parameters (compression algo, compression level, sparse-file min hole size...) would give better de-duplicated ratio, obviously. Also the same compression mask (which file to compress and which file not to compress, and minimum size of file not to compress) should also be chosen the same among backups.

Depending on the de-duplication engine, slicing could slightly reduce the de-duplication ratio as it inserts a slice headers (~ 100 bytes) and splits the compressed/uncompressed data stream into different files. If slice size are big (several gigas) it should marginally impact de-duplication, I think (to be tested!).

But of course you would not use encryption with dar to not totally obliterate the possibility de-duplicate the data.

[stilsch]

Hey Edrusb,
thank you for your detailed answer. Much appreciated!

We don't plan to compress on the client site since the target perfoms dedup -> compression -> encryption (at disk level) where we cannot disable each of them.
Therefore, performing these three processes on the client site would be pointless.

As I understand it, deduplication algorithms rely on chunks appearing consistently across multiple backups. If a file at the beginning of the archive is altered, the chunk boundaries may change, causing the subsequent data to appear different (to older archives).
If I understand you correctly, for this use case (without encryption, not using sparse files), DAR would most likely produce the same results (with view on dedup) as TAR, since the movement of boundaries can also occur.

[Edrusb]

I remember some de-duplication algorithms I had been described by Riverbed more than a decade ago, and it was something much clever than comparing blocks at given offset. AFAIR, there was a dictionary of byte-strings, the dedup process was to find and replace the longest possible string (of arbitrary size and offset in the data flow) by a label, and it was also recursive, a sequence of labels (or a mix of label+strings) could become a string replaced by a single label... Then, from the stream of labels + dictionary, it could then be possible to re-hydrate the data in its original byte stream.

I don't know what algorithm Cohesity uses, but I'm pretty sure that without encryption nor compression, you will not get much difference between dar and tar: the result will be good if this is algorithm is clever, poor it not.

But as always, you have theory in one hand and the experience in the other... you would not climb a tree with a single hand, would you? ;)

[stilsch]

Thank you for the explanation! I'll give that as a test project to my students. :-)

[Edrusb]

If experience diverges from theory, feel free to add a note here...