Stored data de-duplication

[cayolblake]

Hello,

I wonder if data de-duplication is on your roadmap.

Basically, if an attempt to store the same file multiple times would first check if the file chunks already exists somewhere (by using some sort of fast hashing algorithm) and if it previously existed, it would just point to the already existing file without consuming additional space and effort.

[xiaogaozi]

It's an interesting idea. We will have a discussion internally. If you have enough time, contributions are very welcome. 😀

[davies]

De-duplication is hot topic in storage, and useful for some cases, may not justify for the high cost as a general feature, so we don't put that in the roadmap.

JuiceFS does provide some features to dedup the data at application level, for example, hardlink and faster CopyFileRange (without coping the data).

[xyb]

I'm researching how to implement out-of-band deduplication using copy_file_range like btrfs and duperemove. I want to understand the level of support offered by JuiceFS. Does it only work with block boundaries and if so, what is the block size? Alternatively, does JuiceFS support flexible block size similar to the rolling hash generated?

[davies]

There is no limit on CopyFileRange, it work with any length from anywhere.

Ideally, you should take aware of chunk boundary (64MiB), which means two files share the same 64MiB. Otherwise, A file may refer to partial of a slice in another file. When there are too many slices (> 5), it may trigger compaction to duplicate the data.

[xyb]

@davies Thank you for the information. I believe it would be valuable to include this in the JuiceFS documentation.

[davies]

We have update this part recently: https://juicefs.com/docs/zh/community/architecture#how-juicefs-store-files