Allow efficient caching on filesystems without sparse file support?

[groutr]

The CachingFileSystem uses MMapCache to cache parts of files. This works beautifully on filesystems that support sparse files to only store the parts of the file that have been requested. However, on filesystems without sparse file support, mmap does not allow for any efficient use of disk space.

Could the CachingFileSystem use a cache like KnownPartsOfAFile or Blockcache to cache data on filesystems without sparse file support to allow for more efficient use of disk space?

My use case is caching on a FAT32/NTFS filesystem.

[martindurant]

You mean, to have a separate file per chunk or some kind of file index? It could of course be done, but would require some work.

[groutr]

Sorry I should have explained that part a little better. I was envisioning a file storing all of its component chunks. CachingFileSystem already has all the logic from what I can see. If CachingFileSystem was able to switch between BlockCache or MMapCache depending on user preference or filesystem, that should do the trick.

However, BlockCache being based on lru_cache makes the underlying cache inaccessible to the user. Perhaps BlockCache could be modified to use any key/value storage (ie a dictionary, LRUCache, basically anything that behaves like a MutableMapping in python). That would allow CachingFileSystem to more easily manage the cached blocks on the filesystem.

[martindurant]

It sounds like maybe you have a plan for implementation? I would be happy to see it! :)