New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

store tiny files inline in the index #154

Open

sourcefrog opened this issue Oct 27, 2020 · 0 comments

Labels

topic:performance type:format-change

Owner

sourcefrog commented Oct 27, 2020

Files under about 100 bytes in length could be better stored directly in the index:

the file content is potentially smaller than a hash referring to its block
it avoids a separate IO for a data block when reading and writing, and probably still helps even after Combine multiple small files into single blocks #66

There is a complication that the 0.6 index format is encoded in json, which can't directly include binary content. Options here seem to be:

Only inline files whose content is a UTF-8 string. Many small files
base-64 escape the content (not very nice)
Wait to transition to a non-json index encoding, perhaps CBOR

sourcefrog added type:format-change topic:performance labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment