Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

writerFooter is not supported for ORC file format #12418

Open
rui-mo opened this issue Feb 21, 2025 · 2 comments
Open

writerFooter is not supported for ORC file format #12418

rui-mo opened this issue Feb 21, 2025 · 2 comments
Labels
enhancement New feature or request

Comments

@rui-mo
Copy link
Collaborator

rui-mo commented Feb 21, 2025

Description

ORC reader needs to parse footer as photo::orc::Footer.

if (fileFormat() == FileFormat::DWRF) {
footer_ = parseFooter<proto::Footer>(decompressed.get(), arena_.get());
} else {
footer_ = parseFooter<proto::orc::Footer>(decompressed.get(), arena_.get());
}

Velox now only supports writing footer as the DWRF proto::Footer.

void WriterBase::writeFooter(const Type& type) {

When testing the ORC decimal read and write, we find that the ORC reader parses the precision and scale information from footer while the DWRF footer does not contain them. Long decimal type needs to be written as hugeint kind in the ORC footer while the DWRF format does not contain hugeint type.

@majetideepak
Copy link
Collaborator

@rui-mo We need to create/extend the FooterWrapper for the write path similar to the read path.
Orc Format must be written into proto::orc::Footer

@pedroerp
Copy link
Contributor

Cc: @Yuhta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants