Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: conditionalize generic flattening and type casting for streams v1+ #1057

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Conversation

de-sh
Copy link
Contributor

@de-sh de-sh commented Dec 27, 2024

Fixes #XXXX.

Description

  • For streams created after the v1 update, generic flattening will be applied.
  • All numbers will be stored as float64.
  • All string fields with name "date"/"time" that can be parsed as timestamp will be stored as timestamp.

This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

@@ -84,7 +84,7 @@ pub async fn ingest_internal_stream(stream_name: String, body: Bytes) -> Result<
tags: String::default(),
metadata: String::default(),
};
event.into_recordbatch(&schema, None, None)?
event.into_recordbatch(&schema, None, None, SchemaVersion::V0)?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a comment explaining what is V0 vs V1 (and what is the different)

@coveralls
Copy link

coveralls commented Jan 1, 2025

Pull Request Test Coverage Report for Build 12579727646

Details

  • 287 of 417 (68.82%) changed or added relevant lines in 10 files are covered.
  • 8 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.9%) to 12.48%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/handlers/http/ingest.rs 167 168 99.4%
src/storage/mod.rs 0 1 0.0%
src/utils/json/flatten.rs 8 9 88.89%
src/handlers/http/logstream.rs 0 2 0.0%
src/kafka.rs 0 2 0.0%
src/event/format/json.rs 32 35 91.43%
src/utils/json/mod.rs 5 8 62.5%
src/event/format/mod.rs 49 77 63.64%
src/metadata.rs 0 43 0.0%
src/handlers/http/modal/utils/ingest_utils.rs 26 72 36.11%
Files with Coverage Reduction New Missed Lines %
src/handlers/http/modal/utils/ingest_utils.rs 1 30.43%
src/event/format/mod.rs 2 73.04%
src/utils/json/flatten.rs 2 85.09%
src/metadata.rs 3 1.39%
Totals Coverage Status
Change from base Build 12559557684: 0.9%
Covered Lines: 2317
Relevant Lines: 18565

💛 - Coveralls

@de-sh de-sh changed the title feat: conditionalize generic flattening for streams v1+ feat: conditionalize generic flattening and type casting for streams v1+ Jan 1, 2025
@de-sh de-sh requested a review from nitisht January 1, 2025 19:06
Copy link
Contributor

@nikhilsinhaparseable nikhilsinhaparseable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

Copy link
Member

@nitisht nitisht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add detailed comments especially in places we're checking for versions. Also we need comments on what is v0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants