Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): Add support for geoparquet reading #247

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat(cli): Add support for geoparquet reading #247

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bd5cab4
feat(cli): Add support for geoparquet reading
alekzvik Dec 20, 2024
c4fedff
fix: move geoarrow to CLI crate dep and use latest stable
alekzvik Dec 20, 2024
7471d1a
fix: update geoarrow and add more help.
alekzvik Dec 20, 2024
1e3b2e1
fix: remove long help messages
alekzvik Dec 20, 2024
9535a6e
fix: wording in input param help
alekzvik Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ gdal = { version = "0.16", default-features = false }
gdal-sys = "0.9"
geo = "0.26.0"
geo-types = { version = "0.7.11", default-features = false }
geoarrow = { version = "0.3.0" }
alekzvik marked this conversation as resolved.
Show resolved Hide resolved
geojson = { version = "0.24.1", default-features = false }
geos = "9.0"
gpx = { version = "0.9", default-features = false }
Expand Down
2 changes: 2 additions & 0 deletions geozero-cli/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,5 @@ env_logger.workspace = true
flatgeobuf.workspace = true
geozero = { workspace = true, default-features = true, features = ["with-csv"] }
tokio = { workspace = true, default-features = true, features = ["full"] }
geo.workspace = true
geoarrow = { version = "0.4.0-beta.2", features = ["parquet_compression"] }
33 changes: 32 additions & 1 deletion geozero-cli/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
use clap::Parser;
use flatgeobuf::{FgbReader, FgbWriter, GeometryType, HttpFgbReader};
use geo::Rect;
use geoarrow::io::parquet::{GeoParquetReaderOptions, GeoParquetRecordBatchReaderBuilder};
use geoarrow::io::RecordBatchReader;
use geozero::csv::{CsvReader, CsvWriter};
use geozero::error::{GeozeroError, Result};
use geozero::geojson::{GeoJsonLineReader, GeoJsonReader, GeoJsonWriter};
Expand All @@ -24,7 +27,7 @@ struct Cli {
#[arg(short, long, value_parser = parse_extent)]
extent: Option<Extent>,

/// The path or URL to the FlatGeobuf file to read
/// The path or URL to the input file to read
alekzvik marked this conversation as resolved.
Show resolved Hide resolved
input: String,

/// The path to the file to write
Expand Down Expand Up @@ -88,6 +91,26 @@ async fn transform<P: FeatureProcessor>(args: Cli, processor: &mut P) -> Result<
Some("jsonl") | Some("geojsonl") => {
GeozeroDatasource::process(&mut GeoJsonLineReader::new(filein), processor)
}
Some("parquet") | Some("geoparquet") => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The community discourages against .geoparquet as a file extension, but I figure it's fine to accept files with the suffix.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not have a strong opinion here, and tried to be consistent with json/geojson and also with stacrs CLI which also supported both.

let mut geo_options = GeoParquetReaderOptions::default();
if let Some(bbox) = &args.extent {
geo_options = geo_options.with_bbox(
Rect::new((bbox.minx, bbox.miny), (bbox.maxx, bbox.maxy)),
None,
);
}
let reader = GeoParquetRecordBatchReaderBuilder::try_new_with_options(
File::open(path_in)?,
Default::default(),
geo_options,
)
.map_err(arrow_to_geozero_err)?
.build()
.map_err(arrow_to_geozero_err)?;

let mut wrapper = RecordBatchReader::new(Box::new(reader));
wrapper.process(processor)
}
Some("fgb") => {
let ds = FgbReader::open(&mut filein).map_err(fgb_to_geozero_err)?;
let mut ds = if let Some(bbox) = &args.extent {
Expand Down Expand Up @@ -127,6 +150,7 @@ async fn process(args: Cli) -> Result<()> {
}
Ok(())
}

fn set_dimensions(processor: &mut SvgWriter<&mut BufWriter<File>>, extent: Option<Extent>) {
if let Some(extent) = extent {
processor.set_dimensions(extent.minx, extent.miny, extent.maxx, extent.maxy, 800, 600);
Expand All @@ -136,6 +160,13 @@ fn set_dimensions(processor: &mut SvgWriter<&mut BufWriter<File>>, extent: Optio
}
}

fn arrow_to_geozero_err(parquet_err: geoarrow::error::GeoArrowError) -> GeozeroError {
match parquet_err {
geoarrow::error::GeoArrowError::IOError(e) => GeozeroError::IoError(e),
err => GeozeroError::Dataset(format!("Unknown GeoArrow error: {err:?}")),
}
}

fn fgb_to_geozero_err(fgb_err: flatgeobuf::Error) -> GeozeroError {
match fgb_err {
flatgeobuf::Error::MissingMagicBytes => {
Expand Down