Postgres backend parquet problem

[ozgurkalan]

Hi,
When I export any table to parquet, e.g. sample.gzip, and let's say assigned to a variable sample;

sample = ibis.read_parquet("sample.gzip")

I cannot create a new table with sample and have to convert to pandas first:

pg = ibis.postgres.connect(...)
pg.create_table("sample", sample.to_pandas())

also, when I try attaching a parquet file to above pg Backend, ibis complains...

pg.read_parquet("sample.gzip")

>>> NotImplementedError: postgres does not support direct registration of parquet data.

What am I doing wrong?

[gforsyth]

Hi @ozgurkalan !

sample = ibis.read_parquet("sample.gzip")

without an explicit backend set up, this operation happens in DuckDB, so your sample table is a DuckDB table with the contents of your sample.gzip file -- you cannot (currently) arbitrarily pass data from one backend to another, which is why you have to convert it to an intermediate format first (I would suggest using to_pyarrow() instead).

Postgres doesn't have any way to read data from a parquet file, hence the error message you are hitting when you try to call read_parquet on the postgres connection.

[ozgurkalan]

Thank you @gforsyth
I also have another wierd parquet issue, which I will write in a separate post.