Transform PSQL2BigQuery into a CLI #9
Conversation
joaodaher
force-pushed
the
feature/cli
branch
from
c76d2be to
4c72a9a
Compare
It does not behave as expected: since we want to allow running with env vars, the prompt will always ask for input
The auto-detect can cause issues when the sample is not good enough. We've got issues with columns that most of records are numbers, but some records have alpha characters, causing the load to fail.
Now that the import is more reliable, we can be more strict and do not allow any record to be discarded
| @@ -1 +1,12 @@ | |||
| psql2bigquery | |||
| # PostgreSQL to BigQuery | |||
There was a problem hiding this comment.
It's really good add a readme to project =D
What do you think about add a how-to run local to help developers contribute to the project
There was a problem hiding this comment.
Sure, I added a new commit with a Contributing section.
Please tell me if there's anything I could add.
There was a problem hiding this comment.
Hey @joaodaher!
I'd be great to have more usage instructions and examples with the available params. Can you add that as well please?
It could also be in a different issue/PR, no problem. Thanks!
There was a problem hiding this comment.
A note that could be added to the instructions: --include and --exclude are mutually exclusive, so they can't be used together.
And here's the syntax for multiple tables (worth mentioning in the readme or command help):
poetry run psql2bigquery run --db-host localhost \
--db-port 5430 \
--db-user whatsgood \
--db-password xx \
--db-name whatsgood \
--gcp-project whatsgood-dev \
--gcp-dataset api \
--include membership_membership \
--include membership_membershipprogram \
--gcp-credential-path /Users/tferreira/project/psql2bigquery/whatsgood-dev.json
There was a problem hiding this comment.
I added a basic usage example for now.
We can improve later with a proper documentation of each parameter.
There was a problem hiding this comment.
Looks good @joaodaher, thanks!
There was a problem hiding this comment.
@joaodaher, something is not quite right with the logs, can you look into, please? Nothing is being printed to stdout 😱
(but the table is being sent to bigquery correctly ✅ looks like an issue just in the logs)
Here is my command:
poetry run psql2bigquery run --db-host localhost \
--db-port 5430 \
--db-user whatsgood \
--db-password xx \
--db-name whatsgood \
--gcp-project whatsgood-dev \
--gcp-dataset api \
--include crm_crmbackend \
--gcp-credential-path /Users/tferreira/project/psql2bigquery/whatsgood-dev.json
|
@thiagoferreiraw , it was missing the log level for stdout. I added in a new commit. |
Description
Fixes #5 #4