New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write integration tests #1

Open

sahava opened this issue Dec 13, 2018 · 1 comment

Owner

sahava commented Dec 13, 2018

Write an integration test that verifies the payload being written to BQ is valid NDJSON and conforms to BigQuery schema.

jwest75674 commented May 30, 2019

In addition: If too many sites are in the config, then the BQ Upload Jobs fail due to too many concurrent jobs in a short time period.

--> If the table is setup to be partitioned, this increased the limit by a large margin, but does not totally prevent the issue.

Instead, I suspect that if even 1 second can be delayed (sleep) between each process starting, this would likely solve the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment