Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve ParallelTaskHandler to share TaskClient between TaskRunners #14

Closed
gardusig opened this issue Feb 4, 2022 · 3 comments
Closed

Improve ParallelTaskHandler to share TaskClient between TaskRunners #14

gardusig opened this issue Feb 4, 2022 · 3 comments

Comments

@gardusig
Copy link
Contributor

gardusig commented Feb 4, 2022 •
edited
Loading

Currently, TaskHandler spawn a bunch of processes, capable of running tasks in parallel.

Each of these TaskRunners instantiate an ApiClient at each request, due to pickle issues.

Research required to understand alternatives.
@gardusig gardusig self-assigned this Feb 4, 2022
@gardusig
Copy link
Contributor Author

One possible solution:
Instantiate ApiClient at TaskHandler and share between TaskRunners, by overriding pickle.

@gardusig
Copy link
Contributor Author

Removed threadpool from ApiClient, still remaining to remove threadpool from urllib3 at RestClient

@gardusig gardusig removed their assignment May 4, 2022
@gardusig
Copy link
Contributor Author

Issue will be continued here: #97

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gardusig