Synthetic data generates on demand rather than on schedule #63
Conversation
Redis constants enhanced execution_cycle updated to enable warmup stage.
There was a problem hiding this comment.
it's really hard to review this PR because a massive amount of code has been refactored which (a) did not need to be, and (b) is not relevant to the changes we're tryng to make here
I'm also not sure what the PR description means, or how this solution solves the problem. I believe it changes functionality of the whole system too much
| @@ -43,25 +44,25 @@ async def _post_vali_stats(config: Config): | |||
| ) | |||
|
|
|||
|
|
|||
| async def get_nodes_and_contenders(config: Config) -> list[Contender] | None: | |||
| async def get_nodes_and_contenders(config: Config) -> List[Contender] | None: | |||
There was a problem hiding this comment.
Why refactor typehints? Prefer the lowercase
| logger.info("Starting cycle...") | ||
| if config.refresh_nodes: | ||
| logger.info("First refreshing metagraph and storing the nodes") |
There was a problem hiding this comment.
Why is all this refactored?
| @@ -70,29 +71,25 @@ async def main(config: Config) -> None: | |||
| time_to_sleep_if_no_contenders = 20 | |||
| contenders = await get_nodes_and_contenders(config) | |||
|
|
|||
| # date_to_delete = datetime(2024, 10, 13, 30) | |||
| # await delete_task_data_older_than_date(connection, date_to_delete) | ||
| # await delete_contender_history_older_than(connection, date_to_delete) | ||
| # await delete_reward_data_older_than(connection, date_to_delete) | ||
| await warmup_function(config=config) |
There was a problem hiding this comment.
'warmup_function' is a bad name -> what is it doing?
| remaining_requests: int | ||
|
|
||
| def __lt__(self, other: "TaskScheduleInfo"): |
| break | ||
|
|
||
| if i % 100 == 0: |
There was a problem hiding this comment.
Why has all this logic gone? We've completely changed how we schedule synthetic queries now right? All this logic must remain
PR Overview
TLDR
This PR planned to provide on demand synthetic data calculation, where on demands stands for in addition to request received.
Implementation details
This PR consists from two parts:
Main part of PR
Main part consists of the following changes:
refresh_sythetic_data.pyhas been (well it actually wasn't completely but supposed to be) rewritten to store nothing but a cache warmup function.schedule_synthetic_queries.pyis the main place where the most changes have been made_fetch_tasks_in_chunksfunction reads a chunk of new organic tasks to proceed, if any._enhance_taskfunction enhances the initial raw task with the unique data. Currently implemented just forchatandtext-to-imagetasks, because of it's nothing more but a switch by task kind it can be expanded effortlessly._push_to_query_nodefunction pushes enhanced tasks to aquery_nodewarmup_functionfunction implements logic of a service warmup in sake of code intents clarificationschedule_synthetics_until_donefunction has been rewritten to reflect the new approach — enhance any given tasks amount with synthetic dataexecution_cycle.py—warmup_functioncall added on a service start.Work around part of PR
cli.pycli tool that allows developer to manually send arbitrary amount of tasks of any kind to thecontrol_nodeto check the whole-> contol_node -> query_nodepipeline.redis_constants.py—CONTROL_NODE_QUEUE_KEYandCONTROL_TASK_CHUNK_SIZEconstants have been added to be used incli.pyand in the queue that is modifiedschedule_synthetics_until_donefunction reads raw tasks from.Closes: #55