docs(blog): Add Velox Primer Part1 Post #12348
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Adding part of a series of blog posts introducing Velox concepts. Differential Revision: D69694896
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
This pull request was exported from Phabricator. Differential Revision: D69694896 |
mapleFU
reviewed
Feb 15, 2025
| from which the group by reads its input. There are file splits | ||
| (`velox::connector::ConnectorSplit`) and remote splits | ||
| (`velox::exec::RemoteSplit`). The first identifies data to read, the second | ||
| identifies a running Task. |
There was a problem hiding this comment.
May I ask a naive question, so the RemoteSplit is only for input "task" which would used by exchange?
There was a problem hiding this comment.
Yes, you would only add this specific type of split if the task in hand is supposed to read data from a shuffle. If you're reading it directly from a file (table scan) you would use a regular ConnectorSplit.
| somewhere for a consumer to retrieve. This is typically a PartitionedOutput. | ||
| The consumer of the PartitionedOutput is an Exchange in a different task, where | ||
| the Exchange is in the source position. Operators that are neither sources or | ||
| sinks are things like filterProject, HashProbe, HashAggregation and so on, more |
There was a problem hiding this comment.
Suggested change
| sinks are things like filterProject, HashProbe, HashAggregation and so on, more | |
| sinks are things like FilterProject, HashProbe, HashAggregation and so on, more |
Keep same for FilterProject?
mapleFU
reviewed
Feb 17, 2025
| the second stage Tasks (group by), their Splits identify the table scan Tasks | ||
| from which the group by reads its input. There are file splits | ||
| (`velox::connector::ConnectorSplit`) and remote splits | ||
| (`velox::exec::RemoteSplit`). The first identifies data to read, the second |
There was a problem hiding this comment.
It's RemoteConnectorSplit in code?
kKPulla
reviewed
Feb 19, 2025
| An example of a Task with two pipelines is a hash join, with separate pipelines | ||
| for the build and for the probe side. This makes sense because the build must | ||
| be complete before the probe can proceed. We will talk more about this later. | ||
| All vectors are subclasses of `velox::BaseVector`. |
There was a problem hiding this comment.
I think we should move this to next paragraph after introducing vectors as a concept.
| child vector for every column of the relation - it is the equivalent of | ||
| RecordBatch in Arrow. | ||
|
|
||
| ## Operators Souces, Sinks, and State |
There was a problem hiding this comment.
Suggested change
| ## Operators Souces, Sinks, and State | |
| ## Operator Sources, Sinks and State |
| Tasks. Tasks send back statistics, errors and other status information to the | ||
| distributed engine. | ||
|
|
||
| ## Pipelines and Drivers and Operators |
There was a problem hiding this comment.
Suggested change
| ## Pipelines and Drivers and Operators | |
| ## Pipelines, Drivers and Operators |
| Inside a Task, there are *Pipelines*. Each pipeline is a linear sequence of | ||
| operators (`velox::exec::Operator`), and operators are the objects that implement | ||
| relational logic. In the case of the group by example, the first task has one | ||
| pipeline, with a TableScan and a PartitionedOutput. The second Task too has one |
There was a problem hiding this comment.
link these two to code ref? Like velox::exec::TableScan etc.?
Comment on lines
+11
to
+12
| stages, and present Velox concepts such as Tasks, Splits, Pipelines, Drivers, | ||
| and Operators. |
There was a problem hiding this comment.
Probably?
Suggested change
| stages, and present Velox concepts such as Tasks, Splits, Pipelines, Drivers, | |
| and Operators. | |
| stages, and present Velox concepts such as Tasks, Splits, Pipelines, Drivers, | |
| and Operators that enable this in distributed compute engines. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Adding part of a series of blog posts introducing Velox concepts.
Differential Revision: D69694896