Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MB-60029: Accommodate vector_index_optimized_for into vector field mapping #1959

Merged
merged 5 commits into from
Jan 17, 2024

Conversation

metonymic-smokey
Copy link
Member

@metonymic-smokey metonymic-smokey commented Jan 9, 2024

This PR adds the functionality needed to introduce "vector_index_optimized_for" into the vector field mapping to allow each vector fields' index to be optimised for latency/recall.

mapping/field.go Outdated Show resolved Hide resolved
Copy link
Member

@abhinavdangeti abhinavdangeti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the draft tags so github workflows can run.

mapping/field.go Outdated Show resolved Hide resolved
document/field_vector.go Outdated Show resolved Hide resolved
document/field_vector.go Outdated Show resolved Hide resolved
mapping/field.go Outdated Show resolved Hide resolved
document/field_vector.go Outdated Show resolved Hide resolved
document/field_vector.go Outdated Show resolved Hide resolved
mapping/mapping_vectors.go Outdated Show resolved Hide resolved
mapping/mapping_vectors.go Outdated Show resolved Hide resolved
@@ -202,6 +208,12 @@ func validateVectorFieldAlias(field *FieldMapping, parentName string,
reflect.ValueOf(index.SupportedSimilarityMetrics).MapKeys())
}

if _, ok := index.SupportedVectorIndexOptimizations[field.VectorIndexOptimizedFor]; !ok {
return fmt.Errorf("field: '%s', invalid optimization "+
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not error out, with future in mind if we add more options.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought updating index.SupportedVectorIndexOptimizations in bleve_index_api and its version in bleve's go.mod would take care of this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean!? I'm talking about the case in the future - when we add more optimizations and there's a mixed cluster scenario where some nodes won't support it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right.
Removed this snippet so there's no check at this point for valid optimisation metrics.

mapping/mapping_vectors.go Show resolved Hide resolved
@abhinavdangeti abhinavdangeti changed the title MB-60029 - Added vector index type into vector field mapping MB-60029 - Added vector index optimized_for into vector field mapping Jan 17, 2024
@abhinavdangeti abhinavdangeti changed the title MB-60029 - Added vector index optimized_for into vector field mapping MB-60029 - Added vector_index_optimized_for into vector field mapping Jan 17, 2024
@abhinavdangeti abhinavdangeti changed the title MB-60029 - Added vector_index_optimized_for into vector field mapping MB-60029: Accommodate vector_index_optimized_for into vector field mapping Jan 17, 2024
@abhinavdangeti abhinavdangeti merged commit d11e844 into unstable Jan 17, 2024
9 checks passed
@abhinavdangeti abhinavdangeti deleted the index-type branch January 17, 2024 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants