Querying distance based on IDs

[elliotsayes]

Looking through the documentation, the query DSL allows searching for entities based on vector distance.
Is there any functionality for returning the distance between the vectors of particular known IDs? Or, am I expected retrieve their vectors by id (i.e. get_entity_by_id), then calculate their similarity outside of milvus?

Extension of this idea: being able to specify a list of entities (e.g. A, B, C, D) and retrieve the distance between each set of two entities (e.g. A->B, A->C, A->D, B->C, B->D, C->D).

I understand that milvus is intended to fill a niche like elastic search, but it would be convenient for those who want to use milvus as their all-purpose vector store and calculation engine.

[elliotsayes]

workaround:

{
    "bool": {
        "must": [
            {
                "term": {
                    "id_term": 1  # entity B id
                }
            },
            {
                "vector": {
                    "my_vector": {
                        "topk": 1,
                        "query": [[1, 2, 3]],  # retrieved from entity A
                    }
                }
            }
        ]
    }
}

id does not work as a term, so must be duplicated as a term unique to the entity (id_term)
requires first fetching vector data from A, before plugging it into this query

[shiyu22]

Hi, this idea looks interesting, but currently Milvus can only use get_entity_by_id to calculate the similarity of id with other vectors.

The requirement you mentioned is for a two-by-two comparison between feature vectors, can you share the real scenario that needs to be used for this requirement?

[elliotsayes]

The use case I had in mind was generating a 2D/3D visualization of the distance between a subset of vectors in the database. Something like a "mind map".

Some other possible use cases: from a set of university exam or assignment submissions, finding those that are most similar (detecting plagiarism). Also, for searching for duplicate items (which will have the most similar vectors). For searching about what are the most unique items in the database (perhaps finding images that have few similar copies, i.e. only 2 photos of toasters when you should have more). Or for finding unique topics of documents that have not been explored much.

These can all be done on a one by one basis, by querying the individual vectors from the DB and then searching, but this is a convoluted process. In addition, perhaps there are some optimizations for batch searching?

I hope some of these ideas make sense to you, if there is a better way of doing it then please let me know !

[shiyu22]

Thank you for sharing, the scenario you said looks interesting.

In summary, there are two features here:

1. get_distance_by_ids(), which allows you to pass in two sets of ids and then compare the distance between them.
2. get_similar_list(), which can find the most similar list in the specified collection, and as you said return the toasters of all images.

Am I talking about the features you expect? If it is, can you submit a feature issue for Milvus?

[elliotsayes]

Yes those make perfect sense to me, I think we are on the same page. If it sounds like something generally useful, then I'll put together a proposal for them... maybe starting with get_distance_by_ids(), as that would be a bit easier to implement. Thanks!

[shiyu22]

Hi, I have just submitted issue4585 to get distance by ids and issue4586 to getting a similar list, you can follow them for the feature.