[Bug]: client.describe_index shows zero indexed rows despite data being present and ANN search working

[Liqs-v2]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- PyMilvus version: 2.5.0
- Milvus-Lite Version: 2.4.10
- Deployment mode(standalone or cluster): Local DB
- MQ type(rocksmq, pulsar or kafka):
- OS(Ubuntu or CentOS): MacOS 15.1.1 and Windows 11 24H2
- CPU/Memory:
  - Mac: M3 Max, 64GB
  - Windows: Ryzen 7 7735U, 16GB
- GPU: Integrated for both
- Others:

Current Behavior

The Problem

I am trying to use Milvus as DB for a RAG system in a research project related to SWE-bench and AI Agents. My current setup is as follows:

  client = MilvusClient('data/task_embeddings.db')
  schema = MilvusClient.create_schema(
      auto_id=False,
  )

  schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, description="primary id")
  schema.add_field(field_name="instance_id", datatype=DataType.VARCHAR, max_length=512,
                   description="instance id name")
  schema.add_field(field_name="trajectory", datatype=DataType.VARCHAR, max_length=65535,
                   description="agent trajectory")
  schema.add_field(field_name="problem_statement", datatype=DataType.VARCHAR, max_length=65535,
                   description="github issue")
  schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=768,
                   description="problem statement embedding vector")

  # Create the collection
  client.create_collection(
      collection_name="swe_bench_lite",
      dimensions=index_dimensions,
      schema=schema
  )
  index_params = MilvusClient.prepare_index_params()
  index_params.add_index(
      field_name="vector",
      metric_type="COSINE",
      index_type="FLAT",
      index_name="vector_index",
  )

  # Create index for first collection
  client.create_index(
      collection_name="swe_bench_lite",
      index_params=index_params,
      sync=True
  )

Then I compute embeddings for the problem statements in SWE-bench and insert the data as follows:

    client.insert(collection_name='swe_bench_lite', data=[dict(row) for row in swe_bench_lite])

When then running client.describe_index(collection_name='swe_bench_lite', index_name='vector_index') I get the following result:

{'dim': '768', 'field_name': 'vector', 'index_name': 'vector_index', 'index_type': 'FLAT', 'indexed_rows': 0, 'metric_type': 'COSINE', 'pending_index_rows': 0, 'state': 'Finished', 'total_rows': 0}

Indicating that no rows were indexed. I have verified that rows were inserted as expected.

Expected Behavior

I expected client.describe_index to correctly show how many rows are indexed or the search to fail or at least give a warning without an indexation.

Steps To Reproduce

from pymilvus import MilvusClient, DataType, FieldSchema, CollectionSchema, Collection

client = MilvusClient('data/demo.db')
schema = MilvusClient.create_schema(
    auto_id=False,
)

index_dimensions = 2
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, description="primary id")
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=index_dimensions,
                 description="embedding vector")

# Create the collection
client.create_collection(
    collection_name="foo",
    dimensions=index_dimensions,
    schema=schema
)

index_params = MilvusClient.prepare_index_params()
index_params.add_index(
    field_name="vector",
    metric_type="COSINE",
    index_type="FLAT",
    index_name="vector_index",
)

# Create index for first collection
client.create_index(
    collection_name="foo",
    index_params=index_params,
    sync=True
)

data = [
    {
        "id": 1,
        "vector": [0.1, 0.2]
    },
    {
        "id": 2,
        "vector": [0.2, 0.3]
    },
    {
        "id": 3,
        "vector": [0.3, 0.4]
    }
]

client.insert(
    collection_name="foo",
    data=data
)

# Check that the data were correctly inserted
client.get(
    collection_name="foo",
    ids=[1, 2, 3])

# Were the data indexed?
client.describe_index(collection_name='foo', index_name='vector_index') 
# OUTPUT: {'dim': '2', 'field_name': 'vector', 'index_name': 'vector_index', 'index_type': 'FLAT', 'indexed_rows': 0, 'metric_type': 'COSINE', 'pending_index_rows': 0, 'state': 'Finished', 'total_rows': 0}

client.search(collection_name='foo', data=[[0.6,0.1]])
# OUTPUT: data: ["[{'id': 3, 'distance': 0.7233555912971497, 'entity': {}}, {'id': 2, 'distance': 0.683941125869751, 'entity': {}}, {'id': 1, 'distance': 0.5881717205047607, 'entity': {}}]"] 

### Milvus Log

_No response_

### Anything else?

_No response_

[akshatvishu]

I think describe_index show zero row indexed because FLAT index is a brute-force index which does not require index building or provide metrics like “indexed rows.” It’s essentially a baseline configuration that relies on scanning all vectors directly, resulting in 100% recall without any offline indexing step.

[yanliang567]

You did not see the inserted rows in describe_index() because the rows are not flushed onto disk. Usually Milvus will automatically do flush() in a few seconds, you can also manually do client.flush(). But please not manually do flush() frequently, since it is a heavy io operation to the system.

/assign @Liqs-v2
/unassign

[Liqs-v2]

I think describe_index show zero row indexed because FLAT index is a brute-force index which does not require index building or provide metrics like “indexed rows.” It’s essentially a baseline configuration that relies on scanning all vectors directly, resulting in 100% recall without any offline indexing step.

That makes sense to me, but I still think from a UX perspective this is not ideal. To me it seemed like there might be something broken and I lost some degree of trust in the integrity of the results.

[Liqs-v2]

I suppose a way forward would be to at least document this behaviour? I can do this later.

[yanliang567]

Please feel free to create a new doc for this

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.