Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query sets with one query do not work #64

Open
wrigleyDan opened this issue Dec 17, 2024 · 0 comments
Open

Query sets with one query do not work #64

wrigleyDan opened this issue Dec 17, 2024 · 0 comments

Comments

@wrigleyDan
Copy link
Collaborator

wrigleyDan commented Dec 17, 2024

When running the evaluation for a query set containing one query only no metrics are calculated and stored in the metrics index. As soon as a second query is in the query set the metrics are calculated and stored in the metrics index.
This appears to be independent from the judgements available in the judgments index.

Example for creating a query set with only one query:

POST search_quality_eval_query_sets/_doc/1
{
  "sampling": "none",
  "name": "30 inch ceiling fan without light",
  "description": "only 30 inch ceiling fan without light",
  "queries": [
    {
      "30 inch ceiling fan without light": 11
    }
  ],
  "timestamp": "2024-12-12T11:30:20.916Z"
}

The log output indicates that no lookups happen in the judgments index as no frog number (ratio of missing judgements) is reported.

Update

It appears that for a query set with two queries, only metrics for one query are calculated, and for a query set of three only two queries do get metrics calculated and indexed.
The issue may lie in the handling of query sets in general. Query sets of one query seem to only be a special case of these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant