Ability to override stemming in queries

[bnewbold]

It would be good to be able to override stemming at query time. Stemming by default probably makes the most sense, even for phrase queries, but in some cases we would want to return only exact matches.

For example, "dancing" matches "dance". We could have an alternative field or flag so it only matches "dancing".

In terms of implementation, this could definitely be done by including an extra non-stemmed copy of all fields. For example: everything.exact:"dancing".

Related: case-sensitivity, projecting down diacritics (et al), and off-by-N fuzzy typo tolerance.

A blog post noting this issue: https://jurnsearch.wordpress.com/2020/09/28/internet-archive-scholar-is-live/

[bnewbold]

This came up again via a conversation on mastodon: https://mastodon.archive.org/web/@[email protected]/109426257900886328

Which was a good reminder that we do have exact variants for some fields, like biblio.title.exact (https://scholar.archive.org/search?q=biblio.title.exact%3A%22periodicals%22). And others, like author, are always indexed as "exact" (which does still have some normalization, just not stemming).

It should not be too "expensive" in index size to enable "exact" over the core biblio fields. Enabling it for the "everything" field (including body text) would nearly double the entire index size, if I understand correctly. We are already pretty tight on backing disk (SSD) at the moment, and would need to trade off this feature against things like citation graph, indexing additional search-only fulltext works, etc.

Another option might be to somehow execute "exact" match semantics at query time. Eg, taking a large-ish result set (based on the index with stemming enabled), and narrow the results by re-tokenizing the documents in "exact" mode and checking if they still match. If this is an existing elasticsearch/lucene feature I haven't heard of it. I suspect it wouldn't work well if not built-in, because request sets are often quite large and not returned in full (eg, one million or more results would need to be post-processed).

To summarize:

• should enable exact variants of more biblio fields in next full reindexing
• consider enabling exact variants in fatcat release search index (which is smaller)
• include exact index size increases for everything field as part of future hardware budgeting
• help wanted: feasibility of query-time exact filtering without requiring additional index size