Add identify-ids-by-es-query script #107
Conversation
There was a problem hiding this comment.
This looks good!
I had initially come to the scroll api when I was starting my own version of this script, but was diverted by this alert in the docs:
We no longer recommend using the scroll API for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).
I don't imagine that this is a problem for this script, but something to keep in mind for updates in the future. I have a feeling that the search_after might also help with the bug where duplicated results turn up. I can't link to this line, but this in the pagination api docs caught my eye:
Repeat this process by updating the search_after array every time you retrieve a new page of results. If a refresh occurs between these requests, the order of your results may change, causing inconsistent results across pages. To prevent this, you can create a point in time (PIT) to preserve the current index state over your searches.
I think this api is obviously useful for large searches, but I think we could also leverage it for our smaller range pagination on search results to avoid the duplicated search results.
scripts/identify-ids-by-query.js
Outdated
| } | ||
| }) | ||
|
|
||
| castArgsToInts(argv, ['batchSize', 'limit', 'offset']) |
There was a problem hiding this comment.
I think this line shouldn't be here. These arguments are not specified in the preamble.
There was a problem hiding this comment.
Good catch. We do want to support --limit, so keeping that one.
scripts/identify-ids-by-query.js
Outdated
| type: 'string', | ||
| default: '0' | ||
| }, | ||
| outfile: { type: 'string' }, |
There was a problem hiding this comment.
the preamble says this field has a default, but there is not one specified here.
|
Can you update this script name to include ES? |
nonword
changed the title
A script for building a CSV of bib ids for an arbitrary ES query. Useful for building a CSV of records for feeding into the RCI bulk-index script.