RSS feeds for search queries

[bnewbold]

This feature would allow creation of RSS feed endpoints for any search query. The feed would allow users to "subscribe" to new search hits.

Some implementation thoughts:

• if query is embedded in feed URL, no need to retain any server-side state
• utility of this may be dependent on having decent subject/categorization metadata? or maybe not, if keywords are used.
• feed could be sorted/filtered by 1) release date of works 2) index document update time 3) index document creation time. or maybe some combination? index document creation time will not be a stable/long-term metadata field (eg, when re-indexing all document creation times will get incremented)
• it is an assumption that this would not result in many actual queries and search engine load, as RSS usually is only fetched... daily? if there was a lot of load, could cache results in elasticsearch itself (eg, hash the query string, check if there is a cached result from past N hours, only run query and update cache if stale; store results in separate index)

[sckott]

I'd be happy to test if implemented

[bnewbold]

Re-indexing is finally caught up, and the "papers from the past week" type of query should work, so starting to think about this. And i'm excited!

This library seems like a great super-simple way to implement an RSS feed in fastapi; though maybe Atom is preferred?: https://pypi.org/project/fastapi-rss/

Presumably would have a small jinja2 template to render a summary with the existing macros into HTML, and inject that into the items.

Would probably be two new endpoints: a form to help craft a query, with "feed-specific" query parameters, and an RSS endpoint itself (XML).

I think the default parameters should be:

• filter to doc_type:work, so no digitized pages (which are a hack)
• filter to the time range "today to 90 days ago", and sort by recency (date), and then sort by _doc (so there is a stable sort order, within works with the same date)

Then in the generation page have a form for other filters, and a free-form query box (same as the regular search).

Run the query with the same current routine, take the results, transform, and return as the feed.

[sckott]

I don't have a preference between RSS and atom.

Plan sounds great to me

[bnewbold]

I just pushed a minimal version of this. On search result pages, there is an "RSS Feed" link under the search box, which goes straight to an RSS 2.0 file with the search parameters.

I tested with my feed reader and it seems to be working over the past week. Any feedback welcome!

[sckott]

Awesome! Thanks for getting this working. My original use case is gone with changing jobs, but this will be useful for tracking different paper topics in Feedbin

[lstrtstr]

Any feedback welcome!

Many RSS feeds of journal searches seem to have no entries.
Is that on purpose or a bug?

For example, if I search for journal:Catena, I get results as recent as 2022 (https://scholar.archive.org/search?q=journal%3ACatena&sort_order=time_desc). The corresponding RSS feed https://scholar.archive.org/feed/rss?q=journal%3ACatena has no entries, though.