New update mechanisim

[Grotax]

Hey everyone,

with News 25.2.0-beta.1 I added some new logic to the updating mechanism.
A feed is now no longer updated solely on the update interval which by default is every 3600s (1 hour) but we also instead consider a feature offered by feed-io the library that we use for feed-parsing.

feed-io calculates based on how regularly the feed is updated a plausible next update time and we store that in the database.

When the update job is triggered by Nextcloud news will first check if that next update time has been reached. If not it will not update the feed.

Checking my own instance this seems to work fine in the wild.

At least on mysql/mariadb you should be able to use this query:

SELECT
    title,
    next_update_time,
    CASE
        WHEN TIMESTAMPDIFF(MINUTE, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))) < 60 THEN CONCAT(TIMESTAMPDIFF(MINUTE, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))), ' minutes from now')
        WHEN TIMESTAMPDIFF(HOUR, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))) < 24 THEN CONCAT(TIMESTAMPDIFF(HOUR, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))), ' hours from now')
        WHEN TIMESTAMPDIFF(DAY, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))) = 1 THEN 'Tomorrow'
        WHEN TIMESTAMPDIFF(DAY, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))) = 2 THEN 'In 2 days'
        WHEN TIMESTAMPDIFF(DAY, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))) < 7 THEN CONCAT('In ', TIMESTAMPDIFF(DAY, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))), ' days')
        WHEN TIMESTAMPDIFF(WEEK, NOW(), FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED))) = 1 THEN 'Next week'
        ELSE DATE_FORMAT(FROM_UNIXTIME(CAST(next_update_time AS UNSIGNED)), '%Y-%m-%d') 
    END AS relative_time
FROM oc_news_feeds;

To get a human readable relative time.

[wofferl]

Where the next update time seems plausible for most feeds, there seems to be some exceptions where it is several days in the future.

I suspect this is when the feed does not include a proper last_modified. This causes a problem with mostly infrequent feeds, in my feed list mostly some reddit feeds.

It would probably make sense to have a maximum delay of one day as it is handled for staled feeds by feed-io.

Here are some examples:

• The helios feed is staled and have the newest article three years ago, but feed-io seems not detect it as such.
• The Kicker feed is active and had many news yesterday, but is set to 08.01 now.
• Even the news app release notes was set to one week later, which somehow makes sense due to the release cycles, but would still mean that you only find out about a new version days later.

+---------------------------+-------------------------------------------------+---------------------------------+--------------------------+---------------------+---------------------+
| title                     | url                                             | http_last_modified              | last_modified            | next_update_time    | articles_per_update |
+---------------------------+-------------------------------------------------+---------------------------------+--------------------------+---------------------+---------------------+
| helios64                  | https://www.reddit.com/r/helios64/.rss          | Wed, 23 Oct 2024 08:00:36 +0000 | 2024-12-30 20:10:16.7190 | 2025-01-12 17:37:59 |                  26 |
| kicker News               | https://newsfeed.kicker.de/news/aktuell         | NULL                            | 2024-12-31 00:50:25.1100 | 2025-01-08 07:29:50 |                  20 |
| We Are Football           | https://www.reddit.com/r/WeAreFootball/.rss     | Wed, 23 Oct 2024 08:00:35 +0000 | 2024-12-30 20:10:16.2461 | 2025-01-05 19:32:23 |                  27 |
| Release notes from news   | https://github.com/nextcloud/news/releases.atom | Wed, 23 Oct 2024 07:13:55 +0000 | 2024-12-30 20:10:06.9068 | 2025-01-05 00:13:04 |                  10 |
| Steam Play                | https://www.reddit.com/r/SteamPlay/.rss         | Wed, 23 Oct 2024 08:00:38 +0000 | 2024-12-30 20:10:18.5014 | 2025-01-02 02:11:09 |                  26 |
| Path of Exile News        | https://www.pathofexile.com/news/rss            | Wed, 23 Oct 2024 08:00:49 +0000 | 2024-12-30 20:10:25.3664 | 2025-01-01 20:55:05 |                  30 |
| Full Disclosure           | https://seclists.org/rss/fulldisclosure.rss     | Tue, 31 Dec 2024 05:00:02 +0000 | 2024-12-31 06:25:07.7496 | 2025-01-01 16:35:31 |                  15 |
| xkcd.com                  | https://xkcd.com/rss.xml                        | Mon, 21 Oct 2024 04:00:00 +0000 | 2024-12-30 23:40:03.0708 | 2025-01-01 10:48:00 |                   4 |
+---------------------------+-------------------------------------------------+---------------------------------+--------------------------+---------------------+---------------------+

The kicker feed is a very active feed with a lot of news every day

+-------------------------------------------------------------------------------------------+-------------------------+
| title                                                                                     | from_unixtime(pub_date) |
+-------------------------------------------------------------------------------------------+-------------------------+
| Starkes Scoring und Nervenspiel: Littler steht im Viertelfinale                           | 2024-12-30 23:06:08     |
| Torfest im Topspiel: Ingolstadt macht das halbe Dutzend gegen Mannheim voll               | 2024-12-30 22:37:08     |
| Juristischer Weg ausgesch�pft: Verl�sst Dani Olmo Barcelona abl�sefrei?                   | 2024-12-30 22:27:05     |
| Nach desolaten ersten 30 Minuten: ManUnited verliert auch gegen Newcastle                 | 2024-12-30 22:00:08     |
| Sprung auf Platz 2 verpasst: Chelsea unterliegt Kellerkind Ipswich                        | 2024-12-30 21:52:18     |
| Torreiches Topduell: ERC fertigt Adler ab                                                 | 2024-12-30 21:31:46     |
| W�rzburg unterliegt Bamberg - Heidelberg gewinnt Topspiel                                 | 2024-12-30 21:29:00     |
| Van Gerwen erh�ht zum richtigen Zeitpunkt die Schlagzahl                                  | 2024-12-30 21:27:56     |
| Wo ist Cardoso Varela? Ein Talent als Handelsware                                         | 2024-12-30 21:13:03     |
| Inklusive Feazell-Poster: Bambergs souver�ner Derbysieg in W�rzburg                       | 2024-12-30 21:10:13     |
| LIVE! Nur ein Sieg in regul�rer Spielzeit hilft: Deutsche U 20 gegen Lettland unter Druck | 2024-12-30 20:30:00     |
| LIVE! W�rzburg unterliegt Bamberg - Topspiel in Ludwigsburg                               | 2024-12-30 20:00:22     |
| Pietreczkos Rechnung: "Immer eine Runde weiter und in vier Jahren Weltmeister"            | 2024-12-30 18:33:34     |
| Das finale Playoff-Szenario der NFL: Bengals weiter dabei                                 | 2024-12-30 17:50:00     |
| Gebrauchter Nachmittag: Aspinall kocht Pietreczko ab                                      | 2024-12-30 17:35:43     |
| LIVE! W�rzburg empf�ngt Bamberg - Topspiel in Ludwigsburg                                 | 2024-12-30 17:14:34     |
| Antrags-Stopp aufgehoben: "Neustart" der Games-F�rderung                                  | 2024-12-30 17:13:47     |
| Super-Oldie Losilla: Mehr Eins�tze als erwartet                                           | 2024-12-30 17:03:10     |
| Baumgart und Union: Eine romantische und spannende Wiedervereinigung                      | 2024-12-30 16:55:00     |
| Nach Kunz-Abschied: Duisburg verpflichtet Paris aus Schalke                               | 2024-12-30 16:27:06     |
+-------------------------------------------------------------------------------------------+-------------------------+

I will continue to observe whether there are any other feeds that get delayed like the kicker feed.

[wofferl]

Looking at the kickers feed in my environment that I just added next_update was set to in a few minutes... Not sure what went wrong in your case.

it works all day yesterday and as I looked today it had the delayed update time. The last fetch was short after midnight, so maybe some miscalculation on day change. I have reset the next update time and now it works again as expected. I'll be watching to see if it happens again.

By the way would you have capacity and an Idea how to integrate the next_update time in the frontend so that users can see?

I have already thought about that, but have not come to a good idea and the right place yet for all use cases.

Maybe in the feed header itself or as extra info in the feed tooltip, but by default the feed is hidden in the list with no unread items, which makes it difficult, because the feed most likely has no unread items when you are interested in the info.

We could show the feed with a warning when the next update time is several days in the future.

Or we could possibly do some extra info page where the user can see all feeds in a table like I posted above.

[Grotax]

I kind of like the table idea, it could also have options to manage a feed, like rename.

You could see when it was last updated and when the next update is expected maybe more.

[wofferl]

I have digged a bit deeper, what the problems are.

Looking at the helios feed, feed-io should ideally detect that feed as "sleepy" which would set the check to next day. But I guess since reddit generates these feeds on request and the updated date is today that might make it difficult. For this feed it should be fine though to check in a few days.

Yes, this is something where feed-io should take maybe the newest item date into account and mark feeds also stale where the last post is older than the sleepyDuration.

Looking at the kickers feed in my environment that I just added next_update was set to in a few minutes... Not sure what went wrong in your case.

The problem here was that the feed had some older items from august. During the night where the feed get no new items, adding the median interval to the feedTimestamp never extend to a time in the future, so the average was took, which was much higher because of the old articles. This is no longer the case.
But you can see this now with https://www.sportschau.de/fussball/index~rss2.xml, which shows the same behavior.

[Grotax]

Do you think the adjustment to have a max delay for the update of 1 day is enough for these cases?
I still have to check if this actually works and stuff but it is currently the only option I see.

With feed-io in general we might face the issue that maintenance and further development has come to a halt.
And I do not really know what that will mean for news.
Topics like this are affected by it, since there is the desire to not fetch just every x seconds but to do it more intelligent.

I currently see two major options.

• news "Team" takes over the maintenance of feed-io by forking it
• we switch to a different feed parsing library (again)

I think this is not urgent yet but yea...

[wofferl]

Do you think the adjustment to have a max delay for the update of 1 day is enough for these cases? I still have to check if this actually works and stuff but it is currently the only option I see.

I think this is the easiest way to actual workaround this, capping the nextUpdateTime which the fetcher get from feed-io to 1 day.

Having longer delay may be make sense with some special feeds, but in general I think for most feeds you want it to check at least once per day for updates and especially those with very irregular but important updates or information.

[a-mair]

Hi!

This sounds like a good feature to reduce feed updates. I did a quick test to get an idea of when feed updates will happen with this feature.

Some feed's next update is scheduled months in the future, see attached screenshot. Almost any of that feeds has items published in the last week, some even published most recently.

So I must say that the current implementation needs some fine-tuning. I'd suggest that a feed's update interval should never be more than 1 or 2 days. What do you think about it?

[Grotax]

Hmm strange I would have thought that the parameters set in the latest version would limit it to one day max.

But I guess it does not work.

Yea so thing is the library feed-io does not receive updates at this time.

Your only option is to turn the feature off and set the field for each feed back to null

Edit: I think that is not needed

[wofferl]

Hmm strange I would have thought that the parameters set in the latest version would limit it to one day max.

But I guess it does not work.

Yea so thing is the library feed-io does not receive updates at this time.

The news app currently accepts every nextupdatetime that feed-io calculates, which leads to this problem.

Even if it is currently not clear how things will continue with feed-io, I have created a PR (alexdebril/feed-io#434) that solves most of the problems.

However, you should still consider limiting the returned value in the news app to a maximum of 1-2 days.

[Grotax]

I currently have less time again to work on news. So don't expect any contributions from my side any time soon.