-
|
Probably a dumb question: How can I set the status of an URL to "completed", so it gets reported by GetStats as "completed", i.e. how to push custom counts to the url frontier? Is there an example available? 🙃 |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
This is a great question. A URL counts as completed when it is received by the service, is not a DiscoveredURLItem and does not have a refetchable_from_date (or if its value is 0). If a URL has been fetched (i.e. it is not a DiscoveredURLItem) but is given a refetchable_from_date, it won't count towards the completed as it might be refetched later. You can't push custom counts to the frontier. As you can see in the code the completed and active_queues are not in the API and should be considered experimental and implementation dependent. Does that answer your question @rzo1? |
Beta Was this translation helpful? Give feedback.
-
|
Yes This answers the question. I could implement it accordingly. |
Beta Was this translation helpful? Give feedback.
This is a great question.
A URL counts as completed when it is received by the service, is not a DiscoveredURLItem and does not have a refetchable_from_date (or if its value is 0). If a URL has been fetched (i.e. it is not a DiscoveredURLItem) but is given a refetchable_from_date, it won't count towards the completed as it might be refetched later.
You can't push custom counts to the frontier. As you can see in the code the completed and active_queues are not in the API and should be considered experimental and implementation dependent.
Does that answer your question @rzo1?