Unable to load new page, browser needs restart #766

zlodejpapiru · 2025-02-11T14:50:41Z

Hi, since the 1.5 version crawler keep crashing. Save state is saved, warcs not generated. Here is excerpt form log (1.5.3 version) + my config.

workers: 2
limit: 200000
generateWACZ: true
behaviors:

autoscroll
siteSpecific
autoplay
autofetch
#behaviorTimeout: 0
#timeout: 36000
saveState: always
warcinfo:
operator: xxxx
generateCdx: true
#pageLoadTimeout: 90
scopeType: page-spa
extraHops: 50000
#waitUntil: true
#sitemap: true

The text was updated successfully, but these errors were encountered:

ikreymer · 2025-02-11T18:41:55Z

Are you sure there are no WARCs generated? The crawler is interrupted, but should still have WARC files on disk (but not WACZ since it hasn't finished yet).

Yes, the latest version of the browser, we've been seeing this error more often, which results in having to restart the crawler. However, the data should still be there in the ./crawls/collections//archives directory and you should be able to restart..

We're looking into why its happening, something changed in the browser perhaps..

ldko · 2025-02-12T15:41:49Z

I am also seeing issues that seem related to the browser. WARCs are generated (but no WACZ).

I was trying to run another set of EOT crawls similar to ones I ran in early January where I launch 1 crawl at a time (2 workers, 6 second delay on page requests) from a bash script that contains 11 docker crawl commands. Only 2 of the crawls actually produced a WACZ. I had pulled another image from a couple days ago where similarly I was seeing the crawls end with a "Exiting, Crawl status: interrupted" message. Toward the end of the logs for these crawls I start to see messages like:

{"timestamp":"2025-02-11T22:44:28.805Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://www.acf.hhs.gov/ecd?page=1","workerid":0}}
{"timestamp":"2025-02-11T22:44:28.806Z","logLevel":"debug","context":"worker","message":"Getting page in new window","details":{"workerid":0}}
{"timestamp":"2025-02-11T22:44:29.405Z","logLevel":"warn","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":1}}
{"timestamp":"2025-02-11T22:44:29.406Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":1,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:95:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:201:13)\n    at async Promise.allSettled (index 1)\n    at async runWorkers (file:///app/dist/util/worker.js:278:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:1053:9)\n    at async Crawler.run (file:///app/dist/crawler.js:359:13)\n    at async file:///app/dist/main.js:58:1"}}
{"timestamp":"2025-02-11T22:44:29.906Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://www.acf.hhs.gov/ecd?page=2","workerid":1}}
{"timestamp":"2025-02-11T22:44:29.907Z","logLevel":"debug","context":"worker","message":"Getting page in new window","details":{"workerid":1}}
{"timestamp":"2025-02-11T22:44:48.825Z","logLevel":"warn","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":0}}
{"timestamp":"2025-02-11T22:44:48.826Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":0,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:95:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:201:13)\n    at async Promise.allSettled (index 0)\n    at async runWorkers (file:///app/dist/util/worker.js:278:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:1053:9)\n    at async Crawler.run (file:///app/dist/crawler.js:359:13)\n    at async file:///app/dist/main.js:58:1"}}
{"timestamp":"2025-02-11T22:44:48.826Z","logLevel":"error","context":"worker","message":"Worker error, exiting","details":{"type":"exception","message":"Unable to load new page, browser needs restart","stack":"Error: Unable to load new page, browser needs restart\n    at PageWorker.initPage (file:///app/dist/util/worker.js:147:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:201:13)\n    at async Promise.allSettled (index 0)\n    at async runWorkers (file:///app/dist/util/worker.js:278:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:1053:9)\n    at async Crawler.run (file:///app/dist/crawler.js:359:13)\n    at async file:///app/dist/main.js:58:1","workerid":0}}
{"timestamp":"2025-02-11T22:44:49.909Z","logLevel":"warn","conte

zlodejpapiru · 2025-02-13T07:13:51Z

@ikreymer You are right - warcs are there, wacz not. I tried to rerun the crawl but keeps crashing.

gitreich · 2025-02-13T09:53:50Z

Same here, but it only happens with large crawls running longer then 24 hours and stucking on seeds after at least 500 crawled seeds. I thought it's depending on unavailable websites, but it seems like this is not true, and it may also happen if the page is available and does response something correctly. Also on our system warcs are written until the crash.
I use networkidle0 and thought of changing it;
The yaml contains 20.000 seeds build out of 5000 domains with http, http,http://www. and https://www.
Here is a example of the docker start cmd:

docker run -d --name ONB_Btrix_dc_at_test5_BULK_2_20250211013617 -e NODE_OPTIONS='--max-old-space-size=32768' -p 42555:42555 -p 33391:33391 -v /data/browsertrix/crawls/:/crawls/ webrecorder/browsertrix-crawler:1.5.3 crawl --screencastPort 42555 --healthCheckPort 33391 --scopeType domain --headless --delay 0 --behaviorTimeout 60 --pageLoadTimeout 60 --waitUntil networkidle0 --saveState always --logging stats,info --config /crawls/config/dc_at_test5_BULK_2_20250211013617.yaml --depth 0 --workers 1 --warcInfo ONB_CRAWL_dc_at_test5_BULK_2_20250211013617_Depth_0_20250211013618 --userAgentSuffix +ONB_Bot_Btrix_1.5.3, [email protected] --crawlId id_ONB_CRAWL_dc_at_test5_BULK_2_20250211013617_Depth_0_20250211013618 --collection dc_at_test5_BULK_2_20250211013617

As mentioned with the same version there are about 100 other crawls daily, which does not reach this point, but they have all limits set, and are closed by reaching them

ikreymer · 2025-02-16T04:44:41Z

Thanks for the reproducible tests!

Did some testing, can confirm this is happening with Chrome 133, but not Chrome 134 (Beta) or Chrome 135 (Canary).
As soon as there's Brave release on Chromium 134, can try that.

should fix browser timing out on new window, fixes #766 bump to 1.5.4

ikreymer · 2025-02-19T23:48:04Z

Looks like there's a release of Brave with Chromium 134, going to test that and release 1.5.4. On initial testing, it looks like this issue disappears in Chromium 134.

github-project-automation bot added this to Webrecorder Projects Feb 11, 2025

github-project-automation bot moved this to Triage in Webrecorder Projects Feb 11, 2025

ikreymer added a commit that referenced this issue Feb 19, 2025

browser: update brave to 1.77.52 to get Chromium 134

1160f0f

should fix browser timing out on new window, fixes #766 bump to 1.5.4

ikreymer mentioned this issue Feb 19, 2025

browser: update brave to 1.77.52 to get Chromium 134 #773

Merged

ikreymer closed this as completed in #773 Feb 20, 2025

ikreymer closed this as completed in c25c677 Feb 20, 2025

github-project-automation bot moved this from Triage to Done! in Webrecorder Projects Feb 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to load new page, browser needs restart #766

Unable to load new page, browser needs restart #766

zlodejpapiru commented Feb 11, 2025 •

edited

Loading

ikreymer commented Feb 11, 2025

ldko commented Feb 12, 2025 •

edited

Loading

zlodejpapiru commented Feb 13, 2025

gitreich commented Feb 13, 2025

ikreymer commented Feb 16, 2025

ikreymer commented Feb 19, 2025

Unable to load new page, browser needs restart #766

Unable to load new page, browser needs restart #766

Comments

zlodejpapiru commented Feb 11, 2025 • edited Loading

ikreymer commented Feb 11, 2025

ldko commented Feb 12, 2025 • edited Loading

zlodejpapiru commented Feb 13, 2025

gitreich commented Feb 13, 2025

ikreymer commented Feb 16, 2025

ikreymer commented Feb 19, 2025

zlodejpapiru commented Feb 11, 2025 •

edited

Loading

ldko commented Feb 12, 2025 •

edited

Loading