TimoutException after few searches

[Mascherino]

Before submitting a bug report...

• This bug wasn't already reported.
  (I have checked every bug report on GitHub)

• I've cleared the sessions folder.

Title

• The title is no longer "Title" and I edited it with the right error name.

Branch

develop

Commit

750fa7a

Describe the bug

The first searches work as expected, but after a few searches, I get the error below. The browser is on the rewards dashboard, loading forever, resulting in a TimeoutException after about 5 minutes of waiting. It seems like it tries to go to the page of the last search term ("hertha bsc") to start the next search but fails to do so for some reason. Manually refreshing the page returns it to the rewards dashboard, crashing the program.

Copy and paste your error

2024-08-21 02:06:47,423 [INFO] [BING] Remaining searches=24
2024-08-21 02:06:57,884 [DEBUG] terms=['hertha bsc', 'hertha', 'hertha wiesbach', 'hertha bsc news', 'hertha news', 'hertha bsc tickets', 'hertha feiler', 'hertha bsc shop', 'hertha tickets', 'hertha bsc berlin', 'hertha bsc transfermarkt', 'hertha bsc news aktuell']
2024-08-21 02:06:57,885 [DEBUG] rootTerm=hertha
2024-08-21 02:06:58,271 [DEBUG] term=hertha bsc
2024-08-21 02:07:26,005 [INFO] [BING] Remaining searches=23
2024-08-21 02:12:40,238 [DEBUG] terms=['energie cottbus homepage', 'energie cottbus vs dynamo dresden', 'energie cottbus tabelle', 'energie cottbus live', 'energie cottbus tickets', 'energie cottbus pinwand', 'energie cottbus live stream heute', 'energie cottbus news', 'energie cottbus shop', 'energie cottbus fanshop', 'energie cottbus spielplan', 'energie cottbus vs dynamo dresden 2024']
2024-08-21 02:12:40,238 [DEBUG] rootTerm=energie cottbus
2024-08-21 02:13:00,274 [DEBUG] in __exit__ exc_type=<class 'selenium.common.exceptions.TimeoutException'> exc_value=Message: 
Stacktrace:
#0 0x5ed8acad26ca <unknown>
#1 0x5ed8ac7a3600 <unknown>
#2 0x5ed8ac7f2bcb <unknown>
#3 0x5ed8ac7f2eb1 <unknown>
#4 0x5ed8ac836b24 <unknown>
#5 0x5ed8ac8158cd <unknown>
#6 0x5ed8ac83404a <unknown>
#7 0x5ed8ac815643 <unknown>
#8 0x5ed8ac7e5d31 <unknown>
#9 0x5ed8ac7e679e <unknown>
#10 0x5ed8aca9a25b <unknown>
#11 0x5ed8aca9e1f2 <unknown>
#12 0x5ed8aca87615 <unknown>
#13 0x5ed8aca9ed82 <unknown>
#14 0x5ed8aca6c25f <unknown>
#15 0x5ed8acac1e68 <unknown>
#16 0x5ed8acac2040 <unknown>
#17 0x5ed8acad149c <unknown>
#18 0x712972494ac3 <unknown>
 traceback=<traceback object at 0x709248ee3640>
2024-08-21 02:13:01,403 [ERROR] 
Traceback (most recent call last):
  File "/home/dev/MRFarmer-dev/main.py", line 40, in main
    earned_points = executeBot(currentAccount, args)
  File "/home/dev/MRFarmer-dev/main.py", line 249, in executeBot
    searches.bingSearches()
  File "/home/dev/MRFarmer-dev/src/searches.py", line 129, in bingSearches
    self.bingSearch()
  File "/home/dev/MRFarmer-dev/src/searches.py", line 162, in bingSearch
    searchbar = self.browser.utils.waitUntilClickable(
  File "/home/dev/MRFarmer-dev/src/utils.py", line 74, in waitUntilClickable
    return WebDriverWait(self.webdriver, timeToWait).until(
  File "/home/dev/MRFarmer-dev/.venv/lib/python3.10/site-packages/selenium/webdriver/support/wait.py", line 105, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 
Stacktrace:
#0 0x5ed8acad26ca <unknown>
#1 0x5ed8ac7a3600 <unknown>
#2 0x5ed8ac7f2bcb <unknown>
#3 0x5ed8ac7f2eb1 <unknown>
#4 0x5ed8ac836b24 <unknown>
#5 0x5ed8ac8158cd <unknown>
#6 0x5ed8ac83404a <unknown>
#7 0x5ed8ac815643 <unknown>
#8 0x5ed8ac7e5d31 <unknown>
#9 0x5ed8ac7e679e <unknown>
#10 0x5ed8aca9a25b <unknown>
#11 0x5ed8aca9e1f2 <unknown>
#12 0x5ed8aca87615 <unknown>
#13 0x5ed8aca9ed82 <unknown>
#14 0x5ed8aca6c25f <unknown>
#15 0x5ed8acac1e68 <unknown>
#16 0x5ed8acac2040 <unknown>
#17 0x5ed8acad149c <unknown>
#18 0x712972494ac3 <unknown>

Screenshots

Value of dashboard variable

N/A

[mshancock]

I was getting this same error (though only on mobile searches), and this fix worked for me in searches.py (lines 171-173):

                try:
                    searchbar = self.browser.utils.waitUntilClickable(
                        By.ID, "sb_form_q", timeToWait=40
                    )
                except TimeoutException:
                    logging.debug("TimeoutException")
                    self.browser.utils.goToSearch()
                    return

The idea is to just gracefully fail when this happens - the self.browser.utils.goToSearch() was necessary to somehow reset the webpage so the next waitUntilClickable won't just timeout again.

[Mascherino]

I did something similar to try and fix it. Unfortunately I was getting http2 errors after a few errors got caught. Disabling http2 with the --disable-http2 flag in chrome options fixed that and everything works smoothly now.

[jdeath]

Me too. I first tried increasing timetoWait, but did not help. I will try both of these solutions. Suggest doing a PR if they work.

Edit, tried both of your suggestions and still got errors. I caught timeout error, but it just looping until it crashed. For me, the timeouts are intermittent in each run, so these fixes appear to work but then don't.

[28Black]

I was getting this same error (though only on mobile searches), and this fix worked for me in searches.py (lines 171-173):

                try:
                    searchbar = self.browser.utils.waitUntilClickable(
                        By.ID, "sb_form_q", timeToWait=40
                    )
                except TimeoutException:
                    logging.debug("TimeoutException")
                    self.browser.utils.goToSearch()
                    return

The idea is to just gracefully fail when this happens - the self.browser.utils.goToSearch() was necessary to somehow reset the webpage so the next waitUntilClickable won't just timeout again.

This works, the script is not crashing with the mentioned error(s). However, as @jdeath said, it just creates a loop of a search attempt at some point, trying the same search term over and over again while failing it every single time.

[jdeath]

edit: Didn't work today. Got into endless TimeoutException loop...

I finally had it complete all searches on first try and successfully continued through a timeout exception. It is based on @mshancock code. Making a new window is what allowed it to not get stuck in an endless timeout loop. Could close the old tab, but I did not bother. I did not need to add --disable-http2 but did add --headless=new , not sure if necessary

Only worked once, but I will keep testing, I'll do a PR if keeps working

searches.py:

                try:
                    searchbar = self.browser.utils.waitUntilClickable(
                        By.ID, "sb_form_q", timeToWait=40
                    )
                except TimeoutException:
                    logging.debug("TimeoutException - Making New Tab")
                    self.browser.utils.createAndVisitNewTab()
                    self.browser.utils.goToSearch()
                    return

utils.py, add a function:

def createAndVisitNewTab(self, timeToWait: float = 0) -> None:
        self.webdriver.switch_to.new_window('window')

I also added to browser.py browserSetup, but I am not sure if it was necessary.

 options.add_argument("--headless=new")

Function names should probably be changed. I first tried to make a new tab using the existing functions in utils.py, but did not work. I then found the line of code I needed in the selenium documentation. Could change 'window' to 'tab' to see if still works.

[github-actions]

This issue is stale because it has been open for 14 days with no activity.