Refactoring, fixes and some new features

[cal4]

Recommend turning on "Hide whitespace" due to line-ending change (need to settle on project ending eventually

Added

• Configurable retry strategies: exponential backoff and constant (the default)
  • Configurable delay between attempt (default 60 seconds)
  • Configurable max attempts before error (default 6)
• Configurable Apprise summary: every run or on error if there's remaining searches after running
  • Also configurable as a script parameter that takes precedence (useful for development)
• Some useful JetBrains config
• More logging
• More type hints
• Defaults when getting config

Changed

• Simplified search logic and made more resilient
  • Added assertion that terms were correctly typed
• Persist and read Google trends from disk for a given day (helps get rid of duplicate searches even between different runs)

Removed

• Redundant calls to close Chrome
• Calls to close all Chrome processes

Fixed

• Error when executing script from .bat file #113 Reference paths independent of pwd
• [BUG] AttributeError: 'Browser' object has no attribute 'giveMeProxy' #115 Comment out and todo give me proxy
• When logger is set to debug, doesn't include library code now
• [BUG] SessionNotCreatedException on Mobile Browser Creation #126 Sleep a bit after closing browser so ports are freed up

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request includes a comprehensive refactoring of the codebase, bug fixes, and the addition of new features. The changes involve restructuring the main script, enhancing logging, improving account handling, and adding new utility functions. The refactoring also introduces new classes and methods to improve code readability and maintainability.

File-Level Changes

Files	Changes
main.py src/utils.py src/searches.py src/browser.py src/activities.py src/login.py src/dailySet.py src/morePromotions.py src/punchCards.py	Refactored the main script and various modules to improve readability, maintainability, and error handling. Enhanced logging and introduced new utility functions and classes.
config.yaml	Added new configuration options for search attempts and delays.
test/test_main.py	Added unit tests for the main script.
src/account.py	Introduced a new Account dataclass for handling account information.

---

Tips

• Trigger a new Sourcery review by commenting @sourcery-ai review on the pull request.
• You can change your review settings at any time by accessing your dashboard:
  • Enable or disable the Sourcery-generated pull request summary or reviewer's guide;
  • Change the review language;
• You can always contact us if you have any questions or feedback.

[sourcery-ai]

Hey @cal4 - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟡 General issues: 11 issues found
• 🟡 Security: 2 issues found
• 🟡 Testing: 1 issue found
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

[Titaniumtown]

@klept0 the sourcery bot is so annoying lol

[jdeath]

I tried your fork (d48fb60) and had some issues with completing the Daily Set. But the crash was caught and it continued to do all the searches. I made a couple changes to have it work in docker, see sed's below.

On another note, I got "Read to Earn" working in standalone python code, but is via requests library and not selenium. It needs user intervention right now to login, I will see if can make it more automated.

sed -i 's/Path(__file__).parent.parent/Path.cwd()/g' src/utils.py # Make it get config from working directory

sed -i 's/self.webdriver.close()/self.webdriver.quit()/g' src/browser.py && \ # Patch for Docker

sed -i 's/chrome_options.add_argument(\"--headless=new\")/chrome_options.add_argument(\"--headless=new\")\n # Patch for Docker (already in klept0 version)

2024-06-24 06:59:20,509 [ERROR] [DAILY SET] Error Daily Set
Traceback (most recent call last):
  File "/app/src/dailySet.py", line 50, in completeDailySet
    self.activities.completeQuiz()
  File "/app/src/activities.py", line 83, in completeQuiz
    self.browser.utils.waitUntilQuestionRefresh()
  File "/app/src/utils.py", line 76, in waitUntilQuestionRefresh
    return self.waitUntilVisible(By.CLASS_NAME, "rqECredits")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/utils.py", line 64, in waitUntilVisible
    return WebDriverWait(self.webdriver, timeToWait).until(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/support/wait.py", line 105, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

[cal4]

I tried your fork (d48fb60) and had some issues with completing the Daily Set. But the crash was caught and it continued to do all the searches. I made a couple changes to have it work in docker, see sed's below.

On another note, I got "Read to Earn" working in standalone python code, but is via requests library and not selenium. It needs user intervention right now to login, I will see if can make it more automated.

sed -i 's/Path(__file__).parent.parent/Path.cwd()/g' src/utils.py # Make it get config from working directory

sed -i 's/self.webdriver.close()/self.webdriver.quit()/g' src/browser.py && \ # Patch for Docker

sed -i 's/chrome_options.add_argument(\"--headless=new\")/chrome_options.add_argument(\"--headless=new\")\n # Patch for Docker (already in klept0 version)

2024-06-24 06:59:20,509 [ERROR] [DAILY SET] Error Daily Set
Traceback (most recent call last):
  File "/app/src/dailySet.py", line 50, in completeDailySet
    self.activities.completeQuiz()
  File "/app/src/activities.py", line 83, in completeQuiz
    self.browser.utils.waitUntilQuestionRefresh()
  File "/app/src/utils.py", line 76, in waitUntilQuestionRefresh
    return self.waitUntilVisible(By.CLASS_NAME, "rqECredits")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/utils.py", line 64, in waitUntilVisible
    return WebDriverWait(self.webdriver, timeToWait).until(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/support/wait.py", line 105, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

Had same error today actually. Since then I've committed cal4@db016ce which I believe should fix that particular issue. I didn't intend to change the functionality of this class besides adding additional and more thorough logging, but hopefully this is all that needs to be adjusted.

I've also adjusted the try-catch to continue rather than return if there's an error with one of the cards, while still logging the error: cal4@4a633fc. That way the remaining cards, if any, are attempted. Believe this is closer to the OG behavior except with the logging.

Btw, thanks for giving it a go! Definitely keeping an eye on this issue.

[jdeath]

Got an error on new develop branch (46c368e): It got to search 13/30, but looks like got credit for 14 searches.

Edit: On a subsequent run, searches getting multiple attempt failed. But the same term (passedInTerm=debate news) keeps getting passed in for each set of 6 attempts. It did this many times in a row. A restart helped get a few new topics, but then since that term is so popular it got stuck on it again.

024-06-29 04:44:25,364 [DEBUG] in __exit__ exc_type=<class 'AssertionError'> exc_value= traceback=<traceback object at 0x7f677883c440>
2024-06-29 04:44:25,915 [ERROR] 
Traceback (most recent call last):
  File "/app/main.py", line 39, in main
    earned_points = executeBot(currentAccount, args)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/main.py", line 227, in executeBot
    ).bingSearches(remainingSearches.desktop)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/searches.py", line 107, in bingSearches
    pointsCounter = self.bingSearch(googleTrend)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/searches.py", line 119, in bingSearch
    pointsBefore = self.getAccountPoints()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/searches.py", line 179, in getAccountPoints
    return self.browser.utils.getBingInfo()["userInfo"]["balance"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/utils.py", line 121, in getBingInfo
    assert response.status_code == requests.codes.ok
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

2024-06-29 08:27:37,016 [INFO] [BING] 8/16
2024-06-29 08:27:37,016 [DEBUG] self.googleTrendsShelf.keys() = ['trump biden debate time', 'debate news', 'cnn', 'usa panama', 'uruguay  bolivia', 'presidental debate', 'biden vs trump', 'nbc news', '2024 presidential debate', 'what time is the presidential debate', 'thomas massie', 'who won the debate tonight', 'usa vs panama', 'suni lee', 'lebron james', 'trump vs biden', 'covid cases', 'biden', 'cbs news', 'presidential debate highlights', 'how to watch the debate tonight', 'who won the presidential debate', 'kyle filipowski', 'supreme court decisions', 'nato', 'presidential debate', 'when is the presidential debate', 'debate winner', '25th amendment', 'j.d. vance', 'newsom', 'lizzy musi', 'jill biden', 'supreme court decisions today', 'delhi airport roof collapses']
2024-06-29 08:27:37,552 [DEBUG] terms=['debate news', 'debate newsome desantis', 'debate news analysis', 'debate newsome', 'debate news today', 'debate newshub', 'debate newshour', 'debate newsnation', 'debate news tonight', 'debate news who won', 'debate newsome desantis watch', 'desantis newsom debate', 'waspi debate breaking news today', 'fox news debate', 'lebanon debate news', 'newsome vs desantis debate', 'desantis vs newsom debate', 'ron desantis gavin newsom debate']
2024-06-29 08:27:37,552 [DEBUG] passedInTerm=debate news
2024-06-29 08:27:37,628 [DEBUG] term=debate news
2024-06-29 08:27:41,189 [DEBUG] [BING] Search attempt failed 1/6, retrying after sleeping 62 seconds...
2024-06-29 08:28:43,264 [DEBUG] term=debate newsome desantis
2024-06-29 08:28:46,674 [DEBUG] [BING] Search attempt failed 2/6, retrying after sleeping 70 seconds...
2024-06-29 08:29:56,773 [DEBUG] term=debate news analysis
2024-06-29 08:30:00,139 [DEBUG] [BING] Search attempt failed 3/6, retrying after sleeping 74 seconds...
2024-06-29 08:31:14,191 [DEBUG] term=debate newsome
2024-06-29 08:31:17,680 [DEBUG] [BING] Search attempt failed 4/6, retrying after sleeping 78 seconds...
2024-06-29 08:32:35,778 [DEBUG] term=debate news today
2024-06-29 08:32:39,361 [DEBUG] [BING] Search attempt failed 5/6, retrying after sleeping 88 seconds...
2024-06-29 08:34:07,453 [DEBUG] term=debate newshub
2024-06-29 08:34:10,990 [DEBUG] [BING] Search attempt failed 6/6, retrying after sleeping 89 seconds...
2024-06-29 08:35:39,990 [ERROR] [BING] Reached max search attempt retries
2024-06-29 08:35:39,991 [DEBUG] Moving term to end of list
2024-06-29 08:35:39,992 [DEBUG] pointsCounter = XXXX
2024-06-29 08:35:50,992 [INFO] [BING] 9/16
2024-06-29 08:35:50,993 [DEBUG] self.googleTrendsShelf.keys() = ['trump biden debate time', 'debate news', 'cnn', 'usa panama', 'uruguay  bolivia', 'presidental debate', 'biden vs trump', 'nbc news', '2024 presidential debate', 'what time is the presidential debate', 'thomas massie', 'who won the debate tonight', 'usa vs panama', 'suni lee', 'lebron james', 'trump vs biden', 'covid cases', 'biden', 'cbs news', 'presidential debate highlights', 'how to watch the debate tonight', 'who won the presidential debate', 'kyle filipowski', 'supreme court decisions', 'nato', 'presidential debate', 'when is the presidential debate', 'debate winner', '25th amendment', 'j.d. vance', 'newsom', 'lizzy musi', 'jill biden', 'supreme court decisions today', 'delhi airport roof collapses']
2024-06-29 08:35:51,551 [DEBUG] terms=['debate news', 'debate newsome desantis', 'debate news analysis', 'debate newsome', 'debate news today', 'debate newshub', 'debate newshour', 'debate newsnation', 'debate news tonight', 'debate news who won', 'debate newsome desantis watch', 'desantis newsom debate', 'waspi debate breaking news today', 'fox news debate', 'lebanon debate news', 'newsome vs desantis debate', 'desantis vs newsom debate', 'ron desantis gavin newsom debate']
2024-06-29 08:35:51,551 [DEBUG] passedInTerm=debate news
2024-06-29 08:35:51,606 [DEBUG] term=debate news
2024-06-29 08:35:55,279 [DEBUG] [BING] Search attempt failed 1/6, retrying after sleeping 65 seconds...

[cal4]

Closing this PR in favor of #137 to avoid confusion