You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This has been caught 92 times already including 36 times in the last 24 hours. Evidently, it's not uncommon for robots.txt not to be utf-8 encoded: let's see if we can be smarter about this, even if it doesn't cause entire captures to fail.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 27673: invalid continuation byte
File "threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
File "perma/tasks.py", line 613, in robots_txt_thread
content = str(robots_txt_response.content, 'utf-8')
The text was updated successfully, but these errors were encountered:
Sentry Issue: PERMA-B
This has been caught 92 times already including 36 times in the last 24 hours. Evidently, it's not uncommon for robots.txt not to be utf-8 encoded: let's see if we can be smarter about this, even if it doesn't cause entire captures to fail.
The text was updated successfully, but these errors were encountered: