Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: 431 error on unbuntu due to header issue #3037

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

fix: 431 error on unbuntu due to header issue #3037

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion src/you_get/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,10 @@ def urlopen_with_retry(*args, **kwargs):
if i + 1 == retry_time:
raise http_error

def get_domain(url):
domain = parse.urlsplit(url).netloc
return re.sub(r'^www\.', '', domain)


def get_content(url, headers={}, decoded=True):
"""Gets the content of a URL via sending a HTTP GET request.
Expand All @@ -471,8 +475,11 @@ def get_content(url, headers={}, decoded=True):
# - https://bugs.python.org/issue2190
# Here we add cookies to the request headers manually
cookie_strings = []
domain = get_domain(url)
for cookie in list(cookies):
cookie_strings.append(cookie.name + '=' + cookie.value)
if domain in cookie.domain:
# Note this is to fix the 431 error when the cookie is too large in Unbuntu with FireFox's cookes.sqlite
cookie_strings.append(cookie.name + '=' + cookie.value)
cookie_headers = {'Cookie': '; '.join(cookie_strings)}
req.headers.update(cookie_headers)

Expand Down